Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Real-Time 3D Volume Deformation and Visualization by Integrating NeRF, PBD, and Parallel Resampling

NeRF, PBD 및 병렬 리샘플링을 결합한 실시간 3D 볼륨 변형체 시각화

  • Sangmin Kwon (Division of Computer Engineering, Hansung University) ;
  • Sojin Jeon (Division of Computer Engineering, Hansung University) ;
  • Juni Park (Division of Computer Engineering, Hansung University) ;
  • Dasol Kim (Division of Computer Engineering, Hansung University) ;
  • Heewon Kye (Division of Computer Engineering, Hansung University)
  • Received : 2024.06.15
  • Accepted : 2024.07.05
  • Published : 2024.07.25
Download PDF
⟨ Previous Next ⟩

Abstract

Research combining deep learning-based models and physical simulations is making important advances in the medical field. This extracts the necessary information from medical image data and enables fast and accurate prediction of deformation of the skeleton and soft tissue based on physical laws. This study proposes a system that integrates Neural Radiance Fields (NeRF), Position-Based Dynamics (PBD), and Parallel Resampling to generate 3D volume data, and deform and visualize them in real-time. NeRF uses 2D images and camera coordinates to produce high-resolution 3D volume data, while PBD enables real-time deformation and interaction through physics-based simulation. Parallel Resampling improves rendering efficiency by dividing the volume into tetrahedral meshes and utilizing GPU parallel processing. This system renders the deformed volume data using ray casting, leveraging GPU parallel processing for fast real-time visualization. Experimental results show that this system can generate and deform 3D data without expensive equipment, demonstrating potential applications in engineering, education, and medicine.

딥러닝 기반 모델과 물리 시뮬레이션을 결합한 연구는 의료 분야에서 중요한 발전을 이루고 있다. 이는 의료영상 데이터에서 필요한 정보를 추출하고, 물리적 법칙을 기반으로 골격 및 연조직의 변형에 대한 빠르고 정확한 예측을 가능하게 한다. 본 연구는 신경 방사 필드(NeRF), 위치 기반 동역학(PBD), 병렬 리샘플링을 융합하여 3D 볼륨데이터를 쉽게 생성하고 실시간으로 변형 및 시각화하는 시스템을 제안한다. NeRF는 2D 이미지와 카메라 좌표 정보를 사용해 고해상도 3D 볼륨 데이터를 생성하며, PBD는 물리 기반 시뮬레이션으로 획득한 데이터에 대한 실시간 변형과 상호작용을 가능하게 한다. 병렬 리샘플링은 사면체 메쉬와 GPU 병렬 처리를 통해 렌더링 효율성을 높인다. 이 시스템은 광선투사방식으로 렌더링 되어 빠른 실시간 시각화를 제공하며, 비싼 장비 없이 간단하게 3D 데이터를 생성하고 변형할 수 있어 공학, 교육, 의료 등 다양한 분야에서의 활용 가능성을 보여준다.

Keywords

Acknowledgement

본 연구는 한성대학교 교내학술연구비 지원과제임

References

  1. M. Tonutti, G. Gras, and G. -Z. Yang, "A machine learning approach for real-time modelling of tissue deformation in image-guided neurosurgery," Artificial intelligence in medicine, vol. 80, pp. 39-47, 2017.  https://doi.org/10.1016/j.artmed.2017.07.004
  2. N. Lampen, D. Kim, X. Fang, X. Xu, T. Kuang, H. H. Deng, J. C. Barber, J. Gateno, J. Xia and P. Yan "Deep learning for biomechanical modeling of facial tissue deformation in orthognathic surgical planning," International journal of computer assisted radiology and surgery, vol. 17, no. 5, pp. 945-952, 
  3. H. J. Yoon, Y. J. Jeong, H. Kang, J. E. Jeong, and D.-Y. Kang, "Medical image analysis using artificial intelligence," Progress in Medical Physics, vol. 30, no. 2, pp. 49-58, 2019.  https://doi.org/10.14316/pmp.2019.30.2.49
  4. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436-444, 2015.  https://doi.org/10.1038/nature14539
  5. A. Nogales, A. J. Garcia-Tejedor, D. Monge, J. Vara, and C. Anton, "A Survey of Deep Learning Models in Medical Therapeutic Areas," Artificial Intelligence in Medicine. vol. 112, pp. 102020, 2021. 
  6. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, "Nerf: Representing scenes as neural radiance fields for view synthesis," Communications of the ACM, vol. 65, no. 1, pp. 99-106, 2021. 
  7. J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, "Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields," Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855-5864, 2021. 
  8. C. Sun, M. Sun, and H.-T. Chen, "Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5459-5469, 2022.
  9. A. Sodickson, P. F. Baeyens, K. P. Andriole, L. M. Prevedello, R. D. Nawfel, R. Hanson, and R. Khorasani, "Recurrent CT, cumulative radiation exposure, and associated radiation-induced cancer risks from CT of adults," Radiology, vol. 251, no. 1, pp. 175-184, 2009.  https://doi.org/10.1148/radiol.2511081296
  10. A. Corona-Figueroa, J. Frawley, S. Bond-Taylor, S. Bethapudi, H. P. Shum, and C. G. Wilcocks, "Mednerf: Medical neural radiance fields for reconstructing 3d-aware ct-projections from a single x-ray," 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society pp. 3843-3848, 
  11. Y. Fang, L. Mei, C. Li, Y. Liu, W. Wang, Z. Cui, and D. Shen, "Snaf: Sparse-view cbct reconstruction with neural attenuation fields," arXiv preprint arXiv:2211.17048, 
  12. M. Muller, B. Heidelberger, M. Hennix, and J. Ratcliff, "Position based dynamics," Journal of Visual Communication and Image Representation, vol. 18, no. 2, pp. 109-118, 2007.  https://doi.org/10.1016/j.jvcir.2007.01.005
  13. M. Macklin, M. Muller, N. Chentanez, and T.-Y. Kim, "Unified particle physics for real-time applications," ACM Transactions on Graphics (TOG), vol. 33, no. 4, pp. 1-12, 2014. 
  14. A. R. Aguilera, A. L. Salas, D. M. Perandres, and M. A. Otaduy, "A parallel resampling method for interactive deformation of volumetric models," Computers & Graphics, vol. 53, pp. 147-155, 2015.  https://doi.org/10.1016/j.cag.2015.10.002
  15. F. M. Miranda, and W. Celes, "Volume rendering of unstructured hexahedral meshes," The Visual Computer, vol. 28, pp. 1005-1014, 2012.  https://doi.org/10.1007/s00371-012-0742-8
  16. J. Gascon, J. M. Espadero, A. G. Perez, R. Torres, and M. A. Otaduy, "Fast deformation of volume data using tetrahedral mesh rasterization," Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 181-185, 2013. 
  17. J. Nickolls, I. Buck, M. Garland, and K. Skadron, "Scalable parallel programming with cuda: Is cuda the parallel programming model that application developers have been waiting for?," Queue, vol. 6, no. 2, pp. 40-53, 2008. 
  18. J. Seo, C. Park, S. Cho, and H. Kye, "GPU Parallelization Study of Position Based Dynamics for Medical Imaging," The Journal of Korea Institute of Next Generation Computing, vol. 19, no. 3, pp. 19-28, 2023. 
  19. J. L. Schonberger and J. -M. Frahm, "Structure-from-M otion Revisited," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp.4104-4113, 2016. 
  20. M. Macklin, M. Muller, and N. Chentanez, "XPBD: position-based simulation of compliant constrained dynamics," Proceedings of the 9th International Conference on Motion in Games, pp. 49-54, 2016. 
  21. J. Bender, M. Muller, and M. Macklin, "Position-based simulation methods in computer graphics," Eurographics (tutorials), pp. 8, 2015. 
  22. T. Kim and D. L. James, "Physics-based character skinning using multi-domain subspace deformations," Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 63-72, 2011. 
  23. M. Kazhdan, "Reconstruction of solid models from oriented point sets," Third Eurographics Symposium on Geometry Processing, pp. 73-es, 2005. 
  24. M. Kazhdan, M. Bolitho, and H. Hoppe, "Poisson surface reconstruction," Fourth Eurographics Symposium on Geometry Processing, vol. 7, no. 4, 2006. 
  25. D. P. Playne, and K. A. Hawick, "Data Parallel Three-Dimensional Cahn-Hilliard Field Equation Simulation on GPUs with CUDA," In Proceedings of PDPTA, vol. 9, pp. 104-110, 2009. 
  26. C. Park and H. Kye "Efficient Massive Computing for Deformable Volume Data Using Revised Parallel Resampling," Sensors, vol 22, no. 16, pp. 6276, 2022 
  27. I. Herrera, C. Buchart, I. Aguinaga, and D. Borro, "Study of a ray casting technique for the visualization of deformable volumes," IEEE Transactions on Visualization and Computer Graphics, vol. 20, no. 11, pp. 1555-1565, 2014.  https://doi.org/10.1109/TVCG.2014.2337332
  28. T. Muller, A. Evans, C. Schied, and A. Keller, "Instant neural graphics primitives with a multiresolution hash encoding," ACM Transactions on Graphics (TOG), vol. 41, no. 4, pp. 1-15, 2022.