KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Generating 3D Digital Twins of Real Indoor Spaces based on Real-World Point Cloud Data

  • Wonseop Shin (Graduated School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University) ;
  • Jaeseok Yoo (Nextchip) ;
  • Bumsoo Kim (Department of Applied Art and Technology, Chung-Ang University) ;
  • Yonghoon Jung (Department of Applied Art and Technology, Chung-Ang University) ;
  • Muhammad Sajjad (Digital Image Processing Laboratory, Department of Computer Science, Islamia College University Peshawar) ;
  • Youngsup Park (INNOSIMULATION CO., LTD) ;
  • Sanghyun Seo (Department of Applied Art and Technology, Chung-Ang University)
  • Received : 2024.03.28
  • Accepted : 2024.06.03
  • Published : 2024.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

The construction of virtual indoor spaces is crucial for the development of metaverses, virtual production, and other 3D content domains. Traditional methods for creating these spaces are often cost-prohibitive and labor-intensive. To address these challenges, we present a pipeline for generating digital twins of real indoor environments from RGB-D camera-scanned data. Our pipeline synergizes space structure estimation, 3D object detection, and the inpainting of missing areas, utilizing deep learning technologies to automate the creation process. Specifically, we apply deep learning models for object recognition and area inpainting, significantly enhancing the accuracy and efficiency of virtual space construction. Our approach minimizes manual labor and reduces costs, paving the way for the creation of metaverse spaces that closely mimic real-world environments. Experimental results demonstrate the effectiveness of our deep learning applications in overcoming traditional obstacles in digital twin creation, offering high-fidelity digital replicas of indoor spaces. This advancement opens for immersive and realistic virtual content creation, showcasing the potential of deep learning in the field of virtual space construction.

Keywords

Acknowledgement

This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00970, Development of multi-image-based 3D virtual space and dynamic object reconstruction technology for manufacturing site support, 50%) and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No.2022R1A2C1004657, 50%).

References

  1. C.W. Chu, J.Y. Park, H.W. Kim, J.C. Park, S.J. Lim and B.K. Koo, "Recent Trends of 3D Reconstruction Technology," Electronics and Telecommunications Research Institute on Electronics and Telecommunications Trends, vol.22, no.4, Aug. 2007.
  2. G. Pintore, C. Mura, F. Ganovelli, L. Fuentes-Perez, R. Pajarola, and E. Gobbetti, "State-of-theart in Automatic 3D Reconstruction of Structured Indoor Environments," Computer Graphics Forum, vol.39, no.2, pp.667-699, Jul. 2020.
  3. C. R. Qi, O. Litany, K. He, and L. Guibas, "Deep Hough Voting for 3D Object Detection in Point Clouds," in Proc. of 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp.9276-9285, 2019.
  4. T. Czerniawski, B. Sankaran, M. Nahangi, C. Haas and F. Leite, "6D DBSCAN-based segmentation of building point clouds for planar object classification," Automation in Construction, vol.88, pp.44-58, Apr. 2018.
  5. R. Qi Charles, H. Su, M. Kaichun, and L. J. Guibas, "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation," in Proc. of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.77-85, 2017.
  6. C. R. Qi, L. Yi, H. Su, and L. J. Guibas, "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space," in Proc. of NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp.5105-5114, Dec. 2017.
  7. L. Cui, G. Zhang, and J. Wang, "Hole Repairing Algorithm for 3D Point Cloud Model of Symmetrical Objects Grasped by the Manipulator," Sensors, vol.21, no.22, Nov. 2021.
  8. L. P. Tchapmi, V. Kosaraju, H. Rezatofighi, I. Reid, and S. Savarese, "TopNet: Structural Point Cloud Decoder," in Proc. of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.383-392, 2019.
  9. Z. Huang, Y. Yu, J. Xu, F. Ni and X. Le, "PF-Net: Point Fractal Network for 3D Point Cloud Completion," in Proc. of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.7659-7667, 2020.
  10. R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, A. Fitzgibbon, "KinectFusion: Real-time dense surface mapping and tracking," in Proc. of 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pp.127-136, 2011.
  11. S. Choi, Q.-Y. Zhou and V. Koltun, "Robust reconstruction of indoor scenes," in Proc. of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.5556-5565, 2015.
  12. V. Lempitsky, A. Vedaldi and D. Ulyanov, "Deep Image Prior," in Proc. of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.9446-9454, 2018.
  13. E. Rublee, V. Rabaud, K. Konolige and G. Bradski, "ORB: An efficient alternative to SIFT or SURF," in Proc. of 2011 International Conference on Computer Vision, pp.2564-2571, 2011.
  14. D. Nister, "An efficient solution to the five-point relative pose problem," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.26, no.6, pp.756-770, Jun. 2004.
  15. J. Park, Q.-Y. Zhou and V. Koltun, "Colored Point Cloud Registration Revisited," in Proc. of 2017 IEEE International Conference on Computer Vision (ICCV), pp.143-152, 2017.
  16. C.-Y. Wang, A. Bochkovskiy and H.-Y. M. Liao, "YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors," in Proc. of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.7464-7475, 2023.
  17. Dawson-Haggerty et al., Trimesh, 2019. https://github.com/mikedh/trimesh 
  18. D. G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," International Journal of Computer Vision, vol.60, pp.91-110, Nov. 2004.
  19. C. Harris, and M. Stephens, "A Combined Corner and Edge Detector," in Proc. of the Alvey Vision Conference, pp.147-151, 1988.