Implementing a Digital Twin of an Underground Utility Tunnel for Geospatial Feature Extraction Using a Multimodal Image Sensor
Abstract
:1. Introduction
2. Literature Review
2.1. Underground Facility Digital Twins and Structure Performance Analysis
- -
- -
- -
- -
2.2. Geospatial Feature Extraction
- -
- PCL (Point Cloud Library): This is a comprehensive, open-source library specifically designed for the processing of 3D point cloud data [25]. It provides many features and functions such as point filtering, surface normal or curvature estimation, disparate point cloud alignment, and shape recognition. It can support a variety of formats, making it suitable for various applications.
- -
- Open3D: This is another open-source library aimed at processing 3D data. It is known for its speed, and it provides easy-to-use pipelines for processing point clouds, meshes, and depth images [26]. Open3D also includes functions for 3D visualization, 3D reconstruction, and 3D registration (aligning different 3D data sets).
2.3. Finding Changes in 2D and 3D Spatial Objects
- -
- Segmentation: This process involves dividing an image into regions or segments that correspond to objects or their parts [27]. In 2D, this is achieved based on x and y pixel coordinates, disregarding depth (z) information.
- -
- Classification: After segmentation, color information in the form of red, green, and blue (RGB) values from each segment is used for classification [28]. Pixels or regions are grouped into categories or classes based on their RGB values. An alternative approach is using digital elevation model (DEM) data to assign z values to a 2D image, providing a measure of depth or elevation.
3. Methods
- -
- Step 1: Underground Utility Tunnel Digital Twin: Create a virtual replica (digital twin) of the underground utility tunnel to bridge the physical and digital worlds. This involves using sensor data, 3D modeling, and simulation software to create a detailed digital model for monitoring, planning, and a predictive analysis.
- -
- Step 2: Classification of Tunnel Database: Classify the data collected about the underground utility tunnels based on attributes like function, materials, location, and size. Also, categorize the tunnels as point, linear, or planar objects.
- -
- Step 3: Object Segmentation Algorithm: Apply object segmentation algorithms to the digital twin of the underground utility tunnel to identify and distinguish different elements within the model. This involves dividing the digital image into meaningful segments for an easier analysis, enabling tasks such as identifying utility lines, structural components, and sections of the tunnel to support decision making and predictive maintenance.
3.1. Constructions of the Digital Twin of an Underground Utility Tunnel Based on BIM-GIS
3.2. Composition of Digital-Twin-Based Spatial Object Extraction Algorithm
4. Results and Discussion
4.1. Underground Implementation of a Digital-Twin-Based Spatial Object Extraction Algorithm
4.1.1. Linear Spatial Object Extraction Process
4.1.2. Point Spatial Object Extraction Process
4.1.3. Planar Spatial Object Extraction Process
4.1.4. Supplementing the Algorithm via Octree Creation
4.2. Plan for Constructing and Renewing Digital Twin with Spatial Object Type
4.2.1. Linear Spatial Object Extraction Process
4.2.2. Spatial Object Update Plan
- Underground Tunnel Update: This pertains to the update of spatial information for disaster/facilities management. Shapes and properties are updated using data gathered with sensors, which are distinct from a precise update.
- Shape Update: This relates to long-term changes to an object/displacement size of 10 cm or more. Objects that have been moved, such as fire extinguishers, would be exceptions.
- Property Update: This involves updates that generate property information of objects, including information about the object itself, displacement, location value, and time.
- Event Range: This includes movements (point), deflections (line), and cracks (plane).
4.3. Image-Sensor-Based Spatial Object Extraction
4.3.1. Utility Tunnel Data Linear Object Extraction
4.3.2. Utility Tunnel Data Plane Object Analysis
4.3.3. Utility Tunnel Object Displacement Value Derivation
5. Conclusions
- The rationale behind executing real-time updates of 3D spatial information was discussed. This clarification distinguishes automatic updates from real-time updates that directly mirror object movements, preventing potential confusion.
- We described the categories of (near) real-time 3D spatial information updates, the information being updated in them, the update process, and how updates vary based on object types.
- We re-evaluated existing algorithms to enhance (near) real-time spatial information update algorithms for each 3D object type. In doing so, we also redefined the scenario process, data flow, and other essential algorithms to support the existing ones.
- The supplementary algorithm for extracting and updating 3D spatial objects was scrutinized and analyzed. Specifically, the algorithm for extracting point cloud data spatial objects was enhanced, and the algorithms for extracting feature points, spatial objects, and displacement information were examined and analyzed to propose a new integrated algorithm.
- We tested the developed algorithm using point cloud sample data. For this, an environment resembling a utility tunnel was set up, and LiDAR data were collected. The more precise the LiDAR data, the simpler the data analysis. Our results showed that object classification became significantly easier. Lastly, when deriving displacement data, we did not obtain accurate data because relative coordinate data were employed. However, precise data could be obtained through data processing.
- -
- Collecting sufficient data from fixed multimodal sensors: By securing and using multiple sensors that operate in different modalities (for instance, combining LiDAR with photogrammetric or radar sensors), you can gather richer, more comprehensive data. If these sensors are also fixed in position, it could help mitigate issues arising from shifts in sensor installation positions.
- -
- Performing repeated testing in diverse environments: This approach would help validate the effectiveness of the extraction algorithms across various conditions, enhancing their robustness and reliability. By repeatedly testing the system under different scenarios, you can refine the displacement detection information extraction process, making it more efficient and accurate.
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- El Marai, O.; Taleb, T.; Song, J. Roads infrastructure digital twin: A step toward smarter cities realization. IEEE Netw. 2020, 35, 136–143. [Google Scholar] [CrossRef]
- Fan, C.; Zhang, C.; Yahja, A.; Mostafavi, A. Disaster City Digital Twin: A vision for integrating artificial and human intelligence for disaster management. Int. J. Inf. Manag. 2021, 56, 102049. [Google Scholar]
- Singh, M.; Srivastava, R.; Fuenmayor, E.; Kuts, V.; Qiao, Y.; Murray, N.; Devine, D. Applications of digital twin across industries: A review. Appl. Sci. 2022, 12, 5727. [Google Scholar] [CrossRef]
- Lee, J.; Lee, Y.; Hong, C. Development of Geospatial Data Acquisition, Modeling, and Service Technology for Digital Twin Implementation of Underground Utility Tunnel. Appl. Sci. 2023, 13, 4343. [Google Scholar] [CrossRef]
- Park, S.; Hong, C.; Hwang, I.; Lee, J. Comparison of Single-Camera-Based Depth Estimation Technology for Digital Twin Model Synchronization of Underground Utility Tunnels. Appl. Sci. 2023, 13, 2106. [Google Scholar] [CrossRef]
- Marco, Z.; Giuseppe, C.P.; Giuseppe, T.; Marco, S. On the influence of shallow underground structures in the evaluation of the seismic signals. Ing. Sismica 2021, 38, 23–35. [Google Scholar]
- Li, X.; Tang, L.; Ling, J.; Chen, C.; Shen, Y.; Zhu, H. Digital-twin-enabled JIT design of rock tunnel: Methodology and application. Tunn. Undergr. Space Technol. 2023, 140, 105307. [Google Scholar]
- Zhou, Z.; Zhang, J.; Gong, C. Hybrid semantic segmentation for tunnel lining cracks based on Swin Transformer and convolutional neural network. Comput.-Aided Civ. Infrastruct. Eng. 2023, 1–20. [Google Scholar]
- Boje, C.; Guerriero, A.; Kubicki, S.; Rezgui, Y. Towards a semantic Construction Digital Twin: Directions for future research. Autom. Constr. 2020, 114, 103179. [Google Scholar] [CrossRef]
- Jones, D.; Snider, C.; Nassehi, A.; Yon, J.; Hicks, B. Characterising the Digital Twin: A systematic literature review. CIRP J. Manuf. Sci. Technol. 2020, 29, 36–52. [Google Scholar] [CrossRef]
- Agnusdei, G.P.; Elia, V.; Gnoni, M.G. Is digital twin technology supporting safety management? A bibliometric and systematic review. Appl. Sci. 2021, 11, 2767. [Google Scholar] [CrossRef]
- Kritzinger, W.; Karner, M.; Traar, G.; Henjes, J.; Sihn, W. Digital Twin in manufacturing: A categorical literature review and classification. Ifac-PapersOnline 2018, 51, 1016–1022. [Google Scholar] [CrossRef]
- Shao, G.; Helu, M. Framework for a digital twin in manufacturing: Scope and requirements. Manuf. Lett. 2020, 24, 105–107. [Google Scholar]
- Meierhofer, J.; Schweiger, L.; Lu, J.; Züst, S.; West, S.; Stoll, O.; Kiritsis, D. Digital twin-enabled decision support services in industrial ecosystems. Appl. Sci. 2021, 11, 11418. [Google Scholar] [CrossRef]
- Opoku, D.-G.J.; Perera, S.; Osei-Kyei, R.; Rashidi, M. Digital twin application in the construction industry: A literature review. J. Build. Eng. 2021, 40, 102726. [Google Scholar]
- Madubuike, O.C.; Anumba, C.J.; Khallaf, R. A review of digital twin applications in construction. J. Inf. Technol. Constr. 2022, 27, 145–172. [Google Scholar]
- Lee, D.; Lee, S. Digital twin for supply chain coordination in modular construction. Appl. Sci. 2021, 11, 5909. [Google Scholar] [CrossRef]
- Gao, Y.; Qian, S.; Li, Z.; Wang, P.; Wang, F.; He, Q. Digital twin and its application in transportation infrastructure. In Proceedings of the 2021 IEEE 1st International Conference on Digital Twins and Parallel Intelligence (DTPI), Beijing, China, 15 July–15 August 2021; pp. 298–301. [Google Scholar]
- Martínez-Gutiérrez, A.; Díez-González, J.; Ferrero-Guillén, R.; Verde, P.; Álvarez, R.; Perez, H. Digital twin for automatic transportation in industry 4.0. Sensors 2021, 21, 3344. [Google Scholar] [CrossRef]
- Pang, T.Y.; Pelaez Restrepo, J.D.; Cheng, C.-T.; Yasin, A.; Lim, H.; Miletic, M. Developing a digital twin and digital thread framework for an ‘Industry 4.0′Shipyard. Appl. Sci. 2021, 11, 1097. [Google Scholar] [CrossRef]
- Erol, T.; Mendi, A.F.; Doğan, D. The digital twin revolution in healthcare. In Proceedings of the 2020 4th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), Istanbul, Turkey, 22–24 October 2020; pp. 1–7. [Google Scholar]
- Liu, Y.; Zhang, L.; Yang, Y.; Zhou, L.; Ren, L.; Wang, F.; Liu, R.; Pang, Z.; Deen, M.J. A novel cloud-based framework for the elderly healthcare services using digital twin. IEEE Access 2019, 7, 49088–49101. [Google Scholar] [CrossRef]
- Xue, F.; Lu, W.; Chen, Z.; Webster, C.J. From LiDAR point cloud towards digital twin city: Clustering city objects based on Gestalt principles. ISPRS J. Photogramm. Remote Sens. 2020, 167, 418–431. [Google Scholar]
- Tavakolibasti, M.; Meszmer, P.; Böttger, G.; Kettelgerdes, M.; Elger, G.; Erdogan, H.; Seshaditya, A.; Wunderle, B. Thermo-mechanical-optical coupling within a digital twin development for automotive LiDAR. Microelectron. Reliab. 2023, 141, 114871. [Google Scholar] [CrossRef]
- Sommer, M.; Stjepandić, J.; Stobrawa, S.; von Soden, M. Automated generation of a digital twin of a manufacturing system by using scan and convolutional neural networks. In Transdisciplinary Engineering for Complex Socio-Technical Systems—Real-Life Applications; IOS Press: Amsterdam, The Netherlands, 2020; Volume 12, pp. 363–372. [Google Scholar]
- Choi, S.H.; Park, K.-B.; Roh, D.H.; Lee, J.Y.; Mohammed, M.; Ghasemi, Y.; Jeong, H. An integrated mixed reality system for safety-aware human-robot collaboration using deep learning and digital twin generation. Robot. Comput.-Integr. Manuf. 2022, 73, 102258. [Google Scholar]
- Zhang, J.; Fukuda, T.; Yabuki, N. Automatic generation of synthetic datasets from a city digital twin for use in the instance segmentation of building facades. J. Comput. Des. Eng. 2022, 9, 1737–1755. [Google Scholar]
- Kunze, P.; Rein, S.; Hemsendorf, M.; Ramspeck, K.; Demant, M. Learning an empirical digital twin from measurement images for a comprehensive quality inspection of solar cells. Sol. RRL 2022, 6, 2100483. [Google Scholar]
- Nica, E.; Popescu, G.H.; Poliak, M.; Kliestik, T.; Sabie, O.-M. Digital Twin Simulation Tools, Spatial Cognition Algorithms, and Multi-Sensor Fusion Technology in Sustainable Urban Governance Networks. Mathematics 2023, 11, 1981. [Google Scholar]
Major Category | Minor Category | Library | 3D Object Update | Property Information Update | |
---|---|---|---|---|---|
Point | Point Facilities | Fire extinguisher | ○ | Yes | Yes |
Light | ○ | Yes | Yes | ||
Sensor | ○ | Yes | Yes | ||
Sprinkler head | ○ | Yes | Yes | ||
Gate valve | ○ | Yes | Yes | ||
Air valve | ○ | Yes | Yes | ||
Notice sign | ○ | Yes | Yes | ||
Wire splitter | ○ | Yes | Yes | ||
Emergency exit sign | ○ | Yes | Yes | ||
Automatic fire extinguishing equipment | ○ | Yes | Yes | ||
Automatic fire detection equipment | ○ | Yes | Yes | ||
CCTV | ○ | Yes | Yes | ||
Flange | ○ | No | Yes | ||
Distribution panel | ○ | No | Yes | ||
Plumbing equipment | ○ | No | Yes | ||
··· | ··· | ··· | ··· | ||
Point Structures | Bracket | ○ | No | Yes | |
Support | ○ | No | Yes | ||
Branch | ○ | No | No | ||
Pipe protection | ○ | No | No | ||
··· | ··· | ··· | ··· | ||
Line | Line Facilities | Power line | ○ | No | Yes |
Water pipe | ○ | No | Yes | ||
Communications line | ○ | No | Yes | ||
··· | ··· | ··· | ··· | ||
Plane | Plane Structures | Wall | ○ | No | Yes |
Slab | ○ | No | Yes | ||
Floor | ○ | No | Yes | ||
Ventilation opening | ○ | No | Yes | ||
Stairs | ○ | No | Yes | ||
Entrance/Exit | ○ | No | Yes | ||
··· | ··· | ··· | ··· | ||
Other | Other | Random objects | X | Yes | No |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Lee, J.; Lee, Y.; Park, S.; Hong, C. Implementing a Digital Twin of an Underground Utility Tunnel for Geospatial Feature Extraction Using a Multimodal Image Sensor. Appl. Sci. 2023, 13, 9137. https://doi.org/10.3390/app13169137
Lee J, Lee Y, Park S, Hong C. Implementing a Digital Twin of an Underground Utility Tunnel for Geospatial Feature Extraction Using a Multimodal Image Sensor. Applied Sciences. 2023; 13(16):9137. https://doi.org/10.3390/app13169137
Chicago/Turabian StyleLee, Jaewook, Yonghwan Lee, Sangmi Park, and Changhee Hong. 2023. "Implementing a Digital Twin of an Underground Utility Tunnel for Geospatial Feature Extraction Using a Multimodal Image Sensor" Applied Sciences 13, no. 16: 9137. https://doi.org/10.3390/app13169137
APA StyleLee, J., Lee, Y., Park, S., & Hong, C. (2023). Implementing a Digital Twin of an Underground Utility Tunnel for Geospatial Feature Extraction Using a Multimodal Image Sensor. Applied Sciences, 13(16), 9137. https://doi.org/10.3390/app13169137