Transforming Urban Façade Condition Assessments with Semantic Data Visualizations and 3D Spatial Layouts from BIMs ^†

Abstract

Safety inspection of building façades in urban settings is critical for public safety, as many incidents/accidents frequently occur due to falls from façades. This inspection is required for condition assessment of current states and their comparison to assessments conducted in a previous inspection cycle. The current practice of façade condition assessment relies on static, scattered, and textual depictions, which prevents inspectors from having a comprehensive view of façade condition and comparing it to the previous inspection cycle’s findings. Integrated visualization of spatial and semantic data and providing this data based on the preferences of the decision makers are proven to be effective in various decision domains. This study builds on previous research efforts on integrated visualization techniques to identify façade inspectors’ preferences in comprehending inspection findings. Through the design of low-fidelity visualization prototypes, this study first identifies highly frequently preferred visualization techniques by inspectors. Based on these, this study then quantifies the impact of these identified visualization preferences on accuracy and efficiency of decisions in relation to condition assessment of building façades though a set of high-fidelity prototypes and user studies. The results show that integrated visualization of façade conditions has the potential to bring the efficiency of capturing a holistic view of the façade conditions up to 65% and increase the accuracy of decisions up to 41%. The findings of this study directly benefit inspection companies and city agencies who can deploy visualization techniques that are impactful for façade condition assessment in order to holistically assess building façades.

1. Introduction

In twelve major U.S. cities, a mandatory façade inspection program aims to enhance public safety by requiring regular checks on older and taller buildings. Despite its intentions, debris-related incidents in cities such as Boston, Chicago, and New York City continue to pose serious safety risks, highlighting the urgent need for changing façade condition assessment practices [1,2,3]. Earlier work on this research direction highlighted major challenges in the path to effective façade inspection, one of which has been the lack of effective ways to document and review inspection findings [4,5,6].

In current façade safety inspection programs, inspectors are required to assess the integrity of building façades by comparing obtained conditions against those reported in previous inspection cycles. Essential information, such as the types of defects, their precise locations, the affected façade components, and recommended maintenance actions, must be extracted from these reports. However, the efficacy of this process is compromised by the current practice of using disparate documentation forms, including descriptive text, 2D elevation drawings, and numbered images. This format fails to offer a comprehensive view of the façade’s condition relative to its state in the previous cycle. Our shadowing work revealed that inspectors often struggle to navigate through multiple pages of a single report to piece together critical information, as the documentation does not consolidate all relevant data in one place nor provide effective spatial references. This inefficiency indicates a clear need for an improved documentation viewing system that enhances the accessibility and utility of façade inspection data.

To tackle the challenge of inefficient façade inspection assessments, our research vision builds on building information modeling (BIM) due to its hierarchical component representations, property sets, and object-oriented model structures, which facilitate efficient data storage and visualization. The research employed a taxonomy of visualization techniques to encode semantic information on spatial contexts [7], which includes: (a) blending, which modifies visualization attributes such as colors or patterns; (b) embedding, which adds symbols, text annotations, or charts without changing visualization attributes; and (c) multi-views, integrating multiple visualization forms in one interface. Of these, blending and embedding are suitable for encoding façade safety inspection data in spatial layouts, as inspectors need to visualize defect type and location, with defect type being categorical and its location being spatial data. Blending and embedding techniques are effective in displaying such data in 2D/3D [7]. In this vision, generating exterior enclosure models for existing buildings will not require extensive time or effort, as accurate dimensions can be obtained with laser scanning technology, which is widely used in practice today. A captured point cloud can then be used to generate an exterior enclosure model with a level of development (LOD) no greater than LOD 300.

The research method involved developing low-fidelity prototypes of a visualization platform that incorporated all applicable information visualization techniques across both 2D and 3D spatial representations of façades. These prototypes served to determine which visualization methods were preferred by inspectors for depicting façade conditions. Subsequently, a functional prototype was developed based on the preferred techniques to assess their effectiveness in enhancing inspectors’ efficiency and accuracy during façade condition evaluations. For the purposes of this study, efficiency was quantified as the time inspectors required to retrieve relevant information about façade conditions, including defect types, associated components, and overall safety assessments. Accuracy was defined as the proportion of correctly identified façade conditions relative to their verifiable ground truths. This approach allowed us to measure the impact of the visualization techniques on the decision regarding façade safety inspections.

2. Motivating Case Study and Overview of Current Practice of Reviewing Inspection Results

The authors shadowed experienced inspectors to understand the current practices of façade condition assessment [4]. We closely shadowed experienced inspectors to understand the prevalent practices of façade condition assessment, focusing on the ways data, images, and sketches are captured and compared with results from previous inspections [4]. A critical case study involved shadowing the inspection of an 80-year-old, eight-story nursing center, which was thoroughly inspected over eleven hours. This extensive observation allowed us to analyze how inspectors integrate current findings with past reports and the methods they employ.

This particular shadowed case revealed a significant challenge: inspectors struggled to obtain a holistic view of the building façade’s safety condition using current documentation practices. Typically, façade safety inspection records consist of annotations on 2D elevation drawings, separate descriptive texts, and multiple images cataloging identified defects [8]. Crucial information is scattered across these drawings, tables, and images, leading to a fragmented understanding of the façade’s condition, as shown in an example in Figure 1. For example, during the healthcare building inspection, to locate defects spatially on the building façade, the inspector read through more than forty pages of the prior cycle’s report to find the relevant texts and the summary table (Figure 1a). Then, the inspector had to cross-reference between the elevation views (Figure 1c) and key plans (Figure 1d) to pinpoint the exact locations of these defects with respect to the façade components they are associated with. Finally, the inspector checked the images captured in the previous cycle for each defect (Figure 1b). Remembering the locations and types of defects required, the inspector repeatedly consulted various report attachments to verify the specific conditions of the noted defects. This was a mental exercise for the inspector to keep all of this information in mind for all types of the defects reported.

The existing practice of cross-referencing to track defects, group them, and detect changes between inspection cycles is inefficient and time-consuming. There is a need to obtain a holistic but efficient understanding of façade conditions throughout inspection cycles. This paper reports on the impact of integrated visualization (i.e., semantic visualization over spatial contexts) as well as its design and evaluation to improve efficiency and accuracy of decisions regarding façade condition assessments.

3. Related Previous Work

This study builds on previous research on (1) visualization support in the architecture, engineering, construction, and facilities management (AEC/FM) domain, and (2) taxonomies on the visualization of semantic information in spatial contexts. This section provides a synthesis of these related previous studies.

3.1. Visualization Support in the AEC/FM Domain

Information visualization is known to help humans interact with data effectively [9]. Building information modeling (BIM) supports information visualization, sharing, and management in different stages and various visualization techniques have been applied to visualize semantic information within models to support decisions regarding the whole lifecycle of a facility within the AEC/FM domain [10].

Studies for visualization support in the design phase focus on clash avoidance [11,12], planning and simulation practices [13], and design analysis for complex engineered systems [14]. Researchers leveraged augmented reality (AR) [12] and open work frameworks [11] to minimize conflicts with early collaborations. Several tools were developed for clash detection with 3D models [15,16]. For collaborative 4D planning and simulation, researchers proposed adapted views (e.g., textured 3D, line charts, and Gantt charts) based on user roles (e.g., supervisor, subcontractor), industry practice, and identified information needs [13]. Additional visual analytics tools that support various views, such as tree maps, node networks, and matrix visualization, have also been proposed to support efficient design analysis of complex engineered systems [14].

Most of the BIM-based visualization studies focus on the construction phase to improve task scheduling [17,18,19], performance analysis [20], progress monitoring [18,21], and quality control [22]. Data visualization with charts and diagrams was developed to provide easily interpretable support for construction progress monitoring [17], comparing different versions of schedules using Gantt charts and network diagrams [19]. Data visualization was developed to bring efficiency in tracking, modifying, and updating time and cost-based data [20]. Leveraging techniques such as GIS data [21], images [18], point cloud data [22], and computer vision-based approaches, models were mainly the central data repository to provide visualization support.

Visualization of information with BIM for the facilities management (FM) has also been heavily researched [10]. Visualization in the FM phase of construction projects facilitates planning and execution of maintenance decisions [7,23,24,25] and monitoring of the structural health of constructed facilities [26,27,28,29]. User studies found that visualization-based interfaces could improve the accuracy and efficiency of facility operators’ decisions [25]. Combining visualization techniques, such as color coding and chart overlay, with augmented reality [24] and virtual reality [7] technology, researchers developed information visualization platforms to support troubleshooting and monitoring of building systems. For structural health monitoring (SHM) purposes, researchers discussed the importance of an informative display that facilitates decision-making [28], highlighted the use of BIM for dynamic visualization and management of SHM data [27], and offered how dynamic visualization of structural performance can improve maintenance and risk management [26]. Later, with the wide adaptation of Internet of Things (IoT) technology, such IoT-based protocols have been introduced to support expansive network architectures; for example, for improving structural health monitoring [30].

Previous efforts underscored the impact of visualization and related technologies in enhancing decisions related to construction projects at various life-cycle phases. Despite these advancements, a gap remains in applying visualization techniques to façade inspections. This paper fills this gap by designing and evaluating integrated visualization techniques (i.e., semantic and spatial visualization) to improve inspectors’ holistic assessment of façade conditions.

3.2. Taxonomies on Visualization of Semantic Information in Spatial Contexts

Visualization in the AEC-FM domain is broadly classified into scientific and information visualization. Scientific visualization deals with tangible aspects, such as spatial (e.g., location, topological) and geometrical (e.g., dimensions, shape) details of building components, and mainly utilizes 2D/3D representations of products [31]. In contrast, information visualization handles semantic data about products/processes (e.g., time series data), including also the abstract (e.g., parent child relationships) and intangible concepts (e.g., voids represented for windows embedded within walls). Common techniques include floor plans, site [22] and geospatial maps, and schematic diagrams for systems such as HVAC, employing block diagrams to depict configurations and relationships among components [25]. An overview of these existing taxonomies is provided in

Table 1. Taxonomies for scientific and information visualization in AEC-FM.

Visualization Type	Definition	Data Type Focused in Visualization	Example Techniques
Scientific	Visualization that deals with tangible and visible aspects of objects to be visualized, such as spatial (e.g., location, topological) and geometrical (e.g., dimensions, shape) details of building components.	Physical objects Location Material Geometry Topology of physical objects	2D 3D Plan views Elevation views Section views Detail views
Information	Visualization that handles semantic data about products/processes (e.g., time series data), including also the abstract (e.g., parent child relationships) and intangible concepts (e.g., voids represented for windows embedded within walls).	Semantic information of physical objects Generalization relationships Association relationships Composition/aggregation relationships Embeddings	Annotated floor plans Geospatial maps, Schematic diagrams Tree maps scatterplot matrices
Hybrid	Visualization that allows mixed use of scientific and information visualization.	Blend of semantic and spatial data types	Symbol–metaphor Text overlay Chart overlay Color coding Pattern coding Animations

.

Semantic information visualization in AEC-FM can be categorized into two groups: one that represents large, complex datasets without a tangible reference, using standalone visualizations such as tree maps and scatterplot matrices; and another that relates to tangible objects, integrating spatial information with sensor data for a comprehensive view. Techniques for semantic visualization in spatial contexts include blending, embedding, and multi-viewing—each encoding different types of data, such as categorical, scalar, descriptive, and temporal. Categorical data represent distinct categories or groups that data can belong to. These types of data are qualitative and are often used to label or classify items. Examples of categorical data in construction projects are material types (e.g., concrete, steel, wood, and glass), project phase (i.e., planning, design, construction, and maintenance), and worker roles (e.g., labors and carpenters). Scalar data represent numerical values that measure magnitude without direction. These values can be counted or measured and are often used for quantifying and comparing. Examples of scalar data are numerical values representing cost (e.g., material, labor, and equipment cost), measurement dimensions (e.g., length, width, height, and area of construction components), and quantities of units of materials or items. Descriptive data provide detailed descriptions and attributes of items or phenomena. These types of data are qualitative and offer insights into the characteristics and properties, such as project specifications, inspection reports, and material properties. Temporal data represent information related to time, including dates, durations, and sequences of events. These types of data are crucial for tracking and managing time-dependent processes. Typical temporal data can be project schedules, work progress, and historical data of maintenance activities. These visualizations support comparative analysis across components and are vital for effective facility management and troubleshooting, requiring a shared property among the elements being visualized [25].

In this paper, we leveraged building information modeling (BIM), information, and spatial visualization taxonomies. We utilized the component hierarchy representations, associated data, and object-oriented nature of models to efficiently store and visualize data. The views developed in this study are based on the taxonomy of information visualization techniques on spatial layouts, which consist of three groups of visualization techniques capable of encoding semantic information over spatial contexts as shown in

Table 2. Visualization techniques to encode semantic information in spatial contexts [7].

Visualization Techniques		Supports Encoding Semantic Information Types	Supports Single/Multiple Object Visualization
Multi-view		Various types	Multiple objects
Embedding	Symbol/metaphor	Categorical; scalar	Multiple objects
	Text overlay	Categorical; scalar; descriptive	Single objects; multiple objects
	Chart overlay	Temporal	Single objects; multiple objects
Blending	Color/pattern coding	Categorical; scalar	Multiple objects
	4D animation	Spatial–temporal	Multiple objects

[7]: (1) Blending involves utilizing the visual characteristics (e.g., volume, area) of a component, including color coding or pattern coding. An example will be coloring an entire volume of columns with a given color to show a certain defect type. (2) Embedding refers to adding symbols, charts overlay, or text annotation, without altering the visual characteristics of a component. An example will be using a specific symbol to be attached to all defective columns to show the defect type and its locations; and (3) multi-views, which combine multiple visual representations into a single interface, such as visualization of 4D simulation together with Gantt chart and project task information. Symbols and metaphors visually represent ideas, concepts, or objects simply or abstractly. Symbols are often universal icons, while metaphors use imagery to convey deeper meanings. For example, a hard hat icon symbolizes safety. Text overlay places text on visuals such as images or diagrams to add information, such as labeling rooms or dimensions on a floor plan. Chart overlay superimposes charts on visuals, such as maps or diagrams, to provide quantitative data and insights, such as a Gantt chart showing task progress and delays in construction projects. Color/pattern coding uses different colors and patterns to distinguish elements or categories, aiding quick identification. For instance, colors on a site plan indicate zones (e.g., residential, commercial), and patterns represent material types. Additionally, 4D animation combines 3D visualizations with time to create dynamic simulations, showing how a building’s construction progresses over time, including task sequencing and equipment movement.

4. Research Method

The overarching goal of this research is to streamline the current façade safety inspection process with integrated visualization of inspection data within spatial contexts and evaluate the impact of each on the efficiency and accuracy of inspectors’ decisions regarding inspection. The research method includes various steps, as overviewed here:

(1) Defining information requirements to support façade condition assessment: The information requirements for comprehensive façade safety inspections are detailed in Shi [23]. The categories of information that need to be tracked in façade inspection include (1) façade components and their static information, including their location information, geometric information, as well as topological and whole/part relationships; and (2) defect types, associated data (e.g., size, direction, and angle), and their location information.

Table 3. Required façade condition information and applicable visualization techniques.

Information Category	Information Subcategory	Data Type	Applies to Multiple Components?	Applicable Visualization Techniques
Façade component information	Component type	Categorical	Yes	2D/3D
	Location	Spatial	Yes	2D/3D
Defect information	Defect type	Categorical	Yes	Text overlay; color coding; symbol annotation
	Location	Spatial	Yes	2D/3D

summarizes these information items that are essential to review and assess façade conditions. A comparative analysis of visualization taxonomy and applicable information requirements revealed that not all visualization techniques are applicable and are eliminated.

Multi-view, which enables visualizing multiple categories of information simultaneously (such as visualizing control relationships and sensor data of HVAC systems together), is excluded from further evaluation because the required façade condition information are categorical data that embedding and blending are sufficient to visualize. Given Table 1, 4D animations suitable for visualizing spatial-temporal data, multi-views displaying diverse information simultaneously (such as visualizing control relationship and sensor data of HVAC systems together), and chart overlays that display time-series data (e.g., historic data of energy assumption) are not applicable for visualizing the inspection findings on defects since defect types are categorical information, not numerical, complex, or temporal data. Therefore, these visualization techniques that were not applicable are excluded from the rest of the evaluations and only the options given in Figure 2 are carried to the next step.

(2) Developing low-fidelity prototypes and capturing inspectors’ preferences: Low-fidelity prototypes are simple, early-stage representations of tools, interfaces, or processes used in facade inspection. These prototypes are designed to quickly and inexpensively visualize ideas, gather feedback, and refine concepts based on the input from facade inspectors. We developed them by utilizing the taxonomy of visualization techniques for integrated visualization. In total, we designed six integrated visualization views in low-fidelity format, corresponding to each applicable visualization technique. An example is provided in Figure 3. Examples of low-fidelity prototypes: (a) 2D view with color coding and (b) 3D view with color coding using the 2D and 3D spatial view and color-coding information visualization technique. Using these prototypes, we worked with expert inspectors to identify frequently preferred views and use them in the design of high-fidelity prototypes to be deployed in structured user studies.

A total of 12 experienced professional façade inspectors participated in the tests with both low and high-fidelity prototypes. The participants are façade inspectors in consulting companies with their experiences varying from brick masonry renovation to roof replacement. The participants have varying levels of experience, ranging from one year for a recent recruit to 20 years, with an average of nine years (

Table 4. Demographics of inspectors participating in the study.

ID	Experience (Years)	Company Type, Job Title	Professional Experience
1	20	Preservation and restoration of buildings and infrastructure, Associate Principal	Exterior restoration
2	8	Building Envelope Consultants, Senior Designer	Façade design/engineering
3	2	Exterior restoration, Façade Architect	Façade design
4	1	Building enclosure consulting, Façade Architect	Façade design
5	1	Engineering and architectural consulting, Façade Architect	Building envelope investigation
6	2	Full-service engineering, architectural consulting services, Façade Architect	Exterior cladding remediation and replacement projects
7	2	Forensic consulting, construction management, digital intelligence, Façade Architect	Façade design/engineering
8	21	Building enclosure consulting, Vice President Field Operations	Rope access inspection for high-rise buildings
9	13	Forensic consulting, construction management, digital intelligence, Sr. Building Envelope Consultant	Exterior restoration, waterproofing, roofing, and leak remediation
10	7	Civil engineering, building science and related consulting, Assistant Project Manager	Forensic investigations, low-energy building design
11	13	Architecture and Planning, Director of Facade Compliance	FISP quality assurance and control, exterior restoration
12	18	Restoration architect, Senior Project Manager	FISP quality assurance and control

). First, the research team introduced the goal of the study and explained the provided visualization options to participants. Then, for each visualization option, inspectors were instructed to evaluate the value of each given integrated visualization for assessing façade conditions on a Likert scale ranging from 1 (strongly dislike) to 5 (strongly like). At the end, participants were instructed to rank all visualization options from the most preferred to the least.

User feedback collected in this step was evaluated with two metrics: (1) The average Likert score given to each visualization option. The average score for each visualization option is calculated by taking the sum of all inspectors’ ratings on the Likert scale and dividing it by the total number of inspectors, providing a mean value that represents the overall preference (likeability) for the corresponding visualization option; and (2) the ranking of each visualization option, defined as the total number of inspectors choosing a given visualization option as their favorite one.

(3) Implementing high-fidelity prototypes and conducting user studies: High-fidelity prototypes are detailed, interactive representations of a product or system that closely resemble the final version in terms of functionality, design, and user interface. They are used to simulate the user experience as accurately as possible, allowing for comprehensive testing and validation of design choices in functional interfaces. These prototypes have been created based on the feedback gathered from low-fidelity prototype evaluations and used to conduct user studies to evaluate the impact of integrated visualization in improving inspectors’ assessment of inspection conditions.

This user study aims to evaluate how visualization can impact the efficiency and accuracy of decisions made by inspectors while investigating a new building’s inspection history. In this work, the hypothesis is that if inspectors can generate views from a 3D model of the façade using different visualization techniques (i.e., color coding and symbol annotations), they will have a better comprehension of the inspection findings of that façade. Therefore, each inspector was asked to complete three tasks. The tasks were designed for inspectors to look for information by using the functional prototype and a set of reports/sketches/images, which simulates the current practice. Each task simulated a different façade condition scenario with a different combination of defect type, component type, and overall condition of the façade. Participants used the functional prototype and the current means/methods. The time it took for inspectors to complete each task using the prototype and the current means/methods was captured to quantify the efficiency of their performance for both cases. The answers provided by inspectors were used to quantify the accuracy of their performance in understanding the façade condition.

Each inspector participating in the study was instructed to assume that they just joined a façade inspection project and would be reviewing the findings of the last inspection cycle to obtain a holistic understanding of the façade condition captured in that cycle. Three tasks were given to each inspector during the study for each visualization option and the current practice. An overview of these tasks is provided in

Table 5. Tasks assigned to inspectors in the user studies.

Task ID	Task Description	Defect/Component Types Used	Utilized Resources *
1	Ask inspectors to find and mark all instances and locations of a specific defect on the elevation of the façade	Leakage	Current practice resources
		Spall	Implemented prototype (2D + CC)
		Loose	Implemented prototype (2D + SA)
		Missing	Implemented prototype (3D + CC)
		Crack	Implemented prototype (3D + SA)
2	Ask inspectors to find and mark all identify all the defect types associated with a specific component type	Parapet	Current practice resources
		Window	Implemented prototype (2D + CC)
		Coping	Implemented prototype (2D + SA)
		Railing	Implemented prototype (3D + CC)
		4th floor exterior wall	Implemented prototype (3D + SA)
3	Ask inspector to will identify all the defect types on the whole façade of the building	Leakage, spall, loose, missing	Current practice resources
		Spall, loose, missing, crack	Implemented prototype (2D + CC)
		Leakage, spall, missing, crack	Implemented prototype (2D + SA)
		Leakage, spall, loose, crack	Implemented prototype (3D + CC)
		Leakage, loose, missing, crack	Implemented prototype (3D + SA)

* CC: Color coding; SA: symbol annotation.

. Task 1 asked inspectors to find and mark all instances and locations of a specific defect on the elevation of the façade (e.g., “Please find and mark all instances and locations of leakage defect on the east elevation”). Task 2 instructed inspectors to identify all defect types associated with a specific component type (e.g., “Please identify all defect types observed on the copings above 9th floor”). Task 3 asked inspectors to check the report and find all defect types identified from the last inspection cycle (e.g., “Please provide me the number of defect types identified along with their names.”). The defect types simulated are crack, loose, missing, rust/corrosion, spall, leakage, and deflection, which are the top defect types regarding the number of associated component types, as well as the frequencies of observance in inspected buildings.

Components for selected defects were also identified from the historical façade inspection report analysis results.

Table 6. Defect types and associated components included in the developed high-fidelity prototype.

Defect Type	Associated Component
Crack	window, parapet, sill, coping, roof, lintel
Loose	window, sill, appurtenance, railing, parapet, coping
Missing	window, parapet, roof, sealant, railing, coping
Rust/corrosion	lintel, window, railing, roof, bulkhead, parapet
Spall	window, parapet, sill, lintel, coping, bulkhead
Leakage	roof, exterior wall, appurtenance, window, parapet, bulkhead

presents the defect types and their associated façade components simulated in this high-fidelity prototype. When the participants were assigned with the tasks, the research team changed the defect and component types and elevations in each visualization option to eliminate learning bias throughout the experiment. The component types were carefully selected to ensure that each type had the same number of defect types and instances to be identified per task. The number of instances for each defect type is carefully kept the same to eliminate the possible impact on the time it takes to complete each defect-related task. The order of performing the tasks using the current practice vs. the functional prototypes was randomly altered so that the order did not have an impact on the metrics being measured.

Resources provided to inspectors that replicate the current practice of façade condition assessment include (1) a façade safety inspection report, which contains description paragraphs and a summary table for façade conditions, (2) a plot plan, which represents the layout of a building and its surroundings street names, (3) an annotated elevation drawing, which is an elevation drawing marked with symbols and textual descriptions, (4) a key plan, which shows a small-scale layout of the building at a certain floor with window IDs and close-up inspection marks, and (5) images of related defects used in the tasks. Inspectors were also given the functional prototype and had the capability to generate four different views in 2D/3D for each of the three tasks. The views included a 2D view with color coding and symbol annotations, as well as a 3D view with color coding and symbol annotation for given tasks.

The participants were randomly divided into two groups. For the participants in the first group, the research team first provided the resources to replicate the current practice; and for the second group of participants, the research team first provided the developed high-fidelity prototype. After completing the assigned tasks, the research team asked inspectors to pick the top two ranked visualization techniques that they found most helpful in completing the tasks.

Inspectors’ performances were evaluated using both quantitative and qualitative metrics while using various documentation/visualization methods for reviewing the condition of the façade. The quantitative metrics are the accuracy and efficiency of the completed tasks. Accuracy refers to whether the participants correctly identified the number of defects/components over all simulated defects/components in a given task using either the prototype or the current means/methods. It is measured by comparing the number of correctly identified defect types/locations to a set of ground truth answers for each assigned task using mean absolute error (MAE) (Equation (1)). Efficiency refers to the amount of time participants spend on completing each task using either proposed or current visualization methods, measured in seconds. The time that each participant took to complete the assigned three tasks with all visualization options and the current means was recorded and statistically analyzed.

$$Accuracy \left(\%\right)={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\displaystyle \frac{\# of correctly identified responses }{\# of correct answers}}$$

(1)

5. Implementation and Results

This section provides details of the implementation of the prototypes and discusses the results of user studies.

5.1. Low-Fidelity Prototypes and Inspectors’ Preferences

Six low-fidelity prototypes were created for inspectors to use in inspection scenarios. Prototypes were developed using a BIM of a testbed building for the combinations of visualization techniques. These were used to quickly assess different visualization techniques and preferences of inspectors. A real building and its façade model along with a synthetic set of inspection findings were used to generate these visualization prototypes. Locations of defects are displayed directly on the 2D and 3D views in relation to the component they are observed on. Every defect type is associated with a distinct color, symbol, or text annotation that is appended to the component where the defect instance is identified. The six low-fidelity prototypes shown in Figure 4 visualize six selected defect types and their corresponding locations in both 2D (Figure 4a–c) and 3D (Figure 4d–f) views. Three applicable visualization techniques were implemented with each view: color coding (a and d), symbol annotation (b and e), and text overlay (c and f). These were used to capture visualization preferences of expert inspectors.

5.2. Results of Inspectors’ Visualization Preferences for Visualizing Inspection Data

The average scores (i.e., mean value of participants’ rankings on 5-point Likert scale for each visualization option) received for each low-fidelity prototype are shown in Figure 5a. Color coding received the highest preference as the visualization technique both in 2D and 3D views. Symbol annotation over 2D and 3D views ranked as the third and fourth preferred options. Currently, the 2D view with symbol annotation is one of the most common visualization options in the façade safety inspection practice to track inspection findings. The findings show no clear correlation between the inspectors’ visualization preferences and their expertise levels. Participants with less than 5 years of experience and over 15 years of experience selected visualization techniques with both 2D and 3D views as the most preferable visualization, hence seniority/juniority in the practice did not show a clear distinction in preferred spatial visualization techniques.

Figure 5b displays the number of participants that picked each visualization option as their favorite to comprehend the façade inspection results. The results align with conclusion drawn from the average score shown in Figure 5a. Results from this user study with low-fidelity prototypes reveal that color coding (i.e., CC) in 3D and 2D views, as well as symbol annotations (i.e., SA) in 2D and 3D views are options to be evaluated further for their impact on inspectors’ assessments of façade conditions with functional prototypes.

5.3. High-Fidelity Prototypes and Results of User Studies

Efficiency of inspectors in assessing façade conditions using integrated visualization: Based on the user feedback, except the 2D/3D text annotation, we implemented the remaining integrated visualizations as functional prototypes. The prototype takes industry foundation classes (IFC) of a façade along with the façade conditions captured for that façade in an inspection cycle as the input and visualizes the façade conditions with the identified four visualization options (i.e., 2D/3D symbol annotation, 2D/3D color coding). We chose to simulate the top six most frequent defect types because they are not only the most mentioned in previous inspection reports, but are also associated with the widest variety of façade elements (Table 5). For each task, the inspectors were instructed to identify six instances of a defect type (Task 1), or six different defect types associated with one or more components (Task 2), or to list defect types that were simulated on the whole façade (Task 3). The final user interface displays are provided in Figure 6.

Efficiency analysis across tasks: Results of the user study with high-fidelity prototypes are provided in Figure 7. Based on the results, there is a significant reduction in time for accessing façade condition information and assessing the condition across all tasks. By comparing the average completion time using integrated visualization (i.e., four views in Figure 6) and the completion time achieved with the current means/methods, one can obtain the percentage reduction in average completion time for each task. The time used for Task 1 (i.e., finding information about a specific defect) was reduced by 34.8%, for Task 2 (i.e., finding information about a specific component) by 52.3%, and for Task 3 (i.e., finding all defect types identified on the entire façade) by 47.5% as compared to the current practice. These significant reductions in time across all tasks, reaching more than half the time it takes in the current practice, highlight the impact of the integrated visualization prototypes compared to the traditional method of searching through documents and associated sketches/images.

Figure 7 also clearly demonstrates that the 3D model with symbol annotation (3D+SA) provides the greatest reduction in completion time across all tasks, making it the most efficient visualization method. This approach achieved more than a 60% reduction in completion time across all tasks involving finding information about specific defects, components, and all defect types on the given façade. The 3D + SA visualization is particularly effective because it combines the spatial depth of 3D models with the clarity of symbol annotations, allowing inspectors to intuitively and quickly identify and locate defects and components. The consistency of performance is evident from the small range in the min, max, and median times, indicating low variability among participants. This suggests that the 3D + SA visualization provides a uniformly effective user experience, reducing cognitive load and aligning well with natural human perception of spatial environments. The interactive capabilities of 3D models enable inspectors to navigate, zoom, and rotate the façade efficiently, accessing information from various perspectives without being overwhelmed by visual clutter.

In contrast, other visualization methods, such as 2D + CC or 2D + SA, showed longer bars in the performance range, indicating higher variability and less consistent performance. This could be due to the limitations of 2D representations, which lack the perspective evaluations, making it harder for inspectors to process and locate information quickly. Overall, the 3D + SA method stands out as the most efficient and reliable visualization technique for façade inspections, providing significant time savings (~60%) consistent across different tasks and variations across inspectors’ expertise. This task- and expertise-independent efficiency makes it the recommended choice for improving condition assessment of façades.

Efficiency analysis per task: When the efficiency metric is analyzed with respect to individual tasks, Task 3, which involves providing an overview of defect types, shows the greatest reduction in completion time across all visualization options. When asking inspectors to obtain an overview of the façade condition and provide associated defect types, integrated views significantly reduce the time required because they include a legend that facilitates quick reference. Inspectors can easily compare the conditions illustrated in the 2D and 3D views to the legend, allowing them to swiftly identify and categorize defect types. In contrast, the current practice requires inspectors to read through description paragraphs in the report and text annotations of elevation drawings, which is time-consuming. Task 3, designed to study the holistic understanding of the façade, benefits the most from integrated views because it leverages the combined strengths of 2D and 3D visualizations, along with clear symbol annotations, to present a comprehensive and easily interpretable overview. Additionally, the variations in completion time for Task 3 are much less compared to other tasks, indicating consistent performance among inspectors.

When comparing across the three tasks, Task 1 took the longest time across all visualization options. This is because Task 1 required inspectors to mark the exact locations of a specific defect type on the provided elevation drawing, necessitating a thorough and careful examination of the entire visualization to ensure all defect locations were accurately identified. Due to the requirement for accuracy in this assignment, inspectors had to meticulously examine the visualizations, resulting in a naturally longer duration. In contrast, Task 2 required inspectors to check all defects on a specific façade component. The scope of inspection for Task 2 is fixed and relatively small compared to Task 1, allowing inspectors to complete it more quickly. It is crucial to note that across all tasks, the current practice consistently results in the longest completion times (i.e., average 4.5 min to complete all three tasks as compared to average 2.4 min to complete all three tasks with the provided integrated visualizations), which underpins the potential advantages of employing integrated visualization of façade conditions. Beyond the 3D+SA option, the 2D+SA technique demonstrated significant efficiency, showing a 34.8% reduction for Task 1 and a 58.5% reduction for Task 2. For Task 3, the 2D+CC technique provided a substantial reduction of 47.6%. These findings highlight the potential of 3D+SA and 2D+SA visualization techniques in significantly improving task completion times, thereby enhancing the overall inspection process. This study underscores the importance of selecting appropriate visualization tools to optimize defect detection and inspection efficiency. Overall, the developed integrated visualizations can significantly reduce task completion time, thus potentially increasing inspectors’ efficiency in understanding façade conditions.

Accuracy of inspectors in assessing façade conditions using integrated visualization: The accuracy of inspectors in assessing façade conditions using the developed integrated views was compared to their performance using the current practice. The findings are presented in

Table 7. Accuracy of façade conditions identified by inspectors in visualization options.

	Current Practice	2D + CC	2D + SA	3D + CC	3D + SA
Task 1	85%	89%	94%	85%	100%
Task 2	95%	100%	89%	94%	91%
Task 3	87%	100%	97%	100%	96%
Average	89%	96.3%	93.3%	93%	95.7%

.

Accuracy analysis across tasks: Results show variations across tasks, however, clearly showing that any form of integrated visualization is better in helping inspectors to achieve a correct assessment of the façade condition (i.e., average accuracy for three tasks ranging from 93% to 96.3% across four integrated visualizations) as compared to the current practice (average 89% across tasks). The current practice yields the lowest average accuracy at 89%, indicating that the current practice offers certain help in quickly assessing a given façade and its condition. This shows that the inconsistent annotation and scattered documentation in current practice has room for improvement. When there is a clear holistic view of the façade to be obtained (Task 3), or semantic data to be checked (Task 1), then the current practice is significantly at a disadvantage. All visualization alternatives are at a min by 10% across tasks. Additionally, when spatial information is to be found (Task 2), inspectors’ performance does not change drastically regardless of current practice or integrated visualization. For Task 2, inspectors’ performances stay consistent with 2D and 3D views, achieving the best accuracy with 2D + CC visualization.

When comparing the visualizations with color coding (CC) and symbol annotations (SA), both in 2D and 3D formats, there is a marked improvement in accuracy as compared to the current practice. The use of 2D + color coding provides an advantage in increasing the accuracy of inspectors in pinpointing façade conditions (analyzed for specific defects, components, or to obtain a holistic understanding of the façade condition), with 7.3% improvement. This finding is expected because color coding simplifies the process of identifying and distinguishing between different defect types visually. Colors are inherently easier to track and recognize quickly compared to symbols, especially in a complex visual field. Each defect type can be assigned a unique color, allowing inspectors to scan the façade efficiently and identify issues without needing to interpret specific symbols. The cognitive load is reduced as colors provide an immediate visual cue, leading to faster and more accurate identification. The intuitive nature of color coding makes it a more effective tool for quick and precise defect identification in façade inspections. The 2D performance in color coding over 3D with color coding was better since 2D visualizations are often simpler and less visually complex, making it easier for inspectors to quickly identify and track specific colors associated with defect types. The flat, straightforward nature of 2D images allows for faster visual processing without the need to interpret depth and perspective, which can sometimes complicate defect identification in 3D views. This was reversed in the symbol annotation, where 3D+ symbol annotation on the average across tasks was higher (~6.7%) in improving the accuracy of inspectors’ assessments as compared to 2D+ symbol annotation (~4.3%). This could be attributed to the fact that 3D models provide a more realistic and spatially accurate representation of the façade, allowing inspectors to better understand the spatial relationships and context of defects. The added depth and perspective in 3D views help in precisely locating and identifying symbols, which might be harder to differentiate in a flat 2D representation. The same reason applies to the fact that 3D + SA achieved the highest accuracy for Task 1, where defect instances and locations are required, at 100%.

Accuracy analysis per task: When specific tasks and their requirements are analyzed, Task 1 has the highest results in improving accuracy of inspectors in symbol annotation (~15%) as compared to color coding (4%), regardless of how I visualize the corresponding spatial context in 2D vs. 3D. In Task 1, semantic information was to be tracked (i.e., defect). Therefore, symbol annotation is much better than all other options. Decoding a defect (semantic data) with symbols is found to be easier for inspectors to follow, as it aligns with their current practice. For Task 2, which involves tracking spatial information, such as specific component, inspectors need to quickly and accurately identify and differentiate the façade elements. The clear, distinct colors in 2D views make this process more efficient, providing an immediate visual cue that simplifies the identification process, reducing cognitive load and minimizing the chances of errors. Task 3, however, integrated visualization and improved the accuracy compared to current practice by 10% to 15%. This result shows that integrated visualization boosts a holistic understanding of façade conditions, enabling inspectors to quickly grasp the overall state of the structure. For Task 3, color coding, regardless of whether it is in 2D views or 3D views, outperforms symbol annotation because it allows inspectors to easily and immediately identify and categorize different defect types through distinct colors. This visual clarity helps in quickly forming a comprehensive picture of the façade’s condition without the need to interpret symbols, which can be less intuitive and slower to process. The use of color coding facilitates faster recognition and differentiation of defects, making it a more effective method for providing an overall assessment of the façade conditions in Task 3. These findings suggest that the choice of visualization technique should be tailored to the specific nature of the task, emphasizing either spatial clarity or semantic detail as required. Regardless, it should be noted that inspectors take longer to achieve the same accuracy levels in the current practice compared to using visual forms. There is a significant reduction (~46.7%) in time, indicating that visualizations not only improve accuracy, but also efficiency.

6. Conclusions

This study provides details of understanding façade inspectors’ preferences within combinations of visualization techniques that overlay semantic information over spatial contexts. It also examines the impact of integrated visualizations as information support to improve inspectors’ performances in understanding façade conditions. Building upon the previously defined taxonomy of information and spatial visualization techniques, the authors identified applicable ones for façade inspection data (i.e., defect type and defect location) visualization, and conducted a comprehensive work that consists of four steps: (1) developing low-fidelity prototypes to explore various views by utilizing the taxonomy of information visualization techniques, (2) gathering inspectors’ feedback to identify a subset of visualization options, (3) implementing high-fidelity prototypes based on inspectors’ feedback, and (4) conducting user studies to evaluate functional prototypes.

Results with experienced inspectors show that the integrated views result in an average completion time reduction for all tasks of 44.8% and an improvement in accuracy (i.e., ~10%) in inspectors’ performances. The 3D view with symbol annotation is the visualization that achieves most completion time reduction (i.e., 46.7%) for completing all assigned tasks. Additionally, 2D + color coding provides a significant advantage in increasing accuracy of inspectors in pinpointing façade conditions with around 7% improvement. The work clearly demonstrates that visualization enhances inspectors’ ability to quickly gain a comprehensive understanding of a given façade. However, the reported accuracies in identifying condition data are based on the time it takes for inspectors to understand the façade conditions. Although the accuracies are not significantly different, it is important to note that inspectors, on average, spend more time (~46.7%) comprehending the conditions using the current methods.

The contributions of this paper are: (1) identification of applicable visualization techniques for façade safety inspection purposes; (2) identification of inspectors’ preferences over the visualization techniques; and (3) quantification of the different visualization techniques’ impact on inspectors’ performance when reviewing and understanding façade conditions. The effective integrated visualization techniques identified in this research can be applied to obtain fast and accurate assessment of façade conditions, particularly for buildings that have previous inspection findings in earlier inspection cycles. Given that inspection companies can change over these inspection cycles, it will be effective to use these integrated visualization methods to understand the conditions reported in previous cycles by another company. One of the limitations of this study is the size of the participant pool (currently twelve participants), which needs to expanded to include a larger group of inspectors to improve the generalizability of the results. Future work can expand the participant pool and would allow for more comprehensive statistical tests to validate the reliability and replicability of the results, ensuring that the highlighted visualization methods are genuinely the best options for all inspectors. Another limitation is the evaluation of the effect of the level of understanding and acceptance of BIM technology by inspectors on their selection of visualization options. Extended work on this study can incorporate their proficiency in the technology use in practice and further evaluate the results based on this factor.