A Digital 4D Information System on the World Scale: Research Challenges, Approaches, and Preliminary Results

Abstract

Numerous digital media repositories have been set up during recent decades, each containing plenty of data about historic cityscapes. In contrast, digital 3D reconstructions of no longer extant cityscapes have so far almost solely been performed for specific cases and via manual modelling techniques. Within the Jena4D research group, we are investigating and developing methods and technologies for transferring historical media and their contextual information into 4D models. The next step will be to automate this process and extend it to the world scale. Our team are working on different facets of that endeavor: retrieving images, text, and object data from public sources; crowdsourcing and collecting qualitative data from contests and pupil labs; processing historical plans and photographs to extract spatial data; location-based mobile visualization; and collection-browsing interfaces. This article is intended to highlight (1) current challenges, (2) the state of the art, (3) our approach to 4D modelling and visualization on the world scale, and (4) results from testing demo cases in Dresden, Jena, and Amsterdam.

1. Introduction

Imagine you are exploring the historic center of a city with its impressive town houses, churches, and monuments. What if you could just use your mobile device to find out about the historic buildings around you, with detailed visual information about how they were built and the story behind them, making history come alive before your eyes? Photographs, texts, and plans are essential sources for historical research [1,2,3]. Numerous archives and repositories have been set up and nowadays they contain large-scale digital media collections. Within the Jena4D research group, we are investigating and developing methods and technologies for transferring historical media and their contextual information into a 4D—3D spatial and temporal scaled—model to support research and education on urban history. Content is made accessible as a location-dependent, virtual reality, 4D browser application for mobile devices. In previous articles, we presented the research agenda [4] and technological venues [5,6]. The aim of this article is to examine (1) current challenges; (2) the state of the art; (3) our approach to data collection, 4D modelling, and visualization on the world scale; and (4) results from testing within demo cases in Dresden, Jena and Amsterdam.

Against this background, this article is proposed to provide a comprehensive picture of the methods and techniques and their utilization for 4D world visualization. In our work, we apply a highly multifaceted approach to collect, process, and visualize data. The first part of this article provides a comprehensive overview of the state of the art in related fields. In the second part, we present a set of demonstrations and approaches we have conducted to assess and setup components of a 4D information system.

1.1. Motivation

Concerning the use of 4D information systems for cultural heritage, two general modes are discernible: browsing, which a self-directed search of historical information and sources [7], and location- or context-related information that is shared in the course of the presentation of the culture. With our software applications, we strive to serve two main usage scenarios: to supply city visitors with location-based mobile information and to enable scholars to browse media collections via a 4D browser application.

1.1.1. Cultural Tourism as the Application Scenario

Virtual historic cityscapes are employed in various scenarios [8,9,10,11], for example, to teach history and heritage in informal settings such as museum experiences, serious games, or television broadcasts [12,13,14,15,16,17,18,19,20,21,22]. For example, interactive applications for city exploration [23,24,25] allow virtual visits and remote spatial learning [26], guide visitors through the city [25,27,28], provide access to additional information, and allow users to gain a virtual view of temporal changes, historical spaces, buildings and monuments, or covered parts [9,28,29,30,31,32,33]. Extended reality (XR) includes different levels of augmentation of reality with computer-generated content. This is mostly visual content, ranging from augmenting real-world views with information or graphical elements (augmented reality, AR) to fully computer-generated virtual worlds (virtual reality, VR) [34]. Several application scenarios are specifically relevant to cultural heritage (see recent works [8,9,31]). A frequently discussed scenario is the use of XR for 4D city exploration [23,24]. Various sub-scenarios can be supported by XR applications [25]:

• Before visiting the place of interest, XR applications can be used to obtain inspiration or an image of the place [35] or plan a visit [25].
• During a visit, XR applications can help guide visitors through the city [25], offer virtual city tours [27,28], provide information about the city’s history and also about amenities (e.g., restaurants) [36], allow users to obtain a virtual view of temporal changes, historical spaces, buildings, and monuments [9,29,30,31,32], and provide a visual impression of hidden or covered parts [28,33].
• Following a visit, XR applications can assist users in recalling tours and visits, or provide access to places that visitors have not been able to visit [25].

1.1.2. Browsing Collections

Virtual research environments (VREs) are web-based systems that provide a virtual working environment for researchers by including various tools for data processing, analysis, or comparison [37]. For cultural heritage research, a large number of VREs are available, serving the specific demands of researchers in fields like archaeology [38] or architectural history [39]. Depending on the user group, there are different requirements; e.g., history research requires the comparability and contextualization of sources [40,41,42,43]. Furthermore, a transparent relationship between source and representation is essential [44,45]. Finally, visualization should ideally allow the identification of abstract characteristics such as ideas and systems, breaks, or deviations [46,47]. As stated by Beaudoin, the most critical challenge of these repositories is the accessibility of high-quality content [48]. Even if portals such as Europeana [49] make different types of cultural heritage data available via a single user interface, they provide abstract views rather than enabling spatio-temporal contextualization. In this context, 3D and 4D interfaces are used to structure knowledge and information. (a) Digital inventories, such as the spatial book (“Raumbuch”) approach [50,51] in archaeological excavation and similar archaeological excavations and similar approaches for monument documentation [52], focus on the spatial organization of digital data. (b) Digital information spaces, such as the digital twin in manufacturing [53], extend the inventory through fully digital simulation workflows but focus on contemporary data. (c) Four-dimensional models, e.g., city models, add a temporal layer to organize the temporal layer and to organize the data in a 4D inventory, e.g., [54]. Finally, the Metaverse approach tries to include the previously mentioned facets [55].

1.2. The Vision

Our vision is to develop a 4D model and visualization on the world scale. Usage scenarios are (a) virtual visits and on-site assistance, and also (b) browsing through large-scale data collections. The frontend features of the proposed interface are:

• To enable visual 4D impressions and access to further information (e.g., Wikipedia articles about landmarks).
• To work on mobile and desktop devices to enable both location-based and remote access to information.
• To function in a browser rather than a native app, since the mentioned application scenarios target occasional use on different devices [56].

The backend features of the proposed interface are:

• The world scale, created via an automated pipeline based on historical photographs and map data retrieved from various large-scale data sources.
• Tools that operate on application layers with minimal data infrastructure.
• The ability to retrieve information on the fly from multiple open data endpoints.

2. State of the Art

2.1. Data Collection

2.1.1. Citizen Science and Crowdsourcing

The creation of user-generated content is strongly supported by the availability of mobile phones and open-source 3D modeling tools. In terms of the level of participation, most citizen science projects use crowdsourcing to involve “non-scientists to help to analyze or collect data as part of a researcher-led project” ([57], p. 259). Examples include collecting and processing images [58] and crowd-based creation of 3D models [59,60]. Co-design “involves citizens into the research process from its beginnings, or the stimulus for the research project originates from the citizens” ([61], p. 4). Although most prominent in text- or image-based research in humanities [62], co-design is frequently used for 3D content and experience design for museums [63,64] or (serious) history games (e.g., [65]). Besides the challenges of participatory processes such as user activation and management, task definition, and quality control [66], citizen science tasks in humanities must handle complex, non-standardized, and knowledge-intensive tasks and therefore are hard to assess with regard to the scientific quality of processes and outcomes [67,68]. Finally, citizen science approaches have found their way into research and teaching in various university disciplines in recent years [69], including digital heritage (e.g., [70]). In this context, knowledge transfer to citizen scientists creates particular didactic challenges [71]. This is not least because the target or interested group for cultural heritage projects are often not digital natives and has to be addressed via other channels [72]. Knowledge transfer via digital citizen science activities nevertheless appears to be a promising approach, as public involvement in generating knowledge increases the willingness to engage with cultural heritage.

2.1.2. Data Retrieval

During the last two decades, numerous digital image archives containing vast numbers of photographs have been set up [7,73]. These comprise collections of user-generated contemporary photographs, historic photo collections and image databases with geographic coverage, e.g., Street View. For the 3D world, recently, large-scale datasets, such as Objaverse, including 10.2 million 3D models [74], and ShapeNet, including 50 k 3D models [75], and repositories such as Sketchfab, hosting several hundreds of thousands of heritage items [76], have been compiled. As an overlapping area, there are several automated 3D model creation processes that utilize extant imagery [77,78,79,80]. A major task is the provision of sufficient metadata to spatialize and temporalize this material [81].

2.1.3. Teaching Digital Competencies via Heritage

Three-dimensional heritage has become subject of teaching digital literacy and skills, e.g., on modelling techniques or VR technologies, while the historical object is used as a training example. This closely relates to the concept of data literacy comprising data collection, exploration, management, analysis, and visualization skills [82] (see also [83]). Moreover, digital 3D reconstruction techniques have been employed in various educational settings, focusing on the process of the reconstruction of cultural heritage objects (e.g., [84,85]). Despite these activities, there is still no wide consensus on establishing a specific digital visual humanities education paradigm, and larger studies on teaching digital methods in visual humanities are still needed [84].

2.1.4. Summary: Retrieval Challenges

• Despite a large amount of digital and digitized data, a major issue is the findability and missing information about spatial and temporal properties (e.g., the viewport).
• Crowdsourced data collections are particularly well established, but challenging with regard to engaging users on a large scale.
• Teaching digital skills via heritage is frequently used but a consensus is lacking on educational paradigms and methods.

2.2. Four-Dimensional Modelling

The main criticism of interpretative 3D reconstruction of no longer extant objects is that—due to the limitations and specifics of historical sources—source interpretation and the modelling process via expert tools rely heavily on tacit knowledge [86,87]. Thus, digital reconstructions are primarily created by multidisciplinary teams involving specialist modelers and historians [88]. In addition, due to the high level of effort, reconstruction is primarily used only for single objects. Despite various large-scale projects on 3D modelling in cultural heritage (recent overview: [89]), which primarily focused on 3D digitization from contemporary survey data [90,91,92,93], 3D/4D reconstruction from historical imagery still faces challenges. The main currently unsolved challenges in computer visualization based on photographs and vedutas are (1) identifying matching historical images from the multitude of digitized documents available and (2) calculating 4D information as a prerequisite for automated model generation [6,94]. In what follows, relevant technological approaches are highlighted.

2.2.1. Human-Driven 3D Reconstruction

In a digital reconstruction of no longer extant architecture, humans use computer software to create models. The 3D model is generated manually using interactive software tools. Human-driven digital reconstructions mostly utilize standardized software, which originates from different domains like geo information systems (GISs), computer graphics (CGI), computer-aided design (CAD) or building information modelling (BIM) [95,96], each with their own standards. All approaches recommend specific tools and workflows and offer specific benefits, whether this is information on object volumes (BIM), a highly realistic appearance of surfaces (CGI), accurate dimensions (CAD) or large-scale geo-referenced information (GISs). In all cases, human-driven 3D reconstructions are created within a highly versatile (but labor-intensive) and highly experience-based process.

2.2.2. Algebraic Approaches

The algebraic analysis of groups of images to calculate spatial information has a long history. Due to advances in both the quality and availability of camera hardware and image analysis software [90], most digitized models are now produced using photogrammetry. A well-established method is the 3D modelling of architectural heritage from current native digital photographs or video content using photogrammetric algorithms. The development of various feature matching methods [97,98] and their integration into the structure-from-motion (SfM) workflow has led to their widespread use [77,99,100,101,102,103]. Today, a wide variety of image spatialization algorithms are available [104,105,106], and photogrammetry is used in various fields such as landscape surveying [107], underwater archaeology [108], and monument recording [109]. While current algorithms achieve remarkable quality for natively digital (sequential) images, they often fail with unordered historical images due to the high quality and number of input images required [100,110,111]. As a result, current approaches often achieve a low quality or fail to process historical non-native digital imagery due to sparse sampling, incompleteness, missing metadata and a problematic radiometric quality [98]. These are not only technical but fundamental barriers, and despite much research, current photogrammetry dealing with historical images [94,112], such as re-photo approaches [113,114], still requires a lot of manual processing. Initial attempts to advance photogrammetry from historical images using video [115] or—invented by our group—architectural-feature-based photogrammetry [116] and a combination of algebraic and machine learning approaches [117] seem promising avenues for further automation, but algebraic processes still require a lot of prior optimization and multiple images of similar views.

2.2.3. Machine Learning and Hybrid Methods

Traditional algebraic approaches, such as in classical photogrammetry, employ equations, e.g., to detect, describe and match geometric features in images [97]. In contrast, machine learning is based on a statistical model developed via training data (or using self- and cross-attention layers in transformers), e.g., to detect features [118]. Current evolutions in computer vision are linked to the renaissance of machine learning since the 2010s [119], driven by the development of convolutional neural networks (CNNs, e.g., [120]). Machine learning approaches are currently mainly researched and employed for image and 3D point cloud analytics in cultural heritage (recent overview: [121]), but also increasingly serve 3D model creation tasks. There, machine-learning-based technologies are currently primarily used for particular tasks to preselect imagery [122,123], in semantic segmentation to classify parts of images [124,125,126] and to recognize and identify specific objects [126,127,128,129]. Traditional machine-learning-based technologies do require large-scale training data [121,126,127,128] and therefore are mainly capable of recognizing visually distinctive and well-documented landmark buildings [4]. These approaches usually fail to deal with less distinctive architecture, such as houses of a similar style, or fail when few images are available. Even using more advanced machine learning approaches or by combining different algorithms such as DELF and SuperGlue [130] only allows the realization of prototypic scenarios [6,131,132]. Another approach bypasses the modelling stage to generate visualizations directly from imagery [127,133,134], e.g., by transforming or assembling image content (recent image generators like DALL-E [135]). Recent approaches include neural radiance fields (NeRFs) [136,137], which predict shifting spatial perspectives even from single images [138], and can predict 3D geometries [139]. Generative adversarial networks (GANs) are a combination of proposal and evaluation components of machine learning. They are frequently employed as approximative techniques in 3D modelling, e.g., for single photo digitization [140], completion of incomplete 3D digitized models [141,142] and photo-based reconstructions [143]. Regarding transparency, most current machine learning approaches work within black box settings [121,144]. Although some newer machine-learning-based approaches are promising for processing small samples, they fail to process non-digital (historical) imagery with robustness and by design work on predictions; in addition, machine learning is limited by being grounded in historical original sources.

2.2.4. Structure Recognition from Plan Data

Although building footprints are still mainly manually extracted from historical maps, various AI-based approaches support this task. CNNs and—more recently—transformer approaches are used for the segmentation of historical maps [145,146,147,148,149]. Another approach to automatically generating 3D/4D models comprises building footprint recognition and parametric modelling. Footprint recognition via semantic segmentation for aerial/satellite imagery [150,151,152,153] or from current cadastral data [154] and for contemporary photography [155] has been frequently researched. One issue in boundary detection workflows is overlapping building boundaries and texts. Consequently, many approaches combine text recognition and boundary delineation [156,157,158,159] to trace building footprints.

Although current classifiers achieve good results for specific map styles and ages, they are not very transferable, so structure recognition has not reached a sufficient quality yet to be transferable without prior manual retraining of the detection system.

2.2.5. Generative Modelling

Generative modelling creates a 3D geometry iteratively based on pre-programmed rules [160]. In comparison to traditional modelling, which requires defining every single structure manually, this reduces effort and simplifies the modelling process by predefining the structural elements of objects and enables altering them on a parameter level (e.g., the properties and number of windows of a façade). Various projects use generative modelling for the ease of creating large-scale heritage structures such as complex 4D city models [161,162,163,164]. Another generative modelling technique structures the manual modelling process into single steps and translates them into digital-ontology-based workflows [165]. In our case, the combination of generative modelling rulesets with recognized footprints from historical cadaster data is of high relevance to ease model creation. This is currently under investigation in various projects [157,166,167].

2.2.6. Time

Time and non-linear temporal changes are the main elements of history. Current 3D/4D reconstructions still focus mostly on specific timestamps. This is challenging, since it requires multiple sources for these reconstructions, often taken at very different times and with each source a singular document of the represented state. Beside the issue of inter- or extrapolating sources of different times to gain a coherent historical view, dating of sources is challenging. Historical imagery is still primarily dated via metadata captured at recording or amended at later points. Where metadata are not available or certain, change detection can be applied to image series, e.g., to assess if undated images show corresponding states of construction to the dated ones. Current algorithmic change detection focuses on homogenous quality images, such as time series of satellite images [168,169,170] or aerial photos [171,172]. Approaches for heterogeneous photographs can deal with large-scale changes but are limited with subtle changes (overviews: [173,174,175]). Other change detection approaches work with 3D geometries (overview: [176]) or segmentation- and feature-based comparisons between different images to identify changes in architectural features [98,130]. The aim is to detect subtle changes in heterogenous historical imagery over time.

2.2.7. Transparency and Explainable Artificial Intelligence

Since algebraic approaches are reproducible, machine learning approaches are still primarily applied within black box settings with non-transparent decision making [121,144]. Consequently, a key research focus is explainable AI [177], or how to standardize and open AI-based classification processes. Unlike in, for instance, medical AI [178], an optimized heuristic interpretation is not sufficient for historical sources and their singularity [121]. Current approaches to employ AI in 3D cultural heritage applications validate their results only with examples [166]. Taking the singularity of history into consideration, there is a need to establish full-scale cross-validation of AI-based predictions of historical situations, e.g., by cross-validating mixed-methods or human-in-the-loop approaches.

2.2.8. Summary: 3D/4D Reconstruction Challenges

• Three/four-dimensional modelling processes have to deal with non-linear historical spatial situations and must gain information from time-varying singular historical sources of heterogeneous quality.
• Most approaches to automate the 3D/4D modelling process are optimized for specific purposes and therefore unable to cope with sparse samples and to detect differences with small variations.

Consequently, current 3D/4D reconstructions of past buildings are almost exclusively performed in manual workflows and—except in specific cases—fail with automation.

2.3. Visualization

2.3.1. Visualization Technologies

The created virtual 3D or 4D models are presented as dynamic and interactive visualizations, denominated extended reality (XR). XR technologies comprise a wide scope of approaches [179] between real and virtual [34]. This ranges from digital-enriched real environments—augmented realities (ARs) or augmented virtuality (AV)—to virtual reality (VR) as fully computer-generated visualizations. According to Russo [180], a blend between real and virtual elements can include adding information that does not exist, or can be subtractive by hiding or deleting parts of the real world. Despite the hype about glasses-based XR, 3D models and visualscapes are mostly accessed via mobile and desktop screens [181]. Various viewers, e.g., Smithsonian3D, 3DHOP, Potree, and DFG 3D Viewer (Overviews: [182,183,184]), are available and used in multiple projects. Other tools come from the gaming industry (Unreal and Unity engines) and are frequently used to visualize heritage content, while Google Earth or Apple Maps are world viewers but mostly lack the 4D history dimension. Some projects already use 4D viewers for city data, e.g., for Venice [158], Nicosia [185], and Trento [166]. To sum up, most open 3D visualization frameworks deal with single objects or specific scenarios (e.g., 3D specific buildings). Since they rarely include temporal changes, world viewers cannot yet visualize time-variant 3D architectural and city data on a large scale.

2.3.2. Interaction and Motivational Design

Visual appearance, user interaction, and presentation with 3D content pose design variables. In interaction design, XR applications enable multiple freedom degrees and requirements, such as for device motion interpretation [186,187]. For the acceptance of applications, their perceived usefulness is important, which depends on visual representation and interaction [188]. Layouts for user interactions are perspective-dependent and dynamic and require coherence [189,190]. Although the representation of time-independent data in interaction patterns has been investigated in several contexts [191,192], there are no fully validated strategies of how to manage this for time-dependent 4D data. Especially at sites and in cultural institutions, digital 3D applications are embedded in and connected to physical spaces. There are several recommendations for the design of linked physical and virtual spaces, such as in museums [193,194] and at heritage sites [195,196]. Recent evolutions comprise multi-user story-based virtual experiences to enrich physical museum visits by augmented experiences [197]. Motivational design includes linking content so users can easily consume or be motivated to follow it. As an example, gamification, using “elements of games that do not give rise to entire games” ([198], p. 2), is different from playful design that contains no specific rules or goals, and from serious games, which are defined as full-fledged games dedicated to non-entertainment purposes [198]. Storytelling is the use of fictional or non-fictional narratives to present a subject [199]. Psychological [200,201,202] and educational studies [203] have demonstrated that narratives can positively motivate by engaging the audience and making a subject more immersed, and therefore support learning by reducing the cognitive load. Storytelling is also widely used to present heritage content digitally [204] and support heritage education [205,206,207,208,209].

2.3.3. Visual Design and Perception of 3D/4D Content

Visual properties of 3D/4D visualizations heavily rely on device capabilities, e.g., screen properties (2D pixel monitors in smartphones, tablets, and desktops; 3D displays such as holographic displays, holograms, and 3D stereo displays; VR/AR headsets), interaction capabilities (input via mouse vs. touch vs. hand tracking; sensors) and usage scenarios (e.g., location-based vs. remote; mobile vs. desktop) [9,180]. Regarding design, influencing parameters include the level of detail (LoD), which needs to meet both the task and hardware requirements [210]. Detailed visualizations of historical reconstructions are advantageous for imaginability [211]. On the other hand, they are suspected of distracting scholars from their research questions [212] or causing cognitive overload [213]. There is a long-standing debate about the visualization strategies for 3D/4D reconstructions that are appropriate for historic architecture. The main positions include visual styling—to achieve realistic visualizations that are easy to understand and impressive for viewers—or schematic depictions showing hypotheses and schemes [210,214,215]. Visual style is frequently discussed regarding its fit to scholarly recommendations and as a potential distraction for viewers [44]. Since the majority of 3D/4D reconstructions still are aimed to be highly immersive and realistic visualizations [216], the full scope comprises a large variety of photorealistic and non-photorealistic styles [217,218,219]. Much research has been performed on the visualization of different degrees of certainty [214,220,221,222]. Current approaches can be roughly categorized into enrichment of representations by explanatory elements [223] and adaptation of representation quality, e.g., LoD or visual styling [210,214,215,224,225,226,227]. Scaling has been frequently assessed as an important parameter for perceiving architecture [210,228]. Visual acuity is the ability to distinguish details and is mainly influenced by the distance to a virtual or physical object [229,230,231,232]. Perspective depiction and perception include the effect of different fields of view [232,233,234,235]. Lighting refers to the shading of specific parts of an object [236] and is of high relevance for visual comparisons and the realism of virtual visualizations [237,238,239,240,241,242]. Color is also highly relevant for perception—in the case of historical objects, it ranges from realistic coloring and scales to code parameters or support, visually distinguishing model parts [243,244,245,246,247]. Methodologies for investigating the perception of digital visualizations are available but not comprehensively used yet to assess 3D/4D reconstructions. The traditional method is empirical user studies in experimental settings (e.g., [248]). As an extension of this paradigm, several data-driven user observation methods, e.g., eye tracking, are used to verify areas of attention (e.g., [210]). Three-dimensional XR spaces add the methodological challenge of mapping what the user sees, which can be achieved, e.g., via viewshed calculations in fully 3D spaces [232,249,250]. Despite much research, the main criticism is that studies of viewer effects are primarily descriptive rather than proven by user tests [230,251,252].

The quality of perception is influenced by visual parameters, such as size and shading (c.f., Gestalt theory [253], cognition psychology [254,255]); cultural settings including visual semiotics [256]; engagement [257]; and psychological processing such as visuospatial reasoning [211,258,259] or visual research [256,260,261]. User expertise with visualizing specific content is another parameter [262,263,264]: “visuality” [265], or, more precisely, “visual competence” [266] or “visual literacy” [267,268]. Architectural psychology distinguishes between two key factors influencing the perception of architecture: form structure, which includes elements such as color, form, spatial arrangement, movement, and depth [269], and form content [270], which includes dimensions such as cultural meanings, habits, and shock or novelty for the viewer [271]. Increased familiarity with specific content increases the ability to focus on specific aspects and visually recognize and understand even more abstracted content [272,273,274,275]. Although these factors are assumed to influence research processes in architectural history studies, it is currently unknown to what extent form structure and content affect the quality of perception and research.

2.3.4. Summary: 3D/4D Visualization Challenges

• Current visualization technologies lack the capability for large-scale 4D architectural and city visualizations.
• It is rarely empirically investigated what visual qualities are required to enable suitable interactive 3D/4D visualizations of past architecture in specific scenarios.

Within the constraints set by modelling, epistemics, and technical visualization frameworks, there are no validated comprehensive design recommendations for interactive 4D cityscapes.

3. Workflow Design

In our work, we apply a highly multifaceted approach to collect, process, and visualize data. In this chapter, we highlight the high-level design and interlinking of these components. Our pipeline comprises three steps: (1) to collect data as images, models, and information from multiple sources and via automated and community-based approaches and to (2) automatically form a model of a historical architecture and cityscapes via a 4D reconstruction from historical sources. (3) The model visualization is accessed in two ways: via a 4D browser and via location-dependent mobile XR.

3.1. Data Collection

Data were collected in various ways (

Table 1. Data collection and retrieval approaches used by the Jena4D group.

Crowdsourced Image Collection	Crowdsourced 3D Digitization	Data Retrieval Pipeline	Hackathons	School Projects
(b) Participatory virtual history knowledge bases, co-designed by citizens [80,276]	(c) 3D heritage as combined low-end guided workflow to capture photographs of endangered heritage via smartphone and server-based 3D modelling [81]	(a) Location-based data retrieval from open image and 3D repositories and information resources [81]	(d) “Modelathon” as an international student 3D reconstruction competition in 2018 and 2020 [277]	(e) A student presents her digital project to pupils [276,278]

). (a) To aggregate historical photographs of the city of Jena, we conducted a citizen contest in autumn 2022 (see Section 4.1.1). During this contest, over 4000 historical photos were submitted or rephotographed. (b) As another option for gathering 3D content, we implemented a low-end 3D digitization pipeline to document heritage with images taken with a smartphone—this was combined with a server-based 3D reconstruction (Section 4.1.2). (c) Data retrieval comprises approaches to gather images, links to information resources, and 3D models from different sources. So far, ca. 20,000 images and 4000 3D models have been retrieved (see Section 4.1.3). Educational courses comprise (d) two student modelathons to virtually recreate historic cityscapes (see Section 4.1.4) and (e) three courses designed by students (see Section 4.1.5), which were offered to primary and secondary school students in the context of school and extracurricular working groups and a project week in a youth center, and have now been made available to other teachers on a public platform. In total, 95 pupils participated in these educational courses.

3.2. Four-Dimensional Modelling

Our main approach (

Table 2. Workflow for footprint extraction, photo orientation, and parametric generation of textured 3D building models.

Footprint Extraction from Historical Maps	Spatialization of Images	3D Parametric Models	3D Modelling	Data Enrichment
(a) Georeference historical plans and extract building footprints	(b) Detect similar views via overlapping segments in photographs. Calculate the relative position via a feature-based orientation/positioning pipeline	(c) Create low LoD models by extrapolating footprint to building walls, generating roof shapes, and projecting photo texture to the façade	(d) Create higher LoD 3D geometries from imagery for better documentation	(e) Enriching data and connecting to other sources require an overarching ontology and the detection of and map links to text, image, and 3D data

) is (a) to georeference historical plans and to extract building footprints (see Section 4.2.1). A parallel pipeline task is spatialization of historical photographs. The first step is (b) to collect and identify spatiotemporally corresponding images via textual image metadata (e.g., placenames) [78,279]. The next step is to detect similar views via overlapping segments in historical and contemporary photographs and to calculate relative positions via a feature-based orientation/positioning pipeline and combine them with contemporary, oriented data to identify absolute positions (see Section 4.2.2). On the city scale, (c) we created 3D buildings by expanding the footprints (extracted in a) to building walls, calculated the roof shape, and used the position and orientation (calculated in b) to map the photographs onto the façade. These steps mad it possible to create large-scale city models with basic geometric features (see Section 4.2.3). (d) For buildings where a sufficient number of historical photographs were available, we created higher-resolution 3D models from imagery (see Section 4.2.4). (e) The final step was to enrich data (see Section 4.2.5).

3.3. Visualization

Since 2016, our group has been developing a modular software framework for 4D web visualization and content browsing to test and validate design hypotheses in virtual, augmented, and 2.5D visualizations on mobile and desktop devices [215]. As user acceptance of native applications is decreasing [280], especially for specific and short-term use, and as it is relevant for most cityscape scenarios [281], this is a browser-based web application (

Table 3. Frontend applications.

4D City Application	4D Browser Application
Browser-based mobile application showing textured 3D models of historical buildings and points of interest.	Graphical user interface of the 4D browser application.

). Both applications share a backend for handling data I/O (see Section 4.3.1).

The 4D City application for mobile devices (see Section 4.3.2) enables time-variant virtual 3D impressions of historic cities on a multi-device visual interface that can be accessed via desktops, mobile devices, and AR and VR glasses. The 4D City application feeds in real time from other data providers with a minimal database of its own, primarily for caching. It enables virtual city tours and past play (e.g., digitally enhanced discovery games) and provides access to knowledge assets from open-source platforms such as Wikipedia or tourist information platforms like Triposo.

The 4D Browser (see Section 4.3.3) is used for the information presentation and as a research tool [282]. Linking digital images and their actual location makes it possible to directly present resources and therefore proves a valuable support tool for historical research. Users of the virtual archives can benefit extensively from effective tools which enable searches based not only on content and theme, but also location. A timeline provides information on the development of a city via filtering photos of different building conditions.

4. Results

The following sections present information about demonstrations and studies we have performed on the pipeline steps.

4.1. Data Collection

4.1.1. Crowdsourced Data Collection

To crowdsource the collection of historic photographs and re-create these photos to collect spatial metadata, we equipped the mobile application with additional functionalities which can be used worldwide. To increase their use, we conduct local citizen contests. The contest was ran in Jena in 2022, with 4000 images collected, and will take place in the city of Schleiz in Thuringia in 2024.

The Jena citizen contest was conducted in autumn 2022. The citizens were asked to upload private historical photos, postcards, and other historical images from the period between 1900 and 2000 (see [80]). Images could be submitted in three different ways:

• If the photos were digitized, they could be uploaded from within the application.
• If the photos were still in analogue form, they could either be photographed directly within the application or submitted at various collection points such as the Jena City Museum and the Thuringian State and University Library (Thulb). Especially for larger quantities, the images were digitized at Thulb and then we transferred them to the application database.
• To determine the position of the historical photos, citizens were also asked to “rephotograph” images that were already in the database. To do this, the participants had to identify where the respective historical photo was taken and position themselves so as to take a new photo from the same viewpoint and angle. The corresponding information about geolocation, etc., was then automatically transferred from the mobile device to the database and used to project the images on the models.

To advertise the contest, we cooperated with regional newspapers, which regularly reported on its progress. Furthermore, to address the citizens directly, four–five public stands were set up weekly in the Jena City Museum, the University Library, and the Market Square. Here, people could hand in analogue pictures personally. In addition, advertising materials were sent to schools and cultural associations and distributed in bars and pubs. To ensure maximum accessibility, we kept the threshold as low participation. Participants were required to provide their name, email address, and telephone number in the application. This also allowed the participants to choose whether to transfer the rights to the photographs.

Prizes could be won in a total of six categories shown on the official website (Figure 1) created for the competition by an advertising agency. The winners were selected by a high-ranking jury, including the Thuringian State Secretary for Culture and the Mayor of Jena.

In total, over 4000 historical photos and rephotographs were submitted, of which a selection was used to evaluate the 3D reconstruction methods.

4.1.2. Crowdsourced 3D Digitization

As another option for gathering 3D content, we implemented a low-end 3D digitization pipeline to document heritage with images taken with a smartphone (Figure 2). The goal was to document cultural heritage using images and 3D models from user-generated photos and to integrate the results in the DFG 3D-Viewer repository [81]. The web frontend and processing pipeline were in a beta state at the end of 2023, and had already contributed 3D sculptures to our 4D applications.

3DHeritage consists of a webpage frontend in multiple languages (currently English, German, Ukrainian, Russian, and Arabic), providing a guided workflow to take images and upload them to the portal servers with metadata. Metadata can be freely added or retrieved from Wikipedia for object descriptions, from ORCID [283] for user information, and from Geonames for location information [284]. After uploading, a server-side process is initialized, which uses a scripted Meshlab pipeline to create 3D models from these images. We used this tool with an unsupervised pipeline; we are currently automatically uploading the models to Sketchfab and then retrieving them in the DFG 3D-Viewer to reduce error sources and data conversion.

The workflow has been operational in public beta since September 2022 and used in various settings with ~400 images of 21 objects processed so far; of those objects, 19 could be transferred to a 3D model. As a general finding, all 3D meshes produced in the automatically processed pipeline are of low quality with non-watertight and gapped meshes. Despite these limitations, the 3D objects give at least a visual impression of the shape and texture and are of sufficient quality for viewers.

4.1.3. Data Retrieval

To gather data on the world scale, we have setup a server-sided pipeline to retrieve data from different providers. This is currently operational in an alpha version, with 20,000 images, 4000 3D datasets, and 2700 POIs retrieved up to the end of 2023.

Data retrieval comprises approaches to gather images, links to information resources, and 3D models from different sources. To retrieve legally accessible images, we selected CC-0 or CC-BY [285] licensed content only.

For data retrieval, we used a series of server-side scripts in Python and PHP feeding into an SQL database and Unix file storage. The scripts are input by users locating them in our mobile application 4D City—the locations are resolved into placenames via Geonames [284].

To avoid reloading already retrieved positions, each object’s geocoordinates were processed into geotiles. These geotiles were compiled from the Universal Transverse Mercator (UTM) hours and minutes of each value. For instance, the geotile 5105 × 1374 would comprise a bounding box with latitudes 51.05 to 51.06 and longitudes 13.74 to 13.75. Unless this simple approach leads to varying tile sizes, it enables easy coding and decoding of geotiles. To query multiple images, even in query services like Google Street View with one result per query, we further subdivided the coordinates within the bounding box—e.g., to a 3 × 3 position grid—and queried each of these positions.

Since May 2022, 12,983 positions have been collected and processed. By mid-2023, several thousand datasets were retrieved via these sources (

Table 4. Sources and number of retrieved datasets (10/2023).

Type	Source	Grid	Number Retrieved
Points of interest	Triposo	1000 m radius	1684
	Google Place Search	3 × 3 queries within grid 5 top results	1092
	Wikipedia		Not cached
Images	Flickr	1000 m radius	8526
	Europeana	Within grid	3227
	Mapillary	Within grid	3961
	Wikimedia Commons	10,000 m radius	448
	Google Street View	3 × 3 queries 3 images (120° FoV) per position	2310
3D Models	Sketchfab	Keyword search using a location name retrieved via Geonames	2.736
	Europeana	Keyword search using a location name retrieved via Geonames	906
	Mainz 3D	Full dataset	64
	Urban History 4D	Full dataset	214

After retrieval, a series of scripts perform the renaming of files to gain unique filenames and store the metadata in an SQL database.

).

In the next step, we pre-classified retrieved images to sort out images not showing building exteriors. For this, we use a Python-based VGG-16 classifier, which was trained on a set of images derived via the retrieval pipeline. We used 4033 manually classified files belonging to two classes (showing/not showing architectural exterior). For training, 3227 files were used—including nine variants by data augmentation per file—and for validation, we used 806 files.

The accuracy of the validation dataset is above 0.85 (Figure 3). A classifier was used to automate the step to identify architectural exteriors as a prerequisite for further processing in the pipeline. To gain a good ratio between true and false positives and negatives, we currently only include an image if the predictor certainty is 90% or higher.

4.1.4. Student Hackathons

To teach 3D reconstruction skills and apply them in a competitive setting, we organized two student competitions for the digital 3D reconstruction of historical architecture in 2018 [286] and 2020 [277] (Figure 4). Besides teaching 3D modelling skills, these modelling hackathons—named modelathons—were proposed to create 3D models of historic buildings and city quarters.

This format was tested for the first time in 2018 as part of the Digital Humanities in German-speaking Areas (DHd) conference in Cologne. In that setting, over four days during the conference, nine student teams reconstructed the historical Hofburg in Vienna. This reconstruction was supported by various educational and practice sessions teaching 3D skills. Due to the restrictions during the COVID-19 pandemic, the second modelathon was held in the winter semester 2020/2021 within a project funded by the German Rectors’ Conference. In that session, the goal was to reconstruct the lost industrial architecture of the Carl Zeiss AG factory in Jena, extant in the late nineteenth to early twentieth centuries, which only partially exists today. In that modelathon, eleven student teams took part.

4.1.5. Content Creation by School Children

As another approach to create content, we began conducting school courses in 2022 to enhance digital skills but also to create mainly textual information and virtual tours to populate the 4D information system.

Digital learning with primary school children was tested in the religious history project Virtual City Walks to the Churches in Jena [276]. This was carried out in June 2022 in a local primary school with a time frame of three times 60 min with eleven pupils. The aims of the project were to improve the pupils’ digital competence, to deal with the religious and urban history topic of churches in Jena, and to train teamwork skills. In the end, two virtual city tours for children were created, which were to be inserted into the existing 4D City application and thus made available for independent use. In a qualitative pupil survey after the project, it was found that independent work on a computer on a self-selected topic was particularly motivating. The personal reference to some churches and the digital method of the virtual city tour also motivated the learners to study with the topic of city history. However, internet research, especially filtering out credible sources and formulating the most important information on a topic, was difficult for the learners. This is in line with the identified problems of the DigComp framework [287], and shows that this skill needs to be learned in the future, either before or on the course. Building on the identification and replication of this problem, all methods and tools discussed in the digital humanities lab should enable the learning of media and digital literacy with a focus on understanding and evaluating information.

To test and establish digital labs in the humanities, the Digital History Lab, funded by the Stiftung für Innovation in der Hochschullehre, was initiated at the University of Jena in 2022. In the DH Lab, university students learn about cultural and historical topics and test and reflect on them with school pupils. In doing so, the students gain practical experience in the digital research, preparation, and communication of cultural history content, which they can build on in their later professional lives. The thematic focus of the first project year was on Jena’s city history and was intended to motivate children and young people to deal with the cultural heritage, personalities, and historical events of their home city in a source-based and creative way. For this purpose, various digital methods such as digital source research, text analysis, the creation of 3D scans, and contemporary witness interviews were learned and used. The first results of the project were three courses designed by students of art history and history teaching (

Table 5. Example of courses created (images: Sophie-Luisa Hopf, Marlene Kropp).

Tell Us What It Was Like: Oral History Project	Objects Tell Stories, We Listen	Culture of Remembrance Rethought: Digitization of Stolpersteine
Topic: Young people critically examine the oral history method, research topics in modern history, conduct contemporary witness interviews, and prepare them using digital tools. Objectives: Understand the construct nature of historical narratives and be able to research independently using digital resources and critically reflect on the information obtained. Realization: Project week in a youth center, 5 days, participants aged 13–18 Results: Contemporary witness interviews; POIs in the 4D City application; self-evaluation; results of qualitative surveys of teachers and learners Link: https://www.db-thueringen.de/receive/dbt_mods_00059138 (accessed on 10 February 2024)	Topic: Children select historical everyday objects such as old kitchen utensils or tools; examine their function and history; and describe, draw, and digitize them using photography and 3D scans. Objectives: Acquire a basic understanding of source-oriented historical learning and get to know and reflect on the first methods of digitization. Realization: Afternoon work group in a primary school, six sessions, participants aged 7–10 Results: Drawings, photos, 3D scans; POIs in the 4D City application; results of qualitative surveys of teachers and learners; self-evaluation Link: https://www.db-thueringen.de/receive/dbt_mods_00059136?q=Gegenst%C3%A4nde%20erz%C3%A4hlen (accessed on 10 February 2024)	Topic: Pupils research the life stories of Holocaust victims commemorated by Stolpersteine (“stumbling stones”), create short biographies, and present them on the web. Objectives: Learn about regional remembrance culture in a creative way and acquire skills in digital source research and text presentation on the web. Realization: Afternoon working group in the DH Lab, six sessions, participants aged 11 Results: Texts, images; POIs in the 4D City application; results of qualitative surveys of teachers and learners self-evaluations Link: https://www.db-thueringen.de/receive/dbt_mods_00059137?q=Stolpersteine (accessed on 10 February 2024)

), which were offered to primary and secondary school pupils in the context of school and extracurricular working groups, and a project week in a youth center which was made available to other teachers on a public platform. In total, 95 pupils participated in the educational courses.

In a subsequent survey of the students, three challenges were identified: the necessary technical equipment such as internet-capable end devices and digital sound recording devices; the different prerequisites of the learners in terms of prior knowledge, digital skills, and working speed; and legal questions regarding the publication of the work results. The first solutions were the acquisition of the necessary technical equipment, flexible planning, sufficient staff to provide the learners with individual support, differentiated tasks, and obtaining written consent from the parents.

The next steps of the project are to expand the focus of the content from local history to other historical topics, to provide in-depth training in digital skills for teachers and learners, and to expand the target group to include educators in both schools and museums.

4.2. Data Processing

4.2.1. Building Footprint Extraction from Historical Maps

The initial step is to georeference historical plans and to extract building footprints. This step is currently in testing for one historical map of Jena via testing two different approaches—the Deep Learning module of ArcGIS Pro and Segment Anything.

For the first proof of concept, a historical map of Jena in 1936 was provided by the Thuringian State Library (Figure 5). To verify the building footprint extraction with a contemporary map, the historical map was georeferenced by determining multiple control points visible in both maps.

Automatic segmentation of historical maps is considered a difficult challenge due to vast changes in appearance between maps of different epochs [164,288]. While one approach may be successful for the 1936 map, it may easily fail for another map.

For the Jena map, two different strategies were investigated. The first approach involves using the Detect Objects Using Deep Learning module of ArcGIS Pro. This requires users to manually vectorize several building footprints in the historical map and use them as training data [80]. In our research, reasonable results could only be obtained after labelling approximately 50% of the complete map. Still, the post-processing tool Regularize Building Footprints had to be used to create proper edges.

In the second approach, Segment Anything [289] was applied to the historical map. As the resolution of historical map data is usually high, we followed a tile-based approach, cutting the map into equally large parts with a maximum resolution of 500 × 500 pixels. This eases the visual comparison to the first approach. The results can be seen in Figure 6.

Figure 6 shows that the semi-automatic approach outperforms Segment Anything. However, this is only the case after manual segmentation of a large amount of the original map and therefore cannot be seen as a general solution for other historical maps. Without any historical training data, Segment Anything performs quite well in extracting the footprints of the building blocks. However, text and graticule lines provide challenges for the tool and should be removed prior to use. Again, this would require manual work. A similar approach to ArcGIS should be run to regularize the final extracted footprints.

This proof of concept shows the potential for automatic footprint extraction from historical maps and its extrusion to 3D building models. However, the provided data show the challenge of the situation and the research potential for finding a holistic solution.

4.2.2. Spatialization of Contemporary and Historical Photographs

One pipeline task is the spatialization of historical photographs. Part of this task is to collect and identify spatiotemporally corresponding images via textual image metadata (e.g., placenames). The next step is to detect similar views via overlapping segments in historical and contemporary photographs and to calculate a relative position via a feature-based orientation/positioning pipeline, and combine them with contemporary, oriented data to identify absolute positions. While digital photographs already contain data which enable, at least for Streetview data, an out-of-the-box use, historical photographs have to be processed to retrieve orientation information.

Exact determination of the position and orientation of contemporary and historical photographs remains a crucial step for automatic photogrammetric 3D modelling and texturization of 3D models. Several strategies have been developed within the Jena4D research group.

Image Processing for Contemporary Photographs

Regarding the contemporary image material retrieved from different sources, initial experiments deal with the retrieved Google Street View images. The Google Street View API only provides sparse information such as the latitude and longitude of the photograph in UTM coordinates and one angle for the orientation. Thus, several assumptions need to be made:

• As there are always three images taken at one position, we assume a field of view (FOV) of 120°.
• As only one angle is given, we assume that this is the rotation around the yaw axis. We assume that the other two angles are close to 0°.
• As the height coordinate of the image is not given, we estimate the respective elevation using the API of opentopodata.org and the EU digital elevation model EU-DEM with 25 m resolution. We add 2 m to the retrieved height because Google’s camera is usually mounted on a car.

Considering these multiple assumptions for the contemporary images, the accuracy is quite remarkable (Figure 7).

It is planned to use these images in areas where no other (historical) image material is available. Considering the quality of the overlay, it is possible that matching the 3D model and image content may improve the image orientation.

A similar workflow has not yet be established for Mapillary images. The Mapillary API provides even less image orientation information, and the global coordinates seem to be more inaccurate than those for Google Street View, placing the images in the wrong spot in the 4D application. Further, orientation angle information is not given at all, so the viewing direction would have to be estimated by an iterative approach to test matching between images and given OpenStreetMap (OSM) panoramas.

The last retrieval image data source is Wikimedia Commons photographs. While these images are usually not available for all possible urban scenes, they can be especially relevant for specific landmarks. For famous sights, several hundreds of images often exist. However, no orientation information is given for this material. Thus, for the Wikimedia Commons images, we followed the SfM method that is used on historical urban images and explained in the following.

Image Orientation for Historical Photographs

The strategy for orientation of historical images is explained in detail in [130,290]. Here, the process is summarized, and recent improvements are highlighted. For the simultaneous estimation of camera parameters and the generation of a sparse point cloud out of many contemporary and historical photographs, the photogrammetric method SfM was applied using a modified workflow in the software COLMAP (https://colmap.github.io/ (accessed on 12 December 2023)) [291].

The most important aspect for generating a suitable model is finding reliable tie points between image pairs, which is especially challenging for historical or retrieved data. This is why our method uses a combination of the feature detection method SuperPoint [292] and the feature matching method SuperGlue [293]. Recently, small advances were made by using DISK features [294] with LightGlue [295], which has not been tested yet in our workflow.

As the final model often shows slight inaccuracies due to the historical data material, the bundle adjustment is improved using pixel-perfect-sfm [296]. The final product (Figure 8) still needs to be georeferenced in an interactive process as global and local points cannot be found automatically.

This is mainly due to the different appearance and accuracy of the sparse point cloud in the local coordinate system and the LoD2 model in the global coordinate system. We assume that the accuracy could be increased by matching multiple segmented images in the local coordinate system against the LoD2 models.

4.2.3. Parametric Modelling

On the city scale, we created 3D buildings by extrapolating the footprints to building walls, calculating the roof shape, and using the position and orientation to map the photographs onto the façade. These steps made it possible to create large-scale city models with basic geometric features. This feature is already applied on the world scale.

There are a large number of parametric modelling approaches for roof features, most notably the straight skeleton approach [297] to compute a mediated topline of the roof (e.g., [298,299]). In our case, the key objective is to reassemble the roof shape as best as possible; therefore, our roof-generation approach is designed to create different roof shapes, which are to be compared and selected for the best match to photographs.

To ensure a baseline 3D model, building footprints were pulled from OpenStreetMap and other providers, and the outer walls were modelled in the browser of the viewing client. Besides the building geometry, roof features are very important in assessing buildings [215]. The roofs were generated based on the data loaded from OSM. The data returned the following values: type, orientation, height, angle, levels, direction, material, and color. The supported types are as follows: flat, y-shaped, dormer, half-hipped, skillion, gabled, quadruple saltbox, and hipped (Figure 9).

The algorithm can be divided into two main parts: parts for squared and non-squared roofs. Squared roofs are built after calculating the midpoint of the four vertices, and according to the roof shape, the geometry (triangles) is created (Figure 10). For L-shaped roofs, more properties need to be calculated. First of all, the two shortest distances of the six-vertices roof are calculated. These lines cannot be neighboring lines (they cannot have the same starting or ending point) and cannot be parallel (their dot product of vectors, created at midpoint, has to be much smaller than one). Then, at the mentioned midpoints, perpendicular lines are created. The crossing point creates a new vertex which combines two parts of the L-shaped roof. At the given height and shifted by some distance, vertices are added at the midpoints of the shortest lines to create the L-shaped top of the roof. Then, triangles are created based on the direction and the newly created vertices.

Currently, we are able to process gabled, skillion, dome/onion, pyramidal, cone, and flat roofs with different levels of complexity (Figure 11). The algorithm can be divided into six sub-parts. Apart from properties provided from OSM data (like type, angle, or levels), sometimes extra data need to be extracted:

• The center of the roof (midpoint between two coordinates from Cartesian extents of the shape).
• The direction in which the tilt should be created (angle given by longest edge in the shape).

The implementation for creating different types of roofs is currently as follows:

• Gabled/hipped: First, the center of the roof and direction is determined. Next, based on the depth of the geometry (peak point, direction, and outer shape), the highest edge is created, and then the geometry is extruded.
• Skillion: First, the angle of the roof and the pitch is determined. Then, the higher part is extrapolated to the pitch point and the slope faces are added.
• Dome/onion: First, the largest circle within the points and its center are determined. Based on this, a sphere geometry is created and scaled according to the scale (height). For onion domes, this has to be raised by an additional radius.
• Pyramidal: First, the center of the shape is determined. Then, a pyramid geometry at the center point and level height is created.
• Cone: First, the center and number of points of the shape are determined. Then, a cone geometry at the center point, with the height of the levels and base equal to the roof shape, is created.
• Flat: the same shape as the input is created with a height at a given level.

The next steps are to create rarer roof shapes like dome, onion, or cone-shape rooves.

Figure 11. Example roof shapes from the parametric generator for buildings with four faces (left), implementation in the 4D City application for larger numbers of faces (right).

Figure 11. Example roof shapes from the parametric generator for buildings with four faces (left), implementation in the 4D City application for larger numbers of faces (right).

4.2.4. Generation of Historical 3D/4D Models

For buildings where a sufficient number of historical photographs were available, we created higher-resolution 3D models from imagery. This approach has currently been tested with data from single buildings in Jena and Dresden and for a larger area in the inner city of Budapest.

The presented automatic image orientation allows time-dependent 3D models (4D models) to be generated using image content. For this purpose, the textures of the georeferenced images are projected onto the respective LoD2 models. Depending on the selected time period, one or multiple images are used. In the case of multiple images, we developed a shader for smart selection of image content depending on the image angle and distance to the object, as explained in the visualization section.

This approach is useful for real-time applications where heavy data traffic should be avoided. However, textured models usually lack complexity due to the simplicity of LoD2 (Figure 12).

More detailed 3D models from SfM workflows are usually generated using dense matching with its most widespread technology, semi-global matching. As a result of the extremely inhomogeneous nature of historical images, and consequently the large radiometric and geometric differences between image pairs, the result of conventional dense matching is often unusable (Figure 13).

This is why initial experiments on using neural rendering strategies like NeRF [136] have been carried out in the research group. Detailed tests and results [80] are summarized in the following.

The sparse point cloud derived using advanced SfM techniques is transferred into SDFStudio [301], which is a modular framework for neural implicit surface reconstruction. In our experiments, NeuS Facto yielded the best results, even for a small number of historical terrestrial images. However, rendering may also produce artifacts, and the results are still far from perfect detailed representations of the historical buildings (Figure 14).

Consequently, more experiments have to be carried out to improve the automatic generation of high-quality 3D building models. Improvements may be obtained by using and adapting even more recent neural rendering methods such as [302,303], as we think that there is high potential when using these approaches for automatic 3D and 4D modelling.

To test this pipeline on a larger scale, we selected, from ~200 k historic images, 10 k images as belonging to the same continuous city model of Budapest. Figure 15 depicts an SfM reconstruction using SOTA it isand -content-based image retrieval (EigenPlaces) and neural matching algorithms (DISK + LightGlue). Although the overall topology of the city is clearly visible, although partially mirrored, it contains many degenerate cameras (long red lines) and incorrect matches; also, the repetition of structures such as the two sides of the bridge in the center of the view causes problems.

4.2.5. Data Enrichment with Textual Information

To further enrich 3D models with text and image data, turning them into an interlinked dataset, we used labels or annotations [304]. Taken together, the annotations enable a time/space related to browsing source material, but can also support numeric analysis, eliminate bias, and ensure reproducibility [305].

In the digital humanities, the Getty Art & Architecture Thesaurus (AAT) is well established and provides a hierarchy for architectural elements [306]. Furthermore, Wikidata offers a variety of entities, classes, and corresponding semantic relations for art and architectural elements and is also used in cultural heritage [307]. Relevant terms were identified via text analyses of Zwinger literature and Baroque architecture, building history and building research [305]. Finally, a list of about 400 entries (including 140 architectural elements) was compiled, containing both AAT and Wikidata IDs. So far, several Wikipedia articles, scientific publications, and popular scholarly articles have been annotated. In the texts, words or word groups from the compiled term list and terms with a high semantic similarity were annotated (Figure 16) [308].

In our case study, a detailed 3D model of the Kronentor (Crown Gate) from Dresden Zwinger was manually annotated in a modelling environment for testing purposes. In this model, most of the architectural elements are separate objects that carry the corresponding AAT or Wikidata ID in addition to the element name. Some objects are grouped and form a unit in the sense of a hierarchy of elements.

As described in the previous section, 4000 photographs of Dresden have been spatialized in the 4D browser application, either manually or using a semi-automatic approach. Images were also manually segmented and annotated using Label Studio (https://labelstud.io/ (accessed on 12 December 2023)), an application for labelling and annotating data which provides a standardized output format. Each annotation follows the scheme used for the 3D models and contains both AAT and Wikidata IDs (Figure 17). Automatic transfer of annotations from the 3D model to the located photographs is conceivable, but previous approaches have not yet achieved the desired level of reliability and accuracy.

The 4D browser displays annotations as a word cloud, indicating how often a term is mentioned in a text document, but also connects images, texts, and 3D information (Figure 18). The annotations are clickable in the text or image and the word cloud; clicking takes the user to linked external information like the AAT and Wikidata entries and features like searching for connected or similar annotations in other images, 3D models, and text passages within the platform.

4.3. Data Visualization

Multiple types of data have been collected, processed, and/or generated, particularly photographs and 3D building models [309]. To enable users to explore and interact with these data, two applications have been developed, targeting different user groups and scenarios: the 4D City application for mobile devices and the 4D browser for desktop use. Nowadays, all modern devices and browsers are capable of displaying 3D content via WebGL (https://caniuse.com/?search=webgl (accessed on 29 September 2023)). This includes extensive 3D city models due to the increasing computing power of mobile devices [310]. Three-dimensional asset formats like glTF [311] with Draco 3D compression [312] ensure that the data loaded for transmitted 3D content are kept to a minimum.

To put the spatialized photographs and generated 3D models into context, other data sources were incorporated. Digital elevation models (DEMs) on the basis of the SRTM or ALOS World 3D satellite data have a resolution of up to 30 m, are publicly available, and can be queried by Google Elevation API or OpenTopography. With the DEM-Net Elevation API (https://elevationapi.com (accessed on 29 September 2023)), it is possible to query ready-meshed digital terrain models that are optionally textured with a map. DEMs with a higher fidelity (up to 1 m resolution) can be retrieved from national land survey offices. However, they have to be downloaded and processed manually.

The 3D building models, gained from various sources and processed as described above, cover only small areas at very specific places. Hence, there are many void areas. OpenStreetMap (OSM) offers detailed geodata from around the globe, including building footprints. The Overpass API was utilized to query these building footprints that are used to generate 3D geometries [313] during runtime to complement these void areas. Additionally, the 3D city model can be enriched with in-depth information on buildings, places, statues, etc. In this regard, open knowledge graphs such as Wikidata are queried via SPARQL [314] to find and collect points of interest in the vicinity.

4.3.1. Backend Application

The backbone of the two frontend applications is a server-side application with various functions (Figure 19). Primarily, it implements a RESTful API as a main interface to request data. In order to query and serve our aggregated and processed data (i.e., photographs enriched with spatial and temporal information, generated 3D building models, custom terrain and map data, or custom points of interest), we used the graph database Neo4j. This enables us to identify relationships between multiple instances of data (e.g., which images depict which buildings), and to store them to improve the querying and finding of relevant data.

External APIs are requested to complement the custom data: the DEM-Net Elevation API for digital terrain models, the Overpass API for OSM building footprints, and Wikidata for retrieving information for points of interests. Since the requests to those APIs can take up to several seconds and are unlikely to change in the short term, the backend caches the responses. Thus, requests with the same or nearby location query parameters can be answered much faster.

Besides storage and querying data, the backend features some more computation-intensive tasks. When 3D models are uploaded, they are converted into Draco-3D-compressed glTF files. Additionally, which OSM building footprint overlaps with the uploaded model is checked to avoid interfering with 3D models later in the scene. If the spatial information of an image is updated, it is then determined which buildings are visible and to what extent. Experimental pipelines for automated data generation and enrichment (i.e., spatialization of photographs, generation of 3D buildings) are not yet part of the backend, but the results can be pushed via an API endpoint.

4.3.2. Features of the Mobile Application

The 4D City application is a browser-based web application that runs on iOS/WebKit and Android/Chrome browsers. The application is built by using Angular and three.js. Users can explore the scene at their current location. This requires access to device sensors, i.e., geolocation and device orientation sensors. By limiting the user’s geolocation and the view to a first-person perspective, there is no need to load a huge 3D city model, only the buildings in the vicinity, which reduces the amount of data transmitted.

All content with spatial attributes is usually delivered in polar coordinates, i.e., WGS 84 (latitude, longitude). Since the 3D scene requires a Cartesian coordinate system, the spatial attributes are converted to a metric system, specifically UTM coordinates, which preserve the actual dimensions. The following sections present the main features of the 4D City application.

Projective Texturing

The users explore the scene from a first-person perspective. This contrasts to the 4D Browser, where a cityscape is explored more from an aerial perspective. To let users experience historical photographs of the urban scenery in the 4D City application, the photographs are used as textures on the 3D geometries.

A huge number of spatialized photographs can be used as potential textures, but not all of them might be suitable, so only a small number should be selected and assigned to each building geometry. Which photographs are potentially good textures for which buildings is determined in a two-step process.

An offline background process on the backend first determines which buildings of the 3D city model are actually visible from the camera perspective in the year the photo was taken for each photo. The scene at this point in time is rendered from the perspective of the photograph (a) with unique colors for each building and (b) as a depth map. A weight value can then be calculated from the number of pixels a building covers in relation to the total number of pixels in the image. In addition, the distance derived from the gray value of the depth map affects the weight value. A building close to the camera position is given a higher weight than a building further away. The weight value is stored in the database as a relationship between the building and the photo. It is now possible to query the 3D geometry of buildings with potential textures sorted by weight (Figure 20).

Instead of assigning different UV texture coordinates and multiple materials to the 3D building geometry, a custom shader is used to project multiple images onto the building surface at runtime. The number of photographs passed to the shader as texture units is currently limited to six in order to support older devices that have a limited number of fragment uniform vectors. With each texture, the virtual camera’s transformation and projection matrices are also passed to the shader. The fragment shader determines which parts of the textures should be applied to the surface and where. For each pixel, a weight value is calculated for each virtual camera c_i. If the surface position p is outside the frustum of the respective virtual camera, or the surface normal points in the opposite direction, the weight is zero (ω_i = 0). Otherwise, the vector v_i between the camera position and the surface position is calculated and the weight depends on a) the angle α_i between v_i and the surface normal n and b) the length of v_i (i.e., the distance between the two positions). The highest ranked virtual camera or texture is applied to this pixel. Thus, for a given part of the geometry, the texture with a more perpendicular view to the surface and which is closer (i.e., shows more detail) is selected.

UX

User-centered design is a significant principle during user experience (UX) reviews, including checking the design-driven development process and increasing user satisfaction. Gualtieri [315] and Stokes [316] both described a great UX as a useful, usable, and desirable interaction with a system or product. Altogether, UX is the result of a system or product’s manifestation, function, performance, and interactive behavior and covers whole-system or product acceptance. It touches on pragmatic and hedonic aspects of a system or product. Pragmatic or instrumental refers to the utilitarian aspects, such as the usefulness and ease of use, and hedonic or non-instrumental refers to the emotional and experiential aspects of product or system use [317]. As an area of human–computer interactions, UX is an important perspective in the human-centered design of interactive processes.

In 4D City, the challenges include to check the design and interactive elements of development process and improve UX. UX design principles are verified based on scenario needs, including system usability, functional practicality, ease of use, and user interactivity. The UX test process is structured on six levels to test such a complex system. It includes a strategy level (clarifying the application goal, user requirement, and UX), scope level (analyzing related applications and functions), a structure level (designing the information structure), a frame level (designing information interactions), a representation level (designing visual communication and navigation), and an implication and improvement level (evaluations, summarizing implications, and improvements).

In detail, three aspects are reviewed when verifying the application system: user requirements, main steps and vital nodes of the use process, and user experience. In every aspect, three design elements are reviewed in terms of the structure design, interactive design, and visual design. Before analyzing the structure design, the information structure of the application system was listed and analyzed in terms of what, why, and how, which are the significant principles and the foundation of whole UX review process (Figure 21).

For instance, in structure design analysis, the logical classification of the catalogue system and the definition of menu words are important prerequisites for easy understanding and use (Figure 22). A flowchart analysis is used to go through the application system, including the start page, tutorial, main menu, main functions, and popup navigation (Figure 23). For instance, if some button names are ambiguous, clearer names are suggested. Concerning the interactive part of the system, the interface structure should clearly reflect the workflow and avoid repetitive structures, complicated operation steps, or missing significant steps, making it easier to use. Ideally, users should be able to understand the function of the interface and perform the correct action intuitively. The visual design is based on structural design, with reference to the mental model of the target group and task achievement. This includes colors, fonts, pages, etc., to make the system pleasant to use. For instance, the text should be a uniform font, size, and color, making it feel harmonious and comfortable. Also, images should be organized in a uniform typography to feel coherent.

In addition, several potential users were invited to test the system (Figure 24) in both desktop- and mobile-based environments to provide feedback, which was organized to form a report. Finally, the project team was presented with current problems and opportunities for improvement to adjust, improve, and discuss with technical colleagues.

4.3.3. Four-Dimensional Browser Features and Visualizations

The other frontend application is the 4D browser. It is developed as a desktop- and browser-based virtual research environment, targeting primarily (but not only) art, architectural, and urban history scholars. The basic idea originates from shortcomings in conventional media repositories, where data can only be queried by their metadata [309]. To this end, the 4D browser extends conventional search interfaces (e.g., search bars, faceted search) by a 3D viewport that displays spatially oriented photographs within a 3D city model [318]. Since the 3D models only display buildings, (historical) maps have been incorporated as the texture of the terrain model, showing additional clues regarding the building situation, street names, and other infrastructure that supports orientation within a city. A tripartite time slider enables users to select a time range to filter images, choose a point in time that should be represented by the city model, and toggle between (historical) maps.

With this spatial approach, users are able to find images without precise knowledge of the metadata by spatially browsing around the building of interest. Since photographs are linked to the buildings they depict, interaction with the respective 3D objects can additionally filter the search results. Similar images can be browsed by easily navigating to neighboring camera positions. The 3D approach also makes it possible see the photographer’s perspective and to understand their situation while taking the photo [319]. With the time slider, it is possible to detect changes over time in the architecture, cityscape, and distribution of images.

Additional features aim to support scholars in answering their research questions (e.g., Which positions did photographers prefer when taking photos of a given building? Which areas or parts of buildings were never or hardly ever photographed?). Various visualizations extract the spatial information of the images to analyze not only their distribution, but also the acquisition habits of the photographers and the coverage of building parts [320,321]. A conventional heat map is used to identify aggregations of camera positions (Figure 25a), a heat map of buildings shows which parts have been photographed more frequently (Figure 25b), and popular angles of photographs are visualized by cluster-based radial fans (Figure 25c). Current work focuses on the addition and browsing of text documents, as well as annotation of architectural elements in texts, images, and 3D models to make connections across different types of media in order to further improve art and architectural history research [86].

Next to browsing and data exploration, the 4D browser also serves as an authoring and content management tool. Authorized users are able to upload images and 3D models, manually spatialize images, update metadata (including the date the photograph was taken as well as construction and destruction dates of buildings), and add custom POIs. All additions and changes made in the 4D browser also directly affect what can be seen in the mobile 4D City application. It is planned for the future that users are able to upload their own content to feed the system on the one hand (i.e., crowdsourcing), but also to create and analyze custom collections according to their specific research questions on the other hand.

On a technical level, the 4D browser utilizes the same browser-based technologies and frameworks (i.e., Angular, three.js) as the 4D City application, since the same data provided by the backend need to be processed. However, since it is for desktop usage, the application is not limited by the lower computation power and bandwidth of a mobile device. It can load and display more data at once, covering a bigger area. Nonetheless, the amount of data and geometry displayed in the 3D scene still need to be well balanced to ensure smooth interaction, particularly on older devices.

User Study

Usability tests are a common method to gain subjective opinions and insights into what users think about a specific application. How much users actually benefit from applications and interfaces depends on the usability, suitability, and efficiency of technological solutions. Assessment and feedback by user groups is used to adapt applications and interfaces. User studies help to (a) collect information needs and requests of the user groups, (b) filter out the challenges that user groups have when using the tool, (c) identify usability weaknesses and (d) adapt the application to the user’s needs.

Different formats like workshops, surveys, and demo sessions were used to assess the 4D browser application depending on the available time and audience.

The idea of the 4D browser is directly connected to research questions from art and architectural history. Scholars in these fields use images as the main source for analysis and argumentation and therefore turn to (online) digital image repositories. Common requirements of the users of digital heritage collections include ease of understanding the data and tools for accurate search and analysis, as well as an intuitive navigation and interfaces. Previous research has showed that 3D navigation can trouble users who are not used to 3D environments [322]. To evaluate the application, users were involved from the beginning of the development of the 4D browser. A workshop held at the Digital Humanities in the German-speaking Area (DHd 2018) conference in February 2018 involved 25 potential users, who were asked to reflect on their own experiences when using image repositories. Among a variety of mentioned issues and requirements were a dissatisfaction with insufficient filter options for existing platforms, requests for a faceted search functionality, to be able to label buildings within the 3D model, and for a feature to compare images.

Since the topics and tools of the 4D browser application are very specific, it needed to be tested with relevant user scenarios and use cases. Potential users to consider and explicit tasks to solve are particularly useful when a prototype must be tested for initial reactions to the functions and interface. The real user group must be brought in later to ensure that their quality criteria have been met.

The initial study was part of a workshop and contained different realistic tasks connected to the 4D browser [323]. The tasks help users to focus on and assess certain functions. They contain scenarios and problems that correspond to the knowledge and skills of an assigned persona that is part of the actual user group of the application. It is very important to properly have heterogeneous groups of test study participants. The persona approach seemed suitable for this as it helps participants from other academic fields who may not be able to fully envision the genuine intentions of the functions on the spot. Furthermore, thinking aloud was chosen to learn more about user needs and interactions. Instructions on how to approach the assignments as well as consecutive steps helped to guide the participants through the tasks. However, the instructions did not always name the specific tools and functionalities the participants should use. This allowed participants to observe whether they were named and placed in an intuitive manner. The data for the study were collected during a workshop offered by the research group at the DHd 2019 conference in Mainz, Germany. The topic of the workshop was usability testing for software tools in the digital humanities using the example of image repositories. Audio recordings of the participants were transcribed and coded using thematic analysis. This helped to evaluate how well the usability testing and the framework itself worked, revealing how participants dealt with personas and thinking aloud.

The International Committee of Architectural Photogrammetry (CIPA) Conference in 2019 and 2023 also offered opportunities to gather feedback during demo sessions. At the beginning of each session, the 4D browser was introduced via a live tour presenting different features and data. Subsequently, the participants were encouraged to use the application themselves to solve a defined task and provide feedback to the session leaders. At the end, they were asked to complete an online survey on specific design questions.

The demo session approach was reworked into a more extensive test and was carried out with art history students from the University of Munich in July 2023 and with digital humanities scholars in October 2023. Both groups are among the application target users. The two sessions again included a live tour of the 4D browser introducing different features. After a brief discussion to clarify questions concerning the data and the interface, the participants completed an online questionnaire with specific tasks to solve using the 4D browser. The questionnaire allowed every participant to work at their own pace, individually or in groups. The tasks were designed to test the functionalities of the application using the little available test data und to see if users approach solutions as intended during development and are satisfied with the results. The tasks were very simple scenarios connected to art history research and analysis:

• Identifying buildings in a photograph.
• Gathering information for a building (footprint, roof shape) and finding images from all sides of the building.
• Analyzing which perspective of a building was most frequently photographed.
• Analyzing differences between a digitized painting and the city skyline.
• Comparing the ways two photographers staged a certain building.
• Reconstructing the biography of a building.
• Identifying a certain statue in a photograph.

The questionnaire provided all necessary information concerning the task. Additionally, it was possible to seek assistance by displaying a step-by-step approach to the solution, even offering different ways to solve the task. The participants were able to provide feedback on the features and navigation of the application as well as the assigned tasks.

After each test, the feedback was reviewed and the application was adapted accordingly. Subsequent tests revealed if the changes were perceived positively.

Limitations

A general issue in historical research is that available information and data are always fragmented in terms of time and space. Validated knowledge is bound to single points in time, which may then be interpolated. Historical photographs capture a certain moment in the past, but the exact date when they were taken is often unknown. Similarly, the dates when a building was constructed or demolished may be uncertain. This affects the reliability of our visualized data, particularly if images are projected as textures of 3D building models. Due to the different temporal restraints and uncertain knowledge, a building may have the wrong façade texture at specific dates. Inaccuracies in spatialization and in transformations of the photograph, or not using central perspective cameras, lead to mismatches between the projected texture and the 3D surface. If objects have been captured in the foreground of a photograph, they appear bigger when projected onto the building, hiding parts of the façade.

Currently, where data are missing, building footprints extrapolated to 3D geometry and maps are queried from OSM to fill in the gaps. However, these contemporary data only show the current building situation. When exploring the past, buildings may be shown that were not existent at that time. Additionally, since the footprints are extrapolated, they may be hard to identify. This applies in particular to more complex buildings like churches. Furthermore, not all OSM data are properly tagged with a height value or the number of levels for an estimation of the building heights.

Despite using data compression, the amount of data loaded can still be very extensive, which is a crucial issue especially for the mobile application, where the bandwidth may be limited. To decrease the loading time and not exceed the user’s data volume, further optimizations are needed.

5. Demo Cases

5.1. Dresden

The Dresden demo case started in 2016 in a research project. It comprises a specific 3D dataset provided by the municipality of the current city of Dresden, a 3D dataset of Dresden in the 1930s created in a previous project, and a set of images taken from the Deutsche Fotothek (Figure 26). Since 2019, the 4D browser interface has been provided as an alternative interface to the Dresden image collections in the Saxon University and State Library.

5.2. Jena

The city visualization for Jena was created based on LoD2 model data from the municipality of Jena (Figure 27). Within previous projects, a dataset of 5000 images and three time layers of cadastre data had been collected. Beyond the mentioned usage (school projects and image contests), the application is already available in Jena and used for digitally supported tourism, currently including eight different digitally guided city tours on different topics (such as monuments or historical persons) and for different types of users.

5.3. Amsterdam

For the National Maritime Museum in Amsterdam, additional content has been included to enable the application to run on museum kiosk systems and show the transition of the museum building (Figure 28). In July 2023, this was included in an exhibition at the Architecture Centre Amsterdam (ARCAM), entitled Liquid building block—Designing with water in Amsterdam. As part of this exhibition, a 4D representation of the museum building was created and populated in the 4D applications [324].

5.4. Worldwide

The worldwide experience is based on contemporary material and enables a baseline view at any location (Figure 29). This includes a baseline DEM retrieved from OpenDEM, contemporary building footprints retrieved from OSM, and Wikipedia articles retrieved via their Wikibase position information.

6. Future Prospects

The pipeline and applications have been in development by our group since 2016 through various projects at national and European scales. Currently, applications are available to provide contemporary viewer functionality on the world scale, with location-based Wikipedia articles included as POIs. So far, nine PhD and postdoctoral projects in computer science, humanities, geosciences, design and education are linked to the pipeline, its applications, and its use.

6.1. Data Collection

To enhance the user experience, the main challenge is to enrich the current dataset with both (a) contemporary and historical photographs and (b) historical map data. For additional photographs, we are continuing to acquire data from various image sources. We are also developing user management functionalities to allow multiple users to contribute to the applications. We expect this to enable cultural institutions and citizen scientists to contribute content.

Another topic currently under development is the incorporation of 3D models including (a) 3D assets such as sculptures or monuments from Europeana or Sketchfab and (b) 3D geotiles such as those provided by Google [325]. So far, 110 EU-funded projects have also created 3D models of cultural landmarks, although only few of them provide their data. However, they are a high-value asset to safeguard and re-use in applications [326].

6.2. Four-Dimensional Modelling

The next challenge for our group is to improve vectorization and footprint extraction from historical maps. Corresponding research questions for processing photographic data include how to create or augment historical image data by training neural networks in feature extraction and feature matching. Another issue is the number and quality of historical images required to enable the use of dense matching and the creation of historical (generalized) 3D models. The challenge is to further reduce the current number of images required for orientation. The final issue is to cross-validate parametric models and images, for example, to iterate different roof types or to estimate building heights from projected photographs.

6.3. Visualization

A technical challenge in visualization is to merge the different visual material—especially photographs of very different style and quality—into a common visualization. At a higher level, most current collections and guidelines [327] strive for high-quality content, including the best possible quality in a particular feature (e.g., detail, style) used for very different scenarios. As mentioned in our discussion of the literature, we expect that scenarios containing non-detailed views, such as city views of open spaces, may work even with very low detail [215]. We still need to identify usage scenarios and develop and test corresponding visualizations to determine the quality requirements for each of these scenarios.