Once upon a Spacetime: Visual Storytelling in Cognitive and Geotemporal Information Spaces

Abstract

Stories are an essential mode, not only of human communication—but also of thinking. This paper reflects on the internalization of stories from a cognitive perspective and outlines a visualization framework for supporting the analysis of narrative geotemporal data. We discuss the strengths and limitations of standard techniques for representing spatiotemporal data (coordinated views, animation or slideshow, layer superimposition, juxtaposition, and space-time cube representation) and think about their effects on mental representations of a story. Many current visualization systems offer multiple views and allow the user to investigate different aspects of a story. From a cognitive point of view, it is important to assist users in reconnecting these multiple perspectives into a coherent picture—e.g., by utilizing coherence techniques like seamless transitions. A case study involving visualizing biographical narratives illustrates how the design of advanced visualization systems can be cognitively and conceptually grounded to support the construction of an integrated internal representation.

1. Introduction

Stories surround us in many aspects of our daily lives—conversations, news reports, series, films, and books are only some of the most obvious examples. Cultures are flush with thousands of narratives, whether real, fictional, or hybrid; they organize knowledge about our collective past, document our contemporary experiences, emotions, hopes, fears, and visions, and sketch out possible ways into intended futures. Cultures inherit stories and invent new ones, which are collected and shared in everyday communities; they are studied and interpreted by arts, humanities, and history scholars. Stories are used to communicate and present complex contents effectively, and are taught in journalism, media training, and rhetoric seminars.

Geotemporal stories that more or less centrally tell a protagonist’s movement through a geographic space can be found in many different narrative genres. Prototypical examples are travel reports, quest narratives, or biographies. One historical case where movement is highly central in the story is Homer’s Odyssey, as the geographic positioning and the timespan of Odysseus’ stops during his journey are still a topic of discussion (Appendix A). Visualization (cp. Figure 1) can help readers to connect the plot to familiar places and to better understand the time course of the events.

It is not surprising that storytelling has also become a topic of interest in information visualization. The corresponding crossover genre of “narrative visualization” or “visual storytelling” has been explored and elaborated extensively during the last years [2,3,4,5,6,7,8,9,10].

These works developed and discussed a whole panoply of solutions, such as how to bring the framing methods of diagrammatic pictures and stories together—to the mutual benefit of both. Visualization designers reconsidered “author-driven” design elements and strategies (like sequential guidance, ordering, or messaging), which can enrich the usually “reader-driven” reception of visualizations to varying degrees [5]. Since this “narrative turn,” developers of visualization systems have been able to choose more consciously whether complex information should be designed as open installments to be freely explored by the users, or whether they should be guided through these worlds by predefined sequences and narrative paths [11]. However, these reflections were mostly restricted to narrative communication patterns and the visual design space, and did not address the cognitive foundations of storytelling.

What is it that makes storytelling such a powerful mode for processing information and communicating it to experts or general audiences? This paper approaches this question first from a cognitive science perspective, to better understand how narratives are processed and internalized (Section 2). In a second step, it turns to the question of how to support the visual analysis and comprehension of stories by different methods of spatiotemporal information visualization (Section 3). Building on these options, we introduce a case study and a visualization framework that puts emphasis on the cognitive integration and on the coupling of multiple perspectives (Section 4), and we outline options regarding how to further advance this visual storytelling environment (Section 5).

2. Cognitive-Scientific Foundation

The main reason for the cultural prominence and omnipresence of stories is that narration is not only a mode of presentation, but “a fundamental way of organizing human experience and a tool for constructing models of reality” [12] (p. 345) in everyday life. Bruner [13] proposed a “narrative mode of thought” (p. 97) that helps one to construct internal representations about events, human intentions, and actions in the world. The events and actions described in stories closely correspond to everyday sequences and experiences and, therefore, their representation and comprehension is more natural than of other types of information (e.g., descriptive) [14]. Questioning the causes of events and the intentions of actors and organizing incoming information within such chains of events is central to human thinking and sensemaking.

How can we define a story—or rather, a narrative—as it is termed in cognitive science? Wilkens et al. [15] (p. 324) define a narrative as “a chain of events related by cause and effect occurring in time and space and involving some agency.” Following this definition, a story consists of five central information elements: (1) multiple events, (2) a time frame, (3) a space, (4) involved actors or objects, and (5) causal relations between them (Appendix B).

On a cognitive level, story schemata are available concerning how a story is built, how it progresses, and what its constituents are [16], which reduce cognitive load and allow stories to be processed fast and efficiently. Guided by these schemata, the recipient of a story picks up narrative cues, relates the information to scripts and prior knowledge, and actively builds up an internal representation of the events and the involved actors, which results in the so-called “situation model” [17]. How this situation model is actually constructed and what its cognitive constituents are comprise a matter of complex discussions, which will be elaborated further down (see also Figure 3). The situation model is relational and multidimensional in nature and is continuously updated as the story unfolds (Event-indexing model, [18]): Each new event occurring within a story becomes cognitively assigned or connected to a time frame of the narrative, to a space (the scene or location where it happens), to one or multiple causes (the prior event(s) that influence(s) the event), the protagonist(s) actively or passively involved, and to the way the event relates to the protagonists’ goals or intentions. Figure 2 shows a conceptual draft of the event-indexing model, visualized as a time-oriented semantic graph (from left to right).

Depending on how much of these situative indices an event shares with prior events, it can be integrated into the situation model relatively easily [19]. During the construction of a situation model, story recipients try to establish coherence on a local level (no inconsistencies and contradictions between two following events) and on a global level (within the whole story [14]). The recipient aims to close existing coherence gaps or breaks (inconsistencies within one or multiple dimensions) within the story by not only drawing inferences between the pieces of information, but also with prior knowledge. A story can be better understood if it is coherently designed; that is, if it is told consistently, does not contradict itself or miss information within the chain of events, and if its global architecture is plausible with regard to the interplay of the five threads.

But, how are stories (or narrative data in general) internally represented? For multimodal information processing and learning in general, a bi-modal model has been developed by Schnotz [20]. Its basic assumption is a dual layer architecture: Based on Paivio’s dual-coding theory, Schnotz suggested that multimodal information is processed in parallel in (1) a verbal–propositional and (2) a visual–spatial system and leads to the construction and elaboration of multiple internal representations, which can be transferred to one another and are closely connected in an active process that generates a coherent knowledge structure (see Figure 3). Building on this model, we assume that stories—as information in general—are internalized and represented somewhere in these different modes: either verbal or visual, but mostly in a bi- or multimodal fashion. Visual–spatial and verbal–propositional information are drawn together to construct a coherent situation model of the story.

For the geotemporal visualization of stories, two story indices are of specific interest for us: space and time. Though the other indices (intentions, actors, causation, and events) are also highly relevant, they are not the focus of our paper. We will shortly elaborate on them in Section 5.1.

2.1. Space

Space is regarded a difficult dimension of stories, as its nonlinear nature does not match the linear sequence of events [19]. A map-based representation of a story can assist the construction of a spatially structured situation model, which integrates details about a story’s locations [21]. Still, the typical mental representations generated from maps and from stories differ. Tversky [22] postulated that the internal representation of a spatial environment is not a coherent image of the external representation. Rather, recipients only selectively internalize information cues that are relevant to them (and the current task), which may or may not be related to other pieces of information, and which are likely to contain different perspectives. These “cognitive collages” are not coherently organized and, therefore, do not allow perspective-taking, reorientation, and spatial inferences, compared to a fully integrated cognitive map. Rather, the information pieces are related by the spatial relations between them and to larger units or landmarks [23], to personal experiences, and to frames of reference [24]. In contrast to situation models, a cognitive map (or collage) is nonlinear (except maybe for the representation of a spatial path), noncausal, less coherent and less structured.

Therefore, an important question for storytelling with maps is how stories are internally constructed and represented on a cognitive level. Kosara [25] (para 1), emphasized that “pictures don’t tell stories, people do. An image, a visualization, data, etc., can only be the material the story is made from.” The recipient of a narrative visualization plus his or her prior knowledge, intentions, and interests decide what information is attended to and what is internalized. Whether he or she generates a sequential situation model, a cognitive map, or some partial or hybrid form cannot be prescribed by the geovisualization, even though some design cues can help recipients to generate a better integrated, more coherent internal representation.

2.2. Time

Time is one of the most important dimensions of a situation model, as temporal information helps the recipient to establish causal and motivational links between events [19]. Similarly, Kosara and MacKinlay [8] argued that the temporal structure of stories is a fundamental feature for storytelling with visualizations. Therefore, we may assume that sequential and chronological aspects dominate (or strongly structure) the mental representations of stories (or situation models), which goes together perfectly well with the time-oriented nature of speech or reading text.

Yet, if the comprehension of the chronological character of complex stories should also be supported visually, time must be visually encoded and represented, which requires design choices that are far from trivial. “Time presents specific challenges for the representation of data because time is a complex and highly abstract concept” [26] (p. 203). One of the most frequently used solutions is to map time to a spatial (i.e., linear or sequential) dimension of the pictorial space, which often results in timelines [27,28,29,30], or also in linearly juxtaposed representations of sequential art [31,32,33].

Visual cognition support for the comprehension of stories and narrative data often leads to the utilization of either timelines or maps (or a combination of both), which organize the pictorial space—and, thus, the users’ modes of thought—quite differently [34]. Maps represent data that are already spatial, while timelines “spatialize” the abstract concept of time, utilizing the pictorial space in a radically different way [35]. This presumably leads to the construction of two different internal representations, which require additional mental effort to synthesize. To help the recipients generate a situation model of the story that integrates geographic and temporal data more closely and supports cross-dimensional reasoning and perspective taking, a combined or hybrid visualization method is required. Therefore, in the following section, we discuss hybrid visualization techniques that integrate geographical and temporal data aspects—and thereby, facilitate visual storytelling in a spatiotemporally synoptic fashion.

3. Spatiotemporal Visualization Methods

How can we represent narrative data visually on the screens of visualization systems that aim to augment and amplify reasoning with these kinds of data? Numerous solutions to visualize spatiotemporal aspects of narrative data have already been studied and analyzed in the literature. To illustrate some of these techniques, we will zoom in on the example of individual movement data, which consists of “recording the location of a moving point object through time” [36] (p.183). Such data have also been called a “mobile trajectory,” “world line,” “life path,” “space–time path,” or “spatial history” [36]. The benefit of focusing on individual movement data is its elementary nature, which allows the construction of a great variety of story types or narrative forms by combinatory means. Even if individual biographies or space–time paths of objects and actors are only one basic type of spatiotemporal data, almost every other type of narrative utilizes them to draw them as brush strokes into a more complex and intertwined choreography.

With regard to individual movement data, cartographers have introduced multiple methods and means for representing movement, dynamism, and change [37,38], including static and animated maps, space–time cubes, and coordinated linked views (e.g., maps with timelines). In the following section, we take a closer look at a selection of five of these standard techniques, which have been widely discussed and are available in various visualization tools and geo-analytical packages (for a video demonstrating these visualization techniques in the PolyCube system, see https://youtu.be/PTsk-NROJhU). Figure 4 lists these common perspectives, each visualizing spatial and temporal data aspects in a hybrid fashion.

Each of these spatiotemporal representation techniques requires different affordances for the construction of a corresponding internal representation (see also [26]) and, thus, also for the construction of the story situation model. In the following section, we are going to discuss each of them from a cognitive perspective and what they mean for the integrated perception and understanding of a story’s spatial and temporal dimensions.

(a)

Multiple coordinated views combine a standard map with a time graph to visualize spatial and temporal data aspects in parallel [39]. This method utilizes separate representations for the locational and temporal distributions of story events and usually coordinates these views via linked interaction methods, e.g., allowing for linked brushing [40]. For multiple coordinated views, it is likely that users generate two separate internal representations (one for each view) with some links to one another. Depending on the visual work and interaction a user invests to bridge this split of attention [41], these links can be relatively densely knit.

(b)

Animation or slideshows (also “dynamic representations” [42], p. 6) map narrative time orientation to the time dimension of the visual representation [26]. As such, they can represent the movement of objects or actors as a continuous dynamic (i.e., as smooth animation) or as a discrete sequence of steps, which we refer to as a slideshow. These techniques can further be implemented as non-interactive or interactive representations [43] (p. 1588), allowing users to go back and forth in time. It is well known that animation can foster the perception of even subtle changes or display dynamics, but also that the user’s working memory is easily overwhelmed when too much information changes too fast [26]. If the visualization is more complex, a slideshow might be better suited, which reduces the temporal continuum to discrete intervals. Users then can interactively go back and forth from one story event to the next. Still, considering one’s visual view and comparing it to the next one is very demanding for the working memory and increases interaction costs by repeatedly going back and forth.

(c)

Layer superimposition techniques merge multiple temporal positions—or temporal layers—into one integrated representation, while using transparency to see all positions at once [32]. Time is mostly encoded with an additional retinal variable, like color, or with the annotation of temporal values or vectorial references, signifying a temporal sequence of positions in space. In Figure 1, the map on the left-hand side uses a numerical sequence and arrows to encode the time orientation of the narrative. By using such a technique, users can build up a spatiotemporally integrated internal representation of the story. Aside from the expected challenges posed by visual clutter and occlusion, a concern regarding this technique is whether time (e.g., encoded by a color scale) is visually salient enough to be as well integrated into the story situation model as geographic space.

(d)

Layer juxtaposition separates spatiotemporal data into multiple temporal layers, to arrange these layers in parallel—mostly along a spatial reading dimension. This results either in “small multiple” maps [43] or, more generally, in the hybrid genre of data comics [33,44]. In face of juxtaposed views, the user must sequentially read and compare multiple adjacent views to detect the visual changes and comprehend how the story unfolds over time. Though a lot of visual work is required to compare the different views, the user does not need to remember them like in a slideshow; thereby, the interaction costs are lower.

(e)

Space–time cube representations merge maps and timelines orthogonally within a cubic space, which allows one to map every space–time path as a three-dimensional trajectory [45,46,47,48]. Aside from providing such a direct integration of spatiotemporal coordinates, space–time cubes also come with the specific functionality of supporting the cognitive translation and navigation among all other spatiotemporal views [49] (see Section 4). From a cognitive perspective, space–time cube representations offer one perceptually integrated view in which the story can unfold. In contrast to a superimposition view, time is also mapped to space, making the temporal and geographic information of movement paths similarly salient. Therefore, the user can more easily build up a spatiotemporally integrated situation model of a story. However, in such a three-dimensional visualization, visual clutter—and increased interaction costs—are a constant challenge [50]. Still, evaluations confirm that space–time cube visualizations are easy to use and are especially suited for the exploration of spatiotemporal patterns [51,52].

In comparison, these well-established representation techniques find distinctly different solutions to represent spatial and temporal data aspects in an integrated or hybrid fashion.

Table 1. Visual–analytical strengths and limitations of spatiotemporal representation techniques.

	Coordinated Views	Animation/Slideshow	Layer Superimposition	Layer Juxtaposition	Space–Time Cube
Strengths	+ Plain 2D views + Connected by linking & brushing	+ Salient detection of change + Small screen occupancy + “Quasi-integration“ in working memory	+ Spatiotemporally integrated and aggregated vis + Small screen occupancy	+ Parallel visualization + High visual literacy to be expected (comics)	+ Spatiotemporally integrated view + Metashape + Attraction power + Cognitive & conceptual mediator
Limitations	- Split attention effect - High visual work - No spatiotemporal integration	- Memory overload - High interaction costs - Interpolation of animation can create data artifacts - Slideshow can reduced continuous temporal data to an ordinal variable	- Visual clutter - Design too flat, generates little attraction - Time needs to be mapped to an additional visual cue (like color)	- Large occupancy of display space - Temporal data is reduced to an ordinal variable - Split attention effect - High visual work	- Visual clutter - High interaction costs - Medium occupancy of display space

summarizes their visual–analytical strengths and limitations and illustrates that every method comes with a specific profile, combining analytical benefits with particular costs.

Obviously, among these different options, there is no ideal view—but rather, multiple perspectives and methods exist to represent narrative data in a spatiotemporally hybrid way. For this reason, advanced visualization systems provide multiple spatiotemporal views as a “solomonic” design strategy—not to sacrifice the benefits of alternative perspectives, but to offer multiple ones. This allows one to “maximise insight, balance the strengths and weaknesses of individual views, and avoid misinterpretation” [53] (p. 9), and enables the user to select and switch between the most appropriate representations for the data and task at hand (ibidem, p. 10). For storytelling visualizations, multiple views offer different perspectives on a story, highlight different event indices, and let the user construct a more elaborated internal representation.

As time orientation plays a central role in the narrative data domain, we recommend to follow best practices of visualization system design and to deliberately implement multiple views for the complementary and flexible combinations of analytical perspectives. Yet, we also recommend deliberately addressing a neuralgic cognitive challenge, which emerges from using multiple views with their corresponding design strategies. Simply put, the utilization of multiple perspectives allows users to avoid analytical reductions, blind spots, and simplifications. However, multiple perspectives also challenge users’ coherent understanding: How do these different perspectives relate to each other? How do users not only perceptually integrate spatial and temporal perspectives (as all five abovementioned techniques assist them to do), but also cognitively integrate, couple, or synthesize these different spatiotemporal perspectives? This challenge is of special importance in narrative visualizations, as the coherence of the external representation is relevant for the construction of a coherent situation model. Coherence breaks have been shown to interrupt the construction of the internal representation and the recipient’s engagement with the story [14,54]. Similarly, for the design of visualization systems, coherence is an important factor to be considered, since it eases the construction, maintenance, and cognitive utilization of the internal representation [55]. Building on conceptual work, we report on the development of a web-based visualization environment, which emphasizes these exact development gaps that have remained largely unresolved by interface design so far.

4. Towards a Multi-Perspective Interface for Narrative Visualization

The PolyCube framework (PolyCube project: https://www.donau-uni.ac.at/en/polycube) has been developed to offer multiple temporal views on spatiotemporal data, while putting a focus on the overall ease of use, enhanced navigation, and on supporting the cognitive integration of insights from multiple views. Its central operating perspective is provided by a space–time cube representation and also offers access to perspectives of layer juxtaposition, layer superimposition, multiple coordinated views, and animation. One of its main design strategies is to translate or mediate these different perspectives by seamless layout transitions, which are illustrated further down (Figures 7 and 8). The visualization system is currently developed as a web-based visual analytics environment and has been explored with regard to two application scenarios, including a case study on cultural heritage collection data [56] and another on biography data [57].

4.1. Visualizing Biography Data and Historical Narratives

Digitization initiatives are transforming historiographical knowledge collections into semantically structured data and knowledge graphs [58,59]. This translation also involves large biographical lexica, which have been assembled to document the life stories of national cultural heroes—as well as many thousands of other figures that have influenced domains such as the arts, politics, or natural sciences—in a standardized manner. By the means of natural language processing techniques (Appendix C), a variety of entities and relations are extracted from thousands of life path narratives, including actors, organizations, social relations, and geographic places, which are all interwoven by the pathways of individual trajectories and temporally structured according to documented events and dates.

With regard to historical and cultural figures of Austria, the APIS project (https://www.oeaw.ac.at/acdh/projects/apis/) has developed a digital prosopographical information system out of the National Biographical Lexicon [60]. To explore the visual–analytical framework outlined so far, we extracted documented locations and corresponding time stamps for a dozen biographical narratives, which we injected into spatiotemporal visualization systems for further exploration. Figure 5 exemplarily shows the biographical trajectory of Joszef Szabo—an Austro-Hungarian opera singer and actor—visualized as a space–time path by the means of a commercial package (left-hand side, [61]) and by the prototypical PolyCube system, utilizing web technologies, such as d3.js, CSS3D and three.js (right-hand side).

We chose a space–time cube representation as a preferential entry point into narrative visualization because it immediately discloses a variety of spatiotemporal patterns (such as horizontal movements or vertical “stations”) to the highly skilled faculties of 3D gestalt perception (see Figure 5 and Figure 6). Furthermore, it allows on demands switching to all other spatiotemporal perspectives—while maintaining the orientation of analysts in a unique way (see Section 4.2).

With regard to the basic pattern language, three elements define individual choreographies: “The basic concepts in time geography are paths, stations, and prisms. Paths show movement behavior of objects through space and time […]. The stations indicate locations where people stay for longer moments […], and prisms representing space reachable within a given time budget” [62] (p. 57). Therefore, the meandering curves of historic actors disclose their spatiotemporal profile, consisting of their main biographical stations and connected by their movements and travels. As each historical character starts from different spatiotemporal coordinates—and behaves differently until his or her death—their space–time paths can be read and interpreted like a unique diagram. As soon as a minimum amount of familiarity (or visual literacy) with this visualization technique emerges, historians or humanists can utilize it for the purpose of close reading or viewing spatiotemporal story aspects—or zoom out for distant reading of multiple biographies or narratives in a mutual context, connection, or comparison (see Figure 6).

Space–time cube representations allow the direct construction of a model of a historic actor’s behavior and visual analysis of a multitude of movement patterns [63,64], but they also have some analytical limitations (see Table 1). Therefore, if visualization systems want to offer multiple alternative perspectives on spatiotemporal data—how can we maintain their orientation when switching from one perspective to another?

4.2. Supporting the Cognitive Integration of Multiple Perspectives

Bennett and Flach [65] define the extent to which an interface supports the users’ transitions between different perspectives or information activities as “visual momentum.” One of the most effective design strategies to increase visual momentum and enhance the “cognitive coupling” between different views is traceable transformations of arrangement principles, which work like movements of perspective-taking and changing in natural environments.

These seamless transitions work like a coherence technique [55] that connects one view with another by making explicit changes—but also constancies—between different perspectives. Various forms of morphing visually translate from one spatialization principle to another and, thus, help to build up a coherent representation of both views [30]. By changing layouts incrementally—as opposed to abrupt changes or hard cuts between views—the spatial rearrangement of the story-relevant elements into new constellations can be perceptually traced. These techniques correspond to the idea of preserving the mental map [66], the effort of keeping the number of changing elements to a suitable minimum, and the notion of making the new arrangement principles transparent.

With specific regard to the translation between spatiotemporal visualizations, Bach et al. [49] provide a conceptual key element: Space–time cube representations can transform into different spatiotemporal visualizations (including spatial or temporal perspectives only) by quasi-physical operations of flattening, cutting, traversing, or stretching (see Figure 7).

The PolyCube system supports these operations of alternative spatiotemporal perspective taking and cognitive coupling by seamless canvas transitions (for a video demonstration, see https://youtu.be/PTsk-NROJhU) [56,67]. Figure 8 illustrates two of these transitions, leading from space–time cube to layer juxtaposition, and from juxtaposition to a superimposition perspective. The expected cognitive effects on the macrocognitive integration and synthesis of insights, the orientation of the narrative analyst, and overall ease of use are currently being evaluated. In line with this endeavor, we see the development of coherence techniques (including hybrid views and seamless transitions) to be a research and design task of its own for complex interfaces, in order to better support cross-dimensional inferences and versatile model-based visual reasoning with narrative data [68].

5. Discussion

In this paper, we presented some basic assumptions about narrative cognitive processing and their implications for the design of narrative spatiotemporal visualizations. We outlined a novel design guideline and strategy to tackle the “representation challenge” of geovisualization: “To develop new forms of representation that support the understanding of geospatial phenomena and space-time processes. There is a need to take full advantage of technological advances that make it possible to […] generate complex multidimensional and dynamically linked views, merge representation with reality, and […] to develop methods that help users navigate within complex representations” [69] (p. 7). In the following section, we want to shortly discuss different related questions and challenges that mostly go beyond the core topic of this special issue, yet seem to be relevant for us, as they expand fields for further research.

5.1. Going beyond Space and Time

In this paper, we focused on the visualization of two event indices: space and time. However, what role do actors, events, intentions, and causation play in storytelling with geographic visualizations? On one hand, the displayed information content of the story here comes into play; in our case study of life stories, the protagonists pre-defined the main actors and we visualized their life events (as data points in the temporal and spatial information space). While the temporal chain of events already gives the recipient important cues on causes and intentions [19], the actual verbal description of these events will help the recipient to understand these further. On the other hand, we can also go beyond spatiotemporal data and visualize other aspects of a story: Persons or organizations (as collective actors), their relationships, as well as emotions and sentiments (Appendix D) are essential elements of a story, where established visualization techniques can assist recipients in gaining a broader picture of the story. Current reviews on text visualization [70] provide interesting resources for storytelling visualizations, going beyond space and time. In the following section, we will discuss different visualization techniques supporting distant reading on a story macro level (Section 5.2) and close reading on a story micro level (Section 5.3).

5.2. Distant Reading: Combining Geovisualization with Non-Geographic Visualization Techniques

What seems to be of utmost interest, from a more general visualization point of view, is the openness of the PolyCube approach to going beyond the geovisualization frontier and connecting with non-geographic information visualization methods [35]. This allows not only the maintenance of the visualization of geo-spatiotemporal movements as an analytical backbone for the investigation of narrative data, but also the ability to combine this perspective with other relevant imaging procedures. Figure 9 shows how biographical narratives can also be told and visualized against the background of non-geographic information spaces. Exemplarily, the left-hand side illustrates the path of an individual through the social–relational spacetime, generated by interaction, collaboration, or conflict [67], whereas the right-hand side shows how the same system can also visualize trajectories through a knowledge or topic space of a cultural domain [71]. In both cases, the spatiotemporal scaffold is maintained, while the geographic map on the data plane is substituted by a force-directed network graph or a hierarchically structured treemap. With regard to the event-indexing model (cp. Section 2), these methods also allow the development of visualization systems that support the comprehension and connection of other event indices than space or time, like actors and their interactions, as well as the development of sentiments and intentions, as long as they can be categorized and visually encoded into a non-geographic or structural topology.

5.3. Close Reading: Combining Visual Analysis with Textual Analysis

What also seems to be essential in the context of narrative visualizations is to offer optional access to textual representations of narrative data in parallel to graphic representations. This allows the analysis and “close-read” of a source text (or a visualization description or annotation), in comparison to diagrammatic representation, possibly including further supportive text visualization measures, such as coordinated highlighting or colored mark-up of textual entities (cp. [69]). Eccles et al. [3] showed how a system of coordinated multiple views can do this for space–time cube representations and, thus, provided one of many recent instances for the desideratum of a visualization system “that does not destroy the original text in the process” [72]. Another option for combining textual data with a graphic representation is to tell a story sequentially and incrementally on a textual basis, while zooming and panning to selections of a space–time path—as is already offered by tools like StoryMapJS (https://storymap.knightlab.com/) or ESRI storyteller (https://storymaps.arcgis.com/).

5.4. Visualizing Non-Linear Stories

What also becomes possible in the context of parallel views is the visual analysis of (non)linear story structures. Biographies are often told as chronological and linear stories in which the narrative order preserves and mirrors the story order of historical events [73]. In contrast, advanced storytelling frequently mixes things up and works with nonlinear narrative sequencing to create artful or surprising narrative arcs to convey a sense of mystery and tension and elevate the readers’ entertainment. As this narration technique also adds to overall sensemaking challenges, any combined depiction of story order and narrative order (e.g., by the use of dual timelines or the inclusion of textual data) allows the visualization of such nonlinearity of story development—and further adds to the users’ orientation and comprehension [74,75].

5.5. Going beyond the Situation Model—Narrative Effects and Drawbacks

As indicated in Section 2, the processing of narrative information is highly structured by cognitive schemata. Until now, we have only discussed the nature of the internal representation (situation model) and how it is constructed (event indexing). However, many further cognitive and motivational effects of narrative processing can be found in the literature, which are of interest for better understanding the effects of storytelling—positive and negative alike. On the positive side, existing story schemata reduce cognitive load and free cognitive resources, allowing us to process information more deeply and understand more complex information [76]. On an emotional–motivational level, identification with the story is a powerful mechanism, which can also result in narrative engagement, flow, and positive attitudes towards the topic [54]. On the negative side, incoming story information is checked only for plausibility and coherence, but not necessarily for truth [54], nor is it critically reflected. Therefore, it poses the risk of being misused for persuasive effects [77].

5.6. Evaluation Challenge

In their paper on cognitive and usability issues in geovisualization, Slocum et al. [42] emphasized that a usable interface should be built on a metaphor well-known by its users. Story schemata are similar to these metaphors in assisting the user (if activated) to process the information in a quick and efficient manner. But, how do we know that a story schema was activated? How can we know how well the user integrated geotemporal information in the story situation model?

A final, but nevertheless important, challenge is how to survey and evaluate situation models and whether they integrate spatial and temporal data. It is difficult to use the evaluation methods of narrative text comprehension research, as they often use relatively simple stories to single out the effects of interests (see also [78] on the relationship between psychological and cartographical research). When it comes to more complex visual stories, it is important to further develop these methods and combine them with qualitative methods (like the think-aloud technique) to develop a better understanding of the construction and enrichment of situation models from visual stories. Within the PolyCube project, we collected and reviewed a set of methodologies to assess internal representations resulting from the interaction with information visualizations [79]. Further research is needed to empirically evaluate these methodologies and identify the most suitable ones for the study of information visualization, in general, and of story situation models, in particular.

6. Conclusions

This special issue called for design guidelines and best practices for the development and deployment of expressive, perceptually salient, and cognitively supportive online geographic information stories. In this paper, we approached this question from a cognitive perspective and reviewed research on how stories are internally represented and cognitively constructed. The event-indexing model by Zwaan et al. [18] served as a reference for our discussion of geotemporal visualizations for storytelling: Each event of a story is indexed according to actors, their intentions, space, and time, and linked to prior events by causation. This model stems from research on narrative texts; although it was already transferred to other types of media (like films), we are not aware of publications that have transferred the event-indexing model to narrative visualizations. This model can serve as a reference for—and can provide guidance to—future empirical studies on this topic.

The event-indexing model suggests that the spatial and temporal information in a story can serve as a framework for the construction of the story situation model. We explored the design space in which (a) spatial and geographic information can be visually represented in an integrated fashion and (b) multiple perspectives can be coupled in a coherent manner. Standard techniques for visually representing spatiotemporal data (coordinated linked views, animation or slideshows, layer superimposition, juxtaposition, and space–time cube representations) all have different drawbacks and benefits from a cognitive perspective. To develop design guidelines, assisting visualization scholars and professionals in determining the representations that are best suited to supporting the smooth and synoptic construction of situation models (i.e., for which specific kinds of stories and users) is a topic for future research.

In absence of such guidelines and for more complex stories, we suggested offering multiple spatiotemporal representation techniques within one visualization system. We discussed techniques increasing the coherence between multiple views, e.g., by seamless transitions, and assisting users to cognitively integrate these visualizations and construct a coherent internal representation of the story from multiple analytical perspectives. As an outlook, we illustrated options to extend the geovisualization perspective and integrate non-geographic space-time cartographies to a synoptic visual-analytical environment of multiple coordinated cubes.

In a case study, we showed how this kind of visual analytics framework can support storytelling in history and biography contexts. As for further areas of application, we aim to explore how this framework also proves to be useful for visual storytelling in other (digital) humanities and social sciences domains, in order to shed light on a wide range of actor networks and their dynamics throughout the evolution of the human web [80].