Accessible Visual Artworks for Blind and Visually Impaired People: Comparing a Multimodal Approach with Tactile Graphics

Abstract

Despite the use of tactile graphics and audio guides, blind and visually impaired people still face challenges to experience and understand visual artworks independently at art exhibitions. Art museums and other art places are increasingly exploring the use of interactive guides to make their collections more accessible. In this work, we describe our approach to an interactive multimodal guide prototype that uses audio and tactile modalities to improve the autonomous access to information and experience of visual artworks. The prototype is composed of a touch-sensitive 2.5D artwork relief model that can be freely explored by touch. Users can access localized verbal descriptions and audio by performing touch gestures on the surface while listening to themed background music along. We present the design requirements derived from a formative study realized with the help of eight blind and visually impaired participants, art museum and gallery staff, and artists. We extended the formative study by organizing two accessible art exhibitions. There, eighteen participants evaluated and compared multimodal and tactile graphic accessible exhibits. Results from a usability survey indicate that our multimodal approach is simple, easy to use, and improves confidence and independence when exploring visual artworks.

1. Introduction

Museums have traditionally employed several methods to make their collections more accessible in support of the participation of blind and visually impaired people in arts and culture and to comply with laws [1,2] that protect the right to access art. For example, some leading art institutions [3,4,5] offer accessible “touch tours” and workshops similar to Art Beyond Sight [6] and the Mind’s Eye Program [7] where participants can experience art by touching some of the collection artworks while listening to tailored audio descriptions given by the staff. Two additional methods to support access are descriptive audio guides and accessible Braille leaflets of the artworks that may include embossed tactile graphic diagrams. Unfortunately, these methods have limitations. Accessible tours and workshops are available only on specific dates, schedules, and often must be reserved in advance. Moreover, they fail to support independent visits, exploration, and the artworks prepared for touch exploration are not the most prominent collection pieces due to the risk of damage [8]. Audio descriptions and accessible leaflets fail to convey much of the spatial information in the artwork. The latter also requires Braille proficiency, which remains low even in developed countries (about 5% in the UK [9] and less than 10% in the USA [10]).

Nowadays, the development and display of relief models of artworks made using low-cost digital fabrication techniques such as 3D printing are becoming an alternative for improving the accessibility to art. Several art institutions like the Prado Museum [11] and The Andy Warhol Museum [12], among others, have pioneered the use of this alternative in their exhibitions. Compared to tactile graphic diagrams, they offer advantages like improved volume shape, depth, and more diverse texture representation. However, without any verbal descriptions, they might still be challenging to understand. Interactive multimodal guides (IMGs) combine modalities such as audio, tact, smell, flavor, or others to convey and communicate information. Doing so mitigates the individual modalities’ shortcomings and complements their strengths.

In this work, we describe our approach to the design, implementation, and evaluation of an interactive multimodal guide for blind and visually impaired people that uses localized on-demand audio descriptions and tactile relief models to improve the independent access and understanding of visual artworks.

Motivation and Objective

Several challenges prevent the adoption of interactive multimodal guides at art museums and galleries. One of them is the preservation efforts and prioritization of the primacy of vision to experience the art pieces [13]. Also, making and exhibiting models based on artists’ works may lead to ownership, copyright infringement, and artistic integrity arguments [14]. Furthermore, determining effective methods for accessible art representation is challenging. Motivated by these challenges, our objective was to develop an interactive multimodal guide and study its feasibility to improve accessible art representation compared to tactile graphics. Our main contributions are:

• A formative study performed with the help of eight blind and visually impaired participants, art museum and gallery staff, and two artists to understand the different needs of these stakeholders and the current state of the accessibility tools available to experience visual artworks.
• A low-cost alternative implementation of an interactive multimodal guide that enables blind and visually impaired people without previous training to independently access and experience visual artworks.
• In collaboration with an accessible art gallery and a school for blind and visually impaired people, we performed two art exhibitions using the proposed guide. Within those exhibitions, we performed a survey with eighteen blind and visually impaired participants to compare the proposed interactive guide and a tactile graphics alternative.

2. Related Work

2.1. Tactile Graphics

Tactile graphics (TG) are made using raised lines and textures to convey drawings and images by touch. They are frequently used by blind and visually impaired people because the tactile modality is the best for their graphical image comprehension [15]. Their use is recommended where spatial relationships among the graph’s objects are important [16], such as simple graphs, diagrams, and drawings. Unfortunately, they are ineffective to express visual information of complex images [17,18], such as those present in many visual artworks. For this case, adding Braille labels is of limited use due to the large space needed by the Braille characters to be legible. Moreover, including labels within the artwork area obstructs exploration. Advances in low-cost prototyping and 3D printing technologies bring the potential to tackle the complexity of expressing complex images without exploration obstruction by adding interactivity to tactile graphics.

2.2. Interactive Tactile Graphics and 3D Models

In the last decades, researchers have explored the improvement of tactile graphics accessibility by adding interactivity through diverse technologies. Some of the improvements are better content exploration [18], learning facilitation [19], and expansion of the amount of information provided without over-complications [20].

Table A1. Interactive tactile graphics and multimodal guide projects.

Author - Name	Input	Output	Focus
Parkes [21] NOMAD	- Touch (Surface)	- Tactile overlay - Verbal descriptions	- Mathematics, Geometry, Geography, and Biology Education - Orientation & Mobility
Landau et al. [22] The Talking Tablet	- Touch (Surface)	- Tactile overlay - Verbal descriptions	- Mathematics, Geometry, Geography, and Biology Education - Orientation & Mobility
Gardner et al. [23] IVEO	- Touch (Surface)	- Tactile overlay - Verbal descriptions	- Education & Scientific Diagrams
Taylor et al. [24]	- Touch (Touchscreen)	- Tactile overlay - Verbal descriptions	- Orientation & Mobility
Gotzelmann et al. [25] LucentMaps	- Touch (Touchscreen) - Voice	- Tactile overlay - Visual agumentation	- Orientation & Mobility
Brule et al. [26] MapSense	- Touch (Touchscreen) - Tokens (Capacitive)	- Tactile overlay - Smell and taste infused tangible tokens - Verbal descriptions	- Geography Education - Map Exploration - Orientation & Mobility
Shen et al. [27] CamIO	- Touch (Mounted camera)	- Tactile graph - Tactile 3D Map - Tactile Object Verbal descriptions	- Access to 3D objects - Map Exploration - Access to appliances - Access to documents
Baker et al. [28] Tactile Graphics with a Voice	- Touch (Mobile Camera)	- Tactile graph - Verbal descriptions	- STEM Education
Baker et al. [29] Tactile Graphics with a Voice	- Touch (Wearable Camera) - Voice	- Tactile graph - Verbal descriptions	- STEM Education
Fusco et al. [30] The Tactile Graphics Helper	- Touch (Mobile Camera) - Voice	- Tactile graph - Verbal descriptions	- STEM Education - Map Exploration
Holloway et al. [31]	- Touch (Embedded capacitive sensors)	- Tactile 3D Map - Verbal descriptions	- Orientation & Mobility
Vaz et al. [42]	- Touch (Embedded capacitive sensors)	- Tactile Objects - Verbal descriptions - Visual augmentation	- Museum Object Exploration
Anagnostakis et al. [41]	- Touch (PIR and touch sensors)	- Tactile Objects - Verbal descriptions	- Museum Object Exploration
Leporini et al. [43]	- Touch (Physical buttons)	- Tactile 3D Map & Model - Verbal descriptions	- Archeological site exploration - Artwork exploration
Reichinger et al. [44,45,46]	- Touch (Camera) - Hand gestures (Camera)	- Tactile 3D Artwork Model - Verbal descriptions	- Artwork exploration
Landau et al. [35] The Talking Tactile Pen	- Touch (Pen device)	- Tactile graph - Verbal descriptions	- STEM Education - Map Exploration - Games
D’Agnano et al. [36] Tooteko	- Touch (Ring NFC reader)	- Tactile 3D model - Verbal descriptions	- Archelological site exploration - Artwork exploration

summarizes several of these projects and their interaction technologies. Three early works are NOMAD [21], The Talking Tablet [22], and IVEO [23], all of which function by placing a tactile graphic on a high-resolution touch-sensitive pad that detects user touch gestures that trigger audio descriptions. This method provides independent and detailed access to graphic elements, and since it does not rely on Braille, the possible audience is broader. Taylor et al. [24] and LucentMaps [25] make use of the touch screens in mobile devices to detect user–touch interactions in a portable way. They attach 3D printed tactile overlays of city maps to the device screen. Taylor et al. [24] 3D print sections of the overlay using conductive filament to provide interaction points on discrete sections of the map. LucentMaps instead uses translucent filament for their overlays coupled with a mobile application that visually highlights sections of the overlay using the device screen. MapSense [26] also uses a touchscreen to identify user touch gestures and conductive tangible tokens placed on the surface. The tangibles are additionally infused with smell and taste to foster reflective learning and memorization. Using touch-sensitive surfaces to detect user input and trigger audio feedback increases the amount of information communicated to the user. However, this approach is limited to thin overlays. Otherwise, the system can’t recognize the touch gestures.

An alternative approach is using cameras to track either the content or the user’s hands. CamIO [27], Tactile Graphics with a Voice [28,29], and The Tactile Graphics Helper [30] are examples of projects using this approach. The Tactile Graphics with Voice projects work by using a mobile or wearable device’s camera to identify QR codes printed along a tactile graphic. Then, the system tracks the user’s hand to trigger localized verbal descriptions. CamIO and The Tactile Graphics Helper use mounted cameras that identify the content using image processing algorithms, instead of using QR codes or visual markers. With the exception of CamIO, the previous projects focus on adding interactivity to 2D tactile graphics, and mainly propose their use for STEM (science, technology, engineering and mathematics) education and orientation and mobility improvement. Both approaches are effective to improve the amount of information and the comprehension of the spatial arrangement of images. However, to facilitate comprehension, they abstract the complexity of images to contour lines, which hinders the aesthetic aspect and exploration of artwork images.

3D printing opens up the possibility to create low-cost reliefs and 3D models of objects with added expressive volume. Holloway et al. [31] propose a touch interactive prototype that uses 3D printed volumetric representations of map models embedded with discrete capacitive touch points that users can touch to trigger audio descriptions. This approach improved the short term recollection and the understanding of the relative height among the map elements. Other studies focused on symbolic representation on 3D maps models, like Holloway et al. [32] and Gual et al. [33,34], they report improvements in terms of accuracy, efficiency, and memorability compared to two-dimensional symbols. Alternative methods to add interactivity involve using other type of devices. For example, pen-shaped devices like The Talking Tactile Pen [35] or wearables like the ring-shaped Tooteko [36]. In this approach, the user must hold or wear the device, which can detect sensors embedded in the tactile graphic or models on approximation.

2.3. Interactive Multimodal Guides for Blind and Visually Impaired People

The body of work on interactive multimodal guides focused on artwork exploration is limited, as seen in Table A1. However, there are several related works. The American Foundation for the Blind offers guidelines and resources for the use of tactile graphics for the specific case of artworks [37]. Cho et al. [38] present a novel tactile color pictogram system to communicate the color information of visual artworks. Volpe et al. [39] explore the semi-automatic generation of 3D models from digital images of paintings, and classifies four classes of 3D models (tactile outline, textured tactile, flat-layered bas-relief, and bas-relief) for visual artwork representation. After an evaluation with fourteen blind participants, the results indicate that audio guides are still required to make the models understandable. Holloway et al. [14] evaluated three techniques for visual artwork representation: tactile graphic, 3D print (sculpture model), and laser cut. Notably, 3D print and laser cut are preferred by most participants to explore visual artworks. Hinton [40] describes the use of tactile graphics of visual artworks made using thermoforming intended to be explored along with tape recordings. Blind study participants reported that the approach helped them understand the space and perspective of the artworks and found the approach fun, interesting, informative, and even stimulating to their creative efforts.

There are the few projects that add interactivity to visual artwork representations and museum objects. Anagnostakis et al. [41] use proximity and touch sensors to provide audio guidance through a mobile device of museum exhibits. Vaz et al. [42] developed an accessible geological sample exhibitor that reproduces audio descriptions of the samples when picked up. The on-site use evaluation revealed that blind and visually impaired people felt more motivated and improved their mental conceptualization. Leporini et al. [43] explore the use of a three-dimensional archeological map and fascade models to communicate historical, practical, and architectural information on demand, using 3D printed buttons with success to provide autonomous and satisfying exploration. Reichinger et al. [44,45,46] introduce the concept of a gesture-controlled interactive audio guide for visual artworks that uses depth-sensing cameras to sense the location and gestures of the user’s hands during tactile exploration of a bas-relief artwork model. The guide provides location-dependent audio descriptions based on the user’s hand position and gestures.

We designed and implemented an interactive multimodal guide prototype based on the needs found through our preliminary study described in Section 3.1 and inspired mainly in the related works Holloway et al. [31] and Reichinger et al. [44].

Table 1. Features of the proposed interactive multimodal guide and selected related works.

Author	Description
Halloway et al. [31]	- Sensing technology: Capacitive sensor board connected to discrete copper interaction points placed on the surface of the model. - Input: Double tap and long tap gestures on the surface. - Tactile presentation: Tactile 3D map model. - Output: Audio Descriptions. - Objective: Improve Mobility and Orientation.
Reichinger et al. [44,45,46]	- Sensing technology: Color and depth mounted camera. - Input: Tap gestures on the surface and hand gestures above the surface. - Tactile presentation: Tactile bas-relif model. - Output: Audio Descriptions. - Objective: Improve visual artwork exploration.
Cavazos et al. *	- Sensing technology: Capacitive sensor connected to conductive ink-based sensors embedded under the surface of the model. - Input: Double tap and triple tap gestures on the surface. - Tactile presentation: Tactile bas-relief model. - Output: Audio Descriptions, Sound effects, and Background music - Objective: Improve visual artwork exploration.

* This work.

compares the main technical differences between the related works and our approach. Besides these differences, this work introduces a comparison between our approach and using traditional tactile graphics to measure potential improvements of the multimodal approach.

3. Materials and Methods

3.1. Formative Study

To better understand the current state of the accessibility tools available to experience visual artworks and to explore the requirements for the use of interactive multimodal guides, we conducted a formative study with blind and visually impaired participants, art museums and gallery staff, and artists.

3.1.1. Accessible Visual Artworks for Blind and Visually Impaired People

The formative study focused on the current access to visual artworks through tactile graphics and other means with eight blind and visually impaired participants, with an average age of 29.13 (standard deviation of 7.7). Other characteristics of the participants are described in

Table 2. Characteristics of blind and visually impaired participants in our formative study.

Participant	Sex	Age	Occupation	Sight
FP1	Female	24	University student	Total vision loss
FP2	Male	40	Worker	Near vision loss
FP3	Female	42	Worker	Total vision loss
FP4	Female	30	Worker	Profound vision loss
FP5	Male	27	University student	Near vision loss
FP6	Male	24	University student	Total vision loss
FP7	Female	23	University student	Total vision loss
FP8	Female	23	University student	Total vision loss

. Of the eight participants in the study, three (37.5%) are male, and five (62.5%) are female. While five (62.50%) of the participants attend university studies, three (37.5%) of them work. All the participants gave signed informed consent based on the procedures approved by the Sungkyunkwan University Institutional Review Board.

We followed a semi-structured interview focused on the access and availability of tactile materials at museums, galleries, and through their education. Moreover, we inquired about their experience when using tactile graphics and interactive guides, if any. While all the participants stated having experience using tactile graphics, most of the encounters with this type of materials were limited to educational materials and tactile books during their early education or related to STEM subjects and maps. Four participants stated having experience with tactile graphics related to visual artworks. All the participants that said having experience with tactile graphics in the art fields had access to them during their primary and secondary studies. Only two mentioned having experienced them during a visit to a museum or gallery. All of the participants expressed having visited a museum or art gallery; they reported that the most common accessible tools during their visit were guided tours and the use of audio guides. Seven of the participants mentioned that they were accompanied by someone (relatives or friends) during their visits. They added that they mostly relied on that person’s comments and help to use the audio guide during their visit to experience the artworks.

Regarding their experience exploring tactile graphics, the participants mentioned that they are convenient to understand simple diagrams of mathematical concepts or simple graphics in educational fields, learning language characters, and storybooks. Mixed results were reported in their use for tactile maps. Three participants considered tactile graphics easy to understand, while five found them over-complicated or not very useful. However, all of the participants with previous experience with tactile graphics of visual artworks stated dissatisfaction due to their limitations. In particular, one participant commented: “FP2: Using the tactile graphics is a hit and miss. If the contents are simple and separated is easy to get an idea of what the picture looks like, but often there are so many shapes and textures that is difficult to imagine what the picture looks like, it becomes hard, like thinking about math, art is not supposed to be like that." This reflects the known problem of producing tactile graphics of complex images, which is usually dealt with by simplifying and abstracting the objects in the image. However, this approach often doesn’t solve the problem in the case of tactile artworks. “FP3: So much detail is lost when touching a tactile graphic. Even if I can find and feel the silhouette of a person or their face, I cannot know if the person in the painting is smiling or crying, and that’s what people usually talk about." Another problem is the challenge to represent perspective and volume. “FP3: When exploring a tactile graphic everything is on the same level, there’s no depth like in the real world. If it’s a landscape, I don’t know what is in front and what’s on the back. Even something simple like a ball, I only feel a circle, and many things can be a circle. I’m told that in the painting you can know it’s a ball because of the color and shadows, but I just feel a circle." Despite the shortcomings, the participants expressed the need for tactile graphics and desired for them to be available for more artworks and more locations. “FP5: Even when they are not perfect (tactile graphics), they are still useful to know what is where in the painting, I still can be in the conversation. I just hope they were available in more places and for all the works."

3.1.2. Accessible Visual Artwork at Art Museums and Galleries

Some of the participants in the formative study mentioned the shortage in the availability of tactile graphics or other accessibility tools in their visits to art museums and galleries. We met with a couple of administrators and curators at a national art museum, a private art gallery, and an accessible gallery at a social welfare center for blind and visually impaired people, to shed some light on their approach and efforts towards the accessibility to their collection. At the national art museum, they described several of their initiatives towards accessibility. Their current effort is mostly directed to accessible tours. Besides the tours and available audio guides, some of their exhibitions are made accessible through 3D-printed models that can be explored by touch. However, this tool is not always available, and it is used mostly for large modern art installations. The private gallery just offered guided tours by its staff. There were two main concerns. First, any accessible tool or display must be unobtrusive. One of the concerns was that any display co-located with the artwork can become a distraction and deviate the attention from the artwork. The second concern is about the contents. The administrators commented that presenting the artwork through a different medium than the one used by the artist could have implications in the message and intention that the artist wanted to express. Because of this, the use of accessible exhibits is more often available for modern artworks, where the artist can provide guidelines or collaborate in the development of the exhibits or even make their artworks considering accessibility needs.

3.1.3. Accessible Visual Artwork and Artists

We interviewed two artists separately to inquire about the use of accessibility tools and other mediums to experience their art. To generate richer insights, we provided one tactile graphic representation of a painting and discussed it with them. Both artists agreed on the importance of making visual art more accessible to blind and visually impaired people and that it may require the introduction of other tools or mediums. To this end, they strongly suggested collaboration with the author or experts when possible, noting that while the artist may not be an expert on the added medium, it can provide feedback to improve it. One of the artists expressed his concern regarding tactile graphics “Artist 1: I believe too much emphasis is placed on what is in the painting and not the painting itself. Yes, the recognition of shapes, objects, colors, and elements is relevant, but I dare to say it is not the most important aspect. Viewers should not be passive, just saying to them ’this is this’ or ’this means this’ is a failure. The goal of my art is to cause a reaction when someone sees it, they (viewers) should think, they should react. That’s what experiencing art is." We believe that this is a very relevant point, since most of the research literature is centered in the improvement of recognition of the objects in the painting, but there is almost no improvement related to the reaction and interpretation studies when using accessible artwork guides.

3.1.4. Design Requirements

Based on the feedback obtained during the formative study, we identified the following design requirements to develop our interactive multimodal guide. Independent exploration is the most important need derived from the formative study. It is largely derived from two factors, adequate access to the artwork and the information presentation method to facilitate understanding and experience. To improve it, the IMG should tackle the following:

• Simple to learn and use. The guide should offer a low entry barrier to the user. It should avoid the need for Braille literacy for operation and exploration to improve the access for blind and visually impaired people without or limited Braille literacy. It should avoid, as much as possible, the need for training or previous experience for its operation. For example, using a limited set of intuitive and well-known interaction gestures and interfaces to avoid cognitive load.
• Self-contained. The guide should avoid requiring blind and visually impaired people to carry external devices or install software on their own. Blind and visually impaired visitors often already carry several items such as a personal bag, white cane, leaflets, and audio guides. External devices add to their carrying load, add the need to check-in and out the device, as well as to learn the device operation and interface.
• Facilitate access to information. Exploring the artworks by touch is essential to understand the spatial arrangement of the artwork. The design of the model should be simple and abstract enough for easy comprehension, while avoiding oversimplification. Audio descriptions should be detailed but not long. Users should be able to skip them if desired.
• Promote active engagement. The IMG should promote active user engagement by facilitating exploration rather than just providing information. As much as possible, the guide should encourage critical thinking, reflection, and emotional responses.
• Unobtrusive and versatile. The guide should avoid being obtrusive to the original artwork within an art museum and gallery environment such that it can be colocated and avoid user isolation. The IMG should be able to support different artwork styles, sizes, and shapes.

3.2. Interactive Multimodal Guide (IMG)

Based on the design requirements that we identified from the formative study to address the limitations of tactile graphics and audio guides, we decided to develop an interactive multimodal guide. Our IMG will use a combination of tactile and audio modalities to communicate information and promote the exploration of visual artworks such as paintings. The tactile modality is covered by employing a 2.5-dimensional bas-relief model representation of the visual artwork. This model is accessible by touch and will convey the spatial and composition information of the artwork and will be the primary input interface of the IMG. The audio modality will be delivered through speakers or headphones and will include: narrations, sounds, and background music to convey iconographic and iconological information. The following subsections will cover the implementation of the several components of our proposed IMG.

3.2.1. 2.5D Relief Model

Users of the IMG can touch the 2.5-dimensional model to get an idea of the objects, textures, and their locations in the artwork. The main difference between a tactile graphic and a 2.5D model is that the latter can provide depth perception by giving volume to the objects in the model. There are several techniques to extract the topographical information from artworks like paintings to make a 2.5D model. Three of them are 3D laser triangulation, structured light 3D scanning, and focus variation microscopy [47]. The advantage of these techniques is that they are highly automated and provide close to exact information to reproduce the artwork’s surface. Blind and visually impaired people using a model designed using these techniques can perceive the direction of the strokes made by the artist, but often cannot recognize the objects. Only artworks made with simple strokes or rich in textures like splatter, impasto, or sgraffito are good candidates to be experienced with models designed using these techniques. Instead, we decided to use a semi-automated hybrid approach combining a technique known as shape from shading (SFS) [48]. SFS only requires a single image of the painting to generate the depth information to create a 2.5-dimensional model [49]. We chose this technique for three reasons: First, we do not need to have direct access to the artwork. Only a high-resolution image of the artwork is required to generate the depth information. Second, the process is automated and does not need specialized equipment like stereo cameras. Third, the output of the process is a greyscale height-map image that can be easily modified with any image editing software for corrections, or like in our case, to abstract, simplify or accentuate features and objects on the image. The process to design a 2.5-dimensional relief model to use with our IMG is graphically described in Figure 1 and is as follows:

• A high resolution picture of the visual artwork is taken or obtained. Figure 1a.
• The picture is processed using the SFS based methodology proposed in Furferi et al. [50] to obtain a grayscale height-map. Figure 1b.
• The height-map image is modified using a digital image software to correct, modify, abstract, simplify or accentuate features and objects in the painting to improve their legibility and recognition by blind and visually impaired people.
• A three-dimensional model is generated from the original picture and the height-map image using the ’Embossing Tool’ in the ZW3D 3D drawing software. Figure 1c.

Once the digital model of the relief model is ready, there are several methods to produce it. We chose to 3D print it using a fused filament fabrication 3D printer due to the variety and low cost of the materials, as well as the popularity and production services available (Figure 1d). It is also possible to 3D print the model using other 3D printing methods, as long as the material is non-conductive. Such methods are selective laser sintering (SLS) or stereolithography (SLA), which offer improved printing resolution at a cost trade-off. Another alternative is to use a CNC mill to carve the model out of a solid block of material.

The relief model is the primary input interface of our IMG. The touch interactivity on the relief surface is implemented by treating the surface with conductive paint. Conductive paints are electrical conductive solutions composed of dissolved or suspended pigments and conductive materials such as silver, copper, or graphite. We chose to use a water-based conductive paint that uses carbon and graphite for their conductive properties because of its easy to use, safe, and low cost nature. For our IMG, we used electric paint by bare conductive, but there are other suppliers in the market, as well as online guides to self-produce it.

Once the relief model has been 3D printed, making touch-sensitive areas is a simple procedure that only requires painting the areas that must be sensitive using conductive paint. The only requirement is to be careful to paint each touch-sensitive area isolated from the others, as seen in Figure 1e. If two treated areas with conductive paint overlap, they will act as one. The conductive paint dries at room temperature and does not require any special post-processing. One limitation of this method is that while extending or adding zones to the relief model is as simple as painting more areas or extending the existing ones, reducing or splitting existing ones is a more complicated process that involves scrapping or dissolving the paint. Therefore, it is recommended to plan the location and shape of the touch-sensitive areas. Each sensitive area must be connected with a thin conductive thread or wire to the circuit board. To this end, holes can be included in the model design before production or be made using a thin drill. Once the process is complete, the relief model can be sealed using a varnish or coating, preventing smudging and acting like a protective layer. It is possible to add subsequent layers of paint to produce a range of more aesthetic finishes, like a single color finish, a colored reproduction (Figure 1f), or different color palette combinations to improve visibility.

3.2.2. Control Board

The control board is the processing center of the IMG. It receives the touch sensor input from the 2.5D relief model described in Section 3.2.1 and peripherals, processes the signals, and provides audio output feedback. The control board is primarily composed of three components: An Arduino Uno microcontroller (Arduino, Somerville, MA, USA), a WAV Trigger polyphonic audio player board (SparkFun Electronics, Boulder, CO, USA) and an MPR121 proximity capacitive touch sensor controller (Adafruit Industries, New York, NY, USA). The wire leads from each of the touch sensitive areas of the relief model connect to one of the electrode inputs of the MPR121 integrated circuit. The MPR121 processes the capacitance of each of the touch areas in the relief model, which changes when the users touch the area, and it communicates touch and release events to the microcontroller through an I2C interface. One MPR121 integrated circuit is limited to 12 electrodes. It can only handle input for up to 12 touch areas. While this was enough for our prototypes, if more touch areas are required, up to four MPR121 can be connected by configuring different I2C addresses for a total of 48 touch areas. If more areas are needed, an I2C multiplexer, such as the TCA9548A (Adafruit Industries, New York, NY, USA), can be used to extend the number of supported touch zones. The microcontroller acts as the orchestrator of the control board. It receives input signals from the MPR121 and its general purpose input/output ports, processes them, and depending on the current state of execution, issues commands through its UART port to the audio board to trigger audio feedback. The WAV Trigger polyphonic audio player is a board that can play and mix up to 14 audio tracks at the same time and outputs the amplified audio through a mini-plug speaker connector. The audio files are read from an SD card and should be stored using WAV format.

3.2.3. External Hardware

Besides the relief model and the control board, the IMG is composed of an enclosure display. The enclosure was designed for different exploration scenarios. For example, for our preliminary test, a portable box-shaped enclosure is made of laser-cut acrylic. The box itself acts as an exhibit, the relief model is on its top surface, and the control board and electronics are in its interior. Headphones or speakers are connected to listen to the audio, and there is a button that the user can push to start using the IMG prototype. This prototype is meant to be placed on a desk to be used in a seated position during the early preliminary tests to make its use more comfortable for longer periods. For the IMG evaluation, we designed an exhibition display made of plywood for standing up use, as this is the more frequently used display arrangement in art museums and galleries. This version includes three physical buttons with labels in Braille to listen to use instructions, general information of the artwork, and to change the speed of the audio. Headphones are on the right side of the display. Depending on the size of the relief model or the floor space of the gallery, it might be difficult to explore the relief model if it is displayed horizontally or at a near angle, so a full-size vertical display was also developed, as seen in Figure 2c.

3.2.4. Interaction Design

Since there is no standard for interactive relief interfaces, and users are likely to lack previous experience with them, it is important to carefully design the interaction so that using the IMG is intuitive and easy to learn. A session with the IMG starts with the user already located in front of the display. The first task is to wear the exhibit’s headphones. The exhibition stand only has a label in Braille inviting the user to wear the headphones and indicating their location. This is a barrier for blind and visually impaired people with limited Braille literacy. While it is possible to trigger a speaker to inform the user about the location of the headphones using a proximity sensor, from our user test experience, just verbally informing the user one time and maintaining consistency on the location is enough for users to find and wear the headphones independently across different exhibition stands. In our prototypes, we maintained consistency, by placing the headphones hanging on a hook at the right side of the exhibition display.

The interactive session with the IMG starts when the user either touches anywhere on the relief or presses the “Instructions” physical button on the surface to the right of the artwork relief model. At the beginning of the session, the user listens to a short instruction recording that suggests exploring the relief using both hands. Then, it instructs the user to double-tap to hear more localized detailed information about any point of interest in the relief model or triple tap to listen to localized sounds. The recording also introduces the functionality of the other two physical buttons on the surface. The “General description” button provides general information about the artwork. The “Audio Speed” button changes the speed of the audio narrations. The “Instruction” and “General description” narrations can be interrupted any time another button is pressed or by double or triple tapping on the relief model. This is intended to give freedom to the user to skip the narration if desired.

3.2.5. Information Hierarchy

To provide intuitive artwork information access, we divided the information into two layers:

• General information: Refers to the general information of the artwork such as name, author, short visual description, and any information that is not already present in the artwork or related to information that can be accessed in a single point of interest.
• Localized information is information related to a specific point of interest in the artwork such as the object name, detailed description, color, meaning, and their relationship with neighboring points of interest and their sound, among others.

The general information narration of the artwork is accessed only through the physical button on the IMG. Localized information is accessed by double or triple tapping on any of the points of interest in the relief model. Mapping the localized description to the point of interest being touched helps the user to relate what is touched (location, shape, and texture) to what is heard (localized information narration or sound). Sound design plays an important role in the IMG to communicate non-textual information. In collaboration with a music expert, background music was composed for each of the artworks to reflect the artwork’s general mood. This track is reproduced through the entire exploration session. Sound effects representing the objects in each of the points of interest are reproduced on demand. The objective of these sounds is to facilitate the formation of a mental image of the artwork, using familiar sounds instead of images like sighted people would do.

3.3. Evaluation

3.3.1. Accessible Exhibitions Using IMG and Tactile Graphics

We expanded our formative study to receive feedback on our interactive multimodal guide prototype and compare it with a tactile graphics approach as a reference.

3.3.2. Participants

We recruited eighteen participants for the study and divided them into two groups. We held the study with the first group of seven participants at an accessible gallery at a social welfare center for blind and visually impaired people. At a later date, we performed the study with the second group of eleven participants at a school for blind and visually impaired people. Participant age ranged from 15 to 52, with an average of 27.7 years. All of the participants had previous experience using tactile graphics and stated having an interest in arts. None of the participants took part in the formative study. Other characteristics of the participants are described in

Table 3. Characteristics of participants in our Standard Usability Scale evaluation study.

Participant	Sex	Age	Occupation	Sight
EP1	Female	16	High school student	Total vision loss
EP2	Female	16	High school student	Near vision loss
EP3	Female	19	High school student	Profound vision loss
EP4	Male	15	High school student	Total vision loss
EP5	Male	15	High school student	Total vision loss
EP6	Male	18	High school student	Total vision loss
EP7	Female	19	High school student	Profound vision loss
EP8	Female	16	High school student	Total vision loss
EP9	Male	17	High school student	Near vision loss
EP10	Male	18	High school student	Profound vision loss
EP11	Female	15	High school student	Total vision loss
EP12	Female	39	Worker	Total vision loss
EP13	Male	38	Worker	Total vision loss
EP14	Female	43	Worker	Total vision loss
EP15	Male	52	None	Near vision loss
EP16	Male	50	Worker	Near vision loss
EP17	Female	47	Housewife	Near vision loss
EP18	Female	45	Worker	Total vision loss

. All the participants or their legal guardians gave signed informed consent based on the procedures approved by the Sungkyunkwan University Institutional Review Board.

3.3.3. Materials and Apparatus

Two sets of test materials were prepared for the usability study. The IMG set is composed of five standing exhibition IMG prototypes similar to Figure 2b. Each prototype exhibits the 2.5D relief model of a distinct artwork from the selection in Figure 3. Since the participants may not had recently experienced visual artworks through tactile graphics, the second set of materials consisted of tactile graphics reproductions of the same artworks and was produced by a designer with extensive experience in the production of tactile graphics and reading materials for blind and visually impaired people. Descriptions of the artworks in Braille were provided side by side with the tactile graphics.

3.3.4. Methodology

The first group study was held at an accessible gallery located in a social welfare center for blind and visually impaired people. The gallery has a permanent accessible exhibition, and we were able to install our test materials and perform our study in a temporal gallery next to the main gallery, and arranged them as shown in Figure 4a. The second group study was held at a school for blind and visually impaired people. The materials were installed in the main hall of the school, as shown in Figure 4b. The study was performed in the absence of other people.

It began with a short introduction of our team and an interview with the participant to learn about their personal information, level of vision, interests in arts, and their experience at art museums and galleries. Participants were told that they would be experiencing visual artworks through different mediums and would be asked about their experience. A 2 × 2 Latin square test design was used to counterbalance the medium (tactile graphic or IMG) and presentation order, so that the participants would experience both mediums. The artwork selection was random among the five artworks prepared, and the participants responded to a standard usability scale survey immediately after each of the first two interactions with the exhibits. After the survey and a questionnaire, they could freely explore the rest of the exhibits. To replicate the experience that they would face at an art gallery, no training on how to use the exhibits was given to the participants. Only the location of the headphones in the IMG exhibit was communicated. Participants were able to freely explore the artwork exhibit for about ten minutes, after which, they completed the survey and moved to the next exhibit.

4. Results and Discussion

4.1. General Impressions

All the participants received the interactive multimodal guide and tactile exhibits well. The first impression of the IMG was much more exciting for the participants. They expressed surprise since, for most, it was the first time to use such a system, while reading tactile graphs was something they had already experienced. They eagerly expressed their desire to use both tactile graphics and IMG frequently at art galleries, and museums (

Table 4. Tactile Graphics and Interactive Mulitmodal Guide Exhibits Standard Usability Scale report.

1	2	3	4	5	M	SD
S1. I think that I would like to use this system frequently
		2	8	8	4.33	0.69
			9	9	4.50	0.51
S2. I found the system unnecessarily complex.
4	5	4	5		2.56	1.15
11	4	2		1	1.67	1.08
S3. I thought the system was easy to use
		5	8	5	4.00	0.77
			6	12	4.67	0.49
S4. I think that I would need the support of a technical person to be able to use this system.
2	2	5	7	2	3.28	1.18
9	8			1	1.67	0.97
S5. I found the various functions in this system were well integrated.
		2	6	10	4.44	0.70
			4	14	4.78	0.43
S6. I thought there was too much inconsistency in this system
4	6	5		3	2.56	1.34
2	7	3	3	3	2.89	1.32
S7. I would imagine that most people would learn to use this system very quickly
	2	3	9	4	3.83	0.92
		1	5	12	4.61	0.61
S8. I found the system very cumbersome to use.
4	5	4	5		2.56	1.15
12	5	1			1.39	0.61
S9. I felt very confident using the system
		7	9	2	3.72	0.67
			3	15	4.83	0.30
S10. I needed to learn a lot of things before I could get going with this system.
1	3	5	7	2	3.33	1.08
11	5	2			1.50	0.71
	Tactile Graphics
	Interactive Multimodal Guide

SUS score range from 1 (“Strongly disagree”) to 5 (“Strongly agree”).

-S1) and even demanded it, with expressions such as “EP13: I can’t understand why these (tactile graphics) are not available everywhere for every single artwork.".

The IMG was considered extremely easy to use (Table 4-S3), mostly for two reasons; because it requires almost no effort to start using it, “EP7: With this exhibit (IMG) you can feel the artwork from the beginning, you touch it, and it automatically starts telling things to you.", and because it is easier to access and confirm information about their point of interest in the artwork directly “EP11: I think one of the advantages is that with the speaking model (IMG), I can check what I’m touching by tapping two times right there, it is immediate. With the other one (tactile graphics), I need to go and read the Braille and come back, and sometimes I get lost in the graphic or with the Braille." Having to switch between the Braille annotations, texture legends, and the tactile graphics was perceived as the largest factor to perceive the tactile graphics as unnecessarily complex (Table 4-S2).

Participants found the functions of both approaches well integrated (Table 4-S5). Participants were already used to exploring tactile graphics accompanied by Braille annotations. The simple touch interface on the artwork relief of the IMG coupled with the localized audio descriptions was well received. The participants expressed that hearing the localized audio while touching the 3D model area helped them to create a better spatial image of the shape and location of the object to the canvas. A couple of participants perceived background music.

One of them reported two effects; the first was that it made them think about the atmosphere of the scene in the artwork and the second was that it made her wonder about the time and circumstances that the artwork was made. “EP8: When I heard the Korean traditional background music of the painting (Figure 3d) I could feel the solemnity of the painting and I wondered if the painter felt that way when making the painting".

All the participants expressed feeling very confident when using the IMG (Table 4-S10) because they could always revisit the points of interest quickly and trigger the audio descriptions or sounds to confirm the object that they are touching. For the tactile graphics, the opinion was divided between participants that felt very confident and those that didn’t because of the uncertainty of not being sure that they were correctly identifying the point of interest.

In general, the IMG was less cumbersome to use compared to the tactile graphics exhibit (Table 4-S8). Participants stated the following reasons: the difficulty of Braille, “reading Braille is more difficult than listening to a conversation", the cognitive load of switching between the tactile graphic and Braille annotations: “touching the object and getting its information is much better than having to read through Braille text and tactile graphics." which adds up with each session: “after trying several tactile graphics and Braille notes I felt more tired.".

4.2. Interaction

One of our design requirements was to make interaction with the IMG as simple to learn and use as possible. Requiring to remember the location and use of buttons as well as gestures or commands can be burdensome for most people since it will be the first time that they use a device. Moreover, many users often skip instructions, even if they are short. Because of this, the IMG only has three user interactions, pressing buttons with a single-use, and double and triple tapping on the relief model to access localized information and audio. A simple interaction interface has its benefits. It makes the system easy to learn to use (Table 4-S7) and avoiding the feeling of the burden that can come when facing a new device (Table 4-S10) as evidenced by one participant’s response, “EP8: With the talking exhibit you don’t need to know anything, you just stand there, touch something, and it starts talking to you about the picture." By keeping consistency throughout the IMGs, once a user knows how to use one IMG, it knows how to use the rest. Unfortunately, tactile graphics have drawbacks. Experience goes a long way to read tactile graphics proficiently, and every time the user faces a new tactile graphic, it will need to learn the meaning of the texture and line styles to recognize their meaning. As expressed by one of the participants, “EP15: You need to know Braille to read the tactile drawings with Braille and that takes time and effort.” Moreover, the lack of Braille proficiency affects the experience across all the exhibits, since the burden is on the user.

The participants reported a higher degree of inconsistency (Table 4-S6) for the IMG. Upon further investigation, we found out that it was due to a failure in some of the IMG prototypes to register some touch gestures correctly, causing the wrong audio feedback to trigger or not at all. Similarly, at the exhibition, not all the interactive zones in some of the artworks had ambient sound audio feedback, causing some users to believe that the system was malfunctioning or that their gesture was not recognized when they tapped the area and audio was not reproduced. Audio feedback should be added to the interaction zones that lack audio tracks, like empty or background space, to manage user expectations. Non-obtrusive audio or vibrotactile feedback could be added to help the user become aware that their input is sensing.

5. Conclusions and Future Work

In this work, we have presented the development of an interactive multimodal guide for improving the independent access and understanding of visual artworks. The IMG design was developed following the needs uncovered through a formative study in collaboration with people with vision impairments, art museums and gallery staff, and artists. Through an evaluation with eighteen participants, results demonstrate that the multimodal approach coupled with a simple to learn interaction interface is more effective in comparison to tactile graphics guides in providing independent access across a diverse style of artworks. Feedback collected during the multiple exhibition points in new directions for our work. As seen in Figure 5b, the IMG is sometimes used as a collaboration tool to socially interact with art. We would like to explore this possibility, as this could alleviate the perceived burden that some participants expressed when going to the art gallery with an acquaintance. Moreover, our current prototype was designed for use in an exhibition environment. Art educators at schools have expressed their interest in using the guide as an educational tool in class. To this end, more research is needed to explore the difference in audio description content and delivery methods to provide tailored information, while making it manageable for users with different content needs. The current prototypes only make use of tactile and audio modalities. We look forward to develop new experiences with other modalities such as smell, and explore how they might improve visual artworks exploration.