Multimodal Technol. Interact., Volume 6, Issue 11 (November 2022) – 7 articles

Recognizing places of interest (POIs) can be challenging for humans, especially in foreign environments. In this study, we leverage smartphone sensors (i.e., camera, GPS) and deep learning algorithms to propose an intelligent solution to recognize POIs in an urban environment. Recent studies have approached landmark recognition as an image retrieval problem. However, visual similarity alone is not robust against challenging conditions such as extreme appearance variance and perceptual aliasing in urban environments. To this end, we propose to fuse visual, textual, and positioning information. Our contributions are as follows. Firstly, we propose VPR through text reading pipeline (VPRText) that uses off-the-shelf text spotting algorithms for word spotting followed by layout analysis and text similarity search modules. Secondly, we propose a hierarchical architecture that combines VPRText and image retrieval. Thirdly, we perform a comprehensive empirical study on the applicability of state-of-the-art text spotting methods for the VPR task. Additionally, we introduce a challenging purpose-built urban dataset for VPR evaluation. The proposed VPR architecture achieves a superior performance overall, especially in challenging conditions (i.e., perceptually aliased and illuminated environments). Full article

The incorporation of digital functionalities into consumer packaged goods (CPG) has the potential to improve our lives by supporting us in our daily practises. However, despite the increasing availability of data about their use, research is needed to explore how these data can be harnessed to create such digital enhancements. This paper explores how consumers can utilise data about interactions with CPGs to conceptualise their enhanced versions. We devised a data-inspired ideation approach, using data visualisations and design cards to facilitate the conceptualisation of enhanced CPGs. Analysing the role of data as expressed through participants’ comments and designs, we found that data served as a basis for the creation of unique concepts imbued with greater consideration for the experiences of others and attention to their own interests. Our study shows the value of empowering consumers through data to broaden and inform their contributions towards the creation of smart products. Full article

This article explains how the Replica project is a particular case of different professionals coming together to achieve the digitization of a historical photographic archive, intersecting complementary knowledge specific to normally unconnected communities. In particular the community of Art History researchers, brought together here in relation to their common methodologies in the practice of visual pattern research, became protagonists in the construction of a specific tool, the Morphograph, to navigate through the archive’s photos. A specific research problem, the recognition of visual patterns migrating from one work to another, became the key to developing a new technology initially intended for a specific community of users, but with such a generic character in its approach that it could easily be made available to other uninformed users as learning by doing tools. The Morphograph tool also made it possible to demonstrate how, within a community, the partial expertise of individuality needs to be related to each other and benefits enormously from the knowledge densification mechanism made possible by the sharing. The digital context easily makes it possible to create tools that are specific in terms of content but generic in form that can be communicated and shared with even diverse and uninformed communities. Full article

Technology may expand the reach of stress management to broader populations. However, issues with engagement can reduce intervention effectiveness. Technologies with highly social interfaces, such as virtual humans (VH), may offer advantages in this space. However, it is unclear how VH compare to telehealth and e-manuals at delivering psychological interventions. This experiment compared the effects of single laboratory session of Cognitive Behavioral Stress Management (CBSM) delivered by a VH (VH-CBSM), human telehealth (T-CBSM), and an e-manual (E-CBSM) on psychological and physiological outcomes in a community sample of stressed adult women. A pilot randomized controlled trial (RCT) with a parallel, mixed design was conducted. Adult women (M age =43.21, SD = 10.70) who self-identified as stressed were randomly allocated to VH-CBSM, T-CBSM, or E-CBSM involving one 90 min session and homework. Perceived stress, stress management skills, negative affect, optimism, relaxation, and physiological stress were measured. Mixed factorial ANOVAs and pairwise comparisons with Bonferroni correction investigated main and interaction effects of time and condition. Participants’ data (N = 38) were analysed (12 = VH-CBSM; 12 = T-CBSM; 14 = E-CBSM). Each condition significantly improved stress, negative affect, optimism, relaxation, and physiological stress over time with large effect sizes. No significant differences were found between conditions on outcomes. Overall, all three technologies showed promise for remotely delivering CBSM in a controlled setting. The findings suggest feasibility of the VH-CBSM delivery approach and support conducting a fully powered RCT to examine its effectiveness when delivering a full 10-week CBSM intervention. Full article

We investigated the effect of arm pivot joints that are typically used during haptic exploration by evaluating four joints of the human arm (metacarpophalangeal joint of the index finger, wrist, elbow, and shoulder joints). Using a virtual stiffness discrimination task, a four-session psychophysical experiment was conducted with 38 participants (25 male and 13 female); each session was conducted with one of the four joints as the pivot joint during haptic exploration. The participants were asked to judge the stiffness of the top surface of two computer-generated cylinders by determining the stiffer one while using their dominant hand’s index finger. A two-alternative forced-choice procedure was employed by assigning one cylinder a constant stiffness value of 1.0 N/mm (standard side) and the remaining cylinder a variable stiffness value (comparison side). Using a custom-made stylus for the Geomagic TouchTM (3D Systems, Inc., Rock Hill, SC, USA) haptic interface, the participants were able to feel the stiffness of these virtual surfaces only with their index fingers. It was observed that the average Weber fraction monotonically decreased as the pivot joint shifted toward the torso (i.e., a shift from the metacarpophalangeal joint to the shoulder joint); this decrease was not statistically significant, which suggests that the selection of the pivot joint was not a determining factor for the sensitivity to discriminate stiffness. In general, the palpation speed and force exerted by the participants on the standard side during the haptic exploration showed a tendency to increase when the pivot joint shifted toward the torso; the difference in average palpation speed and force across the pivot joints was not statistically significant. Full article

This paper explores the possibilities of motion capture as a tool to preserve and study Intangible Cultural Heritage (ICH) practices such as playing instruments. The Uilleann pipes are both an instrument and a culture with a now strong community following and recovering after being almost extinguished several times in Ireland. The playing and making of the Uilleann pipes was incorporated into the Representative list of the UNESCO Convention of Intangible Cultural Heritage in 2017. This experience was performed in collaboration with several Uillean pipe players who contributed at every stage of the performance recording with comments and orientation. Some of these comments were also later formally kept as interviews to the players. The technical capture of the movements was done using a Rokoko Smart suit and paired Smart gloves which the piper wears for the performance. The resulting motion file was then cleaned and redirected to Blender 3D, a community made software package that allows the incorporation of a renderable avatar that helps for the dissemination of the performance. This recording process, called Inertial system, allows performers to perform almost anywhere and to capture the movements of the players with good accuracy. This process of recording and collaboration with the community is a cost-effective solution that can be of particular interest for scholars as well as for cultural communities wishing to archive these practices quickly. This design of technology and collaborative recording allows for a round experience that combined the detail of the technically enhanced recording with the interpretive nuance of the player that enriches the capturing process with mentions to how it is ‘relatively’ comfortable for the player and how efficient it is in terms of resulting detail. This collaboratively designed experience also provides the three aspects of intangible heritage preservation: supports the community -who can learn from the resulting animation; helps situating the practice within the cultural practice of the community-as they are part of the process; and helps materialise the output permitting its digital cataloguing, archival, and storage. Full article

This paper advocates an epistemological turn in the field of digital art history and cultural heritage studies. This epistemological turn is understood as the elaboration of a new (or renewed) epistemic apparatus that allows us to understand and interpret cultural phenomena from the perspective of a different order of thought. This epistemological labor is conceived of as an «epistemological technical practice», which means integrating epistemological reflection and production into technical making and design. Within this framework of discussion, this paper introduces the idea of a techno-concept, which is defined as a co-production between the machine rationale and the human thought/imagination. As particular cases, this paper argues that the mathematical concepts of n-dimensional, vector and latent spaces constitute examples of techno-concepts that can be reappropriated and reworked for cultural analysis and interpretation. This paper offers a preliminary inquiry, in which certain epistemological propositions are exposed as open gates for further discussions. Full article