Digital Taste in Mulsemedia Augmented Reality: Perspective on Developments and Challenges

Abstract

Digitalization of human taste has been on the back burners of multi-sensory media until the beginning of the decade, with audio, video, and haptic input/output(I/O) taking over as the major sensory mechanisms. This article reviews the consolidated literature on augmented reality (AR) in the modulation and stimulation of the sensation of taste in humans using low-amplitude electrical signals. Describing multiple factors that combine to produce a single taste, various techniques to stimulate/modulate taste artificially are described. The article explores techniques from prominent research pools with an inclination towards taste modulation. The goal is to seamlessly integrate gustatory augmentation into the commercial market. It highlights core benefits and limitations and proposes feasible extensions to the already established technological architecture for taste stimulation and modulation, namely, from the Internet of Things, artificial intelligence, and machine learning. Past research on taste has had a more software-oriented approach, with a few trends getting exceptions presented as taste modulation hardware. Using modern technological extensions, the medium of taste has the potential to merge with audio and video data streams as a viable multichannel medium for the transfer of sensory information.

1. Introduction

The human body possesses the following five physical sensory systems: auditory, optical, olfactory, tactile, and taste. In the current age, there have been many developments to boost the quality of life by enhancing the sensory experience using artificially induced sensory stimuli targeting the senses of sight, sound, and touch. The experience is further made more immersive by combining those sensory modules and creating an integrated deep-dive system. Traditionally, artificial tastes are given to users through a chemical compound, either in a solid or liquid form. Example ingredients for the five basic tastes (sweet, bitter, sour, salty, and umami) are glucose for sweet, citric acid for sour, caffeine/quinine for bitter, sodium chloride for salt, and monosodium glutamate for umami. There have already been quite successful attempts at replicating this taste using electrostimulation, tackling the taste elements both individually [1] and in a collective configuration [2]. The sub-portion of food-texture replication has had work performed previously but was not followed up with more extensions and advancements [3]. Experimental food with indefinitely sustained taste has also been created and can be made commercially viable [4].

There are certain additional components of taste that have not been reached through electrostimulation, such as aftertaste, chilliness, pungency, and throat feel, since it is expected not to work through the same mechanism as taste buds. Further opportunities for exploration can be found upon conducting an in-depth review of the previous literature.

Recent studies conducted on electrically stimulating systems that are developed for inhibiting or enhancing certain gustatory features via ion transfer are described in Section 2 chronologically. The subject of recording the taste of food elements is also briefly explored. Section 3 delves into the recent research into IoT (Internet of Things) with the perspective of extending utility in the domain of taste-oriented Augmented Reality (AR) research. Section 4 deals with the recent works in the domain of artificial intelligence and machine learning that have been implemented into taste recognition tasks using classification algorithms. Section 5 discusses the existing technologies in depth along with possible extensions that can be merged into the existing taste stimulation methods to increase their efficiency. Lastly, the article draws out conclusive recommendations and their potential results based on the analysis of the prior art.

2. Gustatory Taste Stimulation

The earliest records of artificial stimulation of gustatory senses were in 2004, with a food simulator that recreated the biting force as portrayed by every food category. The two-step mechanism had an end-effector equipped with a sleeved pressure sensor to record the biting force and, subsequently, play it back artificially using end effectors with electronically variable force profiles [3]. Four years later, another study experimentally tested the influence of tactile feedback on the sense of taste by placing five swabs equidistantly over the tongue with sucrose and quinine sulfate instantaneously and five seconds post-contact. The experimental study aimed to establish that the tactile sense of the tongue supports the gustatory senses in extension to the taste buds. Three years later, another paper, in an attempt to extend the gustatory palette, introduced a novel hypothesis on whether the taste buds were capable of sensing more extended stimuli. The experimental study was conducted by constructing a combined olfactory and visual AR system (Figure 1) that would display a 6 DoF overlay-visual on top of the food item being consumed while the olfactory module released suggestive flavor-related odors [5,6,7].

The sense of taste is stimulated artificially in conjunction with the digitalization of haptic, visual, and olfactory feedback to obtain a thorough taste profile. This phenomenon has been explored, and the correlation between taste and smell has been quantified [6,8]. A principal component analysis conducted to distinguish liquid sample compositions yielded 100% results from the apparatus. Subsequently, the study featured the analysis of both high and low vapor pressure solutions, which were smell-biased and taste-biased, respectively, yielding the same 100% output [8]. In 2013, a thorough review of the commercially available taste sensors was conducted, and it was found that the artificial e-tongues from that year were capable of discerning astringency in addition to the five basic tastes.

Furthermore, it was stipulated that the pungency could be quantified in a short time [9]. The year 2016 witnessed a major upheaval in the domain of galvanic taste as the taste of sweetness was stimulated artificially by the “Digital Lollipop” by the same author who presented the tongue-mounted stimulation prototype. This mechanism had a customizable input galvanic signal, and the sweetness was induced using an inverse current mechanism [10]. The study employed electro-stimulatory means to modulate the sense of taste using only a single channel stimulation mechanism, as shown in Figure 2.

The following year, another inversion of the taste sensing device was presented in a form factor of a short color-changing bottle that would respond to the following three of the five basic tastes: sourness, saltiness, and bitterness, and would alter its shade to green, blue, and red for each taste, respectively [11]. The system used a microcontroller to deliver PWM signals to the buccal electrodes while outputting GPIO (General Purpose Input-Output) signals to the RGB LEDs, as shown in Figure 3.

A prototype tongue-mounted module for post-current release taste modulation was developed in 2012 and presented at the 16th international wearable computers symposium as a prototype that would take the form of a digital lollipop in its later stages of development [12,13]. Its taste-modulation output characteristics are depicted in Figure 4.

In a study in 2013, based on previous experimentation by Hettinger, a utensil-based approach was employed to build a salt-taste enhancer by using a cathodal current, and an experimental procedure was conducted to establish the long-term effects of the electrical stimulation and to see if it causes the other tastes to get enhanced as well. The experimental study attempted to create a new “electric taste” with the help of electrically energized utensils such as metal straws and forks powered by small batteries with their circuits closing through the mouth. The process behind the phenomena of taste inhibition via GTS (galvanic taste stimulation) was explored in 2017. It was hypothesized that the inhibited tastes of the five basic tastes from GTS were from the migration of their respective ions, which elicited the tastes [14]. Similarly, the whole process, in terms of physical hardware, from the specification of signal to the product range is sequentially documented and thoroughly detailed [15].

Up until 2017, GTS was performed by the provision of a galvanic charge from inside the buccal cavity. The galvanic jaw stimulation module was presented as an alternative to the earlier GTS, wherein the electrodes had to be thoroughly cleaned and disinfected before reuse. Using electrodes applied externally to the jaw on both sides, the modulation of the taste of brine and whether it was successful in inhibiting the saltiness was tested [16]. Using a collective array of sensory feedback modules, a portable AR system with a rechargeable battery unit was designed with a prime focus on portability and integration with smartphones and tablets [17]. An effort to remove the discomfort of wiring inside the mouth was made by experimentally crafting a galvanic taste-stimulating “gum” that is small enough to fit in the mouth with a piezoelectric element that powered it from biting force [4].

The influence of optical visualization over the gustatory interface has been experimentally examined for correlation in 2019. The proposed system uses an image-to-image translation to modulate the appearance of the food being consumed in real-time by implementing a generative adversarial network (GAN) in an AR setup with a limited taste modulating resultant effect observed upon the completion of the experiment [18]. It was initially termed ‘Deep Taste’ [19]. Besides using standalone methods for gustatory stimulation, another approach was tried out using the galvanic modules to test whether sending repetitive square waves through the contacts could enhance tastes other than the salty taste that had been observed before [20].

As an extension to taste modulation, the temporal effect of beverages was examined, and the longevity of their aftertaste was increased using electrode stimulation [21]. The throat feel can be replicated by muscle movement in the esophageal periphery. Its artificial replication has been recently tested by external skin deformation using the ‘grutio’ module [22].

To reproduce preexisting tastes, the initial step would be to record them with a bio-mimetic device called the ‘e-tongue’ that tests the concentration of taste-contributing electrolytic ions and outputs and stores the results in digital format [23]. A system that uses ion electrophoresis to emulate taste using six different electrolytes (five to emulate the five basic taste properties and one for control and current regulation) was coupled with taste-measuring instrumentation and a software GUI (graphical user interface) to reproduce the taste of pre-existing substances [24].

Research works in

Table 1. A brief overview of the key publications on the development of the gustatory sensor module’s hardware.

S. No	Ref. No./ Short Title	Outputs	Potential Extensions
1	[3] Food Simulator	A bite force measurement and replication device with assistive hints for recreating the texture of foods.	The contraption replicated the force of biting with audio and chemical feedback. It could be coupled with an AR overlay headset to replicate the complete experience of consuming virtual food.
2	[10] Digital Lollipop	Application of cathodal current using the body as the closed-circuit conductor for ion transfer. Causes the saltiness to increase upon release of the signal.	The form factor of the output electrodes could be altered to target multiple taste areas simultaneously. More channels could be added to test multiple stimuli.
3	[1] Controlling saltiness without salt	A single-channel bipolar device that is capable of anode/cathode discharge with custom output wave stimulus.	The nature of the conductive electrodes, their respective ion-taste and the toxicity could be experimented with to produce an optimum electrode.
4	[14] Galvanic tongue stimulation inhibits five basic	Externally applied jaw stimulation module for enhancing and inhibiting taste.	The whole system can be made into a compact wearable IoT AR-VR setup.
5	[23] Taste sensor: Electronic tongue with lipid membranes	An electronic tongue that measures taste-inducing electrolytic concentration in food and converts it to digital format.	The e-tongue requires a more compact form factor for mobile application. It could be built as a small embedded system with the lipid sensor in a smaller size as was built in 2013 [25].
6	[24] Taste display that reproduces tastes measured by a taste sensor	Software GUI that controls a 5-channel GTS module that is capable of reproducing any taste and calibrating it.	This system could be used in tandem with edge nodes such as AR visors.

provide a general idea of the bleeding edge research conducted upon taste stimulation. Next, we follow up by reviewing the recent literature on IoT from the perspective of its utility in the area of taste modulation. The domain of IoT shows potential in making the galvanic taste modulators portable, wireless, and backed by powerful prepossessing cloud units.

3. Internet of Things in Augmented Reality

Upon entering into multimedia as a viable medium, the data on the regulation of the modulator signals requires a suitable architecture built for modern transmission protocols and standardization of input/output modules. Since taste-based modulation became an oblique commercial gimmick after the advent of the internet revolution, it has become customary to conceptualize an IoT-compatible mulsemedia architecture. To fulfill this requirement, a four-layered IoT-architectural concept for mulsemedia data to be transferred through edge nodes was devised for immersive multimedia [26]. The quality of the overall experience of multi-sensory inputs in 360° multimedia was appraised in another study in pursuit of alternative extended approaches toward multimedia enrichment [27].

In addition to electrical taste stimulation, IoT can also be used to deliver AR multichannel data via the internet [13] to edge nodes that are capable of breaking it down into trigger inputs for multiple AR sensory modules, including taste [17]. Thermal changes are observed to have a considerable impression on the taste buds, causing the user to experience mild sweetness. An edge device was developed for future VR applications with the intent to pursue thermal taste as a viable sensory extension [28]. Tackling the software side, a web server capable of identifying and classifying three distinct taste characteristics (sweet, bitter, and sour) has been presented in a bid to boost taste-enabled I-T AR devices [29].

Besides taste being an output sensation, in the IoT it also serves as an input, with taste-sensing devices employing non-chemical means to ascertain the quality and composition of substances. There have been multiple use-cases wherein the quality of taste has been indirectly monitored by exploiting congruent variables in the process of monitoring food quality. For instance, the magnitude of soluble sugar content in fruit was tested using millimeter wave-permittivity as the congruent variable [30]. Similarly, in milk, the microbial concentration of Lactobacillus has been used to remotely test for quality [31].

Such alternative pathways prove crucial in areas where organic/chemical interference could affect the output negatively by a significant magnitude. Modern food processing focuses on quality through the precise monitoring of the handling and close surveillance of the assembly lines. Minimization of contact with the product has diverted the testing and monitoring procedures to look into photo-analytic testing methodologies such as thermography and photo-spectroscopy testing. In the case of wine, multiple testing venues have been explored, e.g., testing the water quality in plant leaves using infrared thermography [32], and the wine aging level using silicone-wrapped sensor nodes embedded into wine barrels [33]. The data gathered for these use-cases are transferred using an edge node-based IoT architecture. Once the data is obtained, it can be plugged into various algorithms to obtain ML- (machine learning) and AI (artificial intelligence)-based models to build taste recognition and classification systems.

4. Artificial Intelligence and Machine Learning in AR

Machine learning is an effective domain that can seamlessly merge with any other domain and provide much-needed support to enhance its productivity. When applied in AR, it can be used for the optimization of the output provided to the user by the sensors in various ways to ensure a higher degree of immersion. The hardware used in flavor recognition has been implemented in both edible and inedible products and can be implemented in a broader scope.

In an instance of ML being used for testing, the quality of water from natural water bodies was also tested by using datasets containing 54 attributes and 135 instances. The data was processed, and a confusion matrix was created and plugged into various machine learning algorithms, out of which the K-star algorithm performed the best, using only 6 out of the 54 features with an accuracy of 86.67% [34]. Similarly, it has been attempted to predict the contents of a juice using e-nose data to ascertain the elemental content of fruit juice [35].

Coupled with the data output from the e-tongue used to record taste-stimulating ion-electrolyte concentration parameters, an upgraded LDPP (local discriminant preservation projection) model approach was proposed in contrast to the earlier learning algorithms. It was put to the test along with PCA, LDA, LPP (locality-preserving projections), and LPDP (locality-preserving discriminant projections). The machine learning classifiers tested were the SVM (support vector machine), ELM (extreme learning machine), and KELM (kernelized ELM), displaying a maximum accuracy of 98.22%, as depicted in

Table 2. Average performance characteristics of classifiers used in conjunction with subspace projection learning algorithms over multi-beverage identifier e-tongue data [36].

Accuracy in Percentage Classifier Learning Algorithm	SVM	ELM	KELM
PCA	93	93.18	96.48
PCA (Kernelized)	89.49	87.35	91.23
LDA	94.74	94.5	97.35
LPP-S	93.87	94.94	95.61
LPP-H	94.74	94.84	95.61
LPDP-S	96.48	95.48	96.48
LPDP-H	97.35	96.51	97.35
LDPP-S	97.35	96.69	98.22
LDPP-H	98.22	94.56	98.22

S—using a Simple rule affinity matrix. H—using Heat kernel rule affinity matrix.

[36]. In another similar study, the classifiers used were KNN (K-nearest neighbor), PCA (principal component analysis), NB (naïve Bayes), random forest, EMCC (extended Matthew correlation coefficient), NER (non-error rate), and LDA (linear discriminant analysis), and the maximum accuracy reached was approximately 95% [37].

Using a data set of the physico-chemical parameters of red wine, various machine learning models were pitted against one another to pick out the ideal one for wine classification. Out of the SVM (simple vector machine), ANN (artificial neural network), RR (ridge regression), and GBR (gradient boost regression), the GBR performed the best in the classification of red wine [38].

Another article focusing on the taste characteristics of tea explored the viability of the combined multi-sensory data from e-tongue, e-nose, and camera modules to perceive the taste of tea. It compared three classification algorithms, namely, the 1v1 SVM, VVRKFA (Vector-Valued Regularized Kernel Function Approximation), and ANN, using high and low amplitude pulse and staircase voltammetric datasets with varying sparsity. The results unanimously yielded high accuracy using all three classifiers with low sparsity models [39]. Furthermore, various factors influencing the taste of tea, such as astringency, bitterness, and smell, have been analyzed throughout the course of the decade and have been thoroughly reviewed in a study in 2021 [40]. The study spanned various sensor arrays in combination, such as e-tongue, e-nose, and even computer vision, to perceive and classify tea samples.

Apart from the taste profile, the e-tongue module is also capable of testing the quality of water, which might display certain flavors due to suspended impurities. It used the PCA and the PLS (partial least square regression) methods to quantify the organoleptic analysis data regarding the dissolved impurities [41].

In addition to the taste recognition of edible food products, e-tongue sensors are also used to test the taste profiles of oblique substances such as amino acids. Applying simple linear regression over the data collected from the hydrophobic lipid sensor, the study derived a strong correlation between the sensory score and the values obtained from the lipid sensor [42]. Among several others. These studies followed the trend of taste recognition and classification in AI-ML research about the domain of gustation.

5. Discussion and Recommendations

Upon having explored the domains of IoT and AI-ML for finding applications in taste mulsemedia, a web of intertwined biomedical research pools was uncovered. Once analyzed with the perspective of extending AR utility, numerous potential use cases can be built. Since the connectivity provided by IoT modules brings us closer to the computational processing power of cloud servers, we can employ optimization, filtering, classification, and deep learning models to obtain information that is intrinsically derived from the heaps of data sent in by edge nodes of users’ AR helmets.

The introduction of novel hardware has the potential to redirect this trend toward optimization and filtration of taste signals. A network map of IoT coupled communication technology with AI-assisting AR modules of various types is shown in Figure 5.

The introduction of 3Dprinting and spatial scanning has made it possible to scan an individual’s buccal cavity and print a custom conductive sleeve to fit the user sans any discomfort. Moreover, the custom buccal sleeve would be much more effective with accurate electrode placement and limited movement and chances of slipping. In addition to the fitting comfort, it is also possible to fit an entire circuit into the 3D printed contraption, thus eliminating the chances of wires protruding out of the mouth, causing excessive salivation. The technology can be merged with a modified form of the food simulator that mimics the texture of the food being simulated along with the bite force required for chewing.

In addition to the dental brace serving as an output module for the gustatory simulation module, it could also function as an acquisition module for obtaining a tongue EEG. The spectrum of potential taste sensations that can be stimulated through this approach is still limited. Hence, in the early stages of using taste as a stimulation approach in HCI, chemical stimulation has the advantage of covering a broad range of taste experiences over galvanic stimulation. The most recent methodology of taste reproduction still requires external electrolytic solutions, while galvanic stimulation can merely inhibit/enhance an already consumed electrolytic ion cluster by pulling them together to boost taste density instantaneously. The hardware for detection of ion concentration may not be immune to frequent corrosion from reacting with other acidic compounds present in the food.

Limitations of the current prototype of the biting simulator include its cumbersome structure and the metal at the end of the biting end effector [3]. As a large force must be supported, the device must be fabricated in a structure that is capable of sustaining it while also being non-toxic. When the user has finished biting, he/she can feel an unnatural sensation of the thickness of the linkages. The weight of the linkage causes unwanted vibration. The user’s teeth contact the flat surface of the linkage. This flat surface degrades food texture. An individual feels the force on an independent tooth while biting into real food. Moreover, the current device applies force only to the teeth. The texture of real food is perceived, in part, by the tongue. However, displaying food texture to the tongue is very difficult.

In the case of the electric gum, for the best approach to solidify its foundations as a viable product, the piezoelectric material has to be analyzed for potential toxicity in a long-term use case, its reaction with saliva examined, and the long-term usage effects studied. The alternative research path would be to explore the potential for generating flavors other than the conventional saltiness from the separated ions. Alternative approaches, such as using real gum containing edible organic batteries using safer electrolyte gels and modulating its taste externally, could be tested out [4].

The apparatus used for the study of the effects of visual feedback on taste can be enhanced by combining similar studies to increase immersion. The delay in the deep taste system can be reduced by using an advanced FFT-GAN-based algorithm over the image data from the webcam. The resolution of the visor can be boosted too with a reduction in computational time [5]. Moreover, it can be merged with auditory and olfactory feedback from the original dish as an experiment on the percentage of effect olfaction has over taste.

Beverage taste modulation using a conductive straw to complete a stimulatory circuit could use an added perk in commercial food packaging in the form of flavor-changing drinks, with the downside of short shelf life due to oxidation of the conductive metal contacts. As a precautionary measure, the straw could be coated in plastic, save for the ends. The design of a sustainable GTS soda can (Figure 6) could be a potentially rewarding research pursuit. It would have great value in the patent sector while being relatively easier as a research option.

Currently, the optimum process of artificially recording taste is via measurement of taste-contributing ion concentration, and for classification, the ELM algorithm is ranked at the top. The research areas in the software portion of this particular domain are quite saturated, leaving behind either some potential in chemistry-based research, wherein better electrolytes could be sourced for longer-lasting electrode function, or in structural research, wherein the structure of the recording device could be tweaked to be more compact, portable, and multi-functional. The recording medium could feature a storage add-on that could be interfaced in the form of a flash drive, or it could be made wireless using a radio wave-based data transfer protocol, both long- and short-range.

From the perspective of taste being treated as media, it has to establish standardized data-flow protocols, hardware, and media before it can arrive at the same level as audio and video. Once the data from an e-tongue is obtained, its inversions may range from its channels being mapped to audio frequency bands to experience audio through multiple senses to test whether the patterns are as pleasing to taste as industrial approximation mediums. It could be used to plant more memories into the brain with a taste-based stimulus to trigger them, essentially making mankind a bit more “connected”.

Being the result of a multivariate experience, texture and memory also play a leading role when recognizing taste sensations. The other senses act as support vectors for the taste sensing system. To replicate the results, an accurate duplicate of the initial environment is required. A GTS system works on low-magnitude currents. Taste buds are localized, but their positions can be altered using galvanic electrostimulation. The initial step in the approach is to experiment with the low voltage levels based on personal threshold values. A gustatory range must be affixed within the experienced band of taste change sensation and is only obtained by testing for it.

The design of the stimulation module in prior use cases was not the primary area of focus in the experimentation. So, it could feature a more ergonomic approach in its structure. The single-channel stimulus can be tested to check for a better-distributed multichannel approach. Taste stimulation can be coupled with various areas of technology to generate more use cases in the commercial sector. The gustatory module can be molded into a portable, wearable module that employs state-of-the-art data transfer channels to integrate seamlessly with the most recent devices.

Wave-shaping techniques can be experimented with to create various taste profiles. Sensory analysis can be performed in real-time while tasting a dish, and the taste can be recorded to allow future replication.

6. Conclusions

The domain of taste electrostimulation holds a lot of promise vis-a-vis AR technology, with possibilities such as integration of the hardware’s system into digital multimedia, potential enhancements in memory retention, advancements in gastroenterological research, and many more. The IoT as an extension could provide the required processing prowess to optimize and filter the taste signal. Since the sense of taste usually provides an incentive for appetite, it is indirectly responsible for the amount of nutrients supplied to the body. The currently developed systems require ergonomic structural work to render the technology commercially feasible.

The perception of the utility of GTS systems is still narrowed down to usage in VR systems, rather than a broader radius of inclusivity. For instance, in pursuits such as gardening, farming, or even within the industry, wherein the percentage composition is based on an approach of approximation, the nutrient/threshold levels can be mapped onto different taste channels so that the practitioners can build sensory neural associations with the overall status of the unit. The development of taste stimulation devices is especially critical to the culinary sector, with its prospects extending into online taste sampling, taste copyrighting, establishing a distinguished gustatory range, and taste-based research.

Overall, the entire field of galvanic taste stimulation has tremendous trendsetting potential. This could subsequently tackle major flavor addictions without the user gaining any weight. An alteration of the taste component of nutritious foods to add more flavor could be performed using such modules in daily life. This would boost overall health. The average BMI index could be rigorously pursued using electronic gadgets instead of the tough mechanical weightlifting way. The non-genetic variation of obesity could be potentially eradicated. Moreover, taste stimulation could be used to trigger a state of synesthesia, which could be exploited to boost the power of memory retention. Neural associations could be created to induce certain memories synthetically using a deep dive AR consisting of visual, auditory, olfactory, and gustatory stimulants.