Non-Invasive Intraoral Stand-Alone Tongue Control System Based on RSIC-V Edge Computing

Abstract

The intelligent tongue control system is of great significance for assisting the independent life of patients with a limb disability. In order to more accurately control the assisted living equipment of incompetent patients and solve the power-loss problem of the intelligent tongue control system, this research designs a non-invasive pressure sensor array for tongue touch signal detection in the oral cavity and proposes a tongue control system based on RSIC-V edge computing. The system converts the tongue touch pressure data into specific control instructions on the edge of the RSIC-V chip and transmits them to the receiver, thus reducing the transmission of data. This study takes control of the wheelchair motor as the test object. In the experiment, the speed response time test, the center click task, and the power consumption experiment are carried out, whose results show that the adaptive fuzzy PID control algorithm has good robustness in the system; when the DC motor with a given speed of 750 r/min reaches the steady state, its rise time is 0.108 s and the adjustment time is 0.59 s. The dynamic power consumption of the non-intrusive intraoral stand-alone tongue control system proposed in this paper is found to be 3.745 MW, which is 11.5% lower than the total power consumption of the sTD system.

1. Introduction

According to the statistics, there are 85 million disabled people in China currently, among whom there are 25 million patients with functional disabilities of upper and lower limbs, and number of patients with physical disabilities and disabilities caused by congenital, disease, natural disasters, and traffic accidents is increasing year by year. The number of seniors over the age of 60 is 241 million, and with the severity of aging, the size of the disabled elderly population with quadriplegia caused by spinal cord injury, cerebral palsy, stroke, etc., continues to expand [1]. In order to solve the problems of assisted life for incompetent patients with upper and lower limb disability, researchers have carried out related research such as intelligent brain control, eye control, tongue control auxiliary system, etc., in which the tongue system can more accurately provide complex movement control and manipulation ability because it has a high degree of flexibility and is not affected by the human posture [2]. Therefore, it is of great significance to explore the tongue-controlled interactive assistance technology to help incompetent patients with limb disability have a relatively independent living ability and improve quality of life accordingly.

Relevant Work

The control problems of the tongue control system mainly focus on power loss and control accuracy issues. Currently, popular tongue control research protocols include the low-power independent system of tongue drive (iSTD) based on the magnetic array, extraoral tongue drive system (ETDS), and joystick-controlled tongue control intelligent assistive device (STD) [3,4,5]. For iSTD systems, any bumps on a wheelchair of the patient may lead to the misoperation of magnetic beads on the tongue and cause the control unit to emit misoperation signals, and ETDS and STD sometimes have similar problems. In addition, the above three methods only involve the simple control of wheelchairs in the direction of front, back, left, and right without optimizing the subsequent smoothing control of a wheelchair’s specific motion speed.

To solve this problem, we propose a non-invasive intraoral stand-alone tongue control system based on RSIC-V edge computing by using the Rocket processor as the edge node decision-maker to obtain communication with low power consumption and low bandwidth while maintaining a reasonable cost. By setting the pressure threshold and changing the amount of pressure, the wheelchair is controlled by closed-loop fuzzy PID, so as to avoid the malfunction of the control unit and optimize the users’ experience [6]. The structure diagram of the non-invasive intraoral independent tongue control system based on RSIC-V edge computing is shown in Figure 1 below.

2. System Hardware Architecture

2.1. Master Controller

Due to the particularity of the oral environment, the master control chip placed in the oral cavity for monitoring the tongue muscle pressure needs to have the characteristics of stability, small size, light weight, low power consumption [7], and supporting wireless communication to adapt to the particularity of the oral environment and provide an effective means to monitor tongue control devices. Consequently, the controller adopts the RSIC-V chip based on the Rocket kernel [8], and the characteristics of modularity make RISC-V have the characteristics of miniaturization and low energy consumption, which is crucial for embedded applications, as shown in Figure 2. The dynamic power of the Rocket kernel is 0.034 mW/MHz, which is manufactured by TSMC (Taiwan Semiconductor Manufacturing Company Limited, Taiwan, China) 40GPLUS technology and its area is 0.14 mm². In comparison, the Arm Cortex-A5 (Arm Technology (China) Co., Ltd., Shanghai, China) (32-bit single-phase order) manufactured with the same process has a dynamic power of 0.08 mW/MHz and an area of 0.27 mm². Running at the same frequency (>1 GHz), the 64-bit Rocket scalar kernel consumes less power than the A5 kernel. To reduce power consumption, a power management unit is added to the SoC un the RSIC-V chip. If the SoC or a certain domain of the SoC is not used at certain times or in some applications [9], the power management unit will lower the voltage.

2.2. Pressure Sensor Array Designing for Tongue Touch Signal Detection

Compared with traditional pressure sensors, flexible thin-film pressure sensors are more suitable for closed and humid environments in the mouth. Thus, we use a carbon nanotube-PDMS resistive pressure sensor array to integrate a signal conditioning circuit next to the pressure sensor bridge. Because this pressure sensor uses the inter-integrated circuit (I2C) communication protocol, all six sensors are connected to the same data and clock bus [10], as shown in Figure 3. The I2C protocol specifies that the data and clock bus connect a pull-up resistor between the bus and the power supply, and Ohm’s Law is used to determine the minimum pull-up resistor required for the I2C connection. At this stage, there are no timing requirements; therefore, the minimum value of the pull-up resistor used is 4.7 k. A 3.7 V DTP301120 polymer Li-ion is used to power electronics. The carbon nanotube-PDMS resistance pressure sensor is shown in Figure 3, Figure 4 and Figure 5, showing the layout of the pressure sensor on the artificial palate film.

2.3. Generation and Transmission Mode of Control Commands for Tongue Pressure Signals

Compared with etds, the challenge of this method lies in embedding flexible pressure sensors in a wet oral environment, and the collected pressure signals generate specific action commands through the signal conditioning circuits, and finally, the action commands are transmitted to the wheelchair-side wireless receiver through a wireless communication module. In order to solve the problems of Tx antenna impedance and RF loss, Ali Jafari et al., put forward several ideas [11]. Here, we used a custom-designed intraoral antenna to address this issue. Also, in previous iTDS and eTDS versions, raw magnetic sensor data had to be sent wirelessly to the Rx, and then processed and converted to tongue commands on a PC or smartphone. In this new design, an efficient tongue command detection algorithm processes raw data inside the tongue cavity and generates tongue commands directly in real time without relying on Rx processing power or software. By reducing the amount of data sent wirelessly to Rx, the intralingual pressure sensor data node is independent, and the power consumption of the system is reduced. Furthermore, by standardizing motion commands to match the standard format of Human Interface Devices (HIDs), they can be universally accepted as peripherals by most of today’s computing devices [12].

3. Data Acquisition and Processing of Tongue Control Signal Based on RSIC-V Edge Computing

3.1. Pressure Data Processing Nodes and Transmission

The system edge node is shown in Figure 6 below, whose design consists of four parts: data acquisition part, data processing part, master controller, and wireless transmission. Pressure sensors are distributed on the edge to collect raw pressure data, which is subsequently transmitted to the RSIC-v processor in parallel through the signal conditioning circuit. Due to the parallel transmission mode, the number of pressure sensor nodes will not affect the system data transmission rate. The signal conditioning circuit is mainly composed of a differential mode amplifier and filter. The third part is the finite state machine (FSM), which is a kind of computing model abstracted to study the computation process of finite memory and some language classes. A finite state automaton has a finite number of states, each of which can transmit to zero or more states, and the input string determines which transmission state is performed. This module converts pressure data into action instructions, and it is also the decision-maker of the whole system, which processes all data from sensors and combines them with triggers from the cloud, and then the FSM determines the state of the environment around the node [13]. For example, if the pressure signal is less than the set action threshold, then the decision-maker will judge it as invalid, and the system does not issue any action commands. The AES encryptor is the third part of this edge Internet of Things, which guarantees the security of data transmission. All data from the FSM unit are encrypted before the nodes are transmitted to the cloud. In this paper, the 128-bit version of the RSA module is employed. The last part of the edge node is the wireless transceiver module, which transmits the encrypted output from the node to the cloud [14]. The transceiver module can also send some defined information or instructions to the node from the cloud. In this paper, we use node 2.4 GHZ high-gain antenna as the wireless transceiver module. The wireless sending nodes only need to transmit the modulated action command without transmitting a large amount of raw pressure data, which makes the system require low throughput.

3.2. Tongue Touch Signal Control Command Transmission

The software architecture of the tongue control auxiliary device includes two modules: the software design of the intraoral stand-alone unit and the software design of the extraoral stand-alone unit. For the intraoral stand-alone unit, the communication mode between the CNT-PDMS pressure sensor array and the data-processing unit is iic. Firstly, establish the communication between master and slave machines according to the device address and configure the wireless transmitter Rx. When the pressure sensor captures the pressure signal, the wireless transmitter can choose to send the original pressure data or send the action signal that has been processed by the data-processing unit. Considering that if the original pressure data is sent, more resources and power consumption will be consumed, thus we choose to send the processed action signal directly through Rx. For the extraoral independent unit, communication with the intraoral independent unit needs to be established first. When the wireless receiver Tx receives the action information, the master control chip transmits this action information to the wheelchair through Rx to control the movement of the wheelchair. The intracavitary independent unit software is shown in Figure 7 below. When the pressure sensor detects that the current pressure is larger than the set threshold, the pressure signal is employed to control the PWM of the wheelchair motor through adaptive fuzzy PID to ensure the smooth movement of the wheelchair and increase the comfortability of users.

4. Tongue Control System Control Algorithm

4.1. PID Control of Tongue Control System

What is introduced above is mainly the information-collection method adopted by the tongue control device. For a comprehensive control system, follow-up control methods should also be considered. In engineering practice, the most widely used regulator control law is PID control. Izci D introduced the design of an optimized proportional integral differential (PID) controller for speed control of direct current (DC) motors. By multiplying the time integral by the absolute error objective function, the optimal value of PID gain was obtained. This method performs well in transient and frequency response, as well as robustness and anti-interference, and different practical algorithms are used to adjust the controller [15,16]. When the structure and parameters of PID regulating controlled object cannot be fully mastered or there is no accurate mathematical model, the structure and parameters of the system controller must be determined through experience and field debugging [17].

In practice, there are also PI and PD controls in PID control. PID controller is a systematic error, which is controlled by proportional, integral and differential calculations of the control quantity. When the pressure of the tongue muscle touching the pressure sensor changes, the movement of the wheelchair is also approximately uniform or uniformly accelerated and decelerated accordingly, allowing the user to feel more comfortable during the movement. The flow chart of the traditional PID algorithm is shown in Figure 8 below.

The specifications of the wheelchair DC motor are as follows: rated voltage UN = 24 V, rated current IN = 13.7 A, maximum speed nN = 750 r/min, allowable overload factor = 1.6, the amplification factor of the thyristor triggered rectifier: $K_s=100$, loop resistance $R=10 \mathrm{\Omega }$, electromagnetic time constant:$T_1=0.04$ s. Current feedback coefficient $\beta =0.3$. PI type is selected for the current regulator, and its transfer function is

$$W_{ACR}=K_i\frac{1+\tau {}_{i}S}{\tau {}_{i}S}$$

(1)

ACR preemptive time constant: ${\tau }_i=0.04s$. The scale factor of ACR is

$$K_i=K_i\frac{{\tau }_{i}R}{K_{S}R}=22.3\times \frac{0.04\times 10}{100\times 0.3}=0.293$$

(2)

$$W_{ACR}=K_i\frac{1+\tau {}_{i}s}{\tau {}_{i}s}=0.293\times \frac{1+0.04s}{0.04s}$$

(3)

PI type is selected for the current loop ASR regulator, and its transfer function is

$$W_{ASR}=K_n\frac{1+{\tau }_{n}S}{{\tau }_{n}S}$$

(4)

Taking m = 5, the lead time constant is

$${\tau }_n=m{T}_{{\displaystyle \sum n}}=5\times 0.016=0.08s$$

(5)

Speed loop open loop gain:

$$K_N=\frac{m+1}{2m^2{T}_{{\displaystyle \sum n^2}}}=\frac{6}{2\times 25\times 0.00025}=76$$

(6)

$$W_{ACR}=K_i\frac{1+\tau {}_{i}s}{\tau {}_{i}s}=25\times \frac{1+0.08s}{0.08s}$$

(7)

4.2. Fuzzy PID Control of Tongue Control System

The intelligent wheelchair is a multi-variable, nonlinear, time-varying system, which requires high accuracy and robustness, while it is difficult for traditional PID controllers to achieve this goal. Fuzzy PID control can effectively solve the problem of nonlinearity and rapid change and minimize the influence of disturbance and parameter change. It does not rely on the precise mathematical model of the controlled object; besides it can overcome the influence of nonlinear factors and has a strong robustness to the parameter changes of the controlled object. It does not need to establish a mathematical model. According to the input and output result data of the actual system and referring to the operating experience of field operators, the system can be controlled in real time [18]. Its fuzzy PID control flow chart is as shown in Figure 9 below, and its wheelchair rotation speed can be obtained by the encoder.

The adaptive fuzzy pid quantifies and fuzzifies the input quantity, which is the deviation E between the set output voltage and the feedback voltage of the motor and the deviation change rate $E_c$. Seven fuzzy control states are used to describe the speed deviation E in the motor speed regulation fuzzy controller system: Negative Big (NB), Negative Medium (NM), Negative Small (NS), Zero (ZO), Positive Small (PS), Medium (PM), and Positive Big (PB). The corresponding universe of fuzzy sets after discretization is as follows: E* = {−6, −5, −4, −3, −2, −1, 0, 1, 2, 3, 4, 5, 6}. The fuzzy control rules of DC motors are summarized in

Table 1. Fuzzy control rule table (U).

${{\mathit{E}}_{\mathit{c}}}_{ }$	E
	NB	NM	NS	ZO	PS	PM	PB
NB	NB	NB	NB	NB	NM	ZO	ZO
NM	NB	NB	NB	NB	NM	ZO	ZO
NS	NM	NM	NS	ZO	PS	PM	PM
ZO	NM	NS	ZO	PM	PM	PM	PM
PM	ZO	ZO	PM	PB	PB	PB	PB
PB	ZO	ZO	PM	PB	PB	PB	PB

, with a total of 49 control rules [19].

The fuzzy relation of the output control system is:

$$R_i=A_i\times B_i\times C_i$$

(8)

$$R={\displaystyle \underset{i=1}{\overset{n}{\cup }}{R}_i=}{\displaystyle \underset{i=1}{\overset{n}{\cup }}(A_i\times B_i\times C_i)}$$

(9)

The parameters of the PI controller are determined by the engineering design method, and the output $u_i^{\ast }$ of the fuzzy PID controller obtained from the figure is:

$$u{}_i{}^{\ast }=u_i+u_f$$

(10)

$$u_i=K_i{\displaystyle \sum _i{e}_i}$$

(11)

$$u_f=f(e,e_c)$$

(12)

4.3. Anti-Misoperation Pressure Threshold Setting

First the average simulated mis-touch tongue muscle pressure were collected by four different experimenters using a carbon nanotube-PDMS pressure sensor (the simulated mis-touch tongue muscle pressure included the situation when the experimenters sit in a wheelchair and bump, or they swallow saliva normally). For each experimenter, we measure three times for bumping on the wheelchair as well as normal swallowing of saliva, respectively, and then collect the pressure under normal touch. Each experimenter normally touched the pressure sensor three times, and the experimental results are shown in

Table 2. Normal touch pressure and mis-touch tongue pressure collection form currently.

Number of Experiments	Pressure Data	Object
		A	B	C	D
1	Normal pressure	0.124	0.138	0.131	0.127
	Swallowing saliva pressure	0.042	0.048	0.041	0.043
	Bump pressure	0.022	0.025	0.029	0.021
2	Normal pressure	0.122	0.132	0.135	0.129
	Swallowing saliva pressure	0.05	0.063	0.061	0.057
	Bump pressure	0.024	0.027	0.022	0.035
3	Normal pressure	0.128	0.139	0.134	0.132
	Swallowing saliva pressure	0.037	0.042	0.036	0.034
	Bump pressure	0.028	0.027	0.024	0.022

. According to the data in the table below, we can conclude that when the pressure is greater than 0.12 N, the pressure signal is the action signal.

5. System Testing and Evaluation

In order to evaluate the performance of the device, an accelerated response time test and a central strike experiment are conducted. In order to test the performance of the system, six not-disabled subjects wear the system to perform two tasks for two different algorithm experiments: the speed response over time test, and a center click task [20]. Three healthy men and three healthy women perform the experiment, none of whom has ever used a tongue machine interface device before the experiment. In addition, a 20-s calibration is performed at first use, after which the subjects perform the following tasks according to the test content, respectively.

5.1. Speed Response Time Test

In this test, the response time of the speed control command is detected, and the speed response curve of wheelchair acceleration with time is obtained by matlab simulation.

When a DC motor with a set speed of 750 r/min reaches the steady state, the rise time of the traditional PID model is 0.14 s and the adjustment time is 0.64 s, while the rise time of the adaptive fuzzy PID is 0.108 s, and the adjustment time is 0.59 s. Compared with the traditional PID, the adaptive fuzzy PID control algorithm is more stable in the robustness of this system. Figure 10 shows the simulink model of PID and adaptive fuzzy PID, and Figure 11 shows the simulation results.

The results show that compared with the PID control algorithm, by adopting the fuzzy adaptive PID control algorithm, the system may reduce the overshoot and the time to reach the set speed of the wheelchair during the acceleration process.

5.2. Center Click Task Testing

The center click task evaluates tongue-controlled devices by speed and precision based on Fitts’ law, which states the time to move a pointer to a target is related to the distance D between the current position of the pointer device and the target position and the size S of the target, as shown in Figure 12. The expression formula is:

$$t=a+b{\log }^2(\frac{D}{S}+1)$$

(13)

where a, b represent empirical parameters which are based on the physical characteristics of the pointer device, the subject, and the environment [21].

According to the method described by Fanpeng Kong et al. [22], two circular targets of different diameters are given randomly as 30 and 60 pixels, respectively. The target appears only one at a time at various distances from the center. Subjects manually manipulated the cursor to move from the center to the target and click on the center of the target as possible as they could. A total of 32 circular targets were randomly presented according to the subjects’ clicks, as shown in Figure 13. In this paper, throughput and reaction time are used as reference indexes.

In this paper, throughput is used to represent the information transfer rate from the system to the laptop computer, which is in bits/s. The average throughput is given by the following equation:

$$t=a+b{\log }^2(\frac{D}{S}+1)$$

(14)

$$TP=\frac{1}{n}{\displaystyle \sum _{i=1}^n(\frac{1}{m}{\displaystyle \sum _{j-1}^m\frac{I{D}_{ij}}{M{T}_{ij}}})}$$

(15)

Among them, ID represents the difficulty index under the same conditions; MT represents the mean movement time; m represents the target number of subjects; n represents the total number of subjects, and the difficulty index under the same conditions is calculated according to Shannon’s formula:

$$ID={\log }^2(\frac{D}{W}+1)$$

(16)

where D represents the distance from the pointer device, that is, the manual operation pointer, to the center of the circular target, and W represents the diameter of the circular target.

Response time refers to the time for the system to operate the cursor movement, represented by RT, and the unit is second.

The error rate refers to the proportion of subjects who fall outside the target range when they click on the target, expressed in ER.

Subjects performed 30 tongue touch tasks using iSTD, eSTD, and RSIC-V edge-based tongue control device, respectively, and the analysis results are shown in Figure 14 and Figure 15 below.

From the above results, it can be observed that the throughput of tongue-controlled devices based on RSIC-V edge computing is less than that of iSTD devices and eTDS devices because the raw data of tongue-controlled devices based on RSIC-V edge computing has been processed on the sensor side and specific instructions have been generated, which greatly reduces the requirements for the data-transmission rate and thus reduces the loss of CPU occupancy and energy in data transmission; the reaction time of tongue-controlled devices based on RSIC edge computing in the central click task averages to be 0.16 s, which is faster than that of iSTD devices (0.457 s) and eTDS (0.227 s) devices.

5.3. Power Consumption Test Comparison

The power consumption of the system is tested in the state of normal sending commands and standby as shown in

Table 3. Power consumption analysis of non-invasive intraoral independent tongue control system based on RSIC-V edge computing.

Type	Power Consumption (Mw)
	Typical	Best
Network dynamic	3.300	3.100
Gate dynamic	0.270	0.264
I/O dynamic	0.075	0.060
Core static	0.138	0.015
Bank static	0.054	0.003
Memory	0.460	0.380
Total static	0.242	0.118
Total dynamic	3.745	3.424
Total	3.987	3.542

below:

The low-power wearable independent tongue drive system (sTDS) developed by Ali Jafari et al. has a dynamic power consumption of 4.451 MW and a total power consumption of 4.471 MW. In this paper, the dynamic power consumption of the tongue control device is 3.745 MW, with a total power consumption of 3.987 MW. Compared to this, the dynamic power consumption in this article has been reduced by 18.8%, and the total power consumption has been reduced by 11.5%.

5.4. Labyrinth Navigation Task Experiment and Analysis

The maze navigation task is used to test the sensitivity of the system and is represented by controlling the total time the target passes through a fixed path. As shown in Figure 16, the subject sits in front of the computer, and the computer displays a GUI; the ultimate goal of this experiment is to move the ball from the starting point to the endpoint. The subjects conducted six maze experiments using iSTD devices, eTDS devices, and tongue control devices in this study, with a time of 2 min each time. If they exceeded the time limit, they did not complete the task. The experimental results are shown in

Table 4. Comparison table of labyrinth experiment time for multilingual control equipment.

Type of Tongue Control Device	Completion Time of the First Experiment (s)	Completion Time of the Second Experiment (s)	Completion Time of the Third Experiment (s)	Completion Time of the Fourth Experiment (s)	Completion Time of the Fifth Experiment (s)	Completion Time of the Sixth Experiment (s)	Average Experimental Completion Time (s)
iSTD	76	73	69	72	70	76	72.6
eTDS	83	97	86	83	79	82	85
This device	68	65	60	70	65	67	65.8

. The bottom of the bar chart represents the average time taken by the subject to complete each of the six tasks.

From the results described in the above table, it can be observed that the average time taken to complete the experiment of the tongue control device in this article is 65.8 s, which is less than compared to iSTD devices and eTDS devices.

6. Conclusions

In this paper, a non-intrusive intraoral stand-alone tongue control system based on RSIC-V edge computing is proposed. Patients can detect the pressure signal of tongue muscle touching the palate through a flexible carbon nanotube-PDMS pressure sensor array placed in the mouth, avoid the occurrence of false movement signals through the setting of pressure threshold, and perform adaptive fuzzy PID control on the wheelchair through the normal tongue touching pressure. It can accurately control the movement of the wheelchair, thus increasing the reliability of the device and improving the user’s experience. After many system tests, it is proved that the tongue-controlled system control scheme studied in this paper is reasonable. The problem of tongue control miscontact is solved, the data transmission is reduced, and the power consumption is reduced by 11.5% compared with the sTDS system. On the whole, the tongue-driven system placed in the mouth can achieve better use results after reasonable training and practice. Due to some limitations, improvements can be made in the following areas for future research: (1) The tongue control auxiliary device designed in this article simplifies the setting of tongue muscle pressure threshold. If time permits in the future, the sample tongue muscle pressure capacity should be increased and combined with algorithms that meet the conditions to set the threshold more scientifically. (2) The tongue control auxiliary device designed in this article requires an external power supply, which has brought some inconvenience to patients in practical application. Therefore, the packaging method of implantable power supply in the oral cavity needs to be studied.