1. Introduction
With the development of computer and internet technology, peoples’ lives have undergone major changes. Nowadays, people greatly rely on the use of computers and the internet in their work, information acquisition, shopping, or even entertainment, all of which deeply affect their daily lives. In computer-based devices, in general, a mouse is an important device to interact with the computer, and it is used as a pointer to access the graphical user interface (GUI) on the screen. However, according to the statistics of the World Health Organization, there are about 250,000~500,000 people with disabilities caused by accidents or illnesses every year. Their daily lives almost completely rely on family care, and it is more difficult for them to operate computers using conventional hand-controlled mice. Therefore, to improve the life quality of people with disabilities, it is necessary to design a humanized mouse assistive tool [
1].
Apparently, people with disabilities may be unable to operate computers using conventional mice. People with disabilities, especially those with spinal cord injuries, can flexibly operate only above their neck, so most of the existing mouse assistive tools are implemented based on eyes, head, mouth, or blowing and sucking movements in order to provide the basic mouse functions [
2,
3,
4]. The eye-tracking mouse is one of the popular and often-used mouse assistive tools, and it uses eye-movement tracking to control the cursor movement on a computer screen [
5]. On the other hand, a head-controlled mouse uses sensors such as accelerometers and gyroscopes to detect head actions or captures information about head movements through computer vision [
6]. Brainwave recognition, as a developing technology [
7,
8] allows people with disabilities to use their minds to control mouse movements. Furthermore, some other popular mouse assistive tools use mouth control technologies, which include the control of a sip and puff switch [
9], bite operation [
10], and mouth shape recognition [
11]. All the mentioned mouse assistive technologies are described in detail in
Section 2. However, in addition to their high manufacturing costs, there are some inconveniences and limitations in operating all the aforementioned mouse assistive tools.
Motivated to provide an easy way to access the computers and the internet to people with disabilities, we developed a blowing-controlled mouse to replace the conventional hand-controlled mouse. The main contribution of the proposed mouse is that it employs multiple electret microphones as blowing sensors and uses a technology that converts a blowing signal into a corresponding pulse width. Via the identification of the pulse width, various mouse operations such as a cursor movement, left/right click, drag and scroll can be completed. Unlike the other methods used in an assistive mouse, the proposed blowing-controlled mouse uses signal processing technology that converts a blowing signal into a pulse width, providing the benefits of a fast response and a lower implementation complexity compared with traditional digital signal processing. Since the proposed mouse only needs a slight blow from a user’s mouth and a small swing of a user’s head to operate for people with disabilities or even for paralyzed patients, it is relatively easy to control the computer. In addition, the proposed mouse can be applied to different computer operating systems without installing any driver; it thus possesses the feature of plug-and-play, because it can be set up by just connecting it to a USB (Universal Serial Bus) port.
This paper is organized as follows.
Section 2 reviews the existing control methods of mouse assistive techniques.
Section 3 introduces the proposed blowing-controlled mouse.
Section 4 illustrates the implementation of the proposed mouse and provides the experimental results. Finally,
Section 5 summarizes the features and performance of the proposed mouse.
3. Proposed Blowing-Controlled Mouse
Though the past studies have also used a microphone as a sensor to receive blowing signals, they have mainly focused on the digital signal processing of the blowing signal after amplification andanalog-to-digital conversion, which is similar to speech signal processing [
26,
27]. However, it is necessary to transform the blowing signal to the frequency domain by using a fast Fourier transform (FFT), then analyze and extract the feature values, and finally establish the models to determine whether the blow is long or short. The whole process shown in
Figure 1 is more complicated. When there are multiple microphone inputs, a problem of mutual interference between the microphones may appear, and the hardware cost may increase.
In order to create a more convenient mouse assistive tool for patients with cervical spine injuries or limb defects, we developed a blowing-controlled mouse device based on airflow vibration to assist people with disabilities in manipulating mice.
Figure 2 shows the system architecture of the proposed blowing-controlled mouse, in which six small electret microphones are used as sensing receivers of blowing signals; they represent the cursor’s movements, including up, down, left, and right, as well as left and right composite function keys. A hysteresis comparator and a re-triggerable monostable multivibrator (one shot) are used to complete the conversion of blowing signals to pulse widths. In addition, a microcontroller unit (MCU) is used to identify the position of the blown microphone and determine whether the blowing signal is long or short, and then it sends the pulses of movement coordinate or performs the key actions with the USB mouse controller chip. Finally, a USB mouse control chip with standard human interface device (HID) specification is used to communicate with the computer via a USB protocol to complete various functions of a mouse, such as cursor movement, left/right click, drag, and scroll. The blowing signal processing, the method of blowing position identification, and the basic working principle of the mouse control chip are described in detail in the following subsections.
3.1. Blowing Signal Processing
A technology without ADC (Analog-to-Digital Converter) that converts a blowing signal into the corresponding pulse width is proposed, and it includes the following three steps.
Based on the principle of airflow vibration, a small blowing signal is captured by the sense of an electret microphone.
Passing through a hysteresis comparator, many impulses are obtained in proportion to the time of the blowing signal.
A converted pulse width corresponding to the blowing impulses is generated by a re-triggerable one shot.
Figure 3 illustrates the conversion of the blowing signal for both short and long blows; the converted pulse width is proportional to the blowing time.
In
Figure 4, one of six sense and conversion circuits for capturing the blowing signals is presented. This circuit for blowing signal processing can be divided into three parts, namely a microphone sensor, a hysteresis comparator, and a re-triggerable one shot, they are described in detail in the following.
(1) Microphone sensor
A low-cost electret microphone is a type of electrostatic capacitor-based microphone. It has a stable dielectric material and high resistance, so it can be used as a receiving sensor to capture a blowing signal. During the microphone blowing, a weak signal is sensed via a microphone vibration caused by the airflow, and this small blowing signal is obtained by removing the DC bias by a coupling capacitor CS.
(2) Hysteresis comparator
To achieve digital trigger signals corresponding to a blowing signal, a hysteresis comparator is used to convert the blowing signal into a series of impulses (
VTRG). Here, by changing a variable resistor (
VR), the reference voltage
VREF can adjust the sensitivity to sense the blowing signal, where
VREF should be set to an appropriate value to obtain an optimum sensitivity. Since
VR is equivalent to two resistors
R1 and
R2 in series connection,
VREF can be expressed as:
where
VSAT is the saturation voltage of the comparator output and the hysteresis voltage
VH of the hysteresis comparator is given by:
(3) Re-triggerable one shot
The impulses generated by the comparator are discrete, and their intervals are not the same. Therefore, a re-triggerable one shot is required to make these impulses continuous. When these impulses enter into the re-triggerable one shot, an appropriate RC time constant needs to be selected such that the output pulse width
tp of the one shot is two times greater than the maximum time interval between the impulses (
ti), where
tp can be expressed as:
Through the re-triggering operations, a continuous blowing pulse width (
TW) corresponding to the blowing signal is given by:
where
n denotes the number of impulses generated by a comparator, and it is proportional to the blowing time. Finally, the converted blowing pulse width is sent to the MCU controller for further processing.
3.2. Blowing Position Identification
Since there are multiple microphones on a blowing plate, when one of the microphones is blown into, the adjacent microphones may also sense the blowing signals. In this paper, we propose an identification algorithm to find the maximum blowing pulse width among all the microphones to reduce the mutual interference of multiple microphones; thus, the MCU controller can correctly determine the position of the blown microphone. Of course, there must be appropriate distance in the position arrangement of six microphones so that the identification accuracy can be improved.
Figure 5 shows the flowchart of the identification algorithm for finding the maximum blowing pulse width, where
i denotes the microphone number whose maximum value is 6, DET
i denotes the blowing detection of the
ith microphone, and CNT
i denotes the blowing counter of the
ith microphone. The MCU controller can use a timer interrupt to handle the detection and count of the blowing pulse width in the algorithm. The algorithm steps are as follows.
In the beginning, all microphones have a blowing detection (DETi) with a low level, and the initial value of their blowing counter variable (CNTi) is set to zero.
When a user blows into a particular microphone, the adjacent microphones may also receive a portion of the blowing signal.
In turn, the algorithm detects whether the DETi of each microphone is high. If it is high, the corresponding counter variable starts counting; otherwise, it decreases by one if the counter variable value is not zero.
The algorithm checks whether all the blowing detections (DETi) are returned to zero. If so, it stops the detection; otherwise, it repeats Steps (3)–(4).
Finally, the algorithm compares the blowing counters of all the microphones and finds the maximum blowing pulse width to identify the correct blown microphone position. At the same time, the MCU controller determines whether the signal is a long or short blow according to the counter value of the maximum blowing pulse width.
To more clearly understand the identification algorithm, a case study is provided: When a user blows into the second microphone, its detection signal DET
2 turns to high from low, and then its counter (CNT
2) starts counting until the DET
2 turns to low again. At this moment, it is assumed that the first and third microphones may receive a portion of the blowing signal, which will cause their detection signals (DET
1 and DET
3) to be at high level, but their occurrence times will not necessarily be the same. Their corresponding counters (CNT
1 and CNT
3) also start counting until DET
1 and DET
3 turn to low. When all of the DET signals return to low, the current blowing detection is ended. According to the values of the counters (CNT
1–CNT
3), we can finally find that the value of CNT
2 is maximum; therefore, the second microphone is considered as the main blowing target.
Figure 6 illustrates the principle of the blowing position detection of the identification algorithm.
3.3. USB Mouse Control Chip
A USB mouse controller chip (for instance, TP8833) [
28] with standard HID specification can be used to communicate with a computer via a USB protocol to complete various mouse functions, such as cursor movement, left/right click, drag, and scroll. Such a chip typically includes a USB serial interface engine (SIE) and a mouse functional unit; the SIE handles the transmission via a USB protocol, and the mouse functional unit provides an LED driver and several optical detectors to receive the photo-couple pulse signals caused by mouse movement. In the proposed mouse, due to the existence of the mouse controller chip, the MCU does not need to communicate with the computer as long as it converts the blowing signal into the corresponding input of the mouse button or the photo-couple pulses of the mouse movement. Additionally, it is fully compatible with various computer operating systems and has plug-and-play support without the need to develop any USB driver. This type of mouse controller chip is widely used in a mechanical wheel mouse or an optical mouse. In addition to providing the inputs of the middle, left, and right key switches, the chip also receives the photo-couple signals corresponding to the coordinates (X, Y, Z) where the mouse moves or scrolls. These photo-couple signals include the X1 and X2 pulses that indicate horizontal movement, the Y1 and Y2 pulses that indicate vertical movement, and the Z1 and Z2 pulses that indicate up and down scrolling.
Figure 7 shows the timing relationship between the coordinate pulses; for instance, the cursor’s left movement is when the X1 pulse leads to the X2 pulse; inversely, when the X2 pulse leads to the X1 pulse that is the cursor’s right movement. Using the time difference between the two coordinate pulses in each direction, we can adjust the speed of cursor’s movement along that direction. After identifying the blowing position and the blowing pulse width, the MCU controller reproduces the coordinate pulses corresponding to the original mouse movement, - or sends the button inputs for the mouse controller chip to perform the click action of the left and right keys. Finally, the mouse controller chip directly communicates with the computer via the USB protocol; thereby, various functions of the conventional mouse can be implemented.
4. Implementation and Experimental Results
Figure 8 shows the practically implemented prototype of the proposed blowing-controlled mouse. In addition to the main control device connected to the computer via a USB port, the proposed mouse also includes a blowing panel mounted on the user’s neck. The function configuration of different microphones is shown in
Figure 9, where the left and right microphones represent composite blowing function keys. A short blow into the microphone of the left key performs the clicking action and a long blow realizes the drag mode when it is combined with blows into the microphones labeled with different movement directions. Similarly, a short blow into the microphone of the right key performs the open action, and a long blow activates the scrolling mode. When a user blows into the up and down microphones under such a scrolling mode, the scrolling function can be performed.
4.1. Implementation Procedure
Figure 10 illustrates the implementation procedure of the hardware and software of the proposed blowing-controlled mouse. First, the blowing signal processing was realized in the hardware circuit, which included using the microphone to capture the blowing signal, converting the blowing signal into the corresponding impulses through the hysteresis comparator, and using a re-triggerable one shot to convert the impulses into the corresponding pulse width. To reduce the mutual interference of all the microphones, an identification algorithm to find the maximum blowing pulse width was executed by the MCU. After the identification, the MCU generated the control signals that were sent to the USB mouse controller such as coordinate pulses or the inputs of mouse buttons. Finally, the mouse controller chip directly communicated with the computer via the USB protocol and performed all the mouse functions.
4.2. Waveform Measurement
As shown in
Figure 4, after the hardware blowing signal processing, the output of the re-triggerable one shot was converted into a pulse width that was proportional to the blowing time.
Figure 11 demonstrates the measured waveforms of the blowing signals and their corresponding converted pulse widths.
Figure 11a,b shows the waveforms corresponding to the short blow and long blow, respectively.
4.3. Sensitivity of Blowing Detection
The sensitivity of the blowing detection affects the accuracy of microphone identification, and thus, an optimal reference voltage
VREF of the hysteresis comparator should be determined to facilitate further signal processing. In the practical experiments, the reference voltage
VREF of the hysteresis comparator could be changed by adjusting the variable resistor
VR, and the optimal reference voltage setting could be observed and found. In
Table 2 and
Figure 4, it can be seen that the output of one shot was as follows:
VDET remained high when
VREF was set too low (<0.5 mV), and
VDET had hardly any pulse width output when
VREF was larger than 0.98 V. When
VREF was set at in the range 0.5–219 mV, the comparator generated a long-impulse-interval output at the end of a long blow, and then
VDET had an intermittent pulse width output as shown in
Figure 12a, which could easily cause a misjudgment. Therefore, by choosing a
VREF value close to the range 0.22–0.83 V, once the microphone had been blown into, a complete pulse width output corresponding to the blowing signal (shown in
Figure 12b) could be obtained.
4.4. Identification Rate
In this subsection, we evaluate the identification rate of the proposed blowing control method and compare it with mature speech recognition methods [
29,
30]. In general, the identification rate decreased with the increase in the distance between the sensor and signal source; as such, an appropriate distance is 3–4 cm.
Table 3 shows that the proposed blowing identification method achieved an excellent identification rate of over 90% for the short blow and 85% for the long blow under the conditions without speech interference. The speech identification rate was also higher than 70%; the identification errors were mainly caused by the incorrectness of the semantic analysis or speaker’s inaccurate pronunciation. On the other hand, even when someone spoke alongside the user, the identification rate by using the proposed blowing identification was almost unaffected. However, the speech recognition was easily disturbed by environment sounds, and its identification rate immediately decreased below 60% when multiple voices appeared at the same time.
4.5. Evaluation of Mouse Operation
In the development process, the proposed blowing-controlled mouse was practically operated by different people by repeating the same tests.
Table 4 shows the usage evaluation for different people and includes their age, gender, training time, operational time, and reaction (where the training time is the first practice time to learn how to operate the proposed mouse). Since everyone has a different mastery of blowing skills, some people need more practice time to learn how to blow continuous air to perform a correct long blow, and the average training time was about 5.8 min after an evaluation by ten persons. Therefore, regardless of age and gender, people can get start quickly after knowing the operational points. Because the operational habit of the proposed mouse is similar to that of a hand-controlled mouse and there is no need to blow hard nor to substantially swing one’s head, none of the users felt dizzy after a usage of about 30 min—even elders. Additionally, some people thought it was easy and effortless, with some even not feeling tired. In general, after more practice, people will be more familiar with the proposed mouse and then be able to use it smoothly.
As for the measurement of target acquisition time, we asked our subjects to to move the cursor from 5 cm around the target to the target through short blowing into the movement direction microphones, and we found that the average target acquisition time was about 4.43 s after 10 measurements in a fine-tuned mode. When the distance between the cursor and the target was larger—for example, from the rightmost of the screen to the left most of the screen—the average target acquisition time was about 4.7 s at a fixed speed of automatic movement through long blowing into the left movement microphone. In addition, in order to provide feedback to the users in operation, an LED or a buzzer can be added to indicate whether the blown microphone is sensed to remind users of its status.
The operational skill was very simple and user-friendly, as long as the user slightly blew against the microphone for operation. For instance, if a user wanted to move the cursor from the far-right side to the leftmost icon on a 27-inch computer screen, the user could slightly blow long into the left movement microphone, and then the cursor automatically shifted left; the speed of automatic movement was about 0.13 m/s. When the cursor approached the target icon, the user could shortly blow into any microphone to stop cursor movement. When there was a little position offset between the cursor and the target icon, the cursor could be fine-tuned to the target icon by short blowing into the up, down, left and right microphones. Additionally, the response time from blowing into the microphone to the moment of execution of the corresponding action was only about 100 ms. According to the presented mouse operation principle, a user is less likely to feel tired when manipulating the proposed blowing-controlled mouse, and the mouse can be easily manipulated by simple actions. Due to its high identification rate, as presented in
Section 4.4, almost all of the microphones that are blown into can be quickly and accurately recognized, and the corresponding actions can be immediately performed.
4.6. Discussion on Results
As presented in
Table 1 and described in
Section 2, the existing mouse assistive tools adopt different control methods, but most of them have operational inconveniences and limitations. After profound analysis and research, we have found a simple control method that is easy to implement through blowing identification, especially by using our proposed technology of blowing signal to pulse width conversion, which can achieve better results. Since the control method of the proposed mouse significantly differs from the existing mouse control methods in construction and usage, it is difficult to compare the performance of our and other methods; therefore,
Table 5 provides a comparison of different mouse assistive tools regarding their operational convenience, complexity, and price.
According to the experimental results provided above, as well as further analysis, we found that the sensitivity of blowing detection depends on the reference voltage VREF of the hysteresis comparator, which further affects the blowing identification rate. Namely, when VREF was set to an optimal value of about 0.22–0.83 V, the proposed blowing-controlled mouse could achieve an excellent identification rate of over 90% under conditions without speech interference; regardless of whether the signal was a short or long blow.
To sum up, the aim of the proposed blowing-controlled mouse is to help people with disabilities to easily use computers and the internet, helping them improve their life quality. Moreover, the proposed mouse can be widely promoted to people with disabilities because it has a low-cost implementation.
The novelty and contributions of this work can be summarized as follows:
- (1)
Hardware-based blowing signal processing without ADC is used to convert a blowing signal into a corresponding pulse width.
- (2)
The assistance of an identification algorithm to find the maximum blowing pulse width can reduce the mutual interference of multiple microphones, and the identification rate can thus be increased.
- (3)
The proposed technology requires no effort in operation and provides operational convenience, which makes it especially suitable for people with disabilities.
5. Conclusions
To overcome the operating drawbacks and limitations of conventional mouse assistive tools, a hardware-based blowing signal processing technology has been proposed in this paper to help people with disabilities, especially paralyzed people, to easily control a computer mouse. Using the software co-design of an identification algorithm, the proposed blowing-controlled mouse can be successfully equipped with various mouse functions, and it can achieve a stable and accurate identification rate even in the presence of other-sound interference. The experimental results showed that an identification rate of over 85% can be achieved. Moreover, compared with other mouse assistive tools, the proposed mouse has the advantages of low cost and user-friendly operation; thus, it can help people with disabilities to promote their life quality.