An AIoT-Based Assistance System for Visually Impaired People

Abstract

In this work, an assistance system based on the Artificial Intelligence of Things (AIoT) framework was designed and implemented to provide convenience for visually impaired people. This system aims to be low-cost and multi-functional with object detection, obstacle distance measurement, and text recognition achieved by wearable smart glasses, heart rate detection, fall detection, body temperature measurement, and humidity-temperature monitoring offered by an intelligent walking stick. The total hardware cost is approximately $66.8, as diverse low-cost sensors and modules are embedded. Meanwhile, a voice assistant is adopted, which helps to convey detection results to users. As for the performance evaluation, the accuracies of object detection and text recognition in the wearable smart glasses experiments are 92.16% and 99.91%, respectively, and the maximum deviation rate compared to the mobile app on obstacle distance measurement is 6.32%. In addition, the intelligent walking stick experiments indicate that the maximum deviation rates compared to the commercial devices on heart rate detection, body temperature measurement, and humidity-temperature monitoring are 3.52%, 0.19%, and 3.13%, respectively, and the fall detection accuracy is 87.33%. Such results demonstrate that the proposed assistance system yields reliable performances similar to commercial devices and is impressive when considering the total cost as a primary concern. Consequently, it satisfies the fundamental requirements of daily life, benefiting the safety and well-being of visually impaired people.

1. Introduction

According to recent statistics from the World Health Organization (WHO), the number of visually impaired people worldwide is approximately 285 million, of which 246 million are low-vision and 39 million are entirely blind [1]. With the growth of the population and the deepening of aging, it is expected that such a number will increase three times by 2040 [2]. Consequently, the problem of coping with the ever-increasing high cost of the healthcare system causes a heavy burden in many countries. Commonly, visually impaired people are unable to perceive external information through the visual system, leading to difficulties and inconveniences in daily life [3]. For example, they could easily fall due to hitting obstacles and nearby objects that are impossible to recognize, as well as read content on a book or screen. In addition, health surveillance is vital for visually impaired people because long-term physical status data can monitor health conditions and assess chronic diseases [4]. However, the related commercial devices are usually expensive, bulky, and with poor interaction. Therefore, the way for visually impaired people to obtain health conditions should be further improved.

Previously, several solutions were proposed to help visually impaired people to perceive external information. For instance, the white stick is a traditional tool for visually impaired people to navigate their surroundings and make themselves visible to others. It was recently reinvented to be foldable, lighter, and more visible [5]. Nevertheless, the usage of the white stick has limitations in terms of detecting potential obstacles. Specifically, it is unable to provide a warning before obstacles that may be approaching from a distance, and it also may not recognize those raised obstacles that are situated above the knee. The guide dog is another option that assists visually impaired people, which helps them to walk independently, confidently, and safely without relying on others, and accidents from obstacles and falls can be avoided. In addition, emotional support and social interaction can be provided, helping to reduce anxiety and mental stress levels. However, it requires time and effort to train a guide dog; the entire process usually takes up to 2 years, and the price is about $2379 [6], revealing that the total cost is prohibitive. Currently, with the development of hardware design and artificial intelligence techniques, several prototypes based on wearable [7], voice guidance [8], and hand-held [9] are offered. Most of them integrate various sensors to collect nearby data and then sound the alarms through headphones or other audio equipment. However, these attempts can be further improved by including the function of healthcare surveillance and lowering the total cost simultaneously.

To address those existing shortcomings, inspired by the current prototypes, an assistance system based on the Artificial Intelligence of Things (AIoT) framework was designed and implemented in this work. It aims to be low-cost and multi-functional, which includes two main electronic devices consisting of diverse low-cost sensors and modules connected by AIoT. One is the wearable smart glasses, later illustrated, which can detect objects and estimate their distances from the users. This functionality enables the device to alert users and provide reminders to avoid those obstacles. In addition, it offers text recognition that can be presented through the voice assistant, making the glasses automatically recognize words and transmit the results through Bluetooth audio equipment to the users. To this end, a binocular camera is adopted to acquire images of nearby environments. Subsequently, to assist visually impaired people in better perceiving information about their surroundings and providing real-time operation, all the image-related data processing is performed on the Alibaba Cloud, a platform that offers a range of cloud computing products and services in this field. Therefore, a Raspberry Pi 3B+ with the cloud platform is employed to process the corresponding data. Another device is the intelligent walking stick, which gathers various types of data through specialized sensors and modules for heart rate detection, fall detection, body temperature measurement, and humidity-temperature monitoring. Such data can be uploaded to the cloud platform for collection. Additionally, the reactions of obstacle avoidance control and alarm response are conducted through the STM32F103C8T6 microcontroller, in which the obstacle avoidance control is based on the results from obstacle distance measurement, and the alarm response is applied in the case of a fall. As a result, the collaborative interaction between the wearable smart glasses and the intelligent walking stick exhibits the potential to enhance the safety and well-being of visually impaired people in daily life. In short, the main contributions of this work are summarized as follows:

• To enhance the perceptual capabilities of visually impaired people, the proposed wearable smart glasses can accomplish real-time object detection, obstacle distance measurement, and text recognition. Additionally, the incorporation of a voice assistant enables the presentation of relevant information. The experiments demonstrate the convenience of such functions in facilitating them to acquire pertinent information about their surroundings.
• Compared to the existing prototypes, the proposed intelligent walking stick is equipped with healthcare surveillance of heart rate detection and body temperature measurement, fulfilling the daily needs of visually impaired people to monitor their health conditions. In addition, fall detection and humidity-temperature monitoring are performed to enhance mobility. Meanwhile, obstacle avoidance control and alarm response are achieved in the corresponding cases. The experiments disclose the usefulness of such incorporated functions in ensuring safety.
• The assistance system aims to be low-cost, which relies on a Raspberry Pi 3B+ and STM32F103C8T6 microcontroller, along with diverse low-cost sensors and modules connected by the AIoT. The total hardware cost is approximately $66.8, providing a low-cost solution to design relevant systems in this field.

The rest of this work is organized as follows: Section 2 reviews the related work based on each respective function. Section 3 describes the details of the proposed wearable smart glasses and intelligent walking stick. Section 4 shows the experimental results with discussions. Finally, Section 5 summarizes this work.

2. Related Work

Regarding the assistance system for visually impaired people, several approaches were designed previously, which can be typically classified into three distinct categories based on their technological foundations: vision-based, non-vision-based, and hybrid [10]. The vision-based system employs real-time video streams or images to offer insights into the nearby environments. Conversely, the non-vision-based system utilizes a vast amount of sensors and modules to analyze the surroundings and then convey relevant information to the users. The hybrid system combines vision-based with sensor technologies to derive the benefits of both approaches. Consequently, in this work, the proposed system is hybrid, where vision-based is the wearable smart glasses, and non-vision-based is the intelligent walking stick. Furthermore, as the assistance system aims to be multi-functional, the reviews of related works concentrate on each respective function designed.

2.1. Object Detection

Lowe [11] proposed the Scale Invariant Feature Transform (SIFT) algorithm, which identifies robust feature points unaffected by illumination and noise. Subsequently, an improved version based on SIFT is the Speeded-Up Robust Features (SURF) algorithm developed by Bay et al. [12], which employs a Hessian matrix-based measure for the detector, and a distribution-based descriptor. Although these conventional algorithms are comprehensible for detecting specific objects, they manually extract low-level features and ineffectively process a large variety of multi-class objects [13]. For this purpose, several deep learning approaches have been applied, which not only enable the extraction of more sophisticated features but also encompass the feature extraction, selection, and classification within a unified model. For example, OverFeat [14] is one of the extensively used deep learning approaches for object detection, which can extract image features through a multi-scale sliding window combined with AlexNet to conduct detection. Its mean Average Precision (mAP) on the ILSVRC2013 dataset is about 24.3%, which is a significant improvement compared with the conventional algorithms. However, it still produces a high error rate. The Region-Convolution Neural Network (R-CNN) [15], in which a large Intersection over Union (IoU) threshold is chosen for high-quality samples, and the region proposal network that generates the regions where objects may exist, can helpfully improve detection accuracy. Nonetheless, it is slightly insufficient in real-time requirements [16]. The You Only Look Once (YOLO) [17] is a deep learning regression technique. Unlike the region proposal network, YOLO uses a cell-centered multi-scale region, sacrificing accuracy to obtain faster speeds. It can realize up to 140 Frames Per Second (FPS), properly satisfying real-time requirements. Based on that, Mallikarjuna et al. [18] presented a cognitive IoT system capable of handling occlusion, sudden camera or object movements, and complex rotations. The results displayed real-time detection of diverse objects. Overall, deep learning approaches have greatly surpassed conventional methods in terms of detection accuracy and speed. Therefore, those works hold distinguished potential for assisting visually impaired people in perceiving their surroundings.

2.2. Obstacle Distance Measurement

The obstacle distance measurement can be classified into four categories: ultrasonic, infrared, laser, and vision [19]. For instance, Meshram et al. [20] employed ultrasonic sensors to detect obstacles below the knee so that the priority of obstacles in the path without causing information overload can be achieved. Villanueva and Farcy [21] combined a walking stick with near-infrared sensors to emit and detect infrared pulses reflected from obstacles, enabling visually impaired people to find a wide enough way on the road to pass. Although the above approaches can be used to solve the ranging problem, they are unsuitable for identifying moving objects in nearby environments. Moreover, the propagation speeds in the air are impacted by environmental parameters such as temperature, humidity, and ambient noise [22]. Additionally, the laser could cause damage to human eyes. In this regard, the vision-based method is a better solution. Monteiro et al. [23] utilized video datasets recorded from the perspective of a guide dog to train a convolutional neural network for recognizing the activities occurring around the camera and generating feedback for visually impaired people. Therefore, it can be said that the approaches based on computer vision support visually impaired people to sense their surroundings and avoid obstacles more reliably.

2.3. Text Recognition

Several previous works focus on developing robust systems that recognize text in real-time through wearable smart glasses. They often use computer vision techniques, such as edge detection and Optical Character Recognition (OCR), to extract content from the book or screen. For example, Pei and Zhu [24] introduced an accurate real-time text recognition system based on Tesseract, which is a powerful OCR engine that encloses a long short-term memory (LSTM) architecture integrating new feature words and line recognition, where the LSTM detects and predicts the text for each frame, then translates the per-frame predictions into the final label sequence, thus producing higher recognition accuracy on the images. In another work, Mukhiddinov and Cho [25] accomplished an end-to-end text recognition with the help of the OCR engine, in which the fundamental component is a fully convolutional network modified for text recognition that results in dense per-pixel predictions of sentences or text lines, as it is trained to immediately predict the presence of text occurrences and their geometries from input images.

2.4. Heart Rate Detection

Heart rate detection contains two main manners: contact and non-contact, in which electrocardiograph (ECG) is the gold standard for heart rate detection [26]. The contact manner applies electrodes or sensors that are attached to the subject, so it is limited by the requirement for direct contact with the human body, prone to discomfort, and the complexity of the operation. Conversely, the non-contact way shows convenience and comfort for users as it eliminates the need for physical contact, allowing for a more natural experience. Generally, it can be realized in three solutions: infrared serial image, optical Doppler, and photoplethysmography (PPG) [27]. Among them, PPG obtains the pulsatile variation of blood volume in peripheral microvessels in response to the heartbeat by tracing the light absorption at the measured site (finger, earlobe, nose, etc.). Its main principle is that when external light hits the human skin surface, the absorption of light by the blood in the skin pulsates with the change in its volume, which yields a corresponding periodic variation in the reflected light intensity [28]. In addition, PPG exhibits cost-effective properties, which is more suitable to be a healthcare monitoring solution in this field. Based on that, it has been extensively adopted than the other two techniques. Huang and Selvaraj [29] designed a robust system through PPG to conduct high-quality heart rate long-term monitoring. As heart rate detection functions in an earlier predictive diagnosis of cardiovascular diseases, it plays a vital role in healthcare surveillance for visually impaired people.

2.5. Fall Detection

Fall detection is usually achieved by sensors, such as accelerometer, gyroscope, and magnetometer, to detect fall-related events based on changes in movement and posture [30]. For instance, Pierleoni et al. [31] developed a wearable fall detector by integrating a three-axis accelerometer, a three-axis gyroscope, a three-axis magnetometer, and a barometer sensor, along with the data fusion algorithm derived from Attitude and Heading Reference Systems (AHRS). The vision-based methods that adopt cameras or wearable devices with built-in cameras to analyze visual data for detecting falls through irregularities in the scene are also available in this field [32]. Furthermore, several attempts [33,34,35] not only combine diverse sensors to improve performance and reduce false positives or false negatives, but also incorporate alert mechanisms to notify caregivers or emergency services. Hence, it can be said that the continuous advancement of sensor technologies contributes to accurate fall detection for visually impaired people.

2.6. Body Temperature Measurement

Body temperature measurement for visually impaired people requires specialized considerations to ensure accurate and accessible readings. In this regard, the infrared thermometer is proper, which enables convenient body temperature readings without physical contact, promoting hygiene and reducing the risk of cross-contamination [36]. More importantly, auditory feedback is desired, which caters to visually impaired people, converting body temperatures into audio signals and providing voice-guided instructions accordingly. Meanwhile, user-friendly features, such as tactile markings or braille labels on the thermometer, allow individuals proficient in braille to interpret temperatures through touch conveniently. Following this way, Khan et al. [37] placed the GY-906 infrared thermometer on the top of a portable white stick to monitor the body temperature, and those abnormalities can be wirelessly communicated through a Bluetooth transceiver, then presented by voice for notifications.

2.7. Humidity-Temperature Monitoring

Humidity-temperature monitoring designed for visually impaired people primarily aims to supply real-time information about the environmental conditions, maintaining comfortable and healthy living surroundings [38]. For example, to inform users of unfavorable or extreme conditions, or to sense the presence of water around the floor and offer warnings timely. For this purpose, integrated humidity-temperature sensors are usually applied to track historical trends and patterns, then analyze the data for conducting quantitation of both humidity and temperature levels, in which the humidity element measures the amount of moisture or water vapor present in the air, typically expressed as a percentage relative to the maximum moisture capacity of the air, i.e., Relative Humidity (RH), and temperature element measures the changes in a particular physical property (electrical resistance, voltage, or frequency) that is affected by ambient temperatures. Zhangaskanov et al. [39] modified the white stick by allocating a number of sensors at different angles to perform interactions with surroundings, including the DHT11 sensor for humidity-temperature monitoring.

3. Proposed System

3.1. AIoT-Based Architecture

Considering various situations and the health detection needs of visually impaired people in daily life, the wearable smart glasses and intelligent walking stick should satisfy two fundamental requirements in terms of convenience and reliability. To this end, the proposed design of smart glasses is illustrated in Figure 1, which can perceive surroundings based on the YOLO v5, where a binocular camera captures the images, and a lightweight network model identifies both the external objects with corresponding distances and the content on the book or screen. On the other side, the proposed design of the intelligent walking stick is displayed in Figure 2, which employs a microcontroller combined with diverse low-cost sensors and modules where the microcontroller deals with sensor data acquisition, including fall, air humidity and temperature, the user’s heart rate and body temperature, and so on. All these results can also be uploaded to the cloud platform for data collection. In addition, according to the results of obstacle distance measurement or fall detection, the obstacle avoidance or alarm response can be conducted by the microcontroller to control the stick equipped with the motor driver and omni-directional wheel, or the buzzer and vibration motor, respectively.

Although breaking the IoT architecture into three layers is the most common way, Figure 3 depicts the AIoT-based framework of the proposed assistance system. In this work, the perception layer mainly includes different sensors and modules that are responsible for collecting external data. Subsequently, the network layer uses Narrow Band-IoT (NB-IoT) technology as the cost-effective platform for connecting a wide range of sensors and modules, meanwhile adopting the Hyper Text Transfer Protocol (HTTP) and Message Queuing Telemetry Transport (MQTT) protocols for data transmission. Finally, in the application layer, an app is designed to read the current status information and present it to the users through Bluetooth audio equipment. More details are described in the subsequent sections.

3.2. Design of Wearable Smart Glasses

First, object detection is accomplished by YOLO v5, a cutting-edge technique that can enhance the accuracy and speed of object detection through the utilization of deep neural networks [17,40]. It functions by partitioning an image into a grid and subsequently predicting bounding boxes for each grid cell, so several items in an image while creating bounding boxes around them can be identified. Compared to similar methods and recent YOLO versions, it is a single-pass approach that employs a unified neural network that contributes to a speed of up to 140 FPS. This attribute is important for visually impaired people, as it facilitates immediate feedback regarding the objects existing in their nearby environments without notable delays. Additionally, its lightweight nature renders it compatible with devices possessing limited computational power, enabling its integration into the Raspberry Pi 3B+. Moreover, it exhibits the ability to accurately detect a wide array of objects across diverse categories; such versatility equips it to determine common objects in daily life. Based on that, although YOLO V5 is not the newest version, it is still superior in object detection and can provide reliable performance.

In this work, a pre-trained YOLO v5 network referenced by [41] is applied, as presented in Figure 4, which can be mainly divided into four layers: input, backbone, neck, and output. The input layer defines the initial representation of the input images from the binocular camera. The backbone layer is responsible for extracting features from the input images and, here, the CNN model serves as the backbone that performs a series of convolutions and pooling operations to capture hierarchical features at different scales. The neck connects the backbone layer to the output layer, and the feature fusion is incorporated, which can improve its ability to detect objects of different sizes. Lastly, the output layer shows the final prediction of the object with a bounding box.

More than object detection, it is worth achieving the distance measurement of obstacles in front of visually impaired people so that obstacle avoidance control can be conducted. To this end, the binocular vision ranging method [42] is used for the data recorded from a binocular camera (Intel RealSense D435), as shown in Figure 5, where O_L and O_R refer to the left and right lenses, f denotes the focal length of the camera, B means the distance between the left and right lenses, a is the lens width, P is the target point of the obstacle, P′, P″ are the intersection points of P and the two lenses, m is the lens midpoint to P′, n is the lens endpoint to P″, and Z represents the distance from the obstacle to the baseline. Mathematically, their relationships can be expressed as:

$$\frac{Z-f}{Z}=\frac{B-(m+n)}{B}$$

(1)

$$X_L=\frac{a}{2}+m$$

(2)

$$X_R=\frac{a}{2}-n$$

(3)

Combining (1), (2), and (3), we then obtain (4):

$$Z=\frac{B\times f}{X_L-X_R}=\frac{B\times f}{d}$$

(4)

In (4), Z is related to B, f, and d (the parallax of the corresponding left and right points, i.e., X_L − X_R). As B and f are constant values of the camera, which are relatively fixed, and when the parallax d value of each point is obtained, the Z value (obstacle distance) can be estimated accordingly. As a result, if an obstacle is detected and its distance to the users is measured, the assistance system can timely inform the category of the object and its corresponding distance to the visually impaired people.

Furthermore, text recognition based on OCR is employed in combination with a voice assistant to solve the difficulties encountered when reading from the book or screen. Its implementation relies on a commercial Application Programming Interface (API) provided by the Baidu OCR engine [43]. The entire process involves several stages, as drawn in Figure 6. Initially, the image is acquired from a binocular camera, and then sampling is applied to reduce image noise. Subsequently, pre-processing in terms of grayscale and binarization is adopted to extract the relevant text area from the image. Following this, character slicing is adopted to segment individual characters. The next step involves scanning the segmented image, and feature vectors are extracted from the recognized characters. After that, toward optimization processing, the feature vectors are matched against a template library, which facilitates template fine matching so the characters can be recognized. Finally, the results are outputted through a voice assistant with Bluetooth, addressing the reading difficulties of visually impaired people.

3.3. Design of Intelligent Walking Stick

Aiming at health surveillance, the functions of heart rate detection and body temperature measurement are designed in the intelligent walking stick. Regarding heart rate detection, the key is identifying the number of heartbeats per minute, i.e., the number of pulses within a minute. To this end, the Beat Per Minute (BPM) is acquired by measuring the interval between two adjacent pulses, denoted as Inter-Beat Interval (IBI), and using 60 s to divide this interval, so BPM is equal to 60/IBI. Moreover, the number of pulses is achieved by detecting wave peaks, where a threshold is set, and a pulse is considered to be detected when the signal value is larger than the threshold. As the waveform voltage range is uncertain and the amplitude of the sensor output is randomly changing, the threshold is adjusted according to the signal amplitude to accommodate the detection of different signal peaks, meaning that the solution is to find local maxima that exceed a pre-defined threshold. Such a threshold also determines the acquired voltage value to decide whether it is a waveform. When a valid waveform is received, half of the amplitude (i.e., 50%) of the signal is regarded as a feature point [44], as displayed in Figure 7. Consequently, the IBI is calculated by the time of two adjacent feature points, and the BPM is obtained accordingly.

Second, as the size and wavelength distribution of the infrared radiation energy of an object is closely related to its surface temperature, the surface temperature can be determined by measuring infrared radiation [45]. Based on that, the GY-906 infrared thermometer, a non-contact sensor that measures the infrared radiation emitted by an object to determine its temperature, is embedded in the walking stick to collect the variation caused by the change in body temperature. As for its working principle, the sensor unit converges the magnitude of the infrared radiation energy of the target in the Field of View (FOV), which is a factor in measuring the temperature of a specific area that defines the angle from which it can detect infrared radiation. Subsequently, through analog-to-digital conversion, the collected body temperature value is communicated to STM32F103C8T6 by the Inter-Integrated Circuit (I²C) protocol, allowing it to receive the temperature data and output the results correspondingly.

Next, in order to enhance the perception of surroundings for visually impaired individuals, the design focuses on incorporating humidity-temperature monitoring and fall detection capabilities into the walking stick. For this purpose, on the one hand, referenced by [39], the DHT11 sensor, which includes a calibration unit, is utilized. This sensor uses digital module acquisition and humidity-temperature sensing technologies, integrating a resistive moisture sensing element and a Negative Temperature Coefficient (NTC) temperature measurement element. As for data transmission, when the microcontroller initiates a start signal by pulling down the data bus for at least 800 µs, the DHT11 transitions from sleep mode to high-speed mode. Then, 40 bits of data are transmitted serially through the bus. Upon completion of data transmission, the sensor automatically enters hibernation mode until the subsequent communication is initiated. Additionally, an ATGM336H Global Position System (GPS) module, an MQ-2 smoke sensor, and an MH-RD raindrop sensor are employed to enrich the environmental data, such as location, weather, air quality, and so on, which further ensures the sense of comfortable surroundings. Therefore, all these sensors assist in accurate humidity-temperature monitoring.

On the other hand, a fall is characterized by an abrupt change in posture, resulting in the user falling to the ground. To identify this accident, the intelligent walking stick is equipped with the MPU6050 accelerometer and gyroscope sensor, which plays a vital role in fall detection. It is capable of analyzing acceleration to discern changes in posture when the device falls alongside the user, disclosing that the fall detection mechanism relies on obtaining body acceleration data and monitoring changes in posture. So, when the acceleration exceeds a pre-determined threshold, it signals the occurrence of a fall event. To facilitate fall detection, the MPU6050 sensor is initialized at the first step. Then, its inner Digital Motion Processor (DMP) is calibrated, which allows direct readings of the three Euler angles: roll, pitch, and yaw. These angles represent the stick mapped in the X, Y, and Z axes, as depicted in Figure 8. As a result, by analyzing them, any changes in posture and stick can be identified, helping the detection of a potential fall event.

By defining the acceleration in the X-axis, Y-axis, and Z-axis as a_x, a_y, and a_z, respectively, the combined acceleration (ACLR) can be mathematically obtained by (5), which is a parameter to determine the current status of the stick:

$$ACLR=\sqrt{{a_x}^2+{a_y}^2+{a_z}^2}$$

(5)

Typically, a larger ACLR reveals drastic movement, while it tends to decrease with soft movement. During a fall, a distinct impact phase occurs when the body makes contact with the ground, resulting in a pronounced peak in acceleration, which is more apparent than normal movements, making it a vital indicator for fall detection. As mentioned, a pre-determined threshold is needed, and based on the preliminary tests, it is 2.5 g in this work, where 1 g is equal to 9.8 m/s². In the general case, the ACLR is lower than the threshold. Nevertheless, when the ACLR exceeds the threshold, the walking stick identifies a possible fall event and activates the buzzer and vibration motor to alert. To avoid false alarms, its response continues for 60 s, during which the user has an opportunity to cancel the alarm by pressing a button placed on the stick. Failure to do so within this period indicates an actual fall event, and this alarm will be immediately transmitted to the user’s cell phone through Bluetooth and then send a message to the emergency contact.

4. Results and Discussion

4.1. Experimental Implementation

The experimental implementation mainly includes hardware validation through the designed wearable smart glasses and intelligent walking stick. It aims to check whether this system can conduct those functions according to various conditions. Please note that the experiments were conducted by fixing the STM32F103C8T6 with diverse sensors and modules on the stick to test heart rate detection, fall detection, body temperature measurement, and humidity-temperature monitoring, respectively. On the other side, a binocular camera is embedded into the glasses connected to the Raspberry Pi 3B+ to verify object detection, obstacle distance measurement, and text recognition, respectively. As mentioned, in order to ensure the real-time requirement, the algorithms concerning the functions achieved by the glasses were operated on the Alibaba Cloud.

4.2. Results of Wearable Smart Glasses Experiments

Regarding the validations of object detection and obstacle distance measurement, an evaluation involving 20 cases was conducted, containing single and multiple objects commonly found in indoor and outdoor environments. As object detection and obstacle distance measurement are usually combined in practice applications, the results enclose the predicated object with its respective distance, as shown in Figure 9 for indoors, and Figure 10 for outdoors. In addition, detailed results of object detection under various test cases are summarized in

Table 1. Performances of object detection under different cases.

Cases	Number of Items	Actual Object	Predicated Object	Failure Detection
1	One	Person	Person	None
2	One	Cup	Cup	None
3	One	Pen	Pen	None
4	One	Keyboard	Keyboard	None
5	One	Monitor	Monitor	None
6	Two	Person, Bottle	Person, Bottle	None
7	Two	Chair, Mouse	Chair, Mouse	None
8	Three	Book, Pen, Phone	Book, Pen, Phone	None
9	Three	Person, Cup, Chair	Person, Cup, Chair	None
10	Four	Person, Mouse, Monitor, Keyboard	Person, Mouse, TV, Keyboard	Monitor
11	Four	Person, Bottle, Chair, Phone	Person, Bottle, Chair, Phone	None
12	Four	Book, Pen, Mouse, Keyboard	Book, Pen, Mouse, Keyboard	None
13	Five	Person, Bottle, Chair, Mouse, Monitor	Person, Bottle, Chair, Mouse, TV	Monitor
14	One	Car	Car	None
15	Two	Bicycle, Motorcycle	Bicycle, Motorcycle	None
16	Two	Person, Car	Person, Car	None
17	Three	Person, Car, Tree	Person, Car, Tree	None
18	Three	Person, Bicycle, Car	Person, Bicycle, Car	None
19	Three	Person, Motorcycle, Car	Person, Bicycle, Car	Motorcycle
20	Five	Person, Tree, Car, Bicycle, Motorcycle	Person, Tree, Car, Bicycle	Motorcycle

.

The results of Table 1 present an accurate detection for single and dual-object cases. However, as the number of objects in the image increased, failure cases were observed, particularly for objects with similar appearances. For instance, the computer monitor was erroneously recognized as the Television (TV) in cases 10 and 13, and the motorcycle was misidentified as the bicycle in cases 19 and 20, implying the need for additional data training to improve the recognition capabilities for objects with similar appearances. In short, wearable smart glasses can achieve an object detection accuracy of around 92.16% (47 correct predictions out of 51 total items in 20 cases, with only 4 failure results), demonstrating its promise in offering object information for visually impaired people.

Concerning the distance measurement results, among all cases presented in Table 1, although several objects were misidentified, their respective distances can still be obtained, and the range is 0.71–65.54 m. Compared to a mobile app, the deviation rate (the absolute difference between the distance measured by the glasses and the distance measured by the mobile app, divided by the distance measured by the mobile app) is from 0.28–6.32%, where the longer the distance, the higher the deviation rate. The reason may be due to the binocular vision ranging method, where depth perception is achieved by analyzing the relative displacement of corresponding points in the images captured from the left and right cameras, and the baseline is the separation between the cameras. As the distance increases, the disparity between the corresponding points in the images becomes smaller for those distant objects, reducing the precision of depth estimation [46]. Hence, it has limitations at longer distances because of the way it estimates depth. Based on that, if the distance measured by the glasses Is less than 0.70 m, the user will receive an alarm, and simultaneously, with the help of the cloud platform, such data will be sent to the STM32F103C8T6 to control the stick for obstacle avoidance.

Furthermore, text recognition is designed to beneficently improve real-time reading experiences for visually impaired people, such as reading content on the book or screen, which are usually in indoor environments. Hence, its evaluation is on ten texts extracted from a book, and the example results are displayed in Figure 11, where the top displays the original text, and the bottom shows the outcomes produced by the wearable smart glasses embedded with the Baidu OCR engine. Remarkably, all words can be accurately recognized in this test case. Additional cases have involved various numbers of words, and the results are presented in

Table 2. Performances of text recognition under different cases.

Cases	Original Number of Words	Correct Number of Words	Accuracy (%)
1	363	363	100
2	412	412	100
3	480	480	100
4	557	557	100
5	614	614	100
6	672	672	100
7	754	754	100
8	803	800	99.63
9	872	869	99.66
10	956	954	99.79

. Although the head movements during reading may lead to incomplete camera screenshots, causing very slight errors in real-time text recognition, it still maintains an impressive accuracy (correct number of words divided by the original number of words) of approximately 99.91%, disclosing that the smart glasses properly extract text from the captured images. Therefore, by presenting such words through Bluetooth audio equipment, visually impaired people can enhance their real-time reading experiences in daily life.

4.3. Results of Intelligent Walking Stick Experiments

First, a heart rate module based on PPG was integrated into the stick to validate its heart rate detection capabilities. As individual variations may exist in physical status data, four subjects were recruited for the experiments, where each subject underwent 20 tests in two states: meditation and walking, with each state lasting for 30 s, alternating between the two. The heart rate detection results were not only obtained from the intelligent walking stick but also from a commercial device (Xiaomi smart band) for validation and comparison purposes.

Table 3. Performances of heart rate detection under different cases.

Cases	States	Age	Gender	Heart Rate Detected by Stick (BPM)	Heart Rate Detected by Xiaomi Device (BPM)	Deviation Rates (%)
1	Meditation 30 s	20	Female	72.54	72.02	0.72
2		22	Male	83.89	82.21	2.04
3		45	Female	69.66	71.48	2.55
4		48	Male	75.28	78.03	3.52
5	Walking 30 s	20	Female	82.48	79.76	3.41
6		22	Male	86.31	87.02	0.82
7		45	Female	74.50	76.11	2.11
8		48	Male	86.08	85.27	0.95

discloses the average performance of heart rate detection, in which the BPM values represent the average heart rate value calculated from the 20 tests in each case, and the deviation rate (the absolute difference between the heart rate detected by stick and the heart rate detected by Xiaomi device, divided by the heart rate detected by Xiaomi device) ranges from 0.72% to 3.52%. Such deviations fall within an acceptable range, verifying the reliability of heart rate detection achieved by the intelligent walking stick. Therefore, it proves effective in providing healthcare surveillance by monitoring heart rate variations, which can identify abnormal situations, particularly when the heart beats faster. For visually impaired people, this capability is valuable as it allows for the early detection of potential chronic diseases, leading to timely medical intervention and health management.

Second, toward an investigation of fall detection, the experiment mainly simulated the falls of visually impaired people in three scenarios: walking, upstairs, and downstairs. The embedded MPU6050 sensor was employed to collect acceleration variations during the experiment, and each scenario was tested 100 times.

Table 4. Performances of fall detection under three scenarios.

Scenarios	Number of Correct Detection	Number of Incorrect Detection	Number of Unresponsive Detection	Accuracy (%)
Walking	86	10	4	86.00
Upstairs	88	10	2	88.00
Downstairs	88	8	4	88.00

displays the results, where the correct detection means the stick detects a fall correctly, the incorrect detection refers to when the stick detects a fall, but the user is still in a normal state, and the unresponsive detection denotes the stick being unable to detect a fall when it occurs. Hence, the average accuracy is the total number of correct detections divided by the total number of tests (262/300 = 87.33%). Such an accuracy reveals that the intelligent walking stick can determine whether visually impaired people are in a fall in most cases. Those misjudgments may be due to the behaviors of different subjects, as the manner of the user holding a cane or their movement patterns during a fall may impact the ability of the sensor to detect the event accurately [47]. Consequently, suitable training before daily usage is preferred in this sense.

Third, as for body temperature measurement, its validation was tested 10 times for each subject and then compared with the results measured by a medical forehead thermometer (ORCOE Co., Ltd., Yantai, China, XCH01A).

Table 5. Performances of body temperature measurement from four subjects.

Subjects	Age	Gender	Body Temperature Measured by Stick (°C)	Body Temperature Measured by Medical Forehead Thermometer (°C)	Deviation Rates (%)
A	20	Female	36.47	36.44	0.08
B	22	Male	36.80	36.73	0.19
C	45	Female	36.63	36.67	0.11
D	48	Male	36.86	36.81	0.14

presents the average performance on four subjects, where the maximum deviation rate compared to the commercial device is only 0.19%, proving that body temperature measurement can be accomplished appropriately by the stick, which can benefit the timely assessment of body temperature for visually impaired people. It is particularly essential during epidemics, such as the previous Coronavirus Disease-2019 (COVID-19) pandemic.

Next, humidity-temperature monitoring was evaluated over one week at a consistent time (8 a.m.) and outdoor location (university playground). The obtained data were also compared with the measurements through a Xiaomi humidity-temperature device, as shown in

Table 6. Performances of humidity-temperature monitoring under different cases.

Days	Temperature Measured by Stick (°C)	Humidity Measured by Stick (%RH)	Temperature Measured by Xiaomi Device (°C)	Humidity Measured by Xiaomi Device (%RH)	Deviation Rates of Temperature (%)	Deviation Rates of Humidity (%)
1	16.7	49	16.4	50	1.83	2.00
2	19.2	57	19.3	57	0.52	0.00
3	17.0	62	17.5	64	2.86	3.13
4	20.1	51	20.0	51	0.50	0.00
5	17.8	55	17.9	54	0.56	1.86
6	18.9	60	18.8	59	0.53	1.69
7	17.5	61	17.7	61	1.13	0.00

. The results display that the maximum deviation rates of temperature and humidity are 2.86% and 3.13%, respectively, occurring on Day 3, and the differences were very slight on other days, indicating the walking stick is capable of providing a trustworthy environmental perception of surroundings in terms of humidity and temperature levels.

Lastly, an Android app is designed to visualize the current status information. Figure 12 shows the homepage of the app, and the example results of body temperature measurement, real-time heart rate detection, and humidity monitoring. These data can be played to the users through Bluetooth audio equipment, which helps them to sense health and environmental information. Additionally, the emergency contact or family member can use it to collect the data timely, which enhances their support for visually impaired people.

5. Conclusions

In this work, an assistance system to enhance convenience for visually impaired people is proposed, incorporating wearable smart glasses for object detection, obstacle distance measurement, and text recognition, and the intelligent walking stick equipped with capabilities of heart rate detection, fall detection, body temperature measurement, and humidity-temperature monitoring. This system is mainly designed by integrating a variety of low-cost sensors and modules with their connection based on an AIoT framework. Additionally, a voice assistant that communicates the detection results to the users is included. The glasses experiments indicate high accuracies of 92.16% and 99.91% for object detection and text recognition, respectively. In addition, the maximum deviation rate observed in obstacle distance measurement compared to a mobile app is 6.32%, showing its reliability. Then, the stick experiments focus on evaluating heart rate detection, body temperature measurement, humidity-temperature monitoring, and fall detection. Comparisons with commercial devices reveal maximum deviation rates of 3.52%, 0.19%, and 3.13% for heart rate detection, body temperature measurement, and humidity-temperature monitoring, respectively, and the fall detection accuracy achieves 87.33%. Such outcomes demonstrate that the proposed system offers performances on par with existing commercial devices and, particularly, its total hardware cost amounts to $66.8, which is lower than several previous related works, such as $240 by Lan et al. [48], $97 by Jiang et al. [49], $70 by Rajesh et al. [50], and $68 by Khan et al. [51]. Furthermore, as all image-related processes are performed on the Alibaba Cloud, the results can be available in an average of 1.12 s per image, satisfying a very-high, real-time requirement. Thus, the low-cost and multi-functional system is an appropriate and user-friendly solution to address the needs of visually impaired people, providing them with enhanced safety and well-being.

In the future, to further improve its detection accuracy with the help of sufficiently large data samples, several advanced models such as contrastive learning approach [52], self-attention enhanced deep residual network [53], time-series sequencing method [54], multiscale superpixelwise prophet model [55], and multistage stepwise discrimination with compressed MobileNet [56] will be investigated into the assistance system. Furthermore, additional functions, such as emotion recognition and fatigue detection, will be designed to enhance the overall life quality of visually impaired people.