1. Introduction
Hajj is an annual religious event where millions of pilgrims from all over the world gather to perform certain ritual, which include mobility from one site to another. With this massive gathering of people, crowd management is a significant challenge to ensure safety and security for pilgrims during their movement.
During Hajj, the crowd might walk for long distances between different sites, extending the risk zones and increasing the challenge of providing good crowd management.
Figure 1a shows a crowded street full of pilgrims in one of the Hajj sites called Mina.
Figure 1b shows the whole Mina site and most of its interconnected streets and pathways full of crowds.
Hajj organizing authorities have deployed more than 4000 fixed surveillance cameras in different sites to monitor crowd safety.
Figure 1c shows a control room receiving feeds from various crowd-monitoring cameras. Despite its massive and wide deployment, fixed cameras have a limited view and limited alleviation. They do not cover large geographical areas without the use of complicated image merging and stitching technology. Moreover, fixed cameras lack a bird’s-eye view, which is required for most crowd flow detection. Therefore, given the vast area that required coverage in the case of Hajj, image data from fixed cameras are insufficient for crowd-flow detection applications.
The utilization of Unmanned Ariel Vehicles (UAV) or drones is becoming part of our daily life, especially in critical missions like search and rescue, surveillance, reconnaissance, and emergency response. In crowded situations, drone utilization is becoming increasingly interesting in the discovery of earlier signs of a stampede, congestion, and other crowd problems. Real-time images and videos taken by drones will have a wider field of view, cover huge areas, have zero obstruction in open areas, and have an adjustable alleviation view. In [
1], the author provided a management framework for crowd events. They suggested using technologies to collect the pilgrims’ data in Hajj, and later enhance security measures and crowd management.
Most civilian applications that use UAVs and drones are offline or not time-critical. With mission-critical applications that require real-time imagery feed, capable communication modules should be provided with UAV to transmit real-time feed images.
Finally, if the images captured by the UAVs are transmitted to the base station using typical FANET communication protocols, the FANET applications must wait for the entire image to be received before it can process the image. Image-processing applications require very large amounts of data to be exchanged between the UAV and base-station. In traditional wireless sensor networks that sense light, humidity, pressure, etc., the traffic generated by a sensing node is limited to the scalar data. In most cases, the memory size required to store and send is 16-bits per reading. On the other hand, image data are in kilobytes or even megabytes. For example, a raw Red–Green–Blue (RGB) image of 128 × 128 pixels with 24-bits per pixel (8 bits per color) will be of 128 × 128 × 24 = 393,216 bits (approximately 48 kilobytes). These are magnitudes larger than traditional sensor data. Although image compression reduces the size of data to be transmitted, it is still not enough. Mukherjee et al. [
2] presented a comprehensive survey about the FANET. The author discussed the design issues of FANET. These issues are: Node Mobility, Mobility Models, Node Density, Topology Changes, Radio Propagation Model, Power Consumption and Refueling, and Localization. The networking protocols for FANET are also discussed in detail, such as Static, Proactive, Reactive, Hybrid, and Geographical and Position-based protocols. The author also highlighted some of the research areas of FANET. One of the applications is disaster and crowd management.
This paper proposes an image transmission mechanism called a Priority-Based Routing Framework for Flying Area Networks (PRoFFAN), which will enhance the delivery speed of essential image features from a UAV to the base stations. Our motivation application is crowd management, and we believe that obtaining important image features as early as possible will save lives and enhance the safety and flow of the crowd. In this work, we have simulated the transmission from UAVs to the base-station in Cooja. We implemented our framework in Contiki-NG OS. The simulations were carried out using standard aerial images used in image processing. We used image peak signal-to-noise ratio (PSNR) [
3] and structural similarity index (SSIM) [
4] to show the benefit of using PRoFFAN as compared to typical Zigbee based communication.
As discussed before, the image size can be reduced using image compression. PRoFFAN leverages the ability of image compression algorithms that generate multiple layers of compressed data. The first layer contains the most prominent features of the image, for example, the edges of objects or coarse image data. The subsequent layers contain the details that when merged with the first layer, and restore the original image. Using such image compression algorithms, the response time of FANET applications can be reduced. If the base-station receives the image data required for first pass sooner than data required for subsequent passes, it can start processing the first pass and take action accordingly while the data of subsequent layers arrive.
The rest of the paper is organized as follows. The following section discusses related works.
Section 3 discusses the PRoFFAN methodology.
Section 4 contains details of the experiment setup.
Section 5 elaborates the results, whereas the paper is concluded in
Section 6.
2. Related Work
FANET has become very popular after drones, commercial intervention and usage. Now, in many applications such as monitoring infrastructure, civil works, agriculture monitoring, mass crowd monitoring and many others, drones are used to collect multimedia data. FANETs can be described as a group of Unmanned Aerial Vehicles, which communicate without any access point. In contrast, at least one among the group acts as the server or satellite. These are automated systems, which don’t involve any human input for their basic operations [
5]. All the network activity, including the discovery of its topology or the sending of messages, is carried out by the nodes that integrate it. This type of network has an arbitrary topology. The main consequence of this mobility is that links can be formed and broken frequently, which implies that the network must be self-organized. The path between an origin and a specific destination that crosses several intermediate nodes can be modified. When this happens, the network must re-organize itself to establish a new path in the least possible time. This functionality requires that each member of the network will forward data on behalf of other members who act as a router. The nodes of a FANET network have components or associated factors which are considered when carrying out an efficient routing. These factors include auto-configuration, bandwidth, distance, energy, routing path, partitions/unions, speed, and variation in the routing path. FANET is based on the peer-to-peer (P2P) concept, where peers are mostly UAVs.
Flammini et al. [
6] presented the usage of drones in railways to monitor the situation, issues early warnings, and some decision-support applications. Their study and framework help in automatically checking the railway infrastructure parameters. This monitoring helps provide early warnings, any threats related to faults and security, or any natural hazards or intruder attacks. In railway applications, drones help with three major applications, namely structural monitoring, environmental security monitoring, and physical security monitoring.
Another exciting application area is agriculture monitoring using drones. The drone’s technology helps farmers optimize the resources required for agriculture, like the amount of water required, and to monitor pests and other diseases. It helps the farmer to target the precision in harvesting and fertilizing. The drones help with collecting the data from the fields, its processing analysis, and management. The drones also play an active role in agriculture applications like interventions in the irrigation and fertilizers of crops [
7]. Farmers utilize the drones in many ways, like counting the number of fruits on every tree, the leaf area index measurement, the computation of Normalized Difference Vegetation Index (NDVI), identifying plant diseases, and remote sensing for water management and irrigation control applications, so that the farmer can apply the chemical, and pesticides on time [
8].
Drones are also used in monitoring disasters. FANETs are deployed to the area to encounter the situation as, most of the time, humans cannot reach the remote location due to obstacles [
9]. Drones are also beneficial in search and rescue operations where all the conventional mobile networks have failed or are temporarily unavailable. FANETs can search for hostages and help humans reach those areas which are physically impossible to reach themselves [
7].
With the use of FANETs, the monitoring and analysis of the construction area are easy. Humans can verify the quality, progress, and work condition remotely using FANETs [
9].
Azevedo et al. [
10] discussed the challenges of FANET. The first challenge belongs to signal transmission because most of the directional antennas are omnidirectional, and using them in drones may not be very efficient in terms of quality and energy consumption. The new antenna developed with beamforming technology [
11] is more effective and efficient in drones. The routing protocols can be classified into five major categories, including static routing, position-based routing, and Hierarchical routing, position-based routing, swarm-based routing, hierarchical routing, and topology-based routing.
Noor et al. [
12] provide a review of the FANET communication perspective on wireless technologies, applications, challenges, and open research topics. Some of the applications they cited include monitoring, search and rescue. These applications require multimedia capture and transmission from drones. In addition to the challenges faced by FANETs, they also provide a list of available options for communication between the drones and base stations.
The authors of [
13] designed a FANET with a fleet of drones for monitoring rural areas that do not have Internet access. They argue that monitorung rural areas can be time-consuming, labor-intensive, and expensive due to the limited access and large distances to cover. They use drones to overcome these challenges. In their paper, they presented a FANET video monitoring platform by describing the architecture, the service chains used to realize the video delivery service, and an analytical model used to evaluate the computational load of the platform nodes to allow the network orchestrator to decide the backbone drones where the virtual functions will be run, and the relative resources to be allocated.
Zhang et al. [
14] also provide a similar solution. They argue that public ground networks do not provide sufficient coverage, which has resulted in the problematic backhaul of monitoring data from remote areas of interest. Because of the urgent demand for low-cost data collection in such areas, they propose a drone-enabled IoT relay system to provide high-speed data collection to support remote environmental monitoring. Their solution is based on 5 GHz and LoRa technology. They carried out numerous real-world experiments to validate the designed drone-enabled IoT relay the system’s effectiveness and show its capability of high-speed data collection.
Singh et al. [
15] suggest using drones for surveillance of violent individuals using ScatterNet Hybrid Deep Learning Network. Their system detects the violent individuals in real-time by processing the drone images in the cloud.
Wang et al. [
16] talk about the collaboration of UAVs and unmanned ground vehicles (UVGs) in intelligence gathering, border surveillance, and crowd control. They propose a comprehensive planning and control framework based on a dynamic-data-driven, adaptive multi-scale simulation (DDDAMS). They discussed their solution and shared the results of their framework, algorithms, and testbed.
Almagbile et al. [
17] discuss the estimation of crowd density from images captured from UAVs using corner detection and cluster analysis. They propose a testing procedure based on a feature from accelerated segment test (FAST) algorithms with a high level of correctness. Previously, [
18], they proposed and tested a crowd monitoring system for pedestrians. They deployed a crowd monitoring technique using real-time images taken by UAVs. Captured images were analysed to calculate crowd density using image segmentation procedures.
Motlagh et al. [
19] presented the UAV potential and IoT devices in crowd surveillance. They presented an IoT-based platform and developed a testbed for crowd surveillance. In crowd surveillance, the UAV can help provide safety to people, detect criminal activities, use facial recognition, and many other applications. In their crowd surveillance experimentation, they used a UAV equipped with a video camera, sensing devices and modems, and used an LTE network to connect with the base station. The primary purpose was to evaluate the energy consumption of UAVs during onboard processing and ground station processing. The UAV onboard video and image processing took a lot of time and energy.
Choi et al. [
20] presented a geolocation-based routing protocol for FANETs. Their protocol supports multihop routing. The geolocation-based protocols are dynamic and select the next hop based on demand. Their geolocation-based algorithm operated over the traditional IP layer protocol. The geolocation is added to the sender node’s header, and the node selects the closest possible neighbor based on the location in its header. This mechanism appears proactive and efficient.
Chriki et al. [
21] monitored a crowded area using multiple UAVs and presented a centralized data-oriented communication architecture for crowd surveillance. The data-oriented approach prioritizes the message before sending it. The ground station of the UAV network served as a central coordinator to manage the bandwidth of UAVs. The bandwidth was optimized by prioritizing the data as critical and vital. The central coordinator manages the UAVs according to the priority of information that UAVs want to send. This approach saves the energy and bandwidth of the UAV network.
3. Methodology
Some FANET applications depend on the process of capturing an image at the camera node (through a camera mounted on UAV) and its transmission to the base station (sink node). The information may be passed in sequential order from the camera node to the sink node through intermediate nodes. Typically, packets are not prioritized based on their type, and hence, the critical information may reach the sink node with a delay. In this paper, we propose a methodology to transmit critical image information in FANET with priority.
Figure 2 shows the methodology used in this paper. In FANET, the images are captured by UAVs and need to be transmitted. The captured image is passed to the Encoder for encoding the image.
The first step is to convert the image into grayscale to obtain 8 bits per pixel. We used the bit-plane encoding [
22] to encode the image into two layers: high-priority and low-priority. The bit-plane slicing is a technique used in image processing to slice the images into different planes. The layer formed by these planes contains only binary information. The high-priority layers contain the bit-planes of the bit location 8 to 5 of each pixel, whereas the low-priority layer contains bit planes of bit locations of 4 to 1 of that pixel. One pixel contributes to half a byte of each pixel of the resultant layers.
Figure 3 shows the encoding mechanism of two consecutive pixels. The four most significant bits (MSBs) of Pixel-1 are stored as MSBs of the first pixel of the high-priority layer, and the four MSBs of Pixel-2 are stored as the four least significant bits (LSBs) of the first pixel of the high-priority layer. Similarly, the four LSBs of Pixel-1 become the MSBs of the first pixel of the low-priority layer, followed by four LSBs of Pixel-2. The process continues for the whole image, and at the end of the encoding mechanism, PRoFFAN converts one grayscale image into two layers.
All the packets are arranged as low-priority and high-priority based on MSBs and LSBs. All the high-priority packets contain the MSBs of the image’s two pixels, while the low-priority packets contain the LSBs of two pixels of the image. Typically, to send an image over a network, the packets are sent in a sequence without priority. In PRoFFAN, after encoding, the layers are prioritized before sending them. The prioritization module enqueues all the high-priority layers first. Hence, the transmission of all packets of the high-priority layer occurs first, before the low-priority layer.
The bit-plane encoding mechanism used does not cause much degradation in the image quality, as the high-priority layer contains the MSBs of pixels representing most information of the pixel as it has high intensity compared to LSBs. The primary purpose of using PRoFFAN is to transmit critical information with priority to handle emergencies efficiently. Hence, the image is received at the sink node with the high-priority layers first. The image received at the sink node is low in quality as compared to the original image. Based on the pixel values in the high-priority layers, the application can easily extract low-quality image information. However, the sink node does not yet receive all the pixels. The significant difference lies in the decision-making time. If the image is sent without PRoFFAN, the application cannot decide until it receives the full image. While using the PRoFFAN, where all the high-priority layers arrive first, the application can quickly obtain enough information for decision-making, with a significant time decrease.
PRoFFAN uses the algorithm outlined in the flow chart of
Figure 4 once the bit-planes are ready to be transmitted. The packet generation stage consists of enclosing the maximum number of bytes from a bit-plane into a packet. Depending on the bit-plane, a priority is assigned to each packet. The packet en-queuing stage checks the priority of the packet received and en-queues it either to the head of the transmission queue, if it is a high-priority packet, or at the tail of the transmission queue if the received packet is of low-priority. Finally, the packet transmission stage transmits the packet from the head of the transmission queue. If a collision is detected, the next transmission is delayed based on contention window. The contention window for the high-priority packet is half the size of the contention window for the low-priority packet. In this way, in case of collision, a high-priority packet would be scheduled sooner than a low-priority packet.
4. Experimentation
The scenario shown in
Figure 5 was selected to demonstrate the effectiveness of PRoFFAN in FANETs. In this scenario, the base station (sink node) is located near the field of interest. It has a range R of 500 m. Hence, the base station has a one km radio range. Seventeen UAVs pass by the base station, one by one, but at different speeds.
The first UAV passes at a speed of 1 m/s (3.6 km/h). Therefore, the first UAV will be in the range of the base station for 1000 s. In this time, the UAV has to establish a connection with the sink and transmit one image.
The second UAV passes at a speed of 2 m/s (7.2 km/h). As a result, the second UAV will have 500 s to establish a connection with the sink and transmit one image. The exact process continues for all seventeen UAVs. Hence, the last UAV passes at 17 m/s (61.2 km/h) and has about 1 min to establish a connection and transmit one image to the sink.
All UAVs send the same image of 512 × 512 resolution. As discussed in the Research Methodology section, the image is broken down into two layers. The high-priority layer is sent before the low-priority layer. The packet rate is set at 32 packets per second. Each packet is 100 bytes. Each byte contains either high-priority information of two pixels or low-priority information of two pixels. Hence, one packet contains information of 200 pixels.
In our scenario, the UAVs try to send as much image information. In these experiments, we do not optimize the connection establishment between UAV and base station. All experiments are set with default communication parameters of Contiki-NG OS for RPL routing. Once a connection is established between the base station and the UAV, a UAV at 1 m/s should be able to transmit the entire image data, i.e., 2622 packets, in less than 90 s. Note that the 1 m/s UAV is in range of the base station for 1000 s, so it should be able to transmit the entire image. A UAV at 2 m/s should be able to transmit the entire image as well. However, UAVs at a higher speed would have less time to transmit the entire image. Additionally, packet loss may occur due to other reasons, such as multi-path fading, degradation of transmission, channel characteristics, etc.
As soon as a packet of image information is received at the base station, PRoFFAN starts to decode the information and construct a received image. At the base station, PRoFFAN calculates image Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) while receiving packets on the go. When the PSNR and SSIM reach the threshold required by the FANET application for processing images, PRoFFAN passed the image to the FANET application. Note that the goal of these experiments is to see if the introduction of PRoFFAN reduces the FANET application’s response time.
This paper focuses on single-hop image transmission between the UAV (Camera Node) and the base station (Sink Node). Therefore, only one UAV is in range for the base station. Each UAV travels a distance of 2 km, of which 1 km is in the sink range.
Table 1 summarizes the simulation parameters.
When one UAV reaches its end position, the next UAV starts its journey. In this way, the sink node collects packets from all UAVs, constructs received images, and calculates PSNR and SSIM. Simulations were carried out with PRoFFAN and without PRoFFAN to see the difference. Experiment results are explained in the next section.
5. Results
The following (
Figure 6) shows the packet delivery ratios for high-priority and low-priority packets sent using PRoFFAN and all packets sent with PRoFFAN disabled.
At UAV speeds less than 5 m/s (18 km/h), more than 90 percent of packets were received for all three packet types. The reason is that the UAV speed is less enough than the sink received most packets.
As UAV speed increased from 5 m/s (18 km/h) to 11 m/s (40 km/h), approximately 60 percent of packets made it to the sink when PRoFFAN was disabled. When PRoFFAN was enabled, the delivery ratio of high-priority packets was above 90 percent. The delivery ratio of low priority packets was reduced to 30 percent to compensate for this.
At UAV speeds higher than 11 m/s (40 km/h), it can be seen that the base station receives twice the amount of high-priority packet compared to packets sent with PRoFFAN disabled. Whatever the speed, at any time, the information that the base station receives is of more use to the FANET application than information received with PRoFFAN disabled.
The benefit of using PRoFFAN is more evident when we look at the PSNR and SSIM of received images.
Figure 7a,b show the PSNR and SSIM of received images. Both figures show that as the speed of the UAVs increase, the PSNR and SSIM decrease. However, with PRoFFAN enabled, the quality of images received at the base station is much higher. The curve of PSNR and SSIM for PRoFFAN is analogous with the number of high-priority packets received at the base station. With a higher delivery ratio of high-priority packets, the base station’s image information is more beneficial for the FANET application because its PSNR and SSIM are better. Hence, PRoFFAN assists FANET applications in improving their response time by providing them with better quality images than networks without PRoFFAN under the same conditions.
Figure 8a,b show the PSNR and SSIM calculated after every one second at the base station for packets received from a UAV traveling at 10 km/h. Without PRoFFAN, the UAV’s entire journey in the base station’s range could transmit packets that contributed to PSNR, reaching a max value of 6 and SSIM of 0.28, respectively, in 120 s (represented by marker 4). However, the same quality of PSNR = 6 reached the base station in almost one-third of the time using PRoFFAN at 34 s (represented by marker 3). Assuming that the FANET application required a PSNR = 6 for processing the incoming image, without PRoFFAN, the image of PSNR = 6 was received in 120 s, whereas, with PRoFFAN enabled, the same quality of image reached the base station in 34 s. Hence, PRoFFAN saved 86 s.
The vertical line at the 60 s mark in
Figure 8a depicts the deadline of 60 s. Without PRoFFAN, the image of PSNR = 4 reached the base station (represented by marker 2). On the other hand, with PRoFFAN enabled, an image of PSNR = 14 reached the base station in 60 s (represented by marker 1).
In fact, at 65 s (represented by mark 5), an image with a PSNR of 17.5 had reached the base station. This improvement in the quality of the image was achieved by enabling PRoFFAN. Hence, the application can use images of much better quality for processing. This is how PRoFFAN assists in improving response time and response efficiency in FANET applications with UAVs traveling at low speeds.
Figure 9 shows the image received at Marker 4 in
Figure 8a. The image was received in 120 s with a PSNR of 6 without PRoFFAN.
Figure 9 also shows the image received at Marker 3 in
Figure 8a. This image was received in 34 s with PRoFFAN enabled with the same PSNR of 6. Hence, PRoFFAN saved 86 s of the FANET application.
Figure 10a,b show the PSNR and SSIM calculated after every one second at the base station for packets received from a UAV traveling at 30 km/h. Without PRoFFAN, the UAV’s entire journey in the base station’s range could transmit packets that contributed to PSNR, reaching a max value of 5 and SSIM of 0.19, respectively, in 86 s (represented by marker 4). However, the same quality of PSNR = 5 reached the base station in almost one-third of the time using PRoFFAN at 29 s (represented by marker 3). Assuming that the FANET application required a PSNR = 5 for processing the incoming image, without PRoFFAN, the image of PSNR = 5 was received in 86 s, whereas, with PRoFFAN enabled, the same quality of image reached the base station in 29 s. Hence, PRoFFAN saved 57 s.
The vertical line at 60 s mark in
Figure 10a depicts the deadline of 60 s. Without PRoFFAN, the image of PSNR = 4 reached the base station (represented by marker 2). On the other hand, with PRoFFAN enabled, an image of PSNR = 13.4 had reached the base station in 60 s (represented by marker 1).
In fact, at the 63 s (represented by mark 5), an image with a PSNR of 16.4 reached the base station. This improvement in quality of the image was achieved by enabling PRoFFAN. Hence, the application can use images of much better quality for processing. This is how PRoFFAN assists in improving response time and response efficiency in FANET applications with UAVs traveling at medium speeds.
Figure 11a,b show the PSNR and SSIM calculated at after every one second at the base station for packets received from a UAV traveling at 60 km/h. Without PRoFFAN, the UAV entire journey in the base station’s range could transmit packets that contributed to PSNR reaching a max value of 3.2 and SSIM of 0.16, respectively, in 25 s (represented by marker 4). However, the same quality image reached the base station in 13 s PRoFFAN (represented by marker 3). Assuming that the FANET application required a PSNR = 3.2 for processing the incoming image, without PRoFFAN, the image of PSNR = 3.2 was received in 25 s, whereas, with PRoFFAN enabled, the same quality of image reached the base station in 13 s. Hence, PRoFFAN saved 12 s.
The vertical line at 30 s mark in
Figure 11a depicts the deadline of 30 s. Without PRoFFAN, the image of PSNR = 3.2 reached the base station (represented by marker 2). On the other hand, with PRoFFAN enabled, an image of PSNR = 5.2 reached the base station in 30 s (represented by marker 1).
In fact, at 49 s (represented by mark 5), an image with a PSNR of 9.7 had reached the base station. This improvement in the quality of the image was achieved by enabling PRoFFAN. Hence, the application can use images of much better quality for processing. This is how PRoFFAN improves response time and response efficiency in FANET applications with UAVs traveling at medium speeds and higher speeds.
6. Conclusions
FANET applications require certain the PSNR and/or SSIM of received images at the base-station in a lossy network. Using simulation experimentation, this paper demonstrates the usefulness of PRoFFAN in improving the response time of FANET applications by prioritizing packets that contribute more to higher PSNR and SSIM. In this paper, we demonstrated that PRoFFAN delivers images of higher quality to base-stations for FANET applications in the same amount of time and network conditions as compared to a typical Zigbee-based network. It was also seen that deadline-based FANET applications receive an image of higher quality in the same time as the typical Zigbee based network.
Some areas of PRoFFAN require improvements in future, especially Encoding and Transmission stages, outlined in
Figure 2. More efficient image encoding techniques have to be explored in future. Morevoer, the current implementation of PRoFFAN was tested and compared with Zigbee-based networks. In future, other low-powered transmission techniques have to be considered.
Finally, this paper demonstrated PRoFFAN’s usefulness using simulations. In the future, we plan to implement a real-time FANET test-bed.