A Drone’s 3D Localization and Load Mapping Based on QR Codes for Load Management

Abstract

The ongoing expansion of the Fourth Industrial Revolution has led to a diversification of drone applications. Among them, this paper focuses on the critical technology required for load management using drones. Generally, when using autonomous drones, global positioning system (GPS) receivers attached to the drones are used to determine the drone’s position. However, GPS integrated into commercially available drones have an error margin on the order of several meters. This paper, proposes a method that uses fixed-size quick response (QR) codes to maintain the error of drone 3D localization within a specific range and enable accurate mapping. In the drone’s 3D localization experiment, the errors were maintained within a specific range, with average errors ranging from approximately 0 to 3 cm, showing minimal differences. During the mapping experiment, the average error between the actual and estimated positions of the QR codes was consistently around 0 to 3 cm.

1. Introduction

In this study, we propose technologies for drone logistics management. The key to inventory management using drones is accurate positioning of the drones and identifying the location of the load. Currently, drone positioning technology generally relies on global positioning system (GPS), but it has an error of several meters in commercial drones [1]. To solve this problem, recently developed drones equipped with GPS technology such as RTK-GPS [2] and CDGPS [3], with an error of only a few centimeters, are used to solve this problem. However, the drawback of these GPS-equipped drones is their high cost.

In addition, recent studies have proposed using radio frequency identification(RFID) for load management to identify the location of the load [4,5], but false identification or non-identification of tags depending on the signal sensitivity of the RFID reader, and more importantly, RFID tags cannot generate a load map.

Therefore, in this study, we propose a method using a conventional commercial drone with a basic camera and quick response (QR) code attached to the floor to achieve more precise positioning and generate a load map. The proposed method estimates the drone position through the geometric relationship between the camera and QR code without the need for expensive sensor equipment and further identifies occurs the position of the load QR code based on the estimated position of the drone to generate a load map.

Although this study is similar to simultaneous localization and mapping (SLAM) research in terms of estimating location and a map creating, it cannot use the keyword “simultaneous” because, in conventional SLAM, the location estimation and map creation are not separate tasks. Instead, the trajectory is executed based on the previous map created in the location estimation process, and the previous map is updated based on the current estimated location. Then, a map is created from sensor data measured at the current actual location, and the error is corrected by comparing the two maps. However, in this paper, the estimation is based on QR codes, and only real-time QR code data in the video is used, not the previously acquired sensor information. Therefore, instead of comparing the similarity between the created map and the actual map, this paper compares the actual position coordinates of the drone with the estimated position coordinates. Additionally, the accuracy of load map creation is verified by examining how accurate the measured position of the load QR codes is.

The overall structure of this paper is as follows. Section 2 deals with related works. Section 2.1 is titled Simultaneous Localization And Mapping, Section 2.2 is titled Structure of QR Code and Section 2.3 is titled Camera Pinhole Model. Section 3 deals with the proposed method. Section 3.1 is titled Assumption for Proposed Method, Section 3.2 is titled Proposed Drone 3D Localization Algorithm, and Section 3.3 is titled Load Mapping Algorithm. Section 4 deals with experimental results in which simulator-based experiments (Section 4.1), real-world experiments using drones (Section 4.2), and experiments conducted with GPS attached to actual drones (Section 4.3) were conducted to verify the performance of the proposed method.

2. Related Works

2.1. Simultaneous Localization and Mapping

The SLAM technology is essential for all robots and refers to the process in which a robot detects its surrounding environment using sensors while identifying its position and creating a map [6,7,8,9,10,11,12]. When feature points are input through the sensors to perceive the surrounding situation, the robot identifies its position and posture using these feature points. Afterward, the estimated position and input sensor data are fused to redefine the existing feature points, and if the feature points are not in their previous location, they are adjusted to update the map.

The SLAM method exists in various types depending on the robot form and the sensor. Furthermore, SLAM can be categorized into lidar SLAM [13,14,15,16,17,18], visual SLAM [19,20,21,22,23], and others based on the type of sensor, and can also be divided into feature-based SLAM [15,16], fraph-based SLAM [17,18], etc., based on the method of data processing.

Lidar SLAM refers to the process of using lidar sensors for SLAM. There are various approaches to processing data in lidar SLAM. One of the most representative approaches is lidar odometry and mapping in real-time SLAM [15], which solves the problem of optimizing numerous variables using two separate algorithms. The first algorithm measures the speed of the lidar, and the second algorithm precisely matches and updates the set of sensor data (point cloud) obtained from the lidar sensor. This approach results in high-quality mapping, but because the lidar sensor data occur in real-time while the robot is moving, errors may occur if the sensor data and estimated robot position information do not match. A closed loop must be created to resolve this problem. In addition, LOAM SLAM is also a feature-based SLAM and requires a powerful embedded device to calculate many feature points. Moreover, LeGO-LOAM SLAM [16] was developed to reduce the high computational load of LOAM SLAM. This method reduces computation while improving safety by applying the loop back algorithm. However, it does not address the need for a closed loop, which is a feature of LOAM SLAM. The HDL Graph SLAM method [17,18] is another example of 3D lidar SLAM that uses the normal distribution transformation scan matching method to estimate the robot’s driving distance and loop detection in graph-based SLAM.

Visual SLAM refers to SLAM that uses cameras as sensors. Various types of cameras exist (e.g., monocular cameras, stereo cameras, RGB-D cameras, etc.); thus, many SLAM algorithms use them. Visual SLAM is classified as feature-based SLAM because it works based on various feature points detected in camera images. One advantage of visual SLAM is that it is cheaper than lidar SLAM because it uses cameras. However, due to the nature of the cameras, noise can occur in the image data, such as lighting conditions and object texture, which can result in errors and cumulative errors in creating a map using feature points. Filtering and optimization techniques are methods used to solve these problems by filtering and optimizing feature points. Examples include the Kalman [21,22] and particle [22,23] filters. The Kalman filter maps feature points in the image to landmarks using Gaussian distribution and estimates the position to keep the error of feature points within a specific range. However, if the error of feature points continues to accumulate, the map creation itself can eventually fail. Therefore, existing SLAM algorithms are unsuitable for use in load management. Additionally, machine learning-based localization methods have also been proposed. For instance, there are machine learning and deep learning-based 3D indoor visible light positioning methods [24]. However, this approach is also unsuitable for our problem because additional environmental modifications are required such as ceiling LEDs.

2.2. Structure of QR Code

The algorithm proposed in this paper is based on QR codes. Therefore, this section, explains the basic structure of QR codes, as illustrated in Figure 1 [25,26]. The finder pattern of the QR code is a pattern for detecting the position of the QR code on the photograph. By finding this pattern, the position of the code can be quickly identified, and high-speed reading is possible. The alignment pattern is an auxiliary pattern used to determine the cell location when it deviates due to distortion. The size of the QR code determines these patterns. The timing pattern alternates between black and red along the finder pattern. The coordinates of the cells are estimated using the timing pattern.

2.3. Camera Pinhole Model

Before explaining the details of the algorithm, the camera pinhole model that forms the basis for the mathematical modeling of the algorithm is explained. The camera pinhole model [27] refers to the structure where the actual object passes through a small hole (pinhole) and forms an image on the camera image sensor, as depicted in Figure 2. Assuming that $I_o(x,y)$ is the actual object and $I_i(x/M,y/M)$ is the image formed on the image sensor, $d_o$ represents the distance from the camera to the point on the object, and $d_i$ represents the focal length from the camera to the image sensor.

3. Proposed Method

The full overview of the proposed method can be seen in Figure 3. As shown in Figure 3, initially, the drone acquires both the Floor QR code and the Load QR code simultaneously. Subsequently, the image of the Floor QR code is used as an input for the 3D Localization algorithm. Section 3.1 describes the assumptions necessary for the algorithm proposed. The Load Mapping algorithm operates using the drone’s Load QR code image and its 3D position as inputs. The drone’s 3D Localization algorithm is explained in Section 3.2, while the Load Mapping algorithm is detailed in Section 3.3.

3.1. Assumption for Proposed Method

The QR code-based 3D position estimation and load-map generation method for load management proposed in this paper is based on the following assumptions.

• The size of the QR codes attached to the floor are the same.
• There is a floor camera of the drone to capture the floor QR codes.
• The center of the floor camera is perpendicular to the floor.
• The absolute coordinate of the QR code is contained in the floor QR code.
• The floor QR codes are attached in a fixed direction.
• The size of the QR codes attached to the load are the same.
• There is a camera on the front of the drone to capture the QR codes of the loaded objects.
• During mapping, the central axis of the camera and the face of the loaded object where the QR code is located are perpendicular to each other.
• The drone has basic knowledge of the size of the entire space.

Assumptions 1 and 6 can are depicted in Figure 4. The QR codes are attached to the floor where the drone flies and to the load.

Assumptions 2 and 7 are presented in Figure 5 and Figure 6, displaying the floor camera and the front camera of the drone for capturing the floor and load QR codes, respectively.

Assumptions 3 and 8 are depicted in Figure 7. They represent the perpendicularity between the optical axis of the camera and the surface of each QR code (floor and load).

Assumption 4 is illustrated in Figure 8. When decoding the QR code on the floor, the data “0/2/10” can be read, where 0 represents the QR code type. Zero represents the floor QR code, and 1 represents the load QR code. Additionally, 2 and 10 represent the x-coordinate and y-coordinate, respectively, of the QR code’s position in the absolute coordinate system.

Assumption 5 is depicted in Figure 9. The QR code must be attached in such a way that the finder pattern is fixed in a specific direction, as indicated in Figure 9. This paper assume that the finder pattern of the QR code is attached parallel to the y-axis of the absolute coordinate system.

3.2. Proposed Drone 3D Localization Algorithm

3.2.1. Overall Drone 3D Localization Algorithm

Algorithm 1 presents the overall process of the position estimation algorithm. The basic input for the position estimation algorithm is the image data (FloorFrame) captured through the floor-facing camera of Assumption 2. Once the FloorFrame is input, all QR codes in the image are detected and inserted into the QR_CODE_LIST (line 1). The variable MinDist is initialized as infinity to store the minimum distance between the image center and the QR code center (line 2). The image center point ($F{C}_{f}x$, $F{C}_{f}y$) of the FloorFrame is determined (line 3). The center point ($Q{R}_{f}x$, $Q{R}_{f}y$) of each QR code in the QR_CODE_LIST is determined, and the distance between this center point and the image center point ($F{C}_{f}x$, $F{C}_{f}y$) is calculated (lines 4–6), where Dist(P1, P2) is the formula to calculate the Euclidean distance between points P1 and P2. If the distance between the QR code center and image center is shorter than the existing MinDist, the information for the nearest QR code is updated (lines 7–9). The coordinates of the closest QR code center point ($MQ{R}_{f}x$, $MQ{R}_{f}y$) from the image center are also updated. The updated MinQR contains the position coordinates of the QR code in the world coordinate system ($Q{R}_{f}x$, $Q{R}_{f}y$), the length of one side of the QR code ($QR{L}_r$), and the distance information from the vertex of the QR code to the QR code center. Once the search for the closest QR code in the image to the image center is complete, the algorithm determines whether the minimum distance (MinDist) and distance between the two center points (($Q{R}_{f}x$, $Q{R}_{f}y$), ($F{C}_{f}x$, $F{C}_{f}y$)) match using the threshold value Th. If MinDist is smaller than Th, the MatchedLocalization() algorithm is executed, assuming that the two center points match (lines 10–11). If MinDist is greater than Th, the NotMatchedLocalization() algorithm is executed, if the two center points do not match (lines 12–13).

Algorithm 1 Pseudocode of Drone 3D Localization  Algorithm

Input:

FloorFrame: Floor-direction camera image data

QR_CODE_LIST: All QR codes in the FloorFrame

($F{C}_{f}x$, $F{C}_{f}y$): Frame center point in image coordinates

($Q{R}_{f}x$, $Q{R}_{f}y$): QR code center point in image coordinates

MinDist: Minimum distance between the QR code center and frame center

($MQ{R}_{f}x$, $MQ{R}_{f}y$): position of the QR code closest to the frame center in the image

Th: Threshold to determine whether ($Q{R}_{f}x$, $Q{R}_{f}y$) and ($F{C}_{f}x$, $F{C}_{f}y$) match

MinQR: Information structure of the QR code closest to frame center

Structure of MinQR

[($Q{R}_{r}x$, $Q{R}_{r}y$): x and y coordinates of the QR code in world coordinates,

$QR{L}_r$: The length of one side of the QR code in world coordinates,

$V{C}_r$: Distance from a vertex to the QR code center within the QR code]

Output:

($D_{r}x$, $D_{r}y$, $D_{r}z$): x, y, and z positions of the drone in world coordinates

$D_r\theta$: Angle of the direction of the drone relative to the x-axis in world coordinates

Procedure Drone 3D Localization

1        QR_CODE_LIST ← Insert All QR Code Detection in FloorFrame

2        MinDist ← ∞

3        ($F{C}_{f}x$, $F{C}_{f}y$) ← Center Point of FloorFrame

4        For QR To QR_CODE_LIST

5              ($Q{R}_{f}x$, $Q{R}_{f}y$) ← Center Point of QR

6              If MinDist > Dist($Q{R}_{f}x$, $Q{R}_{f}y$,($F{C}_{f}x$, $F{C}_{f}y$)) Then

7                     MinDist ← Dist(($Q{R}_{f}x$, $Q{R}_{f}y$),($F{C}_{f}x$, $F{C}_{f}y$))

8                     MinQR ← information from the QR

9                     ($MQ{R}_{f}x$, $MQ{R}_{f}y$) ← ($Q{R}_{f}x$, $Q{R}_{f}y$)

10        If MinDist <= Th Then

11                ($D_{r}x$, $D_{r}y$, $D_{r}z$,$D_r\theta$)←MatchedLocalization(MinQR)

12        Else

13                ($D_{r}x$, $D_{r}y$, $D_{r}z$,$D_r\theta$)←NotMatchedLocalization(MinQR,($F{C}_{f}x$, $F{C}_{f}y$),($MQ{R}_{f}x$, $MQ{R}_{f}y$))

END

In Section 3.2.2 the method for detecting all QR codes in line 1 is explained. Section 3.2.3, details the method for detecting the QR code center points in line 5. Section 3.2.4 describes the meaning of the distance between the QR code center and image center in line 6. Then, Section 3.2.5 explains the MatchedLocalization() algorithm in line 11, and Section 3.2.6 clarifies the NotMatchedLocalization() algorithm in line 13.

3.2.2. QR Code Detection Using the QR Code Reader

When an image is input through the floor-facing camera, the QR codes in the image are detected using a QR code reader. The QR code reader uses Pyzbar [21] to detect QR codes, as illustrated in Figure 10. The red area indicates where the QR code is located, and the blue area indicates the segmented area of the detected QR code.

3.2.3. Detecting QR Code Center Points

After the QR code is detected on the image, the center points of all QR codes are detected. The reason for detecting the center point of a QR code is that, even with various rotational movements, the center axis of the QR code remains constant, as illustrated in Figure 11c–f. However, due to Assumption 3, situations like those shown in Figure 11d–f do not occur in this study.

As depicted in Figure 10 in Section 3.2.2, the areas containing the QR code (red rectangle) and the QR code segmentation (blue rectangle) are represented as shown in Figure 12a. The two rectangles have the same center point, so the intersection point of the diagonals of the red rectangle is set as the center point of the QR code, as indicated in Figure 12b.

3.2.4. Distance between Image Center and QR Code Center

The distance between the image center and the QR code center is illustrated in Figure 13, where the purple point represents the image center (($F{C}_{f}x$, $F{C}_{f}y$)), the green point represents the center of the closest QR code to the image center (($MQ{R}_{f}x$, $MQ{R}_{f}y$)), and the blue rectangle represents the closest QR code (MinQR). In this case, if the distance between the green and purple points is the shortest MinDist, and this distance falls within the radius of the blue circle (Th) in the figure, the centers match.

3.2.5. Localization Method of a Drone When the QR Center and Image Center Match

Algorithm 2 presents the localization algorithm in the case where the QR center and image center match(MatchedLocalization). The input is MinQR, which is received from the Localization(). In addition, MinQR contains the position coordinates of the QR code in the world coordinate system ($Q{R}_{r}x$, $Q{R}_{r}y$), the length of one side of the QR code ($QR{L}_r$), and the distance from the vertex of the QR code to its center. In addition, information about the length of one side of the QR code in the image coordinate system ($QR{L}_f$) is required. The algorithm outputs the drone position coordinates ($D_{r}x$, $D_{r}y$, $D_{r}z$) and angle ($D_r\theta$) with respect to the x-axis in the actual coordinate system. Further, FL represents the focal length of the camera.

Algorithm 2 Pseudocode of MatchedLocalization  Algorithm

Input:

MinQR: Information structure of the QR code closest to the frame center

Structure of MinQR

[($Q{R}_{r}x$, $Q{R}_{r}y$): x and y coordinates of the QR code in world coordinates,

$QR{L}_r$: The length of one side of the QR code in world coordinates,

$V{C}_r$: Distance from a vertex to the QR code center within the QR code]

$QR{L}_f$: The length of one side of QR code in image coordinates

Output:

($D_{r}x$, $D_{r}y$, $D_{r}z$): x, y, and z positions of drone in world coordinates

$D_r\theta$: Angle of the direction of the drone relative to the x-axis in world coordinates

Procedure MatchedLocalization

1        $D_r\theta$ ← Extracting angles based on the MinQR shape

2        $QR{L}_f$ ← Average of all side lengths of MinQR

3        $D_{r}z$ ← $MinQR.QR{L}_r$ × FL/$QR{L}_f$

4        $D_{r}y$ ← $MinQR.Q{R}_{r}x$

5        $D_{r}y$ ← $MinQR.Q{R}_{r}y$

END

The algorithm first determines $D_r\theta$ (line 1). The meaning of $D_r\theta$ can be understood through Figure 14a, where $D_r\theta$ represents the angle between the x-axis of the world coordinate system and the direction in which the drone is facing.

The method of estimating the angle the drone is facing using QR codes is related to Assumption 5, where QR codes are attached in a fixed orientation, so the floor QR code rotates in the same direction as the drone. This method is explained in Figure 14b, where the red lines represent the directions of the x- and y-axes in the actual absolute coordinate system. The blue line represents the direction the drone is facing. Therefore, using the degree of rotation of the QR code allows us to determining the degree of rotation of the drone.

The rotated angle of the QR code can be determined, as illustrated in Figure 15. First, when an image containing the QR code is provided, as displaying in Figure 15a, the finder patterns are located. Then, as depicted in Figure 15b, the centroid of the finder patterns is located. Next, the centroid points of the finder patterns are used, as depicted in Figure 15c, to calculate the direction in which the drone is facing, as shown in Figure 15d. The orientation of the QR code is fixed (according to Assumption 5), and the forward direction of the drone corresponds to the 12 o’clock direction in the input image (indicated by the blue arrow). Therefore, the rotated angle $D_r\theta$ can be computed.

The method for estimating the drone position ($D_{r}x$, $D_{r}y$, $D_{r}z$) is described in lines 2–5 in table. The camera pinhole model is used to estimated the altitude of the drone. When applying this to the actual QR code-based approach, it can be modeled as illustrated in Figure 16. In the figure, MinQR$QR{L}_r$ represents the actual length of a QR code side and $QR{L}_f$ denotes the length of the QR code side in the image. In addition, FL denotes the focal length. Equation (1) can be derived according to the geometric properties of the camera pinhole model and by rearranging it, Equation (2) can be obtained. This approach allows us to determine $D_{r}z$ (line 3 in the table), and update $D_{r}x$ and $D_{r}y$ with the QR code’s position coordinates (lines 4–5 in the table).

$$D_{r}z:FL=MinQR.QR{L}_r:QR{L}_f$$

(1)

$$D_{r}z=MinQR.QR{L}_r\times FL/QR{L}_f$$

(2)

3.2.6. Drone Localization Method When the QR Center and Image Center Not-Matched

Algorithm 3 presents the algorithm for estimating the position when the QR code centers do not match. The algorithm takes several inputs: the QR code closest to the image center (MinQR), coordinates of the image center in the image coordinate system ($F{C}_{f}x$, $F{C}_{f}y$), coordinates of the QR code closest to the image center in the image coordinate system ($MQ{R}_{f}x$, $MQ{R}_{f}y$), the length of the QR code side in the image ($QR{L}_f$), the distance from the QR code center to its corner in the image coordinate system ($V{C}_f$), the distance between the QR code and the drone projected onto the xy-plane in the world coordinate system ($D{Q}_{r}xy$), and the angle between the line segment DQrxy and the x-axis in the world coordinate system ($Q{R}_r\theta$). The outputs of the algorithm are the drone’s position coordinates ($D_{r}x$, $D_{r}y$, $D_{r}z$) in the world coordinate system and the angle ($D_r\theta$) with respect to the x-axis.

The algorithm begins by calculating the length of the QR code diagonal in the image ($V{C}_f$: line 1) and computes the average length from the image center to the QR code center and each corner, as illustrated in Figure 17. Then, following the method mentioned in Section 3.2.6, the algorithm determines $D_r\theta$ (line 2).

Algorithm 3 Pseudocode for NotMatchedLocalization  Algorithm

Input:

MinQR: Information structure of the QR code closest to the frame center

Structure of MinQR

[($Q{R}_{r}x$, $Q{R}_{r}y$): x and y coordinates of the QR code in world coordinates,

$QR{L}_r$: The length of one side of the QR code in world coordinates,

$V{C}_r$: Distance from a vertex to the QR code center within the QR code]

($F{C}_{f}x$, $F{C}_{f}y$): Frame center point in image coordinates

($MQ{R}_{f}x$, $MQ{R}_{f}y$): Position of the QR code closest to the frame center in image coordinates

$QR{L}_f$: Length of one side of the QR code in image coordinates

$V{C}_f$: Distance between the vertex to QR code center in image coordinates

$D{Q}_{r}xy$: Distance between the drone and QR code projected on the xy plane in world coordinates

$Q{R}_r\theta$: Angle formed by x-axis and the line $D{Q}_{r}xy$ in world coordinates

Output:

($D_{r}x$, $D_{r}y$, $D_{r}z$): x, y, and z positions of the drone in world coordinates

$D_r\theta$: Angle of the direction of the drone relative to the x-axis in world coordinates

Procedure NotMatchedLocalization

1        $V{C}_f$ ← Average of the distance sum between ($MQ{R}_{f}x$, $MQ{R}_{f}y$) and each vertex of MinQR

2        $D_r\theta$ ← Extracting angles based on the MinQR shape

3        $D_{r}z$ ← MinQR.$V{C}_r$ × FL/$V{C}_f$

4        $D{Q}_{r}xy$ ← $MinQR.QR{L}_r$× Dist(($F{C}_{f}x$, $F{C}_{f}y$),($MQ{R}_{f}x$, $MQ{R}_{f}y$))/$QR{L}_f$

5        $Q{R}_r\theta$ ← Angle between two lines,

y ← tan($D_r\theta$) × (x − $F{C}_{f}x$) + $F{C}_{f}y$ $and$ y = ($Q{R}_{f}x$ − $F{C}_{f}x$)/($Q{R}_{f}y$ − $F{C}_{f}y$) × (x − $F{C}_{f}x$) + $F{C}_{f}y$

6        $D_{r}x$ ← $MinQR.Q{R}_{r}x$ − $D{Q}_{r}xy$ × cos($Q{R}_r\theta$)

7        $D_{r}y$ ← $MinQR.Q{R}_{r}x$ − $D{Q}_{r}xy$ × sin($Q{R}_r\theta$)

END

The next step is to calculate the drone height ($D_{r}z$) (line 3). The modeling method in Figure 18 is used to determine $D_{r}z$. Symmetry is achieved by modeling a triangle with vertices (world vertex, cl, ($MinQR.Q{R}_{r}x$,$MinQR.Q{R}_{r}y$)) and another triangle with vertices (frame vertex, cl, ($\mathrm{M}\mathrm{Q}{\mathrm{R}}_{f}x$, $\mathrm{M}\mathrm{Q}{\mathrm{R}}_{f}y$)) leading to the derivation of Equation (3). Additionally, symmetry is achieved by considering the triangle(($\mathrm{F}{\mathrm{C}}_{f}x$, $\mathrm{F}{\mathrm{C}}_{f}y$), cl, ($\mathrm{MinQR.Q}{\mathrm{R}}_{r}x$,$\mathrm{MinQR.Q}{\mathrm{R}}_{r}y$)) and another triangle(O, cl, ($\mathrm{MinQR.Q}{\mathrm{R}}_{r}x$,$\mathrm{MinQR.Q}{\mathrm{R}}_{r}y$)) allowing the formulation of Equation (4). Therefore, Equation (5) can be derived by rearranging Equations (3) and (4):

$$((MQ{R}_{f}x,MQ{R}_{f}y),cl):((MinQR.Q{R}_{r}x,MinQR.Q{R}_{r}y),cl)=V{C}_f:MinQR.V{C}_r$$

(3)

$$((MQ{R}_{f}x,MQ{R}_{f}y),cl):((MinQR.Q{R}_{r}x,MinQR.Q{R}_{r}y),cl)=FL:D_{r}z$$

(4)

$$D_{r}z=inQR.V{C}_r\times FL/V{C}_f$$

(5)

The next step involves calculating the x- and y-coordinates ($D_{r}x$, $D_{r}y$) of the drone in the world coordinate system (lines 4–7). To do this, we must determine $D{Q}_{r}xy$ and $Q{R}_r\theta$ as indicated in Figure 19, where $D{Q}_{r}xy$ represents the distance between the drone and QR code center projected onto the xy plane in the world coordinate system, and $Q{R}_r\theta$ represents the angle between $D{Q}_{r}xy$ and the x-axis in the world coordinate system.

The variable $D{Q}_{r}xy$ can be modeled as presented in Figure 20 (line 4), where $D{Q}_{r}xy$ corresponds to Dist(($F{C}_{f}x$, $F{C}_{f}y$), ($MQ{R}_{f}x$, $MQ{R}_{f}y$)) and $MinQR.QR{L}_r$ corresponds to $QR{L}_f$, resulting in the derivation of Equation (6). Simplifying Equation (6) leads to Equation (7).

$$D{Q}_{r}xy:MinQR.QR{L}_r=Dist((F{C}_{f}x,F{C}_{f}y),(MQ{R}_{f}x,MQ{R}_{f}y)):QR{L}_f$$

(6)

$$D{Q}_{r}xy=MinQR.QR{L}_r\times Dist((F{C}_{f}x,F{C}_{f}y),(MQ{R}_{f}x,MQ{R}_{f}y))/QR{L}_f$$

(7)

The next step is to calculate $Q{R}_r\theta$ (line 5). The method for calculating $Q{R}_r\theta$ can be explained using Figure 21. In Figure 21a, $D_r\theta$, is already computed. Figure 21b illustrates the relationship between $D_r\theta$, the drone, the QR code, and the xy plane in the image. Moreover, $D_r\theta$ represents the angle between the x-axis and the direction the drone faces in reality and is equivalent to the angle formed between the line crossing the image center point and rx’ as depicted in Figure 21c. Therefore, we can obtain the equations for the straight lines: the line of the segment $D{Q}_{r}xy$ (purple line) with the image center point ($F{C}_{f}x$, $F{C}_{f}y$) as the origin and the line of rx’ (Equations (8) and (9)). $Q{R}_r\theta$ is calculated using these equations:

$$y=tan\left(D_r\theta \right)\times (x-F{C}_{f}x)+F{C}_{f}y$$

(8)

$$y=(Q{R}_{f}x-F{C}_{f}x)/(Q{R}_{f}y-F{C}_{f}y)\times (x-F{C}_{f}x)+F{C}_{f}y$$

(9)

Finally, we compute $D_{r}x$ and $D_{r}y$ (line 6–7). This step, we can establishes a relationship between the actual QR code and drone, as depicted in Figure 22. Based on this relationship, we can derive Equations (10) and (11):

$$D_{r}x=MinQR.Q{R}_{r}x-D{Q}_{r}xy\times cos\left(Q{R}_r\theta \right)$$

(10)

$$D_{r}y=MinQR.Q{R}_{r}y-D{Q}_{r}xy\times sin\left(Q{R}_r\theta \right)$$

(11)

3.3. Proposed Load Mapping Algorithm

3.3.1. Load Mapping Algorithm Overview

Algorithm 4 presents the entire process of the mapping algorithm, which takes the following inputs: the image from the front-facing camera (LoadFrame), coordinates of the image center ($F{C}_{f}x$, $F{C}_{f}y$), the coordinates of the QR code center in the image ($Q{R}_{f}x$, $Q{R}_{f}y$), the threshold value (Th) to determine if the image center and QR code center match, the drone’s position coordinates ($D_{r}x$, $D_{r}y$, $D_{r}z$) obtained through the localization algorithm, and viewing direction angle of the drone with respect to the x-axis in the world coordinate system. The algorithm outputs the position coordinates of all detected QR codes in the LoadFrame.

Algorithm 4 Pseudocode of LoadMapping  Algorithm

Input:

LoadFrame: Face-direction camera image data

($F{C}_{f}x$, $F{C}_{f}y$): Frame center point in image coordinates

($Q{R}_{f}x$, $Q{R}_{f}y$): QR code center point in image coordinates

Th: Threshold to determine whether ($Q{R}_{f}x$, $Q{R}_{f}y$) and ($F{C}_{f}x$, $F{C}_{f}y$) match

($D_{r}x$, $D_{r}y$, $D_{r}z$): x, y, and z positions of drone in world coordinates

$D_r\theta$: Angle of the direction the drone relative the x-axis in world coordinates

Output:

($Q{R}_{r}x$, $Q{R}_{r}y$, $Q{R}_{r}z$): x, y, and z position of the QR code in world coordinates

LOAD_QR_POSITION: Location of all QR codes in image coordinates

Procedure LoadMapping

1        QR_CODE_LIST ← Insert All QR Code Detection in LoadFrame

2        ($F{C}_{f}x$, $F{C}_{f}y$) ← Center Point of LoadFrame

3        For QR To QR_CODE_LIST

4                ($Q{R}_{f}x$, $Q{R}_{f}y$) ← Center point of QR

5                If Dist(($Q{R}_{f}x$, $Q{R}_{f}y$),($F{C}_{f}x$, $F{C}_{f}y$)) <= Th Then

6                        ($Q{R}_{r}x$, $Q{R}_{r}y$, $Q{R}_{r}z$) ← MatchedMapping(QR, $D_{r}x$, $D_{r}y$, $D_{r}z$, $D_r\theta$)

7                Else

8                        ($Q{R}_{r}x$, $Q{R}_{r}y$, $Q{R}_{r}z$) ←NotMatchedMapping(QR, $D_{r}x$, $D_{r}y$, $D_{r}z$,$D_r\theta$,

($F{C}_{f}x$, $F{C}_{f}y$),($Q{R}_{f}x$, $Q{R}_{f}y$))

9                LOAD_QR_POSITION ← Insert ($Q{R}_{r}x$, $Q{R}_{r}y$, $Q{R}_{r}z$)

END

The LoadMapping algorithm detects all QR codes using a QR code reader (line 1) and determines the image center (line 2). For each QR code, it calculates the center point (line 4) and compares the Euclidean distance between the QR code’s center point and the image center with a threshold value (Th) to determine if they match (line 5). If the distance is less than or equal to the threshold, it generates a load map for the matched case (line 6). Otherwise, the algorithm generates a load map for the unmatched case (lines 7–8). The algorithm identifies and stores the positions of the QR codes by performing these steps for all QR codes in the image (line 9). Section 3.3.2 describes the load map generation algorithm (MatchedMapping()) for matched QR codes where the center point matches the image center, while Section 3.3.3 explains the load map generation algorithm (NotMatchedMapping()) for situations where the center point does not match the image center.

3.3.2. Load QR Code Mapping Algorithm When Center Points Matched

Algorithm 5 describes the MatchedMapping() algorithm operation. It requires the following input values: basic information about the load QR code (QR), estimated drone position information ($D_{r}x$, $D_{r}y$, $D_{r}z$), drone viewing angle ($D_r\theta$), QR code side length in the image coordinate system ($QR{L}_f$), and distance between the QR code and drone in the world coordinate system ($DQ{W}_{r}xy$). However, this algorithm focuses on determining the position of the load QR code and generating a map; thus, the QR code information does not include its coordinates in the world coordinate system.

Algorithm 5 Pseudocode of MatchedMapping  Algorithm

Input:

QR: Information structure of the load QR code

Structure

[$QR{L}_r$: The length of one side of the QR code in world coordinates,

$V{C}_r$: Distance from a vertex to the QR code center within the QR code]

($D_{r}x$, $D_{r}y$, $D_{r}z$): x, y, and z positions of the drone in world coordinates

$D_r\theta$: Angle of the direction of the drone relative to the x-axis in world coordinates

$QR{L}_f$: The length of one side of the QR code in image coordinates

$DQ{W}_{r}xy$: Distance between the drone and wall attaching the load QR code in world coordinates

Output:

($Q{R}_{r}x$, $Q{R}_{r}y$, $Q{R}_{r}z$): x, y, and z positions of QR code in world coordinates

Procedure MatchedMapping

1        $D{Q}_{r}xy$ ← FL × $QR.QR{L}_r$/$QR{L}_f$

2        $Q{R}_{r}x$ ← $D_{r}x$ + $D{Q}_{r}xy$ × cos($D_r\theta$)

3        $Q{R}_{r}y$ ← $D_{r}y$ + $D{Q}_{r}xy$ × sin($D_r\theta$)

4        $Q{R}_{r}z$ ← $D_{r}z$

END

The situation assumed in MatchedMapping starts from Assumption 8, referring to a scenario where the frontal camera center aligns with the load QR code center of the drone, as depicted in Figure 23a. In this case, the height ($Q{R}_{r}z$) of the load QR code in the world coordinates matches the current height of the drone ($D_{r}z$). Additionally, $Q{R}_{r}x$ and $Q{R}_{r}y$ are modeled as illustrated in Figure 23b, representing the situation viewed in the xy-plane where the centers align. The blue dot represents the drone’s position, “load” represents the load, and the blue rectangle inside the load represents the load QR code. Given the current drone position ($D_{r}x$, $D_{r}y$, $D_{r}z$) and the drone’s frontal angle of view ($D_r\theta$) obtained from the localization algorithm, we can derive the equations as shown in lines 2–4.

The variable $DQ{W}_{r}xy$ represents the distance from the actual QR code to the drone. To calculate this distance, we can model it similarly to the situation where the center points are aligned in the localization algorithm, as depicted in Figure 24 (line 1).

3.3.3. Load QR Code Mapping Algorithm with Center Point Not-Matched

Algorithm 6 describes the NotMatchedMapping algorithm operation. The algorithm requires the following input values: basic information on the load QR code (QR), image coordinates of the image center ($F{C}_{f}x$, $F{C}_{f}y$) and QR code center ($Q{R}_{f}x$, $Q{R}_{f}y$), estimated drone-position information ($D_{r}x$, $D_{r}y$, $D_{r}z$), angle the drone is facing ($D_r\theta$), length of the QR code side in the image coordinate system ($QR{L}_f$), distances from the image center to the QR code center along the x-axis and y-axis ($QRF{C}_{f}x$, QR$F{C}_{f}y$), corresponding distances in the world coordinate system ($QRF{C}_{r}x$, $QRF{C}_{r}y$), distance from the drone to the wall where the QR code is attached in the world coordinate system ($DQ{W}_{r}xy$), coordinates of the intersection point between segment $DQ{W}_{r}xy$ and the wall where the QR code is attached ($Inte{r}_{r}x$, $Inte{r}_{r}y$), and the angle between the x-axis and the wall where the QR code is attached in the world coordinate system ($QR{W}_r\theta$).

Algorithm 6 Pseudocode of NotMatchedMapping  Algorithm

Input:

QR: Information structure of load QR code

Structure

[$QR{L}_r$: The length of one side of the QR code in world coordinates,

$V{C}_r$: Distance from a vertex to the QR code center within the QR code]

($F{C}_{f}x$, $F{C}_{f}y$): Frame center point in image coordinates

($Q{R}_{f}x$, $Q{R}_{f}y$): QR code center point in image coordinates

($D_{r}x$, $D_{r}y$, $D_{r}z$): x, y, and z positions of the drone in world coordinates

$D_r\theta$: Angle of the direction of the drone relative to the x-axis in world coordinates

$QR{L}_f$: The length of one side of the QR code in image coordinates

($QRF{C}_{f}x$, QR$F{C}_{f}y$): The x and y-axes distance from the QR code

center to the frame center in image coordinates

($QRF{C}_{r}x$, $QRF{C}_{r}y$): Distance corresponding to ($QRF{C}_{f}x$, QR$F{C}_{f}y$)

in world coordinates

$DQ{W}_{r}xy$: Distance between the drone and the wall attaching the load QR code in world coordinates

($Inte{r}_{r}x$, $Inte{r}_{r}y$): Position where the wall attaching the load QR code and the line

$DQ{W}_{r}xy$ intersect in world coordinates

$QR{W}_r\theta$: Angle between the wall attaching the QR code and the x-axis in world coordinates

Output:

($Q{R}_{r}x$, $Q{R}_{r}y$, $Q{R}_{r}z$): x, y, and z positions of the QR code in world coordinates

Procedure NotMatchedMapping

1        ($QRF{C}_{f}x$, QR$F{C}_{f}y$) ←(|$\mathrm{F}{\mathrm{C}}_{f}x$ − $\mathrm{Q}{\mathrm{R}}_{f}x$|, |$F{C}_{f}y$ − $Q{R}_{f}y$|)

2        ($QRF{C}_{r}x$, $QRF{C}_{r}y$) ← ($QRF{C}_{f}x$, QR$F{C}_{f}y$) × $QR.QR{L}_r$/$QR{L}_f$

3        $V{C}_f$ ← Average of the distance sum between ($MQ{R}_{f}x$, $MQ{R}_{f}y$) and each vertex of QR

4        $DQ{W}_{r}xy$ ← QR.$V{C}_r$ × FL/$V{C}_f$

5        ($Inte{r}_{r}x$, $Inte{r}_{r}y$) ←($D_{r}x$ + $DQ{W}_{r}xy$ × cos($D_r\theta$), $D_{r}y$ + $DQ{W}_{r}xy$ × sin($D_r\theta$))

6        $QR{W}_r\theta$ ←($D_r\theta$ + 90)% 360

7        If $F{C}_{f}y$ > $Q{R}_{f}y$

8                 $Q{R}_{r}z$ ← $D_{r}z$ + $QRF{C}_{r}y$

9        Else

10                $\mathrm{Q}{\mathrm{R}}_{r}z$ ← $D_{r}z$ – $QRF{C}_{r}y$

11         If $D_r\theta$ = 0

12                $Q{R}_{r}x$ ← $D_{r}x$ + $DQ{W}_{r}xy$

13                If $Q{R}_{f}x$ > $F{C}_{f}x$

14                        $Q{R}_{r}y$ ← $D_{r}y$ + $QRF{C}_{r}x$

15                Else

16                        $Q{R}_{r}y$ ← $D_{r}y$ − $QRF{C}_{r}x$

17         Else If $D_r\theta$ = 90

18                $Q{R}_{r}y$ = $D_{r}y$ + $DQ{W}_{r}xy$

19                If $Q{R}_{f}x$ > $F{C}_{f}x$

20                        $Q{R}_{r}x$ ← $D_{r}x$ − $QRF{C}_{r}x$

21                Else

22                        $Q{R}_{r}x$ ← $D_{r}x$ + $QRF{C}_{r}x$

23         Else If $D_r\theta$ = 180

24                $Q{R}_{r}x$ ← $D_{r}x$ – $DQ{W}_{r}xy$

25                If $Q{R}_{f}x$ > $F{C}_{f}x$

26                        $Q{R}_{r}y$ ← $D_{r}y$ − $QRF{C}_{r}x$

27                Else

28                        $Q{R}_{r}y$ ← $D_{r}y$ + $QRF{C}_{r}x$

29         Else If $D_r\theta$ = 270

30                $Q{R}_{r}y$ ← $D_{r}y$ − $DQ{W}_{r}xy$

31                If $Q{R}_{f}x$ > $F{C}_{f}x$

32                        $Q{R}_{r}x$ ← $D_{r}x$ − $QRF{C}_{r}x$

33                Else

34                        $Q{R}_{r}x$ ← $D_{r}x$ + $QRF{C}_{r}x$

35         Else

36                If $Q{R}_{f}x$ > $F{C}_{f}x$

37                        $Q{R}_{r}x$ ← $Inte{r}_{r}x$ + $QRF{C}_{r}x$ × cos($QR{W}_r\theta$)

38                        $Q{R}_{r}y$ ← $Inte{r}_{r}y$ + $QRF{C}_{r}x$ × sin($QR{W}_r\theta$)

39                Else If $Q{R}_{f}x$ = $F{C}_{f}x$

40                        $Q{R}_{r}x$ ← $Inte{r}_{r}x$

41                        $Q{R}_{r}y$ ← $Inte{r}_{r}y$

42                Else

43                        $Q{R}_{r}x$ ← $Inte{r}_{r}x$ + $QRF{C}_{r}x$ × cos(($QR{W}_r\theta$ + 180)%360)

44                        $Q{R}_{r}y$ ← $Inte{r}_{r}y$ + $QRF{C}_{r}x$ × sin(($QR{W}_r\theta$ + 180)%360)

END

The NotMatchedMapping algorithm operates in a situation where the image center of the front (load) camera does not align with the center of the load QR code, as illustrated in Figure 25a. When observing this situation through the front (load) camera, the resulting image is depicted in Figure 25b. With this information, $QRF{C}_{f}x$ and QR$F{C}_{f}y$ can be obtained in line 1 and $QRF{C}_{r}x$ and $QRF{C}_{r}y$ can be derived in line 2. To calculate $DQ{W}_{r}xy$, we can use the modeling in Figure 26, which is the same as that the localization algorithm for the mismatch situation. With this, $DQ{W}_{r}xy$ can be calculated in line 4. Next, we determine the height of the QR code ($Q{R}_{r}z$), which can be derived from Figure 25b (lines 7–10). If the QR code center is above the image center in the image coordinate system (line 7), then $Q{R}_{r}z$ in the world coordinate system is equal to $D_{r}z$ + $QRF{C}_{r}y$ (line 8). If the opposite situation occurs (line 9), $Q{R}_{r}z$ becomes $D_{r}z$−$QRF{C}_{r}y$ (line 10).

To obtain $Q{R}_{r}x$ and $Q{R}_{r}y$, we need to determine ($Inte{r}_{r}x$, $Inte{r}_{r}y$) as indicated in Figure 27. In addition, ($Inte{r}_{r}x$, $Inte{r}_{r}y$) refers to the x and y coordinates where the line passing through the center of the front camera and the wall where the QR code is attached intersects. We can derive this in line 5 of the algorithm.

To determine the angle $QR{W}_r\theta$ formed by the wall where the QR code is attached and the x-axis in the absolute coordinate system, we refer to Figure 28. In NotMatchedMapping algorithm, line 6 can be derived based on this information. The reason the direction the drone is facing and the wall where the QR code is attached are perpendicular is Assumption 8.

The next step involves the modeling process to determine $Q{R}_{r}x$ and $Q{R}_{r}y$, which varies slightly depending on the magnitude of $D_r\theta$, as illustrated in Figure 29. When $D_r\theta$ is 90 (Figure 29b), we can determine $Q{R}_{r}x$ and $Q{R}_{r}y$ based on the position of the QR code in the image captured by the drone’s front camera. This can be performed using lines 17–22 of the algorithm. In the case where $D_r\theta$ is 180 (Figure 29c), we can determine $Q{R}_{r}x$ and $Q{R}_{r}y$ based on the position of the QR code in the image captured by the front camera of the drone using lines 23–28 of the algorithm. When $D_r\theta$ is 270 (Figure 29d), we can determine $Q{R}_{r}x$ and $Q{R}_{r}y$ based on the position of the QR code in the image captured by the front camera of the drone. This can be done using lines 29–34 of the algorithm. For any other value of $D_r\theta$, (Figure 29e), we can determine QR$r_x$ and $Q{R}_{r}y$ based on the position of the QR code in the image captured by the front camera using lines 35–44 of the algorithm.

4. Experimental Results

4.1. Experiment on Simulator

4.1.1. Experimental Environment on Simulator

Simulator experiments and actual drone experiments were conducted to evaluate the performance of the proposed algorithm based on the given assumptions. The simulator, depicted in Figure 30a, allows for a flexible setup of the experimental environment. Cameras were attached underneath and in the front of the drone, Figure 30b, enabling the simultaneous recording of two videos to easily synchronize between them. Additionally, the simulator incorporated the positional information of the drone automatically, as illustrated in Figure 30c, enabling real-time logging of data for comparison.

4.1.2. Experimental Results for Drone’s 3D Localization

Table 1. Example experimental results of localization on simulator (unit: x,y,z: cm, $D_r\theta$: angle).

Image Number	Axis	Real Position (a)	Estimation Results (b)	|(a) − (b)|
	x	0.00	0.16	0.16
image1	y	190.00	191.09	0.25
	z	234.00	231.59	2.41
	$D_r\theta$	90.00	90.00	0.00
	x	0.00	0.16	0.16
image2	y	205.99	191.09	1.09
	z	190.00	231.59	2.41
	$D_r\theta$	90.00	90.00	0.00
	x	167.99	168.68	0.69
image3	y	200.00	199.94	0.06
	z	90.00	85.59	4.11
	$D_r\theta$	90.00	89.89	0.11
	x	250.23	252.42	2.19
image4	y	597.81	598.27	0.46
	z	155.51	159.00	3.14
	$D_r\theta$	97.19	97.22	0.03
	x	1.31	1.31	0
image5	y	600.52	600.52	0
	z	97.71	97.71	0
	$r_{\theta }$	64.80	65.74	0.94

presents example data for the experimental results. The performance measure of the experiment compares the actual drone position (a) obtained from the simulator with the estimated drone position (b) derived from the images, specifically in terms of the differences in the x-, y-, and z-coordinates (|(a) − (b)|). In addition, D$r\theta$ represents the forward direction of the current drone. Full experimental results can be found in

Table A1. Experimental results of localization on simulator (unit: $x,y,z$: cm, $D_r\theta$: angle).

Image Number	Axis	Real Position (a)	Estimation Results (b)	|(a) − (b)|
	x	0.00	0.16	0.16
image1	y	190.00	191.09	0.25
	z	234.00	231.59	2.41
	$D_r\theta$	90.00	90.00	0.00
	x	0.00	0.16	0.16
image2	y	205.99	191.09	1.09
	z	190.00	231.59	2.41
	$D_r\theta$	90.00	90.00	0.00
	x	167.99	168.68	0.69
image3	y	200.00	199.94	0.06
	z	90.00	85.59	4.11
	$D_r\theta$	90.00	89.89	0.11
	x	250.23	252.42	2.19
image4	y	597.81	598.27	0.46
	z	155.51	159.00	3.14
	$D_r\theta$	97.19	97.22	0.03
	x	1.31	1.31	0
image5	y	600.52	600.52	0
	z	97.71	97.71	0
	$r_{\theta }$	64.80	65.74	0.94
	x	610.20	609.97	0.23
image6	y	606.52	606.44	0.08
	z	132.00	128.34	3.66
	$r_{\theta }$	270.00	270.00	0.00
	x	0.0	0.16	0.16
image7	y	205.99	205.74	0.25
	z	239.99	231.79	2.20
	$r_{\theta }$	90.00	90.00	0.00
	x	424.20	420.78	3.42
image8	y	603.52	603.46	0.06
	z	210.0	206.89	3.11
	$r_{\theta }$	270.00	270.27	0.27
	x	2.99	3.16	0.17
image9	y	211.99	211.65	0.34
	z	287.99	287.25	0.74
	$r_{\theta }$	90.00	90.00	90.00
	x	13.20	12.94	0.26
image10	y	603.52	603.50	0.02
	z	81.00	76.49	4.51
	$r_{\theta }$	270.00	270.10	0.10
	x	13.20	12.94	0.26
image11	y	603.52	603.50	0.02
	z	81.00	76.49	4.51
	$r_{\theta }$	270.00	270.10	0.10
	x	2.99	3.09	0.10
image12	y	211.99	211.68	0.31
	z	254.99	249.97	5.02
	$r_{\theta }$	90.00	90.00	0.00
	x	2.99	3.09	0.10
image13	y	211.99	211.68	0.31
	z	254.99	249.97	5.02
	$r_{\theta }$	90.00	90.00	0.00

–

Table A4. Experimental results of localization on simulator (unit: $x,y,z$: cm, $D_r\theta$: angle).

Image Number	Axis	Real Position (a)	Estimation Results (b)	|(a) − (b)|
	x	272.55	270.86	1.69
image42	y	597.81	598.27	0.46
	z	155.51	159.00	3.49
	$r_{\theta }$	97.19	97.22	0.03
	x	875.13	876.76	1.63
image43	y	208.74	209.13	0.39
	z	177.00	174.00	3.00
	$r_{\theta }$	74.70	73.96	0.74
	x	357.40	358.02	0.62
image44	y	591.03	591.13	0.10
	z	159.00	155.98	3.02
	$r_{\theta }$	86.39	86.39	0.00
	x	880.87	882.54	1.67
image45	y	210.46	211.30	0.84
	z	177.00	174.34	2.66
	$r_{\theta }$	72.90	71.85	1.05
	x	862.07	858.01	4.06
image46	y	564.84	565.70	0.86
	z	177.00	174.13	2.87
	$r_{\theta }$	277.20	276.49	0.71
	x	753.52	752.24	1.28
image47	y	585.35	583.67	1.68
	z	456.00	457.11	1.11
	$r_{\theta }$	324.00	323.37	0.63
	x	581.2	579.08	2.12
image48	y	460.15	461.73	1.58
	z	456.00	456.03	0.03
	$r_{\theta }$	324.00	324.06	0.06
	x	835.20	834.43	0.77
image49	y	567.52	568.20	0.68
	z	177.00	173.96	3.04
	$r_{\theta }$	270.00	270.00	0.00
	x	0.00	0.00	0.00
image50	y	2510.01	2509.79	0.22
	z	233.99	231.71	2.28
	$r_{\theta }$	110.69	111.08	0.39

.

We collected 50 data points similar to those in Table 1. Figure 31 is a box plot graph summarizing the results of the entire dataset. According to the experimental results, the average errors along the x, y, and z axes were approximately 0.83, 0.32, and 2.95 cm, respectively. The angular error was approximately 1.08 degrees.

Figure 32 visually represents the logged data of the actual path taken by the drone in the simulator and the estimated path obtained by the proposed positioning estimation algorithm. The red line in the figure represents the actual path taken by the drone in the simulator, whereas the yellow line represents the estimated path generated by the proposed positioning estimation algorithm. In Figure 32a, viewed from a vertical perspective, the x- and y-axes closely align, appearing overlapped. However, in Figure 32b, viewed from a side angle a consistent error occurs along the z-axis.

4.1.3. Experimental Results for LoadMapping Algorithm

Simulator experiments were conducted to measure the accuracy of the mapping algorithm. The simulator in Figure 33 was created where an environment was set up with floor QR codes and attached payload QR codes (Figure 33a). The drone was then maneuvered in the simulator while simultaneously capturing images of the floor and payload. The mapping results for the payload QR code positions were visualized, as shown in Figure 33b. In the figure, the purple spheres represent the visualized positions of the QR codes estimated by the mapping algorithm. The white rectangles (dark-colored rectangles) attached to the floor in Figure 33b represent the floor QR codes that are not estimated. The positions of the payload QR codes (red rectangles) are represented by the center coordinates of the purple spheres. The center coordinates of the purple spheres are observed in Figure 33c. Subsequently, the averages were compared by randomly selecting five purple spheres (estimated results) from QR codes with five or more purple spheres.

Table 2. Experimental result example of Loadmapping on simulator (unit: cm).

QR No.	Axis	Real Position (a)	Result 1	Result 2	Result 3	Result 4	Result 5	Average Result (b)	|(a) − (b)|
	x	32.50	31.29	31.28	31.73	31.62	31.33	31.45	1.05
QR 1	y	285.30	271.42	287.77	284.16	290.51	284.61	283.69	1.61
	z	236.50	236.05	245.41	239.06	242.55	233.50	239.31	2.81
	x	32.50	32.28	32.00	33.45	32.87	32.28	32.57	0.07
QR 2	y	286.40	289.44	290.37	291.45	284.35	281.21	287.36	0.96
	z	126.40	122.73	121.78	123.56	121.78	124.21	122.81	3.59
	x	399.20	399.21	399.26	399.22	398.45	401.26	399.48	0.28
QR 3	y	291.40	293.70	293.70	293.78	294.54	292.97	293.74	2.34
	z	75.60	74.44	71.11	70.84	78.24	76.21	74.17	1.43

presents the some results of the mapping algorithm measured using the simulator. For the QR codes in the simulator, the average errors were approximately 0.87 cm along the x-axis, 1.64 cm along the y-axis, and 2.11 cm along the z-axis. This result can also be confirmed through Figure 34. These results are consistent with the observations in Figure 33c, where the estimated positions (purple spheres) are densely clustered around the actual QR code positions (white rectangles). Full experimental results can be found in

Table A5. Experimental results of Loadmapping on simulator (unit: cm).

QR No.	Axis	Real Position (a)	Result1	Result2	Result3	Result4	Result5	Average Result (b)	|(a) − (b)|
	x	32.50	31.29	31.28	31.73	31.62	31.33	31.45	1.05
QR 1	y	285.30	271.42	287.77	284.16	290.51	284.61	283.69	1.61
	z	236.50	236.05	245.41	239.06	242.55	233.50	239.31	2.81
	x	32.50	32.28	32.00	33.45	32.87	32.28	32.57	0.07
QR 2	y	286.40	289.44	290.37	291.45	284.35	281.21	287.36	0.96
	z	126.40	122.73	121.78	123.56	121.78	124.21	122.81	3.59
	x	823.46	826.85	826.94	826.40	819.92	821.22	824.27	0.81
QR 3	y	291.40	293.70	293.70	293.78	294.54	292.97	293.74	2.34
	z	75.60	74.44	71.11	70.84	78.24	76.21	74.17	1.43
	x	399.20	399.21	399.26	399.22	398.45	401.26	399.48	0.28
QR 4	y	291.20	290.90	290.78	291.02	295.73	294.26	292.54	1.34
	z	171.31	173.01	169.43	170.31	169.23	170.09	170.41	0.90
	x	27.00	26.90	25.99	27.71	26.81	26.15	26.71	0.29
QR 5	y	500.10	502.69	498.98	504.39	505.89	502.82	502.95	2.85
	z	82.40	81.02	77.71	77.66	77.77	81.02	79.04	3.36
	x	604.00	603.63	604.30	610.26	598.46	610.26	605.38	1.38
QR 6	y	498.20	496.87	496.87	496.88	496.77	496.87	496.85	1.35
	z	73.54	75.74	69.29	69.22	69.31	69.22	70.56	2.98
	x	26.50	26.20	25.92	26.02	25.92	26.19	26.05	0.45
QR 7	y	683.80	682.33	687.98	685.06	687.98	682.33	685.14	1.34
	z	78.10	73.53	79.83	73.55	76.89	73.52	75.46	2.64
	x	382.90	377.42	379.89	380.35	380.55	382.72	380.19	2.71
QR 8	y	675.10	673.02	672.06	674.15	676.32	674.25	673.96	1.14
	z	286.30	284.06	284.78	286.05	285.86	286.90	285.53	0.77
	x	387.40	387.33	390.22	392.28	387.31	386.49	388.73	1.33
QR 9	y	190.90	190.26	187.28	187.73	187.64	193.67	189.32	1.58
	z	681.17	680.77	680.99	680.89	680.85	680.79	680.86	0.31
	x	808.10	811.70	807.17	811.74	807.35	807.17	809.03	0.93
QR 10	y	180.30	183.46	183.24	183.46	177.47	183.24	182.17	1.87
	z	682.70	679.35	681.09	679.35	681.09	680.99	680.37	2.33

.

4.2. Experiment Using Actual Drone

4.2.1. Experimental Environment Using Actual Drone

The experimental setup using an actual drone was configured as depicted in Figure 35. As shown in Figure 35a, an origin point and coordinate plane were established on the floor, with markings (black lines) made every 1 m on the floor. Floor QR codes were then attached at the intersections of these markings. In Figure 35b, the drone was flown to capture images of the floor QR codes. The proposed algorithm was applied to the captured footage to extract predicted results, which were compared to the actual position of the drone. For the actual position of the drone, as shown in Figure 35b, a sinker was attached to the drone. The actual x and y coordinates of the drone were measured relative to the position of the sinker on the floor, while the z-axis was measured based on the length of the line connected to the sinker. The drone used in the experiment was the DJI Mavic Mini 3 Pro(manufacturer: DJI, country: China).

4.2.2. The Experimental Results of Localization Algorithm Using an Actual Drone

Table 3. The experimental results of localization using an actual drone (unit: cm).

Position Number	Axis	Real Position (a)	Estimation Results (b)	|(a) − (b)|
	x	37.70	65.10	189.70
Position1	y	37.73	65.08	0.02
	z	189.70	189.22	0.48
	x	151.70	146.98	4.72
Position2	y	107.30	110.00	2.7
	z	189.70	182.76	3.06
	x	50.70	48.32	2.38
Position3	y	144.10	144.64	0.54
	z	189.70	187.67	2.03
	x	187.40	186.12	1.28
Position4	y	128.90	128.73	0.17
	z	189.70	195.73	6.09
	x	80.40	80.15	0.25
Position5	y	86.50	86.70	0.20
	z	189.70	190.45	0.75
	x	184.60	184.11	0.49
Position6	y	38.10	38.07	0.03
	z	189.70	186.79	2.91
	x	93.80	97.64	3.84
Position7	y	144.70	145.13	0.43
	z	189.70	188.33	1.37
	x	115.70	115.93	0.23
Position8	y	183.70	184.35	0.65
	z	189.70	193.17	3.47
	x	56.70	58.89	2.19
Position9	y	56.70	58.89	2.19
	z	189.70	189.38	0.32
	x	223.40	223.91	0.51
Position10	y	125.40	125.12	0.28
	z	189.70	193.50	3.8
	x	123.30	120.6	2.7
Position11	y	52.70	52.12	0.58
	z	189.70	189.51	0.19

presents the experimental results of localization accuracy based on actual drone experiments. In this experiment, due to the inability to accurately measure angles, only information along the x, y, and z axes was utilized to assess accuracy.

The experimental results show an average error of 1.55 cm, 0.51 cm, and 2.04 cm along the x, y, and z axes, respectively. Comparing these results with simulator experiments, the average errors along the x and y axes increased by approximately 0.57 cm and 0.41 cm, respectively. However, the overall error remained relatively consistent.

4.2.3. The Experimental Results of Load Mapping Algorithm Using an Actual Drone

In the load mapping experiment using an actual drone, an environment was set up by adding cargo QR codes, as shown in Figure 36. Building upon the setup described in Figure 35a, load QR codes were added to the wall to create the experimental environment.

The experimental results of the load mapping algorithm using an actual drone are presented in

Table 4. The experimental results of load mapping algorithm using an actual drone (unit: cm.)

Load QR Code Number	Axis	Real Position (a)	Estimation Results (b)	|(a) − (b)|
	x	174.60	178.22	3.62
Load QR Code1	y	0.00	−1.48	1.48
	z	135.10	130.14	4.96
	x	115.90	119.36	3.46
Load QR Code2	y	0.00	−0.37	0.37
	z	76.60	70.19	6.41
	x	27.40	29.94	2.54
Load QR Code3	y	0.00	−1.93	1.93
	z	151.40	147.96	3.44
	x	0.00	2.18	2.18
Load QR Code4	y	55.40	54.12	1.28
	z	132.60	127.73	4.87
	x	0.00	0.19	0.19
Load QR Code5	y	110.20	109.46	0.74
	z	101.06	97.26	4.34
	x	0.00	−1.14	1.14
Load QR Code6	y	145.10	155.53	10.43
	z	156.30	142.14	14.16

. The average errors observed were approximately 2.19 cm along the x-axis, 2.70 cm along the y-axis, and 6.36 cm along the z-axis. It is evident that these errors are larger than those presented in the simulator. Several reasons can account for these results. Firstly, errors may arise due to lens distortion. Secondly, changes in the drone’s position during hovering may contribute to inaccuracies. Lastly, difficulties in achieving perfect vertical alignment between the front camera and the wall surface could also contribute to the observed discrepancies.

4.3. Commercial Drone GPS-Based Average Error Experiment

In this section, we aim to measure the errors generated by the GPS-based positioning system mounted on the actual drone used in the experiment. To conduct the experiment, we set up the experimental environment as depicted in Figure 37a. QR codes were placed on the floor at 1 m intervals, with a red circle serving as the reference point. The drone was then positioned on top of these QR codes, and latitude and longitude were measured using the drone’s GPS. Additionally, as shown in Figure 37b, we assigned numbers to the QR codes and used QR1 as the reference point to establish the distances to each QR code (QR2 9) for comparison, serving as the ground truth for evaluation.

Before explaining the experimental results, it is necessary to determine the distance equivalent to one degree of latitude and longitude around the experimental location. Within the latitude range of 35 to 40 degrees, the distance per degree of latitude is approximately 110.941 to 111.034 km, while the distance per degree of longitude ranges from about 91.290 to 85.397 km. Since South Korea spans approximately latitude 37 degrees, we can calculate that one degree of latitude corresponds to approximately 110.988 km (11,098,800 cm) and one degree of longitude corresponds to approximately 88.343 km (8,834,300 cm). These values will be used to convert latitude and longitude differences into centimeters in

Table 5. Commercial drone GPS-based average error experiment results.

QR Code Number	The Distance to QR1 (a) (Unit: cm)	(Latitude, Longitude)	Difference in Latitude and Longitude from QR1	Converting Latitude and Longitude Differences into Centimeters (Unit: cm)	Distance to QR1 Calculated Using Latitude and Longitude (b) (Unit: cm)	|(a) − (b)|
QR1 (Base)	0.000000	(37.556171, 127.001319)	(0.000000, 0.000000)	(0.000000, 0.000000)	0.000000	0.000000
QR2	100.000000	(37.556181, 127.001312)	(0.000010, 0.000007)	(110.988000, 61.840100)	127.053273	27.053273
QR3	200.000000	(37.556186, 127.001312)	(0.000015, 0.000007)	(166.482000, 61.840100)	177.596324	22.403676
QR4	100.000000	(37.556187, 127.001323)	(0.000016, 0.000004)	(177.580800, 35.337200)	181.062581	81.062581
QR5	141.421350	(37.556188, 127.001303)	(0.000017, 0.000016)	(188.679600, 141.348800)	235.752995	94.331645
QR6	223.606790	(37.556183, 127.001284)	(0.000012, 0.000035)	(133.185600, 309.200500)	336.665046	113.058256
QR7	200.000000	(37.556189, 127.001312)	(0.000018, 0.000007)	(199.778400, 61.840100)	209.130598	9.130598
QR8	223.606790	(37.556197, 127.001301)	(0.000026, 0.000018)	(288.568800, 159.017400)	329.482148	105.875358
QR9	282.842712	(37.556197, 127.001293)	(0.000026, 0.000026)	(288.568800, 229.691800)	368.822824	85.980111

.

The experimental results can be observed through Table 5. In summarizing the experimental results, latitude and longitude values were recorded up to the sixth decimal point to ensure maximum precision. Using the drone, latitude and longitude values were measured for QR2 through QR9, and the differences from QR1 were calculated. These differences were then converted into centimeters and used in distance measurement formulas to calculate the distances from QR1.

The results indicated that QR7 exhibited the least error at 9.130598 cm, while QR6 showed the largest error at 113.058256 cm. On average, an error of 67.361937 cm was observed. Comparing these results to the method presented in this paper, which showed errors ranging from 0 to 3 cm, it is evident that the proposed method outperforms the GPS mounted on the actual drone. Additionally, while GPS-based errors have a wide range, the proposed method offers a much narrower error range, highlighting its advantages.

5. Conclusions

The utilization of drones has diversified as the Fourth Industrial Revolution has continued to expand. Among various applications, this paper focuses on load management using drones. In general, when using autonomous drones, GPS receivers attached to the drones are used to determine the drone position. However, GPSs integrated into commercially available drones have an error margin on the order of several meters. To compensate for this, high-precision GPSs or drones equipped with such systems could be used, but they are expensive.

Additionally, in the context of load management applying drones, recent research has explored methods utilizing RFID and OCR. OCR techniques face the challenge that drones need to be in close proximity to the load, whereas RFID only determines the presence or absence of the load without providing accurate positioning information. Therefore, this study proposes a study on the drone position estimation and load-map generation method for load management, using the commonly available RGB camera and QR codes already integrated into commercially available drones. In terms of the related technology for position estimation and map generation, SLAM is one such method, while SLAM can generate the current drone position and a surrounding map, errors in drone trajectory caused by sensor noise can incorrectly generate maps and make this method impractical for generating maps specifically for load management. In this regard, this paper demonstrates that by utilizing fixed QR codes, the geometric relationship between the drone’s camera and the QR codes enables us to maintain the error of drone position estimation within a certain range, thereby enabling accurate mapping. This is evident from the experiments conducted.

In the Drone’s 3D localization experiment, the errors were maintained within a certain range, with average errors ranging from approximately 0 to 3 cm, showing minimal differences. During the mapping experiment, the average error between the actual and estimated positions of the QR codes was consistently around 0 to 3 cm. These metrics are expected to be useful in various drone industries, such as drone delivery and drone logistics, where more sophisticated positioning technologies are required.

This study focuses on utilizing QR codes to determine the position of robots and, based on their position, identify the location of cargo. However, when QR codes become unreadable due to factors such as strong light, shadows, distortion, tearing, or crumpling in complex environments, there may be situations where QR codes are obscured, making position determination challenging. Nonetheless, since the floor QR code is not always obstructed and the drone is continuously moving, it is conceivable to estimate the current position of the drone by calculating the timing of the last visible floor QR code and incorporating the drone’s direction and speed.