Autonomous Loading System for Load-Haul-Dump (LHD) Machines Used in Underground Mining

Abstract

This paper describes an autonomous loading system for load-haul-dump (LHD) machines used in underground mining. The loading of fragmented rocks from draw points is a complex task due to many factors including: bucket-rock interaction forces that are difficult to model, humidity that increases cohesion forces, and the possible presence of boulders. The proposed system is designed to integrate all the relevant tasks required for ore loading: rock pile identification, LHD positioning in front of the ore pile, charging and excavating into the ore pile, pull back and payload weighing. The system follows the shared autonomy paradigm: given that the loading process may not be completed autonomously in some cases, it takes into account that the machine/agent can detect this situation and ask a human operator for assistance. The most novel component of the proposed autonomous loading system is the excavation algorithm, and the disclosure of the results obtained from its application in a real underground production environment. The excavation method is based on the way that human operators excavate: while excavating, the bucket is tilted intermittently in order to penetrate the material, and the boom of the LHD is lifted on demand to prevent or correct wheel skidding. Wheel skidding is detected with a patented method that uses LIDAR-based odometry and internal measurements of the LHD. While a complete loading system was designed, the validation had to be divided in two stages. One stage included the rock pile identification and positioning, and the other included the charging, excavation, pull back, and weighting processes. The stage concerning the excavation algorithm was validated using full-scale experiments with a real-size LHD in an underground copper mine in the north of Chile, while the stage concerning the rock pile identification was later validated using real data. The tests showed that the excavation algorithm is able to load the material with an average of 90% bucket fill factor using between three and four attempts (professional human operators required between two and three loading attempts in this mine).

1. Introduction

The automation of mining equipment is an important requirement in the mining industry. This is because mining operations need to increase the safety of the workers, as well as to augment the productivity, efficiency, and predictability of the processes. Safety is, without doubt, a key factor, and has been the top priority of mining companies during the past decades. This is particularly true for underground operations, with hazardous environments where workers are exposed to constant risks of rock falls, rock bursts, and mud rushes, and where the presence of dust in the air can result in a number of associated occupational diseases in workers [1]. All of these hazards have been steadily increasing as mine operations have gone deeper, and geomechanical conditions therefore become more extreme. As a consequence, great effort has been invested in increasing the automation level of underground mining machines, especially those that operate in high-risk areas [1], one of them being load-haul-dump (LHD) machines (also known as scoop trams).

An LHD is a four-wheeled, center-articulated vehicle with a frontal bucket used to load and transport ore on the production levels of an underground mine (see Figure 1). The automation of the transport phase has been covered in many previous publications [2,3,4,5,6] and is currently commercially available. However, although the loading part has also been a subject of scientific research, [7,8,9,10,11,12,13,14,15,16,17], its application in mining environments is not as widely used as other types of automation solutions such as autonomous navigation, assisted tele-operation, or collision avoidance. The slower-paced progress in this area could be due to the complexity of the addressed problem: during excavation, the interaction between the bucket and the material is difficult to model, because the bucket-material interaction forces may vary greatly depending on the properties of the material (e.g., humidity, hardness, fragmentation), the rock pile geometry, and the LHD dynamics (mass, speed, etc.). For this reason, the development of autonomous loading solutions requires full-scale experiments under conditions that can be found only in real mining environments, since physical interactions between the machine and the fragmented rock cannot be easily reproduced elsewhere. This adds to the fact that loading maneuvers and techniques are mostly developed and perfected through the experiences of human operators.

In this paper, a complete system for the automation of the loading process using LHDs is proposed. This system is designed to integrate all the relevant tasks required for ore loading: rock pile identification, the LHD’s positioning in front of the ore pile, charging, excavation, pull back, and payload weighing (see Figure 2). Assuming that in some cases the loading may not be completed autonomously, the system can detect this situation and request the help of a human operator (by tele-operation). Thus, the proposed system falls under the shared autonomy paradigm. In the mining exploitation context, fleets of LHDs are normally supervised by humans, so providing assistance to an autonomous equipment is not an unusual requirement, especially considering that the most common practice is for LHD automation systems to rely on tele-operators to handle the loading task. It must be noticed that providing assistance on demand, i.e., only in the few cases that the autonomous system is not able to load, is much more efficient than the current semi-autonomous operation used in most commercial systems, where navigation is executed autonomously but loading is teleoperated.

The core of the autonomous loading system is the excavation algorithm, which is based on the way that human operators excavate: the bucket is tilted intermittently while excavating in order to penetrate the material, and the boom of the LHD is lifted on demand to prevent or correct wheel skidding. Wheel skidding is detected with a patented method that uses LIDAR-based odometry and internal measurements of the LHD’s actuators [18]. The excavation algorithm was validated in an underground mine (a sublevel stoping copper mine located in Chile), using full-scale excavation experiments with a real LHD and a typical production rock pile. The validation process in a productive area of a real mine and the lessons learned are fully disclosed here. The 2D-pile modeling was validated afterwards with data from these tests.

It is also important to mention that commercial solutions to the problems of autonomous loading need to be compatible with the 24 × 7 operation of mines, where production throughput is one of the main requirements. That means that the LHD needs to be able to load without stopping during the transition from navigation to loading. The proposed automation framework considers this requirement, and it is able to characterize the rock pile, without stopping the LHD, while it approaches the pile.

The main contributions of this paper are:

• A system that models and implements the whole loading process required in underground mining operations, from rock pile identification to payload weighing. The system is based on the shared autonomy paradigm, which allows the system to obtain assistance on demand from a human operator.
• An excavation algorithm that is based on the way that human operators excavate, and that uses a patented method for detecting wheel skidding, which is required for successful excavation of fragmented material in draw points.
• Full-scale rock excavation experiments using a commercial LHD in a production level of a sublevel stoping mine.

This paper is organized as follows: Section 2 presents the background and related work on loading automation for LHDs. Section 3 describes the proposed autonomous loading system. In Section 4, results of the full-scale experiments are shown and discussed. Finally, in Section 5, the main conclusions of this work are drawn.

2. Background and Related Work

2.1. Problem Description

Autonomous loading of fragmented rock involves a sequence of steps: As the LHD’s operator approaches the draw point, a swift assessment of the condition of the rock pile is made. The bucket of the LHD is tilted downwards until the tip is pressed against the ground. Next, the machine is commanded to charge at the rock pile. After the LHD makes contact, it buries the bucket in the rock pile, and bucket tilt and lift movements are issued as it advances through the fragmented rock. When sufficient penetration is achieved, the bucket is fully retracted, and the vehicle is withdrawn with a reverse motion. Then, the operator shakes the bucket briefly and assesses whether or not enough ore has been loaded. If necessary, a new loading maneuver is carried out.

The step in which the machine is being controlled to dig into the rock pile will herein be referred to as the excavation process. During excavation, bucket-rock interaction forces affect the bucket motion through the material, and ultimately define the amount of ore loaded. Granular material inter-particle forces, as well as bucket-rock interactions, have been studied previously, but their complexity has led only to stochastic modeling [19]. In consequence, an analytical approach to bucket motion control is not possible, and accurately replicating these interactions to develop a system in simulation is highly unlikely. Hence, only a real environment offers suitable conditions for development of, and experimentation with, an excavation procedure. Furthermore, fragmented rock, unlike piled sand or gravel, is much more difficult to load. This is aggravated due to the environmental conditions in underground mining. Depending on the mining method, the column of blasted rock might exert large compression forces on the open face of the draw point (see Figure 3). Humidity can also increase cohesion forces, making the overall process more difficult [20]. Moreover, large rocks can be present in the draw point. For this reason, the operator needs to detect the presence of large rocks while approaching the draw point, and, depending on their size, could be forced to change the goal of the maneuver from filling the bucket to just loading one large rock. If the rock is too big to be loaded, the task is aborted. Furthermore, other conditions, such as narrow tunnels, uneven or unprepared terrain, and/or significant amounts of water and mud near the draw point, can make the loading process very challenging. An autonomous system should be prepared to handle all these situations in addition to performing certain processing and steps that a human operator deals with while driving the LHD, such as detecting an accurate enough position of the rock pile before charging at it, making sure not to hit the tunnel while pulling back from the rock, and estimating if the bucket is full enough.

2.2. How Human Operators Excavate

The excavation process is the core of the loading maneuver; it is the crucial step and, since bucket-rock interactions cannot be predicted in advance, there is no optimal method for performing this operation. Nevertheless, human operators have had to deal with this situation by learning the loading technique from experience and shared good practices. It is natural, therefore, to analyze which ideas can be borrowed from them. Through several interviews with LHD operators of Chilean block caving and sublevel stoping mining companies, and discussions and analysis of data from mining operation manuals, some common points in their practices were identified:

• Before engaging with the rock pile, the bucket must be fully extended.
• Forward motion is selected for maximum traction (first gear).
• Intermittent tilt commands are the basis of the excavation process.
• Lift commands are mainly issued to prevent or correct wheel skidding.

Figure 4 shows a graphical representation of an excavation maneuver. First, the LHD engages with the rock pile, then the bucket is tilted in order to excavate on the muck pile. When the wheels lose traction, the boom is lifted to regain traction. After filling the bucket, the LHD pulls back. Figure 5 shows a graphic of some of the relevant variables during a line-of-sight remote-controlled loading maneuver. As mentioned, bucket tilting (“tilt command”) is activated intermittently. Its duration and shape vary among operators and loading attempts. Techniques also using negative tilt commands (downwards tilting of the bucket) were also encountered in combination with positive lift commands (“lift command”). The common factor in all of them was the intermittency of the commands. As for the lift command, its use was seen to be highly variable, which correlates with the notion of using it mainly to prevent wheel skidding. In the example of Figure 5, the lift command was not used at all. The “pedal command” is the one used for machine acceleration and thus, in turn, engine revolutions per minute (RPM). “Tilt angle” and “lift angle” are the bucket tilt and boom lift angle values, estimated through the encoders installed in the machine. “Transmission pressure” is the hydrostatic transmission pressure of the machine, a variable that can be associated with the engine’s power output. “Skidding factor” is an estimated skidding factor that is explained in Section 3.3.

2.3. Related Work

Mikrived analyzed the interaction forces in excavation processes and proposed that measurements of the resistive forces could be used for controlling them [21]. Hemami studied the LHD loading process and dealt with theoretical formulations on bucket trajectory and resistive forces during a scooping motion [7,8,10]. Other work has addressed the loading problem using laboratory-scale prototypes, under less realistic conditions [9,11,12,22]. Research on autonomous loading showed scarce progress up until the past decade. In [13], full-scale experiments with a real LHD were performed using a proposed admittance-based controller for bucket commands. This work was further extended in [15,16,17] with support from the LHD original equipment manufacturer (OEM) Atlas Copco (now Epiroc), a study that led to a patented method [23]. It should be noted that this method addresses only the problem of ore excavating once the bucket is inside the rock pile. On the other hand, our approach comprehends the complete loading process, while also benefiting from human experience.

Caterpillar, another LHD OEM, is known to have been working on technical solutions for the autonomous loading problem, as demonstrated by patents [24,25,26]. While their patents show that they have tackled different issues of the autonomous loading problem, a general system has not been described, and their research has not been made available to the scientific community. However, it is uncertain if either of these solutions are able to handle the loading problem in block/panel caving draw points, which is significantly harder than loading from an ore pile or from draw points originated by other mining methods. As mentioned, loading from draw points is difficult due to many factors including: bucket-rock interaction forces that are difficult to model, humidity that increases cohesion forces, and the possible presence of boulders.

According to the public information shared on the web sites of LHD OEMs, only Epiroc, Sandvik, and RCT Global have commercially available products that focus on the autonomous loading task. However, specific features, limitations, and applicability for block/panel caving or sublevel stoping underground mining are not publicly disclosed, and therefore no comments can be made on them. At least in Chile, where all major mining companies operate and where one third of the world’s copper is produced, no autonomous loading system for LHD is in use.

Work associated with standard front-end loaders was not considered in this analysis, because they operate on the surface, where the difficulties associated with underground draw points are not present. In addition, these types of vehicles usually deal with material that has already been processed by milling equipment.

Our previous work on the topic of autonomous loading with LHDs was focused on laboratory-scale trials [27]. While useful for an initial approach to an experimental setup, laboratory experiments are limited due to the large deviation between real-world operating conditions and data from a small-scale and controlled environment. Other laboratory work is presented by Shi [22], which has some similarities with the here-presented method, as excavation is modeled after the experience of human operators. However, since their experiments are carried out using a robotic arm, it is difficult to apply those findings to a four-wheeled machine such as an LHD.

As can be seen, little work has been published on the particular case of autonomous loading with LHD machines in real underground mining environments, and, as far as we know, no previous method has validated an excavation technique using human operator experiential knowledge under real underground mining conditions.

3. Proposed Autonomous Loading System

3.1. Methodology Overview

The hardware used in this work included a full-scale LHD machine (a GHH LF11H with 11-ton tramming capacity and hydraulic powertrain) retrofitted with two 2D laser scanners (SICK LMS511), encoders in all joints (IFM RN7012), pressure sensors along the bucket’s tilt and the boom’s lift hydraulic lines (manufacturer specific, 600 bar), and an onboard industrial computer (Advantech ARK-2151-59AIE) (see Figure 1). All components are of industrial grade, suitable for the rigorous environment of underground mining, and readily available as off-the-shelf components used widely in a broad range of commercial applications. In order to better integrate the system with commercial autonomous navigation solutions for LHDs, which only use 2D scanners, 3D laser scanners were not used in the current version of the proposed system. The software implementation was performed using ROS as the central middleware [28], CABSL library [29] for state-machine codification, and ROS actions to interface with the main processing function in each state. The system was initially developed and tested in a simulated environment using Gazebo [30].

An autonomous loading routine can be thought of as a sequence of several steps that perform specific actions, as presented in the diagram of Figure 2. There, rock pile identification is the process of finding the draw point’s location and status (whether it is suited for loading or not). Positioning refers to orienting the LHD machine to ensure that a forward thrust will end in a collision with the rock pile. Charging is the step that considers lowering the bucket and accelerating towards the pile. Excavation is the machine control that performs the digging action. Pull back comprises the backward motion of the vehicle and bucket shake. Finally, Payload weighing estimates the amount of ore loaded. Most of the mentioned steps could fail due to the highly variable environment in underground mining. It is possible that the onboard sensors fail to find the rock pile accurately or that the machine positioning cannot avoid colliding with a tunnel wall. The collision between the machine and the rock pile might not be detected, and the LHD’s bucket could become stuck during excavation. In any of these cases, human assistance would be required to overcome the problematic situation. In fact, human interaction with automated mobile machines for mining operations is needed in the state-of-the-art systems, at the very least to supervise a fleet of machines. Since large-scale operations have a fleet of vehicles, it is common for one or more operators to supervise and assist multiple semi-autonomous machinery, as each machine does not require constant attention. For these reasons, in our view, it is imperative that this human interaction is considered at the design stage of the autonomous loading system and, thus, included in the formulation of the agent’s behavior. Figure 6 shows the proposed state machine for the autonomous loading process with the human in the loop. There are two assistance states: positioning assistance allows the operator to take control of the machine to position it in front of the rock pile, while loading assistance provides the operator with full control of the LHD to perform the remaining parts of the loading maneuver up until payload weighing, which is needed to record the performance of the task. The positioning assistance is called either when the rock pile identification cannot characterize the rock or when positioning is not able to place the LHD to start the charging procedure, while loading assistance is called when the charging, the excavation, or the pull back procedures cannot be successfully completed.

3.2. Environment Modeling and Rock Pile Identification

Of the six steps of the autonomous loading routine, the first one, rock pile identification, is the one that most strongly relies on the processing of data acquired from the environment. The aim of this step is to find the location and status of the draw point accurately, in real time, and without the need of extra steps or delays. This means that the LHD is required to continue moving while scanning the pile. The characterization of the rock pile is performed through integration of consecutive laser scans from the frontal 2D LIDAR sensor mounted on the LHD. The integration of the laser scans relies on a LIDAR-based odometry method [31], and the output of the machine’s inclinometer data, which is filtered and integrated in the local coordinate frame in order to benefit from the natural tilting of the LHD while moving across uneven terrain. The aforementioned odometry method consists of an adaptive frame rate modification of the range-flow based odometry (RF2O) [32]. While RF2O computes displacement by comparing two scans, using the “range flow constraint equation”, the adaptive frame rate implements a control over the processing frequency, increasing it at higher machine speeds.

A block diagram for the pile identification step is presented in Figure 7. The LIDAR odometry enables the integration of 2D laser scans into a 3D point cloud of the environment. Then, the rock pile is segmented, and its features are computed (center, limits, and slope).

A conceptual diagram of the complete pile identification process is shown in Figure 8. The succession of images (a, b, and c) illustrates how the integration is carried out. For each new scan (colored circles), the data is registered in an accumulated point cloud, using the computed odometry. It is important to note that the whole process must be executed in real time without stopping, or even slowing down, the LHD. Moreover, the fact that the LIDAR reaches the bottom of the pile from a long distance away, because it is tilted down 2.5 degrees, results in the point cloud having very low density, all of which increases the complexity of the problem.

Once enough scans have been integrated, the rock pile is segmented using the normal vectors of the surface, a curvature filter, and neighbors count in a small radius. The normal vectors provide information about the points belonging to the walls or to the pile (normal ny is negative on the left wall, near zero on the pile, and positive on the right wall). The curvature provides a rotation-invariant description of the points, which can be used for further segmenting the pile. Finally, neighbor counts provide information valuable for rejecting outliers (points that are isolated from all others). Once a pile is segmented successfully, its mean slope is estimated and is used to classify the status of the pile: a too shallow slope indicates there is not enough ore to perform a loading, and a too steep slope signals the presence of a tunnel wall instead of a rock pile, or even a rock pile in a “hang-up” state, a condition that can arise in certain underground mining methods, such as block caving, in which the fragmented rock stops flowing, generating an unstable and hazardous condition. The determination of the status of the pile is performed by a simple thresholding of its slope.

For clarity purposes, the described algorithm for segmenting and characterizing the pile is detailed in Figure 9.

Besides the pile status, the locations of the horizontal bounds are needed for the next step of the loading routine, the machine positioning. The left (E_L) and right (E_R) limits of the rock pile are required to be able to direct the machine without colliding with the tunnel walls. Using the information available from the pile segmentation algorithm, the limits are estimated as E_L = max_y, and E_R = min_y (see definitions in Figure 9).

The normal vectors of the points provide valuable information for segmenting the pile, as shown in Figure 10. In this figure, low values of the normal component ny are colored in blue, while high values are colored in red.

Oversized rock detection can also be performed in this identification step; however, the description of the functionality is outside the scope of this paper (see, for example, the detection system described in [33]).

If the identification of the rock pile has not finished by the time the LHD is closer than a certain predetermined distance to the pile (in the experiments, a threshold of 15 m was used), a transition to the positioning assistance state is forced in the behavior state machine from Figure 6. In the experiments, a threshold of 15 m was used as the predetermined distance. This value was deemed sufficient for the LHD to successfully achieve the following positioning in front of the rock pile. The overall loading routine is, nevertheless, not sensitive to small variations of the selected parameter, and its value should be selected based on the particular conditions of the operating environment.

3.3. Positioning and Charging

After the draw point location and status have been determined, the loading routine continues with the positioning of the machine. Here, the LHD is continued to be driven forward using a “guidance” algorithm [34] until the machine’s reference point (in this case its central articulation joint) is at a set distance from the center of the pile (computed as the center of gravity of the segmented points). Once this distance is met, small corrections to the steering angle are issued to ensure that the projected path lands within the limits of the rock pile. Finally, the bucket is lowered until a tight contact with the ground is achieved. A block diagram of this process is shown in Figure 11.

The projected path for the machine aligning procedure (see Figure 12) is computed using the LHD’s kinematic equations, reported in several previous publications, such as [35]:

$$\partial \left[x,y,\theta \right]/\partial t=\left[v \cos \left(\theta \right),v \sin \left(\theta \right),\left(v \sin \left(\gamma \right)+L_r\omega \right)/\left(L_f\cos \left(\gamma \right)+L_r\right)\right]$$

(1)

$$\partial \gamma /\partial t=\omega$$

(2)

where $\left[x,y,\theta \right]$ is the pose of the LHD, $v$ is the linear speed, $\gamma$ is the steering angle of the LHD’s pivot, $\omega =\dot{\gamma }$ is the steering speed of the LHD’s pivot, $L_r$ is the length from the LHD’s pivot to the rear wheel axis, and $L_f$ is the length from the LHD’s pivot to the front wheel axis.

Since the next step of the autonomous loading routine, the charging against the rock pile, does not involve steering commands, a fixed steering angle (constant γ, ω = 0) can be considered, which simplifies the model:

$$\partial \left[x,y,\theta \right]/\partial t=\left[v \cos \left(\theta \right),v \sin \left(\theta \right),v·K\right]$$

(3)

where $K$ is a constant value that depends on the LHD dimensions and the fixed steering angle. Equation (3) shows that an LHD charging against a rock pile will follow a circular trajectory, characterized by a center ($C$) and a curvature radius ($R$). Because the rock pile identification step computed the left ($E_L$) and right ($E_R$) pile edges, the problem of determining if the LHD will collide with the rock pile simplifies to checking if the line between $E_L$ and $E_R$ intersects with both the circular path of the left and right sides of the LHD. Figure 12 shows a diagram of the described scenario.

The solution for the intersection between a circle and a line segment is well known in the common literature and it involves computing the following discriminant value:

$$\mathsf{\Delta }={\left(2 \stackrel{\rightharpoonup }{E}·\stackrel{\rightharpoonup }{EC}\right)}^2-4{\left|\stackrel{\rightharpoonup }{E}\right|}^2\left({\left|\stackrel{\rightharpoonup }{EC}\right|}^2-R^2\right)$$

(4)

where $\stackrel{\rightharpoonup }{E}$ is the vector going from $E_R$ to $E_L$ and $\stackrel{\rightharpoonup }{EC}$ is the vector going from $E_R$ to $C$. If $\mathsf{\Delta }$ results in a negative value, then the infinite line defined by the points $E_R$ and $E_L$ never intersects the given circular path. On the other hand, if $\mathsf{\Delta }$ is positive, a solution exists. Let $S_1$ and $S_2$ be solutions to the intersection between the infinite line defined by $E_R$ and $E_L$ and the circle of center $C$ and curvature radius $R$:

$$\begin{gathered} S_1=\frac{-2 \stackrel{\rightharpoonup }{E}·\stackrel{\rightharpoonup }{EC}-\sqrt{\mathsf{\Delta }}}{2\left|\stackrel{\rightharpoonup }{E}\right|} \\ {S}_2=\frac{-2 \stackrel{\rightharpoonup }{E}·\stackrel{\rightharpoonup }{EC}+\sqrt{\mathsf{\Delta }}}{2\left|\stackrel{\rightharpoonup }{E}\right|} \end{gathered}$$

(5)

then, if $0<S_1<1$ or $0<S_2<1$, the circular path intersects the segment $\overline{E_L{E}_R}$. Otherwise, if $\min \left(\left|S_1\right|, \left|S_2\right|\right)<\min \left(\left|S_1-1\right|, \left|S_2-1\right|\right)$, the circular path passes to the right of the segment $\overline{E_L{E}_R}$. If none of the previous conditions is met, then the path passes to the left of segment $\overline{E_L{E}_R}$.

The above process is executed for the circular paths defined by the left and right side of the LHD. If one of them does not intersect the pile, the steering angle is adjusted in the appropriate direction until both paths land within the rock pile limits. During this steering adjustment, a simple collision check is computed periodically over a bounding box for each body of the LHD. Figure 13 shows a diagram of the collision check for the front body of the LHD. A bounding box is defined for the front body. Then, vectors going from the bottom left corner to the top left corner ($\stackrel{\rightharpoonup }{S_f}$) and to the bottom right corner ($\stackrel{\rightharpoonup }{B_f}$) are defined. In order to check if point $P$ of the detected tunnel wall is inside the bounding box (and hence causing a collision), the following condition is evaluated:

$$0<\stackrel{\rightharpoonup }{S_f}·\stackrel{\rightharpoonup }{d_f}<{\left|S_f\right|}^2 \wedge 0<\stackrel{\rightharpoonup }{B_f}·\stackrel{\rightharpoonup }{d_f}<{\left|B_f\right|}^2$$

(6)

where $\stackrel{\rightharpoonup }{d_f}$ is the vector going from the bottom left corner of the bounding box to point $P$. If the condition is true, then point $P$ is inside the collision bounding box.

Once the machine’s steering angle has been adjusted so that its projected path lands within the rock pile limits, the bucket of the LHD is lowered using the tilt hydraulic cylinder. The tilt cylinder is commanded to extend until the pressure in its hydraulic line rises over a predefined threshold, signaling a tight contact between the bucket tip and the ground. An example from full-scale experimental data is shown in Figure 14, where a threshold of 23 bars was selected.

If the above positioning process is not completed within a predefined time limit (in the experiments, a threshold of 30 s was used), an operator assistance request is generated, effectively putting the system in the positioning assistance state from Figure 6. Otherwise, once the vehicle is in position and with the bucket down, it is commanded to drive forward in first gear. The transition to the excavation phase occurs as the collision with the rock pile is detected by a patented method that analyzes the machine’s transmission pressure (engine load) and a skidding factor, estimated from the LIDAR odometry and the machine’s tachometer data [18]. A block diagram is shown in Figure 15.

The skidding estimator from Figure 15 computes the skidding factor with the following formula:

$$skid=\left(v_T-v_L\right)/v_{Mm}$$

(7)

where $skid$ is the skidding factor; $v_T$ is the speed measured from the machine’s tachometer; $v_L$ is the speed estimated with the LIDAR-based odometry method mentioned in Section 3.2., and $v_{Mm}$ is a normalizing factor, computed as the maximum between $v_T$ and $v_L$ in a fixed time window.

Simultaneously, the collision detector module monitors the pressure of the hydrostatic transmission system (for the used LF11H LHD, this variable correlates with the engine’s power output). If either the skidding factor or the transmission pressure rises above a predefined threshold, a collision detection event is flagged. Figure 16 shows a plot of the skid factor and the transmission pressure signals, alongside the tilt command, for a teleoperated loading maneuver. The collision with the rock pile happens just before the human operator activates the tilt command. As mentioned above, a significant rise in the transmission pressure, and of the skidding factor, can be seen at that point. Extensive analysis of experimental data, such as shown in Figure 16, led to transmission pressure threshold to be set at 300 bar, and the skidding factor threshold at 0.5.

If a collision detection is not generated within a predefined time limit (in the experiments, a threshold of 10 s was used), an operator assistance request is issued, forcing the system into loading assistance state from Figure 6.

3.4. Excavation Algorithm

The excavation algorithm is based on the techniques used by experienced human operators from block caving, panel caving and sublevel stoping operations (which are the most common type of underground operations in Chile). From a control theory point of view, the proposed method acts as a traction controller during the excavation process. A diagram is shown in Figure 17. The pregenerated commands is simply a module in charge of generating precomputed commands for the pedal and tilt signals. These commands are similar to the commands observed during manual operation. Specifically, the acceleration pedal is maintained at a constant output value and the bucket tilt is intermittently activated in the form of a ramp function. A ramp function was only selected in order to have an extra parameter, although a parameter sensibility analysis could not be performed in the full-scale experiments due to time constrains. The traction controller acts on the lift command in order to suppress wheel skidding, a variable that is estimated from Equation (7), as explained in Section 3.3. In practice, a simple “on/off” controller was implemented, where the lift command was set at the maximum value if the estimated skidding factor raised over a predefined threshold. This choice was made to keep the methods simple in favor of achieving a validation of the complete system.

The implementation of the excavation method was carried out using a two-state machine. In the first state, called $S_{tilt}$, a fixed acceleration and a ramp-shaped bucket tilt command are selected. In the second state, called $S_{push}$, the fixed acceleration command persists, and wheel skidding is analyzed. If the wheel-skidding factor rises above a predefined threshold, the boom lift command of the machine is activated in order to compensate for it. A machine-specific option, named “full RPM”, is also activated during this state. This option forces the engine to maximize its power output, providing additional penetration force. Formally, the following equations apply for the machine commands:

$$\begin{array}{lll}pedal& =P_0& \\ tilt& =\left\{\begin{array}{l}{R}_0\left(t-t_{tilt}\right)\\ 1\\ 0\end{array}\right.& \begin{array}{l}ifstate=S_{tilt}\wedge b\left(t-t_{tilt}\right)\le 1\\ ifstate=S_{tilt}\wedge b\left(t-t_{tilt}\right)>1\\ ifstate=S_{push}\end{array}\\ lift& =\left\{\begin{array}{l}0\\ 1\end{array}\right.& \begin{array}{l}ifstate=S_{tilt}\vee skid<ski{d}_{max}\\ ifstate=S_{push}\wedge skid\ge ski{d}_{max}\end{array}\\ fullRpm& =\left\{\begin{array}{l}0\\ 1\end{array}\right.& \begin{array}{l}ifstate=S_{tilt}\\ ifstate=S_{push}\end{array}\end{array}$$

(8)

where $P_0$, $R_0$, and $ski{d}_{max}$ are predefined constant values; $t_{tilt}$ is the start time of state $S_{tilt}$, and $skid$ is the skidding factor. As for the output commands, $pedal$ is the signal sent to the machine’s internal control unit that regulates acceleration (similar to the signal sent when pressing the physical pedal in the machine’s cabin), $tilt$ is the signal sent to retract (positive values) or extend (negative values) the cylinder that controls the movement of the bucket of the LHD machine, and $lift$ is the signal that commands the “boom” cylinder of the machine, that is, the arm that can lift (positive values) or lower (negative values) the bucket. Finally, the $fullRpm$ signal is, as mentioned, a machine-specific command that forces the engine’s RPM to the maximum, temporarily increasing the force of the hydraulic mechanism. It should be noted that analysis and correction of wheel skidding is only performed during the $S_{push}$ state. This decision was made based on the empirical observation that wheel skidding is much less likely to occur during bucket retraction and the fact that activating the lift cylinder effectively reduces the power available for the tilt hydraulics. In Figure 18, a flow chart of the excavation process is presented, where EP represents engine power (a variable observed, in this case, through the transmission pressure of the machine).

Both states, $S_{tilt}$ (tilt state) and $S_{push}$ (push state), have upper and lower bounds to their execution time, so it is guaranteed that their execution alternates during the excavation. Besides the time limit of $S_{push}$, the system can transition back to $S_{tilt}$ if the hydrostatic transmission pressure (or, more generally, the engine power output) climbs over a threshold value, meaning that the resistive force from the rock pile is too large, and the bucket is not able to penetrate further without a scooping motion. This alternation between states continues until any of the following stop criteria are met: the bucket is fully retracted, the bucket is lifted over a maximum angle, or the machine has advanced further than a maximum displacement value. This method was patented as reported in [36]. For clarity purposes, a pseudo code of the described algorithm is shown in Figure 19.

Table 1. Excavation method parameters used for the full-scale validation.

Parameter	Value
$P_0$	0.5
$R_0$	1.2
$ski{d}_{max}$	0.2
${\beta }_{max}$	0.5 [rad]
${\alpha }_{max}$	0.0 [rad]
$d_{max}$	2.0 [m]
$\mathsf{\Delta }{t}_{max}$	1.0 [s]
$\mathsf{\Delta }{t}_{min}$	0.5 [s]
$E{P}_{max}$	300 [bar]

shows the value of the parameters used for the full-scale validation experiments reported in Section 4. The numerical values were selected based on operation data.

In the case that the described excavation process is not completed within a predefined time limit (in the experiments, a threshold of 30 s was used), an operator assistance request is generated, thus putting the system in the loading assistance state from Figure 6.

3.5. Pull Back and Payload Weighing

Once the excavation is finished, the bucket gets fully retracted and the vehicle is driven backwards, using the same “guidance” navigation module from the positioning step. A predefined time limit is assigned to this step (in the experiments, 60 s were used). If a timeout occurs, the system is put in the loading assistance state from Figure 6.

After a predefined distance has been covered within the aforementioned time limit, the bucket is commanded to perform short downward and upward tilting movements, known as the “bucket shake”. Then, the bucket is lifted up to a fixed angle (${\beta }_{weigh}$), and the pressure sensors in the boom hydraulic line are used to estimate the bucket fill factor. A fill factor is used instead of the mass weight, since it simplifies calibration, while providing the same information for the evaluation process. The relationship between the lift cylinder pressure ($P$) and the fill factor was found experimentally and corresponds to the following affine model:

$$f_{fill}=mP-n$$

(9)

where $f_{fill}$ is the relative fill factor (0 for an empty bucket and 1 for a full bucket), and $m$ and $n$ are the constants to be calibrated through experimentation.

Table 2. Weight model parameters used for the full-scale validation.

Parameter	Value
${\beta }_{weigh}$	0.2
$m$	0.0207
$n$	1.0774

shows the parameters used for the particular conditions of the field trials presented in this work (machine and rock characteristics).

4. Experiments and Analysis

Full-scale experiments with a GHH LF11H model LHD (see picture in Figure 1) were carried out in a production tunnel of a sublevel stoping mine located in the Coquimbo Region (Chile), facilitated by the Chilean mining company, CMSG. Pictures of the mine site and the LHD inside its tunnel are shown in Figure 20. Figure 20a shows the mechanical workshop just outside the mine’s main entrance, and Figure 20b shows the LHD during surface tests. Figure 20c shows the LHD outside the test tunnel with an almost fully loaded bucket.

Figure 21 shows the layout of the tunnel where experiments were carried out. The LHD traveled from the “start–end point” shown in the figure to the end of the red line, where the stope was originally located. A stope is a large open underground space that is produced by the sublevel stoping mining method. The LHDs load ore inside the stope from a large pile of material, also sometimes referred to as the loading point or draw point. Once the LHD has a fully loaded bucket, it has to travel back to the “start–end point” to dump the ore.

The experiments reported here were carried out in a fully functional tunnel of the mine, i.e., a tunnel used for extracting mineral. In fact, the experiments took place during day hours, while the mine’s night shift used the LHD in manual mode for production. As a result of this, the shape and location of the stope and the draw point changed every day (as the ore runs out, the inner wall of the stope is blasted as part of the mining process).

Because of this, and since in situ experimentation was time-limited, experiments were separated in two parts. First, the field trial experiments regarding charging and collision detection of the LHD and the pile, and the LHD’s excavation and payload weighing, are reported in Section 4.1. Then, pile identification experiments from an offline processing of the collected field data are presented in Section 4.2.

The mining tunnel was narrower than recommended by the LHD manufacturer, so the LHD’s positioning was more complex than initially expected. This also contributed to dividing the experiments into the two groups described above. Figure 22 shows a picture of the draw point as seen from the teleoperation system cameras.

4.1. Full-Scale Rock Excavation Experiments

Since the pile identification method was not fully optimized at the time of the onsite tests, and time for executing the experiments in the mine was limited, the development team divided the loading procedure in two stages in order to maximize the total number of full-scale rock excavation experiments and evaluate each stage separately. One stage included rock pile identification and positioning, and the other included the charging, excavation, pull back, and weighting processes. To perform this, loading experiments required a human operator to position the machine five to ten meters in front of the draw point, effectively bypassing the rock pile identification and positioning stages, and starting the system’s state machine in “positioning assistance”, as described in Section 3.

A total of 30 complete loading experiments were carried out. Similar to in manual operation, multiple loading attempts were required in order to fill the bucket during each experiment. Manual operation of LHD in sublevel stoping mines usually needs two or three attempts to achieve a proper bucket fill factor, since the procedure is performed using line-of-sight radio control with limited visibility behind the LHD (safety regulations forbid operators to enter the stope area). In some attempts, the tip of the bucket was lowered too much, causing high resistive forces from the ground during charging, and thus triggering the collision detection too early. A total of 81 loading attempts were performed in order to fulfill the 30 complete loads. Of these, 7 failed (early collision trigger) and 74 were successful, resulting in the 91% success rate of the charging and collision detection method.

For each experiment, a fixed value of 0.5 was used for the pedal command. An operator monitoring the system’s behavior during the complete loading process from a remote-control station decided when the bucket was full enough, and hence if a new attempt was needed. The operator relied only on the visual input of the machine’s forward camera to make this decision.

Figure 23 shows an example of the relevant variables in an autonomous loading attempt, which has some resemblance to the operator-controlled loading shown in Figure 5.

Table 3. Loading experiments and average fill factor per number of loading attempts. N-Attempts: number of attempts required to fill bucket. N-Exp: number of experiments. Total-Attempts: total number of performed attempts. %-Experiments: % of the experiments. Fill-Factor: average fill factor.

N-Attempts	N-Exp	Total-Attempts	%-Experiments	Fill-Factor
1 attempt	9	9	30%	88%
2 attempts	4	8	13%	107%
3 attempts	11	33	37%	87%
4 attempts	6	24	20%	90%
TOTAL	30	74	100%	-

shows the number of experiments carried out, classified by the number of attempts to achieve a full bucket, and the average fill factor for each category. It can be seen that most experiments required three loading attempts, and that the operator continued to reattempt loading until the fill factor was about 90%. This can be seen more easily on

Table 4. Total average number of attempts and fill factors.

Total Average Number of Attempts	Total Average Fill Factor
3.6	90%

, where the average fill factor and average number of attempts for all experiments are shown. It must be noted that while the fill factor at the end of each experiment met the criteria for manual operation (since it was the same operator deciding when the bucket was full), the average number of attempts was found to be higher than that in manual operation, which, according to manual operators was between two and three attempts.

Table 5. Duration and fill factor in consecutive attempts. P-Attempts: progressive sequence of attempts. N-Exp: number of experiments. Duration: average duration. Fill-Factor: average fill factor.

P-Attempts	N-Exp	Percentage	Duration	Fill-Factor
First attempt	30	100%	12.7 s	62%
Second attempt	21	70%	10.2 s	77%
Third attempt	17	57%	10.8 s	82%
Fourth attempt	6	20%	10.7 s	90%

and Figure 24 and Figure 25 show more detail about progressive filling of the bucket through loading attempts for the 30 loading experiments. Table 5 shows the average durations of the excavation step and the resulting fill factor, classified according to the progressive sequence of attempts, as well as the total number of experiments in each case. On average, the first attempt managed to fill 62% of the bucket capacity; then, the second attempt achieved 77%; the third attempt achieved 82%; and for the six experiments that needed a fourth attempt, the average that resulted was a 90% fill factor. The average excavation duration was about 10–12 s, which comprised the time between the collision of the LHD with the rock pile and the end of the excavation algorithm. The required time for positioning, charging, and weighing was not taken into account in this measurement. It can be seen that the average duration of the excavation in the first attempt is slightly longer than the others.

Figure 24 shows a box plot of the excavation step duration for all loading attempts. The time taken for the excavation ranges from 6 to almost 20 s, depending on the conditions of the rock pile and the specific interaction between the bucket and the rock pile. Figure 25 shows a box plot of the fill factor that was obtained progressively through loading attempts. It can be seen that the biggest jump in performance happened in the second loading attempt. Some outliers with performance above 100% are also depicted for the second and third attempts and appear as a consequence of loading a large boulder.

During experimentation, the value of accurately placing the bucket of the machine against the ground, prior to the charging against the rock pile, was recognized. A bucket tilted too far down would cause an early trigger of a collision detection, while a bucket tilted just slightly upwards has a relevant impact on how far the machine can penetrate into the rock pile. An even more relevant factor that affects the performance of the loading maneuver is the status of the rock pile. Newly blasted ore is far easier to load than draw points that have had more time to settle and compress due to the weight of the rock. Draw points with large amounts of ore are also easier to load than those that have little material left. Despite these relevant factors, the proposed method was able to extract full buckets each time, at the expense of the operator having to perform more than the usual number of reattempts required under manual operation.

It is important to note that in a real mining operation, the efficiency of an LHD is measured in the amount of ore that it is able to haul from the extraction point to the dumping point (usually in tons/hour). Therefore, the percentage of bucket filling is not the only relevant factor when evaluating the system, but also the amount of time it uses to load the bucket. This is especially important when deciding if the system should make another attempt at the excavation procedure to achieve a fuller bucket. This criterion varies among different operations, as the dumping and haulage time is different.

4.2. Offline Results Using Field Data: 2.5D Modeling of the Extraction Point

Personnel cannot be inside a stope as it is forbidden for security reasons (i.e., rock can fall from the ceiling of the stope), so it was not possible to obtain accurate measurements of the ore pile characteristics. However, an accurate modeling of the ore pile is not really necessary, as it only needs to be detected, as well as having its width and inclination roughly estimated in order for the excavation algorithm to work.

To obtain the datasets required to characterize our 2.5D modeling algorithm, the LHD was driven from a fixed distance of about 50 m from the loading point to the entrance of the stope, while all sensor and machine data was recorded. These datasets were captured at different times so the drawing point was not located in the same place, nor did the ore pile have the same shape. First, in each case the point cloud was computed and then the pile’s width and inclination estimated. Figure 26 shows an example of the point cloud obtained while the LHD is approaching an extraction point, after registering the position of the points using the LIDAR-based odometry and the inclinometer data (see details in Section 3.2). The color of the points represents the distance to the LHD (in the x-axis) at the time of registration. In the case of the rock pile at the end of the tunnel, red indicates that the machine was further away when that reading was taken.

The results of several modeling attempts where the pile’s width and inclination were estimated are summarized in

Table 6. Results of modeling extraction points.

Modeling Attempt	Ground Truth Width (m)	Predicted Width (m)	Ground Truth Inclination (°)	Predicted Inclination (°)
1	3.25	2.89	−53.0	−52.6
2	3.10	5.27	−50.1	−48.7
3	3.10	3.16	−52.4	−50.5
4	3.44	3.29	−50.6	−50.5
5	3.20	3.72	−50.0	−53.4
6	3.03	3.25	−49.5	−46.5

. They consider ground truth width, predicted width, ground truth inclination, and predicted inclination. Both ground truth width and ground truth inclination were determined manually from the integrated point clouds. The average absolute error in the predicted width is 0.58 m, while the average absolute error in the inclination is 1.7 degrees. The precision achieved by the characterization procedure is sufficient for the task of autonomous loading.

5. Conclusions

A complete autonomous loading system for LHD machines for underground mining was presented. The loading system considers identification of the rock pile, positioning of the machine in front of it, charging against the rock pile, excavating, moving away from the draw point, and estimating the bucket weight or fill factor. Despite that the proposed system was fully implemented (including the necessity for a human operator to be involved in the process in order to complete the task when the system fails), it could not be tested as a whole. The proposed system which implements the whole loading process, the excavation algorithm, and the tests in a real production environment are the most important contributions of this work.

The experimentation phase was divided between onsite execution and validation of the excavation algorithm, and offline data processing for the rock pile identification method. In the onsite experiments, identification and positioning were bypassed in favor of teleoperation by a human operator. It was also the operator’s choice whether or not to perform multiple loading attempts in order to fill the bucket. A total number of 30 excavation experiments were carried out, most of them requiring multiple attempts to achieve a full bucket. An average of 3.6 attempts per experiment was needed in order to obtain a bucket fill factor of 90%. By comparison, manual operation usually needs two or three attempts to achieve this bucket fill factor. This performance may seem inferior to previous published work: however, there are other factors involved that prevent direct comparison of the results. Equipment, type of ore, and mining method should be considered. Thus, comparison to the human operators of the specific mining site should be preferred. It is important to mention that the main driver for automating the loading process is to increase the safety of workers rather than obtain higher efficiency. In this regard, the proposed excavation algorithm fulfills our expectations.

Since these experiments are time-consuming in an industry where time is an expensive resource, only enough experiments needed to validate the excavation algorithm were able to be executed. Part of our future work will be to carry out new experiments in an underground mine in order to validate the complete autonomous loading system, and to measure its performance more accurately. Despite this, the problem remains relevant and mining companies are looking forward to integrating autonomous loading to LHDs in their operations, as it would enable them to close the loop for a fully autonomous production cycle.

A video showing the operator’s graphic interface while the system is autonomously performing an excavation procedure can be found in https://youtu.be/Oa11kTBJf2Y (accessed on 7 September 2021).

The system is now being installed and tested in a room and pillar mine in Germany, where it will be tested as a whole system.