Fragmentation Size Distribution Measurement by GNSS-Aided Photogrammetry at Real Mine Site

Abstract

In mining operations that employ explosives and mineral processing, one of the important factors for efficient and low-cost operation is the fragmentation size distribution of rock after it has been blasted. Automatic scaling is a critical component of fragmentation size distribution measurement as it will directly determine the accuracy of the size estimation. In this study, we propose a method to create a system for creating a scaled 3D CG model, without the use of ground truth data such as GCPs (Ground Control Points), for the purpose of improving fragmentation size distribution measurement using positional data such as GNSS (Global Navigation Satellite System)-aided photogrammetry. We confirmed the validation of the method through an experimental evaluation of actual muckpiles. The results showed evidence of improving the scaling aspect of 3D fragmentation measurement systems without using GCPs or manual scales, specifically in surface mines where GNSS data are available.

1. Introduction

A mining operation contains several factors that affect its efficient extraction of resources. One such factor is fragmentation size, a key parameter across almost all of the stages of production from mine (drill, blasting, haulage, etc.) to mill (mineral processing). Several studies have explored the effect of fragmentation size on other factors such as drilling and blasting cost [1] and the performance of crushing and grinding circuits [2,3]. Therefore, it can be considered that it is vital for companies to continuously monitor fragmentation size and make necessary changes to mine planning and execution to keep the fragmentation size that is most beneficial to the operation as a whole. Traditionally, methods such as manual sieving, boulder counting, and visual estimation have been used for fragmentation size measurement. However, due to the generally large amount of material being mined and its innately heterogeneous nature, difficulties arise from using traditional methods. In addition, limitations on sampling as well as bias make these methods relatively inefficient [4]. As such, there exists a need for a quick and accessible method of rock fragmentation size distribution determination that can surmount the limitations of physical sampling and laboratory analysis. A currently used digital solution to this problem is to employ image-based particle size analysis software. Commercial products such as WipFrag [5,6] and Split Desktop [7,8] use images of muckpiles or orthomosaics to measure fragmentation size distribution, where the conventional studies use 2D information and the 3D size estimation accuracy is relatively low.

A 3-Dimensional Fragmentation Measurement (3DFM) system was developed that uses 3D photogrammetry to measure fragmentation size distribution at accuracies greater than that of conventional methods [9]. The 3DFM extracts the fragmentation sizes from a 3D muckpile model, which facilitates accurate fragmentation size distribution measurement without scale objects and eliminates the need for excessive manual editing. Furthermore, a representative fragmentation of an entire blasting shot can easily be analyzed by generating a 3D model of an entire blasted muckpile. A workflow for this system when applied to a mining operation, and scope of this work, is shown in Figure 1.

The developed system is divided into stages, utilizing multiple computational techniques in order to achieve its purpose. In a hypothetical application of the system, pictures of the muckpiles from the products of blasting are taken. The sizes of muckpiles vary considerably depending on the specifications of the hauling equipment as well as the mine plan that the operation employs. In situations where the muckpiles are too large or have parts inaccessible to photo-taking, the system can reconstruct only a representative slice of the muckpiles. The images are then processed in a high-power computer by a sequence of 3D imaging techniques that will ultimately output a scaled 3D model of the muckpiles in the form of a point cloud. The dimensional data can be used to compute the fragment size distribution of the muckpiles. Using this information, the blasting product can be judged if it is up to the expected specification. Adjustments are then made to the blasting design, such as the amount and type of explosive and blasting patterns to achieve the required distribution.

Scaling is a critical component of fragmentation size distribution measurement using photogrammetry as it will directly determine the accuracy of the size estimation. In creating a 3D model, extrinsic data such as ground truths are needed to create a properly-scaled reconstruction of the scene. Several methods are used to resolve scale in photogrammetry. Most of these methods have the same basic idea in that once the exact distance between at least two different points in a scene as a ground truth is known, a scale factor can be applied to the 3D model. One way to do this is to include an object of known length, such as scale bars, in the scene. In larger applications such as aerial mapping, GCPs (Ground Control Points) are used, marked points of known absolute or relative coordinates.

A previous work by authors [10] proposed a method using GNSS (Global Navigation Satellite System, such as GPS) to get absolute (but rough) camera positions for scaling muckpile fragmentation. The authors showed the potential of the fragmentation size distribution estimation method using rocks similar to muckpiles in the previous work, but the verification test with actual muckpiles and fragmentation size distribution estimation remained an issue.

This paper validates the method’s effectiveness through practical experiments using actual muckpiles generated in a quarry site, and show that the method can be applied to fragmentation size distribution measurement.

2. Muckpile Scaling by GNSS-Aided Photogrammetry

2.1. Structure-from-Motion-Multi-View-Stereo (SfM-MVS)

Structure from Motion (SfM) has been collectively defined as the photogrammetric technique [11], the process [12], as well as the tools [13] used to generate 3D models from 2D images taken at different angles. It has been developed since the 1980s, resulting in various applications such as photogrammetric surveys, virtual reality model reconstruction, determination of camera motion [14], and odometrical scale estimation [15]. Compared to traditional photogrammetry, where the calculation is more direct, SfM uses repeating algorithms to identify matching features in a set of overlapping images, and use these matched features to calculate camera location and orientation [13]. SfM can be computed in several ways, depending on numerous factors such as camera type, image ordering, capture format, and more.

Mathematically, SfM can be described as the conversion of four coordinate systems, illustrated in Figure 2:

i.

Image pixel coordinate system, which concerns the pixels on the 2D image.

ii.

Imaging plane coordinate system, which lies on the same plane of the previous system, but whose origin is the plane’s intersection with the camera’s optical axis.

iii.

Camera coordinate system, which concerns a pinhole camera’s point of view of the image.

iv.

World coordinate system, which is a reference system to describe the position of the camera and the objects being taken pictures of.

The conversion of these four coordinates systems can be described by Equation (1). The $u$ and $v$ describe the axes in the imaging planes. The $u_0$ and $v_0$ are the coordinates of the origins of the imaging plane in the pixel coordinate system. The ${\delta }_x$ and ${\delta }_y$ represent the physical size of each pixel in the image in the imaging plane (zoom ratio). The $f$ describe the focal length, which is the distance from the optical center of the camera to the pixel plane. The $\mathit{R}\in {ℝ}^{3\times 3}$ and $\mathit{t}\in {ℝ}^3$ describes the rotational and translational vectors that relate the camera and the world coordinate systems. The $X_w$, $Y_w$, and $Z_w$ are the actual coordinates of a point in the world coordinate system.

$$Z_c\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{{\delta }_x}& 0& u_o\\ 0& \frac{1}{{\delta }_y}& v_o\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}f& 0& 0\\ 0& f& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cc}\mathit{R}& \mathit{t}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cc}\mathit{R}& \mathit{t}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$$

(1)

Equation (1) represents the fact that in order to estimate the position of a point in the real world, the external parameter matrix of the camera (i.e., $\mathit{R}$ and $\mathit{t}$) needs to be measured first. Once $\mathit{R}$ is known, the relative position of the object in the world coordinate system can be estimated, and once t is known, the absolute position can also be acquired. Basic SfM can relatively estimate $\mathit{R}$ and $\mathit{t}$.

It can be inferred that there is no single correct workflow or process in the conversion of 2D images into models. However, key processes are present in almost all applications of the method, as shown in Figure 3. Firstly, an image dataset must be taken by cameras with a GNSS receiver. As many images as possible from various viewpoints should be taken because the number of images and variation of viewpoints effect the accuracy of the result of 3D reconstruction. SfM process is composed of keypoint detection, keypoint matching, matched filtering (filtering valid corresponding points), and sparse reconstruction (bundle adjustment for 3D model with point cloud). In the bundle adjustment step, a scaled sparce 3D model is generated using the georeferenced data in the input. By combining texture information and the scaled sparse 3D model through SfM and MVS processes, the scaled and dense 3D model can be obtained.

2.2. GNSS-Aided Scaling

The study proposes a method that makes use of GNSS (Global Navigation Satellite System) data to create scaled 3D models without the need for post-reconstruction rescaling. GNSS positional data and its sub-systems such as GPS, Beidou [16], GLONASS (GLObal NAvigation Satellite System) [17], and QZSS (Quasi-Zenith Satellite System) [18] can be utilized. A previous study was performed with regards to using GPS in reconstruction, but mostly in the context of UAV (Unmanned Aerial Vehicle) Mapping [19]. This study aimed to develop a system without needing ground truth data such as GCPs to create a properly scaled 3D model of muckpiles. This would aid greatly in the fragmentation size distribution measurement of muckpiles using photogrammetry.

It is a known fact that an inherent error exists within GNSS and its subsets, and even high-end geodetic GNSS receivers have errors in the centimeter range [20]. For this study, a smartphone is used as a GNSS receiver for the digital camera. This decision is due to the end goal of this research which is to be able use both image data and GNSS data from a smartphone, as this practicality can be important in a mining operation environment. This comes at a drawback to the GNSS accuracy, as recreational-grade GNSS chips like those found in smartphones typically have errors in the meter range [21]. To overcome this error, the study proposes to make use of an increasing number of georeferenced images to statistically decrease the scaling error of the constructed 3D model. Figure 4 shows a general overview of the proposed system for this study. Utilizing a smartphone’s built-in GNSS receiver, GNSS data can be logged and sent to a camera. At the moment an image is taken, GNSS data can be embedded into the image’s metadata.

Bundle adjustment is an essential process of SfM. In the bundle adjustment phase, the previous and imperfect solutions regarding camera positions and 3D features of the scene are refined [22]. More specifically, bundle adjustment is a nonlinear minimization procedure that jointly optimizes camera parameters and point position by minimizing the reprojection error between the image locations of observed and predicted image points. This minimization is done using nonlinear least squares algorithms [23]. Numerous studies have been done since its inception in the 90′s regarding bundle adjustment, with most of the research going into reducing its computational burden and accelerating the problem-solving process [24]. One such proposition is the fusion of positional data and bundle adjustment, with GNSS data being used as constraints for solving reprojection errors [25]. This concept is what this research aims to use to produce accurately-scaled 3D reconstructions of muckpiles. GNSS data are used to provide position and covariance estimates for the bundle adjustment process. The nominal form of these solutions is:

$$r_{GNSS}^M\left(t\right)=r_c^M\left(t\right)+R_c^M\left(t\right)r_{GNSS}^c+\left(b_{GNSS}^M+d_{GNSS}^M\left(t-t_0\right)\right),$$

(2)

where $r_{GNSS}^M\left(t\right)$ is the position of the GNSS receiver, $r_c^M\left(t\right)$ denotes the camera position, and $R_c^M\left(t\right)$ is the rotational matrix that aligns camera and mapping space axes, and $r_{GNSS}^c$ is the difference between the GNSS receiver and camera position. The $b_{GNSS}^M$ and $d_{GNSS}^M$ denote bias and drift terms and are included to account for data inconsistencies and the inherent errors that exist within GNSS. A previous study [26] applied GNSS-assisted terrestrial photogrammetry to model coastal areas without the use of GCPs. With bundle adjustment being an error minimization problem with multiple factors, weights can be assigned to them, as is the case in the study’s SfM workflow.

3. Evaluation Experiment in a Real Mine Site

For this experiment, actual muckpiles found in an active mining (quarry) site are recreated in 3D space. The aim of this experiment is to assess the performance of GNSS-aided photogrammetry in a practical application in an open mining environment.

The figures in this section are labeled “CGI” (Computer Generated Image) or “Photo” (real photo) in the lower left corner of each figure.

3.1. Experimental Site

The site chosen for this experiment was the Mikurahana Quarry Site of TOSEKI MATERIAL Inc. https://www.materiaru.jp/ (accessed on 12 June 2022), found in Hachirogata-town in Akita Prefecture, Japan, and the experiment was performed on 15 November 2021. The muckpiles were composed mainly of limestone and fragments that were generally large, with fragments as large as 2 m wide in its largest dimension. Satellite imagery is shown in Figure 5 to provide locational context for the quarry and the muckpiles in question.

3.2. Experiment with Real Muckpiles

A total of 400 images of the muckpiles were taken from varying angles facing the muckpiles using a smartphone (Xiaomi Mi 9T Pro) with a GNSS receiver on. Each image size is 4464 $\times$ 2976 pixels. Sample images are shown in Figure 6, which also feature the yellow cardboard box that was used as an absolute reference for scaling error. The box measures 15 cm $\times$ 15 cm $\times$ $60$ cm, with the 60 cm-long side used for measuring scaling error.

3.3. Results of 3D Reconstruction and Scaling

Using 3DF Zephyr (version 6.010) [27], a 3D photogrammetry software, the images are used to create 3D models at different image numbers, namely 50, 100, 200, and 400 images. The camera positions, measured by GNSS and by SfM, are shown in Figure 7. The red dots in the left figure represent photo-taking positions obtained using GNSS, and green objects in the right represent actual photo-taking positions measured by SfM. The figure on the right was created by manually superimposing an orthorectified image on a satellite image. The orthorectified image was generated by a 3D model obtained by SfM. We tried to take as many images as we could from various viewpoints, as Figure 7 shows.

The scaling error of each model with respect to the yellow box is measured and plotted in a graph to observe the relationship between scaling error and image number. The sparse and dense reconstruction of the muckpiles at 400 images is shown in Figure 8 at different angles for reference. A mesh reconstruction featuring measurement of the yellow box in 3DF zephyr is also shown in Figure 9.

As shown in

Table 1. Results of the experiment on the Mikurahana quarry muckpiles.

Number of Images	Measured (m)	Real Measurement (m)	Difference from Real Measurement (m)	Error (%)
50	0.82	0.60	0.22	36.7
100	0.76	0.60	0.16	26.7
200	0.74	0.60	0.14	23.3
400	0.67	0.60	0.07	11.7

and Figure 10, there is a trend that at an increasing number of images used in reconstruction, the difference from the real measurement decreases. This result, as with the previous experiments, lends more credence to the hypothesis that using more images for reconstruction has the tendency to lessen scale error in 3D models, with a relatively linear relationship between image number and scaling error as shown by the trendline with an R-value of 0.93. It is noted, however, that the improvement in scaling error by increasing the number of images is lessened, as the scaling error at using only 50 images is already 36.67%, which is much higher than when using 50 images in the other experiments. The study hypothesizes that this is due to the lack of vegetation in the area, which improved the GNSS accuracy. The study achieved 11.7% scaling error at 400 images, which is close to the extrapolated 10% scaling error at 400 images in the previous experiment on monuments [10]. The error is comparable or more accurate than WipFrag [5,6].

3.4. Muckpile Fragmentation Size Distribution Estimation

In the previous chapters, it was confirmed that scaling by GNSS-aided 3D photogrammetry is effective for actual muckpiles. In this chapter, based on the scaling results, we measured the muckpile size distribution of actual muckpiles. The results of manually measuring the muckpile size distribution of the generated 3D model of muckpiles are shown in Figure 11. Green bars represent the percentage of each rock size measured by the 3D model, and the gray bars represent that measured by real measurements. The blue line represents the cumulative percentage of rock size measured by the 3D model, and the black dotted line represents that measured by real measurements.

Figure 11 shows that the proposed method achieved a close result to the real measurements. Referring to the result, we concluded that it is possible to measure the muckpile fragmentation size distribution in an actual quarry.

Although it is difficult to count rocks that are hidden from view by other rocks, it is possible to count all large rocks that require heavy machines to transport. Scaling of very small rocks (a few centimeters) is also difficult due to the resolution of the camera, but this is expected to improve as camera performance improves. Many and various viewpoints of photos should be taken for improving the accuracy. It is hard to achieve by handheld cameras, but it can be easier with videos by drone. An end-to-end fragmentation size distribution estimation system using drones is one of the ultimate goals of this study.

4. Conclusions

This paper showed that the method of creating an accurately scaled 3D model by constraining camera positions through georeferenced images as input for SfM can be applied to real muckpiles and can be used to measure the fragmentation size distribution. Monitoring fragmentation size is an important procedure in optimizing mining operations that perform blasting. In recent years, a new method that involves using 3D photogrammetry to measure fragment sizes has been developed that has the potential to surpass traditional techniques. For this particular process to be accurate, a method for properly scaling 3D models with georeferenced images using GNSS was investigated. An experiment was performed on actual muckpiles in the Mikurahana quarry to test the system’s accuracy in a practical application. Three-dimensional reconstructions were created at image numbers of 50, 100, 200, and 400 limestone muckpiles, and the scaling error was measured and graphed against the image number. It also showed a linear pattern with an R-squared value of 0.93. The scaling error decreases with increasing image number, albeit at a lower ratio than has been hypothesized due to the lack of interference from vegetation and buildings.

Two observations can be drawn from the experimental result: (1) increasing the number of georeferenced images in SfM will incrementally improve the scaling error of the reconstruction. These observations can help improve scale accuracy in GNSS-aided 3D fragmentation measurement; and (2) the proposed method can be applied to actual muckpiles at real mine sites. These results show examples of improving the scaling aspect of 3D fragmentation measurement systems without the use of GCPs or manual scales, specifically in surface mines where GNSS data are generally readily available. This shows that monitoring the fragmentation distribution can potentially be performed using just a camera and GNSS-enabled devices, such as smartphones or drones.