Application of Feature Point Matching Technology to Identify Images of Free-Swimming Tuna Schools in a Purse Seine Fishery

Abstract

Tuna fish school detection provides information on the fishing decisions of purse seine fleets. Here, we present a recognition system that included fish shoal image acquisition, point extraction, point matching, and data storage. Points are a crucial characteristic for images of free-swimming tuna schools, and point algorithm analysis and point matching were studied for their applications in fish shoal recognition. The feature points were obtained by using one of the best point algorithms (scale invariant feature transform, speeded up robust features, oriented fast and rotated brief). The k-nearest neighbors (KNN) algorithm uses ‘feature similarity’ to predict the values of new points, which means that new data points will be assigned a value based on how closely they match the points that exist in the database. Finally, we tested the model, and the experimental results show that the proposed method can accurately and effectively recognize tuna free-swimming schools.

1. Introduction

Purse seine fishing is a sophisticated fishing method that involves a series of advanced technologies to aid the process of fish detection, attraction, and capture. In general, fish schools targeted by purse seining can be divided into unassociated schools (e.g., free-swimming schools) and log- or fish aggregating device (FAD)-associated schools. Although catches for FAD-associated fish schools account for approximately 78% of the total landings in the Chinese tuna seine fishery [1,2], due to the negative effects that FAD exerts on pelagic ecosystems [3,4,5,6,7,8,9,10], this fishing method has been limited by tuna Regional Fisheries Management Organizations to conserve and manage tuna resources. For example, in 2008, the Western and Central Pacific Fisheries Commission proposed the implementation of a seasonal ban on tuna fishing with driftwood (or FADs) in the exclusive economic zones and high seas of the members of the Nauru Agreement. Since 2009, the fishing ban has been gradually extended from two months to four months, and the fishery ban provisions of the conservation and management measures for fishing member states in 2019 included a three-month closure of FADs in exclusive economic waters (July to September, including releasing FADs and casting nets) and two consecutive months (April to May and November to December) of high-seas FAD closures. Resolution 18/08 of the Indian Ocean Tuna Commission stipulates that the number of FAD satellite sonar buoys activated by each operating vessel at sea at any time should not exceed 350. Resolution c-16-01 of the Inter-American Tropical Tuna Commission stipulates that the maximum number of FADs released by large tuna purse seine vessels (with a total hold capacity of 1200 m³ or more) at any time is 450.

At present, many purse seine ships in Japan, China Taibei, the United States, and other countries or regions rent or equip fishing helicopters for fish detection. However, leasing helicopters is associated with high costs and safety risks for workers. Currently, unmanned aerial vehicle (UAV) technology is gradually replacing the use of aircrafts to obtain remote sensing information; as time progresses, UAVs will become more and more commonplace, and their potential uses will be adapted to fishery. For example, public and private actors involved in fisheries management used camera-equipped UAVs for marine surveillance [11,12,13]; because of decreasing costs, increasing flight times, and an improving capacity for an easy launch and retrieval at sea, UAV is rapidly expanding its utility for surveys of marine life [14,15,16]; the development of UAVs has brought technological innovation to the recreational fishing sector, and recreational anglers have also begun using UAVs to physically assist in the capture of fish [17]; a drone coupled with automated image processing has the potential to be used more widely in ecological monitoring [18,19,20].

At present, UAVs have been already studied in areas such as marine recreation fishing, as well as monitoring illegal fishing and marine fauna, but used UAVs to detect free-swimming tuna schools in tuna purse seine fisheries need further discussion. Consequently, it is possible to use UAVs to detect tuna free-swimming shoals and automatic recognition goals. The application of UAVs covering areas beyond the visual line of sight and image identifying technology in tuna purse seine fisheries will reduce costs, save energy consumption, and ameliorate stock monitoring for assessments.

2. Materials and Methods

2.1. Materials

From 26 October 2018 to 9 December 2018, a researcher of Shanghai Ocean University boarded a tuna purse seine fishing vessel, “Xiefeng 789”, operated in the central and western Pacific Ocean, and used a SONY FDR-AX60 4K HDR camera to shoot fish shoals from the lookout platform of the main mast of the seine vessel and the helicopter launched from the vessel. The researcher obtained a total of 424 GB audio-visual data, and all video data (1920 × 1080) were 25 frames per second (fps). The test data of free-swimming tuna schools that was clear and stable was selected from the original audio-visual data.

2.2. Methods

Based on feature point matching technology, we based the construction of the tuna free-swimming school and free-swimming school image recognition models on the technical process of “image acquisition-image processing-image feature library construction-fish shoal identification” to identify images.

This paper summarizes and identifies the most suitable method from three robust feature detection methods: Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Oriented FAST and Rotated Brief (ORB) [21,22,23], from the computing speed and robustness performance indicators such as robustness, rotation, blur, and illumination changes. This study applied KNN (K-Nearest Neighbor) and RANSAC (Random Sample Consensus) to the three detection methods to analyze the results of the methods’ applications in feature recognition [24]. In the experiment, we used repeatability measurements [25] and the number of correct matches for the evaluation measurements. The repeatability measurement is computed as a ratio between the number of point-to-point correspondences that can be established for detected points and the mean number of points detected in two images [25]:

$$r_{1,2}=\frac{C\left(I_1,I_2\right)}{mean\left(m_1,m_2\right)}$$

(1)

where $C\left(I_1,I_2\right)$ denotes the number of corresponding couples, and m₁ and m₂ means the numbers of the detector. This measurement represents the performance of finding matches.

The special image processing method is used to process the grays in an image and reduce noise before obtaining feature points [26]. In this study, Mongodb was used as the tuna shoal feature database to store the feature point descriptors marked by a feature algorithm. The image feature points were extracted, and feature matching was performed using the KNN algorithm, which predicts the values of new points based on how closely they match the points in the database [27]. This method allows for the intelligent recognition of fish shoals, as shown in Figure 1.

The methods used in this research were all based on Opencv [26]. All analyses were performed on an Intel^(R) core^(TM) i7-7700 CPU with 8.0 GB RAM, with Windows 10 as the operating system. We used the image dataset photographed by our own researchers (Figure 2). This included image preprocessing, a comparison of feature point algorithms, the construction of the Mongodb feature library, and a recognition experiment. The procedures 1–6 are described below:

(1)

Image preprocessing. To improve the computing speed and reduce data storage, we used the perceptually weighted formula [26]:

$$\mathrm{Y}=\left(0.299\right)\mathrm{R}+\left(0.587\right)\mathrm{G}+\left(0.114\right)\mathrm{B}$$

(2)

Figure 2. Part of a test image. (A) rotation image; (B) blurred images; (C) illumination-changed image; (D) tested computing speed.

Figure 2. Part of a test image. (A) rotation image; (B) blurred images; (C) illumination-changed image; (D) tested computing speed.

Among them, Y is the grey value, and R, G, and B are pixel values of the three-channel RGB image (Figure 3).

Noise reduction. Image noise is a random variation of brightness or color information in images and an undesirable by-product of images that obscures the desired information. Therefore, we chose Non-Local Means to reserve the details of each image while removing the random noise from the image as much as possible [28] (Figure 4).

(2)

Comparison of Rotational Variation Robustness. Image rotation is essential in feature detection application. When the image is rotated, the corresponding angle of each pixel, the gradient value and direction information of the pixels around the original feature point, and the main direction of the feature point will change. The method was applied as follows: image tuna1 as the test image was selected from the tuna image set (Figure 2A) and rotated 0, 45, 90, 135, 180, 225, 270, and 315 degrees clockwise to get T-Rotate1 to T-Rotate8. The RANSAC algorithm was used to optimize the rotated tuna1 image, and the feature points of the rotated tuna1 image were matched with the original image.

(3)

Fuzzy Transformation Robustness Comparison. In the actual application environment, the fuzzy degree of image acquisition changes with the external environment and the working state of the remote sensing equipment. After a fuzzy transformation, the resolution of an image decreases. With a decrease in resolution, the performance of the image feature point recognition also decreases. In contrast, with an increase in resolution, the recognition performance of the feature point algorithm gradually improves. We tested the robustness of the fuzzy transformation as follows: the t-Gaussian blur image in the free-swimming image set (Figure 2B), through altering differently sized ksizes (1,1), (3,3), (5,5), (7,7) and (9,9) to obtain T-Bulr1, T-Bulr2, T-Bulr3, T-Bulr4, and T-Bulr5. The processed image was matched with the original image. The matching feature points were optimized using RANSAC and the number of feature points that could be correctly matched to the analysis. Finally, the change range of the characteristic points was observed and recorded.

(4)

Brightness Transformation Robustness Comparison. In the actual identification, the illumination intensity will differ according to different operation times. Under different illumination conditions, there are large intra-class divergences between images, and some key feature points become more prominent or weakened under the influence of illumination. This makes the feature points differ in gray scale spaces, which is not an ideal condition for feature point analysis. We tested the robustness of the brightness transformation as follows: we randomly selected an image, T-Light (Figure 2C), and adjusted the bias parameter to 0, 25, 50, 75, 100 and 125 to obtain T-Light1 to T-Light6, the processed image was matched with the original image, and the matched feature points were optimized using RANSAC. Then, we compared the number of feature points that could be correctly matched and observed the change range of the feature points.

(5)

Construction of Mongodb Feature Library. Mongodb is a database that supports a variety of data structures and complex data types [29]. In this study, Mongodb was used as the feature database of a free-swimming shoal to store the feature point descriptors marked by the ORB feature algorithm. When constructing the tuna free-swimming school feature database, it is necessary to recruit technicians engaged in tuna purse seine fishing to identify and annotate tuna free-swimming school videos. Figure 5 shows an example of the tuna shoal feature points labeled by the ORB algorithm. The red frame represents the marked area in which the feature points were identified and stored. For this video acquisition, more than 200,000 feature points were obtained.

The rough matching method of the brute-force matcher was used to match the feature points of the test set to the sample set in Mongodb. Euclidean distances were then calculated, and KNN values were obtained using the KNN calculation, as follows:

Supposing the database sample x belongs to n-dimensional space ${\mathrm{R}}^{\mathrm{n}}$, set the i^th sample $X_i=x_1^i,x_2^i\dots x_n^i\in {\mathrm{R}}^n$, where the n^th characteristic represents the point from ORB that attributes value to the i^th sample. Then, the Euclidean distance between the two samples was defined as:

$$\mathrm{d}(X_i,X_j)=\sqrt{{\displaystyle \sum }_{\mathrm{n}=1}^n (x_n^i-x_n^j{)}^2}$$

(3)

(6)

Recognition experiment. For the ratio parameter of the descriptor distance, n = 0.7 and n = 0.6 were selected for the recognition experiment. The recognition experiment was conducted as follows: using the k-fold cross-validation method, the test dataset was selected from the tuna database and divided into 10 mutually exclusive subsets (10 small videos, each containing 50 frames). Then, 10 training sets were obtained and tested 10 times. For the correct identification of the fish school (or to not recognize the non-fish school), the identification system was compared and analyzed.

3. Results

3.1. Overview of the Three Feature Algorithms

3.1.1. Matching Speed Comparison

The time evaluation only indicated the tendency of the time cost of the three methods. We chose the images (Figure 2D) that displayed the same size and quality in order to maintain consistency in the experiment. Time was counted for the complete processing, which included feature detection and matching.

From the perspective of the algorithm execution speed, the SIFT algorithm took an average of 10.832 s to complete, which was the most time-consuming compared to that of the ORB and SURF algorithms (

Table 1. Execution speed of the same image match by the Oriented FAST and Rotation Brief (ORB), Speeded Up Robust Features (SURF), and Scale Invariant Feature Transform (SIFT) image detection algorithms.

Algorithm	SIFT	SURF	ORB
time consumed in the first test	10.729 s	6.063 s	0.193 s
time consumed in the second test	10.859 s	6.012 s	0.200 s
time consumed in the third test	11.081 s	6.010 s	0.185 s
time consumed in the fourth test	10.703 s	5.979 s	0.188 s
time consumed in the fifth test	10.790 s	5.965 s	0.190 s
mean	10.832 s	6.001 s	0.191 s
number of matches	24,256	22,379	500

). The average execution time of the SURF algorithm was 6.001 s. However, the fastest algorithm was the ORB algorithm, which took 0.191 s to complete on average. The ORB algorithm was two orders of magnitude faster than the SIFT algorithm and one order of magnitude faster than the SURF algorithm. The SIFT algorithm displayed the most matching logarithm, which was 24,256, followed by that of the SURF algorithm (22,379), and then ORB (500).

3.1.2. Comparison of Rotational Variation Robustness

Here are the results of SIFT, SURF, and ORB on the number of matching feature points after the image rotation of T-Rotate1, T-Rotate2, …… T-Rotate7, and T-Rotate8, respectively representing the images of tuna1 after eight different rotation angles. Although the logarithm of the feature point matching of the three algorithms showed a downward trend after eight rotations at different angles, the SIFT and ORB algorithms tended to stabilize more quickly, while the SURF algorithm remained in an unstable state; meanwhile, SIFT had the largest repeatability at 53%, and SURF showed the same repeatability performance as ORB (

Table 2. Rotation changes comparison. The data represents the repeatability and the average of the repeatability.

Data	SIFT	SURF	ORB
T-Rotate1	100%	100%	100%
T-Rotate2	53%	8%	13%
T-Rotate3	50%	25%	13%
T-Rotate4	60%	9%	18%
T-Rotate5	51%	28%	11%
T-Rotate6	53%	8%	12%
T-Rotate7	50%	27%	12%
T-Rotate8	54%	9%	12%
average	53%	16%	13%

, Figure 6).

3.1.3. Fuzzy Transformation Robustness Comparison

Here are the results of SIFT, SURF, and ORB on the number of matching feature points after the image fuzzy transformation of T-Blur1, T- Blur 2, T- Blur 4, and T- Blur 5, which respectively represent the images of tuna1 after five different fuzzy degrees. After the image was Gaussian-blurred by altering the size of the Gaussian kernel, the logarithm of the feature point matching of each algorithm decreased significantly as the scale of the blur became larger. The SIFT algorithm displayed the worst repeatability at 29%, while SURF appeared stable in the later stages, and ORB performed best and had the largest repeatability at 48% (

Table 3. Blur changes comparison. The data represents the repeatability and the average of the repeatability.

Data	SIFT	SURF	ORB
T-Blur1	100%	100%	100%
T-Blur2	53%	71%	72%
T-Blur3	32%	51%	56%
T-Blur4	17%	26%	37%
T-Blur5	12%	16%	27%
average	29%	41%	48%

, Figure 7).

3.1.4. Brightness Transformation Robustness Comparison

Here are the results of SIFT, SURF, and ORB on the number of matching feature points after the image brightness transformation of T-Light1, T- Light 2, T- Light 5, and T- Light 6, which respectively represent the images of tuna1 after six different brightness degrees. As the scale of the image brightness increases, the logarithm of the feature point matching of each algorithm decreases significantly. SIFT and SURF showed a good stability as the brightness changed slightly, and their repeatability was 48% and 43%. ORB displayed a poor stability when the illumination changed significantly and had the smallest repeatability at 27% (

Table 4. Brightness changes comparison. The data represents the repeatability and the average of the repeatability.

Data	SIFT	SURF	ORB
T-Light1	100%	100%	100%
T-Light2	96%	96%	63%
T-Light3	78%	72%	39%
T-Light4	46%	34%	21%
T-Light5	15%	9%	9%
T-Light6	3%	2%	2%
average	48%	43%	27%

, Figure 8).

3.2. Comparison of the Results

There was no best method for all deformations. Hence, when choosing a feature detection method, it is best to determine the performance of the specific methods used, as the results of this experiment were not consistent for all cases (

Table 5. Comparison of the experimental results.

Method	Time	Rotation	Blur	Illumination
SIFT	weak	best	weak	best
SURF	good	good	good	good
ORB	best	weak	best	weak

).

ORB showed stability and fast speed in the experiments. It is known as the “Fast” detector, as ORB is faster than SIFT and SURF. The computing time for SIFT could be improved; however, it still performs well in most situations. SIFT appears best under illumination and rotation; however, under large extents of blur, the performance could be further improved. Taking these factors into account and, in particular, the need for identification in real time, we chose ORB to computer the feature point descriptors stored in the feature library and to computer the new point.

3.3. Test Results

In the recognition system, different ratio parameters make different recognition rates. During the test of the tuna shoal image recognition, we figured out that the recognition success rate of the tuna school recognition model was approximately 70% (

Table 6. Recognition results with the ratio parameter, n = 0.6.

	Content	Recognition Occurrences	Artificial Judgment	Result
Videos
T-video1		5	Fish shoal	correct
T-video2		8	Fish shoal	correct
T-video3		4	Fish shoal	correct
T-video4		0	Fish shoal	incorrect
T-video5		0	Fish shoal	incorrect
T-video6		0	Fish shoal	incorrect
T-video7		0	Non-Fish shoal	correct
T-video8		1	Fish shoal	correct
T-video9		0	Non-Fish shoal	correct
T-video10		0	Non-Fish shoal	correct

and

Table 7. Recognition results with the ratio parameter, n = 0.7.

	Content	Recognition Occurrences	Artificial Judgment	Result
Videos
T-video1		5	Fish shoal	correct
T-video2		17	Fish shoal	correct
T-video3		6	Fish shoal	correct
T-video4		0	Fish shoal	incorrect
T-video5		0	Fish shoal	incorrect
T-video6		0	Fish shoal	incorrect
T-video7		0	Non-Fish shoal	correct
T-video8		1	Fish shoal	correct
T-video9		0	Non-Fish shoal	correct
T-video10		0	Non-Fish shoal	correct

). The identified feature points are framed (Figure 9). When n = 0.6, the system could recognize fewer images but did so with a higher recognition accuracy. Meanwhile, for n = 0.7, the system could recognize more feature images, but the recognition accuracy was reduced.

4. Conclusions

A technical process of "acquisition-processing-recognition" of tuna school images based on videos taken from the main mast and a helicopter can be applied on the UAV aerial photography. The framework mainly included image preprocessing, image feature extraction, database construction, and image matching recognition, which preliminarily implemented the computer intelligent recognition of tuna school images.

The advantages of the ORB algorithm in tuna fish school image recognition applications are apparent. A comparative analysis of three algorithms, ORB, SURF, and SIFT, showed that the ORB algorithm was the fastest and could meet real-time calculation requirements. The stability of the SURF algorithm fluctuated in terms of the rotation robustness. Both ORB and SURF performed relatively well in terms of the fuzzy transformation robustness. The performance of ORB was relatively poor in terms of the robustness brightness transformation, and both SIFT and ORB performed better in terms of the identified proportion of feature points in the feature area. As mentioned above, ORB is suitable for the identifying models.

A tuna fish school image recognition model was constructed based on the brute-force matching method and the KNN algorithm. Computer simulation experiments showed that the tuna fish school recognition model had a recognition success rate of approximately 70%. The recognition effect was optimal when n = 0.6. The model for identifying images of free-swimming tuna schools is a crucial part of fish detection based on the UAV system and is equivalent to a scout on a helicopter. Thus, as the next step we will take UAV with the recognition system to the tuna purse seiner.