Deep Learning Model Size Performance Evaluation for Lightning Whistler Detection on Arase Satellite Dataset

Abstract

The plasmasphere within Earth’s magnetosphere plays a crucial role in space physics, with its electron density distribution being pivotal and strongly influenced by solar activity. Very Low Frequency (VLF) waves, including whistlers, provide valuable insights into this distribution, making the study of their propagation through the plasmasphere essential for predicting space weather impacts on various technologies. In this study, we evaluate the performance of different deep learning model sizes for lightning whistler detection using the YOLO (You Only Look Once) architecture. To achieve this, we transformed the entirety of raw data from the Arase (ERG) Satellite for August 2017 into 2736 images, which were then used to train the models. Our approach involves exposing the models to spectrogram diagrams—visual representations of the frequency content of signals—derived from the Arase Satellite’s WFC (WaveForm Capture) subsystem, with a focus on analyzing whistler-mode plasma waves. We experimented with various model sizes, adjusting epochs, and conducted performance analysis using a partial set of labeled data. The testing phase confirmed the effectiveness of the models, with YOLOv5n emerging as the optimal choice due to its compact size (3.7 MB) and impressive detection speed, making it suitable for resource-constrained applications. Despite challenges such as image quality and the detection of smaller whistlers, YOLOv5n demonstrated commendable accuracy in identifying scenarios with simple shapes, thereby contributing to a deeper understanding of whistlers’ impact on Earth’s magnetosphere and fulfilling the core objectives of this study.

1. Introduction

To understand space physics, we need to clarify the features of the plasmasphere, a region within the Earth’s magnetosphere. One crucial aspect of studying the plasmasphere involves analyzing its electron density distribution, which is strongly affected by solar activity and changes day by day [1]. Very Low Frequency (VLF) waves, such as whistlers and OMEGA signals, provided valuable information for assessing electron density within the plasmasphere [2]. By analyzing the propagation of these signals through the plasmasphere, scientists could gain insights into its electron density distribution and better understand its behavior. This knowledge is essential for various applications, including predicting space weather impacts on communication systems, satellite operations, and navigation technologies.

Suarjaya et al. [3] developed a systematic approach for the automated detection of OMEGA signals in PFX (Poynting Flux Analyzer) data from a VLF wave instrument onboard the Akebono satellite [4]. Their approach involved steps such as identifying the transmission station, calculating the signal delay time, and estimating signal intensity. They showcased the reliability and effectiveness of this automated detection system, successfully confirming the detection of OMEGA signals, propagation, and connection with solar activity [5]. The OMEGA navigation system ceased transmissions in 2005. In 2016, JAXA launched a new satellite named Arase to explore the magnetosphere. One of the space weather phenomena that can still be captured today is the signal from lightning whistlers. Therefore, researchers now rely primarily on whistler waves for studying the plasmasphere.

Storey [6] provided evidence that lightning generates various electromagnetic waves. Some of these waves traverse the Earth’s magnetic field and influence the magnetosphere. The propagation path of these lightning-induced waves is affected by the inner-magnetospheric plasma, making it crucial for estimating plasma distribution. Electromagnetic waves originating from lightning interact with the ambient magnetic field of the Earth and the plasma in the magnetosphere, resulting in whistler waves. These waves, typically below 30 kHz, propagate through the plasma in the magnetosphere, resulting in an intriguing phenomenon. They have different frequencies that travel at different speeds, causing a delay where lower frequencies arrive later than higher frequencies [7]. This delay is due to the interactions between the whistler waves and the plasma within the magnetosphere.

Scientists have made significant progress in studying whistlers using advanced spacecraft data collection and computer modeling. These advancements have enhanced our comprehension of whistler waves and their impacts, resulting in significant scientific revelations about their characteristics and behaviors in the geospace environment. Bayupati et al. [8] studied lightning whistlers in two events using data from the Akebono satellite. By comparing observed and theoretical patterns and refining electron density profiles, scientists can learn more about how electrons are distributed and behave in the magnetosphere. Putri et al. [9] proposed a new method to reconstruct the global electron density using lightning whistlers observed by the Arase satellite. They demonstrated the possibility of the estimation of the electron density distribution along the whistler path. The facts that lightning strikes happen daily worldwide, and whistlers are a result of these strikes, suggest that whistlers also occur daily. This means that it is possible to reconstruct the global electron density using whistlers observed by Arase or other satellites along their paths. Note, that a tremendous amount of lightning whistlers are necessary to obtain the global electron density, and the whistlers should be automatically detected for practical use.

In Ahmad et al.’s prior research [10], a decision tree was employed for lightning whistler detection. It is expected that using deep learning will enhance the detection outcomes. This approach is essential due to deep learning’s capacity to capture intricate patterns and nuances in data, potentially leading to more accurate and robust detection results. Therefore, the implementation of automated detection holds significant importance. It is also important to acknowledge that the resources onboard the satellite are limited, thus requiring a meticulous evaluation of the minimum size of the deep learning model.

In this study, we evaluate the deep learning model size performance for lightning whistler detection using YOLO (You Only Look Once) [11]. Our proposed method trains on lightning whistlers using numerous spectrogram diagrams, which are visual representations of the frequency content of signals.We evaluate the effectiveness of the proposed method using the image dataset of the lightning whistlers observed by the Arase satellite. We also evaluate the detection accuracy of lightning whistlers for different sizes of the YOLO architecture and discuss the necessary and sufficient size of the deep learning architecture to precisely detect lightning whistlers. This preliminary research introduces a novel approach by systematically evaluating the impact of deep learning model size on the detection accuracy of lightning whistlers using the YOLO architecture. This study not only establishes the necessary and sufficient model size for precise detection but also lays the groundwork for future research aimed at quantifying whistler types, analyzing their temporal and spectral occurrence patterns, and determining key characteristics such as the dispersion parameter.

2. Lightning Whistlers Data Set

2.1. Arase

The Arase satellite, also known as the Exploration of Energization and Radiation in Geospace (ERG) satellite, is a Japanese spacecraft dedicated to studying the Earth’s inner magnetosphere and plasma dynamics with space weather phenomena [12]. Launched in 2016, it is equipped with a suite of instruments and sensors specifically designed to observe and analyze various phenomena occurring in the magnetosphere.

The Arase satellite utilizes multiple instruments and techniques, including the Plasma Wave Experiment (PWE), to study various phenomena in space. One notable phenomenon that can be observed is the lightning whistler [13]. The PWE instrument comprises two primary sensors: the Wire Probe Antenna (WPT) [14] and a Magnetic Search Coil (MSC) [15]. These sensors work together alongside other subsystems such as the Electric Field Detector (EFD) [14], Waveform Capture and Onboard Frequency Analyzer (WFC/OFA) [16], and High-Frequency Analyzer (HFA) [17], enabling the accurate measurement and analysis of plasma waves onboard the Arase spacecraft [16].

The Arase satellite’s sensors efficiently detect and record the electric and magnetic field signatures of whistlers. The captured data are then transmitted back to Earth for further analysis and interpretation. We analyze the recorded electric and magnetic field data to gain insights into the properties, characteristics, and behavior of the whistlers. By observing whistlers using the Arase satellite, we can gain valuable insights into the dynamics of the magnetosphere, the interaction between lightning discharges and the magnetosphere, and the distribution of electron density within this region. These observations contribute to our deeper understanding of the Earth’s magnetosphere and significantly advance our knowledge of space weather and its potential impacts on technological systems and infrastructure.

2.2. Lightning Whistlers Observed by Arase

The data collection process for studying whistlers using the WFC involves the measurement of two electric field components and three magnetic field components. The WFC is a waveform receiver specifically designed to capture and measure these field components up to a frequency of 20 kHz [18]. The data used for this study were Lv.2 magnetic field spectrum data [19] with time and frequency resolutions of 7.8125 ms and 32 Hz, respectively. Our focus lies on analyzing the magnetic field (three magnetic field components) spectra due to their superior signal-to-noise ratio, as noise can affect the shape of the whistlers and influence the detection results. We made the spectrograms using PySPEDAS library [20], which contains the data from the ERG science center [21].

The type of whistlers that are used in the classifications are [22]:

(a)

Nose whistler: This type of whistler is characterized by a frequency–time curve displaying both ascending and descending branches. The minimal delay occurs at the frequency corresponding to the nose.

(b)

Short whistler: This type has a simple one-way path that goes from a higher frequency to a lower frequency. It has a duration of less than a second.

(c)

Middle whistler: This type has a one-way path with a slight curve that goes from a higher frequency to a lower frequency. It has a duration between 1 and 2 s.

(d)

Long whistler: This type has a one-way path that goes from a higher frequency to a lower frequency. It has a duration of more than 2 s.

Figure 1 displays the dynamic power spectra produced from the WFC data captured on 15 August 2017 at 02:01:59–02:02:06 UT. This visual representation illustrates the frequency components and their corresponding power levels, enabling further analysis and interpretation of the lightning whistlers’ characteristics. From the figure, it is evident that several lightning whistler events occurred. Between 02:02:01 and 02:02:03, a moderate-intensity middle whistler event is observed. Subsequently, at 02:02:04, a higher-intensity middle whistler event is recorded. Moreover, two nose whistler events are identified after 02:02:05.

3. Detection Process

3.1. System Overview

Figure 2 provides an overview of our research pipeline, which focuses on the detection and analysis of whistler-mode plasma waves using data from the Arase Satellite, specifically from the WFC subsystem. The analysis requires examining both electric and magnetic fields, with manual waveform acquisition based on observations from the OFA spectrum data. After selecting relevant events, the data are reproduced from the Mission Data Recorder (MDR).

The workflow then continued with the generation of spectrograms—visual representations of the frequency content over time—using SPEDAS/PySPEDAS library version 1.4.47. These spectrograms were subsequently fed into a YOLOv5 deep learning object detection model, specifically trained to identify whistlers. The model outputs bounding boxes around the detected whistlers, which were then classified into different categories such as long, middle, short, and nose whistlers.

During the data preprocessing phase, manual data labeling played a crucial role. A subset of labeled data was used for training, where various models were experimented with, and the number of epochs was adjusted. This process was followed by performance analysis and testing to evaluate the model’s accuracy in detecting and classifying whistlers effectively.

3.2. Data Acquisition

In this research, the initial step involved reading the CDF file format followed by the generation of spectrograms, which were then visualized through multiple images, depicting both the frequency and time domains. This conversion was accomplished using the PySPEDAS (Python Space Physics Environment Data Analysis Software) library [20]. PySPEDAS is a Python-based framework designed to facilitate the retrieval, analysis, and visualization of heliophysics time series data from various missions and instruments. Derived from the original SPEDAS framework developed in Interactive Data Language (IDL), PySPEDAS aims to provide similar functionality and user experience but utilizes Python, a more widely used and accessible language in scientific computing.

The entirety of raw data from the Arase Satellite for August 2017 has been transformed into 2736 images. All of the data used are 32-bit color PNG files with dimensions of 1200 × 500 pixels, as they are compatible with YOLO’s input data format (eight-bit depth, three-channel png/jpg images). For the dynamic range of intensity, we used the default value from PySPEDAS as mentioned in Section 2.2. If this value changes, it will affect the image produced by PySPEDAS and also impact the detection results. The next step involves identifying images containing whistlers and assigning appropriate labels.

3.3. Data Preprocessing

In this study, we classified whistlers into four classes: nose whistlers, short whistlers, middle whistlers, and long whistlers, as mentioned in the previous section. This classification allows the model to distinguish between different variations of whistlers that may be encountered in the image data.

We used polygon annotation to label the images, which allows for more accurate and flexible labeling compared to rectangle annotation. This is especially important for whistlers, which can have irregular shapes and sizes. Although polygon annotation is more time-consuming and requires more human effort, it provides more detailed information about the whistlers in the images, which can be beneficial for training the object-detection model. All labeling processes were performed manually by using an online annotation tool.

After preprocessing the data, the next step was to use the data on YOLO (You Only Look Once) [11]. YOLO is a convolutional neural network [23]-based object detection method for 2D images. In contrast to methods based on classifiers, YOLO undergoes training using a loss function directly linked to detection performance, and the entire model is trained simultaneously [11]. YOLO has a very simple architecture and is suitable to be considered for implementation on board a scientific satellite. There are different versions of YOLO, such as YOLOv3, YOLOv4, and YOLOv8. Each version has different features and improvements over the previous ones. For example, YOLOv8 introduces a novel NAS method to design optimal model architectures automatically [24]. It also uses super-gradients to accelerate the training process and improve the model performance. Unfortunately, this version still has some issues with labeling using polygons [25]. Therefore, in this study we used YOLOv5.

3.4. Training

As we mentioned in the first section, we needed to evaluate the minimum size of the YOLO model, so we compared four YOLO models to evaluate their performance. Out of 2736 images in total, we manually identified and labeled 204 images containing whistler events for training purposes, along with 20 additional images for evaluation. Among these, we classified 461 as short whistlers, 87 as long whistlers, 189 as middle whistlers, and 20 as nose whistler events for training. Additionally, there were 55 short whistlers, 2 long whistlers, 10 middle whistlers, and 1 nose whistler for validation. In the context of YOLOv5 training, the nano model (YOLOv5n) features a unique architectural configuration with a depth multiple of 0.33 and a width multiple of 0.25, specifically tailored for its compact design. Notably, the layer channel multiple is set as the default for other YOLOv5 variants. These parameters governing the depth and width of the model play a pivotal role in shaping the nano model’s architecture, distinguishing it from other YOLOv5 models in terms of performance and overall accuracy.

In this research, we are trying to see the performance of all models provided by YOLOv5. There are four models: YOLOv5n (nano), YOLOv5s (small), YOLOv5m (medium), and YOLOv5l (large). These models have differences in their complexities, performances, and overall accuracy [26].

Figure 3 shows a graph of detection parameters used in object detection. The ‘Box’ graph depicts the loss incurred when the predicted bounding boxes fail to cover objects accurately. The ‘Objectness’ graph signifies the loss attributed to erroneous object predictions. ‘Classification’ loss reflects inaccuracies in identifying the correct object class. These metrics collectively highlight the model’s proficiency in object detection, as evidenced by the accurate prediction of bounding box sizes and high confidence in object presence. YOLO generated this figure after the training process was completed.

Moreover, the ‘Precision’ metric evaluates object detection accuracy by determining the ratio of correctly detected objects to the total number of predicted objects, showcasing the model’s effectiveness in correctly identifying objects. Similarly, ’Recall’ assesses the model’s ability to detect all existing objects by dividing the number of correctly detected objects by the total number of objects in the image, demonstrating the model’s comprehensiveness in object detection. ‘${\mathrm{mAP}}^{\mathrm{val}}$@0.5’ represents the mean Average Precision (mAP) at an Intersection over Union (IoU) threshold of 0.5, indicating the model’s accuracy in detecting objects with an IoU of 0.5. Conversely, ‘${\mathrm{mAP}}^{\mathrm{val}}$@0.5:0.95’ averages precision across IoU thresholds ranging from 0.5 to 0.95, offering insights into the model’s accuracy across a broader range of IoU values. These parameters collectively underscore the model’s robust performance in object detection. Figure 3 shows that this model performs quite well in object detection, since the precision reaches above 0.8 and the recall value reaches above 0.3.

3.5. Result and Evaluation

Four models were trained for 1000 epochs and all models achieved their best mAP at varied epochs. Notably, the nano model attained its best mAP at epoch 503. The result is shown in

Table 1. Best mAP values of YOLOv5 models.

Model	Filesize	${\mathbf{mAP}}^{\mathbf{val}}$ 0.5	${\mathbf{mAP}}^{\mathbf{val}}$ 0.5:0.95
YOLOv5l	89.5MB	0.245	0.101
YOLOv5m	40.6MB	0.371	0.114
YOLOv5s	13.8MB	0.32	0.103
YOLOv5n	3.7MB	0.186	0.0773

. YOLOv5l, despite having the largest file size with 46,638,261 parameters and the best mAP (0.245) for ${\mathrm{mAP}}^{\mathrm{val}}$ 0.5, experienced a notable decline in precision for ${\mathrm{mAP}}^{\mathrm{val}}$ 0.5:0.95 (0.101). YOLOv5m, featuring a smaller file size and 7,072,789 parameters, demonstrated a slightly higher mAP for ${\mathrm{mAP}}^{\mathrm{val}}$ 0.5 and thus reveals an improved precision for ${\mathrm{mAP}}^{\mathrm{val}}$ 0.5:0.95 (0.114). YOLOv5s, with a significantly smaller file size and 21,065,925 parameters, obtained a commendable mAP for ${\mathrm{mAP}}^{\mathrm{val}}$ 0.5 but underwent a reduction in precision for ${\mathrm{mAP}}^{\mathrm{val}}$ 0.5:0.95 (0.103). YOLOv5n, having the smallest file size and 780,293 parameters, registered the lowest mAP for ${\mathrm{mAP}}^{\mathrm{val}}$ 0.5 and an even lower precision for ${\mathrm{mAP}}^{\mathrm{val}}$ 0.5:0.95 (0.0773).

Considering the parameter details, it becomes apparent that the precision of the models is influenced not only by the file size but also by the number of parameters. Smaller models may achieve a balance between file size and precision, as seen with YOLOv5m. The larger models, while demonstrating higher precision for certain thresholds, may experience diminishing returns in precision for broader threshold ranges, as observed with YOLOv5l. The choice of a YOLOv5 model should thus factor in file size, parameter count, and precision, aligning with the specific requirements of the application.

YOLOv5n may prove adequate for detecting whistlers in images characterized by simple shapes. This suitability stems from several key factors. First and foremost, YOLOv5n boasts a small model size, specifically 3.7 MB, rendering it ideal for applications with resource constraints, such as mobile devices. Additionally, the model exhibits a high detection speed, surpassing other YOLOv5 variants, which is particularly advantageous for real-time applications like surveillance systems. Furthermore, given that whistlers typically feature uncomplicated shapes, such as straight or curved lines, YOLOv5n demonstrates the capability to accurately detect these straightforward configurations. However, it is essential to acknowledge that various factors may impact YOLOv5n’s performance in whistler detection. Issues such as image quality, with blurry or noisy images potentially diminishing YOLOv5n’s efficacy, and the size of whistlers, especially smaller and thinner ones, might pose challenges for detection. Moreover, the complexity of the background, particularly in busy and intricate scenarios, could potentially disrupt YOLOv5n’s overall performance.

Figure 4 illustrates how effectively the four models can detect four types of whistlers: short, long, middle, and nose whistlers. The red rectangles indicate the bounding box detections, accompanied by the type of whistler and the corresponding confidence scores. From this, we can observe two important points. First, the nano model performed well, comparable to the other models, detecting all types of whistlers and registering a similar number with confidence values above the threshold of 0.5.

Additionally, the shapes of the whistlers that the models found are not very complicated. This suggests that even though more complex models exist, they might not significantly affect how well they find whistlers. Therefore, it seems like the nano model is a good choice for finding whistlers. The high confidence score of the nano model, exceeding the threshold of 0.5, indicates its capability to detect whistlers effectively. It performs similarly to other models, but its smaller size makes it efficient and saves resources for different uses.

4. Discussion

Table 2. Confusion matrix of test data for YOLOv5 nano (Bg: Background; FN: False Negative; FP: False Positive).

Predicted	Short	0.65	0.50	0.20	0	0.89
	Long	0	0	0	0	0
	Middle	0.07	0	0.60	0	0.11
	Nose	0	0	0	0	0
	Bg FN	0.27	0.50	0.20	1.00	0
		Short	Long	Middle	Nose	Bg FP
True

shows the confusion matrix of test data for the YOLOv5 nano model of whistlers. There, 65% of short whistlers and 60% of middle whistlers were detected correctly. It is crucial to acknowledge that the dataset we employed is predominantly characterized by short and middle whistlers, with a limited occurrence of nose whistlers and long whistler events during August 2017. Hence, it is crucial to acknowledge that the results might be influenced by unlabeled whistlers in the dataset, especially regarding long whistlers and nose whistlers. Despite the nano model’s performance not being optimal, it remains a viable option for onboard detection and model selection on satellites, as its confidence score still surpasses the detection threshold, as previously mentioned.

We show one of the examples of the unlabeled dataset in Figure 5. From the annotation image, we can see that not all whistlers are labeled. This is due to our limitations in manually labeling the entire dataset. However, testing results indicate that the model can successfully detect all whistlers, even those that are not labeled. This suggests that the model shows promise for application in automated detection, given that whistler events occur frequently at all times in the atmosphere. In the test results, we observe that this model is unable to detect certain low-intensity whistlers within the time range of 16:44:54 to 16:44:55 due to faint and discontinuous lines that do not meet the detection criteria. This limitation arises because we only labeled whistlers with clear shapes and higher intensities, specifically those above ${10}^{-2},{\mathrm{pT}}^2/\mathrm{Hz}$, based on our current spectrogram.

The nano model’s compact size distinguishes it from others, offering significant efficiency benefits for computational performance and resource utilization, particularly in whistler detection. Despite the YOLOv5n model’s slightly lower accuracy compared to alternatives, the nano model’s efficiency advantages highlight its practical suitability, especially in scenarios where manual labeling of whistlers presents challenges. In such cases, this model excels in detecting unlabeled whistlers effectively.

5. Conclusions

In this study, we evaluated the deep learning model (YOLOv5) size performance for lightning whistler detection by using the dataset for a month observed by the Arase satellite. We compared four different sizes of YOLOv5 models to clarify the appropriate size for lightning whistler detection.

Based on the training results, YOLOv5n stands out as a favorable choice for whistler detection, given its compact size (3.7 MB) and impressive detection speed, making it particularly well-suited for resource-constrained applications like mobile devices or satellites. Despite potential challenges with image quality, smaller whistlers, and complex backgrounds, YOLOv5n’s capabilities make it a commendable option for accurate and efficient detection in scenarios characterized by simple shapes.

As mentioned in the discussion section, future investigations could be refined by focusing on improving the dataset, fine-tuning parameters for enhanced mAP accuracy, and exploring model optimization. This strategic approach seeks to achieve an optimal balance between accuracy and efficiency, ultimately enhancing the model’s effectiveness in real-world applications.