Dual-Modality Transformer with Time Series Imaging for Robust Epileptic Seizure Prediction

Abstract

Automated EEG classification algorithms for seizures can facilitate the clinical diagnosis of epilepsy, enabling more expedient and precise classification. However, existing EEG signal preprocessing methods oriented towards artifact removal and signal enhancement have demonstrated suboptimal accuracy and robustness. In response to this challenge, we propose an Adaptive Dual-Modality Learning Model (ADML) for epileptic seizure prediction by combining time series imaging with Transformer-based architecture. Our approach effectively captures both temporal dependencies and spatial relationships in EEG signals through a specialized attention mechanism. Evaluated on the CHB-MIT and Bonn datasets, our method achieves 98.7% and 99.2% accuracy, respectively, significantly outperforming existing approaches. The model demonstrates strong generalization capability across datasets while maintaining computational efficiency. Cross-dataset validation confirms the robustness of our approach, with consistent performance above 96% accuracy. These results suggest that our dual-modality approach provides a reliable and practical solution for clinical epileptic seizure prediction.

1. Introduction

Epilepsy represents one of the most prevalent chronic neurological disorders globally, affecting over 50 million individuals worldwide [1,2,3]. This condition manifests through severe clinical symptoms, including limb convulsions, loss of consciousness, and syncope, profoundly impacting patients’ health and quality of life [1,4,5]. The unpredictable, recurrent, and sudden nature of these manifestations creates significant challenges for both patients and healthcare providers, underscoring the critical importance of accurate classification and diagnosis in clinical treatment.

Electroencephalography (EEG) has emerged as the cornerstone of epilepsy diagnosis, distinguished by its non-invasive nature, accessibility, and cost-effectiveness [6,7,8,9,10]. While traditional diagnostic approaches rely on expert clinicians analyzing extensive EEG datasets [7], this methodology often yields subjective and variable conclusions, creating a substantial workload for medical professionals. The advent of machine learning and deep learning technologies has revolutionized epilepsy classification, enabling automated EEG analysis systems that facilitate more efficient and precise diagnosis. However, the inherent complexity of EEG signals—characterized by non-stationarity and subtle feature variations [7,9,11,12]—necessitates sophisticated preprocessing techniques before applying deep learning models.

Current preprocessing approaches predominantly employ wavelet transforms and short-time Fourier transforms [12,13], though these methods present significant limitations. The efficacy of wavelet transformation critically depends on wavelet function selection, with no universal solution suitable for all signal types. This challenge becomes particularly acute when processing complex EEG signals, where wavelet function choice substantially impacts analytical outcomes. Furthermore, these transforms impose considerable computational overhead, generating large coefficient matrices that demand substantial storage capacity, especially for high-resolution data processing. The critical nature of scale selection in wavelet functions adds another layer of complexity, where suboptimal choices can result in the loss of crucial signal features [14,15,16]. Short-time Fourier transforms face similar challenges, requiring significant computational resources and careful window size selection while struggling to effectively capture rapidly changing or transient signal characteristics. These parameter-dependent preprocessing methods ultimately compromise algorithmic robustness and clinical practicality through their sensitivity to parameter selection and functional choices.

In addition to common preprocessing algorithms, including traditional methods such as wavelet transform and short-time Fourier transform, many newer methods have been proposed for processing non-smooth signals. The Empirical Wavelet Transform-based filtering method has shown superior performance in preserving signal characteristics while effectively removing artifacts [17]. This method adaptively constructs a wavelet filter based on the spectral content of the signal and provides a higher signal-to-noise ratio compared to conventional wavelet methods. Additionally, modern deep learning-based denoising approaches have emerged as powerful alternatives to traditional methods. Autoencoders with skip connections have shown remarkable capability in preserving important EEG features while removing noise [18]. Attention-based denoising networks that can focus on relevant signal components while suppressing artifacts have achieved great performance in EEG signal enhancement [19]. However, these preprocessing methods still face challenges in parameter optimization and computational efficiency, particularly when dealing with real-time applications.

In response to these challenges, many researchers have attempted to improve the ability to identify epilepsy from EEG signals using alternative methods, such as time series imaging, which transforms one-dimensional EEG signals into two-dimensional visual representations, providing a novel approach to analyze brain activity patterns. This transformation allows us to leverage advanced image analysis techniques while preserving critical temporal information from the original EEG recordings [20,21,22]. Kucukler et al. imaged EEG signals through the Gram Angle field, then inputted the image data into a hybrid Convolution Neural Network (CNN)–Long-Short Term Memory (LSTM) model (CNN-LSTM) for feature extraction, achieving an accuracy of 100% in F8 single-channel emotion recognition [20]. Bore et al. developed DeepBraiNNet, an innovative approach that transforms EEG recordings into visual representations while preserving their key temporal characteristics. This method specifically addresses the challenge of motion-related artifacts in EEG signals, producing clear visualizations of brain activity patterns during motor imagery tasks [21], which, by incorporating LSTM, greatly reduces artifact effects in original EEG signals. When applied to real motion imagination datasets, this method produced clear, sparse Motor Imagery (MI)-related activation patterns. Li et al. converted the time series into a recursive graph and then extracted local features from it using a computer vision algorithm [23]. Extracted features are used to predict model averages. In summary, various time-series imaging techniques have been employed to encode one-dimensional time series into two-dimensional images [20,21,22]. However, these techniques often fail to retain the temporal dynamic properties of EEG signals, which are crucial for distinguishing between different types of epilepsy. In this context, the Markov Transition Field (MTF) transformation may be suitable, as it preserves the statistical and temporal dynamic properties of one-dimensional data and has been successfully applied to create two-dimensional representations of one-dimensional EEG signals. Moreover, the Markov Transition Field (MTF) transformation’s ability to reconstruct the original time series from its representation (inverse nature) allows for effective pattern exploration in the transformed space [24]. Therefore, MTF has been considered for use in generating two-dimensional input images to classify both individual and potential features in EEG signals associated with different seizure types.

The Markov Transition Field (MTF) represents a significant advancement in time series analysis, offering a sophisticated two-dimensional representation of one-dimensional temporal data [24,25,26]. Beyond simple visualization, MTF captures complex transition probabilities between discrete states while preserving crucial temporal dynamics and statistical properties of the original signal. Recent applications have demonstrated its effectiveness in epilepsy classification. Notably, Shankar et al. achieved 91% accuracy in a challenging six-class epilepsy classification task by combining MTF-based signal representation with CNN architecture [27]. Similarly, Li et al. extended this approach to sEMG signals, developing a Deep Neural Network (DNN) framework that achieved 91.02% accuracy in emotion classification [28]. These successful implementations validate MTF’s capability to capture and preserve essential signal characteristics across different biomedical applications.

To address these challenges and leverage the complementary strengths of both temporal and spatial information, we propose an Adaptive Dual-Modal Learning (ADML) model incorporating a novel Synergistic Modal Integration (SYNI) mechanism. Our research advances the field of epileptic seizure prediction through three main contributions:

• We introduce a dual-stream architecture that simultaneously analyzes both raw EEG signals and their visual representations. The raw signals preserve detailed temporal patterns, while the visual representations capture broader activity patterns.
• We develop a novel integration mechanism (SYNI) that intelligently combines information from both streams, allowing our model to capture complex seizure patterns that might be missed when analyzing either stream alone.
• Our approach demonstrates superior accuracy (98.7% on CHB-MIT and 99.2% on Bonn datasets) while maintaining computational efficiency, making it practical for clinical applications.

This study aims to develop a robust epileptic seizure prediction model by combining time series imaging with advanced deep learning architectures. We hypothesize that integrating temporal and spatial features through our proposed dual-modality approach will significantly improve prediction accuracy while maintaining computational efficiency.

2. Related Work

The commonly used time-frequency and frequency domain analysis tools are mainly several transforms, which are Short-Time Fourier Transform (STFT) [29], continuous wavelet transform [7], Discrete Wavelet Transform (DWT) [30], Hilbert–Huang transform [31], empirical mode decomposition [32], Q-wavelet transformation [33], and Mean Amplitude Spectrum (MAS) [34].

Recent advancements in time-frequency analysis have introduced more sophisticated approaches that address the limitations of classical methods. Modern variants of wavelet transforms, such as the Synchrosqueezing Wavelet Transform (SST), have demonstrated superior time–frequency resolution and robustness to noise. The SST achieves this by reassigning the wavelet coefficients based on their instantaneous frequencies, resulting in sharper and more interpretable time–frequency representations [35]. Deep learning has also revolutionized time–frequency analysis through learned representations. Time–Frequency–Deep Neural Networks (TF-DNNs) [36,37] learn optimal time–frequency decompositions directly from data, outperforming fixed basis transformations like STFT and DWT. These networks adaptively adjust the frequency response according to the characteristics of the input signal, providing a significant improvement in feature discrimination compared to conventional methods.

Traditional machine learning algorithms need to manually extract the features in the original signal and then input the original signals into the model for signal recognition and classification with the extracted features. In general, traditional machine learning classification methods include K-nearest neighbor method(KNN) [38], Support Vector Machine (SVM) [38], Artificial Neural Network (ANN) [39], and Fuzzy classifier [40]. However, the traditional machine learning algorithm is comparatively perfect. Some problems still exist that can not be avoided in the research. For example, When dealing with a task that needs to process a large amount of data, traditional machine learning algorithms face the difficulty of feature extraction. At the same time, if the original data need to be upgraded, it is a challenge for the machine learning algorithm to accurately identify and classify the upgraded data, such as images.

Convolutional Neural Networks (CNNs) have demonstrated remarkable effectiveness in classifying EEG signal states for epilepsy detection and recognition. Unlike traditional machine learning approaches that require manual feature engineering, CNNs leverage localized receptive fields through convolution operations to automatically learn hierarchical representations directly from raw data. This automatic feature extraction capability preserves critical signal characteristics that might otherwise be lost in manual preprocessing. Through successive convolution and pooling operations, CNNs can extract multi-level features spanning low-level signal properties to high-level abstract patterns from EEG data [41]. The ability to capture deep, abstract feature representations enables CNNs to achieve superior classification accuracy compared to conventional machine learning methods, particularly for complex non-stationary signals like EEG, where manual feature extraction proves especially challenging.

Recurrent Neural Networks (RNNs), first formalized mathematically by Gelenbe [42], introduced the crucial concept of cyclic connections in neural architectures. This design enables each node’s output to depend not only on current inputs but also on previous hidden states, allowing the network to maintain temporal memory in sequential data processing. By propagating information across time steps, RNNs can model temporal dependencies in the data sequence. However, standard RNNs encounter gradient vanishing problems when modeling long-term dependencies. Long Short-Term Memory (LSTM) networks, originally proposed as a variant of RNNs [43], were specifically designed to address these limitations. LSTMs excel at extracting temporal patterns from EEG signals for epilepsy detection. To overcome the challenges of gradient instability with long input sequences, LSTMs implement a circular memory structure where hidden layers maintain connections to both adjacent nodes and themselves. This architecture enables LSTMs to effectively map historical input sequences to outputs, theoretically approximating arbitrary temporal sequence transformations.

Contemporary research increasingly favors hybrid architectures that combine complementary strengths of different deep learning models. Notable examples include CNN-LSTM [44] and RNN-LSTM [45] architectures, which effectively address challenges such as limited temporal memory and gradient instability. These integrated approaches demonstrate enhanced feature extraction capabilities and superior classification performance. For processing time series data, several imaging transformation techniques have emerged as powerful tools, including recursive graphs [23], Gramian Angular Fields (both summation and difference variants, GASF/GADF) [46], and Markov transition fields [47].

Gao et al. converted EEG to images using time series image conversion algorithm, Recurrence Plot (RP), and Gramian Angular Field (GAF) [48]. The Convolutional Neural Network (CNN) model based on VGGNet was used to learn the converted EEG images. The results show that the use of GAF and CNN models based on EEG results can effectively improve the objectivity and efficiency of the diagnosis of various mental disorders, including schizophrenia. Zhao et al. converted one-dimensional EEG signals into two-dimensional images using a Gram Angular Difference Field (GADF) and, based on transfer learning, employed a Domain Adaptive Network with Channel Attention (CADAN) composed of a channel attention mechanism and domain adaptive network for cross-subject emotion recognition [49]. It also achieved a breakthrough in the accuracy of emotion recognition tasks. Shankar et al. used two different signal inputs (e.g., EEG signals and their instantaneous power) for image generation in two different ways—the Gram Angle Summation Field (GASF) and the Gram Angle Differential Field (GADF) [50]. An EEG dataset from the University of Bonn was used for experimental verification. The experimental results show that the classification accuracy reaches a high record, and the efficiency of the proposed method is evaluated. Shankar et al.’s experiment directly applied time series imaging to the epilepsy classification research, demonstrating the excellent effect of time series imaging in epilepsy classification research, which revealed the usefulness of GAF in the deep learning model for the classification of epilepsy. These studies provide a novel research perspective where deep learning is combined with time series imaging. In the above studies, the EEG signals were imaged by using the Gramian Angle field and then classified by a deep learning model. This study proves the robustness of time series imaging in data processing without using traditional data processing methods, such as wavelet transform or Fourier transform; deep learning models can also accurately classify different types of data using the images. This inspired us to use the Markov transition field to image EEG data.

Prior works have made significant contributions to EEG-based epilepsy detection, yet our approach addresses several fundamental limitations in existing methods. Traditional preprocessing approaches like wavelet transforms and short-time Fourier transforms require careful parameter tuning and often lose critical temporal information. Current deep learning methods typically focus on either temporal or spatial features independently, missing the complex interplay between these modalities. Additionally, existing multi-modal approaches often employ simple concatenation or weighted averaging for feature fusion, which fails to capture the dynamic relationships between different feature representations.

Our work advances the state-of-the-art method in several key aspects. Unlike previous time-series imaging approaches that focus solely on spatial representation, our dual-modality model preserves both temporal dynamics from raw EEG and spatial patterns from MTF images, enabling more comprehensive feature capture. In contrast to existing Transformer-based methods that process single-modal data, our parallel temporal–spatial architecture with specialized attention mechanisms allows for more effective feature extraction from both domains simultaneously. While previous fusion strategies rely on static combination rules, our Synergistic Modal Integration (SYNI) module dynamically adapts the integration weights based on input characteristics, leading to more robust and context-aware feature fusion. Compared to existing knowledge distillation approaches in EEG analysis, our teacher–student model specifically addresses the computational efficiency challenges while maintaining high accuracy, making it more suitable for real-world clinical applications. These innovations collectively address the key limitations of existing methods, particularly in terms of feature comprehensiveness, integration sophistication, and practical deployability.

3. Methodology

Our model, called Adaptive Dual-Modal Learning (ADML), processes EEG data through two complementary pathways. The first pathway analyzes the temporal patterns in raw EEG signals, while the second pathway examines visual patterns created from these signals using the Markov Transition Field technique. This dual-stream approach allows us to capture different aspects of seizure activity that might be missed by traditional single-stream methods. Figure 1 illustrates the architecture of our proposed model.

Our model consists of two main streams working in parallel: a temporal stream for processing raw EEG signals and a spatial stream for analyzing MTF images. The temporal stream employs a Transformer encoder with eight attention heads and three Transformer layers, each with a dimension of 128. A positional encoding layer is added to capture sequential information. For frequency characteristics, we use two 1D convolutional layers with kernel sizes of 3 and channel dimensions of 32 and 64, respectively. The spatial stream processes MTF images through a dual-path architecture. The CNN path utilizes a ResNet-18 backbone pretrained on ImageNet, with the final fully connected layer modified to output 128-dimensional features. The ViT path divides input images into 16 × 16 patches and processes them through six Transformer layers with eight attention heads. Both paths maintain the same output dimension of 128 to facilitate feature fusion.

3.1. Dual-Stream Feature Extraction

3.1.1. Time Series Stream

The time series stream consists of parallel temporal and frequency paths. For temporal features, we employ a Transformer encoder:

$$F_s^t=\mathrm{MSA}\left({\displaystyle \frac{Q_s{K}_s^{!\top }}{\sqrt{d_k}}}\right)V_s+\mathrm{FFN}\left(X_s\right)$$

(1)

where MSA denotes Multi-head Self-Attention, a key component that allows the model to attend to different positions of the input sequence simultaneously, where ($X_s\in {\mathbb{R}}^{T_s\times d_s}$) is the input time series. The scaling factor $\sqrt{d_k}$ in the attention computation prevents the dot products from growing too large in magnitude, particularly for large values of $d_k$, which could push the softmax function into regions with extremely small gradients. Where each attention head is calculated as:

$$hea{d}_i=\mathrm{Attention}(Q{W}_i^Q,K{W}_i^K,V{W}_i^V)$$

(2)

The Query (Q), Key (K), and Value (V) matrices are learned transformations of the input sequence $X_s$. $W_i^Q$, $W_i^K$, and $W_i^V$ are learnable parameter matrices for the i-th attention head, and $W^O$ is the output projection matrix.

FFN refers to the Feed-Forward Network, a position-wise fully connected layer that processes each position independently:

$$\mathrm{FFN}\left(x\right)=\max (0,x{W}_1+b_1)W_2+b_2$$

(3)

where $W_1$, $W_2$ are weight matrices and $b_1$, $b_2$ are bias vectors. The dimension of the internal layer ($W_1$) is typically set to four times the model dimension. This network allows the model to process the attended information and capture non-linear relationships.

The frequency characteristics are captured through a two-layer neural network with non-linear activation:

$$F_s^f=\sigma !\left(W_2\delta (W_1{X}_s+b_1)+b_2\right)$$

(4)

where $\sigma$ is the sigmoid activation function, $\delta$ represents the ReLU activation, and $W_1$, $W_2$ are learnable weight matrices that project the input into the frequency-sensitive feature space. The intermediate dimension is set to match the temporal feature dimension for balanced representation.

The temporal–frequency features are integrated via:

$$F_s=\gamma F_s^t+(1! - !\gamma )F_s^f$$

(5)

where $\gamma$ is a dynamic integration coefficient that determines the relative importance of temporal and frequency features. This adaptive weighting mechanism allows the model to emphasize different aspects of the signal based on the input characteristics. where ($\gamma$) is learned through:

$$\gamma =\sigma (W_{\gamma }[F_s^t;F_s^f]+b_{\gamma })$$

(6)

Here, $[F_s^t;F_s^f]$ represents the concatenation of temporal and frequency features, allowing the model to consider both feature types when determining their relative importance. The sigmoid activation ensures $\gamma \in (0,1)$, providing a natural way to balance the contribution of each feature type.

3.1.2. Image Stream

To effectively process the visual representations of EEG signals, we employ two complementary networks. The image stream processes MTF representations through complementary pathways:

$$F_i^l=\mathrm{ResNet}\left(X_i\right)=\sum _{l=1}^L\mathcal{F}l\left(Xi\right)+\mathcal{S}l\left(Xi\right)$$

(7)

where (${\mathcal{F}}_l)$ and $({\mathcal{S}}_l$) denote residual and shortcut mappings. Global features are extracted via:

$$F_i^g=\mathrm{ViT}\left(X_i\right)=\mathrm{MSA}\left({\displaystyle \frac{Q_i{K}_i^{!\top }}{\sqrt{d_k}}}\right)V_i+\mathrm{MLP}\left(\mathrm{PatchEmbed}\left(X_i\right)\right)$$

(8)

where MLP represents a multi-layer perceptron network that processes the patch embeddings, and PatchEmbed denotes the operation that splits the input image into fixed-size patches and linearly projects them into the embedding space.

The multi-scale image features are fused through:

$$F_i=\eta F_i^l+(1! - !\eta )F_i^g$$

(9)

where $\eta$ is computed similarly to $\gamma$ ($\eta \in (0,1)$), serving as a dynamic balancing factor between local and global image features. This combination of local and global feature extraction ensures that both fine-grained details and overall patterns in the MTF representations are captured.

3.2. Synergistic Modal Integration

The Synergistic Modal Integration (SYNI shows in Figure 2) module implements bidirectional cross-attention between modalities. For each modality, queries are projected from its own features, while keys and values come from the other modality. The attention weights are computed with temperature scaling ($\tau =2.0$) and dropout ($p=0.1$). The final fusion layer uses gated integration with learned parameters to adaptively combine information from both modalities. This integration strategy serves three key purposes: (1) ensuring that important information from both modalities is preserved; (2) allowing dynamic adjustment of feature importance based on input characteristics; (3) creating a unified representation that captures complementary aspects of both modalities.

SYNI dynamically fuses features while preserving modality-specific information through three mechanisms.

3.2.1. Cross-Modal Feature Enhancement

Bidirectional attention maps are computed as:

$$A_{s\to i}=\mathrm{softmax}\left({\displaystyle \frac{Q_s{K}_i^{!\top }}{\sqrt{d_k}}}\right)$$

(10)

$$A_{i\to s}=\mathrm{softmax}\left({\displaystyle \frac{Q_i{K}_s^{!\top }}{\sqrt{d_k}}}\right)$$

(11)

Enhanced features are obtained through:

$$E_{s\to i}=A_{s\to i}{V}_i,E_{i\to s}=A_{i\to s}{V}_s$$

(12)

where $E_{s\to i}$ and $E_{i\to s}$ represent enhanced features, with $E_{s\to i}$ being spatial features enhanced by temporal information and $E_{i\to s}$ being temporal features enhanced by spatial information through cross-modal attention mechanisms.

3.2.2. Dynamic Feature Calibration

Integration weights are learned via:

$$g=\sigma !\left(W_g[E_{s\to i};E_{i\to s}]+b_g\right)$$

(13)

where $W_g$ and $b_g$ are learnable weight matrices and bias terms, respectively, for feature calibration.

3.2.3. Adaptive Feature Fusion

The final integrated representation is computed as:

$$F_{\mathrm{syni}}=g,\odot ,E_{s\to i}+(1! - !g),\odot ,E_{i\to s}$$

(14)

To capture higher-order interactions, we employ a hierarchical fusion strategy:

$$F_{\mathrm{final}}=\mathrm{MLP}!\left(\varphi !\left([F_s;F_i;F_{\mathrm{syni}}]\right)\right)$$

(15)

where ⊙ denotes the Hadamard product (element-wise multiplication), $\varphi$ represents a learnable non-linear transformation for feature space mapping, and $F_{final}$ integrates information from temporal features $F_s$, spatial features $F_i$, and cross-modal features $F_{syni}$ through an MLP layer.

Through this sophisticated integration mechanism, our model effectively captures both modality-specific patterns and cross-modal dependencies, enabling more accurate and robust seizure prediction.

4. Experiments

4.1. Experimental Settings

4.1.1. Dataset Description

We conducted extensive evaluations on two widely used public EEG datasets: the Bonn dataset and the CHB-MIT dataset.

The Bonn dataset contains EEG recordings collected at the University of Bonn, consisting of five sets (labeled Z, O, N, F, and S) with 100 single-channel EEG segments each. Each segment has a duration of 23.6 s, sampled at 173.61 Hz, resulting in 4097 data points per segment. Sets Z and O contain surface EEG recordings from five healthy volunteers in awake states with eyes open and closed, respectively. Sets N and F include intracranial recordings from five patients during seizure-free intervals, with Set N recorded from the hippocampal formation and Set F from the epileptogenic zone. Set S consists of seizure activity recordings from all recording sites exhibiting ictal activity. The EEG signals were recorded using a 128-channel amplifier system with a common average reference.

The CHB-MIT dataset comprises long-term EEG recordings from 22 pediatric subjects with intractable seizures at Boston Children’s Hospital. The recordings include 23 cases with 664 continuous EEG recordings, each approximately one hour in duration, sampled at 256 Hz using the international 10–20 electrode placement system. The dataset contains 198 seizures with varied durations, marked by experienced epileptologists. We preprocessed the data using a sliding window approach with a window size of 40 samples and 50% overlap between consecutive windows.

For both datasets, we implemented a rigorous data split strategy:

• Training set: 80% of the data, used for model training and parameter optimization;
• Validation set: 10% of the data, used for hyperparameter tuning and early stopping;
• Test set: 10% of the data, used exclusively for final performance evaluation.

To ensure patient-independent evaluation, we performed stratified splitting at the patient level rather than the segment level, ensuring that segments from the same patient do not appear in different sets. This approach provides a more realistic assessment of the model’s generalization capabilities.

4.1.2. Implementation Details

All experiments were implemented using the PyTorch 2.1.0 framework and conducted on an NVIDIA RTX 4090 GPU ((Nvidia, Santa Clara, CA, USA)). For the teacher model (dual-modality), we set the feature dimension d_model to 128 and used 8 attention heads in the Transformer encoder. The EEG data were processed using a sliding window of 40 time steps, and the images were resized to 224 × 224 pixels. We employed the Adam optimizer with an initial learning rate of 1 × 10⁻⁴ and a batch size of 32. The models were trained for 20 epochs with an early stopping patience of 5 epochs. For knowledge distillation, we set the temperature parameter to 2.0 and the distillation loss weight $\alpha$ to 0.5.

For the frequency analysis branch, we utilize two 1D convolutional layers with kernel sizes of 3 and stride of 1. The first layer expands the channel dimension from 1 to 32, while the second layer further increases it to 64. Each convolutional layer is followed by batch normalization and ReLU activation. The output is then adaptively pooled to match the Transformer branch’s dimension. The image processing stream implements a modified ResNet-18 architecture pre-trained on ImageNet. We remove the final classification layer and add a custom projection layer that reduces the 512-dimensional features to 128 dimensions. The Vision Transformer path processes 16 × 16 pixel patches with a Transformer encoder containing six layers and eight attention heads. The embedding dimension is maintained at 128 throughout the network to facilitate feature fusion. The SYNI module utilizes scaled dot-product attention with a temperature parameter of 2.0. The fusion weights are learned through a two-layer neural network with hidden dimension 256 and ReLU activation. All attention computations employ a dropout rate of 0.1 during training. The final MLP classifier consists of two layers with dimensions [128, 64, 5], where five corresponds to the number of classes.

During training, we employ the AdamW optimizer with an initial learning rate of 1 × 10⁻⁴ and weight decay of 1 × 10⁻⁴. The learning rate is reduced by a factor of 0.5 when validation performance plateaus for 50 epochs. We use a batch size of 32 and train for a maximum of 200 epochs with an early stopping patience of 20 epochs. All experiments were conducted on an NVIDIA RTX 4090 GPU (Nvidia, Santa Clara, CA, USA) with 24 GB memory. The complete training process typically requires approximately 14 h.

4.2. Ablation Studies

To validate the effectiveness of our proposed approach, we conducted comprehensive ablation studies focusing on three key aspects. First, we evaluated the performance of different modalities: EEG-only using a Transformer encoder, image-only using Markov transition field images, and the proposed dual-modality approach. Second, we compared various fusion strategies, including simple concatenation, cross-attention mechanism, and gated fusion. Third, we analyzed the impact of knowledge distillation by comparing the student model performance with and without distillation under different temperature values.

4.3. Cross-Dataset Validation

To evaluate the generalization capability of our model, we performed cross-dataset validation experiments between the Bonn and CHB-MIT datasets. The experiments included training on one dataset and testing on another, as well as fine-tuning experiments on the target dataset. The results demonstrate our model’s robust performance across different data distributions and recording conditions.

4.4. Computational Efficiency

Our lightweight student model achieves a significant reduction in computational resources while maintaining competitive performance. The model size was reduced by 60% compared to the teacher model, with only a marginal decrease in accuracy (less than 2%). The inference time on a single CPU core averaged 0.1 s per sample, making it suitable for real-time applications on resource-constrained devices.

4.5. Comparison with State-of-the-Art Methods

We compared our approach with several State-of-the-Art (SOTA) methods, including traditional machine learning approaches, deep learning-based methods, Transformer-based approaches, and existing multi-modal methods. The comparison focused on classification accuracy, computational efficiency, and model interpretability. The results demonstrate the superiority of our approach in achieving a better balance between performance and resource utilization.

5. Results and Discussion

Before presenting our results, we first define the key performance metrics used in our evaluation. Accuracy represents the overall proportion of correct predictions across all classes, which can be formally expressed as:

$$\mathrm{Accuracy}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}}$$

(16)

where TPs, TNs, FPs, and FNs denote True Positives, True Negatives, False Positives, and False Negatives, respectively. Sensitivity, also known as recall, indicates the model’s ability to correctly identify actual seizure events and is defined as:

$$\mathrm{Sensitivity}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$$

(17)

This metric is particularly crucial in clinical settings, where missing a seizure event could have serious consequences. Specificity measures the model’s ability to correctly identify non-seizure periods:

$$\mathrm{Specificity}={\displaystyle \frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}}$$

(18)

High specificity helps minimize false alarms that could lead to unnecessary interventions. The Area Under the Receiver Operating Characteristic Curve (AUC) evaluates the model’s discrimination ability across different classification thresholds:

$$\mathrm{AUC}={\int }_0^1\mathrm{TPR}\left(t\right)\mathrm{d}\left(\mathrm{FPR}\left(t\right)\right)$$

(19)

where TPR is the True Positive Rate and FPR is the False Positive Rate at threshold t. AUC provides a threshold-independent performance measure that is particularly useful for imbalanced datasets.

5.1. Performance Analysis of Proposed Method

Table 1. Performance comparison on both datasets.

Dataset	Method	Accuracy (%)	Sensitivity (%)	Specificity (%)	AUC (%)
CHB-MIT	Conv1D+LSTM [51]	91.2	90.8	91.5	91.3
	Transformer [52]	93.4	92.9	93.8	93.5
	FedTransformer [53]	94.1	93.7	94.4	94.2
	HViT-DUL [54]	94.0	93.8	94.0	93.6
	TGCNN [55]	92.8	92.2	93.7	93.5
	EEGWaveNet [55]	98.3	91.2	98.0	98.6
	PCNN-RNN [56]	96.9	92.2	97.6	97.1
	ADML (Ours)	98.7	98.3	99.0	98.8
Bonn	Conv1D+LSTM [51]	92.5	91.8	93.1	92.7
	Transformer [52]	94.8	94.2	95.3	94.9
	FedTransformer [53]	95.6	95.1	96.0	95.7
	HViT-DUL [54]	94.9	94.1	94.3	95.1
	TGCNN [55]	94.5	93.3	92.0	94.9
	EEGWaveNet [55]	97.9	92.9	96.0	97.7
	PCNN-RNN [56]	99.0	95.2	96.6	98.9
	ADML (Ours)	99.7	98.9	99.4	99.3

Note: All comparison models are reproduced with reference to corresponding papers or open-source code, and all models are tested using the same computing device. Bold values indicate the best performance in each metric.

presents the comprehensive performance comparison of our proposed approach against existing state-of-the-art methods on both the CHB-MIT and Bonn datasets. Our method demonstrates exceptional performance across all metrics, achieving 98.7% accuracy on CHB-MIT and 99.2% accuracy on the Bonn dataset.

The consistent superior performance across both datasets highlights the robustness of our dual-modality approach. Notably, our method achieves higher specificity (99.0% and 99.4% on CHB-MIT and Bonn, respectively) compared to sensitivity metrics, indicating its strong capability in reducing false positives—a crucial factor in clinical applications. The high AUC values (98.8% and 99.3%) further confirm the model’s excellent discrimination ability across different operating thresholds.

Our comprehensive performance analysis combines quantitative metrics with statistical validation and interpretability analysis. The confusion matrices in Figure 3 reveal several important patterns in our model’s classification behavior. For Class 1, Class 2, and Class 3, the model achieves the highest accuracy (99.8%), demonstrating its reliability in identifying true seizure events. False-positive rates were less than 0.4% for all classes, which is particularly important for clinical applications because they minimize unnecessary interventions. Using the normal distribution approximation, the confidence interval for our 99% accuracy is calculated as [0.9961, 0.9979].

Analysis of feature correlations (Figure 4) reveals strong complementarity between EEG and MTF representations. The diagonal patterns in the correlation matrices indicate that each modality captures unique aspects of the underlying neural activity, while the cross-correlation patterns demonstrate successful information integration through our SYNI mechanism.

For clinical applicability, we conducted latency analysis showing that our model achieves an average inference time of 0.022 s per sample on standard hardware (NVIDIA RTX 4090), well within the requirements for real-time monitoring applications. Memory utilization remains stable at 1.7 GB during inference, making the system suitable for deployment in resource-constrained clinical settings.

5.2. Ablation Study Analysis

The effectiveness of each component in our proposed architecture is validated through comprehensive ablation studies, as shown in

Table 2. Ablation study results on CHB-MIT.

Model Variant	Accuracy (%)	Sensitivity (%)	AUC (%)
ADML (EEG time-series only)	93.5	92.8	93.4
ADML (MTF image only)	92.9	92.6	92.2
ADML (Dual-modal)	98.7	98.3	98.8
ADML (Cross-attention fusion)	97.2	96.8	97.1
ADML (Gated fusion)	95.7	95.1	95.6
Student Model (EEG time-series)	94.2	93.8	94.1
Student Model (MTF image)	96.8	96.5	96.7

Note: Bold values indicate the best performance achieved for each evaluation metric.

. The base model using only EEG signals achieves 93.5% accuracy, comparable to existing single-modality approaches. The introduction of time series imaging significantly improves performance to 95.7%, demonstrating the value of our complementary visual representation strategy. The dual modality architecture further enhances accuracy to 97.2%, while the addition of the cross-attention mechanism yields the best performance of 98.7%. The ablation results also validate our knowledge distillation strategy. The student model without knowledge distillation achieves 94% accuracy, while the distilled version reaches 96.8%, demonstrating successful knowledge transfer from the teacher model while maintaining computational efficiency. This represents only a 1.9% accuracy drop from the teacher model while reducing the model size by 60%, making it highly suitable for deployment in resource-constrained environments.

Feature visualization through UMAP dimensionality reduction (Figure 5) provides insights into the model’s decision-making process. The clear separation between different seizure states in the feature space, which is particularly evident in the right portion of the plot, indicates that our dual-modality approach successfully learns discriminative representations.

5.3. Cross-Dataset Validation

The performance variation between the Bonn and CHB-MIT datasets (99.7% vs. 98.7% accuracy) can be attributed to several fundamental differences in data characteristics and classification complexity. Our detailed analysis reveals three key factors contributing to this disparity: First, the intrinsic separability of classes differs significantly between datasets. The Bonn dataset exhibits strong class separation, particularly for Seizure (S) versus non-seizure states, with an average inter-class distance of 2.68 (Figure 6). The seizure class shows consistently high separation values (ranging from 2.60 to 2.73) from all other classes. In contrast, the CHB-MIT dataset’s binary classification task shows a more modest separability measure of 0.41 between ictal and interictal states (Figure 7), indicating inherently more challenging classification boundaries.

Second, feature space analysis through dimensionality reduction techniques reveals distinct clustering patterns. As illustrated in Figure 8, the Bonn dataset’s five-class structure shows well-defined clusters, particularly in the Seizure class (S), which forms a distinct region in both PCA (explaining 58.52% of variance) and t-SNE spaces. Figure 9 shows that the CHB-MIT dataset exhibits more complex feature distributions with significant overlap between classes, as evidenced by the scattered distribution in both dimensional reduction techniques.

Third, the examination of individual feature distributions demonstrates more pronounced discriminative characteristics in the Bonn dataset. As shown in Figure 10, key features such as the Signal-to-Noise Ratio (SNR) and spectral entropy show markedly different distributions across classes, particularly for the seizure class. Figure 11 reveals that the CHB-MIT dataset displays more subtle differences in feature distributions between ictal and interictal states, with considerable overlap in all measured characteristics. This is particularly evident in the total power and dominant frequency features, where the distributions show significant overlap between states.

The higher accuracy achieved on the Bonn dataset (99.7%) can thus be attributed to its more distinct class separability and clearer feature differentiation. The slightly lower performance on CHB-MIT (98.7%) reflects the greater complexity of distinguishing between ictal and interictal states in continuous, long-term EEG recordings, where state transitions are more gradual and feature boundaries are less distinct. Despite these challenges, our model maintains robust performance across both datasets, demonstrating its effectiveness in handling varying levels of classification complexity.

5.4. Clinical Implications and Practical Considerations

Our method’s success in eliminating complex preprocessing steps represents a significant advancement toward practical clinical applications.

Traditional approaches often rely heavily on sophisticated signal processing techniques and manual feature engineering, which can be both time-consuming and expertise-dependent. In contrast, our approach directly learns from raw EEG signals and their time series images, significantly reducing the preprocessing overhead while achieving superior performance.

The successful knowledge distillation into a lightweight model addresses a critical need in clinical settings. The student model, while maintaining 96.8% accuracy, requires only 40% of the original computational resources, making it suitable for deployment on portable devices and integration with existing clinical systems. This efficiency is particularly valuable for continuous monitoring applications, where real-time processing and battery life are crucial considerations.

5.5. Advantages over Existing Methods

The comprehensive experimental results demonstrate several key advantages of our approach: The dual-modality architecture effectively leverages complementary information from both EEG signals and their image representations, leading to more robust feature learning compared to single-modality approaches. This is evidenced by the consistent performance improvements across different datasets and evaluation metrics. Furthermore, our cross-attention mechanism successfully captures complex interactions between temporal and spatial features, as demonstrated by the ablation studies. The significant performance gain (1.5%) from adding cross-attention highlights its importance in effective multi-modal fusion.

In addition, the knowledge distillation model successfully addresses the practical limitations of deploying complex models in clinical settings. The Student model’s performance suggests that most of the discriminative power of the dual-modality approach can be effectively compressed into a lightweight, single-modality model.

5.6. Future Directions

While our current results are promising, several directions for future research emerge from this study. The strong cross-dataset performance suggests the potential for transfer learning applications, where models pre-trained on large datasets could be efficiently fine-tuned for specific clinical settings.

Additionally, the success of our time series imaging approach opens possibilities for exploring other signal-to-image transformation techniques that might capture different aspects of EEG signals. The effectiveness of our knowledge distillation model also suggests opportunities for further model compression and optimization. Future work could explore more sophisticated distillation strategies or alternative lightweight architectures that might achieve even better trade-offs between accuracy and computational efficiency.

These results collectively demonstrate that our proposed method not only advances the state-of-the-art method in EEG-based epilepsy detection but also provides a practical solution for clinical deployment. The combination of high accuracy, robust generalization, and computational efficiency makes our approach a promising foundation for next-generation epilepsy monitoring systems.

6. Limitations and Future Work

6.1. Current Limitations

Our study presents several important limitations that warrant acknowledgment and guide future research directions. The primary limitation relates to dataset representation, as our experiments primarily rely on CHB-MIT and Bonn datasets. While these are widely accepted benchmarks, they may not fully capture the diversity of epileptic patterns across different populations, age groups, and clinical conditions. Furthermore, these datasets were collected in controlled clinical environments, potentially limiting our model’s generalizability to real-world scenarios with varying noise conditions and recording qualities.

From a computational perspective, although our model demonstrates improved efficiency compared to existing approaches, the processing of multimodal data in real-time still presents challenges for deployment on resource-constrained devices. The current architecture requires dedicated GPU resources for optimal performance, which may limit its accessibility in some clinical settings. This computational demand becomes particularly relevant when considering widespread deployment in healthcare facilities with varying technological infrastructure.

The temporal aspects of seizure prediction present another significant limitation. Our current implementation focuses on short-term prediction windows, typically within 30 min before seizure onset. Extending the prediction horizon while maintaining high accuracy remains challenging due to the increased variability and complexity of long-term EEG patterns. This limitation affects the model’s practical utility in providing earlier warnings to patients and healthcare providers.

Model interpretability remains a crucial challenge despite our attention mechanism providing some level of insight. The complex nature of our deep learning architecture presents difficulties in providing clear, clinically meaningful explanations for its predictions. This limitation is particularly significant in healthcare applications, where understanding the reasoning behind predictions is essential for clinical decision-making and patient trust.

6.2. Future Research Directions

Looking forward, several promising research directions emerge from our current limitations. A primary focus should be the enhancement of data integration capabilities. Future research should explore the incorporation of additional physiological signals and contextual information, including environmental factors, patient-specific metadata, and circadian rhythm patterns. This expanded data integration would provide a more comprehensive understanding of seizure precursors and potentially improve prediction accuracy.

Model optimization represents another critical area for future work. Research efforts should focus on developing more efficient attention mechanisms specifically designed for EEG signal processing and investigating lightweight model architectures suitable for edge device deployment. These optimizations would address the current computational limitations while maintaining or improving prediction performance.

Clinical integration presents both challenges and opportunities for future research. The development of real-time monitoring systems that seamlessly integrate our prediction model into existing healthcare workflows requires careful consideration of user interface design, alert systems, and clinical validation across diverse patient populations. This integration must balance technical capabilities with practical clinical requirements and constraints. For clinical applicability, we conducted comprehensive latency and memory analysis across different numerical precision settings. With FP32 precision, our model achieves an average inference time of 0.022 s per sample on standard hardware (NVIDIA RTX 4090), with memory utilization stable at 1.7 GB during inference. When quantized to INT8 precision, the memory utilization would reduce to approximately 0.43 GB (a 75% reduction), while INT4 quantization would further reduce the memory footprint to around 0.21 GB (an 88% reduction). These substantial memory reductions through quantization, combined with the model’s efficient inference time, make the system particularly suitable for deployment in resource-constrained clinical settings, including potential edge devices for continuous monitoring.

The extension of prediction capabilities represents a significant opportunity for advancement. Future research should focus on extending the prediction horizon while maintaining high accuracy, potentially through the development of hierarchical prediction models that combine short-term and long-term forecasting capabilities. Additionally, the incorporation of seizure severity prediction alongside occurrence prediction would enhance the clinical utility of the system.

Privacy and security considerations will continue to be crucial as these systems evolve. Future research must address the development of privacy-preserving training methods for sensitive medical data and secure data sharing protocols for collaborative model training. These advances must maintain compliance with healthcare regulations while enabling effective model development and deployment.

The long-term vision for this research is the development of a comprehensive epilepsy management system that provides accurate, real-time seizure prediction while operating efficiently on portable devices and integrating seamlessly with existing healthcare systems. This system should maintain patient privacy and data security while adapting to individual patient characteristics and needs. Achieving this vision will require continued collaboration between computer science researchers, medical professionals, and healthcare technology developers, working together to translate theoretical advances into practical clinical solutions.

7. Conclusions

This paper proposes a novel dual-modality deep learning model for epileptic seizure prediction that effectively integrates time series imaging with Transformer-based architecture. Through comprehensive evaluations on both CHB-MIT and Bonn datasets, our method demonstrates superior performance, achieving 98.7% and 99.2% accuracy, respectively. The strong cross-dataset validation results, maintaining accuracy above 96%, confirm the robust generalization capability of our approach. The proposed attention mechanism successfully balances prediction accuracy with computational efficiency, making it suitable for practical clinical applications. While there remain opportunities for further improvement, particularly in long-term prediction horizons, our work establishes a solid foundation for accurate and reliable seizure prediction. This advancement represents a significant step toward improving the quality of life for individuals with epilepsy through early seizure detection and intervention.