Next Article in Journal
A Framework for Evaluating Dynamic Directed Brain Connectivity Estimation Methods Using Synthetic EEG Signal Generation
Previous Article in Journal
Data-Driven Formation Control for Multi-Vehicle Systems Induced by Leader Motion
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Topological Reinforcement Adaptive Algorithm (TOREADA) Application to the Alerting of Convulsive Seizures and Validation with Monte Carlo Numerical Simulations

by
Stiliyan Kalitzin
1,2
1
Stichting Epilepsie Instellingen Nederland (SEIN), Achterweg 5, 2103 SW Heemstede, The Netherlands
2
Image Sciences Institute, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
Algorithms 2024, 17(11), 516; https://doi.org/10.3390/a17110516
Submission received: 14 October 2024 / Revised: 1 November 2024 / Accepted: 5 November 2024 / Published: 8 November 2024
(This article belongs to the Section Algorithms for Multidisciplinary Applications)

Abstract

:
The detection of adverse events—for example, convulsive epileptic seizures—can be critical for patients suffering from a variety of pathological syndromes. Algorithms using remote sensing modalities, such as a video camera input, can be effective for real-time alerting, but the broad variability of environments and numerous nonstationary factors may limit their precision. In this work, we address the issue of adaptive reinforcement that can provide flexible applications in alerting devices. The general concept of our approach is the topological reinforced adaptive algorithm (TOREADA). Three essential steps—embedding, assessment, and envelope—act iteratively during the operation of the system, thus providing continuous, on-the-fly, reinforced learning. We apply this concept in the case of detecting convulsive epileptic seizures, where three parameters define the decision manifold. Monte Carlo-type simulations validate the effectiveness and robustness of the approach. We show that the adaptive procedure finds the correct detection parameters, providing optimal accuracy from a large variety of initial states. With respect to the separation quality between simulated seizure and normal epochs, the detection reinforcement algorithm is robust within the broad margins of signal-generation scenarios. We conclude that our technique is applicable to a large variety of event detection systems.

Graphical Abstract

1. Introduction

Motor disorders—epileptic seizures with convulsive patterns, for example—can have a debilitating impact on the lives of the affected patients. In cases where therapy has failed, constant safeguarding may be necessary to prevent major complications, injuries, or even death as a result of sudden attacks or seizures [1,2]. This requires around-the-clock monitoring in both dedicated facilities and home situations. Although remote video monitoring is now readily available and incorporated in many care systems, it poses certain challenges when operated by human observers. It is extremely labor-intensive, and operators have to pay uninterrupted attention to multiple screens. In addition, privacy issues may prevent the use of this modality in a variety of cases. Alternative solutions based on attachable devices are also available on the market [3,4], but they also require frequent attention and control for charging, proper positioning, etc. Certain patients, due to the stigmatizing effect of wearing them, can also reject the attachable devices.
All of the above arguments motivated us to develop an automated video-input-based alerting system that is capable of detecting a variety of adverse events such as convulsive motor seizures, falls, and nonobstructive apnea, as well as possible warnings for post-seizure cortical electrographic depression. Such an approach combines the advantages of remote nonobstructive monitoring, a holistic representation of the patient state, and automated privacy-respecting alerting.
A central issue for such a development is the wide variety of clinical conditions, camera parameters, and other environmental factors that can challenge the universality of the algorithms used to provide real-time alerting. To address these challenges, we developed an adaptive reinforcement concept that can account for nonstationary applications and optimize the detection process accordingly.
In an earlier work, we addressed the problem of reducing false-positive detections by restricting the detection to only validated true-positive events. This, however, left open the question of missing events, or false negatives. Here, a solution is proposed that can also expand the detection space and gradually reduce the chance of undetected true events.
To verify the effectiveness of our algorithms, synthetic data that can account for a variety of dynamic situations and nonstationary environments are used.
This paper is organized as follows. In the Materials and Methods (Section 2), the general concept of TOREADA (Topological Reinforcement Adaptive Algorithm) is described. Then, we present its particular application to convulsive seizure alerting. We introduce the concept of dual detection, which allows for the accounting of missed true seizures and expansion of the detection parameters.
A brief introduction to the MC (Monte Carlo) type of simulating input data is included, and its role in the validation of the classifier and adaptive algorithms is explained.
In the Results (Section 3), an array of trials demonstrating the effectiveness of our approach for cases of different initial detector states and different levels of separation between events and background signal controlled by the signal-generation parameters is presented.
In the Discussion (Section 4), we state the main motivations for our concept and the assumptions behind it. We also acknowledge certain limitations of the approach and propose some directions for further extensions.
Finally, our major findings are summarized in the Conclusions (Section 5).

2. Materials and Methods

2.1. General Concept of TOREADA

Our approach is of the type of reinforcement learning that uses constant assessments of the detector performance and optimizes the parameters accordingly. For a general overview and full description of this type of machine learning, we refer to a few of the numerous publications available [5,6,7,8,9,10], and here, we describe our specific methodology. We illustrate the overall process for the proposed adaptive classification of events in Figure 1.
The generic description of our approach is illustrated in Figure 1. The green blocks represent the input and the output of an automated system. The blue block “Decision Manifold” represents a direct (without any adaptive reinforcement) reactive process. The reinforcement learning loop consists of an embedding (brown oval containing “Embedding”) step where the decision criteria are relaxed and define the “Observation Manifold” (blue box) in order to account for a larger variety of input events (the training data). An assessment procedure (brown rectangle containing “Assessment”) that can be either supervised or automated, labels the events as true or false (blue box containing “Labelled Data”). The labeled set of events provides an input for constructing an envelope of decision criteria (brown oval containing “Envelope”) that optimizes the detection of the true events and the rejection of the false ones. This decision envelope forms the reinforced, adapted, new decision manifold. The assumption is that our system detects a certain class of events from an input stream of data. Upon a detection, the system will proceed with a predefined action, like raising an alert or simply logging the event. The detection algorithm is encapsulated as M d , the “Detection Manifold”. The latter can be explicit or implicit (like in the case of trained classifiers), but in all cases, we assume that the event features fall into a certain area, or manifold, that defines them. The adaptive reinforcement involves on-the-move deforming or adapting M d according to the performance of the detector. This process should lead to both decreasing the number of false detections and avoiding undetected events. As our training process includes only events that are detected and not on screening the whole data history, we introduce an embedding procedure:
M d M o M d
This step expands the decision manifold into a larger manifold that may include events missed by the initial detection. The extra events will enter the dynamic training set for the adaptive detector but will not affect directly the output (for example, they will be logged separately and no alerts will be generated).
The assessment of the whole set of events is outside the scope of this work. In an earlier work [11], a clustering-based algorithm for separating the true from the false events was proposed. Alternatively, a trained operator can perform the assessment. Here, we assume that a labeled set of events is available for the next phase of the training process, the construction of the envelope manifold:
M o M d M o
Finally, the reinforcement cycle closes by substituting the initial detection manifold with the envelope from (2):
M d M d
The above procedure repeats with every new event or in blocks of predefined number of events. It is a learning during the operation scheme that does not require precollected datasets.
Equations (1)–(3) are very abstract and do not reveal any specific algorithm of performing the process. In Section 2.2 and Section 2.3, we provide the specific application to the task of alerting for convulsive epileptic seizures.

2.2. Detection of Convulsive Seizures

For a complete description of seizure detection algorithms, we refer to previous work [11] and the references therein. Here, we note that an optical flow technique processes live feed from an optical camera and, subsequently, a set of Gabor filters defines a single time series signal that represents the “epileptic content” in the video sequence. Our focus here is only on the last stage of the detection techniques, the decision of whether in a given time window a convulsive seizure has taken place. We illustrate the decision algorithm in Figure 2.
The following condition defines a time interval preceding any time point t as containing a seizure event:
# { s | s N & ( E t s > T ) } n
Here, E is the epileptic marker as an input signal, T is the selected threshold, N is the retrospective time interval (seconds or sample points), and n N is the selected minimal amount of time (or number of samples) when the epileptic marker is above the threshold. Note that in (4) we allow for the marker to move up and down across the threshold during the observation period as long as the total time above is longer than n .
According to (4), the three parameters T , N , n define a seizure event. It is clear that the higher the threshold, the more conservative will be the detection. Also, the longer the interval N for a given n , the more “liberal” is the detection; more events will fulfil the condition (4). A larger n for a fixed N defines a stronger condition for an event to be detected as a seizure. Finally, a “black-out” period T B is introduced in order to avoid multiple detections of the same event. After a detection, the algorithm skips the next T B samples. In all our detections, we selected a black-out period of 90 samples.
Presented in Section 2.3 are the embedding and the envelope implementations for this case.

2.3. Dual Detection Approach

The enlargement or embedding of the decision space formed by the parameters T , N , n in inequality (4) is postulated by the following substitution:
T , N , n α T , N + 1 , n α T , N 1 , n 1
In the transition (5), the union means that if either of the triple parameters fulfils the condition (4), the event is considered as a training event. The parameter α is the relative threshold decrease, and a reasonable choice is α = 0.9 . This choice is a compromise between taking too many fault positive events into the training set ( α is very small) that will make the validation difficult, or taking it to close to one; this will make the adaptation too slow. The embedding (5) is by no means unique but it provides a minimal extension of the decision manifold that is close to the detection one. Larger embedding may speed up the inclusion process but may also introduce too many training events which will make the validation phase challenging.
To complete the enforcement procedure, the envelope-construction algorithm is also needed. In earlier work [11], this stage is introduced in detail. Here, we recall that after the assessment of the training events, we introduce the cost function:
( T , N ) = T 2 n ( T , N ) N n T , N m i n S | T , N ( n )
In the second equation, the value of the parameter n , the filling number, is derived for each pair of ( T , N ) values as the minimal number of threshold-exceeding epochs, n, over all detected and validated seizure events. This way, we can guarantee that, for any choice of the pair ( T , N ) , all the previously detected true seizure events will be preserved. Therefore, we can postulate the optimized choice of parameters as follows:
T , N , n : T , N = a r g m a x ( C T , N ) ; n = n ( T , N )

2.4. Generating Synthetic Data

Commonly referred to as Monte Carlo simulation, the synthetic data-generating algorithm consists of a random process driven by a known probability distribution over the states of the system [12,13]. Figure 3 introduces our implementation that generates a synthetic epileptic marker according to probabilistic rules with variable predefined parameters.
The stochastic model introduces the following probabilities for the epimarker during seizures and outside seizure periods:
P E > T s s e i z u r e = P s P E > T n n o r m a l = P n
In Equation (8), T s , n are the thresholds for, correspondingly, the ictal and normal periods, and P s , n are the associated probabilities. We also postulate random ictal onset times. The seizure duration is a random number in the interval L / 2 , L , where L is a chosen maximal duration. The generated events are used as a ground truth in order to assess the performance of the detection and reinforcement processes. Figure 4 illustrates a trace of the epimarker from a real seizure as observed by our seizure-detection algorithm (top frame) and from a generated event (bottom frame). The parameters for the latter are { T s , T n , P s ,   P n ,   L } = { 0.8 ,   0.1 ,   0.96 ,   0.05 ,   30 } .
In the results below, a variety of scenarios are considered with both stationary and nonstationary seizure-generation parameters { T s , T n , P s , P n , L}.
To quantify the separation between normal and seizure states, one can use the “confusion” metric between two distributions p n ,   p s defined over the states of any system:
C ( p n ,   p s ) s t a t e s min ( p n ,   p s )
Assuming in our case that T s T n , we obtain from Equation (9)
C = T n min 1 P n , 1 P s +   T s T n min P n , 1 P s + 1 T s min P n , P s

2.5. Simulation Protocol and Performance Quantification

Here, the different seizure-generating scenarios and initial detection parameters T , N , n are introduced.
Unless otherwise stated, the typical stimulation session includes a number of epochs, each of a length of 3600 simulation steps. In each epoch, we randomly generate five seizure events according to the stochastic rules explained in the previous subsection. After each epoch, we detect the seizures with the current detector parameters according to Section 2.2. Based on the data-generation log, only the true positive events are selected to enter the parameter optimization algorithm. Next, a minimum number of 20 retrospectively detected true events is required in order to start performing the dual detection parameter optimization procedure explained in Section 2.3. If the number of events is lower than 20, the current parameters are kept. The update is performed after each new simulation epoch; no more than 60 previously detected true events are taken, in order to account for possible nonstationary conditions.
After each epoch, the detector performance is evaluated by collecting the statistics from the last 8 epochs (40 generated true events) according to the following standard definitions for the sensitivity and specificity:
S e n s T P T P + F N S p e c T P T P + F P
In Equation (11), T P , F P , F N are the true positive, false positive, and false negative (undetected events) detections, correspondingly. The ground truth for the events is in accordance with the simulation log.
All the calculus is performed in Matlab®, version 2024a. The statistical distribution plots are produced by the standard “boxplot” function from the “Statistics and Machine Learning Toolbox” version 24.1.

3. Results

3.1. Static Seizure-Generation Scenarios and Various Initial Detector Parameters

In this set of simulations and detection scenarios, the seizure-generation parameters { T s , T n , P s ,   P n , L} = {0.4, 0.6, 0.1, 0.96, 0.064, 60} are constant. Only the initial detector parameters T , N , n are taken different for the different simulation trials, as given in Table 1, and their evolution during the reinforcement process is registered. The results of the sensitivity and specificity recovery are presented in Figure 5.
To show the asymptotic behavior and stability of the reinforcement dynamics, the boxplots from the distributions of the sensitivity and specificity values taken from the last 50 epochs are presented in Figure 6.
In Figure 7, we show the evolution of the detection parameters.

3.2. Change of Seizure-Generation Parameters

Here, two simulation experiments aimed at testing the limits of the proposed reinforcement paradigm are presented. Both consist of three separate tests, starting with the three different sets of initial detector parameters shown in Table 2.
In the first experiment, we generate seizures with the same maximal duration (60 samples) but with a variety of parameters, T s , T n , P s ,   P n , presented in Table 3.
In Figure 8, the results of this reinforcement simulation experiment are shown. For better inspection of the results, the vertical axes are showing the confusion factor from Equation (10) associated with the content of Table 3.
Figure 9 represents the statistics of the asymptotic behavior of the reinforcement procedure for these tests, taking, again, the last 50 epochs. Statistical distributions of sensitivity (the left column of the plots) and specificity (the right column of the plots) are shown as boxplots starting from the three various initial detector settings (each horizontal line of the plots) given in Table 2.
In the second experiment, the reinforcement paradigm is applied to generated seizures of six different maximal lengths, increasing from 10 to 60 samples with a 10-sample step. In this experiment, the other generation parameters were kept constant { T s , T n , P s ,   P n } = { 0.1 ,   0.9 ,   0.02 ,   0.98 } , corresponding to a low confusion factor of 0.02 for high separation between the simulated normal and seizure states. Results are shown in Figure 10.
Also for this set of data, the sensitivity and specificity asymptotic behavior are shown in the boxplots of Figure 11. Statistical distributions of sensitivity (the left column of the plots) and specificity (the right column of the plots) derived from the last 50 epochs of the simulations of reinforcement procedure are shown as boxplots starting from the three various initial detector settings (each horizontal line of the plots) given in Table 2.

4. Discussion

In this work, a general method for adaptive detection and reinforcement is introduced. We call it TOREADA (from TOpological REinforcement Detection Algorithm) and we explain here, in short, the role of the term “topology”. In addition to the formal description in Section 2, a more intuitive insight might be helpful. Our major assumption, or axiom, is that the events that are to be detected form upon appropriate parameterization of a connected continuous manifold. An ad hoc detector setting, or the detection manifold, may partially cover the “true” event manifold, therefore resulting in false positive (FP) detections and false negative (FN) detections in the detected nonevents and the undetected true events, correspondingly. Therefore, the reinforcement process, or learning paradigm, is associated with deforming the detection manifold so that it can possibly cover as precisely as possible the true event manifold. To do so, two complimentary transformations are introduced: embedding and envelope. The embedding step expands the detection manifold to an “embedding” one that contains the former. All events that fall into the larger, embedding manifold form the training set of events. In this way, if the detector is not initially detecting certain true events, they may still fall into the training set. The second stage constructs an envelope around all classified true positive events. This step will form a “tighter” manifold within the embedding one that will possibly help to reject false positive detections. Finally, the envelope manifold is taken as the new, updated detection manifold. Subsequently, the process is iteratively repeated according to the appropriate schedule.
The two steps of embedding and envelope forming are by far not unique. Taking a too-wide embedding manifold around the current detection one, for example, may speed up the process of reducing the FN events, but at the same time it may introduce too many FP events, which would make the assessment of the true events quite complicated. A very tight envelope may reduce the FP detections but may also counteract the embedding process and, consequently, slow down the reduction in undetected true events.
We also left outside the scope of this work the procedure of validating the detected events. Here, both supervised and automated labeling are possible, using for the latter the clustering algorithm, as proposed, for example, in our previous work [11]. Including this form of automated validation into the simulation tests will be a subject of our future research. Another extension of the current work may involve simulation of the optical flow output signals and not just the resulting epifactor. This provides the possibility of testing our real-world adaptive algorithm, also presented in [11], that also involves the adaptive reinforcement of the frequency interval relevant for the particular seizure-generating system. We remind the reader that our objective here was to concentrate on the steps described by Equations (5) and (6).
Another hypothesis used here is that seizure events are characterized by an epimarker that, during a seizure, takes values from a different distribution than during “normal” periods. More specifically, we assumed that the ictal distribution favors higher values. As this is a highly nonparametric assumption, we implemented the simulation model with a couple of thresholds and couple of probabilities in order to explore various levels of separation between the states. In a more general scenario, the epimarker may define a seizure event not by its value but by some features of its distribution statistics. It can be, for example, the fluctuation rate or, more generally, certain momenta of the distribution. We believe that the topological reinforcement paradigm proposed here can be applicable to those cases by properly parameterizing the embedding and envelope manifolds and algorithms. As for the simulation of data, one can apply a similar technique to the one used here to generate synthetic signals, by selecting two model distributions representing seizures and normal states. This and possibly other generalized implementations of TOREADA will be considered in our future research.
The classic approaches to machine learning, including applications targeting detection of convulsive epileptic states, rely on available big collections of labeled data [14]. The founding belief is that these collections contain the necessary information and that they can account for all the variability of the real-world scenarios. Our concept does not require, although it may use, a prerecorded labeled sets of events. It acquires its training sets along with the operation of the detector and, therefore, is a paradigm of continuous learning from experience. This way, the method is applicable to nonstationary situations and, more generally, can cope with the real-world variability of conditions. Moreover, in contrast to the static data collection paradigm, the TOREADA concept can potentially apply adaptive reinforcement to the data acquisition protocols. It can, for example, adjust spectral ranges, apertures, and other sensor properties in order to achieve optimal performance.
The above topological paradigm also presents certain restrictions. If the events that we are attempting to detect are of several different classes, corresponding to disconnected manifolds, our approach is unlikely to work. A possible solution would be to introduce multiple initial detection manifolds and apply the TOREADA algorithm to each one separately.
As a particular real-world application, we presented here our development of systems for convulsive seizures detection. This explicit example of TOREADA is an operational implementation undergoing field trials in clinical and patient care settings. We use this specific realization, called also the dual detection algorithm, to provide numerical validation and performance assessment of the general concept. In this particular application, the embedding stage consists of decreasing the detection threshold and simultaneously relaxing the duration criteria. There are no absolute rules to determine the boundaries of the embedding in general. The optimal choice may depend on the data variability. It might be interesting to explore a second level of reinforcement and to optimize the embedding parameters according to the rates of false positive versus negative training detections. This possibility may also be addressed in our future investigations.
In this work, instead of collected clinical data, data generated by the Monte Carlo (MC) type of numerical simulations were used. There are several reasons for this. First, validating and exploring the limits of adaptive and machine learning algorithms in clinical conditions may present certain ethical issues. If the alerting system departs from its optimal setting and fails to alert for convulsive events, this may place the patient at risk. Conversely, if an unsuccessful reinforcement trial leads to many false alerts, the caregiving personnel may switch the system off. Second, no amount of real-life examples can provide enough data for all possible scenarios and exceptions that the system may encounter in general. Finally, the use of simulated data avoids the issues of privacy, especially present in the cases of supervised video data validation.
Concerning our results from the trials described in Section 3, we again point out the “asymmetry” between the performance boundaries of our approach with respect to the sensitivity and the specificity. The detector loses specificity earlier than it loses sensitivity from the decrease in seizure/normal separation. The latter can be either due to confusion increase or to seizure length decrease, as seen from Figure 8, Figure 9, Figure 10 and Figure 11. As mentioned earlier, it is a result of the “competition” between the embedding process described by Equation (5) and the envelope described by Equations (6) and (7) in this particular implementation. It is clear that if only the embedding is active, the system can only enlarge the detection manifold, for example, according to Equation (5) it can only decrease the threshold. The opposite process, the envelope making, can counter this and increase the threshold, as indeed seen from the results in Figure 7. Other choices, for example, a more conservative embedding and tighter envelope construction, may reverse the balance, and the system will lose sensitivity while the reinforcement algorithm still achieves high specificity. Adding to this point, the current results are favorable in the view of clinical and care practice. The harm of undetected seizure events may be potentially higher than that of false alarming, without diminishing, of course, the need to reduce the latter as much as possible.

5. Conclusions

As demonstrated in Section 3.1 and visually presented in Figure 5, the proposed reinforcement parameter adaptation successfully converges to a detector with both high sensitivity and specificity performance. This is achieved from a variety of initial settings, some very “liberal”, allowing for many false positive detections, and others very “conservative”, skipping some real events. In all cases, here, we use the same event-generation parameters. The later correspond to a relatively good separation between normal and seizure epochs with a confusion factor of 0.02. From Figure 7, it is clear that for all initial values, the detector parameters reach approximately the same optimal levels due to the reinforcement algorithm. As the process is continuously active, certain fluctuations are present, but the system recovers its target parameters and shows asymptotic stability.
To answer the question of how far one can trust the method with respect to the separation between seizures and normal periods, the experiments reported in Section 3.2 are performed. From the results of the first test, shown in Figure 8 and Figure 9, it is clear that, independently of the initial detection parameters, the reinforcement adaptation provides stable sensitivity for generation parameters corresponding to confusion factors of less than 0.2. For the specificity, the adaptive process fails already for confusion factors greater than 0.1. The conclusion is that for systems with marginal or poor distinction between normal and seizure states, the detector can still adapt for proper detection of the events but will also produce false positive detections.
The final test explores the limits of the seizure event durations so that the approach will still be useful. The results shown in Figure 10 and Figure 11 show that, again, the sensitivity reaches high values independently of the initial detector settings for all maximal event durations ranging from 10 up to 60 simulation steps. The specificity trace, however, is not stable for short seizure events. For short seizures the algorithm will manage to restore an optimal detection sensitivity but at the expense of letting through false positive detections.

Funding

This research was partially funded by “De Christelijke Vereniging voor de Verpleging van Lijders aan Epilepsie”, Program 35401, Remote Detection of Motor Paroxysms (REDEMP).

Data Availability Statement

All data used in this work were synthetically generated by the algorithms described and, upon request, can be made available by the corresponding author.

Conflicts of Interest

The authors declare no conflicts of interest, The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

References

  1. Harden, C.; Tomson, T.; Gloss, D.; Buchhalter, J.; Cross, J.H.; Donner, E.; French, J.A.; Gil-Nagel, A.; Hesdorffer, D.C.; Smithson, W.H.; et al. Practice guideline summary: Sudden unexpected death in epilepsy incidence rates and risk factors: Report of the Guideline Development, Dissemination, and Implementation Subcommittee of the American Academy of Neurology and the American Epilepsy Society. Neurology 2017, 88, 1674–1680. [Google Scholar] [CrossRef] [PubMed]
  2. Lamberts, R.; Thijs, R.; Laffan, A.; Langan, Y.; Sander, J.W. Sudden unexpected death in epilepsy: People with nocturnal seizures may be at highest risk. Epilepsia 2017, 23, 253–260. [Google Scholar] [CrossRef] [PubMed]
  3. Herrera-Fortin, T.; Assi, E.B.; Gagnon, M.-P.; Nguyen, D.K. Seizure detection devices: A survey of needs and preferences of patients and caregivers. Epilepsy Behav. 2021, 114, 107607. [Google Scholar] [CrossRef] [PubMed]
  4. Bruno, E.; Viana, P.F.; Sperling, M.R.; Richardson, M.P. Seizure detection at home: Do devices on the market match the needs of people living with epilepsy and their caregivers? Epilepsia 2020, 61, S11–S24. [Google Scholar] [CrossRef] [PubMed]
  5. Naeem, M.; Rizvi, S.S.; Coronato, A. A Gentle Introduction to Reinforcement Learning and its Application in Different Fields. IEEE Access 2020, 8, 209320–209344. [Google Scholar] [CrossRef]
  6. Kaelbling Leslie, P.; Littman Michael, L.; Moore Andrew, W. Reinforcement Learning: A Survey. J. Artif. Intell. Res. 1996, 4, 237–285. [Google Scholar] [CrossRef]
  7. François-Lavet, V.; Henderson, P.; Islam, R.; Bellemare, M.G.; Pineau, J. An introduction to deep reinforcement learning. Found. Trends Mach. Learn. 2018, 11, 219–354. [Google Scholar] [CrossRef]
  8. Sutton, R.S. Introduction: The challenge of reinforcement learning. In Reinforcement Learning; Springer US: Boston, MA, USA, 1992; pp. 1–3. [Google Scholar]
  9. Wiering, M.A.; Van Otterlo, M. Reinforcement learning. Adapt. Learn. Optim. 2012, 12, 729. [Google Scholar]
  10. Fayaz, S.A.; Jahangeer Sidiq, S.; Zaman, M.; Butt, M.A. Machine learning: An introduction to reinforcement learning. In Machine Learning and Data Science: Fundamentals and Applications; John Wiley & Sons: New York, NY, USA, 2022; pp. 1–22. [Google Scholar]
  11. Kalitzin, S. Adaptive Remote Sensing Paradigm for Real-Time Alerting of Convulsive Epileptic Seizures. Sensors 2023, 23, 968. [Google Scholar] [CrossRef] [PubMed]
  12. Kroese, D.P.; Taimre, T.; Botev, Z.I. Handbook of Monte Carlo Methods; John Wiley & Sons: New York, NY, USA, 2011; p. 772. ISBN 978-0-470-17793-8. [Google Scholar]
  13. Binder, K. Introduction: Theory and “technical” aspects of Monte Carlo simulations. In Monte Carlo Methods in Statistical Physics; Springer: Berlin/Heidelberg, Germany, 1986; pp. 1–45. [Google Scholar]
  14. Yang, Y.; Sarkis, R.A.; El Atrache, R.; Loddenkemper, T.; Meisel, C. Video-based detection of generalized tonic-clonic seizures using deep learning. IEEE J. Biomed. Health Inform. 2021, 25, 2997–3008. [Google Scholar] [CrossRef] [PubMed]
Figure 1. “Category” representation of the TOREADA approach to reinforcement.
Figure 1. “Category” representation of the TOREADA approach to reinforcement.
Algorithms 17 00516 g001
Figure 2. Illustration of the quantities used in the seizure-detection decision algorithm. The vertical axis represents the output of the optical flow and filtering preprocessing. The horizontal axis is the running time. The dashed blue line is the selected threshold and the blue arrow on the top is the selected observation elapsed time, or the detection depth. The gray box represents the preconvulsive seizure onset (most typically the tonic phase) and the yellow area is the observed convulsive, clonic phase.
Figure 2. Illustration of the quantities used in the seizure-detection decision algorithm. The vertical axis represents the output of the optical flow and filtering preprocessing. The horizontal axis is the running time. The dashed blue line is the selected threshold and the blue arrow on the top is the selected observation elapsed time, or the detection depth. The gray box represents the preconvulsive seizure onset (most typically the tonic phase) and the yellow area is the observed convulsive, clonic phase.
Algorithms 17 00516 g002
Figure 3. General scheme for Monte Carlo type of simulating epimarker. The output signal, the synthetic epimarker, is generated according to a probabilistic model that determines the distribution of values at each time point depending on whether the system is in a predefined ictal (seizure) state or in a “normal” state.
Figure 3. General scheme for Monte Carlo type of simulating epimarker. The output signal, the synthetic epimarker, is generated according to a probabilistic model that determines the distribution of values at each time point depending on whether the system is in a predefined ictal (seizure) state or in a “normal” state.
Algorithms 17 00516 g003
Figure 4. Real (top frame) and synthetic (bottom frame) epimarker. In both plots, the epimarker is the blue trace and the vertical axes indicate its values. Note that the negative values on the top frame are due to a conveniently introduced offset as a compensation for white noise background. The horizontal axes on both plots represent the real and simulation time (sample steps), correspondingly. The vertical red line on the top frame represents the beginning of the seizure as detected in real time by the detector. On the bottom frame, the two red lines are at the beginning and the end of the model seizure event; the red star is the detected onset by the algorithm.
Figure 4. Real (top frame) and synthetic (bottom frame) epimarker. In both plots, the epimarker is the blue trace and the vertical axes indicate its values. Note that the negative values on the top frame are due to a conveniently introduced offset as a compensation for white noise background. The horizontal axes on both plots represent the real and simulation time (sample steps), correspondingly. The vertical red line on the top frame represents the beginning of the seizure as detected in real time by the detector. On the bottom frame, the two red lines are at the beginning and the end of the model seizure event; the red star is the detected onset by the algorithm.
Algorithms 17 00516 g004
Figure 5. Results from simulations of the reinforcement procedure starting from various initial detector settings. The left plot is the sensitivity presented in pseudo-color code from the color bar. The vertical axis is the trial number; the corresponding parameters are given in Table 1. The horizontal axis is the epoch number (each epoch of 3600 simulation steps; see Section 2.5) from the start of the simulation experiment. The right plot has the same notations but shows the specificity evolution.
Figure 5. Results from simulations of the reinforcement procedure starting from various initial detector settings. The left plot is the sensitivity presented in pseudo-color code from the color bar. The vertical axis is the trial number; the corresponding parameters are given in Table 1. The horizontal axis is the epoch number (each epoch of 3600 simulation steps; see Section 2.5) from the start of the simulation experiment. The right plot has the same notations but shows the specificity evolution.
Algorithms 17 00516 g005
Figure 6. Statistical distributions of the sensitivity (left plot) and the specificity (right plot) derived from the last 50 epochs of the simulations of the reinforcement procedure starting from various initial detector settings, using the same data as shown in Figure 5. The vertical axes are the corresponding quantities and the horizontal axes represent the number of initial parameter sets, as given in Table 1. The boxes are the 25th–75th percentiles, the red lines are the median values, the whiskers denote the 5th–95th percentile values, and the red crosses are the outliers.
Figure 6. Statistical distributions of the sensitivity (left plot) and the specificity (right plot) derived from the last 50 epochs of the simulations of the reinforcement procedure starting from various initial detector settings, using the same data as shown in Figure 5. The vertical axes are the corresponding quantities and the horizontal axes represent the number of initial parameter sets, as given in Table 1. The boxes are the 25th–75th percentiles, the red lines are the median values, the whiskers denote the 5th–95th percentile values, and the red crosses are the outliers.
Algorithms 17 00516 g006
Figure 7. Traces (black lines) of the detector parameter adaptation from simulations of the reinforcement procedure starting from various initial detector settings given in Table 1. The horizontal axis is the epoch number (each epoch of 3600 simulation steps; see Section 2.5) from the start of the simulation experiment. The labels on the vertical axes denote the corresponding parameter.
Figure 7. Traces (black lines) of the detector parameter adaptation from simulations of the reinforcement procedure starting from various initial detector settings given in Table 1. The horizontal axis is the epoch number (each epoch of 3600 simulation steps; see Section 2.5) from the start of the simulation experiment. The labels on the vertical axes denote the corresponding parameter.
Algorithms 17 00516 g007
Figure 8. The notations are the same as those of Figure 5. Each row of images represents a session with the different initial detection parameters given in Table 2.
Figure 8. The notations are the same as those of Figure 5. Each row of images represents a session with the different initial detection parameters given in Table 2.
Algorithms 17 00516 g008
Figure 9. Statistical distributions of sensitivity (the left column of the plots) and specificity (the right column of the plots) for the same data as shown in Figure 8. Vertical axes represent the quantities; the labels on the horizontal axes indicate the various confusion factors as given in the sixth column of Table 3. The boxplot features are the same as those of Figure 6.
Figure 9. Statistical distributions of sensitivity (the left column of the plots) and specificity (the right column of the plots) for the same data as shown in Figure 8. Vertical axes represent the quantities; the labels on the horizontal axes indicate the various confusion factors as given in the sixth column of Table 3. The boxplot features are the same as those of Figure 6.
Algorithms 17 00516 g009
Figure 10. The same notations as in Figure 6. The vertical axes here represent the simulated maximal seizure duration in simulation steps. The three lines of images correspond to the three different initial detection parameters from Table 2.
Figure 10. The same notations as in Figure 6. The vertical axes here represent the simulated maximal seizure duration in simulation steps. The three lines of images correspond to the three different initial detection parameters from Table 2.
Algorithms 17 00516 g010
Figure 11. Statistical distributions of sensitivity (the left column of the plots) and specificity (the right column of the plots) for the same data as shown in Figure 10. Vertical axes represent the quantities; the labels on the horizontal axes indicate the various maximal seizure lengths. The boxplot features are the same as those of Figure 6.
Figure 11. Statistical distributions of sensitivity (the left column of the plots) and specificity (the right column of the plots) for the same data as shown in Figure 10. Vertical axes represent the quantities; the labels on the horizontal axes indicate the various maximal seizure lengths. The boxplot features are the same as those of Figure 6.
Algorithms 17 00516 g011
Table 1. Initial values of the detector parameters before starting the reinforcement learning. The first column is the trial number, the second: detection threshold, the third: the detection window length; the fourth: the minimal number of values above the threshold for detecting a seizure event. The fifth column is the black-out period, taken the same (90 samples) throughout all simulations.
Table 1. Initial values of the detector parameters before starting the reinforcement learning. The first column is the trial number, the second: detection threshold, the third: the detection window length; the fourth: the minimal number of values above the threshold for detecting a seizure event. The fifth column is the black-out period, taken the same (90 samples) throughout all simulations.
Trial NoThresholdNnBlack-Out
10.110.009.0090.00
20.28.007.0090.00
3 0.36.005.0090.00
4 0.47.006.0090.00
5 0.510.009.0090.00
60.68.007.0090.00
70.76.005.0090.00
80.85.004.0090.00
90.94.003.0090.00
Table 2. Three different initial detection parameters used for the two experiments described in this section. The columns are the same as in Table 1.
Table 2. Three different initial detection parameters used for the two experiments described in this section. The columns are the same as in Table 1.
SessionThresholdNnBlack-Out
10.25390
20.510990
30.8101090
Table 3. Seizure-generation parameters for the first experiment in this section. The first column is the trial number, the second: the threshold of normal state, the third: the threshold of simulated seizure state, the fourth: probability of exceeding the threshold in normal state, the fifth: probability of exceeding the threshold during seizure state. The sixth column is the confusion factor for the corresponding generation parameters as given in Equation (10). In this experiment, the maximal seizure duration is always 60 samples.
Table 3. Seizure-generation parameters for the first experiment in this section. The first column is the trial number, the second: the threshold of normal state, the third: the threshold of simulated seizure state, the fourth: probability of exceeding the threshold in normal state, the fifth: probability of exceeding the threshold during seizure state. The sixth column is the confusion factor for the corresponding generation parameters as given in Equation (10). In this experiment, the maximal seizure duration is always 60 samples.
Trial No T n T s P n P s C
10.10.90.020.980.02
20.40.60.10.960.064
30.30.40.10.90.1
40.20.40.150.750.17
50.40.50.20.70.24
60.50.50.30.60.35
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Kalitzin, S. Topological Reinforcement Adaptive Algorithm (TOREADA) Application to the Alerting of Convulsive Seizures and Validation with Monte Carlo Numerical Simulations. Algorithms 2024, 17, 516. https://doi.org/10.3390/a17110516

AMA Style

Kalitzin S. Topological Reinforcement Adaptive Algorithm (TOREADA) Application to the Alerting of Convulsive Seizures and Validation with Monte Carlo Numerical Simulations. Algorithms. 2024; 17(11):516. https://doi.org/10.3390/a17110516

Chicago/Turabian Style

Kalitzin, Stiliyan. 2024. "Topological Reinforcement Adaptive Algorithm (TOREADA) Application to the Alerting of Convulsive Seizures and Validation with Monte Carlo Numerical Simulations" Algorithms 17, no. 11: 516. https://doi.org/10.3390/a17110516

APA Style

Kalitzin, S. (2024). Topological Reinforcement Adaptive Algorithm (TOREADA) Application to the Alerting of Convulsive Seizures and Validation with Monte Carlo Numerical Simulations. Algorithms, 17(11), 516. https://doi.org/10.3390/a17110516

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop