Prospective validation of a seizure diary forecasting falls short.
Ontology highlight
ABSTRACT: Objective
Recently, a deep learning AI model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm.Methods
We recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSS) compared random forecasts and simple moving average forecasts to the AI.Results
The AI had an AUC of 0.82. At the group level, the AI outperformed random forecasting (BSS=0.53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre-enrollment (non-verified) diaries (with presumed under-reporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor quality LOW-RISK or HIGH-RISK forecasts, yet 91% wanted access to these forecasts.Significance
The previously developed AI forecasting tool did not outperform a very simple moving average forecasting this prospective cohort, suggesting that the AI model should be replaced.
SUBMITTER: Goldenholz DM
PROVIDER: S-EPMC10802655 | biostudies-literature | 2024 Jan
REPOSITORIES: biostudies-literature
Publications
Prospective validation of a seizure diary forecasting falls short.
Goldenholz Daniel M DM Eccleston Celena C Moss Robert R Westover M Brandon MB
medRxiv : the preprint server for health sciences 20240113
<h4>Objective</h4>Recently, a deep learning AI model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm.<h4>Methods</h4>We recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSS) compared random fore ...[more]
Similar Datasets
Project description:BackgroundPeople with epilepsy are burdened with the apparent unpredictability of seizures. In the past decade, converging evidence from studies using chronic EEG (cEEG) revealed that epileptic brain activity shows robust cycles, operating over hours (circadian) and days (multidien). We hypothesised that these cycles can be leveraged to estimate future seizure probability, and we tested the feasibility of forecasting seizures days in advance.MethodsWe did a feasibility study in distinct development and validation cohorts, involving retrospective analysis of cEEG data recorded with an implanted device in adults (age ≥18 years) with drug-resistant focal epilepsy followed at 35 centres across the USA between Jan 19, 2004, and May 18, 2018. Patients were required to have had 20 or more electrographic seizures (development cohort) or self-reported seizures (validation cohort). In all patients, the device recorded interictal epileptiform activity (IEA; ≥6 months of continuous hourly data), the fluctuations in which helped estimate varying seizure risk. Point process statistical models trained on initial portions of each patient's cEEG data (both cohorts) generated forecasts of seizure probability that were tested on subsequent unseen seizure data and evaluated against surrogate time-series. The primary outcome was the percentage of patients with forecasts showing improvement over chance (IoC).FindingsWe screened 72 and 256 patients, and included 18 and 157 patients in the development and validation cohorts, respectively. Models incorporating information about multidien IEA cycles alone generated daily seizure forecasts for the next calendar day with IoC in 15 (83%) patients in the development cohort and 103 (66%) patients in the validation cohort. The forecasting horizon could be extended up to 3 days while maintaining IoC in two (11%) of 18 patients and 61 (39%) of 157 patients. Forecasts with a shorter horizon of 1 h, possible only for electrographic seizures in the development cohort, showed IoC in all 18 (100%) patients.InterpretationThis study shows that seizure probability can be forecasted days in advance by leveraging multidien IEA cycles recorded with an implanted device. This study will serve as a basis for prospective clinical trials to establish how people with epilepsy might benefit from seizure forecasting over long horizons.FundingNone. VIDEO ABSTRACT.
| S-EPMC7968722 | biostudies-literature
Project description:A subset of patients with epilepsy successfully self-predicted seizures in a paper diary study. We conducted an e-diary study to ensure that prediction precedes seizures, and to characterize the prodromal features and time windows that underlie self-prediction.Subjects 18 or older with localization-related epilepsy (LRE) and ?3 seizures per month maintained an e-diary, reporting a.m./p.m. data daily, including mood, premonitory symptoms, and all seizures. Self-prediction was rated by, "How likely are you to experience a seizure (time frame)?" Five choices ranged from almost certain (>95% chance) to very unlikely. Relative odds of seizure (odds ratio, OR) within time frames was examined using Poisson models with log normal random effects to adjust for multiple observations.Nineteen subjects reported 244 eligible seizures. OR for prediction choices within 6 h was as high as 9.31 (CI 1.92-45.23) for "almost certain." Prediction was most robust within 6 h of diary entry, and remained significant up to 12 h. For nine best predictors, average sensitivity was 50%. Older age contributed to successful self-prediction, and self-prediction appeared to be driven by mood and premonitory symptoms. In multivariate modeling of seizure occurrence, self-prediction (2.84; CI 1.68-4.81), favorable change in mood (0.82; CI 0.67-0.99), and number of premonitory symptoms (1.11; CI 1.00-1.24) were significant.Some persons with epilepsy can self-predict seizures. In these individuals, the odds of a seizure following a positive prediction are high. Predictions were robust, not attributable to recall bias, and were related to self-awareness of mood and premonitory features. The 6-h prediction window is suitable for the development of preemptive therapy.
| S-EPMC3833277 | biostudies-literature
Project description:Objectives.People with refractory epilepsy are overwhelmed by the uncertainty of their next seizures. Accurate prediction of future seizures could greatly improve the quality of life for these patients. New evidence suggests that seizure occurrences can have cyclical patterns for some patients. Even though these cyclicalities are not intuitive, they can be identified by machine learning (ML), to identify patients with predictable vs unpredictable seizure patterns.Approach.Self-reported seizure logs of 153 patients from the Human Epilepsy Project with more than three reported seizures (totaling 8337 seizures) were used to obtain inter-seizure interval time-series for training and evaluation of the forecasting models. Two classes of prediction methods were studied: (1) statistical approaches using Bayesian fusion of population-wise and individual-wise seizure patterns; and (2) ML-based algorithms including least squares, least absolute shrinkage and selection operator, support vector machine (SVM) regression, and long short-term memory regression. Leave-one-person-out cross-validation was used for training and evaluation, by training on seizure diaries of all except one subject and testing on the left-out subject.Main results.The leading forecasting models were the SVM regression and a statistical model that combined the median of population-wise seizure time-intervals with a test subject's prior seizure intervals. SVM was able to forecast 50%, 70%, 81%, 84%, and 87% of seizures of unseen subjects within 0, 1, 2, 3 to 4 d of mean absolute forecasting error, respectively. The subject-wise performances show that patients with more frequent seizures were generally better predicted.Significance.ML models can leverage non-random patterns within self-reported seizure diaries to forecast future seizures. While diary-based seizure forecasting alone is only one of many aspects of clinical care of patients with epilepsy, studying the level of predictability across seizures and patients paves the path towards a better understanding of predictable vs unpredictable seizures on individualized and population-wise bases.
| S-EPMC9940727 | biostudies-literature
Project description:The unpredictability of epileptic seizures exposes people with epilepsy to potential physical harm, restricts day-to-day activities, and impacts mental well-being. Accurate seizure forecasters would reduce the uncertainty associated with seizures but need to be feasible and accessible in the long-term. Wearable devices are perfect candidates to develop non-invasive, accessible forecasts but are yet to be investigated in long-term studies. We hypothesized that machine learning models could utilize heart rate as a biomarker for well-established cycles of seizures and epileptic activity, in addition to other wearable signals, to forecast high and low risk seizure periods. This feasibility study tracked participants' (n = 11) heart rates, sleep, and step counts using wearable smartwatches and seizure occurrence using smartphone seizure diaries for at least 6 months (mean = 14.6 months, SD = 3.8 months). Eligible participants had a diagnosis of refractory epilepsy and reported at least 20 seizures (mean = 135, SD = 123) during the recording period. An ensembled machine learning and neural network model estimated seizure risk either daily or hourly, with retraining occurring on a weekly basis as additional data was collected. Performance was evaluated retrospectively against a rate-matched random forecast using the area under the receiver operating curve. A pseudo-prospective evaluation was also conducted on a held-out dataset. Of the 11 participants, seizures were predicted above chance in all (100%) participants using an hourly forecast and in ten (91%) participants using a daily forecast. The average time spent in high risk (prediction time) before a seizure occurred was 37 min in the hourly forecast and 3 days in the daily forecast. Cyclic features added the most predictive value to the forecasts, particularly circadian and multiday heart rate cycles. Wearable devices can be used to produce patient-specific seizure forecasts, particularly when biomarkers of seizure and epileptic activity cycles are utilized.
| S-EPMC8320020 | biostudies-literature
Project description:Accurate seizure forecasting is emerging as a near-term possibility due to recent advancements in machine learning and EEG technology improvements. Large-scale data curation and new data element collection through consumer wearables and digital health tools such as longitudinal seizure diary data has uncovered new possibilities for personalized algorithm development that may be used to predict the likelihood of future seizures. The Epilepsy Foundation recognized the unmet need for development in seizure forecasting following a 2016 survey where an overwhelming majority of respondents across all seizure types and frequencies reported that unpredictability of seizures had the strongest impact on their life while living with or caring for someone living with epilepsy. In early 2021, the Epilepsy Foundation conducted an updated survey among those living with epilepsies and/or their caregivers to better understand the use-cases that best suit the needs of our community as seizure forecast research advances. These results will provide researchers with insight into user-acceptance of using a forecasting tool and incorporation into their daily life. Ultimately, this input from people living with epilepsy and caregivers will provide timely feedback on what the community needs are and ensure researchers and companies first and foremost consider these needs in seizure forecasting tools/product development.
| S-EPMC8488220 | biostudies-literature
Project description:PurposeData about maternal recall accuracy for classifying early pregnancy medication exposure are meager. Nonetheless, studies often rely on recall to evaluate potential impact of pharmaceuticals on the developing fetus.MethodsRight from the Start is a community-based pregnancy cohort that enrolled women from North Carolina, Tennessee, and Texas. A subset of 318 women participated in daily medication diaries initiated before conception (2006-2012). We examined nonsteroidal anti-inflammatory drugs (NSAIDs) as an example of a drug type that is difficult to study due to its intermittent and primarily over-the-counter use as well as its incomplete documentation in medical and pharmaceutical records. Selective serotonin reuptake inhibitors (SSRI) were assessed as a prescription medication comparator. Maternal recall of NSAID and SSRI use in early pregnancy was examined by comparing diary data (gold standard) to first-trimester interview.ResultsSensitivity and specificity for recall of NSAID exposure were 78.6% and 62.3%, respectively (kappa statistic: 0.41), with 72.3% agreement for exposure classification. Sensitivity and specificity for recall of SSRI exposure were 77.8% and 99.0%, respectively (kappa statistic: 0.79), with 97.8% agreement.ConclusionsOur findings suggest the validity of maternal recall varies with medication type and prospective data collection should be prioritized when studying early pregnancy drug exposures.
| S-EPMC5685497 | biostudies-literature
Project description:BackgroundSeizure risk forecasting could reduce injuries and even deaths in people with epilepsy. There is great interest in using non-invasive wearable devices to generate forecasts of seizure risk. Forecasts based on cycles of epileptic activity, seizure times or heart rate have provided promising forecasting results. This study validates a forecasting method using multimodal cycles recorded from wearable devices.MethodSeizure and heart rate cycles were extracted from 13 participants. The mean period of heart rate data from a smartwatch was 562 days, with a mean of 125 self-reported seizures from a smartphone app. The relationship between seizure onset time and phases of seizure and heart rate cycles was investigated. An additive regression model was used to project heart rate cycles. The results of forecasts using seizure cycles, heart rate cycles, and a combination of both were compared. Forecasting performance was evaluated in 6 of 13 participants in a prospective setting, using long-term data collected after algorithms were developed.FindingsThe results showed that the best forecasts achieved a mean area under the receiver-operating characteristic curve (AUC) of 0.73 for 9/13 participants showing performance above chance during retrospective validation. Subject-specific forecasts evaluated with prospective data showed a mean AUC of 0.77 with 4/6 participants showing performance above chance.InterpretationThe results of this study demonstrate that cycles detected from multimodal data can be combined within a single, scalable seizure risk forecasting algorithm to provide robust performance. The presented forecasting method enabled seizure risk to be estimated for an arbitrary future period and could be generalised across a range of data types. In contrast to earlier work, the current study evaluated forecasts prospectively, in subjects blinded to their seizure risk outputs, representing a critical step towards clinical applications.FundingThis study was funded by an Australian Government National Health & Medical Research Council and BioMedTech Horizons grant. The study also received support from the Epilepsy Foundation of America's 'My Seizure Gauge' grant.
| S-EPMC10300292 | biostudies-literature
Project description:People with epilepsy frequently under- or inaccurately report their seizures, which poses a challenge for evaluating their treatment. The introduction of epilepsy health apps provides a novel approach that could improve seizure documentation. This study assessed the documentation performance of an app-based seizure diary and a conventional paper seizure diary. At two tertiary epilepsy centers patients were asked to use one of two offered methods to report their seizures (paper or app diary) during their stay in the epilepsy monitoring unit. The performances of both methods were assessed based on the gold standard of video-EEG annotations. In total 89 adults (54 paper and 35 app users) with focal epilepsy were included in the analysis, of which 58 (33 paper and 25 app users) experienced at least one seizure and made at least one seizure diary entry. We observed a high precision of 85.7% for the app group, whereas the paper group's precision was lower due to overreporting (66.9%). Sensitivity was similar for both methods. Our findings imply that performance of seizure self-reporting is patient-dependent but is more precise for patients who are willing to use digital apps. This may be relevant for treatment decisions and future clinical trial design.
| S-EPMC11233562 | biostudies-literature
Project description:ObjectiveWe conducted clinical testing of an automated Bayesian machine learning algorithm (Epilepsy Seizure Assessment Tool [EpiSAT]) for outpatient seizure risk assessment using seizure counting data, and validated performance against specialized epilepsy clinician experts.MethodsWe conducted a prospective longitudinal study of EpiSAT performance against 24 specialized clinician experts at three tertiary referral epilepsy centers in the United States. Accuracy, interrater reliability, and intra-rater reliability of EpiSAT for correctly identifying changes in seizure risk (improvements, worsening, or no change) were evaluated using 120 seizures from four synthetic seizure diaries (seizure risk known) and 120 seizures from four real seizure diaries (seizure risk unknown). The proportion of observed agreement between EpiSAT and clinicians was evaluated to assess compatibility of EpiSAT with clinical decision patterns by epilepsy experts.ResultsEpiSAT exhibited substantial observed agreement (75.4%) with clinicians for assessing seizure risk. The mean accuracy of epilepsy providers for correctly assessing seizure risk was 74.7%. EpiSAT accurately identified seizure risk in 87.5% of seizure diary entries, corresponding to a significant improvement of 17.4% (P = .002). Clinicians exhibited low-to-moderate interrater reliability for seizure risk assessment (Krippendorff's α = 0.46) with good intrarater reliability across a 4- to 12-week evaluation period (Scott's π = 0.89).SignificanceThese results validate the ability of EpiSAT to yield objective clinical recommendations on seizure risk which follow decision patterns similar to those from specialized epilepsy providers, but with improved accuracy and reproducibility. This algorithm may serve as a useful clinical decision support system for quantitative analysis of clinical seizure frequency in clinical epilepsy practice.
| S-EPMC6980278 | biostudies-literature
Project description:ObjectiveNon-invasive biomarkers have recently shown promise for seizure forecasting in people with epilepsy. In this work, we developed a seizure-day forecasting algorithm based on nocturnal sleep features acquired using a smart shirt.MethodsSeventy-eight individuals with epilepsy admitted to the Centre hospitalier de l'Université de Montréal epilepsy monitoring unit wore the Hexoskin biometric smart shirt during their stay. The shirt continuously measures electrocardiography, respiratory, and accelerometry activity. Ten sleep features, including sleep efficiency, sleep latency, sleep duration, time spent in non-rapid eye movement sleep (NREM) and rapid eye movement sleep (REM), wakefulness after sleep onset, average heart and breathing rates, high-frequency heart rate variability, and the number of position changes, were automatically computed using the Hexoskin sleep algorithm. Each night's features were then normalized using a reference night for each patient. A support vector machine classifier was trained for pseudo-prospective seizure-day forecasting, with forecasting horizons of 16- and 24-h to include both diurnal and nocturnal seizures (24-h) or diurnal seizures only (16-h). The algorithm's performance was assessed using a nested leave-one-patient-out cross-validation approach.ResultsImprovement over chance (IoC) performances were achieved for 48.7% and 40% of patients with the 16- and 24-h forecasting horizons, respectively. For patients with IoC performances, the proposed algorithm reached mean IoC, sensitivity and time in warning of 34.3%, 86.0%, and 51.7%, respectively for the 16-h horizon, and 34.2%, 64.4% and 30.2%, respectively, for the 24-h horizon.SignificanceSmart shirt-based nocturnal sleep analysis holds promise as a non-invasive approach for seizure-day forecasting in a subset of people with epilepsy. Further investigations, particularly in a residential setting with long-term recordings, could pave the way for the development of innovative and practical seizure forecasting devices.Plain language summarySeizure forecasting with wearable devices may improve the quality of life of people living with epilepsy who experience unpredictable, recurrent seizures. In this study, we have developed a seizure forecasting algorithm using sleep characteristics obtained from a smart shirt worn at night by a large number of hospitalized patients with epilepsy (78). A daily seizure forecast was generated following each night using machine learning methods. Our results show that around half of people with epilepsy may benefit from such an approach.
| S-EPMC11450616 | biostudies-literature