Dataset Information

Simulating Study Design Choice Effects on Observed Performance of Predictive Patient Monitoring Alarm Algorithms.

ABSTRACT: There are multiple study design choices to be selected in order to perform evaluations of predictive patient monitoring algorithms related to the event and true positive alarm definitions (e.g., how far ahead of the event is a true positive alarm). Often, passively collected patient monitoring datasets from clinical environments are available to perform these types of studies, so that the effects of different study design choices can be simulated to evaluate the robustness of an algorithm to those choices. Here, we simulate the effects of varying alarm and event definition criteria on the reported performance of the early warning score to predict hypotensive events. A total of 432 combinations of study design choices were simulated. Area under the receiver-operating characteristic curve varied from greater than 0.8 to less than 0.5 by adjusting alarm and event definition criteria. Traditional metrics for evaluating diagnostic systems were modulated across a wide range by adjusting study design choices for a predictive algorithm using a patient monitoring dataset. This highlights the importance of examining study design choices for new predictive patient monitoring algorithms and presents an approach to simulate different study designs with retrospective patient monitoring data as part of a robustness evaluation.

SUBMITTER: Nahmias DO

PROVIDER: S-EPMC8392319 | biostudies-literature | 2021 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Simulating Study Design Choice Effects on Observed Performance of Predictive Patient Monitoring Alarm Algorithms.

Nahmias David O DO Scully Christopher G CG

... IEEE-EMBS International Conference on Biomedical and Health Informatics. IEEE-EMBS International Conference on Biomedical and Health Informatics 20210701

There are multiple study design choices to be selected in order to perform evaluations of predictive patient monitoring algorithms related to the event and true positive alarm definitions (e.g., how far ahead of the event is a true positive alarm). Often, passively collected patient monitoring datasets from clinical environments are available to perform these types of studies, so that the effects of different study design choices can be simulated to evaluate the robustness of an algorithm to tho ...[more]

PMID: 34458854

Similar Datasets

Project description:We elicited patient experiences from clinical trial simulations to aid in future trial development and to improve patient recruitment and retention.Two simulations of draft Phase II and Phase III anifrolumab studies for systemic lupus erythematosus (SLE)/lupus nephritis (LN) were performed involving African-American patients from Grady Hospital, an indigent care hospital in Atlanta, GA, USA, and white patients from Altoona Arthritis and Osteoporosis Center in Altoona, PA, USA. The clinical trial simulation included an informed consent procedure, a mock screening visit, a mock dosing visit, and a debriefing period for patients and staff. Patients and staff were interviewed to obtain sentiments and perceptions related to the simulated visits.The Atlanta study involved 6 African-American patients (5 female) aged 27-60 years with moderate to severe SLE/LN. The Altoona study involved 12 white females aged 32-75 years with mild to moderate SLE/LN. Patient experiences had an impact on four patient-centric care domains: 1) information, communication, and education; 2) responsiveness to needs; 3) access to care; and 4) coordination of care; and continuity and transition. Patients in both studies desired background material, knowledgeable staff, family and friend support, personal results, comfortable settings, shorter wait times, and greater scheduling flexibility. Compared with the Altoona study patients, Atlanta study patients reported greater preferences for information from the Internet, need for strong community and online support, difficulties in discussing SLE, emphasis on transportation and child care help during the visits, and concerns related to financial matters; and they placed greater importance on time commitment, understanding of potential personal benefit, trust, and confidentiality of patient data as factors for participation. Using these results, we present recommendations to improve study procedures to increase retention, recruitment, and compliance for clinical trials.Insights from these two studies can be applied to the development and implementation of future clinical trials to improve patient recruitment, retention, compliance, and advocacy.

Project description:Monitoring one's performance helps detect errors and adapt to prevent future mistakes. However, elevated performance monitoring is associated with increased checking behaviors and perfectionism and is characteristic of multiple psychiatric disorders. Understanding how heightened performance monitoring in early childhood relates to subsequent brain connectivity may elucidate mechanistic risk factors that influence brain and psychiatric outcomes. The aim of this study was to examine the association between performance monitoring in preschool-aged children and functional connectivity during adolescence. In the current prospective longitudinal study, we performed seed-based functional connectivity analysis using a dorsal anterior cingulate cortex (dACC) seed to assess brain-behavior relationships between observationally coded performance monitoring in preschool-aged children and adolescent functional connectivity (n = 79). We also utilized enrichment analysis to investigate network-level connectome-wide associations. Seed-based analysis revealed negative correlations between preschool performance monitoring and adolescent fc between dACC and orbitofrontal and dorsolateral prefrontal cortex while a positive correlation was observed between dACC-occipital cortex connectivity. Enrichment analysis revealed a negative correlation between preschool performance monitoring and connectivity between motor (MOT) - cingulo-opercular (CO) and salience (SN) - Reward (REW) and a positive correlation with MOT-DMN, and cerebellum (CB) - motor connectivity. Elevated performance monitoring in early childhood is associated with functional connectivity during adolescence in regions and networks associated with cognitive control, sensorimotor processing and cortico-striatal-thalamic-cortico (CTSC) aberrations. These regions and networks are implicated in psychiatric disorders characterized by elevated performance monitoring. Findings shed light on a mechanistic risk factor in early childhood with long-term associations with neural functioning.

Project description:ObjectivesTo determine whether an innovative graphical tool for accurate measurement of individual surgeon performance metrics, adjusted for both surgeon-specific and patient-specific factors, significantly alters interpretation of performance data.DesignRetrospective analysis of all total knee replacements (TKRs) conducted at the host institution between 1996 and 2009. The database was randomly divided into training and testing datasets. Using multivariate generalised estimating equation regression models, the training dataset enabled generation of patient-risk and surgeon-experience adjustment factors. To simulate prospective monitoring of individual surgeon outcomes, the testing dataset was mapped on control charts. Weighted ? statistics were calculated to measure the agreement between patient-risk adjusted and fully adjusted control charts.SettingTertiary care academic hospital.ParticipantsAll patients undergoing TKR at the host institution 1996-2009.Main outcome measureOperative efficiency.Results5313 procedures were analysed. Adjusted control charts were generated using a training dataset comprising 3756 procedures performed by 13 surgeons. The operative time gradually declined by 121 min with 25 years of experience (p<0.0001). Charts were tested by monitoring four other surgeons, performing an average of 389 procedures each. Adjustment for surgeon experience significantly altered the interpretation of operative efficiency (?=0.29 (95% CI 0.11 to 0.47)), and enhanced assessment of a surgeon's improvement or diminishment in efficiency over time. Specifically, experience adjustment inverted the interpretation of surgeon efficiency from above average to below average, or from improving to declining performance.ConclusionsAdjustment for surgeon experience is necessary for accurate interpretation of metrics over the course of a surgeon's career. Patient-adjusted and surgeon-adjusted control charts provide an accurate method of monitoring individual operative efficiency.

Dataset Information

Simulating Study Design Choice Effects on Observed Performance of Predictive Patient Monitoring Alarm Algorithms.

Publications

Simulating Study Design Choice Effects on Observed Performance of Predictive Patient Monitoring Alarm Algorithms.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets