Dataset Information

Normal Workflow and Key Strategies for Data Cleaning Toward Real-World Data: Viewpoint.

ABSTRACT: With the rapid development of science, technology, and engineering, large amounts of data have been generated in many fields in the past 20 years. In the process of medical research, data are constantly generated, and large amounts of real-world data form a "data disaster." Effective data analysis and mining are based on data availability and high data quality. The premise of high data quality is the need to clean the data. Data cleaning is the process of detecting and correcting "dirty data," which is the basis of data analysis and management. Moreover, data cleaning is a common technology for improving data quality. However, the current literature on real-world research provides little guidance on how to efficiently and ethically set up and perform data cleaning. To address this issue, we proposed a data cleaning framework for real-world research, focusing on the 3 most common types of dirty data (duplicate, missing, and outlier data), and a normal workflow for data cleaning to serve as a reference for the application of such technologies in future studies. We also provided relevant suggestions for common problems in data cleaning.

SUBMITTER: Guo M

PROVIDER: S-EPMC10557005 | biostudies-literature | 2023 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Normal Workflow and Key Strategies for Data Cleaning Toward Real-World Data: Viewpoint.

Guo Manping M Wang Yiming Y Yang Qiaoning Q Li Rui R Zhao Yang Y Li Chenfei C Zhu Mingbo M Cui Yao Y Jiang Xin X Sheng Song S Li Qingna Q Gao Rui R

Interactive journal of medical research 20230921

With the rapid development of science, technology, and engineering, large amounts of data have been generated in many fields in the past 20 years. In the process of medical research, data are constantly generated, and large amounts of real-world data form a "data disaster." Effective data analysis and mining are based on data availability and high data quality. The premise of high data quality is the need to clean the data. Data cleaning is the process of detecting and correcting "dirty data," w ...[more]

PMID: 37733421

Similar Datasets

Project description:PurposeHealthcare utilization databases often lack information on glycemic control, a key confounder when studying the safety of antidiabetic treatments, since patients with worse control are channeled to second-line agents, in particular insulin, versus first-line agents such as metformin. We evaluated whether adjustment for measured characteristics attains balance in glycemic control when comparing antidiabetic treatment strategies in pregnant women with pregestational type 2 diabetes (T2DM).MethodsIn a US insurance claims database, we identified 3360 women with T2DM pregnant between 2004 and 2015, of whom a subset of 996 had data on hemoglobin A1c (HbA1c ) levels. We selected insulin only as the comparator group and used propensity score (PS)-matching on comorbidities and proxies of diabetes severity, but not on HbA1c , to adjust for confounding. We used standardized differences (st.diff) to assess balance in claims-based covariates and mean HbA1c (% ± SD) in the subset.ResultsThere were imbalances in claims-based covariates before PS-matching, with smaller differences when both treatment strategies included insulin. After PS-matching, balance was achieved in most claims-based covariates (st.diff <0.1). Mean HbA1c was similar before and after PS-matching when both treatments included insulin (e.g., 7.1 ± 1.5 vs. 7.7 ± 1.8 and 7.1 ± 1.5 vs. 7.5 ± 1.7, respectively, for metformin + insulin vs. insulin only). Differences in mean HbA1c remained after PS-matching when non-insulin treatments were compared to treatments including insulin (e.g., 6.3 ± 1.1 vs. 7.6 ± 1.7 for metformin only vs. insulin only).ConclusionsBalance in both claims-based characteristics and glycemic control was attained after restricting the population to women with T2DM and comparing treatment strategies indicated for patients with similar diabetes severity. When comparing treatment strategies with versus without insulin, differences in glycemic control persisted after PS-matching even when balance was attained for other measured characteristics.

Project description:BackgroundCoronary heart disease (CHD) has become the leading cause of death and one of the most serious epidemic diseases worldwide. CHD is characterized by urgency, danger and severity, and dynamic treatment strategies for CHD patients are needed. We aimed to build and validate an AI model for dynamic treatment recommendations for CHD patients with the goal of improving patient outcomes and learning best practices from clinicians to help clinical decision support for treating CHD patients.MethodsWe formed the treatment strategy as a sequential decision problem, and applied an AI supervised reinforcement learning-long short-term memory (SRL-LSTM) framework that combined supervised learning (SL) and reinforcement learning (RL) with an LSTM network to track patients' states to learn a recommendation model that took a patient's diagnosis and evolving health status as input and provided a treatment recommendation in the form of whether to take specific drugs. The experiments were conducted by leveraging a real-world intensive care unit (ICU) database with 13,762 admitted patients diagnosed with CHD. We compared the performance of the applied SRL-LSTM model and several state-of-the-art SL and RL models in reducing the estimated in-hospital mortality and the Jaccard similarity with clinicians' decisions. We used a random forest algorithm to calculate the feature importance of both the clinician policy and the AI policy to illustrate the interpretability of the AI model.ResultsOur experimental study demonstrated that the AI model could help reduce the estimated in-hospital mortality through its RL function and learn the best practice from clinicians through its SL function. The similarity between the clinician policy and the AI policy regarding the surviving patients was high, while for the expired patients, it was much lower. The dynamic treatment strategies made by the AI model were clinically interpretable and relied on sensible clinical features extracted according to monitoring indexes and risk factors for CHD patients.ConclusionsWe proposed a pipeline for constructing an AI model to learn dynamic treatment strategies for CHD patients that could improve patient outcomes and mimic the best practices of clinicians. And a lot of further studies and efforts are needed to make it practical.

Dataset Information

Normal Workflow and Key Strategies for Data Cleaning Toward Real-World Data: Viewpoint.

Publications

Normal Workflow and Key Strategies for Data Cleaning Toward Real-World Data: Viewpoint.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets