Dataset Information

Using set visualisation to find and explain patterns of missing values: a case study with NHS hospital episode statistics data.

ABSTRACT:

Objectives

Missing data is the most common data quality issue in electronic health records (EHRs). Missing data checks implemented in common analytical software are typically limited to counting the number of missing values in individual fields, but researchers and organisations also need to understand multifield missing data patterns to better inform advanced missing data strategies for which counts or numerical summaries are poorly suited. This study shows how set-based visualisation enables multifield missing data patterns to be discovered and investigated.

Design

Development and evaluation of interactive set visualisation techniques to find patterns of missing data and generate actionable insights. The visualisations comprised easily interpretable bar charts for sets, heatmaps for set intersections and histograms for distributions of both sets and intersections.

Setting and participants

Anonymised admitted patient care health records for National Health Service (NHS) hospitals and independent sector providers in England. The visualisation and data mining software was run over 16 million records and 86 fields in the dataset.

Results

The dataset contained 960 million missing values. Set visualisation bar charts showed how those values were distributed across the fields, including several fields that, unexpectedly, were not complete. Set intersection heatmaps revealed unexpected gaps in diagnosis, operation and date fields because diagnosis and operation fields were not filled up sequentially and some operations did not have corresponding dates. Information gain ratio and entropy calculations allowed us to identify the origin of each unexpected pattern, in terms of the values of other fields.

Conclusions

Our findings show how set visualisation reveals important insights about multifield missing data patterns in large EHR datasets. The study revealed both rare and widespread data quality issues that were previously unknown, and allowed a particular part of a specific hospital to be pinpointed as the origin of rare issues that NHS Digital did not know exist.

SUBMITTER: Ruddle RA

PROVIDER: S-EPMC9680176 | biostudies-literature | 2022 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Using set visualisation to find and explain patterns of missing values: a case study with NHS hospital episode statistics data.

Ruddle Roy A RA Adnan Muhammad M Hall Marlous M

BMJ open 20221121 11

<h4>Objectives</h4>Missing data is the most common data quality issue in electronic health records (EHRs). Missing data checks implemented in common analytical software are typically limited to counting the number of missing values in individual fields, but researchers and organisations also need to understand multifield missing data patterns to better inform advanced missing data strategies for which counts or numerical summaries are poorly suited. This study shows how set-based visualisation e ...[more]

PMID: 36410820

Similar Datasets

Project description:ObjectiveIt is typical in epidemiological research of osteoarthritis (OA) to collect data for the hand, hip, and knee. However, little population-based data exist for this disease in the foot. Thus, we addressed patterns of OA for the foot compared with the hand, hip, and knee spanning 2000/2001 to 2017/2018 in England.MethodsSecondary-care data from 3 143 928 patients with OA of the foot, hand, hip, and knee were derived from the National Health Service (NHS) Hospital Episode Statistics (HES) database. Distribution, population prevalence, and incidence of joint-specific OA were stratified by age and sex.ResultsOA incidence increased significantly at the foot [3.8% (95% confidence interval [CI] 3.0, 4.6)], hand [10.9% (10.1, 11.7)], hip [3.8% (2.9, 4.7)], and knee [2.9% (2.2, 3.6)] per year from 2000/2001 to 2017/2018. A higher proportion of women were diagnosed with OA, whereas greater incidence in men was estimated for the hand and hip. Foot OA presented comparable diagnosis numbers to the hand. More recently during 2012/2013 to 2017/2018, a significant rise in hip OA was estimated among younger adults, whereas knee OA decreased across all age groups. Incidence of OA in the foot and hand were particularly significant among the 75 or older age group, though bimodal age distributions were observed for both sites.ConclusionThe significant increase in secondary care records for OA in England underscores the importance of exploring possible causative factors and identifying groups most at risk. Further detailed data may be particularly important for the hip, which represents significant incidence among younger adults. Greater incidence of OA in the foot compared with the knee emphasizes the need for well-conducted epidemiological research in this area. Monitoring the performance of surgical outcomes at the population-level for this frequently affected yet understudied site could have substantial potential to reduce the socioeconomic burden it represents to the NHS.

Project description:BackgroundIn England emergency hospital admissions among children are increasing and the under 25s are the most frequent attenders of A&E departments. Children of lower socio-economic status (SES) have poorer health outcomes and higher hospital admission rates. NHS Hospital Episode Statistics (HES) are increasingly being used for research but lack detailed data on individual characteristics such as SES. We report the results of an Avon Longitudinal Study of Parents and Children (ALSPAC) study that linked the data of 3,189 consenting participants to HES. We describe rates of hospital admission, emergency readmissions, and A&E attendances and examine socio-demographic correlates of these.MethodsSubjects were singletons and twins enrolled in ALSPAC who had provided consent for linkage to their health records by the study cut-off date (31.02.12). Linkage was carried out by the Health and Social Care Information Centre (now NHS Digital). We examined rates of admissions between birth and age 20 and A&E attendances between 14 and 20 years. Socio-demographic information collected in ALSPAC questionnaires during pregnancy were used to examine factors associated with admissions, emergency readmissions (an emergency admission within 30 days of discharge) and A&E attendances.ResultsExcluding birth records, we found at least one admission for 1,792/3,189 (56.2%) participants and 4,305 admissions in total. Admission rates were highest in the first year of life. Among males, admissions declined until about age 5 and then remained relatively stable; conversely, among females, they increased sharply from the age of 15. ICD 10 chapters for diseases of the digestive system and injury and poisoning accounted for the largest proportions of admissions (15.8 and 14.5%, respectively). Tooth decay was the highest single cause of admission for those aged 5-9 years. Overall, 1,518/3,189 (47.6%) of participants attended A&E at least once, with a total of 3,613 attendances between age 14 and 20 years. Individuals from more deprived backgrounds had higher rates of admissions, readmissions and A&E attendances.ConclusionsLinkage between cohort studies such as ALSPAC and HES data provides unique opportunities for detailed insights into socio-demographic and other determinants of hospital activity, which can inform secondary care demand management in the NHS.

Dataset Information

Using set visualisation to find and explain patterns of missing values: a case study with NHS hospital episode statistics data.

Objectives

Design

Setting and participants

Results

Conclusions

Publications

Using set visualisation to find and explain patterns of missing values: a case study with NHS hospital episode statistics data.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets