Dataset Information

A workflow reproducibility scale for automatic validation of biological interpretation results.

ABSTRACT:

Background

Reproducibility of data analysis workflow is a key issue in the field of bioinformatics. Recent computing technologies, such as virtualization, have made it possible to reproduce workflow execution with ease. However, the reproducibility of results is not well discussed; that is, there is no standard way to verify whether the biological interpretation of reproduced results is the same. Therefore, it still remains a challenge to automatically evaluate the reproducibility of results.

Results

We propose a new metric, a reproducibility scale of workflow execution results, to evaluate the reproducibility of results. This metric is based on the idea of evaluating the reproducibility of results using biological feature values (e.g., number of reads, mapping rate, and variant frequency) representing their biological interpretation. We also implemented a prototype system that automatically evaluates the reproducibility of results using the proposed metric. To demonstrate our approach, we conducted an experiment using workflows used by researchers in real research projects and the use cases that are frequently encountered in the field of bioinformatics.

Conclusions

Our approach enables automatic evaluation of the reproducibility of results using a fine-grained scale. By introducing our approach, it is possible to evolve from a binary view of whether the results are superficially identical or not to a more graduated view. We believe that our approach will contribute to more informed discussion on reproducibility in bioinformatics.

SUBMITTER: Suetake H

PROVIDER: S-EPMC10164546 | biostudies-literature | 2022 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A workflow reproducibility scale for automatic validation of biological interpretation results.

Suetake Hirotaka H Fukusato Tsukasa T Igarashi Takeo T Ohta Tazro T

GigaScience 20221201

<h4>Background</h4>Reproducibility of data analysis workflow is a key issue in the field of bioinformatics. Recent computing technologies, such as virtualization, have made it possible to reproduce workflow execution with ease. However, the reproducibility of results is not well discussed; that is, there is no standard way to verify whether the biological interpretation of reproduced results is the same. Therefore, it still remains a challenge to automatically evaluate the reproducibility of res ...[more]

PMID: 37150537

Similar Datasets

Project description:ContextThe reproducibility of adrenal vein sampling (AVS) is unknown.ObjectiveThis work aimed to determine reproducibility of biochemical results and diagnostic lateralization in patients undergoing repeat AVS.MethodsA retrospective chart review was conducted of single-center, single-operator AVS procedures at a tertiary care center. Included were patients with confirmed primary aldosteronism (PA) undergoing repeat AVS because of concerns about technical success or discordant diagnostic results. Simultaneous AVS was performed by an experienced operator using a consistent protocol of precosyntropin and postcosyntropin infusion. Among successfully catheterized adrenal veins (selectivity index ≥ 2), the correlation of the adrenalaldosterone/cortisol (A/C) ratio was measured between the first and second AVS. The secondary outcome measure was diagnostic agreement on repeat AVS lateralization (lateralization index ≥ 3).ResultsThere were 46 sets of AVS from 23 patients at a median of 3 months apart. There was moderate correlation in A/C ratios in the adrenal veins and inferior vena cava (Spearman r = 0.49-0.59, P < .05) pre cosyntropin. Post cosyntropin, the correlation was better (Spearman r = 0.67-0.76, P < .05). In technically successful AVS, there was moderate correlation between the repeated lateralization indices (Spearman r = 0.53, P < .05). In 15 patients in whom repeat AVS was performed because of apparent lateralization discordance with computed tomography imaging, the final diagnosis was the same in the second AVS procedure. Initial failed AVS was successful 75% of the time on repeat attempt.ConclusionRepeat AVS was feasible and usually successful when an initial attempt failed. There was modest correlation between individual repeat adrenal A/C ratios and lateralization indices when AVS was performed twice. The final lateralization diagnosis was identical in all cases. This demonstrates that AVS is a reliable and reproducible localizing test in PA.

Project description:ObjectiveAmid the COVID-19 pandemic, social stigma towards COVID-19 infection has become a major component of public discourse and social phenomena. As such, we aimed to develop and validate the COVID-19 Public Stigma Scale (COVID-PSS).Design and settingNational-based survey cross-sectional study during the lockdown in Thailand.ParticipantsWe invited the 4004 adult public to complete a set of measurement tools, including the COVID-PSS, global fear of COVID-19, perceived risk of COVID-19 infection, Bogardus Social Distance Scale, Pain Intensity Scale and Insomnia Severity Index.MethodsFactor structure dimensionality was constructed and reaffirmed with model fit by exploratory and confirmatory factor analyses and non-parametric item response theory (IRT) analysis. Psychometric properties for validity and reliability were tested. An anchor-based approach was performed for classifying the proper cut-off scores.ResultsAfter factor analysis, IRT analysis and test for model fit, we created the final 10-item COVID-PSS with a three-factor structure: stereotype, prejudice and fear. Face and content validity were established through the public and experts' perspectives. The COVID-PSS was significantly correlated (Spearman rank, 95% CI) with the global fear of COVID-19 (0.68, 95% CI 0.66 to 0.70), perceived risk of COVID-19 infection (0.79, 95% CI 0.77 to 0.80) and the Bogardus Social Distance Scale (0.50, 95% CI 0.48 to 0.53), indicating good convergent validity. The correlation statistics between the COVID-PSS and the Pain Intensity Scale and Insomnia Severity Index were <0.2, supporting the discriminant validity. The reliability of the COVID-PSS was satisfactory, with good internal consistency (Cronbach's α of 0.85, 95% CI 0.84 to 0.86) and test-retest reproducibility (intraclass correlation of 0.94, 95% CI 0.86 to 0.96). The proposed cut-off scores were as follows: no/minimal (≤18), moderate (19-25) and high (≥26) public stigma towards COVID-19 infection.ConclusionsThe COVID-PSS is practical and suitable for measuring stigma towards COVID-19 in a public health survey. However, cross-cultural adaptation may be needed.

Project description:BackgroundPoint-of-care arterial blood gas (ABG) is a blood measurement test and a useful diagnostic tool that assists with treatment and therefore improves clinical outcomes. However, numerically reported test results make rapid interpretation difficult or open to interpretation. The arterial blood gas algorithm (ABG-a) is a new digital diagnostics solution that can provide clinicians with real-time interpretation of preliminary data on safety features, oxygenation, acid-base disturbances and renal profile. The main aim of this study was to clinically validate the algorithm against senior experienced clinicians, for acid-base interpretation, in a clinical context.MethodsWe conducted a prospective international multicentre observational cross-sectional study. 346 sample sets and 64 inpatients eligible for ABG met strict sampling criteria. Agreement was evaluated using Cohen's kappa index, diagnostic accuracy was evaluated with sensitivity, specificity, efficiency or global accuracy and positive predictive values (PPV) and negative predictive values (NPV) for the prevalence in the study population.ResultsThe concordance rates between the interpretations of the clinicians and the ABG-a for acid-base disorders were an observed global agreement of 84,3% with a Cohen's kappa coefficient 0.81; 95% CI 0.77 to 0.86; p < 0.001. For detecting accuracy normal acid-base status the algorithm has a sensitivity of 90.0% (95% CI 79.9 to 95.3), a specificity 97.2% (95% CI 94.5 to 98.6) and a global accuracy of 95.9% (95% CI 93.3 to 97.6). For the four simple acid-base disorders, respiratory alkalosis: sensitivity of 91.2 (77.0 to 97.0), a specificity 100.0 (98.8 to 100.0) and global accuracy of 99.1 (97.5 to 99.7); respiratory acidosis: sensitivity of 61.1 (38.6 to 79.7), a specificity of 100.0 (98.8 to 100.0) and global accuracy of 98.0 (95.9 to 99.0); metabolic acidosis: sensitivity of 75.8 (59.0 to 87.2), a specificity of 99.7 (98.2 to 99.9) and a global accuracy of 97.4 (95.1 to 98.6); metabolic alkalosis sensitivity of 72.2 (56.0 to 84.2), a specificity of 95.5 (92.5 to 97.3) and a global accuracy of 93.0 (88.8 to 95.3); the four complex acid-base disorders, respiratory and metabolic alkalosis, respiratory and metabolic acidosis, respiratory alkalosis and metabolic acidosis, respiratory acidosis and metabolic alkalosis, the sensitivity, specificity and global accuracy was also high. For normal acid-base status the algorithm has PPV 87.1 (95% CI 76.6 to 93.3) %, and NPV 97.9 (95% CI 95.4 to 99.0) for a prevalence of 17.4 (95% CI 13.8 to 21.8). For the four-simple acid-base disorders and the four complex acid-base disorders the PPV and NPV were also statistically significant.ConclusionsThe ABG-a showed very high agreement and diagnostic accuracy with experienced senior clinicians in the acid-base disorders in a clinical context. The method also provides refinement and deep complex analysis at the point-of-care that a clinician could have at the bedside on a day-to-day basis. The ABG-a method could also have the potential to reduce human errors by checking for imminent life-threatening situations, analysing the internal consistency of the results, the oxygenation and renal status of the patient.

Project description:BackgroundOne of the current major factors of not following up on the abnormal test results is the lack of information about the test results and missing interpretations. Clinical decision support systems (CDSS) can become a solution to this problem. However, little is known how patients react to the automatically generated interpretations of the test results, and how this can affect a decision to follow up. In this research, we study how patients perceive the interpretations of the laboratory tests automatically generated by a clinical decision support system depending on how they receive these recommendations and how this affects the follow-up rate.MethodsA study of 3200 patients was done querying the regional patient registry. The patients were divided into 4 groups who received: 1. Recommendations automatically generated by a CDSS with a clear indication of their automatic nature. 2. Recommendations received personally from a doctor with a clear indication of their automatic nature. 3. Recommendations from a doctor with no indication of their automated generation. 4. No recommendations, only the test results. A follow-up rate was calculated as the proportion of patients referred to a laboratory service for a follow-up investigation after receiving a recommendation within two weeks after the first test with abnormal test results had been completed and the interpretation was delivered to the patient. The second phase of the study was a research of the patients' motivation. It was performed with a group of 789 patients.ResultsAll the patients who received interpretations on the abnormal test results demonstrated a significantly higher rate of follow-up (71%) in comparison to the patients who received only test results without interpretations (49%). Patients mention a time factor as a significant benefit of the automatically generated interpretations in comparison to the interpretations they can receive from a doctor.ConclusionThe results of the study show that delivering automatically generated interpretations of test results can support patients in making a decision to follow up. They are trusted by patients and raise their motivations and engagement.

Dataset Information

A workflow reproducibility scale for automatic validation of biological interpretation results.

Background

Results

Conclusions

Publications

A workflow reproducibility scale for automatic validation of biological interpretation results.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets