Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Asymptotic analysis of reliability measures for an imperfect dichotomous test.

ABSTRACT: To access the reliability of a new dichotomous test and to capture the random variability of its results in the absence of a gold standard, two measures, the inconsistent acceptance probability (IAP) and inconsistent rejection probability (IRP), were introduced in the literature. In this paper, we first analyze the limiting behavior of both measures as the number of test repetitions increases and derive the corresponding accuracy estimates and rates of convergence. To overcome possible limitations of IRP and IAP, we then introduce a one-parameter family of refined reliability measures, Δ(k,s) . Such measures characterize the consistency of the results of a dichotomous test in the absence of a gold standard as the threshold for a positive aggregate test result varies. Similar to IRP and IAP, we also derive corresponding accuracy estimates and rates of convergence for Δ(k,s) as the number k of test repetitions increases.

Supplementary information

The online version supplementary material available at 10.1007/s00362-021-01266-9.

SUBMITTER: Slynko A

PROVIDER: S-EPMC8492041 | biostudies-literature | 2021 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Asymptotic analysis of reliability measures for an imperfect dichotomous test.

Slynko Alla A

Statistical papers (Berlin, Germany) 20211005 4

To access the reliability of a new dichotomous test and to capture the random variability of its results in the absence of a gold standard, two measures, the inconsistent acceptance probability (IAP) and inconsistent rejection probability (IRP), were introduced in the literature. In this paper, we first analyze the limiting behavior of both measures as the number of test repetitions increases and derive the corresponding accuracy estimates and rates of convergence. To overcome possible limitatio ...[more]

PMID: 34629758

Similar Datasets

Meta-analysis of dichotomous and ordinal tests with an imperfect gold standard.

Project description:Standard methods for the meta-analysis of medical tests, without assuming a gold standard, are limited to dichotomous data. Multivariate probit models are used to analyse correlated dichotomous data, and can be extended to model ordinal data. Within the context of an imperfect gold standard, they have previously been used for the analysis of dichotomous and ordinal test data from a single study, and for the meta-analysis of dichotomous tests. However, they have not previously been used for the meta-analysis of ordinal tests. In this article, we developed a Bayesian multivariate probit latent class model for the simultaneous meta-analysis of ordinal and dichotomous tests without assuming a gold standard, which also allows one to obtain summary estimates of joint test accuracy. We fitted the models using the software Stan, which uses a state-of-the-art Hamiltonian Monte Carlo algorithm, and we applied the models to a dataset in which studies evaluated the accuracy of tests, and test combinations, for deep vein thrombosis. We demonstrate the issues with dichotomising ordinal test accuracy data in the presence of an imperfect gold standard, before applying and comparing several variations of our proposed model which do not require the data to be dichotomised. The models proposed will allow researchers to more appropriately meta-analyse ordinal and dichotomous tests without a gold standard, potentially leading to less biased estimates of test accuracy. This may lead to a better understanding of which tests, and test combinations, should be used for any given medical condition.

| S-EPMC9541315 | biostudies-literature

Conditional asymptotic inference for the kernel association test.

Project description:MotivationThe kernel association test (KAT) is popular in biological studies for its ability to combine weak effects potentially of opposite direction. Its P-value is typically assessed via its (unconditional) asymptotic distribution. However, such an asymptotic distribution is known only for continuous traits and for dichotomous traits. Furthermore, the derived P-values are known to be conservative when sample size is small, especially for the important case of dichotomous traits. One alternative is the permutation test, a widely accepted approximation to the exact finite sample conditional inference. But it is time-consuming to use in practice due to stringent significance criteria commonly seen in these analyses.ResultsBased on a previous theoretical result a conditional asymptotic distribution for the KAT is introduced. This distribution provides an alternative approximation to the exact distribution of the KAT. An explicit expression of this distribution is provided from which P-values can be easily computed. This method applies to any type of traits. The usefulness of this approach is demonstrated via extensive simulation studies using real genotype data and an analysis of genetic data from the Ocular Hypertension Treatment Study. Numerical results showed that the new method can control the type I error rate and is a bit conservative when compared to the permutation method. Nevertheless the proposed method may be used as a fast screening method. A time-consuming permutation procedure may be conducted at locations that show signals of association.Availability and implementationAn implementation of the proposed method is provided in the R package iGasso.Contactkai-wang@uiowa.edu.

| S-EPMC5860324 | biostudies-literature

Alternative Devices for Heart Rate Variability Measures: A Comparative Test-Retest Reliability Study.

Project description:Using healthy adult participants, seven measures of heart rate variability were obtained simultaneously from four devices in five behavioral conditions. Two devices were ECG-based and two utilized photoplethysmography. The 140 numerical values (measure, condition, device) are presented. The comparative operational reliability of the four devices was assessed, and it was found that the two ECG-base devices were more reliable than the photoplethysmographic devices. The interchangeability of devices was assessed by determining the between-device Limits of Agreement. Intraclass correlation coefficients were determined and used to calculate the standard error of measurement and the Minimal Detectable Difference. The Minimal Detectable Difference, MDD, quantifies the smallest statistically significant change in a measure and is therefore critical when HRV measures are used longitudinally to assess treatment response or disease progression.

| S-EPMC8147413 | biostudies-literature

Test-retest reliability of behavioral and computational measures of advice taking under volatility.

Project description:The development of computational models for studying mental disorders is on the rise. However, their psychometric properties remain understudied, posing a risk of undermining their use in empirical research and clinical translation. Here we investigated test-retest reliability (with a 2-week interval) of a computational assay probing advice-taking under volatility with a Hierarchical Gaussian Filter (HGF) model. In a sample of 39 healthy participants, we found the computational measures to have largely poor reliability (intra-class correlation coefficient or ICC < 0.5), on par with the behavioral measures of task performance. Further analysis revealed that reliability was substantially impacted by intrinsic measurement noise (indicated by parameter recovery analysis) and to a smaller extent by practice effects. However, a large portion of within-subject variance remained unexplained and may be attributable to state-like fluctuations. Despite the poor test-retest reliability, we found the assay to have face validity at the group level. Overall, our work highlights that the different sources of variance affecting test-retest reliability need to be studied in greater detail. A better understanding of these sources would facilitate the design of more psychometrically sound assays, which would improve the quality of future research and increase the probability of clinical translation.

| S-EPMC11573178 | biostudies-literature

Quantifying test-retest reliability of repeated objective attentional measures in Lewy body dementia.

Project description:Objective cognitive impairment is a feature of Lewy body dementia (LBD), and computerised attentional tasks are commonly used as outcome measures in interventional trials. However, the reliability of these measures, in the absence of interventions, are unknown. This study examined the reliability of these attentional measures at short-term and longer-term follow-up stages. LBD patients (n = 36) completed computerised attentional tasks [simple and choice reaction time, and digit vigilance (SRT, CRT, DV)] at short-term (Day 0-Day 5) and longer-term (4 and 12 weeks) follow-up. Intra-class correlations (ICCs) were calculated to assess test-retest reliability. At short-term, the reciprocal SRT, CRT and DV mean reaction time to correct answers, the reciprocal DV coefficient of variation, and reciprocal power of attention (PoA) all showed excellent levels of reliability (all ICCs > 0.90). The reciprocal PoA showed the highest level of reliability (ICC = 0.978). At longer-term follow-up, only the reciprocal PoA had excellent levels of reliability (ICC = 0.927). Reciprocal SRT, CRT and DV reaction time to correct answers, and the CRT coefficient of variation values, showed good levels of test-retest reliability (ICCs ≥ 0.85). Contrary to expectations, most attentional measures demonstrated high levels of test-retest reliability at both short-term and longer-term follow-up time points. The reciprocal PoA composite measure demonstrated excellent levels of test-retest reliability, both in the short-term and long-term. This indicates that objective attentional tasks are suitable outcome measures in LBD studies and that the composite PoA measure may offer the highest levels of reliability.

| S-EPMC9217900 | biostudies-literature

What Is the Test-Retest Reliability of Common Task-Functional MRI Measures? New Empirical Evidence and a Meta-Analysis.

Project description:Identifying brain biomarkers of disease risk is a growing priority in neuroscience. The ability to identify meaningful biomarkers is limited by measurement reliability; unreliable measures are unsuitable for predicting clinical outcomes. Measuring brain activity using task functional MRI (fMRI) is a major focus of biomarker development; however, the reliability of task fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. First, a meta-analysis of 90 experiments (N = 1,008) revealed poor overall reliability-mean intraclass correlation coefficient (ICC) = .397. Second, the test-retest reliabilities of activity in a priori regions of interest across 11 common fMRI tasks collected by the Human Connectome Project (N = 45) and the Dunedin Study (N = 20) were poor (ICCs = .067-.485). Collectively, these findings demonstrate that common task-fMRI measures are not currently suitable for brain biomarker discovery or for individual-differences research. We review how this state of affairs came to be and highlight avenues for improving task-fMRI reliability.

| S-EPMC7370246 | biostudies-literature

Reliability modelling and evaluating of wind turbine considering imperfect repair.

Project description:To model and evaluate the reliability of wind turbine (WT) under imperfect repair, an improved Log-linear Proportional Intensity Model (LPIM)-based method was proposed. Initially, using the three-parameter bounded intensity process (3-BIP) as the benchmark failure intensity function of LPIM, an imperfect repair effect-aware WT reliability description model was developed. Among them, the 3-BIP was used to describe the evolution process of the failure intensity in the stable operation stage with running time, while the LPIM reflected the repair effect. Second, the estimation problem for model parameters was transformed into a minimum solution problem for a nonlinear objective function, which was then solved using the Particle Swarm Optimization algorithm. The confidence interval of model parameters was finally estimated using the inverse Fisher information matrix method. Key reliability indices interval estimation based on the Delta method and point estimation was derived. The proposed method was applied to a wind farm's WT failure truncation time. The proposed method has a higher goodness of fit based on verification and comparison. As a result, it can bring the evaluated reliability closer to engineering practice.

| S-EPMC10067845 | biostudies-literature

Psychometric properties of measures of substance use: a systematic review and meta-analysis of reliability, validity and diagnostic test accuracy.

Project description:BackgroundSynthesis of psychometric properties of substance use measures to identify patterns of use and substance use disorders remains limited. To address this gap, we sought to systematically evaluate the psychometric properties of measures to detect substance use and misuse.MethodsWe conducted a systematic review and meta-analysis of literature on measures of substance classes associated with HIV risk (heroin, methamphetamine, cocaine, ecstasy, alcohol) that were published in English before June 2016 that reported at least one of the following psychometric outcomes of interest: internal consistency (alpha), test-retest/inter-rater reliability (kappa), sensitivity, specificity, positive predictive value, and negative predictive value. We used meta-analytic techniques to generate pooled summary estimates for these outcomes using random effects and hierarchical logistic regression models.ResultsFindings across 387 paper revealed that overall, 65% of pooled estimates for alpha were in the range of fair-to-excellent; 44% of estimates for kappa were in the range of fair-to-excellent. In addition, 69, 97, 37 and 96% of pooled estimates for sensitivity, specificity, positive predictive value, and negative predictive value, respectively, were in the range of moderate-to-excellent.ConclusionWe conclude that many substance use measures had pooled summary estimates that were at the fair/moderate-to-excellent range across different psychometric outcomes. Most scales were conducted in English, within the United States, highlighting the need to test and validate these measures in more diverse settings. Additionally, the majority of studies had high risk of bias, indicating a need for more studies with higher methodological quality.

| S-EPMC7203822 | biostudies-literature

A generalized family-based association test for dichotomous traits.

Project description:Recent advances in genotyping technology make it possible to utilize large-scale association analysis for disease-gene mapping. Powerful and robust family-based association methods are crucial for successful gene mapping. We propose a family-based association method, the generalized disequilibrium test (GDT), in which the genotype differences of all discordant relative pairs are utilized in assessing association within a family. The improvement of the GDT over existing methods is threefold: (1) information beyond first-degree relatives is incorporated efficiently, yielding substantial gains in power in comparison to existing tests; (2) the GDT statistic is implemented via a robust technique that does not rely on large sample theory, resulting in further power gains, especially at high levels of significance; and (3) covariates and weights based on family size are incorporated. Advantages of the GDT over existing methods are demonstrated by extensive computer simulations and by application to recently published large-scale genome-wide linkage data from the Type 1 Diabetes Genetics Consortium (T1DGC). In our simulations, the GDT consistently outperforms other tests for a common disease and frequently outperforms other tests for a rare disease; the power improvement is > 13% in 6 out of 8 extended pedigree scenarios. All of the six strongest associations identified by the GDT have been reported by other studies, whereas only three or four of these associations can be identified by existing methods. For the T1D association at gene UBASH3A, the GDT resulted in a genome-wide significance (p = 4.3 x 10(-6)), much stronger than the published significance (p = 10(-4)).

| S-EPMC2771538 | biostudies-literature

Alterations and test-retest reliability of functional connectivity network measures in cerebral small vessel disease.

Project description:While structural network analysis consolidated the hypothesis of cerebral small vessel disease (SVD) being a disconnection syndrome, little is known about functional changes on the level of brain networks. In patients with genetically defined SVD (CADASIL, n = 41) and sporadic SVD (n = 46), we independently tested the hypothesis that functional networks change with SVD burden and mediate the effect of disease burden on cognitive performance, in particular slowing of processing speed. We further determined test-retest reliability of functional network measures in sporadic SVD patients participating in a high-frequency (monthly) serial imaging study (RUN DMC-InTENse, median: 8 MRIs per participant). Functional networks for the whole brain and major subsystems (i.e., default mode network, DMN; fronto-parietal task control network, FPCN; visual network, VN; hand somatosensory-motor network, HSMN) were constructed based on resting-state multi-band functional MRI. In CADASIL, global efficiency (a graph metric capturing network integration) of the DMN was lower in patients with high disease burden (standardized beta = -.44; p [corrected] = .035) and mediated the negative effect of disease burden on processing speed (indirect path: std. beta = -.20, p = .047; direct path: std. beta = -.19, p = .25; total effect: std. beta = -.39, p = .02). The corresponding analyses in sporadic SVD showed no effect. Intraclass correlations in the high-frequency serial MRI dataset of the sporadic SVD patients revealed poor test-retest reliability and analysis of individual variability suggested an influence of age, but not disease burden, on global efficiency. In conclusion, our results suggest that changes in functional connectivity networks mediate the effect of SVD-related brain damage on cognitive deficits. However, limited reliability of functional network measures, possibly due to age-related comorbidities, impedes the analysis in elderly SVD patients.

| S-EPMC7294060 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data