Dataset Information

Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool.

ABSTRACT: In studies of diagnostic test accuracy, authors sometimes report results only for a range of cutoff points around data-driven "optimal" cutoffs. We assessed selective cutoff reporting in studies of the diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool. We compared conventional meta-analysis of published results only with individual-patient-data meta-analysis of results derived from all cutoff points, using data from 13 of 16 studies published during 2004-2009 that were included in a published conventional meta-analysis. For the "standard" PHQ-9 cutoff of 10, accuracy results had been published by 11 of the studies. For all other relevant cutoffs, 3-6 studies published accuracy results. For all cutoffs examined, specificity estimates in conventional and individual-patient-data meta-analyses were within 1% of each other. Sensitivity estimates were similar for the cutoff of 10 but differed by 5%-15% for other cutoffs. In samples where the PHQ-9 was poorly sensitive at the standard cutoff, authors tended to report results for lower cutoffs that yielded optimal results. When the PHQ-9 was highly sensitive, authors more often reported results for higher cutoffs. Consequently, in the conventional meta-analysis, sensitivity increased as cutoff severity increased across part of the cutoff range-an impossibility if all data are analyzed. In sum, selective reporting by primary study authors of only results from cutoffs that perform well in their study can bias accuracy estimates in meta-analyses of published results.

SUBMITTER: Levis B

PROVIDER: S-EPMC5430941 | biostudies-literature | 2017 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool.

Levis Brooke B Benedetti Andrea A Levis Alexander W AW Ioannidis John P A JPA Shrier Ian I Cuijpers Pim P Gilbody Simon S Kloda Lorie A LA McMillan Dean D Patten Scott B SB Steele Russell J RJ Ziegelstein Roy C RC Bombardier Charles H CH de Lima Osório Flavia F Fann Jesse R JR Gjerdingen Dwenda D Lamers Femke F Lotrakul Manote M Loureiro Sonia R SR Löwe Bernd B Shaaban Juwita J Stafford Lesley L van Weert Henk C P M HCPM Whooley Mary A MA Williams Linda S LS Wittkampf Karin A KA Yeung Albert S AS Thombs Brett D BD

American journal of epidemiology 20170501 10

In studies of diagnostic test accuracy, authors sometimes report results only for a range of cutoff points around data-driven "optimal" cutoffs. We assessed selective cutoff reporting in studies of the diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool. We compared conventional meta-analysis of published results only with individual-patient-data meta-analysis of results derived from all cutoff points, using data from 13 of 16 studies published during 2004 ...[more]

PMID: 28419203

Similar Datasets

Project description:ImportanceTest accuracy studies often use small datasets to simultaneously select an optimal cutoff score that maximizes test accuracy and generate accuracy estimates.ObjectiveTo evaluate the degree to which using data-driven methods to simultaneously select an optimal Patient Health Questionnaire-9 (PHQ-9) cutoff score and estimate accuracy yields (1) optimal cutoff scores that differ from the population-level optimal cutoff score and (2) biased accuracy estimates.Design, setting, and participantsThis study used cross-sectional data from an existing individual participant data meta-analysis (IPDMA) database on PHQ-9 screening accuracy to represent a hypothetical population. Studies in the IPDMA database compared participant PHQ-9 scores with a major depression classification. From the IPDMA population, 1000 studies of 100, 200, 500, and 1000 participants each were resampled.Main outcomes and measuresFor the full IPDMA population and each simulated study, an optimal cutoff score was selected by maximizing the Youden index. Accuracy estimates for optimal cutoff scores in simulated studies were compared with accuracy in the full population.ResultsThe IPDMA database included 100 primary studies with 44 503 participants (4541 [10%] cases of major depression). The population-level optimal cutoff score was 8 or higher. Optimal cutoff scores in simulated studies ranged from 2 or higher to 21 or higher in samples of 100 participants and 5 or higher to 11 or higher in samples of 1000 participants. The percentage of simulated studies that identified the true optimal cutoff score of 8 or higher was 17% for samples of 100 participants and 33% for samples of 1000 participants. Compared with estimates for a cutoff score of 8 or higher in the population, sensitivity was overestimated by 6.4 (95% CI, 5.7-7.1) percentage points in samples of 100 participants, 4.9 (95% CI, 4.3-5.5) percentage points in samples of 200 participants, 2.2 (95% CI, 1.8-2.6) percentage points in samples of 500 participants, and 1.8 (95% CI, 1.5-2.1) percentage points in samples of 1000 participants. Specificity was within 1 percentage point across sample sizes.Conclusions and relevanceThis study of cross-sectional data found that optimal cutoff scores and accuracy estimates differed substantially from population values when data-driven methods were used to simultaneously identify an optimal cutoff score and estimate accuracy. Users of diagnostic accuracy evidence should evaluate studies of accuracy with caution and ensure that cutoff score recommendations are based on adequately powered research or well-conducted meta-analyses.

Project description:ObjectiveTo determine the accuracy of the Patient Health Questionnaire-9 (PHQ-9) for screening to detect major depression.DesignIndividual participant data meta-analysis.Data sourcesMedline, Medline In-Process and Other Non-Indexed Citations, PsycINFO, and Web of Science (January 2000-February 2015).Inclusion criteriaEligible studies compared PHQ-9 scores with major depression diagnoses from validated diagnostic interviews. Primary study data and study level data extracted from primary reports were synthesized. For PHQ-9 cut-off scores 5-15, bivariate random effects meta-analysis was used to estimate pooled sensitivity and specificity, separately, among studies that used semistructured diagnostic interviews, which are designed for administration by clinicians; fully structured interviews, which are designed for lay administration; and the Mini International Neuropsychiatric (MINI) diagnostic interviews, a brief fully structured interview. Sensitivity and specificity were examined among participant subgroups and, separately, using meta-regression, considering all subgroup variables in a single model.ResultsData were obtained for 58 of 72 eligible studies (total n=17 357; major depression cases n=2312). Combined sensitivity and specificity was maximized at a cut-off score of 10 or above among studies using a semistructured interview (29 studies, 6725 participants; sensitivity 0.88, 95% confidence interval 0.83 to 0.92; specificity 0.85, 0.82 to 0.88). Across cut-off scores 5-15, sensitivity with semistructured interviews was 5-22% higher than for fully structured interviews (MINI excluded; 14 studies, 7680 participants) and 2-15% higher than for the MINI (15 studies, 2952 participants). Specificity was similar across diagnostic interviews. The PHQ-9 seems to be similarly sensitive but may be less specific for younger patients than for older patients; a cut-off score of 10 or above can be used regardless of age..ConclusionsPHQ-9 sensitivity compared with semistructured diagnostic interviews was greater than in previous conventional meta-analyses that combined reference standards. A cut-off score of 10 or above maximized combined sensitivity and specificity overall and for subgroups.RegistrationPROSPERO CRD42014010673.

Project description:BackgroundIndividual patient data meta-analyses (IPD-MA) are regarded as the gold standard for systematic reviews, which also applies to systematic reviews of diagnostic test accuracy (DTA) studies. An increasing number of DTA systematic reviews with IPD-MA have been published in recent years, but there is much variation in how these IPD-MA were performed. A number of existing methods were found, but there is no consensus as to which methods are preferred as the standard methods for statistical analysis in DTA IPD-MA.ObjectivesTo create a web-based tool which integrates recommended statistical analyses for DTA IPD-MA, and allows researchers to analyse the data and visualize the results with interactive plots.MethodsA systematic methodological review was performed to identify statistical analyses and data visualization methods used in DTA IPD-MA. Methods were evaluated by the authors and recommended analyses were integrated into the IPDmada tool which is freely available online with the user interface developed with R Shiny package.ResultsIPDmada allows users to upload their own data, perform the meta-analysis with both continuous and dichotomized tests, and incorporate individual level covariate-adjusted analysis. All tables and figures can be exported as .csv or .pdf files. A hypothetical dataset was used to illustrate the application of IPDmada.ConclusionsIPDmada will be very helpful to researchers doing DTA IPD-MA, since it not only facilitates the statistical analysis but also provides a standard framework. The introduction of IPDmada will harmonize the methods used in DTA IPD-MA and ensure the quality of such analyses.HighlightsIPDmada is a newly developed web-based tool for performing statistical analysis of individual patient data meta-analysis of diagnostic accuracy and visualizing the results. The tool is freely available to all the researchers, and requiring no installation of statistical software/packages. The tool has an user-friendly interface, and allows meta-analysis on both dichotomized and continuous test results. Researchers can easily use this tool to investigate the threshold effect and covariate effect on the summary accuracy. The introduction and implementation of IPDmada will serve as a useful tool for DTA IPD-MA and increase the quality of such studies.

Project description:ObjectiveTo update a previous individual participant data meta-analysis and determine the accuracy of the Patient Health Questionnaire-9 (PHQ-9), the most commonly used depression screening tool in general practice, for detecting major depression overall and by study or participant subgroups.DesignSystematic review and individual participant data meta-analysis.Data sourcesMedline, Medline In-Process, and Other Non-Indexed Citations via Ovid, PsycINFO, Web of Science searched through 9 May 2018.Review methodsEligible studies administered the PHQ-9 and classified current major depression status using a validated semistructured diagnostic interview (designed for clinician administration), fully structured interview (designed for lay administration), or the Mini International Neuropsychiatric Interview (MINI; a brief interview designed for lay administration). A bivariate random effects meta-analytic model was used to obtain point and interval estimates of pooled PHQ-9 sensitivity and specificity at cut-off values 5-15, separately, among studies that used semistructured diagnostic interviews (eg, Structured Clinical Interview for Diagnostic and Statistical Manual), fully structured interviews (eg, Composite International Diagnostic Interview), and the MINI. Meta-regression was used to investigate whether PHQ-9 accuracy correlated with reference standard categories and participant characteristics.ResultsData from 44 503 total participants (27 146 additional from the update) were obtained from 100 of 127 eligible studies (42 additional studies; 79% eligible studies; 86% eligible participants). Among studies with a semistructured interview reference standard, pooled PHQ-9 sensitivity and specificity (95% confidence interval) at the standard cut-off value of ≥10, which maximised combined sensitivity and specificity, were 0.85 (0.79 to 0.89) and 0.85 (0.82 to 0.87), respectively. Specificity was similar across reference standards, but sensitivity in studies with semistructured interviews was 7-24% (median 21%) higher than with fully structured reference standards and 2-14% (median 11%) higher than with the MINI across cut-off values. Across reference standards and cut-off values, specificity was 0-10% (median 3%) higher for men and 0-12 (median 5%) higher for people aged 60 or older.ConclusionsResearchers and clinicians could use results to determine outcomes, such as total number of positive screens and false positive screens, at different PHQ-9 cut-off values for different clinical settings using the knowledge translation tool at www.depressionscreening100.com/phq.Study registrationPROSPERO CRD42014010673.

Project description:BackgroundSelective reporting of results from only well-performing cut-offs leads to biased estimates of accuracy in primary studies of questionnaire-based screening tools and in meta-analyses that synthesize results. Individual participant data meta-analysis (IPDMA) of sensitivity and specificity at each cut-off via bivariate random-effects models (BREMs) can overcome this problem. However, IPDMA is laborious and depends on the ability to successfully obtain primary datasets, and BREMs ignore the correlation between cut-offs within primary studies.MethodsWe compared the performance of three recent multiple cut-off models developed by Steinhauser et al., Jones et al., and Hoyer and Kuss, that account for missing cut-offs when meta-analyzing diagnostic accuracy studies with multiple cut-offs, to BREMs fitted at each cut-off. We used data from 22 studies of the accuracy of the Edinburgh Postnatal Depression Scale (EPDS; 4475 participants, 758 major depression cases). We fitted each of the three multiple cut-off models and BREMs to a dataset with results from only published cut-offs from each study (published data) and an IPD dataset with results for all cut-offs (full IPD data). We estimated pooled sensitivity and specificity with 95% confidence intervals (CIs) for each cut-off and the area under the curve.ResultsCompared to the BREMs fitted to the full IPD data, the Steinhauser et al., Jones et al., and Hoyer and Kuss models fitted to the published data produced similar receiver operating characteristic curves; though, the Hoyer and Kuss model had lower area under the curve, mainly due to estimating slightly lower sensitivity at lower cut-offs. When fitting the three multiple cut-off models to the full IPD data, a similar pattern of results was observed. Importantly, all models had similar 95% CIs for sensitivity and specificity, and the CI width increased with cut-off levels for sensitivity and decreased with an increasing cut-off for specificity, even the BREMs which treat each cut-off separately.ConclusionsMultiple cut-off models appear to be the favorable methods when only published data are available. While collecting IPD is expensive and time consuming, IPD can facilitate subgroup analyses that cannot be conducted with published data only.

Dataset Information

Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool.

Publications

Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets