Dataset Information

Therapist-Observer Concordance in Ratings of EBP Strategy Delivery: Challenges and Targeted Directions in Pursuing Pragmatic Measurement in Children's Mental Health Services.

ABSTRACT: Pragmatic measures of therapist delivery of evidence-based practice (EBP) are critical to assessing the impact of large-scale, multiple EBP implementation efforts. As an initial step in the development of pragmatic measurement, the current study examined the concordance between therapist and observer ratings of items assessing delivery of EBP strategies considered essential for common child EBP targets. Possible EBP-, session-, and therapist-levels factors associated with concordance were also explored. Therapists and independent observers rated the extensiveness of therapist (n = 103) EBP strategy delivery in 680 community psychotherapy sessions in which six EBPs were used. Concordance between therapist- and observer-report of the extensiveness of therapist EBP strategy use was at least fair (ICC ≥ .40) for approximately half of the items. Greater therapist-observer concordance was observed in sessions where a structured EBP was delivered and in sessions where therapists reported being able to carry out planned activities. Findings highlighted conditions that may improve or hinder therapists' ability to report on their own EBP strategy delivery in a way that is consistent with independent observers. These results can help inform the development of pragmatic therapist-report measures of EBP strategy delivery and implementation efforts more broadly.

SUBMITTER: Brookman-Frazee L

PROVIDER: S-EPMC7722101 | biostudies-literature | 2021 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Therapist-Observer Concordance in Ratings of EBP Strategy Delivery: Challenges and Targeted Directions in Pursuing Pragmatic Measurement in Children's Mental Health Services.

Brookman-Frazee Lauren L Stadnick Nicole A NA Lind Teresa T Roesch Scott S Terrones Laura L Barnett Miya L ML Regan Jennifer J Kennedy Catherine A CA F Garland Ann A Lau Anna S AS

Administration and policy in mental health 20210101 1

Pragmatic measures of therapist delivery of evidence-based practice (EBP) are critical to assessing the impact of large-scale, multiple EBP implementation efforts. As an initial step in the development of pragmatic measurement, the current study examined the concordance between therapist and observer ratings of items assessing delivery of EBP strategies considered essential for common child EBP targets. Possible EBP-, session-, and therapist-levels factors associated with concordance were also e ...[more]

PMID: 32507982

Similar Datasets

Project description:BackgroundNumerous studies have examined the association between safety and primary school-aged children's forms of active mobility. However, variations in studies' measurement methods and the elements addressed have contributed to inconsistencies in research outcomes, which may be forming a barrier to advancing researchers' knowledge about this field. To assess where current research stands, we have synthesised the methodological measures in studies that examined the effects of neighbourhood safety exposure (perceived and measured) on children's outdoor active mobility behaviour and used this analysis to propose future research directions.MethodA systematic search of the literature in six electronic databases was conducted using pre-defined eligibility criteria and was concluded in July 2020. Two reviewers screened the literature abstracts to determine the studies' inclusion, and two reviewers independently conducted a methodological quality assessment to rate the included studies.ResultsTwenty-five peer-reviewed studies met the inclusion criteria. Active mobility behaviour and health characteristics were measured objectively in 12 out of the 25 studies and were reported in another 13 studies. Twenty-one studies overlooked spatiotemporal dimensions in their analyses and outputs. Delineations of children's neighbourhoods varied within 10 studies' objective measures, and the 15 studies that opted for subjective measures. Safety perceptions obtained in 22 studies were mostly static and primarily collected via parents, and dissimilarities in actual safety measurement methods were present in 6 studies. The identified schematic constraints in studies' measurement methods assisted in outlining a three-dimensional relationship between 'what' (determinants), 'where' (spatial) and 'when' (time) within a methodological conceptual framework.ConclusionsThe absence of standardised measurement methods among relevant studies may have led to the current diversity in findings regarding active mobility, spatial (locality) and temporal (time) characteristics, the neighbourhood, and the representation of safety. Ignorance of the existing gaps and heterogeneity in measures may impact the reliability of evidence and poses a limitation when synthesising findings, which could result in serious biases for policymakers. Given the increasing interest in children's health studies, we suggested alternatives in the design and method of measures that may guide future evidence-based research for policymakers who aim to improve children's active mobility and safety.

Project description:The Double Empathy Problem suggests that communicative difficulties between autistic and non-autistic people are due to bi-directional differences in communicative style and a reciprocal lack of understanding. If true, there should be increased similarity in interaction style, resulting in higher rapport during interactions between pairs of the same neurotype. Here, we provide two empirical tests of rapport, with data revealing whether self- and observer- rated rapport varies depending on the match or mismatch in autism status within a pair. An additional opportunity afforded by these data is to examine the effect of the autism status of the rater on the perceived rapport between matched and mismatched pairs. In Study 1 72 participants were allocated to one of three dyad conditions: autistic pairs (n = 24), non-autistic pairs (n = 24) and mixed pairs (n = 12 autistic; n = 12 non-autistic). Each participant completed three semi-structured interactions with their partner, rating rapport after each interaction. Non-autistic pairs experienced higher self-rated rapport than mixed and autistic pairs, and autistic pairs experienced higher rapport than mixed pairs. In Study 2 (n = 80) autistic and non-autistic observers rated interactional rapport while watching videoed interactions between autistic pairs, non-autistic pairs, and mixed pairs (n = 18, a subset of participants in Study 1). Mixed pairs were rated significantly lower on rapport than autistic and non-autistic pairs, and autistic pairs were rated more highly for rapport than non-autistic pairs. Both autistic and non-autistic observers show similar patterns in how they rate the rapport of autistic, non-autistic, and mixed pairs. In summary, autistic people experience high interactional rapport when interacting with other autistic people, and this is also detected by external observers. Rather than autistic people experiencing low rapport in all contexts, their rapport ratings are influenced by a mismatch of diagnosis. These findings suggest that autistic people possess a distinct mode of social interaction style, rather than demonstrating social skills deficits. These data are considered in terms of their implications for psychological theories of autism, as well as practical impact on educational and clinical practice.

Project description:ImportanceThe Centers for Medicare & Medicaid Services (CMS) Hospital Compare star ratings are widely used summaries of hospital quality that can influence patient choice and organizational reputation.ObjectiveTo identify the changes in hospital ratings and rankings associated with alternative methodological choices in the calculation of the 2021 CMS Hospital Compare star ratings.Design setting and participantsThis cross-sectional study used publicly available 2021 Hospital Compare data for 3339 US hospitals from the October 2020 data release. Change in apparent hospital performance was assessed, arising when plausible alternatives to current methods are used for calculating star ratings in relation to individual measure standardization, domain derivation, and domain weighting. Three example changes were examined in detail, with more comprehensive changes considered using Monte Carlo simulation. Changes in centile of hospital ranks and in star rating overall were examined, as well as separately in CMS peer groupings defined by the number of reported quality domains.Main outcomes and measuresProportion of hospitals receiving a different star rating under the alternative technical specifications than under the current (2021) CMS approach and mean absolute change in centile rank of hospitals under alternative technical specifications compared with the current (2021) CMS approach.ResultsA total of 3339 US hospitals were included in the analysis. Of the specific changes to current specifications considered, the alternative method of standardization was associated with the most substantial changes, with 55.4% (95% CI, 53.7%-57.1%) of hospitals having their star rating reclassified. The change in domain weights was associated with the smallest differences, but even that resulted in reclassification of the star rating in approximately 1 in 4 (24.5%; 95% CI, 23.0%-26.0%) hospitals. In the simulation study, which covered a range of possible changes, on average half of hospitals (51.8%; IQR, 44.2%-59.7%) were assigned a different star rating from that produced by the current specification, usually into an adjacent star category.Conclusions and relevanceIn this cross-sectional study of data on US hospitals, CMS Hospital Compare star ratings were found to be highly sensitive to how performance ratings are calculated, demonstrating the need for transparent justification of the technical approaches used in calculating composite performance ratings.

Project description:ImportanceThe Press Ganey Outpatient Medical Practice Survey is used to measure the patient experience. An understanding of the patient- and physician-related determinants of the patient experience may help identify opportunities to improve health care delivery and physician ratings.ObjectiveTo evaluate the associations between the patient experience as measured by scores on the Press Ganey survey and patient-physician racial/ethnic and gender concordance.Design, setting, and participantsA cross-sectional analysis of Press Ganey surveys returned for outpatient visits within the University of Pennsylvania Health System between 2014 and 2017 was performed. Participants included adult patient and physician dyads for whom surveys were returned. Data analysis was performed from January to June 2019.ExposuresPatient-physician racial/ethnic and gender concordance.Main outcomes and measuresThe primary outcome was receipt of the maximum score for the "likelihood of your recommending this care provider to others" question in the Care Provider domain of the Press Ganey survey. Secondary outcomes included each of the remaining 9 questions in the Care Provider domain. Generalized estimating equations clustering on physicians with exchangeable intracluster correlations and cluster-robust standard errors were used to investigate associations between the outcomes and patient-physician racial/ethnic and gender concordance.ResultsIn total, 117 589 surveys were evaluated, corresponding to 92 238 unique patients (mean [SD] age, 57.7 [15.6] years; 37 002 men [40.1%]; 75 307 White patients [81.6%]) and 747 unique physicians (mean [SD] age 45.5 [10.6] years; 472 men [63.2%]; 533 White physicians [71.4%]). Compared with racially/ethnically concordant patient-physician dyads, discordance was associated with a lower likelihood of physicians receiving the maximum score (adjusted odds ratio [OR], 0.88; 95% CI, 0.82-0.94; P < .001). Black (adjusted OR, 0.73; 95% CI, 0.68-0.78; P < .001) and Asian (adjusted OR, 0.55; 95% CI, 0.50-0.60; P < .001) patient race were both associated with lower patient experience ratings. Patient-physician gender concordance was not associated with Press Ganey scores (adjusted OR, 1.00; 95% CI, 0.96-1.04; P = .90).Conclusions and relevanceIn this study, higher Press Ganey survey scores were associated with racial/ethnic concordance between patients and their physicians. Efforts to improve physician workforce diversity are imperative. Delivery of health care in a culturally mindful manner between racially/ethnically discordant patient-physician dyads is also essential. Furthermore, Press Ganey scores may differ by a physician's patient demographic mix; thus, care must be taken when publicly reporting or using Press Ganey scores to evaluate physicians on an individual level.

Dataset Information

Therapist-Observer Concordance in Ratings of EBP Strategy Delivery: Challenges and Targeted Directions in Pursuing Pragmatic Measurement in Children's Mental Health Services.

Publications

Therapist-Observer Concordance in Ratings of EBP Strategy Delivery: Challenges and Targeted Directions in Pursuing Pragmatic Measurement in Children's Mental Health Services.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets