Dataset Information

Testing and improving the acceptability of a web-based platform for collective intelligence to improve diagnostic accuracy in primary care clinics.

ABSTRACT:

Objectives

Usable tools to support individual primary care clinicians in their diagnostic processes could help to reduce preventable harm from diagnostic errors. We conducted a formative study with primary care providers to identify key requisites to optimize the acceptability of 1 online collective intelligence platform (Human Diagnosis Project; Human Dx).

Materials and methods

We conducted semistructured interviews with practicing primary care clinicians in a sample of the US community-based clinics to examine the acceptability and early usability of the collective intelligence online platform using standardized clinical cases and real-world clinical cases from the participants' own practice. We used an integrated inductive-deductive qualitative analysis approach to analyze the interview transcripts.

Results and discussion

Perceived usefulness, perceived accuracy, quality assurance, trust, and ease of use emerged as essential domains of acceptability required for providers to use a collective intelligence tool in clinical practice. Participants conveyed that the collective opinion should: (1) contribute to their clinical reasoning, (2) boost their confidence, (3) be generated in a timely manner, and (4) be relevant to their clinical settings and use cases. Trust in the technology platform and the clinical accuracy of its collective intelligence output emerged as an incontrovertible requirement for user acceptance and engagement.

Conclusion

We documented key requisites to building a collective intelligence technology platform that is trustworthy, useful, and acceptable to target end users for assistance in the diagnostic process. These key lessons may be applicable to other provider-facing decision support platforms.

SUBMITTER: Fontil V

PROVIDER: S-EPMC6952011 | biostudies-literature | 2019 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Testing and improving the acceptability of a web-based platform for collective intelligence to improve diagnostic accuracy in primary care clinics.

Fontil Valy V Radcliffe Kate K Lyson Helena C HC Ratanawongsa Neda N Lyles Courtney C Tuot Delphine D Yuen Kaeli K Sarkar Urmimala U

JAMIA open 20190201 1

<h4>Objectives</h4>Usable tools to support individual primary care clinicians in their diagnostic processes could help to reduce preventable harm from diagnostic errors. We conducted a formative study with primary care providers to identify key requisites to optimize the acceptability of 1 online collective intelligence platform (Human Diagnosis Project; Human Dx).<h4>Materials and methods</h4>We conducted semistructured interviews with practicing primary care clinicians in a sample of the US co ...[more]

PMID: 31984344

Similar Datasets

Project description:BACKGROUND:The diagnosis of pigmented skin lesion is error prone and requires domain-specific expertise, which is not readily available in many parts of the world. Collective intelligence could potentially decrease the error rates of nonexperts. OBJECTIVE:The aim of this study was to evaluate the feasibility and impact of collective intelligence for the detection of skin cancer. METHODS:We created a gamified study platform on a stack of established Web technologies and presented 4216 dermatoscopic images of the most common benign and malignant pigmented skin lesions to 1245 human raters with different levels of experience. Raters were recruited via scientific meetings, mailing lists, and social media posts. Education was self-declared, and domain-specific experience was tested by screening tests. In the target test, the readers had to assign 30 dermatoscopic images to 1 of the 7 disease categories. The readers could repeat the test with different lesions at their own discretion. Collective human intelligence was achieved by sampling answers from multiple readers. The disease category with most votes was regarded as the collective vote per image. RESULTS:We collected 111,019 single ratings, with a mean of 25.2 (SD 18.5) ratings per image. As single raters, nonexperts achieved a lower mean accuracy (58.6%) than experts (68.4%; mean difference=-9.4%; 95% CI -10.74% to -8.1%; P<.001). Collectives of nonexperts achieved higher accuracies than single raters, and the improvement increased with the size of the collective. A collective of 4 nonexperts surpassed single nonexperts in accuracy by 6.3% (95% CI 6.1% to 6.6%; P<.001). The accuracy of a collective of 8 nonexperts was 9.7% higher (95% CI 9.5% to 10.29%; P<.001) than that of single nonexperts, an improvement similar to single experts (P=.73). The sensitivity for malignant images increased for nonexperts (66.3% to 77.6%) and experts (64.6% to 79.4%) for answers given faster than the intrarater mean. CONCLUSIONS:A high number of raters can be attracted by elements of gamification and Web-based marketing via mailing lists and social media. Nonexperts increase their accuracy to expert level when acting as a collective, and faster answers correspond to higher accuracy. This information could be useful in a teledermatology setting.

Project description:BackgroundU.S. hospitals and dialysis centers are penalized for 30-day hospital readmissions of dialysis patients, despite little infrastructure to facilitate care transitions between these settings. We are developing a third-party web-based information exchange platform, DialysisConnect, to enable clinicians to view and exchange information about dialysis patients during admission, hospitalization, and discharge. This health information technology solution could serve as a flexible and relatively affordable solution for dialysis facilities and hospitals across the nation who are seeking to serve as true partners in the improved care of dialysis patients. The purpose of this study was to evaluate the perceived coherence of DialysisConnect to key clinical stakeholders, to prepare messaging for implementation.MethodsAs part of a hybrid effectiveness-implementation study guided by Normalization Process Theory, we collected data on stakeholder perceptions of continuity of care for patients receiving maintenance dialysis and a DialysisConnect prototype before completing development and piloting the system. We conducted four focus groups with stakeholders from one academic hospital and associated dialysis centers [hospitalists (n = 5), hospital staff (social workers, nurses, pharmacists; n = 9), nephrologists (n = 7), and dialysis clinic staff (social workers, nurses; n = 10)]. Transcriptions were analyzed thematically within each component of the construct of coherence (differentiation, communal specification, individual specification, and internalization).ResultsParticipants differentiated DialysisConnect from usual care variously as an information dashboard, a quick-exchange communication channel, and improved discharge information delivery; some could not differentiate it in terms of workflow. The purpose of DialysisConnect (communal specification) was viewed as fully coherent only for communicating outside of the same healthcare system. Current system workarounds were acknowledged as deterrents for practice change. All groups delegated DialysisConnect tasks (individual specification) to personnel besides themselves. Partial internalization of DialysisConnect was achieved only by dialysis clinic staff, based on experience with similar technology.ConclusionsImplementing DialysisConnect for clinical users in both settings will require presenting a composite picture of current communication processes from all stakeholder groups to correct single-group misunderstandings, as well as providing data about care transitions communication beyond the local context to ease resistance to practice change.

Project description:The collective intelligence of crowds could potentially be harnessed to address global challenges, such as biodiversity loss and species' extinction. For wisdom to emerge from the crowd, certain conditions are required. Importantly, the crowd should be diverse and people's contributions should be independent of one another. Here we investigate a global citizen-science platform-iNaturalist-on which citizens report on wildlife observations, collectively producing maps of species' spatiotemporal distribution. The organization of global platforms such as iNaturalist around local projects compromises the assumption of diversity and independence, and thus raises concerns regarding the quality of such collectively-generated data. We spent four years closely immersing ourselves in a local community of citizen scientists who reported their wildlife sightings on iNaturalist. Our ethnographic study involved the use of questionnaires, interviews, and analysis of archival materials. Our analysis revealed observers' nuanced considerations as they chose where, when, and what type of species to monitor, and which observations to report. Following a thematic analysis of the data, we organized observers' preferences and constraints into four main categories: recordability, community value, personal preferences, and convenience. We show that while some individual partialities can "cancel each other out", others are commonly shared among members of the community, potentially biasing the aggregate database of observations. Our discussion draws attention to the way in which widely-shared individual preferences might manifest as spatial, temporal, and crucially, taxonomic biases in the collectively-created database. We offer avenues for continued research that will help better understand-and tackle-individual preferences, with the goal of attenuating collective bias in data, and facilitating the generation of reliable state-of-nature reports. Finally, we offer insights into the broader literature on biases in collective intelligence systems.

Project description:BackgroundDiagnostic error in ambulatory care, a frequent cause of preventable harm, may be mitigated using the collective intelligence of multiple clinicians. The National Academy of Medicine has identified enhanced clinician collaboration and digital tools as a means to improve the diagnostic process.ObjectiveThis study aims to assess the efficacy of a collective intelligence output to improve diagnostic confidence and accuracy in ambulatory care cases (from primary care and urgent care clinic visits) with diagnostic uncertainty.MethodsThis is a pragmatic randomized controlled trial of using collective intelligence in cases with diagnostic uncertainty from clinicians at primary care and urgent care clinics in 2 health care systems in San Francisco. Real-life cases, identified for having an element of diagnostic uncertainty, will be entered into a collective intelligence digital platform to acquire collective intelligence from at least 5 clinician contributors on the platform. Cases will be randomized to an intervention group (where clinicians will view the collective intelligence output) or control (where clinicians will not view the collective intelligence output). Clinicians will complete a postvisit questionnaire that assesses their diagnostic confidence for each case; in the intervention cases, clinicians will complete the questionnaire after reviewing the collective intelligence output for the case. Using logistic regression accounting for clinician clustering, we will compare the primary outcome of diagnostic confidence and the secondary outcome of time with diagnosis (the time it takes for a clinician to reach a diagnosis), for intervention versus control cases. We will also assess the usability and satisfaction with the digital tool using measures adapted from the Technology Acceptance Model and Net Promoter Score.ResultsWe have recruited 32 out of our recruitment goal of 33 participants. This study is funded until May 2020 and is approved by the University of California San Francisco Institutional Review Board until January 2020. We have completed data collection as of June 2019 and will complete our proposed analysis by December 2019.ConclusionsThis study will determine if the use of a digital platform for collective intelligence is acceptable, useful, and efficacious in improving diagnostic confidence and accuracy in outpatient cases with diagnostic uncertainty. If shown to be valuable in improving clinicians' diagnostic process, this type of digital tool may be one of the first innovations used for reducing diagnostic errors in outpatient care. The findings of this study may provide a path forward for improving the diagnostic process.International registered report identifier (irrid)DERR1-10.2196/13151.

Dataset Information

Testing and improving the acceptability of a web-based platform for collective intelligence to improve diagnostic accuracy in primary care clinics.

Objectives

Materials and methods

Results and discussion

Conclusion

Publications

Testing and improving the acceptability of a web-based platform for collective intelligence to improve diagnostic accuracy in primary care clinics.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets