Trends and Focus of Machine Learning Applications for Health Research.
ABSTRACT: Importance:The use of machine learning applications related to health is rapidly increasing and may have the potential to profoundly affect the field of health care. Objective:To analyze submissions to a popular machine learning for health venue to assess the current state of research, including areas of methodologic and clinical focus, limitations, and underexplored areas. Design, Setting, and Participants:In this data-driven qualitative analysis, 166 accepted manuscript submissions to the Third Annual Machine Learning for Health workshop at the 32nd Conference on Neural Information Processing Systems on December 8, 2018, were analyzed to understand research focus, progress, and trends. Experts reviewed each submission against a rubric to identify key data points, statistical modeling and analysis of submitting authors was performed, and research topics were quantitatively modeled. Finally, an iterative discussion of topics common in submissions and invited speakers at the workshop was held to identify key trends. Main Outcomes and Measures:Frequency and statistical measures of methods, topics, goals, and author attributes were derived from an expert review of submissions guided by a rubric. Results:Of the 166 accepted submissions, 58 (34.9%) had clinician involvement and 83 submissions (50.0%) that focused on clinical practice included clinical collaborators. A total of 97 data sets (58.4%) used in submissions were publicly available or required a standard registration process. Clinical practice was the most common application area (70 manuscripts [42.2%]), with brain and mental health (25 [15.1%]), oncology (21 [12.7%]), and cardiovascular (19 [11.4%]) being the most common specialties. Conclusions and Relevance:Trends in machine learning for health research indicate the importance of well-annotated, easily accessed data and the benefit from greater clinician involvement in the development of translational applications.
Project description:The study evaluated whether a modified version of the information literacy Valid Assessment of Learning in Undergraduate Education (VALUE) rubric would be useful for assessing the information literacy skills of graduate health sciences students.Through facilitated calibration workshops, an interdepartmental six-person team of librarians and faculty engaged in guided discussion about the meaning of the rubric criteria. They applied the rubric to score student work for a peer-review essay assignment in the "Information Literacy for Evidence-Based Practice" course. To determine inter-rater reliability, the raters participated in a follow-up exercise in which they independently applied the rubric to ten samples of work from a research project in the doctor of physical therapy program: the patient case report assignment.For the peer-review essay, a high level of consistency in scoring was achieved for the second workshop, with statistically significant intra-class correlation coefficients above 0.8 for 3 criteria: "Determine the extent of evidence needed," "Use evidence effectively to accomplish a specific purpose," and "Access the needed evidence." Participants concurred that the essay prompt and rubric criteria adequately discriminated the quality of student work for the peer-review essay assignment. When raters independently scored the patient case report assignment, inter-rater agreement was low and statistically insignificant for all rubric criteria (kappa=-0.16, p>0.05-kappa=0.12, p>0.05).While the peer-review essay assignment lent itself well to rubric calibration, scorers had a difficult time with the patient case report. Lack of familiarity among some raters with the specifics of the patient case report assignment and subject matter might have accounted for low inter-rater reliability. When norming, it is important to hold conversations about search strategies and expectations of performance. Overall, the authors found the rubric to be appropriate for assessing information literacy skills of graduate health sciences students.
Project description:This report summarizes the Joint FDA-MIPS Workshop on Methods for the Evaluation of Imaging and Computer-Assist Devices. The purpose of the workshop was to gather information on the current state of the science and facilitate consensus development on statistical methods and study designs for the evaluation of imaging devices to support US Food and Drug Administration submissions. Additionally, participants expected to identify gaps in knowledge and unmet needs that should be addressed in future research. This summary is intended to document the topics that were discussed at the meeting and disseminate the lessons that have been learned through past studies of imaging and computer-aided detection and diagnosis device performance.
Project description:The clinical distinction between Alzheimer's disease (AD) and behavioral variant frontotemporal dementia (bvFTD) remains challenging and largely dependent on the experience of the clinician. This study investigates whether objective machine learning algorithms using supportive neuroimaging and neuropsychological clinical features can aid the distinction between both diseases. Retrospective neuroimaging and neuropsychological data of 166 participants (54 AD; 55 bvFTD; 57 healthy controls) was analyzed via a Naïve Bayes classification model. A subgroup of patients (n = 22) had pathologically-confirmed diagnoses. Results show that a combination of gray matter atrophy and neuropsychological features allowed a correct classification of 61.47% of cases at clinical presentation. More importantly, there was a clear dissociation between imaging and neuropsychological features, with the latter having the greater diagnostic accuracy (respectively 51.38 vs. 62.39%). These findings indicate that, at presentation, machine learning classification of bvFTD and AD is mostly based on cognitive and not imaging features. This clearly highlights the urgent need to develop better biomarkers for both diseases, but also emphasizes the value of machine learning in determining the predictive diagnostic features in neurodegeneration.
Project description:Introduction:Social networking sites (or social media [SM]) are powerful web-based technologies used to bolster communication. SM have changed not only how information is communicated but also the dissemination and reception of a variety of topics. This workshop highlighted the benefits of SM for clinician educators. The use of SM was explored as a way to maximize opportunities for clinician educators to network, establish themselves as experts, and build a national reputation leading to promotion. The target audience for this submission is faculty developers who would like to implement a similar workshop, and clinician-educator faculty motivated by promotion and advancement. Methods:The training workshop involved an interactive session, with approximately 20 minutes of content, 20 minutes of individual and small-group activities, and 15 minutes of large-group discussion. The effectiveness of the workshop was evaluated by asking participants to complete a postsession survey of SM knowledge, attitude, and action. Results:Survey responses (n = 14) demonstrated an increase in participants' knowledge of SM platforms, ability to identify benefits of SM, skills to disseminate their work, and eagerness to build their personal brand. Discussion:This workshop provided a foundation for clinician educators to think strategically about SM use in ways that highlight access to a broader network of colleagues and potential collaborators and that influence the impact of publications and work.
Project description:Human learning is one of the main topics in psychology and cognitive neuroscience. The analysis of experimental data, e.g. from category learning experiments, is a major challenge due to confounding factors related to perceptual processing, feedback value, response selection, as well as inter-individual differences in learning progress due to differing strategies or skills. We use machine learning to investigate (Q1) how participants of an auditory category-learning experiment evolve towards learning, (Q2) how participant performance saturates and (Q3) how early we can differentiate whether a participant has learned the categories or not. We found that a Gaussian Mixture Model describes well the evolution of participant performance and serves as basis for identifying influencing factors of task configuration (Q1). We found early saturation trends (Q2) and that CatBoost, an advanced classification algorithm, can separate between participants who learned the categories and those who did not, well before the end of the learning session, without much degradation of separation quality (Q3). Our results show that machine learning can model participant dynamics, identify influencing factors of task design and performance trends. This will help to improve computational models of auditory category learning and define suitable time points for interventions into learning, e.g. by tutorial systems.
Project description:Evaluating the biomedical literature and health-related websites for quality are challenging information retrieval tasks. Current commonly used methods include impact factor for journals, PubMed's clinical query filters and machine learning-based filter models for articles, and PageRank for websites. Previous work has focused on the average performance of these methods without considering the topic, and it is unknown how performance varies for specific topics or focused searches. Clinicians, researchers, and users should be aware when expected performance is not achieved for specific topics. The present work analyzes the behavior of these methods for a variety of topics. Impact factor, clinical query filters, and PageRank vary widely across different topics while a topic-specific impact factor and machine learning-based filter models are more stable. The results demonstrate that a method may perform excellently on average but struggle when used on a number of narrower topics. Topic-adjusted metrics and other topic robust methods have an advantage in such situations. Users of traditional topic-sensitive metrics should be aware of their limitations.
Project description:<h4>Background</h4>In Canada, requests for public reimbursement of cancer drugs are predominately initiated by pharmaceutical manufacturers. Clinician-led submissions provide a mechanism to initiate the drug funding process when industry does not submit a request for funding consideration. Although such requests are resource-intensive to produce, Cancer Care Ontario (cco) has the capacity to facilitate clinician-led submissions. In 2014, cco began developing a cancer drug prioritization framework that allocates resources to systematically address a growing number of clinician-identified funding gaps with clinician-led submissions.<h4>Methods</h4>Cancer site-specific drug advisory committees established by cco consist of health care practitioners whose roles include identifying and prioritizing funding gaps. The committees submit their identified gaps to a cross-cancer-site prioritization exercise in which the requests are ranked based on a set of guiding principles derived from health technology assessment. The requests are then sequentially allocated the resources needed to meet submission requirements. Whether the funding gap is of provincial or pan-Canadian relevance determines where the submission is filed for assessment.<h4>Results</h4>Since its inception, the cco framework has identified 17 funding gaps in 9 cancer sites. In 4 prioritizations, the framework supported 6 submissions. As of June 2018, the framework had contributed to the eventual funding of more than 9 new drug-indication pairs, with more awaiting funding consideration.<h4>Conclusions</h4>The cco prioritization framework has enabled clinicians to effectively and systematically identify, prioritize, and fill funding gaps not addressed by industry. Ultimately, the framework helps to ensure that patients can access evidence-informed and cost-effective therapies. The framework will continue to evolve as it encounters new challenges, including funding requests for rare indications.
Project description:Introduction:Advance care planning (ACP) is an essential discussion between a health care provider and a patient about their future care during serious illness. In clinical practice, high-quality ACP may be addressed with an interprofessional approach. Role-playing is an ideal method to practice both ACP and shared decision-making before having these conversations with patients. Methods:This asynchronous role-playing workshop is prefaced with two prerecorded 25-minute videos for faculty and student preparation with one introducing ACP concepts, and one depicting a patient-physician ACP discussion. During the 2-hour workshop, students complete four role-play ACP scenarios with the following roles: patient, family member, nurse, nurse practitioner, and physician. Students rotate through different roles guided by scripts, and have a fact sheet for each scenario detailing prognostic information for disease processes. The role-play works optimally with three nursing students, three medical students, and one faculty facilitator per group. Facilitators are provided with a timeline, a guide for debriefing, and an evaluation rubric. Results:The survey data from 85 students spread over four course offerings were summarized. When asked both if learning objectives were met, and to reflect on the clinical relevance, teaching effectiveness, and the overall workshop experience, most participants reported a good to excellent rating. Discussion:This role-play activity allows students to practice ACP and shared decision-making, both with patient and family presence, and in premeeting rounds with the health care team. ACP exposure during student training helps trainees recognize the impact of high-quality interprofessional conversations on the care patients want and ultimately receive.
Project description:Introduction:Although studies surveying internal medicine (IM) residency program directors identify geriatric women's health as an essential curriculum topic, there are limited published women's health curricula for IM residents. Our IM residency program performed a needs assessment, which revealed that the majority of residents were unsatisfied with our current curricula and most were not confident managing geriatric women's health. We developed and assessed a structured curriculum to improve IM residents' knowledge and confidence in addressing geriatric women's health. Methods:This 2-hour interactive workshop used the jigsaw teaching method (a cooperative learning strategy where peers deliver specific content in teams) to teach 84 categorical IM residents of all PGY levels about the diagnosis and management of menopause, osteoporosis, urinary incontinence, and abnormal uterine bleeding. Participants completed a pretest and immediate posttest to assess knowledge and confidence about the targeted topics. We compared baseline and postworkshop responses using chi-square and Wilcoxon signed rank tests. Results:Seventy-four (88%) IM residents completed the pretest, and 62 (74%) completed the posttest. Mean knowledge scores improved from 51% to 69% (p < .0001). Residents who reported feeling somewhat confident or confident in addressing women's health topics increased from 14% to 44% (p < .0001). The majority were satisfied or very satisfied with the workshop (94%) and requested additional women's health education (92%). Discussion:Our results suggest that workshops using the jigsaw teaching method can effectively increase IM resident knowledge and confidence in managing geriatric women's health.
Project description:INTRODUCTION: Publishing a case report demonstrates scholarly productivity for trainees and clinician-educators. AIM: To assess the learning outcomes from a case report writing workshop. SETTING: Medical students, residents, fellows and clinician-educators attending a workshop. PROGRAM DESCRIPTION: Case report writing workshop conducted nine times at different venues. PROGRAM EVALUATION: Before and after each workshop, participants self-rated their perceived competence to write a case report, likelihood of submitting a case report to a meeting or for publication in the next 6-12 months, and perceived career benefit of writing a case report (on a five-point Likert scale). The 214 participants were from 3 countries and 27 states or provinces; most participants were trainees (64.5 %). Self-rated competence for writing a case report improved from a mean of 2.5 to 3.5 (a 0.99 increase; 95% CI, 0.88-1.12, p < 0.001). The perceived likelihood of submitting a case report, and the perceived career benefit of writing one, also showed statistically significant improvements (p = 0.002, p = 0.001; respectively). Nine of 98 participants published a case report 16-41 months after workshop completion. DISCUSSION: The workshop increased participants' perception that they could present or publish a case report.