Dataset Information

Humans in the Loop: Incorporating Expert and Crowd-Sourced Knowledge for Predictions Using Survey Data.

ABSTRACT: Survey data sets are often wider than they are long. This high ratio of variables to observations raises concerns about overfitting during prediction, making informed variable selection important. Recent applications in computer science have sought to incorporate human knowledge into machine-learning methods to address these problems. The authors implement such a "human-in-the-loop" approach in the Fragile Families Challenge. The authors use surveys to elicit knowledge from experts and laypeople about the importance of different variables to different outcomes. This strategy offers the option to subset the data before prediction or to incorporate human knowledge as scores in prediction models, or both together. The authors find that human intervention is not obviously helpful. Human-informed subsetting reduces predictive performance, and considered alone, approaches incorporating scores perform marginally worse than approaches that do not. However, incorporating human knowledge may still improve predictive performance, and future research should consider new ways of doing so.

SUBMITTER: Filippova A

PROVIDER: S-EPMC8112737 | biostudies-literature | 2019 Jan-Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Humans in the Loop: Incorporating Expert and Crowd-Sourced Knowledge for Predictions Using Survey Data.

Filippova Anna A Gilroy Connor C Kashyap Ridhi R Kirchner Antje A Morgan Allison C AC Polimis Kivan K Usmani Adaner A Wang Tong T

Socius : sociological research for a dynamic world 20190101

Survey data sets are often wider than they are long. This high ratio of variables to observations raises concerns about overfitting during prediction, making informed variable selection important. Recent applications in computer science have sought to incorporate human knowledge into machine-learning methods to address these problems. The authors implement such a "human-in-the-loop" approach in the Fragile Families Challenge. The authors use surveys to elicit knowledge from experts and laypeople ...[more]

PMID: 33981842

Similar Datasets

Project description:BackgroundVideo-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review.MethodsTwo surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10 min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT).ResultsLess than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = - 0.82 to 0.02), operative time (r = - 0.42 to 0.07), and the MIE-CAT scores (r = - 0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = - 0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42).ConclusionGOALS assessments by crowd workers lacked convergent validity and showed poor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.

Project description:IntroductionTo mitigate the COVID-19 pandemic and prevent overwhelming the healthcare system, social-distancing policies such as school closure, stay-at-home orders, and indoor dining closure have been utilized worldwide. These policies function by reducing the rate of close contact within populations and result in decreased human mobility. Adherence to social distancing can substantially reduce disease spread. Thus, quantifying human mobility and social-distancing compliance, especially at high temporal resolution, can provide great insight into the impact of social distancing policies.MethodsWe used the movement of individuals around New York City (NYC), measured via traffic levels, as a proxy for human mobility and the impact of social-distancing policies (i.e., work from home policies, school closure, indoor dining closure etc.). By data mining Google traffic in real-time, and applying image processing, we derived high resolution time series of traffic in NYC. We used time series decomposition and generalized additive models to quantify changes in rush hour/non-rush hour, and weekday/weekend traffic, pre-pandemic and following the roll-out of multiple social distancing interventions.ResultsMobility decreased sharply on March 14, 2020 following declaration of the pandemic. However, levels began rebounding by approximately April 13, almost 2 months before stay-at-home orders were lifted, indicating premature increase in mobility, which we term social-distancing fatigue. We also observed large impacts on diurnal traffic congestion, such that the pre-pandemic bi-modal weekday congestion representing morning and evening rush hour was dramatically altered. By September, traffic congestion rebounded to approximately 75% of pre-pandemic levels.ConclusionUsing crowd-sourced traffic congestion data, we described changes in mobility in Manhattan, NYC, during the COVID-19 pandemic. These data can be used to inform human mobility changes during the current pandemic, in planning for responses to future pandemics, and in understanding the potential impact of large-scale traffic interventions such as congestion pricing policies.

Dataset Information

Humans in the Loop: Incorporating Expert and Crowd-Sourced Knowledge for Predictions Using Survey Data.

Publications

Humans in the Loop: Incorporating Expert and Crowd-Sourced Knowledge for Predictions Using Survey Data.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets