Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds.

ABSTRACT: Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural networks (DNNs). In this work we propose learning IRT models using RPs generated from artificial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Parameters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive correlations. We demonstrate a use-case for latent difficulty item parameters, namely training s

SUBMITTER: Lalor JP 

PROVIDER: S-EPMC6892593 | biostudies-literature | 2019 Nov

REPOSITORIES: biostudies-literature