Dataset Information

Investigating cross-lingual training for offensive language detection.

ABSTRACT: Platforms that feature user-generated content (social media, online forums, newspaper comment sections etc.) have to detect and filter offensive speech within large, fast-changing datasets. While many automatic methods have been proposed and achieve good accuracies, most of these focus on the English language, and are hard to apply directly to languages in which few labeled datasets exist. Recent work has therefore investigated the use of cross-lingual transfer learning to solve this problem, training a model in a well-resourced language and transferring to a less-resourced target language; but performance has so far been significantly less impressive. In this paper, we investigate the reasons for this performance drop, via a systematic comparison of pre-trained models and intermediate training regimes on five different languages. We show that using a better pre-trained language model results in a large gain in overall performance and in zero-shot transfer, and that intermediate training on other languages is effective when little target-language data is available. We then use multiple analyses of classifier confidence and language model vocabulary to shed light on exactly where these gains come from and gain insight into the sources of the most typical mistakes.

SUBMITTER: Pelicon A

PROVIDER: S-EPMC8237322 | biostudies-literature | 2021

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Investigating cross-lingual training for offensive language detection.

Pelicon Andraž A Shekhar Ravi R Škrlj Blaž B Purver Matthew M Pollak Senja S

PeerJ. Computer science 20210625

Platforms that feature user-generated content (social media, online forums, newspaper comment sections etc.) have to detect and filter offensive speech within large, fast-changing datasets. While many automatic methods have been proposed and achieve good accuracies, most of these focus on the English language, and are hard to apply directly to languages in which few labeled datasets exist. Recent work has therefore investigated the use of <i>cross-lingual transfer learning</i> to solve this prob ...[more]

PMID: 34239970

Similar Datasets

Project description:Speech change is a biometric marker for Parkinson's disease (PD). However, evaluating speech variability across diverse languages is challenging. We aimed to develop a cross-language algorithm differentiating between PD patients and healthy controls using a Taiwanese and Korean speech data set. We recruited 299 healthy controls and 347 patients with PD from Taiwan and Korea. Participants with PD underwent smartphone-based speech recordings during the "on" phase. Each Korean participant performed various speech texts, while the Taiwanese participant read a standardized, fixed-length article. Korean short-speech (≦15 syllables) and long-speech (> 15 syllables) recordings were combined with the Taiwanese speech dataset. The merged dataset was split into a training set (controls vs. early-stage PD) and a validation set (controls vs. advanced-stage PD) to evaluate the model's effectiveness in differentiating PD patients from controls across languages based on speech length. Numerous acoustic and linguistic speech features were extracted and combined with machine learning algorithms to distinguish PD patients from controls. The area under the receiver operating characteristic (AUROC) curve was calculated to assess diagnostic performance. Random forest and AdaBoost classifiers showed an AUROC 0.82 for distinguishing patients with early-stage PD from controls. In the validation cohort, the random forest algorithm maintained this value (0.90) for discriminating advanced-stage PD patients. The model showed superior performance in the combined language cohort (AUROC 0.90) than either the Korean (AUROC 0.87) or Taiwanese (AUROC 0.88) cohorts individually. However, with another merged speech data set of short-speech recordings < 25 characters, the diagnostic performance to identify early-stage PD patients from controls dropped to 0.72 and showed a further limited ability to discriminate advanced-stage patients. Leveraging multifaceted speech features, including both acoustic and linguistic characteristics, could aid in distinguishing PD patients from healthy individuals, even across different languages.

Dataset Information

Investigating cross-lingual training for offensive language detection.

Publications

Investigating cross-lingual training for offensive language detection.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets