Dataset Information

Enhancing African low-resource languages: Swahili data for language modelling.

ABSTRACT: Language modelling using neural networks requires adequate data to guarantee quality word representation which is important for natural language processing (NLP) tasks. However, African languages, Swahili in particular, have been disadvantaged and most of them are classified as low resource languages because of inadequate data for NLP. In this article, we derive and contribute unannotated Swahili dataset, Swahili syllabic alphabet and Swahili word analogy dataset to address the need for language processing resources especially for low resource languages. Therefore, we derive the unannotated Swahili dataset by pre-processing raw Swahili data using a Python script, formulate the syllabic alphabet and develop the Swahili word analogy dataset based on an existing English dataset. We envisage that the datasets will not only support language models but also other NLP downstream tasks such as part-of-speech tagging, machine translation and sentiment analysis.

SUBMITTER: Shikali CS

PROVIDER: S-EPMC7339006 | biostudies-literature | 2020 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Enhancing African low-resource languages: Swahili data for language modelling.

Shikali Casper S CS Mokhosi Refuoe R

Data in brief 20200630

Language modelling using neural networks requires adequate data to guarantee quality word representation which is important for natural language processing (NLP) tasks. However, African languages, Swahili in particular, have been disadvantaged and most of them are classified as low resource languages because of inadequate data for NLP. In this article, we derive and contribute unannotated Swahili dataset, Swahili syllabic alphabet and Swahili word analogy dataset to address the need for language ...[more]

PMID: 32671155

Similar Datasets

Project description:Recent advancements in large language models (LLMs) have opened new possibilities for developing conversational agents (CAs) in various subfields of mental healthcare. However, this progress is hindered by limited access to high-quality training data, often due to privacy concerns and high annotation costs for low-resource languages. A potential solution is to create human-AI annotation systems that utilize extensive public domain user-to-user and user-to-professional discussions on social media. These discussions, however, are extremely noisy, necessitating the adaptation of LLMs for fully automatic cleaning and pre-classification to reduce human annotation effort. To date, research on LLM-based annotation in the mental health domain is extremely scarce. In this article, we explore the potential of zero-shot classification using four LLMs to select and pre-classify texts into topics representing psychiatric disorders, in order to facilitate the future development of CAs for disorder-specific counseling. We use 64,404 Russian-language texts from online discussion threads labeled with seven most commonly discussed disorders: depression, neurosis, paranoia, anxiety disorder, bipolar disorder, obsessive-compulsive disorder, and borderline personality disorder. Our research shows that while preliminary data filtering using zero-shot technology slightly improves classification, LLM fine-tuning makes a far larger contribution to its quality. Both standard and natural language inference (NLI) modes of fine-tuning increase classification accuracy by more than three times compared to non-fine-tuned training with preliminarily filtered data. Although NLI fine-tuning achieves slightly higher accuracy (0.64) than the standard approach, it is six times slower, indicating a need for further experimentation with NLI hypothesis engineering. Additionally, we demonstrate that lemmatization does not affect classification quality and that multilingual models using texts in their original language perform slightly better than English-only models using automatically translated texts. Finally, we introduce our dataset and model as the first openly available Russian-language resource for developing conversational agents in the domain of mental health counseling.

Project description:ImportanceEffective communication between patients and health care teams is essential in the health care setting for delivering optimal cancer care and increasing cancer awareness. While the significance of communication in health care is widely acknowledged, the topic is largely understudied within African settings.ObjectiveTo assess how the medical language of cancer and oncology translates into African languages and what these translations mean within their cultural context.Design, setting, and participantsIn this multinational survey study in Africa, health professionals, community health workers, researchers, and scientists involved in cancer care and research and traditional healers were invited to participate in an online survey on a voluntary basis through online platforms. The survey provided 16 cancer and oncologic terms used in cancer diagnosis and treatment (eg, cancer, radiotherapy) to participants, mostly health care workers, who were asked to provide these terms in their local languages (if the terms existed) followed by a direct or close translation of the meaning in English. The survey was open from February to April 2023.Main outcomes and measuresPatterns of meaning that recurred across languages were identified using thematic analysis of 16 English-translated terms categorized into 5 themes (neutral, negative, positive, phonetic or borrowed, and unknown).ResultsA total of 107 responses (response rate was unavailable given the open and widespread distribution strategy) were collected from 32 countries spanning 44 African languages, with most participants (63 [59%]) aged 18 to 40 years; 54 (50%) were female. Translations for cancer were classified as phonetic or borrowed (34 [32%]), unknown (30 [28%]), neutral (24 [22%]), and negative (19 [18%]), with the latter category including universal connotations of fear, tragedy, incurability, and fatality. Similar elements connoting fear or tragedy were found in translations of terms such as malignant, chronic, and radiotherapy. The term radiotherapy yielded a high percentage of negative connotations (24 [22%]), with a prevailing theme of describing the treatment as being burned or burning with fire, heat, or electricity, which may potentially hinder treatment.Conclusions and relevanceIn this survey study of cancer communication and the translation of oncology terminology in African languages, the findings suggest that the terminology may contribute to fear, health disparities, and barriers to care and pose communication difficulties for health professionals. The results reinforce the need for culturally sensitive cancer terminology for improving cancer awareness and communication.

Dataset Information

Enhancing African low-resource languages: Swahili data for language modelling.

Publications

Enhancing African low-resource languages: Swahili data for language modelling.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets