Language Resources and Evaluation

Mahibha, C. Jerin; Thenmozhi, Durairaj

doi: 10.1007/s10579-025-09899-0pmid: N/A

Hate speech represents the use of abusive or threatening speech or writing that expresses hatred towards an individual or a particular group based on race, religion, gender, or sexual orientation. The detection of hate speech has become important, as it can increase the crime rate in society. Manual identification of hate speech is a very tedious process, as social media is flooded with an enormous number of posts continuously. There is a notable amount of research in this field considering the English language, but this should also be extended to the Indian languages, as there is only limited research on this, considering the Indian languages. This paper provides a survey on the process associated with hate speech detection and the different methodologies associated with it, considering Indian languages. The survey has also categorized the research work on hate speech detection based on language, extracted features, classification techniques, and the corpus used for the research. The challenges associated with the process are also highlighted, as they can pave the way for further effective research on the detection of hate speech based on Indian languages.

Tang, Yao; Ali, Nor Liza

doi: 10.1007/s10579-025-09901-9pmid: N/A

Social media is flooded with a large volume of slang. Whilst many scholars have focused on the impact of slang through sentiment analysis, few have paid attention to the emotions represented by domain-specific slang. This study takes cat-related slang as an example to construct the CatSlang-SA dataset and evaluates its effectiveness in sentiment analysis tasks. The dataset consists of 10,025 posts from Reddit, each containing cat-related slang from the Urban Dictionary. The dataset is annotated with three sentiment labels: positive (1,545 instances), negative (2,528 instances), and neutral (5,952 instances). The dataset’s suitability is evaluated by the term frequency–inverse document frequency (TF-IDF) vectorisation technique and pre-trained transformer models, including four classical machine learning models and four pre-trained models. The results of the paired bootstrap test show that the DistilBERT-base-cased model achieved the best performance on the test set, with a Macro-F1 score of 0.9483. The transfer-comparative experiment demonstrates that, at the category level, the DistilBERT-base-cased model fine-tuned on CatSlang-SA significantly outperforms the model fine-tuned on the relevant portions of the Multi-Source, Multi-Language Social Media Dataset in the positive category. The main contribution of this study lies in constructing the first sentiment analysis dataset related to cat slang, providing a new resource for the study of domain-specific slang, and highlighting the role of domain-specific corpora in improving model performance.

Tomar, Vishakha; Dixit, Shubhra; Sangwan, Pardeep

doi: 10.1007/s10579-025-09877-6pmid: N/A

Spoken language identification (SLID) under real-world class imbalance remains a bottleneck for multilingual voice applications. This study introduces an incremental, imbalance-aware framework that fuses self-supervised XLS-R embeddings through a dual-stream gated-attention classifier and augments minority classes with diffusion-based audio synthesis. Class-balanced focal loss and elastic-weight consolidation jointly preserve recall for low-resource languages while preventing catastrophic forgetting when new languages are added. On the Indian languages audio dataset, the framework attains a macro-\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\hbox {F}_1$$\end{document} of 0.995 for three languages and sustains 0.818 after expansion to ten languages, surpassing the strongest published baseline by 6 percentage points. A Friedman aligned-rank test yields \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\chi ^2(3)=0.067$$\end{document}, \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$p=0.9955$$\end{document}, and a critical difference of 2.708 at \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\alpha =0.05$$\end{document}, indicating no significant differences among the compared configurations. These results demonstrate the framework’s robustness under severe class imbalance and its suitability for scalable, low-resource SLID deployment in linguistically diverse environments.

Ocal, Mustafa; Xie, Ning; Finlayson, Mark A.

doi: 10.1007/s10579-025-09883-8pmid: N/A

A timeline provides a total ordering of events and times, and is useful for a number of natural language understanding tasks. However, qualitative temporal graphs that can be derived directly from text—such as TimeML annotations—usually explicitly reveal only partial orderings of events and times. In this work, we apply prior work on solving point algebra problems to the task of extracting timelines from TimeML annotated texts, and develop an exact, end-to-end solution which we call tlex (TimeLine EXtraction). tlex transforms TimeML annotations into a collection of timelines arranged in a trunk-and-branch structure. Like what has been done in prior work, tlex checks the consistency of the temporal graph and solves it; however, it adds two novel functionalities. First, it identifies specific relations involved in an inconsistency (which could then be manually corrected) and, second, tlex performs a novel identification of sections of the timelines that have indeterminate order, information critical for downstream tasks such as aligning events from different timelines. We provide detailed descriptions and analysis of the algorithmic components in tlex, and conduct experimental evaluations by applying tlex to 385 TimeML annotated texts from four corpora. We show that 123 of the texts are inconsistent, 181 of them have more than one “real world” or main timeline, and there are 2,541 indeterminate sections across all four corpora. A sampling evaluation showed that tlex is 98–100% accurate with 95% confidence along five dimensions: the ordering of time-points, the number of main timelines, the placement of time-points on main versus subordinate timelines, the connecting point of branch timelines, and the location of the indeterminate sections. We provide the extracted timelines for all texts, the manual corrections to the temporal graphs of the inconsistent texts, as well as code to reproduce our experiments.

Fernandes, Leandro Carísio; Ribeiro, Leandro dos Santos; de Castro, Marcos Vinícius Borela; da Silva Pacheco, Leonardo Augusto; de Oliveira Sandes, Edans Flávius

doi: 10.1007/s10579-025-09881-wpmid: N/A

This paper introduces JurisTCU, a Brazilian Portuguese dataset for legal information retrieval (LIR). The dataset is freely available (https://huggingface.co/datasets/LeandroRibeiro/JurisTCU) and consists of 16,045 jurisprudential documents from the Brazilian Federal Court of Accounts, along with 150 queries annotated with relevance judgments. It addresses the scarcity of Portuguese-language LIR datasets with query relevance annotations. The queries are organized into three groups: real user keyword-based queries, synthetic keyword-based queries, and synthetic question-based queries. Relevance judgments were produced through a hybrid approach combining LLM-based scoring with expert domain validation. We used JurisTCU in 14 experiments using lexical search (document expansion methods) and semantic search (BERT-based and OpenAI embeddings). We show that the document expansion methods significantly improve the performance of standard BM25 search on this dataset, with improvements exceeding 45% in P@10, R@10, and nDCG@10 metrics when evaluating short keyword-based queries. Among the embedding models, the OpenAI models produced the best results, with improvements of approximately 70% in P@10, R@10, and nDCG@10 metrics for short keyword-based queries, suggesting that these dense embeddings capture semantic relationships in this domain, surpassing the reliance on lexical terms. Besides offering a dataset for the Portuguese-language IR research community, suitable for evaluating search systems, the results also contribute to enhancing a search system highly relevant to Brazilian citizens.

Ring, Hiram

doi: 10.1007/s10579-026-09902-2pmid: N/A

Existing datasets available for crosslinguistic investigations have tended to focus on large amounts of data for a small group of languages or a small amount of data for a large number of languages. This means that claims based on these datasets are limited in what they reveal about universal properties of the human language faculty. While this has begun to change through the efforts of projects seeking to develop tagged corpora for a large number of languages, such efforts are still constrained by limits on resources. The current paper reports on a large tagged parallel dataset which has been developed to partially address this issue. The taggedPBC contains POS-tagged parallel text data from more than 1,940 languages, representing 155 language families and 78 isolates, dwarfing previously available resources. The accuracy of particular tags in this dataset is shown to correlate well with existing SOTA taggers for high-resource languages (SpaCy, Trankit). Additionally, a novel measure derived from this dataset, the N1 ratio, correlates with expert determinations of intransitive word order in three typological databases (WALS, Grambank, AUTOTYP) such that a Gaussian Naive Bayes classifier trained on this feature can accurately identify basic intransitive word order for languages not in those databases. While much work is still needed to expand and develop this dataset, the taggedPBC is an important step to enable corpus-based crosslinguistic investigations, and is made available for research and collaboration via GitHub.

Trakuekul, Pontakorn; Leong, Wei Qi; Polpanumas, Charin; Sawatphol, Jitkapat; Tjhi, William Chandra; Rutherford, Attapol T.

doi: 10.1007/s10579-025-09898-1pmid: N/A

While coreference resolution is a well-established research area in Natural Language Processing (NLP), research focusing on Thai language remains limited due to the lack of large annotated corpora. In this work, we introduce ThaiCoref, a dataset for Thai coreference resolution. Our dataset comprises 777,271 tokens, 44,082 mentions and 10,429 entities across four text genres: university essays, newspapers, speeches, and Wikipedia. Our annotation scheme is built upon the OntoNotes benchmark with adjustments to address Thai-specific phenomena and cover more cases. Utilizing ThaiCoref, we train models employing a multilingual encoder and cross-lingual transfer techniques, achieving a best F1 score of 67.88% on the test set. Our error analysis reveals challenges posed by Thai’s unique morphological and syntactic features. To benefit the NLP community, we make the dataset and the model publicly available at http://www.github.com/nlp-chula/thai-coref.

Vicente, Aileen Joan; Amamampang, Theresse Faith; Lahaylahay, Dunn Dexter; Cheng, Charibeth

doi: 10.1007/s10579-025-09888-3pmid: N/A

Chavacano, formally known as Philippine Creole Spanish, is the only Creole language spoken in the Philippines. As with many languages, particularly Creoles, computational research on Chavacano remains limited due to the scarcity of available corpora. This paper presents ChavacanoMT, a benchmark corpus developed to support machine translation research on Philippine Creole Spanish. ChavacanoMT comprises 767,053 parallel sentences between Chavacano and its related languages: Spanish, Cebuano, Hiligaynon, Tagalog, and English. The corpus was constructed using data scraped from Bible translations and articles published on the Jehovah’s Witnesses website. This paper also presents the performance of a multilingual neural machine translation model generated using ChavacanoMT. We report an overall 17 BLEU score on a fine-tuned mT5 model, outperforming an mT5-based model trained from scratch. Our experiments show that ChavacanoMT can generate models on par with similar systems that translate between English and some Philippines languages despite having fewer sentence samples used in training. We also report an improved Chavacano translation to and from its related languages that can be used as benchmark data. In particular, we highlight more than 13 BLEU points of improvement in the translation from Chavacano to English. The study opens avenues for exploring cross-linguistic interactions of Chavacano and its related languages in its translation that may benefit other low-resource languages.

Ul Hassan, Muhammad Naeem; Yu, Zhengtao; Ullah, Khalil; Wang, Jian; Li, Ying; Gao, Shengxiang; Yang, Shuwan; Mao, Cunli

doi: 10.1007/s10579-025-09889-2pmid: N/A

To address the issue of semantic distortion in translation caused by structural differences between Urdu and English, a linguistically informed large-model-based Urdu-English neural machine translation method is proposed. Given that current general-purpose large models are predominantly trained on high-resource languages like English, they struggle to capture the specific grammar, complex morphological variations, and idiomatic expressions of Urdu. In this paper, first designs an adaptive multi-layer linguistic injection method that integrates lexical, syntactic, and semantic features into the large language model. These features include key linguistic markers such as tense, gender, and politeness; syntactic transformations between Subject-Verb-Object and Subject-Object-Verb structures; as well as idiomatic expressions and cultural differences. Next, a multi-knowledge integration and prompting technique is employed to dynamically adjust translations based on sentence complexity. Finally, Low-Rank Adaptation is used for efficient parameter fine-tuning, further enhancing translation performance. Experimental results demonstrate that this method significantly outperforms traditional neural machine translation systems, achieving a notable improvement in + 4.7 BLEU scores.

Song, Jayoung; Lim, KyungTae; Park, Jungyeul

doi: 10.1007/s10579-025-09882-9pmid: N/A

Despite growing global interest in Korean language education, learner corpora tailored to Korean L2 writing remain scarce. This paper introduces KoLLA v2.0, the first Korean learner corpus to incorporate both multi-reference GEC annotations and rubric-based essay scoring. We extend the original KoLLA dataset by adding a second human correction for each sentence, creating the first multi-reference GEC resource for Korean. This design captures the variability of valid corrections in an agglutinative and morphologically complex language, enabling fairer and more realistic evaluation of GEC systems. In parallel, we enrich the corpus with rubric-based scores based on criteria from the Korean National Language Institute, providing standardized, multi-dimensional assessments of grammatical accuracy, coherence, and lexical diversity. These enhancements position KoLLA v2.0 as a standardized resource for research in Korean L2 learning and instruction, and as a benchmark for evaluating automated error correction and essay scoring systems.