SQAC-SQUAD 2024: Question Answering

This is an extractive text comprehension task formulated in terms of question-answering. The task consists of answering questions about a text in such a way that the answer is a fragment extracted directly from the text. The texts are academic news from the CSIC (Centro Superior de Investigaciones Científicas) from several scientific domains. In all cases, the answers are fragments of the text and questions that cannot be answered from the text are not included.

 

Language
Spanish
NLP topic
question answering
Abstract task
Sequence labeling
Dataset
SQUAD-SQAC 2024 ES
Year
2024
Ranking metric
F1

Task results

System Precision Recall F1 CEM Accuracy MacroPrecision MacroRecall MacroF1 RMSE MicroPrecision MicroRecall MicroF1 MAE MAP UAS LAS MLAS BLEX Pearson correlation Spearman correlation MeasureC BERTScore EMR Exact Match F0.5 Hierarchical F ICM MeasureC Propensity F Reliability Sensitivity Sentiment Graph F1 WAC b2 erde30 sent weighted f1
Hermes-3-Llama-3.1-8B 0.6791 0.6791 0.6791 0.6791 0.68
Hermes-3-Llama-3.1-8B_2 0.6791 0.6791 0.6791 0.6791 0.68
Gemma-2B-IT 0.4738 0.4738 0.4738 0.4738 0.47
PlanTL GOB ES roberta large bne 0.4640 0.4640 0.4640 0.4640 0.46
Xlm roberta large 0.4589 0.4589 0.4589 0.4589 0.46
Bertin roberta base spanish 0.4172 0.4172 0.4172 0.4172 0.42
Dccuchile bert base spanish wwm cased 0.4118 0.4118 0.4118 0.4118 0.41
PlanTL GOB ES roberta base bne 0.4061 0.4061 0.4061 0.4061 0.41
XLM-RoBERTa-large-v3 0.4000 0.4000 0.4000 0.4000 0.40
Xlm roberta base 0.3691 0.3691 0.3691 0.3691 0.37

If you have published a result better than those on the list, send a message to odesia-comunicacion@lsi.uned.es indicating the result and the DOI of the article, along with a copy of it if it is not published openly.