SQAC-SQUAD 2024: Question answering

This is an extractive text comprehension task formulated in terms of question-answering. The task consists of answering questions about a text in such a way that the answer is a fragment extracted directly from the text. The texts are academic news from Cambridge University from several scientific domains. In all cases, the answers are fragments of the text and questions that cannot be answered from the text are not included.

 

Language
English
NLP topic
question answering
Abstract task
Sequence labeling
Dataset
SQUAD-SQAC 2024 ES
Year
2024
Ranking metric
F1

Task results

System F1 Accuracy MacroF1 Pearson correlation ICM
Roberta large 0.4626 0.4626 0.4626 0.4626 0.46
Xlm roberta large 0.4163 0.4163 0.4163 0.4163 0.42
Roberta base 0.3746 0.3746 0.3746 0.3746 0.37
Xlm roberta base 0.3251 0.3251 0.3251 0.3251 0.33
Ixa ehu ixambert base cased 0.3222 0.3222 0.3222 0.3222 0.32
Bert base cased 0.2996 0.2996 0.2996 0.2996 0.30
Bert base multilingual cased 0.2948 0.2948 0.2948 0.2948 0.29
Distilbert base uncased 0.2670 0.2670 0.2670 0.2670 0.27
Distilbert base multilingual cased 0.1994 0.1994 0.1994 0.1994 0.20

If you have published a result better than those on the list, send a message to odesia-comunicacion@lsi.uned.es indicating the result and the DOI of the article, along with a copy of it if it is not published openly.