EXIST-2023: Source intention (soft-soft)

EXIST is a series of scientific events and shared tasks on sexism identification in social networks. EXIST aims to capture sexism in a broad sense, from explicit misogyny to other subtle expressions that involve implicit sexist behaviours.This task includes a soft-soft evaluation in which the probability of each label predicted by the system is compared with the probability defined from the disagreement in the gold standard annotation.

Publication
Plaza, L. et al. (2023). Overview of EXIST 2023 – Learning with Disagreement for Sexism Identification and Characterization. In: Arampatzis, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2023. Lecture Notes in Computer Science, vol 14163. Springer, Cham. https://doi.org/10.1007/978-3-031-42448-9_23
Language
English
NLP topic
Abstract task
Dataset
Year
2023
Ranking metric
ICM

Task results

System F1 Sort ascending Accuracy MacroF1 Pearson correlation ICM
Xlm roberta large 0.4029 0.4029 0.4029 0.4029 0.40
Roberta large 0.3962 0.3962 0.3962 0.3962 0.40
Distilbert base uncased 0.3844 0.3844 0.3844 0.3844 0.38
Roberta base 0.3779 0.3779 0.3779 0.3779 0.38
Xlm roberta base 0.3735 0.3735 0.3735 0.3735 0.37
Bert base cased 0.3701 0.3701 0.3701 0.3701 0.37
Ixa ehu ixambert base cased 0.3622 0.3622 0.3622 0.3622 0.36
Bert base multilingual cased 0.3618 0.3618 0.3618 0.3618 0.36
Distilbert base multilingual cased 0.3576 0.3576 0.3576 0.3576 0.36

If you have published a result better than those on the list, send a message to odesia-comunicacion@lsi.uned.es indicating the result and the DOI of the article, along with a copy of it if it is not published openly.