EXIST-2023: Source intention (soft-soft)

EXIST is a series of scientific events and shared tasks on sexism identification in social networks. EXIST aims to capture sexism in a broad sense, from explicit misogyny to other subtle expressions that involve implicit sexist behaviours.This task includes a soft-soft evaluation in which the probability of each label predicted by the system is compared with the probability defined from the disagreement in the gold standard annotation.

Publication
Plaza, L. et al. (2023). Overview of EXIST 2023 – Learning with Disagreement for Sexism Identification and Characterization. In: Arampatzis, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2023. Lecture Notes in Computer Science, vol 14163. Springer, Cham. https://doi.org/10.1007/978-3-031-42448-9_23
Competition
EXIST: sEXism Identification in Social neTworks
Language
English
URL Task
http://nlp.uned.es/exist2023/
NLP topic
hate detection
Abstract task
Classification
Dataset
EXIST-2023-EN
Year
2023
Publication link
https://link.springer.com/cha…
Ranking metric
ICM

Task results

System Precision Recall F1 CEM Accuracy MacroPrecision MacroRecall MacroF1 RMSE MicroPrecision MicroRecall MicroF1 MAE MAP UAS LAS MLAS BLEX Pearson correlation Spearman correlation MeasureC BERTScore EMR Exact Match F0.5 Hierarchical F ICM MeasureC Propensity F Reliability Sensitivity Sentiment Graph F1 WAC b2 erde30 sent weighted f1
Xlm roberta large 0.4029 0.4029 0.4029 0.4029 0.40
Roberta large 0.3962 0.3962 0.3962 0.3962 0.40
Distilbert base uncased 0.3844 0.3844 0.3844 0.3844 0.38
Roberta base 0.3779 0.3779 0.3779 0.3779 0.38
Xlm roberta base 0.3735 0.3735 0.3735 0.3735 0.37
Bert base cased 0.3701 0.3701 0.3701 0.3701 0.37
Ixa ehu ixambert base cased 0.3622 0.3622 0.3622 0.3622 0.36
Bert base multilingual cased 0.3618 0.3618 0.3618 0.3618 0.36
Distilbert base multilingual cased 0.3576 0.3576 0.3576 0.3576 0.36

If you have published a result better than those on the list, send a message to odesia-comunicacion@lsi.uned.es indicating the result and the DOI of the article, along with a copy of it if it is not published openly.