The task aims at finding the best techniques to identify and categorize propagandistic tweets from governmental and diplomatic sources on a dataset of 9501 tweets in Spanish, posted by authorities of China, Russia, United States and the European Union. The task consists on categorising propagandistic tweets in 15 propagandistic techniques.
Pablo Moral, Guillermo Marco, Julio Gonzalo, Jorge Carrillo-de-Albornoz, Iván Gonzalo-Verdugo (2023) Overview of DIPROMATS 2023: automatic detection and characterization of propaganda techniques in messages from diplomats and authorities of world powers. Procesamiento del Lenguaje Natural, Revista nº 71, septiembre de 2023, pp. 397-407.
Publication
Pablo Moral, Guillermo Marco, Julio Gonzalo, Jorge Carrillo-de-Albornoz, Iván Gonzalo-Verdugo (2023) Overview of DIPROMATS 2023: automatic detection and characterization of propaganda techniques in messages from diplomats and authorities of world powers. Procesamiento del Lenguaje Natural, Revista nº 71, septiembre de 2023, pp. 397-407.
Language
Spanish
NLP topic
Abstract task
Dataset
Year
2023
Publication link
Ranking metric
ICM
Task results
| System | F1 Sort ascending | Accuracy | MacroF1 | Pearson correlation | ICM |
|---|---|---|---|---|---|
| Hermes-3-Llama-3.1-8B_2 | 0.4855 | 0.4855 | 0.4855 | 0.4855 | 0.49 |
| Hermes-3-Llama-3.1-8B | 0.4675 | 0.4675 | 0.4675 | 0.4675 | 0.47 |
| XLM-RoBERTa-large | 0.4581 | 0.4581 | 0.4581 | 0.4581 | 0.46 |
| XLM-RoBERTa-large-2 | 0.4581 | 0.4581 | 0.4581 | 0.4581 | 0.46 |
| XLM-RoBERTa-large-v3 | 0.4581 | 0.4581 | 0.4581 | 0.4581 | 0.46 |
| Xlm roberta large | 0.4527 | 0.4527 | 0.4527 | 0.4527 | 0.45 |
| Gemma-2B-IT | 0.4303 | 0.4303 | 0.4303 | 0.4303 | 0.43 |
| PlanTL GOB ES roberta large bne | 0.3894 | 0.3894 | 0.3894 | 0.3894 | 0.39 |
| PlanTL GOB ES roberta base bne | 0.2944 | 0.2944 | 0.2944 | 0.2944 | 0.29 |
| Dccuchile bert base spanish wwm cased | 0.2931 | 0.2931 | 0.2931 | 0.2931 | 0.29 |
