SemEval-2024 Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes

Back to summary metadata
dc.contributor.affiliationUniversity of Helsinki - Mickus, Timothee
dc.contributor.affiliationUniversity of Helsinki - Zosa, Elaine
dc.contributor.affiliationUniversity of Helsinki - Vázquez, Raúl
dc.contributor.affiliationUniversity of Helsinki - Vahtola, Teemu
dc.contributor.affiliationUniversity of Helsinki - Tiedemann, Jörg
dc.contributor.affiliationUniversité Bretagne Sud - Segonne, Vincent
dc.contributor.affiliationUniversity of Milano-Bicocca - Raganato, Alessandro
dc.contributor.affiliationUniversity of Pennsylvania - Apidianaki, Marianna
dc.contributor.authorMickus, Timothee
dc.contributor.authorZosa, Elaine
dc.contributor.authorVázquez, Raúl
dc.contributor.authorVahtola, Teemu
dc.contributor.authorTiedemann, Jörg
dc.contributor.authorSegonne, Vincent
dc.contributor.authorRaganato, Alessandro
dc.contributor.authorApidianaki, Marianna
dc.date.accessioned2025-03-24T15:11:23Z
dc.date.issued2024-05-16
dc.date.issued2024-05-16
dc.descriptionTask description: SHROOM participants will need to detect grammatically sound output that contains incorrect semantic information (i.e. unsupported or inconsistent with the source input), with or without having access to the model that produced the output. Overview of the task: The modern NLG landscape is plagued by two interlinked problems: On the one hand, our current neural models have a propensity to produce inaccurate but fluent outputs; on the other hand, our metrics are most apt at describing fluency, rather than correctness. This leads neural networks to “hallucinate”, i.e., produce fluent but incorrect outputs that we currently struggle to detect automatically. For many NLG applications, the correctness of an output is however mission critical. For instance, producing a plausible-sounding translation that is inconsistent with the source text puts in jeopardy the usefulness of a machine translation pipeline. With our shared task, we hope to foster the growing interest in this topic in the community. With SHROOM we adopt a post hoc setting, where models have already been trained and outputs already produced: participants will be asked to perform binary classification to identify cases of fluent overgeneration hallucinations in two different setups: model-aware and model-agnostic tracks. That is, participants must detect grammatically sound outputs which contain incorrect or unsupported semantic information, inconsistent with the source input, with or without having access to the model that produced the output. To that end, we will provide participants with a collection of checkpoints, inputs, references and outputs of systems covering three different NLG tasks: definition modeling (DM), machine translation (MT) and paraphrase generation (PG), trained with varying degrees of accuracy. The development set will provide binary annotations from at least five different annotators and a majority vote gold label.
dc.identifierhttps://doi.org/10.5281/zenodo.11202503
dc.identifier.urihttps://hydatakatalogi-test-24.it.helsinki.fi/handle/123456789/9151
dc.rightsOpen
dc.rights.licensecc-by-4.0
dc.subjectSemeval-2024
dc.subjectHallucinations
dc.titleSemEval-2024 Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes
dc.typedataset
dc.typedataset

Files

Repositories

Zenodo