Publications

Home / Publications

Selected Publications

feb 2024

"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models

This paper investigates to what extent the first token probabilities of large language models match their final answers to multiple-choice questions.
dec 2023

Establishing Trustworthiness: Rethinking Tasks and Model Evaluation

Decades of NLP research have traditionally compartmentalized linguistic tasks, but the emergence of large language models is reshaping this approach, emphasizing the need for holistic, task-agnostic evaluation methods that prioritize trustworthiness.
may 2023

A Survey of Corpora for Germanic Low-Resource Languages and Dialects

This paper provides an overview of more than 80 corpora to support NLP research in resource-poor and non-standardized languages of the Germanic language family.

All Publications

jun 2024

Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models

Mondorf, Philipp and Plank, Barbara

arXiv
may 2024

Slot and Intent Detection Resources for Bavarian and Lithuanian: Assessing Translations vs Natural Queries to Digital Assistants

Winkler, Miriam and Juozapaityte, Virginija and van der Goot, Rob and Plank, Barbara

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
may 2024

Sebastian, Basti, Wastl?! Recognizing Named Entities in Bavarian Dialectal Data

Peng, Siyao and Sun, Zihang and Shan, Huangyan and Kolm, Marie and Blaschke, Verena and Artemova, Ekaterina and Plank, Barbara

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
may 2024

MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebank

Blaschke, Verena and Kovačić, Barbara and Peng, Siyao and Schütze, Hinrich and Plank, Barbara

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
may 2024

IndirectQA: Understanding Indirect Answers to Implicit Polar Questions in French and Spanish

Müller, Christin and Plank, Barbara

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
may 2024

How to Encode Domain Information in Relation Classification

Bassignana, Elisa and Gascou, Viggo Unmack and Laustsen, Frida Nøhr and Kristensen, Gustav and Petersen, Marie Haahr and van der Goot, Rob and Plank, Barbara

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
apr 2024

Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models – A Survey

Mondorf, Philipp and Plank, Barbara

arXiv
apr 2024

MaiNLP at SemEval-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatedness

Zhou, Shijia and Shan, Huangyan and Plank, Barbara and Litschko, Robert

arXiv
apr 2024

Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think

Wang, Xinpeng and Hu, Chengzhi and Ma, Bolei and Röttger, Paul and Plank, Barbara

arXiv
mar 2024

EEVEE: An Easy Annotation Tool for Natural Language Processing

Sorensen, Axel and Peng, Siyao and Plank, Barbara and Van Der Goot, Rob

Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII)
mar 2024

More Labels or Cases? Assessing Label Variation in Natural Language Inference

Gruber, Cornelia and Hechinger, Katharina and Assenmacher, Matthias and Kauermann, Göran and Plank, Barbara

Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language
mar 2024

Rethinking Skill Extraction in the Job Market Domain using Large Language Models

Nguyen, Khanh and Zhang, Mike and Montariol, Syrielle and Bosselut, Antoine

Proceedings of the First Workshop on Natural Language Processing for Human Resources (NLP4HR 2024)
mar 2024

Deep Learning-based Computational Job Market Analysis: A Survey on Skill Extraction and Classification from Job Postings

Senger, Elena and Zhang, Mike and Goot, Rob and Plank, Barbara

Proceedings of the First Workshop on Natural Language Processing for Human Resources (NLP4HR 2024)
mar 2024

Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations

Peng, Siyao and Sun, Zihang and Loftus, Sebastian and Plank, Barbara

Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language
mar 2024

Entity Linking in the Job Market Domain

Zhang, Mike and Goot, Rob and Plank, Barbara

Findings of the Association for Computational Linguistics: EACL 2024
mar 2024

Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?

Baan, Joris and Fernández, Raquel and Plank, Barbara and Aziz, Wilker

Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
mar 2024

Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties

Artemova, Ekaterina and Blaschke, Verena and Plank, Barbara

Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
mar 2024

Donkii: Characterizing and Detecting Errors in Instruction-Tuning Datasets

Weber, Leon and Litschko, Robert and Artemova, Ekaterina and Plank, Barbara

Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII)
mar 2024

JobSkape: A Framework for Generating Synthetic Job Postings to Enhance Skill Matching

Magron, Antoine and Dai, Anna and Zhang, Mike and Montariol, Syrielle and Bosselut, Antoine

Proceedings of the First Workshop on Natural Language Processing for Human Resources (NLP4HR 2024)
mar 2024

NNOSE: Nearest Neighbor Occupational Skill Extraction

Zhang, Mike and van der Goot, Rob and Kan, Min-Yen and Plank, Barbara

Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
dec 2023

Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training

Müller-Eberstein, Max and van der Goot, Rob and Plank, Barbara and Titov, Ivan

Findings of the Association for Computational Linguistics: EMNLP 2023
dec 2023

What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability

Giulianelli, Mario and Baan, Joris and Aziz, Wilker and Fernández, Raquel and Plank, Barbara

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)
dec 2023

ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation

Wang, Xinpeng and Plank, Barbara

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
dec 2023

Establishing Trustworthiness: Rethinking Tasks and Model Evaluation

Litschko, Robert and Müller-Eberstein, Max and van der Goot, Rob and Weber-Genzel, Leon and Plank, Barbara

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
dec 2023

From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification

Xu, Shanshan and T.y.s.s, Santosh and Ichim, Oana and Risini, Isabella and Plank, Barbara and Grabmair, Matthias

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
jul 2023

Boosting Zero-shot Cross-lingual Retrieval by Training on Artificially Code-Switched Data

Litschko, Robert and Artemova, Ekaterina and Plank, Barbara

Findings of the Association for Computational Linguistics: ACL 2023
jul 2023

SemEval-2023 Task 11: Learning with Disagreements (LeWiDi)

Leonardelli, Elisa and Abercrombie, Gavin and Almanea, Dina and Basile, Valerio and Fornaciari, Tommaso and Plank, Barbara and Rieser, Verena and Uma, Alexandra and Poesio, Massimo

Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)
jul 2023

ActiveAED: A Human in the Loop Improves Annotation Error Detection

Weber, Leon and Plank, Barbara

Findings of the Association for Computational Linguistics: ACL 2023
jul 2023

Silver Syntax Pre-training for Cross-Domain Relation Extraction

Bassignana, Elisa and Ginter, Filip and Pyysalo, Sampo and van der Goot, Rob and Plank, Barbara

Findings of the Association for Computational Linguistics: ACL 2023
jul 2023

How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives

Wang, Xinpeng and Weissweiler, Leonie and Schütze, Hinrich and Plank, Barbara

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
jul 2023

ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain

Zhang, Mike and van der Goot, Rob and Plank, Barbara

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
may 2023

Low-resource Bilingual Dialect Lexicon Induction with Large Language Models

Artemova, Ekaterina and Plank, Barbara

Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)
may 2023

A Survey of Corpora for Germanic Low-Resource Languages and Dialects

Blaschke, Verena and Schuetze, Hinrich and Plank, Barbara

Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)
may 2023

Multi-CrossRE A Multi-Lingual Multi-Domain Dataset for Relation Extraction

Bassignana, Elisa and Ginter, Filip and Pyysalo, Sampo and van der Goot, Rob and Plank, Barbara

Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)
may 2023

Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages

Blaschke, Verena and Schütze, Hinrich and Plank, Barbara

Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)
may 2023

Findings of the VarDial Evaluation Campaign 2023

Aepli, Noëmi and Çöltekin, Çağrı and Van Der Goot, Rob and Jauhiainen, Tommi and Kazzaz, Mourhaf and Ljubešić, Nikola and North, Kai and Plank, Barbara and Scherrer, Yves and Zampieri, Marcos

Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)
dec 2022

Experimental Standards for Deep Learning in Natural Language Processing Research

Ulmer, Dennis and Bassignana, Elisa and Müller-Eberstein, Max and Varab, Daniel and Zhang, Mike and van der Goot, Rob and Hardmeier, Christian and Plank, Barbara

Findings of the Association for Computational Linguistics: EMNLP 2022
dec 2022

Spectral Probing

Müller-Eberstein, Max and van der Goot, Rob and Plank, Barbara

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
dec 2022

The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation

Plank, Barbara

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
dec 2022

Stop Measuring Calibration When Humans Disagree

Baan, Joris and Aziz, Wilker and Plank, Barbara and Fernandez, Raquel

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
dec 2022

Evidence \textgreater Intuition: Transferability Estimation for Encoder Selection

Bassignana, Elisa and Müller-Eberstein, Max and Zhang, Mike and Plank, Barbara

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
dec 2022

CrossRE: A Cross-Domain Dataset for Relation Extraction

Bassignana, Elisa and Plank, Barbara

Findings of the Association for Computational Linguistics: EMNLP 2022

Publications

Selected Publications

"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models

Establishing Trustworthiness: Rethinking Tasks and Model Evaluation

A Survey of Corpora for Germanic Low-Resource Languages and Dialects

All Publications