Parameterized Argumentative Reasoning Task (PART)

The task

Teams are tasked with solving reasoning puzzles concerning witness testimony. Based on a synthetic case text, the goal is to answer binary yes-or-no questions about a particular testimony, or to answer which witnesses can be believed. The puzzles are represented in natural language generated from formal argumentation graphs, where certain testimonies attack others thus rendering them “unbelievable”.

The underlying structure of each puzzle is a directed attack graph, which language models will need to extrapolate from the natural language case text representation in order to solve the task.

Subtasks

We split the overall goal of the shared task into two subtasks: a binary task and a multi-label task. Participants can take part in either or both.

Dataset

The dataset is based on the dynamic benchmarks introduced in Steging et al. (2025) and is available on Hugging Face. Each item has an id, a list of all witnesses, and two prompts describing the same case: binary_prompt and multi_label_prompt (testimonies may appear in a different order). Training items also include the answers (binary and multi_label); for test items these are withheld.

Every prompt states the rule that decides the answer: witnesses should always be believed unless there is believable testimony that they are lying (or another attack word, such as “not reliable”). The attack graphs are chains and star-shaped trees, so they contain no cycles, and this rule gives exactly one correct answer for every puzzle.

Puzzles vary in complexity, expressed primarily by the number of nodes and shapes of the underlying argumentation graph. Witness testimonies may be presented in order, shuffled, and irrelevant testimonies unrelated to the main argument may be included. Different types of attack relations, statements, witness names, irrelevant information, and prompt configurations may be expected to ensure generalizability.

Note on distribution: the test set may differ in distribution from the training set. Specifically, expect shifts in the number of nodes and shapes of the underlying argumentation graphs, the order in which witness testimonies are presented (in order or shuffled), the presence of irrelevant testimonies, the types of attack relations, the statements and witness names, and the prompt configurations. Teams should not assume that test puzzles follow the same distribution as the training data.

Modeling restrictions

• Models are restricted to open-weight models with a total of 8B parameters or less. Intermediate steps may use larger open-weight models, for instance for distillation.
• Mixture-of-experts models count as the total parameter count; ensembles of several models count as the sum of their weights.
• Teams may use any training data, models, and frameworks, provided the model weights can be released under a license with unrestricted non-commercial use (e.g. Apache, MIT, CC-BY).
• The use of closed commercial APIs is prohibited at every stage, including intermediate steps such as distillation or data generation.
• Per subtask, teams may submit a maximum of three runs (see how to submit).

How to submit

Submissions are made by email: send your predictions to argpart2027@gmail.com before the submission deadline (Dec 11, 2026, 23:59 AoE), using this exact subject line so it is not missed:

[PART2027 SUBMISSION] <team name> - run <n>

For example, [PART2027 SUBMISSION] Team Owl - run 1. Attach one JSONL file per run, named <team>_<run>.jsonl (for example owl_1.jsonl). You may send each run in its own email or several runs in one email; in the latter case, list all run numbers in the subject, e.g. [PART2027 SUBMISSION] Team Owl - runs 1, 2, 3.

Number your runs in one sequence per team (1, 2, 3, …), regardless of subtask. The fields in each file decide which subtask(s) the run counts toward, up to three runs per subtask. If you resend a run, the last version received before the deadline counts.

For questions, email the same address with the subject [PART2027 QUESTION] <short topic>.

Submission file format

Each run is a single JSON Lines file (.jsonl, UTF-8), with one JSON object per line and one line per test item:

{"id": "test-0001", "binary": "no", "multi_label": ["Melody"]}
{"id": "test-0002", "binary": "yes", "multi_label": ["Arwen", "Miles"]}
{"id": "test-0003", "binary": "no", "multi_label": []}

• id: the item id, copied exactly from the test file.
• binary (Task 1): your answer to the binary prompt, either "yes" or "no".
• multi_label (Task 2): a list of the names of all witnesses who can be believed, spelled as in the prompt. Order and capitalization do not matter.
• If you only take part in one subtask, leave the other field out. If a file contains both fields, it counts as one run for each subtask.
• Include every test item. A missing answer counts as wrong, and a file with unknown or duplicate ids is rejected.

Organizers

Send questions to argpart2027@gmail.com with the subject [PART2027 QUESTION] <short topic>. For submissions, see How to submit.

References

1. Cor Steging, Silja Renooij, and Bart Verheij. 2025. Parameterized argumentation-based reasoning tasks for benchmarking generative language models. In Proceedings of the Twentieth International Conference on Artificial Intelligence and Law, ICAIL ’25, pages 455–459, New York, NY, USA. Association for Computing Machinery.