Touché Task 1: Argument Retrieval for Controversial Questions
- Task: Given a query about a controversial topic, retrieve and rank a relevant pair of sentences from a collection of arguments.
- Input: [data]
- Submission: [submit]
The goal of Task 1 is to support users who search for arguments to be used in conversations (e.g., getting an overview of pros and cons or just looking for arguments in line with a user's stance). However, in contrast to previous iterations of this task, we now require retrieving a pair of sentences from a collection of arguments. Each sentence in this pair must be argumentative (e.g., a claim, a supporting premise, or a conclusion). Also, to encourage diversity of the retrieved results, sentences in this pair may come from two different arguments.
The corpus for Task 1 (available here) is a pre-processed version of the args.me corpus (version 2020-04-01) where each argument is split into sentences; you may index these sentences (and the complete arguments if you wish to) with your favorite retrieval system. To ease participation, you may also directly use the args.me search engine's API for a baseline retrieval and then extract the candidate pair of sentences.
Participants can access the query relevance judgments and quality judgments of the past edition of this task (where complete arguments were retrieved instead of sentence pairs). This data may serve as a source of distant supervision, but there is no strict requirement to harness it.
Touché 2021 topics (only titles) for Task 1 can be downloaded from here. The topics are provided as an XML file.
<title>Is human activity primarily responsible for global climate change?</title>
[Download quality judgments from Touché 2021 (args.me corpus version 2020-04-01)]
Be sure to retrieve a pair of ''strong'' argumentative sentences. Our human assessors will label the retrieved pairs manually, both for their general topical relevance and for their argument quality, i.e.,: (1) whether each sentence in the retrieved pair is argumentative (claim, supporting premise, or conclusion), (2) whether the sentence pair forms a coherent text (sentences in a pair must not contradict each other), (3) whether the sentence pair forms a short summary of the corresponding arguments from which these sentences come from; each sentence in the pair must ideally be the most representative / most important sentence of its corresponding argument.The format of the relevance/quality judgment file:
qid 0 pair rel
qid: The topic number.
0: Unused, always 0.
pair: The pair of sentence IDs (from the provided version of the args.me corpus).
rel: The relevance judgment: -2 non-argument (spam), 0 (not relevant) to 3 (highly relevant). The quality judgment: 0 (low) to 3 (high).
We encourage participants to use TIRA for their submissions to allow for a better reproducibility. Please also have a look at our TIRA quickstart—in case of problems we will be able to assist you. Even though the preferred way of run submission is TIRA, in case of problems you may also submit runs via email. We will try to quickly review your TIRA or email submissions and provide feedback.
Runs may be either automatic or manual. An automatic run must not "manipulate" the topic titles via manual intervention. A manual run is anything that is not an automatic run. Upon submission, please let us know which of your runs are manual. For each topic, include up to 1,000 retrieved documents. Each team can submit up to 5 different runs.
The submission format for the task will follow the standard TREC format:
qid Q0 pair rank score tag
qid: The topic number.
Q0: Unused, should always be Q0.
pair: The pair of sentence IDs (from the provided version of the args.me corpus) returned by your system for the topic
rank: The rank the document is retrieved at.
score: The score (integer or floating point) that generated the ranking. The score must be in descending (non-increasing) order: it is important to handle tie scores.
tag: A tag that identifies your group and the method you used to produce the run.
An example run for Task 1 is:
1 Q0 Sf9294c83-Af186e851_sent1||Sf9294c83-A9a4e056e_sent3 1 17.89 myGroupMyMethod
1 Q0 Sf9294c83-A9a4e056e_sent2||Sf9294c83-A9a4e056e_sent4 2 16.43 myGroupMyMethod
1 Q0 S96f2396e-Aaf079b43_sent1||Sf9294c83-Af186e851_sent5 3 16.42 myGroupMyMethod
Participants have to upload (through SSH or RDP) their retrieval models in a dedicated TIRA virtual machine, so that their runs can be reproduced and so that they can be easily applied to different data (of same format) in the future. You can find host ports for your VM in the web interface, same login as to your VM. If you cannot connect to your VM, please make sure it is powered on: you can check and power on your machine in the web interface.
Your software is expected to accept two arguments:
- An input directory (named
$inputDatasetin TIRA). The variable
$inputDatasetpoints to a directory which contains one csv file:
args_processed_04_01.csv(the provided corpus), and
topics.xml(the topics for which documents should be retrieved).
- An output directory (named
$outputDirin TIRA). Your software should create a standard trec run file in
args_processed_04_01.csvfile to produce the run file.
As soon as your Software is installed in your VM, you can register it in TIRA. Assume that your software is started with a bash script in your home directory called
my-software.shwhich expects an argument
-ispecifying the input directory, and an argument
-ospecifying the output directory. Click on "Add software" and specify the command
./my-software.sh -i $inputDataset -o $outputDir. The other fields can stay with default settings.
Click on "Run" to execute your software in TIRA. Note that your VM will not be accessible while your system is running – it will be “sandboxed”, detached from the internet, and after the run the state of the VM before the run will be restored. Your run will be reviewed and evaluated by the organizers.
NOTE: By submitting your software you retain full copyrights. You agree to grant us usage rights for evaluation of the corresponding data generated by your software. We agree not to share your software with a third party or use it for any purpose other than research.
Once the run of your system completes, please also run the evaluator on the output of your system to verify that your output is a valid submission. These are two separate actions and both should be invoked through the web interface of TIRA. You don’t have to install the evaluator in your VM. It is already prepared in TIRA. You should see it in the web interface, under your software, labeled “Evaluator”. Before clicking the “Run” button, you will use a drop-down menu to select the “Input run”, i.e. one of the completed runs of your system. The output files from the selected run will be evaluated.
You can see and download STDOUT and STDERR as well as the outputs of your system.
In the evaluator run you will see only STDOUT and STDERR, which will tell you if one or more of your output files is not valid.
If you think something went wrong with your run, send us an e-mail.
Additionally, we review your submissions and contact you on demand.
You can register more than one system (“software/ model”) per virtual machine using the web interface. TIRA gives systems automatic names “Software 1”, “Software 2” etc. You can perform several runs per system.