Touché Task 3: Image Retrieval for Arguments

Synopsis

Register now

Data

This task uses a focused crawl of about 20,000 images (and associated web pages) as document collection. See the collection's README for more information on its contents and file formats. We may still modify the collection until the end of 2021 based on feedback and own considerations. The collection also contains a small set of training judgements for basic testing. [collection] [training judgements]

Evaluation

Systems are evaluated on Touché topics 1–50 by the ratio of relevant images among 20 retrieved images (Precision) for each topic, namely 10 images for each stance (file format explained in the README). [topics]

Submission

We encourage participants to use TIRA for their submissions to allow for a better reproducibility (see the Quickstart section below). Email submission is allowed as a fallback. For each topic and stance, include 10 retrieved images. Each team can submit up to 5 different runs.

The submission format adapts the standard TREC format. Each line corresponds to an image retrieved for some topic and stance at a certain rank, making a run file 1000 lines long (50 topics, 2 stances, 10 ranks). Each line contains the following fields, separated by single whitespaces:

  • The topic number (1 to 50).
  • The stance ("PRO" or "CON").
  • The image's ID (corresponds to the name of the image's directory in the collection; always 17 characters long and starts with "I").
  • The rank (1 to 10 in increasing order per topic and stance). Not used in this year's evaluation.
  • A score (integer or floating point; non-increasing per topic and stance). Not used in this year's evaluation.
  • A tag that identifies your group and the method you used to produce the run.
For example:
1 PRO I000330ba4ea0ad13 1 17.89 myGroupMyMethod
1 PRO I0005e6fe00ea17fd 2 16.43 myGroupMyMethod
...
1 CON I0009d5f038fe6f2e 1 15.89 myGroupMyMethod
1 CON I000f34bd3f8cb030 2 14.43 myGroupMyMethod
...

If you have questions, please ask in the forum. You will get a combined TIRA-and-forum account on registration. Announcements are published here.

TIRA Quickstart

Coming soon

Task Committee