Natural Language Inference

Annotation manual

The following sentences are descriptions of photographs. You cannot see the photos, however, you can imagine their content.

The aim is to decide for each pair of sentences (premise, hypothesis) if:

• the hypothesis follows from the premise (entailment)

• the hypothesis cannot follow from the premise (contradiction)

• the hypothesis neither follows nor does not follow from the premise (neutral)

• one or both sentences from the pair do not make sense (bad translation)

The annotation is made via the tool ​LabelStudio. Log into the tool using the credentials obtained by email (student@localhost).

Next, log into one of the packages using your NLP credentials.

How to annotate one sample

Read the premise and hypothesis and decide whether they make sense. If not, select bad translation and continue. You can check your understanding by revealing the original texts.

If the sentences make sense, decide whether the relation between them is entailment, contradiction, or neutral.

Please assume the hypothesis and premise are connected. E.g., if the premise says something about a man, and the hypothesis mentions a human, assume they are the same entity in the picture.

Also, use common sense when judging the impossibility of following. E.g., if the premise says a man rides a bicycle, and the hypothesis says the man is washing his hands, annotate it as false, even though you can imagine someone washing his hands while riding a bike.

Typically, if the hypothesis is a more general statement than the premise, it is entailment. If the premise adds new information or denotes an element of a hypothesis by a more specific name, it is neutral. If the hypothesis and the premise describe a non-related action with different actors or they describe mutually exclusive actions, it is a contradiction.

Next, support your selection by selecting the relevant part in the premise, hypothesis, or both.

Select relevant part

Click on the relevant button under the premise or hypothesis.

Mark one or more words (the tool selects whole words and removes trailing spaces from the selection).

In case of neutral, there is often some new information in the hypothesis. Mark this information with no relation to the premise.

Create relation

In case of entailment, mark the elements that are the same thing but are described with different words. In case of contradiction, mark the elements that are mutually exclusive (e.g., man-woman) in the premise and hypothesis. In case of neutral, there may be a more specific term (e.g., woman-human) or some independent term; the relation would be orange.

(It can happen that there are no such elements.)

Set up the relation between the elements in the premise and hypothesis. Click the relevant element in premise (1) and click the Create relation button, a chain icon (2). Next, click the relevant part in the hypothesis (3).

Select the relation type: click on the Relations tab, and select the relation by clicking the triple dot button. Next, click into the Select labels box and select a relation.

In the case of entailment, the relation would be green, i.e., generalization or similar. In the case of contradiction, the relation would be red, i.e., exclusion.

Submit

Finally, Submit your annotation.

Further information

You can check the instructions by using the Instructions button on the top every time you need it.