Natural Language Inference

Annotation manual

The aim is to decide for each pair of sentences (premise, hypothesis) if:

• the hypothesis follows from the premise (entailment)

• the hypothesis cannot follow from the premise (contradiction)

• the hypothesis neither follow nor does not follow from the premise (neutral)

• one or both sentences from the pair do not make sense (bad translation)

The annotation is made via the tool ​LabelStudio. You log into one of the packages using your NLP credentials.

How to annotate one sample

Read the premise and hypothesis and decide whether they make sense. If not, select bad translation and continue. If the sentences make sense, decide whether the relation between them is entailment, contradiction, or neutral.

Next, support your selection by selecting the relevant part in the premise, hypothesis or both.

Finally, Submit your annotation.

You can check the instructions by using the Instructions button on the top every time you need it.