= Natural Language Inference =
== Annotation manual ==
[[Image(annotation.png,width=50%,right)]]

The aim is to decide for each pair of sentences (premise, hypothesis) if:

 * the hypothesis follows from the premise (**entailment**)

 * the hypothesis cannot follow from the premise (**contradiction**)

 * the hypothesis neither follow nor does not follow from the premise (**neutral**)

 * one or both sentences from the pair do not make sense (**bad translation**)

The annotation is made via the tool [https://nlp.fi.muni.cz/projekty/labelstudio/projects LabelStudio]. You log into one of the packages using your NLP credentials.

=== How to annotate one sample ===

Read the premise and hypothesis and decide whether they make sense. If not, select bad translation and continue.
If the sentences make sense, decide whether the relation between them is entailment, contradiction, or neutral.

Next, support your selection by selecting the relevant part in the premise, hypothesis or both.

==== Select relevant part ====

Click on the **relevant** button under the premise or hypothesis.

Mark one or more words (the tool selects whole words and removes trailing spaces from the selection).

In case of *neutral*, there is often some new information in the hypothesis. Mark this information with no relation to the premise.

==== Create relation ====

In case of *entailment*, mark the elements that are the same thing but are described with different words. In case of *contradiction*, mark the elements that are mutually exclusive (e.g., man-woman) in the premise and hypothesis. In case of *neutral*, there may be a more specific term (e.g., woman-human) or some independent term; the relation would be orange.

[[Image(entailment1.png,width=50%)]]

(It can happen, there are no such elements.)

[[Image(entailment2.png,width=50%)]]

Set up relation between the elements in premise and hypothesis. Click the relevant element in premise (1) and click the *Create relation* button a chain icon (2). Next, click the relevant part in the hypothesis (3).

[[Image(entailment3.png,width=50%)]]

Select the relation type: click on the Relations tab, select the relation by clicking the triple dot button. Next, click into the *Select labels* box and select a relation.

In case of *entailment*, the relation would be green, i.e., generalization or similar. In case of *contradiction*, the relation would be red, i.e. exclusion.

[[Image(entailment4.png,width=15%)]]


=== Submit ===

Finally, **Submit** your annotation.

[[Image(annotation_submit.png,width=50%)]]

=== Further information ===

You can check the instructions by using the **Instructions** button on the top every time you need it.

[[Image(annotation_instruction.png,width=50%)]]