= Natural Language Inference =
== Annotation manual ==
[[Image(annotation.png,width=50%,right)]]

The following sentences are descriptions of photographs (machine translated from English to Czech). You cannot see the photos, however, you can imagine their content.

The aim is to decide for each pair of sentences (premise and hypothesis) whether:

 * the hypothesis **follows** from the premise (**entailment**)

 * the hypothesis **cannot follow** from the premise (**contradiction**)

 * the hypothesis **neither follows nor does not follow** from the premise (**neutral**)

 * one or both sentences from the pair **do not make sense** (**bad translation**)

The annotation is made via the [https://nlp.fi.muni.cz/projekty/labelstudio LabelStudio] tool. The procedure requires a double login process. 

First, log into the tool using the credentials obtained by email (`student@localhost`).

Next, log into the assigned data packages using your faculty credentials (x..../password).

=== How to annotate one sample ===

==== 1. Rule out bad translations

Read the premise and hypothesis and decide whether they make sense. If not, select **bad translation** and continue. You can check your understanding by revealing the original texts.

==== 2. Decide the sentence relationship

If the sentences make sense, decide whether the relation between them is **entailment**, **contradiction**, or **neutral**.

Please assume that the hypothesis and the premise are connected. For example, if the premise says something about a man and the hypothesis mentions a human, assume they are **the same entity** in the picture.

Also, use **common sense** when judging the impossibility of ''following''. For example, if the premise says "''a man rides a bicycle''", and the hypothesis says "''the man is washing his hands''", annotate it as **false**, even though you can imagine someone washing his hands while riding a bike (being a very uncommon situation).

Typically:
1. If the hypothesis is a more general statement than the premise, the result is **entailment**. 
1. If the premise adds new information or denotes an element of a hypothesis by a more specific name, it is **neutral** (cannot say). 
1. If the hypothesis and the premise describe a non-related action with different actors or they describe mutually exclusive actions, it is a **contradiction**.

==== 3. Select relevant parts ====

[[Image(entailment1.png,width=50%,right)]]


After deciding the sentence relationship, support your selection by selecting the relevant part in the premise, hypothesis, or both.

Click on the **relevant** button under the premise or hypothesis.

Mark one or more words (the tool selects whole words and removes trailing spaces from the selection).

In case of **entailment**, mark the elements that are the same thing but are described with different words. 

In case of **contradiction**, mark the elements that are mutually exclusive (e.g., man-woman) in the premise and hypothesis. 

In case of **neutral**, there is often some new information in the hypothesis, for instance a more specific term (woman-human) or some independent term.

It can happen that there are no such elements.


==== 4. Create a supporting relation ====

Set up the relation between the elements in the premise and hypothesis. Click the relevant element in premise (1), select the **Info** tab and click the **Create relation** button, a chain icon (2). Next, click the relevant part in the hypothesis (3).

[[Image(entailment2.png,width=50%,center)]]


[[Image(entailment3.png,width=50%,center)]]

Select the relation type: click on the Relations tab, and select the relation by clicking the triple dot button. Next, click into the **Select labels** box and select a relation.

In the case of **entailment**, the relation would be green, i.e., generalization or similar. In the case of **contradiction**, the relation would be red, i.e., exclusion.
In case of **neutral**, the relation would be orange. 


[[Image(entailment4.png,width=15%,center)]]


=== Submit ===

Finally, **Submit** your annotation.

[[Image(annotation_submit.png,width=50%,center)]]

=== Further information ===

You can check the instructions by using the **Instructions** button on the top when you need it.

[[Image(annotation_instruction.png,width=50%,center)]]