Changes between Version 12 and Version 13 of AnnotationManual/CZSNLI


Ignore:
Timestamp:
Mar 3, 2024, 9:14:52 PM (3 months ago)
Author:
Ales Horak
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • AnnotationManual/CZSNLI

    v12 v13  
    33[[Image(annotation.png,width=50%,right)]]
    44
    5 The following sentences are descriptions of photographs. You cannot see the photos, however, you can imagine their content.
     5The following sentences are descriptions of photographs (machine translated from English to Czech). You cannot see the photos, however, you can imagine their content.
    66
    7 The aim is to decide for each pair of sentences (premise, hypothesis) if:
     7The aim is to decide for each pair of sentences (premise and hypothesis) whether:
    88
    9  * the hypothesis follows from the premise (**entailment**)
     9 * the hypothesis **follows** from the premise (**entailment**)
    1010
    11  * the hypothesis cannot follow from the premise (**contradiction**)
     11 * the hypothesis **cannot follow** from the premise (**contradiction**)
    1212
    13  * the hypothesis neither follows nor does not follow from the premise (**neutral**)
     13 * the hypothesis **neither follows nor does not follow** from the premise (**neutral**)
    1414
    15  * one or both sentences from the pair do not make sense (**bad translation**)
     15 * one or both sentences from the pair **do not make sense** (**bad translation**)
    1616
    17 The annotation is made via the tool [https://nlp.fi.muni.cz/projekty/labelstudio/projects LabelStudio]. Log into the tool using the credentials obtained by email (`student@localhost`).
     17The annotation is made via the [https://nlp.fi.muni.cz/projekty/labelstudio/projects LabelStudio] tool. The procedure requires double login process.
    1818
    19 Next, log into one of the packages using your NLP credentials.
     19First, log into the tool using the credentials obtained by email (`student@localhost`).
     20
     21Next, log into one of the data packages using your NLP credentials.
    2022
    2123=== How to annotate one sample ===
    2224
    23 Read the premise and hypothesis and decide whether they make sense. If not, select bad translation and continue. You can check your understanding by revealing the **original texts**.
     25==== 1. Rule out bad translations
    2426
    25 If the sentences make sense, decide whether the relation between them is entailment, contradiction, or neutral.
     27Read the premise and hypothesis and decide whether they make sense. If not, select **bad translation** and continue. You can check your understanding by revealing the original texts.
    2628
    27 Please assume the hypothesis and premise are connected. E.g., if the premise says something about a man, and the hypothesis mentions a human, assume they are the same entity in the picture.
     29==== 2. Decide the sentence relationship
    2830
    29 Also, use common sense when judging the impossibility of following. E.g., if the premise says a man rides a bicycle, and the hypothesis says the man is washing his hands, annotate it as false, even though you can imagine someone washing his hands while riding a bike.
     31If the sentences make sense, decide whether the relation between them is **entailment**, **contradiction**, or **neutral**.
    3032
    31 Typically, if the hypothesis is a more general statement than the premise, it is **entailment**. If the premise adds new information or denotes an element of a hypothesis by a more specific name, it is **neutral**. If the hypothesis and the premise describe a non-related action with different actors or they describe mutually exclusive actions, it is a **contradiction**.
     33Please assume that the hypothesis and the premise are connected. For example, if the premise says something about a man and the hypothesis mentions a human, assume they are **the same entity** in the picture.
    3234
    33 Next, support your selection by selecting the relevant part in the premise, hypothesis, or both.
     35Also, use **common sense** when judging the impossibility of ''following''. For example, if the premise says "''a man rides a bicycle''", and the hypothesis says "''the man is washing his hands''", annotate it as **false**, even though you can imagine someone washing his hands while riding a bike (being a very uncommon situation).
    3436
    35 ==== Select relevant part ====
     37Typically:
     381. If the hypothesis is a more general statement than the premise, the result is **entailment**.
     391. If the premise adds new information or denotes an element of a hypothesis by a more specific name, it is **neutral** (cannot say).
     401. If the hypothesis and the premise describe a non-related action with different actors or they describe mutually exclusive actions, it is a **contradiction**.
     41
     42==== 3. Select relevant parts ====
     43
     44[[Image(entailment1.png,width=50%,right)]]
     45
     46
     47After deciding the sentence relationship, support your selection by selecting the relevant part in the premise, hypothesis, or both.
    3648
    3749Click on the **relevant** button under the premise or hypothesis.
    … …  
    4153In case of **neutral**, there is often some new information in the hypothesis. Mark this information with no relation to the premise.
    4254
    43 ==== Create relation ====
     55It can happen that there are no such elements.
    4456
    45 In case of **entailment**, mark the elements that are the same thing but are described with different words. In case of **contradiction**, mark the elements that are mutually exclusive (e.g., man-woman) in the premise and hypothesis. In case of **neutral**, there may be a more specific term (e.g., woman-human) or some independent term; the relation would be orange.
    4657
    47 [[Image(entailment1.png,width=30%)]]
     58==== 4. Create a supporting relation ====
    4859
    49 (It can happen that there are no such elements.)
     60In case of **entailment**, mark the elements that are the same thing but are described with different words.
     61
     62In case of **contradiction**, mark the elements that are mutually exclusive (e.g., man-woman) in the premise and hypothesis.
     63
     64In case of **neutral**, there may be a more specific term (e.g., woman-human) or some independent term; the relation would be orange.
     65
    5066
    5167[[Image(entailment2.png,width=50%)]]
    … …  
    7086=== Further information ===
    7187
    72 You can check the instructions by using the **Instructions** button on the top every time you need it.
     88You can check the instructions by using the **Instructions** button on the top when you need it.
    7389
    7490[[Image(annotation_instruction.png,width=50%)]]