Changes between Version 22 and Version 23 of AnnotationManual/CZSNLI


Ignore:
Timestamp:
Mar 5, 2024, 1:00:36 PM (2 months ago)
Author:
Ales Horak
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • AnnotationManual/CZSNLI

    v22 v23  
    33[[Image(annotation.png,width=50%,right)]]
    44
    5 The following sentences are descriptions of photographs (machine-translated from English to Czech). You cannot see the photos; however, you can imagine their content.
     5The annotated sentences are descriptions of photographs. You cannot see the photos; however, you can imagine their content. The individual sentences are machine-translated from English to Czech.
    66
    7 The aim is to decide for each pair of sentences (premise and hypothesis) whether:
     7The aim is to decide, for each pair of sentences (premise and hypothesis) belonging to a single photo, whether:
    88
    99 * the hypothesis **follows** from the premise (**entailment**)
    … …  
    1919First, log into the tool using the credentials obtained by email (`student@localhost`).
    2020
    21 Next, log into the assigned data packages using your faculty credentials (x..../password).
     21Next, log into the assigned data packages using your faculty credentials (`x....` and your password).
    2222
    2323=== How to annotate one sample ===
    2424
    25 !LabelStudio limitation: Do not simultaneously open one labeling task in two tabs.
     25!LabelStudio limitation: Do not simultaneously open one labeling task in two tabs or windows.
    2626
    2727More details about the labeling process: https://labelstud.io/guide/labeling.html
    2828
     29The annotation steps are the following.
     30
    2931==== 1. Rule out bad translations
    3032
    31 Read the premise and hypothesis and decide whether they make sense. If not, select **bad translation** and continue. You can check your understanding by revealing the original texts.
     33Read the premise and the hypothesis and decide whether they make sense. If not, select **bad translation** and continue to another pair. You can check your understanding by revealing the original texts.
    3234
    3335==== 2. Decide the sentence relationship
    … …  
    3537If the sentences make sense, decide whether the relation between them is **entailment**, **contradiction**, or **neutral**.
    3638
    37 Please assume that the hypothesis and the premise are connected. For example, if the premise says something about a man and the hypothesis mentions a human, assume they are **the same entity** in the picture.
     39Please assume that the hypothesis and the premise are connected by the photograph content. For example, if the premise says something about a man and the hypothesis mentions a human, assume they are **the same entity** in the picture.
    3840
    39 Also, use **common sense** when judging the impossibility of ''following''. For example, if the premise says "''a man rides a bicycle''", and the hypothesis says "''the man is washing his hands''", annotate it as **contradiction**, even though you can imagine someone washing his hands while riding a bike (being a very uncommon situation).
     41Also, use **common sense** when judging the impossibility of ''following''. For example, if the premise says "''a man rides a bicycle''", and the hypothesis says "''the man is washing his hands''", annotate the pair as **contradiction**, even though you ''can'' imagine someone washing his hands while riding a bike (being a very uncommon situation).
    4042
    4143Typically:
    … …  
    5456Mark one or more words (the tool selects whole words and removes trailing spaces from the selection).
    5557
    56 In the case of **entailment**, mark the elements that are the same thing but are described with different words.
     58In case of **entailment**, mark the elements that are the same thing but are described with different words.
    5759
    58 In the case of **contradiction**, mark the elements that are mutually exclusive (e.g., man-woman) in the premise and hypothesis. If the hypothesis and the premise are completely unrelated, do not mark anything.
     60In case of **contradiction**, mark the elements that are mutually exclusive (e.g., man-woman) in the premise and hypothesis. If the hypothesis and the premise are completely unrelated, do not mark anything.
    5961
    60 In the case of **neutral**, there is often some new information in the hypothesis, for instance, a more specific term (woman-human) or some independent term. In the neutral pair, the hypothesis may have one element with no relation to the premise.
     62In case of **neutral**, there is often some new information in the hypothesis, for instance, a more specific term (woman-human) or some independent term. In the neutral pair, the hypothesis may have one element with no relation to the premise.
    6163
    6264==== 4. Create a supporting relation ====
    6365
    64 Set up the relation between the elements in the premise and hypothesis. Click the relevant element in premise (1), select the **Info** tab and click the **Create relation** button, a chain icon (2). Next, click the relevant part in the hypothesis (3).
     66Set up the relation between the elements in the premise and hypothesis:
     671. Click the relevant element in premise (1).
     681. select the **Info** tab and click the **Create relation** button, a chain icon (2).
     691. Next, click the relevant part in the hypothesis (3).
    6570
    6671[[Image(entailment2.png,width=50%,center)]]
    … …  
    6974[[Image(entailment3.png,width=50%,center)]]
    7075
    71 Select the relation type: click on the Relations tab, and select the relation by clicking the triple dot button. Next, click into the **Select labels** box and select a relation.
     76Then select the relation type: click on the **Relations** tab, and select the relation by clicking the triple dot button. Next, click into the **Select labels** box and choose a relation type:
     77* In the case of **entailment**, the relation would be green, i.e., generalization or similar.
     78* In the case of **contradiction**, the relation would be red, i.e., exclusion.
     79* In the case of **neutral**, the relation would be orange.
    7280
    73 In the case of **entailment**, the relation would be green, i.e., generalization or similar. In the case of **contradiction**, the relation would be red, i.e., exclusion.
    74 In the case of **neutral**, the relation would be orange.
    75 
    76 It does not matter if you mark prepositional phrases (e.g., "v obýváku") or noun phrases (e.g., "obýváku").
     81It does not matter if you mark prepositional phrases (e.g., "''v obýváku''") or noun phrases (e.g., "''obýváku''").
    7782
    7883[[Image(entailment4.png,width=15%,center)]]
    … …  
    8287==== Example 1 ====
    8388
    84 **Premise:** Vousatý muž v malířské čepici hraje na xylofon na straně rušné lávky.
     89 **Premise:** ''Vousatý muž v malířské čepici hraje na xylofon na straně rušné lávky.''
    8590
    86 **Hypothesis:** Muž s plnovousem hraje na nástroj na chodníku.
     91 **Hypothesis:** ''Muž s plnovousem hraje na nástroj na chodníku.''
    8792
    88 The information in the hypothesis is more general, so mark it as **entailment**. For support of the decision, mark "xylofon" and "nástroj" and connect with the **generalization** relation, and "Vousatý muž" and "Muž s plnovousem" with the **similar** relation.
     93 The information in the hypothesis is more general, so mark it as **entailment**. For support of the decision, mark "xylofon" and "nástroj" and connect with the **generalization** relation, and "Vousatý muž" and "Muž s plnovousem" with the **similar** relation.
    8994
    9095==== Example 2 ====
    9196
    92 **Premise:** Chlapec je zajištěn mezi dvěma bungee lany a visí ve vzduchu.
     97 **Premise:** ''Chlapec je zajištěn mezi dvěma bungee lany a visí ve vzduchu.''
    9398
    94 **Hypothesis:** Chlapec se prochází v parku.
     99 **Hypothesis:** ''Chlapec se prochází v parku.''
    95100
    96 The boy cannot do both things. Mark as **contradiction**. To support the decision, mark "visí" (is suspended) and "se prochází" (is walking) as **exclusion**.
     101 The boy cannot do both things. Mark as **contradiction**. To support the decision, mark "visí" (is suspended) and "se prochází" (is walking) as **exclusion**.
    97102
    98103==== Example 3 ====
    99104
    100 **Premise:** Surfař v zeleném pruhovaném neoprenu jede na vlně.
     105 **Premise:** ''Surfař v zeleném pruhovaném neoprenu jede na vlně.''
    101106
    102 **Hypothesis:** Muž, který surfuje, má zelený pruhovaný neopren se svým psem.
     107 **Hypothesis:** ''Muž, který surfuje, má zelený pruhovaný neopren se svým psem.''
    103108
    104 The information about the dog is more specific, therefore mark as **neutral**. To support the decision, mark "se svým psem" without relation to the premise.
     109 The information about the dog is more specific, therefore mark as **neutral**. To support the decision, mark "se svým psem" without relation to the premise.
    105110
    106111==== Example 4 ====
    107112
    108 **Premise:** Dvě dámy se střetnou se třemi osly.
     113 **Premise:** ''Dvě dámy se střetnou se třemi osly.''
    109114
    110 **Hypothesis:** Dvě dámy půjdou nakrmit tři osly do zoo.
     115 **Hypothesis:** ''Dvě dámy půjdou nakrmit tři osly do zoo.''
    111116
    112 We cannot tell whether the ladies will feed the donkeys if we only know they meet the donkeys.
     117 We cannot tell whether the ladies will feed the donkeys if we only know they meet the donkeys.
    113118
    114 Mark as **neutral**. For support of the decision, mark "se střetnou" (meet) and "půjdou nakrmit" (will feed) as **independence**.
     119 Mark as **neutral**. For support of the decision, mark "se střetnou" (meet) and "půjdou nakrmit" (will feed) as **independence**.
    115120
    116121==== Example 5 ====
    117122
    118 **Premise:** Starší muž hrající na podivný nástroj podobný kytaře na něčem, co vypadá jako park nebo otevřené prostranství.
     123 **Premise:** ''Starší muž hrající na podivný nástroj podobný kytaře na něčem, co vypadá jako park nebo otevřené prostranství.''
    119124
    120 **Hypothesis:** Žena hraje v obýváku na klavír.
     125 **Hypothesis:** ''Žena hraje v obýváku na klavír.''
    121126
    122 Since a man is not a woman, mark it as **contradiction**. To support the decision, mark "Starší muž" and "Žena" with the **exclusion** relation. Mark also "park nebo otevřené prostranství" and "obýváku" with the **exclusion** relation.
     127 Since a man is not a woman, mark it as **contradiction**. To support the decision, mark "Starší muž" and "Žena" with the **exclusion** relation. Mark also "park nebo otevřené prostranství" and "obýváku" with the **exclusion** relation.
    123128
    124129==== Example 6 ====
    125130
    126 **Premise:** Kůň vlevo má bílou hřívu.
     131 **Premise:** ''Kůň vlevo má bílou hřívu.''
    127132
    128 **Hypothesis:** Je tam víc než jeden kůň.
     133 **Hypothesis:** ''Je tam víc než jeden kůň.''
    129134
    130 From the premise, it follows that there are more horses. Mark the relation as **entailment** and connect "Kůň vlevo" (the left horse) and "víc než jeden kůň" (more than one horse) with the **generalization** relation.
     135 From the premise, it follows that there are more horses. Mark the relation as **entailment** and connect "Kůň vlevo" (the left horse) and "víc než jeden kůň" (more than one horse) with the **generalization** relation.
    131136
    132137
    133138==== Example 7 ====
    134139
     140 **Premise:** ''Několik lidí pohybujících nějakou stavbou.''
    135141
    136 **Premise:** Několik lidí pohybujících nějakou stavbou.
     142 **Hypothesis:** ''Několik lidí stěhuje kůlnu na nářadí do zadní části dvora.''
    137143
    138 **Hypothesis:** Několik lidí stěhuje kůlnu na nářadí do zadní části dvora.
    139 
    140 We don't know the nature of the structure. So, mark it as **neutral**. To support the decision, mark "nějakou stavbou" and "kůlnu na nářadí" with the **specification** relation.
     144 We don't know the nature of the structure. So, mark it as **neutral**. To support the decision, mark "nějakou stavbou" and "kůlnu na nářadí" with the **specification** relation.
    141145
    142146==== Example 8 ====
    143147
    144 **Premise:** Ta mladá žena s šátkem dává někomu dárek.
     148 **Premise:** ''Ta mladá žena s šátkem dává někomu dárek.''
    145149
    146 **Hypothesis:** Současnost je malá.
     150 **Hypothesis:** ''Současnost je malá.''
    147151
    148 Apparently, the two sentences have no relation; moreover, the hypothesis is suspicious.
     152 Apparently, the two sentences have no relation; moreover, the hypothesis is suspicious.
    149153
    150 Check the English origin (The young woman with the scarf is giving a present to someone. The present is small.) and mark it as **bad translation**.
     154 Check the English origin (The young woman with the scarf is giving a present to someone. The present is small.) and mark it as **bad translation**.
    151155
    152156==== Example 9 ====
    153157
    154 **Premise:** Žena a dvě malé holčičky slaví narozeniny s mužem přes webovou kameru.
     158 **Premise:** ''Žena a dvě malé holčičky slaví narozeniny s mužem přes webovou kameru.''
    155159
    156 **Hypothesis:** Muž běží po ulici
     160 **Hypothesis:** ''Muž běží po ulici''
    157161
    158 Completely unrelated stories. Annotated as **contradiction** and do not mark any relations.
     162 Completely unrelated stories. Annotated as **contradiction** and do not mark any relations.
    159163
    160164=== Submit ===