Changes between Version 22 and Version 23 of AnnotationManual/CZSNLI
- Timestamp:
- Mar 5, 2024, 1:00:36 PM (5 months ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
AnnotationManual/CZSNLI
v22 v23 3 3 [[Image(annotation.png,width=50%,right)]] 4 4 5 The following sentences are descriptions of photographs (machine-translated from English to Czech). You cannot see the photos; however, you can imagine their content.5 The annotated sentences are descriptions of photographs. You cannot see the photos; however, you can imagine their content. The individual sentences are machine-translated from English to Czech. 6 6 7 The aim is to decide for each pair of sentences (premise and hypothesis)whether:7 The aim is to decide, for each pair of sentences (premise and hypothesis) belonging to a single photo, whether: 8 8 9 9 * the hypothesis **follows** from the premise (**entailment**) … … 19 19 First, log into the tool using the credentials obtained by email (`student@localhost`). 20 20 21 Next, log into the assigned data packages using your faculty credentials ( x..../password).21 Next, log into the assigned data packages using your faculty credentials (`x....` and your password). 22 22 23 23 === How to annotate one sample === 24 24 25 !LabelStudio limitation: Do not simultaneously open one labeling task in two tabs .25 !LabelStudio limitation: Do not simultaneously open one labeling task in two tabs or windows. 26 26 27 27 More details about the labeling process: https://labelstud.io/guide/labeling.html 28 28 29 The annotation steps are the following. 30 29 31 ==== 1. Rule out bad translations 30 32 31 Read the premise and hypothesis and decide whether they make sense. If not, select **bad translation** and continue. You can check your understanding by revealing the original texts.33 Read the premise and the hypothesis and decide whether they make sense. If not, select **bad translation** and continue to another pair. You can check your understanding by revealing the original texts. 32 34 33 35 ==== 2. Decide the sentence relationship … … 35 37 If the sentences make sense, decide whether the relation between them is **entailment**, **contradiction**, or **neutral**. 36 38 37 Please assume that the hypothesis and the premise are connected . For example, if the premise says something about a man and the hypothesis mentions a human, assume they are **the same entity** in the picture.39 Please assume that the hypothesis and the premise are connected by the photograph content. For example, if the premise says something about a man and the hypothesis mentions a human, assume they are **the same entity** in the picture. 38 40 39 Also, use **common sense** when judging the impossibility of ''following''. For example, if the premise says "''a man rides a bicycle''", and the hypothesis says "''the man is washing his hands''", annotate it as **contradiction**, even though you canimagine someone washing his hands while riding a bike (being a very uncommon situation).41 Also, use **common sense** when judging the impossibility of ''following''. For example, if the premise says "''a man rides a bicycle''", and the hypothesis says "''the man is washing his hands''", annotate the pair as **contradiction**, even though you ''can'' imagine someone washing his hands while riding a bike (being a very uncommon situation). 40 42 41 43 Typically: … … 54 56 Mark one or more words (the tool selects whole words and removes trailing spaces from the selection). 55 57 56 In thecase of **entailment**, mark the elements that are the same thing but are described with different words.58 In case of **entailment**, mark the elements that are the same thing but are described with different words. 57 59 58 In thecase of **contradiction**, mark the elements that are mutually exclusive (e.g., man-woman) in the premise and hypothesis. If the hypothesis and the premise are completely unrelated, do not mark anything.60 In case of **contradiction**, mark the elements that are mutually exclusive (e.g., man-woman) in the premise and hypothesis. If the hypothesis and the premise are completely unrelated, do not mark anything. 59 61 60 In thecase of **neutral**, there is often some new information in the hypothesis, for instance, a more specific term (woman-human) or some independent term. In the neutral pair, the hypothesis may have one element with no relation to the premise.62 In case of **neutral**, there is often some new information in the hypothesis, for instance, a more specific term (woman-human) or some independent term. In the neutral pair, the hypothesis may have one element with no relation to the premise. 61 63 62 64 ==== 4. Create a supporting relation ==== 63 65 64 Set up the relation between the elements in the premise and hypothesis. Click the relevant element in premise (1), select the **Info** tab and click the **Create relation** button, a chain icon (2). Next, click the relevant part in the hypothesis (3). 66 Set up the relation between the elements in the premise and hypothesis: 67 1. Click the relevant element in premise (1). 68 1. select the **Info** tab and click the **Create relation** button, a chain icon (2). 69 1. Next, click the relevant part in the hypothesis (3). 65 70 66 71 [[Image(entailment2.png,width=50%,center)]] … … 69 74 [[Image(entailment3.png,width=50%,center)]] 70 75 71 Select the relation type: click on the Relations tab, and select the relation by clicking the triple dot button. Next, click into the **Select labels** box and select a relation. 76 Then select the relation type: click on the **Relations** tab, and select the relation by clicking the triple dot button. Next, click into the **Select labels** box and choose a relation type: 77 * In the case of **entailment**, the relation would be green, i.e., generalization or similar. 78 * In the case of **contradiction**, the relation would be red, i.e., exclusion. 79 * In the case of **neutral**, the relation would be orange. 72 80 73 In the case of **entailment**, the relation would be green, i.e., generalization or similar. In the case of **contradiction**, the relation would be red, i.e., exclusion. 74 In the case of **neutral**, the relation would be orange. 75 76 It does not matter if you mark prepositional phrases (e.g., "v obýváku") or noun phrases (e.g., "obýváku"). 81 It does not matter if you mark prepositional phrases (e.g., "''v obýváku''") or noun phrases (e.g., "''obýváku''"). 77 82 78 83 [[Image(entailment4.png,width=15%,center)]] … … 82 87 ==== Example 1 ==== 83 88 84 **Premise:** Vousatý muž v malířské čepici hraje na xylofon na straně rušné lávky. 89 **Premise:** ''Vousatý muž v malířské čepici hraje na xylofon na straně rušné lávky.'' 85 90 86 **Hypothesis:** Muž s plnovousem hraje na nástroj na chodníku. 91 **Hypothesis:** ''Muž s plnovousem hraje na nástroj na chodníku.'' 87 92 88 The information in the hypothesis is more general, so mark it as **entailment**. For support of the decision, mark "xylofon" and "nástroj" and connect with the **generalization** relation, and "Vousatý muž" and "Muž s plnovousem" with the **similar** relation.93 The information in the hypothesis is more general, so mark it as **entailment**. For support of the decision, mark "xylofon" and "nástroj" and connect with the **generalization** relation, and "Vousatý muž" and "Muž s plnovousem" with the **similar** relation. 89 94 90 95 ==== Example 2 ==== 91 96 92 **Premise:** Chlapec je zajištěn mezi dvěma bungee lany a visí ve vzduchu. 97 **Premise:** ''Chlapec je zajištěn mezi dvěma bungee lany a visí ve vzduchu.'' 93 98 94 **Hypothesis:** Chlapec se prochází v parku. 99 **Hypothesis:** ''Chlapec se prochází v parku.'' 95 100 96 The boy cannot do both things. Mark as **contradiction**. To support the decision, mark "visí" (is suspended) and "se prochází" (is walking) as **exclusion**.101 The boy cannot do both things. Mark as **contradiction**. To support the decision, mark "visí" (is suspended) and "se prochází" (is walking) as **exclusion**. 97 102 98 103 ==== Example 3 ==== 99 104 100 **Premise:** Surfař v zeleném pruhovaném neoprenu jede na vlně. 105 **Premise:** ''Surfař v zeleném pruhovaném neoprenu jede na vlně.'' 101 106 102 **Hypothesis:** Muž, který surfuje, má zelený pruhovaný neopren se svým psem. 107 **Hypothesis:** ''Muž, který surfuje, má zelený pruhovaný neopren se svým psem.'' 103 108 104 The information about the dog is more specific, therefore mark as **neutral**. To support the decision, mark "se svým psem" without relation to the premise.109 The information about the dog is more specific, therefore mark as **neutral**. To support the decision, mark "se svým psem" without relation to the premise. 105 110 106 111 ==== Example 4 ==== 107 112 108 **Premise:** Dvě dámy se střetnou se třemi osly. 113 **Premise:** ''Dvě dámy se střetnou se třemi osly.'' 109 114 110 **Hypothesis:** Dvě dámy půjdou nakrmit tři osly do zoo. 115 **Hypothesis:** ''Dvě dámy půjdou nakrmit tři osly do zoo.'' 111 116 112 We cannot tell whether the ladies will feed the donkeys if we only know they meet the donkeys.117 We cannot tell whether the ladies will feed the donkeys if we only know they meet the donkeys. 113 118 114 Mark as **neutral**. For support of the decision, mark "se střetnou" (meet) and "půjdou nakrmit" (will feed) as **independence**.119 Mark as **neutral**. For support of the decision, mark "se střetnou" (meet) and "půjdou nakrmit" (will feed) as **independence**. 115 120 116 121 ==== Example 5 ==== 117 122 118 **Premise:** Starší muž hrající na podivný nástroj podobný kytaře na něčem, co vypadá jako park nebo otevřené prostranství. 123 **Premise:** ''Starší muž hrající na podivný nástroj podobný kytaře na něčem, co vypadá jako park nebo otevřené prostranství.'' 119 124 120 **Hypothesis:** Žena hraje v obýváku na klavír. 125 **Hypothesis:** ''Žena hraje v obýváku na klavír.'' 121 126 122 Since a man is not a woman, mark it as **contradiction**. To support the decision, mark "Starší muž" and "Žena" with the **exclusion** relation. Mark also "park nebo otevřené prostranství" and "obýváku" with the **exclusion** relation.127 Since a man is not a woman, mark it as **contradiction**. To support the decision, mark "Starší muž" and "Žena" with the **exclusion** relation. Mark also "park nebo otevřené prostranství" and "obýváku" with the **exclusion** relation. 123 128 124 129 ==== Example 6 ==== 125 130 126 **Premise:** Kůň vlevo má bílou hřívu. 131 **Premise:** ''Kůň vlevo má bílou hřívu.'' 127 132 128 **Hypothesis:** Je tam víc než jeden kůň. 133 **Hypothesis:** ''Je tam víc než jeden kůň.'' 129 134 130 From the premise, it follows that there are more horses. Mark the relation as **entailment** and connect "Kůň vlevo" (the left horse) and "víc než jeden kůň" (more than one horse) with the **generalization** relation.135 From the premise, it follows that there are more horses. Mark the relation as **entailment** and connect "Kůň vlevo" (the left horse) and "víc než jeden kůň" (more than one horse) with the **generalization** relation. 131 136 132 137 133 138 ==== Example 7 ==== 134 139 140 **Premise:** ''Několik lidí pohybujících nějakou stavbou.'' 135 141 136 **Premise:** Několik lidí pohybujících nějakou stavbou. 142 **Hypothesis:** ''Několik lidí stěhuje kůlnu na nářadí do zadní části dvora.'' 137 143 138 **Hypothesis:** Několik lidí stěhuje kůlnu na nářadí do zadní části dvora. 139 140 We don't know the nature of the structure. So, mark it as **neutral**. To support the decision, mark "nějakou stavbou" and "kůlnu na nářadí" with the **specification** relation. 144 We don't know the nature of the structure. So, mark it as **neutral**. To support the decision, mark "nějakou stavbou" and "kůlnu na nářadí" with the **specification** relation. 141 145 142 146 ==== Example 8 ==== 143 147 144 **Premise:** Ta mladá žena s šátkem dává někomu dárek. 148 **Premise:** ''Ta mladá žena s šátkem dává někomu dárek.'' 145 149 146 **Hypothesis:** Současnost je malá. 150 **Hypothesis:** ''Současnost je malá.'' 147 151 148 Apparently, the two sentences have no relation; moreover, the hypothesis is suspicious.152 Apparently, the two sentences have no relation; moreover, the hypothesis is suspicious. 149 153 150 Check the English origin (The young woman with the scarf is giving a present to someone. The present is small.) and mark it as **bad translation**.154 Check the English origin (The young woman with the scarf is giving a present to someone. The present is small.) and mark it as **bad translation**. 151 155 152 156 ==== Example 9 ==== 153 157 154 **Premise:** Žena a dvě malé holčičky slaví narozeniny s mužem přes webovou kameru. 158 **Premise:** ''Žena a dvě malé holčičky slaví narozeniny s mužem přes webovou kameru.'' 155 159 156 **Hypothesis:** Muž běží po ulici 160 **Hypothesis:** ''Muž běží po ulici'' 157 161 158 Completely unrelated stories. Annotated as **contradiction** and do not mark any relations.162 Completely unrelated stories. Annotated as **contradiction** and do not mark any relations. 159 163 160 164 === Submit ===