Changes between Version 24 and Version 25 of AnnotationManual/CZSNLI
- Timestamp:
- Mar 5, 2024, 1:55:07 PM (2 months ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
AnnotationManual/CZSNLI
v24 v25 15 15 * one or both sentences from the pair **do not make sense** (**bad translation**) 16 16 17 The annotation is made via the [https://nlp.fi.muni.cz/projekty/labelstudio LabelStudio] tool. The procedure requires a double login process. 17 The annotation is made via the [https://nlp.fi.muni.cz/projekty/labelstudio LabelStudio] tool. The procedure requires a double login process. 18 18 19 19 First, log into the tool using the credentials obtained by email (`student@localhost`). … … 22 22 23 23 === How to annotate one sample === 24 25 24 !LabelStudio limitation: Do not simultaneously open one labeling task in two tabs or windows. 26 25 27 26 The annotation steps are the following. 28 27 29 ==== 1. Rule out bad translations 30 28 ==== 1. Rule out bad translations ==== 31 29 Read the premise and the hypothesis and decide whether they make sense. If not, select **bad translation** and continue to another pair. You can check your understanding by revealing the original texts. 32 30 33 ==== 2. Decide the sentence relationship 34 31 ==== 2. Decide the sentence relationship ==== 35 32 If the sentences make sense, decide whether the relation between them is **entailment**, **contradiction**, or **neutral**. 36 33 … … 40 37 41 38 Typically: 42 1. If the hypothesis is a more general statement than the premise, the result is **entailment**. 43 1. If the premise adds new information or denotes an element of a hypothesis by a more specific name, it is **neutral** (cannot say). 44 1. If the hypothesis and the premise describe a non-related action with different actors or they describe mutually exclusive actions, it is a **contradiction**. 39 40 1. If the hypothesis is a more general statement than the premise, the result is **entailment**. 41 1. If the premise adds new information or denotes an element of a hypothesis by a more specific name, it is **neutral** (cannot say). 42 1. If the hypothesis and the premise describe a non-related action with different actors or they describe mutually exclusive actions, it is a **contradiction**. 45 43 46 44 ==== 3. Select relevant parts ==== 47 48 45 [[Image(entailment1.png,width=50%,right)]] 49 46 … … 54 51 Mark one or more words (the tool selects whole words and removes trailing spaces from the selection). 55 52 56 In case of **entailment**, mark the elements that are the same thing but are described with different words. 53 In case of **entailment**, mark the elements that are the same thing but are described with different words. 57 54 58 55 In case of **contradiction**, mark the elements that are mutually exclusive (e.g., man-woman) in the premise and hypothesis. If the hypothesis and the premise are completely unrelated, do not mark anything. … … 60 57 In case of **neutral**, there is often some new information in the hypothesis, for instance, a more specific term (woman-human) or some independent term. In the neutral pair, the hypothesis may have one element with no relation to the premise. 61 58 59 You can remove selection by selecting it and clicking a red garbage bin in the Selection details tab. 60 62 61 ==== 4. Create a supporting relation ==== 62 Set up the relation between the elements in the premise and hypothesis: 63 63 64 Set up the relation between the elements in the premise and hypothesis: 65 1. Click the relevant element in premise (1). 66 1. select the **Info** tab and click the **Create relation** button, a chain icon (2). 67 1. Next, click the relevant part in the hypothesis (3). 64 1. Click the relevant element in premise (1). 65 1. select the **Info** tab and click the **Create relation** button, a chain icon (2). 66 1. Next, click the relevant part in the hypothesis (3). 68 67 69 68 [[Image(entailment2.png,width=50%,center)]] 70 71 69 72 70 [[Image(entailment3.png,width=50%,center)]] 73 71 74 72 Then select the relation type: click on the **Relations** tab, and select the relation by clicking the triple dot button. Next, click into the **Select labels** box and choose a relation type: 75 * In the case of **entailment**, the relation would be green, i.e., generalization or similar. 76 * In the case of **contradiction**, the relation would be red, i.e., exclusion. 77 * In the case of **neutral**, the relation would be orange. 73 74 * In the case of **entailment**, the relation would be green, i.e., generalization or similar. 75 * In the case of **contradiction**, the relation would be red, i.e., exclusion. 76 * In the case of **neutral**, the relation would be orange. 78 77 79 78 It does not matter if you mark prepositional phrases (e.g., "''v obýváku''") or noun phrases (e.g., "''obýváku''"). 79 80 You can remove a relation by clicking a red garbage bin in the Relations tab. 80 81 81 82 [[Image(entailment4.png,width=15%,center)]] 82 83 83 84 === Examples of annotations === 85 ==== Example 1 ==== 86 **Premise:** '' Vousatý muž v malířské čepici hraje na xylofon na straně rušné lávky.'' 84 87 85 ==== Example 1 ==== 88 **Hypothesis:** '' Muž s plnovousem hraje na nástroj na chodníku.'' 86 89 87 **Premise:** ''Vousatý muž v malířské čepici hraje na xylofon na straně rušné lávky.'' 88 89 **Hypothesis:** ''Muž s plnovousem hraje na nástroj na chodníku.'' 90 91 The information in the hypothesis is more general, so mark it as **entailment**. For support of the decision, mark "xylofon" and "nástroj" and connect with the **generalization** relation, and "Vousatý muž" and "Muž s plnovousem" with the **similar** relation. 90 The information in the hypothesis is more general, so mark it as **entailment**. For support of the decision, mark "xylofon" and "nástroj" and connect with the **generalization** relation, and "Vousatý muž" and "Muž s plnovousem" with the **similar** relation. 92 91 93 92 ==== Example 2 ==== 93 **Premise:** '' Chlapec je zajištěn mezi dvěma bungee lany a visí ve vzduchu.'' 94 94 95 **Premise:** ''Chlapec je zajištěn mezi dvěma bungee lany a visí ve vzduchu.''95 **Hypothesis:** '' Chlapec se prochází v parku.'' 96 96 97 **Hypothesis:** ''Chlapec se prochází v parku.'' 98 99 The boy cannot do both things. Mark as **contradiction**. To support the decision, mark "visí" (is suspended) and "se prochází" (is walking) as **exclusion**. 97 The boy cannot do both things. Mark as **contradiction**. To support the decision, mark "visí" (is suspended) and "se prochází" (is walking) as **exclusion**. 100 98 101 99 ==== Example 3 ==== 100 **Premise:** '' Surfař v zeleném pruhovaném neoprenu jede na vlně.'' 102 101 103 **Premise:** ''Surfař v zeleném pruhovaném neoprenu jede na vlně.''102 **Hypothesis:** '' Muž, který surfuje, má zelený pruhovaný neopren se svým psem.'' 104 103 105 **Hypothesis:** ''Muž, který surfuje, má zelený pruhovaný neopren se svým psem.'' 106 107 The information about the dog is more specific, therefore mark as **neutral**. To support the decision, mark "se svým psem" without relation to the premise. 104 The information about the dog is more specific, therefore mark as **neutral**. To support the decision, mark "se svým psem" without relation to the premise. 108 105 109 106 ==== Example 4 ==== 107 **Premise:** '' Dvě dámy se střetnou se třemi osly.'' 110 108 111 **Premise:** ''Dvě dámy se střetnou se třemi osly.''109 **Hypothesis:** '' Dvě dámy půjdou nakrmit tři osly do zoo.'' 112 110 113 **Hypothesis:** ''Dvě dámy půjdou nakrmit tři osly do zoo.''111 We cannot tell whether the ladies will feed the donkeys if we only know they meet the donkeys. 114 112 115 We cannot tell whether the ladies will feed the donkeys if we only know they meet the donkeys. 116 117 Mark as **neutral**. For support of the decision, mark "se střetnou" (meet) and "půjdou nakrmit" (will feed) as **independence**. 113 Mark as **neutral**. For support of the decision, mark "se střetnou" (meet) and "půjdou nakrmit" (will feed) as **independence**. 118 114 119 115 ==== Example 5 ==== 116 **Premise:** '' Starší muž hrající na podivný nástroj podobný kytaře na něčem, co vypadá jako park nebo otevřené prostranství.'' 120 117 121 **Premise:** ''Starší muž hrající na podivný nástroj podobný kytaře na něčem, co vypadá jako park nebo otevřené prostranství.''118 **Hypothesis:** '' Žena hraje v obýváku na klavír.'' 122 119 123 **Hypothesis:** ''Žena hraje v obýváku na klavír.'' 124 125 Since a man is not a woman, mark it as **contradiction**. To support the decision, mark "Starší muž" and "Žena" with the **exclusion** relation. Mark also "park nebo otevřené prostranství" and "obýváku" with the **exclusion** relation. 120 Since a man is not a woman, mark it as **contradiction**. To support the decision, mark "Starší muž" and "Žena" with the **exclusion** relation. Mark also "park nebo otevřené prostranství" and "obýváku" with the **exclusion** relation. 126 121 127 122 ==== Example 6 ==== 123 **Premise:** '' Kůň vlevo má bílou hřívu.'' 128 124 129 **Premise:** ''Kůň vlevo má bílou hřívu.''125 **Hypothesis:** '' Je tam víc než jeden kůň.'' 130 126 131 **Hypothesis:** ''Je tam víc než jeden kůň.'' 132 133 From the premise, it follows that there are more horses. Mark the relation as **entailment** and connect "Kůň vlevo" (the left horse) and "víc než jeden kůň" (more than one horse) with the **generalization** relation. 134 127 From the premise, it follows that there are more horses. Mark the relation as **entailment** and connect "Kůň vlevo" (the left horse) and "víc než jeden kůň" (more than one horse) with the **generalization** relation. 135 128 136 129 ==== Example 7 ==== 130 **Premise:** '' Několik lidí pohybujících nějakou stavbou.'' 137 131 138 **Premise:** ''Několik lidí pohybujících nějakou stavbou.''132 **Hypothesis:** '' Několik lidí stěhuje kůlnu na nářadí do zadní části dvora.'' 139 133 140 **Hypothesis:** ''Několik lidí stěhuje kůlnu na nářadí do zadní části dvora.'' 141 142 We don't know the nature of the structure. So, mark it as **neutral**. To support the decision, mark "nějakou stavbou" and "kůlnu na nářadí" with the **specification** relation. 134 We don't know the nature of the structure. So, mark it as **neutral**. To support the decision, mark "nějakou stavbou" and "kůlnu na nářadí" with the **specification** relation. 143 135 144 136 ==== Example 8 ==== 137 **Premise:** '' Ta mladá žena s šátkem dává někomu dárek.'' 145 138 146 **Premise:** ''Ta mladá žena s šátkem dává někomu dárek.''139 **Hypothesis:** '' Současnost je malá.'' 147 140 148 **Hypothesis:** ''Současnost je malá.''141 Apparently, the two sentences have no relation; moreover, the hypothesis is suspicious. 149 142 150 Apparently, the two sentences have no relation; moreover, the hypothesis is suspicious. 151 152 Check the English origin (The young woman with the scarf is giving a present to someone. The present is small.) and mark it as **bad translation**. 143 Check the English origin (The young woman with the scarf is giving a present to someone. The present is small.) and mark it as **bad translation**. 153 144 154 145 ==== Example 9 ==== 146 **Premise:** '' Žena a dvě malé holčičky slaví narozeniny s mužem přes webovou kameru.'' 155 147 156 **Premise:** ''Žena a dvě malé holčičky slaví narozeniny s mužem přes webovou kameru.''148 **Hypothesis:** '' Muž běží po ulici'' 157 149 158 **Hypothesis:** ''Muž běží po ulici'' 159 160 Completely unrelated stories. Annotated as **contradiction** and do not mark any relations. 150 Completely unrelated stories. Annotated as **contradiction** and do not mark any relations. 161 151 162 152 === Submit === 163 164 153 Finally, **Submit** your annotation. 165 154 … … 167 156 168 157 === Further information === 169 170 158 You can check the instructions by using the **Instructions** button on the top when you need it. 171 159 … … 173 161 174 162 You can try a short [https://nlp.fi.muni.cz/trac/research/wiki/AnnotationManual/CZSNLI/training training] on your own. 175
