Changes between Version 17 and Version 18 of AnnotationManual/CZSNLI

Timestamp:

Mar 4, 2024, 9:36:00 AM (2 months ago)

Author:

Zuzana Nevěřilová

Comment:

--

AnnotationManual/CZSNLI

@@ -3 +3 @@
 [[Image(annotation.png,width=50%,right)]]
 
-The following sentences are descriptions of photographs (machine translated from English to Czech). You cannot see the photos, however, you can imagine their content.
+The following sentences are descriptions of photographs (machine-translated from English to Czech). You cannot see the photos; however, you can imagine their content.
 
 The aim is to decide for each pair of sentences (premise and hypothesis) whether:
@@ -23 +23 @@
 === How to annotate one sample ===
 
+!LabelStudio limitation: Please **do not open** one labeling task in two tabs simultaneously.
+
+More details about the labeling process: https://labelstud.io/guide/labeling.html
+
 ==== 1. Rule out bad translations
 
@@ -33 +37 @@
 Please assume that the hypothesis and the premise are connected. For example, if the premise says something about a man and the hypothesis mentions a human, assume they are **the same entity** in the picture.
 
-Also, use **common sense** when judging the impossibility of ''following''. For example, if the premise says "''a man rides a bicycle''", and the hypothesis says "''the man is washing his hands''", annotate it as **false**, even though you can imagine someone washing his hands while riding a bike (being a very uncommon situation).
+Also, use **common sense** when judging the impossibility of ''following''. For example, if the premise says "''a man rides a bicycle''", and the hypothesis says "''the man is washing his hands''", annotate it as **contradiction**, even though you can imagine someone washing his hands while riding a bike (being a very uncommon situation).
 
 Typically:
@@ -74 +78 @@
 In case of **neutral**, the relation would be orange. 
 
+It does not matter if you mark prepositional phrases (e.g., "v obýváku") or noun phrases (e.g., "obýváku").
 
 [[Image(entailment4.png,width=15%,center)]]
 
+=== Examples of annotations ===
+
+==== Example 1 ====
+
+**Premise:** Vousatý muž v malířské čepici hraje na xylofon na straně rušné lávky.
+
+**Hypothesis:** Muž s plnovousem hraje na nástroj na chodníku.
+
+The information in the hypothesis is more general, so mark it as **entailment**. For support of the decision, mark "xylofon" and "nástroj" and connect with the **generalization** relation, and "Vousatý muž" and "Muž s plnovousem" with the "similar" relation.
+
+==== Example 2 ====
+
+**Premise:** Chlapec je zajištěn mezi dvěma bungee lany a visí ve vzduchu.
+
+**Hypothesis:** Chlapec se prochází v parku.
+
+The boy cannot do both things. Mark as **contradiction**. To support the decision, mark "visí" (is suspended) and "se prochází" (is walking) as **exclusion**. 
+
+==== Example 3 ====
+
+**Premise:** Surfař v zeleném pruhovaném neoprenu jede na vlně.
+
+**Hypothesis:** Muž, který surfuje, má zelený pruhovaný neopren se svým psem.
+
+The information about the dog is more specific, therefore mark as **neutral**. To support the decision, mark "se svým psem" without relation to the premise.
+
+==== Example 4 ====
+
+**Premise:** Dvě dámy se střetnou se třemi osly.
+
+**Hypothesis:** Dvě dámy půjdou nakrmit tři osly do zoo.
+
+We cannot tell whether the ladies will feed the donkeys if we only know they meet the donkeys.
+
+Mark as **neutral**. For support of the decision, mark "se střetnou" (meet) and "půjdou nakrmit" (will feed) as **independence**.
+
+==== Example 5 ====
+
+**Premise:** Starší muž hrající na podivný nástroj podobný kytaře na něčem, co vypadá jako park nebo otevřené prostranství.
+
+**Hypothesis:** Žena hraje v obýváku na klavír.
+
+Since a man is not a woman, mark it as **contradiction**. To support the decision, mark "Starší muž" and "Žena" with the **exclusion** relation. Mark also "park nebo otevřené prostranství" and "obýváku" with the **exclusion** relation.
+
+==== Example 6 ====
+
+**Premise:** Kůň vlevo má bílou hřívu.
+
+**Hypothesis:** Je tam víc než jeden kůň.
+
+From the premise, it follows that there are more horses. Mark the relation as **entailment** and mark "Kůň vlevo" (the left horse) without any relation to the hypothesis.
+
+
+==== Example 7 ====
+
+**Premise:** Motocyklista jede po úbočí zelené hory, zatímco jiné kolo zůstává stát.
+
+**Hypothesis:** Jsou tam dva motocykly, ale jeden je rozbitý.
+
+We don't know the reason why the bike remains stationary. So, mark it as **neutral**. To support the decision, mark "zůstává stát" and "je rozbitý" with the **specification** relation.
+
+==== Example 8 ====
+
+**Premise:** Ta mladá žena s šátkem dává někomu dárek.
+
+**Hypothesis:** Současnost je malá.
+
+Apparently, the two sentences have no relation; moreover, the hypothesis is suspicious.
+
+Check the English origin (The young woman with the scarf is giving a present to someone. The present is small.) and mark it as **bad translation**.
 
 === Submit ===