First Time Login
Thank you for participating in this experiment, which evaluates the
output of several systems. The experiment should take you around one
INSTRUCTIONS (please read this carefully, these are
============ not some ignorable small print stuff)
You will be presented with pairs (s1,s2) of textual units. The
*general* point of the task is to determine whether the two units are
matching in that they are two valid options of describing the same
event. In particular, you will look at phrases in which some words
have been substituted with slots or variables (e.g., "X1 defeated
X2"). An example of such a pair is:
s1 = "NUM1 people were killed and NUM2 wounded in X1"
s2 = "A bomb in X1 killed NUM1 people; NUM2 were taken to emergency
You will be asked to identify each pair as matching or not.
The guidelines for your judgments are:
* You should identify the pair as a match if one of the phrases can
generally be substituted for the other in documents regarding the
context which is news reports of terror events in the Israeli
Note: This does not need to be symmetric. For instance, in the
above example anywhere "The bomb killed X people; Y were taken to
emergency room" appears, one could replace it with "X people were
killed and Y wounded", so the pair should be identified as a match
even though the ``other direction'' doesn't hold (people can be
killed in ways other than by bombs).
* The given context should be used in this judgment, for example, "the
Israeli army" does not match "the army" in general, but in the
context of a news report that established the identity of the army
elsewhere in the report, these can be interchangeable. A more
extreme example is "NUM people died, police said" and "at least NUM
people died" -- a newsreporter hearing an official statement can
choose either sentences to convey the content of this message,
because usually the official death toll raises in time.
* By "generally substituted", we mean that you can think of many
values of these variables (again, in the context of news reports on
the Israeli-Palestinian conflict) for which the meaning is the same
(or very similar).
* The phrases omit articles (for example, s2 lacks "the" before
"emergency room"). Do not take this into account in your judgments.
In addition, you need to rate your decision as difficult or not. You
can also include other information in an optional comment box. These
comments can be as long as you want -- please provide any relevant
comments, for example, a reason for a non-obvious choice, assumptions
you made, or any other relevant information.
* variables are capitalized;
* some variables are typed: NUM stands for numbers, DATE stands for a
date -- either a full date or a day name.
IMPORTANT: before proceeding, carefully go through the examples below,
which were created by following the above guidelines, to check your
understanding of the task. If you do have questions, please ask us
for clarification BEFORE starting the experiment.
1>> "X defeated Y", "Y lost against X"
We expect the answers to be "yes" and "not difficult" (e.g., "The
government defeated the rebels." can be substituted for "The
rebels lost against the government").
2>> "X informed Y", "Y was informed by X"
We expect the answers to be "yes" and "not difficult."
3>> "Teenager X driving in red car was arrested", "X was arrested"
We expect the answers to be "yes" and "not difficult" (even though
the second phrase does not contain all the information in the
first phrase, the second sentence can be substituted for the
4>> "hit in X by Y", "Y wound in X"
We expect the answer to be "no", but "difficult". (Even though "a
bullet wound in the head" sounds like "hit in the head by a
bullet", these two phrases are different parts of speech: "bullet
wound in head" is a noun phrase, and "hit in head by bullet" is a
verb phrase. Therefore, there is no sentence in which one phrase
can be substituted by the other.)
5>> "- DATE : bombing at X1 in X2 kills NUM1 people and injures NUM2
"Suicide bomber blew himself up at X1 in X2 DATE, killing NUM1
people and wounding around NUM2 others, police said."
We expect the answers to be "yes" and "not difficult". (This is
similar to example 3, the first can be substituted for the second,
but the second can not be substituted for the first, since not all
bombings are suicide bombings).
6>> "Y entered territory in X", "Y raided X town"
We expect answer "no", and "difficult." (Although "raiding X
town" seems to imply "entering territory in X", substituting the
first for the second results in great loss of information,
significantly weakening its meaning).
At the top of every page you will have a new pair to judge with
controls to make your selections, and afterwards a list of some of
your recent selections, which you can use to review and modify them in
case of errors (a link at the bottom will show you all previous
Please do not do more than 60 judgments in one sitting. Take a break
between each group of judgments that you do.
Some pairs that you will see will look extremely similar to others,
but no two pairs will be identical. Please read every pair carefully.
Regina and Lillian.
Eli's Scheme Server at sardine.cs.columbia.edu:8080 - firstname.lastname@example.org