Sem Experiment

First Time Login

Thank you for participating in this experiment, which evaluates the
output of several systems.  The experiment should take you around one
hour.

INSTRUCTIONS          (please read this carefully, these are
============           not some ignorable small print stuff)

You will be presented with pairs (s1,s2) of textual units.  The
*general* point of the task is to determine whether the two units are
matching in that they are two valid options of describing the same
event.  In particular, you will look at phrases in which some words
have been substituted with slots or variables (e.g., "X1 defeated
X2").  An example of such a pair is:

   s1 = "NUM1 people were killed and NUM2 wounded in X1"
   s2 = "A bomb in X1 killed NUM1 people; NUM2 were taken to emergency
         room"

You will be asked to identify each pair as matching or not.

The guidelines for your judgments are:

* You should identify the pair as a match if one of the phrases can
  generally be substituted for the other in documents regarding the
  context which is news reports of terror events in the Israeli
  Palestinian context.

  Note: This does not need to be symmetric.  For instance, in the
  above example anywhere "The bomb killed X people; Y were taken to
  emergency room" appears, one could replace it with "X people were
  killed and Y wounded", so the pair should be identified as a match
  even though the ``other direction'' doesn't hold (people can be
  killed in ways other than by bombs).

* The given context should be used in this judgment, for example, "the
  Israeli army" does not match "the army" in general, but in the
  context of a news report that established the identity of the army
  elsewhere in the report, these can be interchangeable.  A more
  extreme example is "NUM people died, police said" and "at least NUM
  people died" -- a newsreporter hearing an official statement can
  choose either sentences to convey the content of this message,
  because usually the official death toll raises in time.

* By "generally substituted", we mean that you can think of many
  values of these variables (again, in the context of news reports on
  the Israeli-Palestinian conflict) for which the meaning is the same
  (or very similar).

* The phrases omit articles (for example, s2 lacks "the" before
  "emergency room").  Do not take this into account in your judgments.

In addition, you need to rate your decision as difficult or not.  You
can also include other information in an optional comment box.  These
comments can be as long as you want -- please provide any relevant
comments, for example, a reason for a non-obvious choice, assumptions
you made, or any other relevant information.

Notation notes:

* variables are capitalized;
* some variables are typed: NUM stands for numbers, DATE stands for a
  date -- either a full date or a day name.

IMPORTANT: before proceeding, carefully go through the examples below,
which were created by following the above guidelines, to check your
understanding of the task.  If you do have questions, please ask us
for clarification BEFORE starting the experiment.

1>> "X defeated Y", "Y lost against X"

    We expect the answers to be "yes" and "not difficult" (e.g., "The
    government defeated the rebels." can be substituted for "The
    rebels lost against the government").

2>> "X informed Y", "Y was informed by X"

    We expect the answers to be "yes" and "not difficult."

3>> "Teenager X driving in red car was arrested", "X was arrested"
    We expect the answers to be "yes" and "not difficult" (even though
    the second phrase does not contain all the information in the
    first phrase, the second sentence can be substituted for the
    first).

4>> "hit in X by Y", "Y wound in X"

    We expect the answer to be "no", but "difficult".  (Even though "a
    bullet wound in the head" sounds like "hit in the head by a
    bullet", these two phrases are different parts of speech: "bullet
    wound in head" is a noun phrase, and "hit in head by bullet" is a
    verb phrase.  Therefore, there is no sentence in which one phrase
    can be substituted by the other.)

5>> "- DATE : bombing at X1 in X2 kills NUM1 people and injures NUM2
     others."

    "Suicide bomber blew himself up at X1 in X2 DATE, killing NUM1
     people and wounding around NUM2 others, police said."
    We expect the answers to be "yes" and "not difficult".  (This is
    similar to example 3, the first can be substituted for the second,
    but the second can not be substituted for the first, since not all
    bombings are suicide bombings).

6>> "Y entered territory in X", "Y raided X town"
    We expect answer "no", and "difficult."  (Although "raiding X
    town" seems to imply "entering territory in X", substituting the
    first for the second results in great loss of information,
    significantly weakening its meaning).

At the top of every page you will have a new pair to judge with
controls to make your selections, and afterwards a list of some of
your recent selections, which you can use to review and modify them in
case of errors (a link at the bottom will show you all previous
selections).

Please do not do more than 60 judgments in one sitting.  Take a break
between each group of judgments that you do.

Some pairs that you will see will look extremely similar to others,
but no two pairs will be identical.  Please read every pair carefully.

Your browser should have JavaScript enabled.

Thanks,
Regina and Lillian.