Labelings for the paper: For the sake of simplicity: Unsupervised extraction of lexical simplifications from Wikipedia. Mark Yatskar, Bo Pang, Cristian Danescu-Niculescu-Mizil and Lillian Lee. Proceedings of NAACL 2010. Short paper. http://www.cs.cornell.edu/home/llee/papers/simple.home.html version 1.0, released April 1, 2010. =============== The files consist of the pairs that were labeled by our 6 annotators (3 native English speakers and three non-native English speakers). The pairs are in the order in which they were presented to the labeler. The labels are: 1: simpler 0: more complex e: equal u: unrelated ?: hard to judge The instructions provided to the human labelers are as follows. Instructions! This survey asks questions in the following form: 'PHRASE ONE' is simpler than 'PHRASE TWO' y/n/?/u/e/q=quit/h=help/c=toLabel: This question is asking two things: (a) can 'PHRASE ONE' replace 'PHRASE TWO' in SOME context (perhaps not every context!)? (b) would replacing 'PHRASE TWO' with 'PHRASE ONE' result in a simpler expression? (1) If your answer to (a) is no, then you will always give the answer 'u' (unreplacable). (2) If your answer to (a) is yes and the two phrases are equally difficult/simple, give the answer 'e' (equal) (2) If your answer to (a) is yes,and 'PHRASE ONE' is simpler than 'PHRASE TWO' give the answer 'y' (yes). (3) If your answer to (a) is yes, and 'PHRASE ONE' is more complicated than 'PHRASE TWO' give answer 'n' (no). (4) If your answer to (a) is yes, but it's difficult to answer (b), give answer '?' (not sure). (5) [garbage case] If either 'PHRASE ONE' or 'PHRASE TWO' seem to contain easily preventable computer errors (for example, if you see stray HTML markup: ' kittens' vs 'kittens'), please also answer '?'. You can quit at any time by hitting 'q' or 'ctrl-c'. YOUR COMPLETED RESPONSES WILL NOT BE LOST. Feel free to do this in more than one sitting. Also, to get this help again, hit 'h'. To see how many you have left to label, hit 'c'. The survey begins with a set of questions to test whether or not you understand the annotation scheme. You only have to get by this portion once. So, after you pass, you will not be prompted to do it again if you choose to stop and come back later. EXAMPLES: 'juvenile' is simpler than 'young' y/n/?/u/e/q=quit/h=help/c=toLabel:n 'good' is simpler than 'impeccable' y/n/?/u/e/q=quit/h=help/c=toLabel:y 'Obama' is simpler than 'Hitler' y/n/?/u/e/q=quit/h=help/c=toLabel:u 'infinitesimal' is simpler than 'very small' y/n/?/u/e/q=quit/h=help/c=toLabel:n 'creative drive' is simpler than 'willingness to experiment' y/n/?/u/e/q=quit/h=help/c=toLabel:u 'small' is simpler than 'not big' y/n/?/u/e/q=quit/h=help/c=toLabel:e -because both are surely simple. 'happy' is simpler than 'happy' y/n/?/u/e/q=quit/h=help/c=toLabel:? -clear computer error.