CS/INFO 6742 2021 Assignment 1
Task: Propose a research idea related to one
of the readings below and execute a pilot empirical study using one of the
listed datasets. Most crucial to is that (a) your idea
is interesting, and (b) your pilot empirical study demonstrates that you can
quickly evaluate feasibility and estimate the chances of an interesting result.
It is not expected that your proposal for this assignment
will relate to your final course project.
Please strive to post your proposal well in advance of the actual due date
(a suggested goal: Mon Sep 6, 11:59pm), for two reasons.
First, I (and, I hope, your classmates)
need time to be able to post useful replies and feedback
— indeed, perhaps more than one round, time permitting —
to help you refine or adapt it. Second, you are encouraged to work in groups,
and early posting will facilitate linking up with classmates having similar interests.
After posting your proposal, continue to monitor and participate on the course discussion site.
After all, your classmates have read the same papers and are using the same data,
so we have a lot of common ground.
Example things to post: feedback on other people's proposals;
some oddity of the datasets you've found that is worth alerting others to;
unexpected early results that are interesting or that you need help interpreting.
Basically, I would like us all to act as a team; we're all in this together!
The two required readings
- Excerpts from anaesthetica's “Attacked from within”,
2009.
- Dirk Hovy and Diyi Yang, 2021.
The Importance of Modeling Social Factors of Language: Theory and Practice
.
NAACL, pp.588–602.
These readings were chosen because they are thought-provoking, accessible, short,
and together represent a wide range of possibilities.
I do not necessarily agree with any particular point made in these readings.
The two datasets — you are required to use one
- Cornell ChangeMyView data, November 2016 version. (If download fails, it may be that your browser isn't allowing .gz downloads. Use "download link as" or similar right/ctrl-click functionality to check, and if that is the problem, try temporarily disabling Safe Browsing.)
- README for the January 2016 version — still mostly applicable, since the file format did not change. Use the DeltaBot's comments to infer which comments received deltas, rather than trying to identify the various ways in which people assign a delta.
- It is also acceptable to use the ConvoKit version of the data. Persuasion-success labels are included.
- Discussion and example code
- Optional reading: the original paper
in which this dataset was introduced, Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee, 2016,
Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions,
WWW, pp. 613–624.
- Slashdot portion of the British Columbia Conversation Corpora BC3-Blog Corpus
- Sample file: 09_05_25_212203.instancedata.txt
- README, constructed
from the paper in which this dataset was introduced: Nicholas FitzGerald, Giuseppe Carenini, Gabriel Murray, and Shafiq R. Joty, 2011,
Exploiting conversational features to detect high-quality blog comments,
the Canadian Conference on AI, pp. 122–127.
[official link] [author-posted version]
- As stated in the README, some comments are missing. For example,
compare this
original slashdot tree
(http://slashdot.org/story/09/05/25/212203/public-notices-going-online-not-in-newspapers)
with the file slashdot_part_1/09_05_25_212203.instancedata.txt (notice the correspondence between file name and URL). This makes this dataset less than ideal in representing full conversations, but should be fine for our pilot-study purposes.
- The average length of threads in the first split of the data is significantly different
from that in the second portion.
- The "semanticweb.org" URLs given in the files appear to be broken.
- Miscellaneous notes:
(i) Here is the full UBC Conversation Corpora website.
(ii) Why the UBC corpus and not the CAW 2.0 dataset? While the latter may have more complete Slashdot conversation trees, it has fewer of them.
(iii) am aware of this criticism of both the CAW
2.0 dataset and looking at Slashdot in general.
Collaboration
Teamwork is encouraged.
Groups of any size can be formed, where each group jointly submits a single project report
at the end on the official course management system, CMS. However, each individual remains
individually responsible for posting feedback on other people's/group's proposals.
There are further notes on how to find/work as a group below.
Due dates
- Wed Sep 1, 11:59pm:
- Sign in to https://edstem.org using your Cornell NetID
and password, and check/update your settings, such as your display name and notification preferences.
- Send an email to
llee@cs.cornell.edu with subject line "CS/IS 6742 account request"
containing all the following information.
- Your Cornell NetID
(example: LJL2)
- Name you prefer to be referred to
by in this class (example: I prefer to be called "Lillian". Some
other "Lillian" prefer to be called "Lil", but not
me
- Your goals for taking this course
- What background
you have, including but not limited to how you satisfy the three
prerequisites ((a) CS 2110 or equivalent programming experience; (b) a
course in artificial intelligence or any relevant subfield (e.g., NLP,
information retrieval, machine learning, Cornell CS courses numbered 47xx
or 67xx);(c) proficiency with using machine learning tools (e.g., fluency
at training an SVM, comfort with assessing a classifier’s performance
using cross-validation))
Once you send this email, you will be (manually) given access to CMS.
- Thu Sep 9, 11:59pm (as mentioned in the "Task" description above, aim for an earlier date of Mon Sep 6, 11:59pm): Post pilot-study idea(s) to
the course discussion site. (Click the "New Thread"
button and select "Post" (not "Question" or "Announcement".)
- Each idea should be a separate post, to keep discussions organized.
- Choose a title that summarizes your project idea (e.g., “Identifying reviewers with nefarious schemes” as opposed to “Random ideas”).
- For category, select "A1".
- Length expectation: 3+ paragraphs. Be as detailed as possible while remaining sensible. In the ideal case, you'll already have peeked at the data to make sure your idea is going to be feasible.
If persons A, B, and C have already decided to work together, then A should make the idea post, and B and C should each individually comment on A's post stating that they've agreed to work together. This way, can tell who has finished this part of the assignment.
- If, subsequently, D and E want to join forces with A, B and C because your proposals are similar, please arrange to do so among yourselves, and repost a combined proposal to the course discussion site. The deadline for CMS group formation is a bit later than the proposal submission deadline precisely to allow for this possibility.
- Fri Sep 10, 11:59pm: form groups on CMS. CMS group formation requires invitations and acceptance of invitations via the system, i.e., action by two people per person added; please check the official CMS documentation or this more graphically-oriented guide for instructions. need the group information from CMS to schedule the group presentations.
- wed Sep 15, 11:59pm: As a comment to your (group's) pilot-study proposal on the course discussion site, post an update. Being informal is expected, and including any problems you're encountering is most useful and welcome. Lecture the next day will be group meetings with me to discuss the state of your explorations.
- Mon Sep 20, 11:59pm: Submit a project report on CMS. One group = one CMS submission: any
group member can upload a version, which will overwrite any previous versions
by any other members of the group.
Required information: (a) the overall research problem you proposed; (b) relation
of your research problem to the reading(s) (this description should provide
evidence that you read the relevant parts of the readings carefully);
(c) proposed techniques; steps employed to process/clean/select data;
(d) results (probably preliminary, possibly negative); (e) what you learned;
(f) a list of the roles that each member of the group played, if there is more than one person in your group.
(g) If you collaborated a bit with people outside your group, acknowledge those
other people by name and explain their contribution in the writeup.
- Tue Sep 21, in class: Group presentations. Don't worry about being particularly formal, but do be precise about your findings. You can bring handouts (often most effective for discussions, since people can refer to things out of order) or project slides off a laptop.
Depending on the room setup, you may be able to use the projector wirelessly using the Crestron AirMedia app: look for the text "Click here to download" on https://www.crestron.com/Products/Featured-Solutions/AirMedia.