CS/INFO 5306:
Crowdsourcing and Human Computation
Fall 2022

Al Feldstein (1953). Ahead of the Game, Weird Fantasy pp. 9-15, EC Comics, 17b(1).

Course Staff:

Instructor: Prof. Haym Hirsh, Gates 352
Office Hours: Tuesdays 11:30-12:30
CIS Enrollment Administrator: courses@cis.cornell.edu
Project TAs: Zhan Wu and Xiying Zhang
Office Hours: Fridays 2:00-3:00
Course email: Please use cs5306-l@cornell.edu, and include "[5306]" at the start of the subject. Emails to this address will go to all course staff and you will likely get a speedier reply than emailing the instructor directly. However, if you have something you would like to specifically discuss with the instructor, please do so, but still include "[5306]" in the subject.
Office Hours will be given in a Google calendar that will be linked to here once course staffing is finalized and office hours are set.

General Course Information:

CS 5306 versus IS 5306: This course spans topics in both Computer Science and Information Science and thus is cross-listed in both departments. It does not matter which you sign up for, the workload is the same and grading is done across all students regardless of course enrollment.
Prerequisites: This is a 5000-level course in CS and IS. The projects - which count for 75% of your grade - will require having experience programming. If you are unsure, please discuss it with the instructor.
Lectures: Mondays and Wednesdays, 7:30 - 8:45pm, Olin Hall 245.

CS 5306 / INFO 5306 online:

Canvas: The Canvas website for the course will be used as the entry point to all online material for the course, such as the slides for each lecture, and will include links to the other online resources for the course. It will be readable to all Cornell students for the first two weeks of the course, then will be restricted to enrolled students.
Zoom: Lectures will not be available on Zoom unless Cornell's COVID-19 status changes.
Ed Discussion: Online course discussions will take place on Ed Discussion, and is accessible via the course's Canvas page. You will be automatically enrolled in it. You are required to read it - it will be the primary place where we send information to students.
Assignment submissions:
- Projects: All project-related assignments will be submitted via CMS.
- Readings reactions: All readings reactions will be submitted via Canvas.
Textbook/Readings: The course has no textbook. Instead, you will primarily read technical papers and news articles, provided online, for each lecture. The list of readings can be found here.

Grading

The coursework is centered on three things: readings for each lecture, one-off sporadic assignments, and three team projects, according to the following percentages:

20% Readings:
- Readings reactions: For each reading of each lecture you will have some short-answer questions that you will need to answer by 11:59am the day before the lecture. There will be four possible grades:
  - 0: No submission
  - 1: Deficient in some way
  - 2: Acceptable
  - 3: Superb (think of this as an A+)
  The expectation is that students will mostly get 2's as grades, demonstrating that they have read each technical paper.
5% Sporadic assignments: There will be various relatively small assignments given over the course of the semester. Many of these will be about the use of online resources, and will typically be graded 0="submitted"/1="not submitted" unless specified otherwise.
75% Projects: There will be three projects spread out across the semester.
- 25% Project 1: Understanding A Crowdsourced Systems
  - In class project discussions: 7 September 2022
  - Professor meeting slots (optional): 8 September 2022
  - 5% Proposal: 9 September 2022
  - 20% Final deliverables:
    - One-slide summary: 2 October 2022
    - Report/in-class "blurb": 3 October 2022
- 25% Project 2: Mining the Wisdom of a Crowd
  - In class project discussions: 5 October 2022
  - Professor meeting slots (optional): 6 October 2022
  - 5% Proposal: 7 October 2022
  - 20% Final deliverables:
    - Report: 11 November 2022
    - One-slide summary: 13 November 2022
    - In-class "blurb": 14 November 2022
- 25% Project 3: Harnessing Microlabor
  - In class project discussions: 14 November 2022
  - Professor meeting slots (optional): 15 November 2022
  - 5% Proposal: 16 November 2022
  - 20% Final deliverables:
    - One-slide summary: 4 December 2022
    - In-class "blurb": 5 December 2022
    - Report: 15 December 2022
    This final project will require spending money on recruiting crowd workers and running experiments. This is in lieu of textbook costs and should not exceed $50. Please contact the instructor if such costs will pose a hardship
Further information:
- You will work in teams of 3. Occasionally students ask if they can work in a team of 2. Note that any such exceptions must be approved in advance (and may not be approved) and you will still be graded on the expectations of a team of 3.
- The CS Department will be running a teammate-finding event, currently tentatively scheduled for 6 September 2022 at 4pm. Further information will be provided as it becomes available. There will also be teammate-finding opportunities as part of our class meetings.
- The projects will be selected by you. Project ideas will be discussed in class, with meetings with the professor available before the proposal due date.
- Proposals will be graded only as submitted/not submitted. The goal is to give you feedback so you can do a great project.
- You can use any programming language you want, and any existing resources that might be available and relevant to your project - but they must be fully documented. Your project is what you do on top of that.
- Your final deliverables will be:
  - An approximately 5 page report. Although the project will involve programming, your submission is not code, but rather documentation of the outcomes of your project.
  - A one-slide summary of your project outcomes for presentation as a one-minute "blurb" in class.

Attendance and Participation:

The course is in some sense a sampling of data points - examples of work that has been done - in this general area. The format of the class is that each class has readings that all students must do - exposing them to key data points - and lectures amplify beyond the readings and give further data points. Attendance is thus mandatory, and non-attendance will impact the readings component of your grade.

That said, things happen. Be communicative about events intruding in your life. Let us know in advance so we can be responsive to them, rather than telling us much later when it is often far more difficult to do so.

Name cards: You will be given "name tents" during the second week of class. You will be required to bring it to each lecture.

Karma Points: Students are required to attend all class meetings and to be engaged in online discussions. This is captured by "Karma points" for participation. Karma points are not a direct part of the course grading but rather is used for students on the borderline between grades at the end of the semester, and can affect students in either direction. Attendance will be taken at random class meetings. Absences yield negative karma.

Excused Absences:
- Interviews: You are looking for jobs. Your potential employers know that you are a student and know you will need to work around your class schedule. Nonetheless, some absences are unavoidable. You are forgiven two absences due to interviews. To receive credit you must:
  - Email courses@cis.cornell.edu in advance of the lecture
  - Include "[5306 interview]" in the subject field
  - Attach documentation of the interview requiring you to be absent
  - Read the additional papers for that week and submit readings responses for them
- Illness: If you are sick for any reason do not come to class.
  - COVID-19: You must follow Cornell's rules for staying thoughtful about COVID.
    - If you test positive you will receive a letter documenting it. Please email it to courses@cis.cornell.edu and include "[5306 COVID]" in the subject field. Please also let your teammembers on your projects know if relevant.
    - After the first two weeks of the semester we'll arrange for each student to have a "COVID buddy". If you have a mild case and the primary impact is being quarantined, your COVID buddy will be allowed to sit at the front of the class and run zoom so you can attend remotely.
    - If you have a more severe case please take care of yourself and email us when you are feeling better. We will respond thoughtfully to it depending on the situation, including if it involves a team project.
  - Any other illness: Please contact us as soon as possible and we will be responsive to it.
- If anything else intrudes in your studies (such as, for example, a death in the family) please let us know.

Policies

Technology Policy: Discussions will center on papers that you've likely read online and will want to refer to. There may also be occasional in-class exercises online. Technology should only be used in lecture for course related purposes. Frankly, it is obvious when students do not follow this rule, and it will be recorded with your record.
Teams: All three projects in this course will be done in teams. Teamwork is a valuable skill to learn; teams are the norm in industry and you are typically evaluated each year (for promotion or retention) on what you bring to the team and your ability to work in a team. It's a skill that we will work on developing and improving this semester. For each project, your team will be asked to establish expectations in a team contract, and a peer assessment process will be used throughout the semester. In some cases the resulting information can lead to differential grades among team members.

Teams are groups of people, so there is no doubt that conflicts will arise. Part of teamwork is managing that conflict. We expect you to manage conflict professionally and courteously, as you do everything else in this course. The course staff is available to help you with team conflict. In the worst case, the instructor has the power to effect large-scale team change: remove a member, reorganize teams, etc. However, this will not be done until there is an attempt to address issues at hand within the group.
Late work is not accepted, except in case of excused absences as discussed above.

Academic Integrity

From Cornell's code of academic integrity:

Absolute integrity is expected of every Cornell student in all academic undertakings. Integrity entails a firm adherence to a set of values, and the values most essential to an academic community are grounded on the concept of honesty with respect to the intellectual efforts of oneself and others. Academic integrity is expected not only in formal coursework situations, but in all University relationships and interactions connected to the educational process, including the use of University resources. ... A Cornell student's submission of work for academic credit indicates that the work is the student's own. All outside assistance should be acknowledged, and the student's academic position truthfully reported at all times. In addition, Cornell students have a right to expect academic integrity from each of their peers.

Accommodations

This course complies with the Cornell University policy and equal access laws to ensure that students with disabilities can still participate fully in this course. Requests for academic accommodations should be made during the first three weeks of the semester, except for unusual circumstances, so arrangements can be made as soon as possible. Students are encouraged to register with Student Disability Services, as we may require verification of eligibility to provide appropriate accommodations.

Respect for Others

Everyone, the instructors, TAs, and students, must be respectful of everyone else in this class. All communication, in class and online, will be held to a high standard for thoughtfulness and inclusiveness: it may never target individuals or groups for harassment, and it may not exclude specific groups. If any of the communication in this class doesn't meet these standards, please don't ignore it, but also don't escalate it by responding in kind. Instead, contact the instructor as early as possible; if for whatever reason you don't feel comfortable discussing something directly with the instructor please contact your advising office or the department chair.

CS/INFO 5306: Crowdsourcing and Human Computation Fall 2022