Class Schedule for CS/INFO 6306: Advanced Human Computation
Week 1:
- August 25: Introduction (Slides)
Assignments for next time
- Email haym.hirsh@cornell.edu: Your name and email, Program of study and year, and Interests and background
- Initiate setting up a worker account at Amazon Mechanical Turk: Go to mturk.com, click on “Sign in as a Worker” at upper right, enter your email, click on “No, I am a new customer” and proceed from there
- Complete Cornell IRB Training
Readings for next time:
- Required readings:
- Additional readings:
- Doan, Anhai, Raghu Ramakrishnan, and Alon Y. Halevy. "Crowdsourcing systems on the world-wide web." Communications of the ACM 54.4 (2011): 86-96.
- Howe, Jeff. "The rise of crowdsourcing." Wired magazine 14.6 (2006): 1-4.
- Kalil, Thomas A. "Leveraging cyberspace." IEEE Communications Magazine 34.7 (1996): 82-86.
- Law, Edith, and Luis von Ahn. "Human computation." Synthesis Lectures on Artificial Intelligence and Machine Learning 5.3 (2011): 1-121. Chapters 1 and 2.
- Malone, Thomas W., Robert Laubacher, and Chrysanthos Dellarocas. "The collective intelligence genome." MIT Sloan Management Review 51.3 (2010): 21.
- Surowiecki, James. The wisdom of crowds. Anchor, 2005. Chapter 1.
Week 2
- August 30: Introduction (continued) (Slides)
Readings for next time:
- Required readings:
- Khatib, Firas, Seth Cooper, Michael D. Tyka, Kefan Xu, Ilya Makedon, Zoran Popovi?, and David Baker. "Algorithm discovery by protein folding game players." Proceedings of the National Academy of Sciences 108, no. 47 (2011): 18949-18953.
- Tuite, Kathleen, Noah Snavely, Dun-Yu Hsiao, Adam M. Smith, and Zoran Popovi?. "Reconstructing the world in 3D: bringing games with a purpose outdoors." In Proceedings of the Fifth International Conference on the Foundations of Digital Games, pp. 232-239. ACM, 2010.
- Additional readings:
- September 1: Games with a Purpose (overt) (Slides)
Readings for next time:
- Required readings:
- Barowy, D. W., Curtsinger, C., Berger, E. D., & McGregor, A. (2012). “Automan: A platform for integrating human-based and digital computation.” ACM SIGPLAN Notices, 47(10), 639-654.
- Little, Greg, Lydia B. Chilton, Max Goldman, and Robert C. Miller. "Turkit: human computation algorithms on mechanical turk." In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 57-66. ACM, 2010.
- Additional readings:
- Kittur, Aniket, Boris Smus, Susheel Khamkar, and Robert E. Kraut. "Crowdforge: Crowdsourcing complex work." In Proceedings of the 24th annual ACM symposium on User interface software and technology, pp. 43-52. ACM, 2011.
- Patrick Minder, Abraham Bernstein, ”CrowdLang: A Programming Language for the Systematic Exploration of Human Computation Systems.” Fourth International Conference on Social Informatics, 2012.
- Atsuyuki Morishima, Norihide Shinagawa, Tomomi Mitsuishi, Hideto Aoki, Shun Fukusumi. “CyLog/Crowd4U: A Declarative Platform for Complex Data-centric Crowdsourcing.” PVLDB 5(12): 1918-1921 (2012)
- Tosch, Emma, and Emery D. Berger. "Surveyman: Programming and automatically debugging surveys." ACM SIGPLAN Notices. Vol. 49. No. 10. ACM, 2014.
- Tranquillini, S., Daniel, F., Kucherbaev, P., & Casati, F. (2015). “Modeling, enacting, and integrating custom crowdsourcing processes.” ACM Transactions on the Web, 9(2), 7.
Week 3
- September 6: Programming Languages (Slides)
Readings for next time:
- Required Readings:
- Franklin, M.J., Kossmann, D., Kraska, T., Ramesh, S. and Xin, R., 2011. “CrowdDB: answering queries with crowdsourcing.” In Proceedings of the 2011 ACM SIGMOD International Conference on Management of data(pp. 61-72). ACM.
- Marcus, A., Wu, E., Karger, D.R., Madden, S. and Miller, R.C., 2011. “Crowdsourced databases: Query processing with people.” In Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR 2011).
- Additional Readings:
- Davidson, S.B., Khanna, S., Milo, T. and Roy, S., 2013. “Using the crowd for top-k and group-by queries.” In Proceedings of the 16th International Conference on Database Theory (pp. 225-236). ACM.
- Guo, S., Parameswaran, A. and Garcia-Molina, H., 2012. “So who won?: dynamic max discovery with the crowd.” In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (pp. 385-396). ACM.
- Marcus, A., Wu, E., Karger, D., Madden, S. and Miller, R., 2011. “Human-powered sorts and joins.” Proceedings of VLDB, 5(1), pp.13-24.
- Parameswaran, A.G., Park, H., Garcia-Molina, H., Polyzotis, N. and Widom, J., 2012. “Deco: declarative crowdsourcing.” In Proceedings of the 21st ACM international conference on Information and knowledge management (pp. 1203-1212). ACM.
- Wang, J., Kraska, T., Franklin, M.J. and Feng, J., 2012. “Crowder: Crowdsourcing entity resolution.” Proceedings of VLDB,5(11), pp.1483-1494.5
- September 8: Database Perspective (Slides)
Homework Assignment 1: Due Thursday, 15 September 2016
Readings for next time:
- Required Readings:
- Additional readings:
- Bachrach, Yoram, Thore Graepel, Gjergji Kasneci, Michal Kosinski, and Jurgen Van Gael. "Crowd IQ: aggregating opinions to boost performance." In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, pp. 535-542.
- Dasgupta, A. and Ghosh, A., 2013, May. "Crowdsourced judgement elicitation with endogenous proficiency." In Proceedings of the 22nd international conference on World Wide Web (pp. 319-330). ACM.
- Easley, D. and Ghosh, A., 2013, June. "Incentives, gamification, and game theory: an economic approach to badge design." In Proceedings of the fourteenth ACM conference on electronic commerce (pp. 359-376). ACM.
- Jain, S., Chen, Y. and Parkes, D.C., 2009, July. "Designing incentives for online question and answer forums." In Proceedings of the 10th ACM conference on electronic commerce (pp. 129-138). ACM.
- Jain, S. and Parkes, D.C., 2013. "A game-theoretic analysis of the ESP game. ACM Transactions on Economics and Computation, 1(1), p.3.
- Kamar, E. and Horvitz, E., 2012. "Incentives and truthful reporting in consensus-centric crowdsourcing." Technical report, MSR-TR-2012-16, Microsoft Research.
- Moshfeghi, Y., Rosero, A.F.H. and Jose, J.M., 2016. A Game-Theory Approach for Effective Crowdsource-Based Relevance Assessment. ACM Transactions on Intelligent Systems and Technology (TIST), 7(4), p.55.
- Naroditskiy, V., Jennings, N.R., Van Hentenryck, P. and Cebrian, M., 2014. "Crowdsourcing contest dilemma." Journal of The Royal Society Interface,11(99), p.20140532.
- Pickard, G., Pan, W., Rahwan, I., Cebrian, M., Crane, R., Madan, A. and Pentland, A., 2011. "Time-critical social mobilization." Science,334(6055), pp.509-512.
- Shah, N.B. and Zhou, D., 2015. "Double or nothing: Multiplicative incentive mechanisms for crowdsourcing." In Advances in Neural Information Processing Systems (pp. 1-9).
- Shah, N.B., Zhou, D. and Peres, Y., 2015. "Approval voting and incentives in crowdsourcing. In International Conference on Machine Learning (ICML).
- Ugander, J., Drapeau, R. and Guestrin, C., 2015. "The Wisdom of Multiple Guesses. In Proceedings of the Sixteenth ACM Conference on Economics and Computation (pp. 643-660). ACM.
- Wu, W., Daskalakis, C., Kaashoek, N., Tzamos, C. and Weinberg, M., 2015. "Game theory based peer grading mechanisms for MOOCs." In Proceedings of the Second ACM Conference on Learning@Scale (pp. 281-286). ACM.
Week 4
- September 13: Game Theoretic Approaches (Slides)
Readings for September 20:
- Required Readings:
- Martin, D., Hanrahan, B.V., O'Neill, J. and Gupta, N., 2014. "Being a turker.” In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (pp. 224-235). ACM.
- Irani, L.C. and Silberman, M., 2013. “Turkopticon: interrupting worker invisibility in Amazon Mechanical Turk.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 611-620). ACM.
- Additional readings:
- Borokhovich, M., Chatterjee, A., Rogers, J., Varshney, L.R. and Vishwanath, S., 2015. “Improving impact sourcing via efficient global service delivery.” In Proceedings Data for Good Exchange (D4GX).
- Brawley, A.M. and Pury, C.L., 2016. “Work experiences on MTurk: Job satisfaction, turnover, and information sharing.” Computers in Human Behavior, 54, pp.531-546.
- Gray, M.L., Suri, S., Ali, S.S. and Kulkarni, D., 2016. “The crowd is a collaborative network.” In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 134-147). ACM.
- Gupta, N., Martin, D., Hanrahan, B.V. and O'Neill, J., 2014. “Turk-life in India.” In Proceedings of the 18th International Conference on Supporting Group Work (pp. 1-11). ACM.
- Kokkalis, N., Köhn, T., Pfeiffer, C., Chornyi, D., Bernstein, M.S. and Klemmer, S.R., 2013. “EmailValet: Managing email overload through private, accountable crowdsourcing.” In Proceedings of the 2013 conference on Computer supported cooperative work (pp. 1291-1300). ACM.
- Lee, M.K., Kusbit, D., Metsky, E. and Dabbish, L., 2015. “Working with machines: The impact of algorithmic and data-driven management on human workers.” In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 1603-1612). ACM.
- McInnis, B., Cosley, D., Nam, C. and Leshed, G., 2016. “Taking a HIT: Designing around Rejection, Mistrust, Risk, and Workers’ Experiences in Amazon Mechanical Turk.” In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 2271-2282). ACM.
- Salehi, N., Irani, L.C., Bernstein, M.S., Alkhatib, A., Ogbe, E. and Milland, K., 2015. “We are dynamo: Overcoming stalling and friction in collective action for crowd workers.” In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 1621-1630). ACM.
- September 15: Game Theoretic Approaches (Ian Arawjo)
Week 5
- September 20: Human Side of Human Computation
- September 22: Human Side of Human Computation (Sydney Zink)
Week 6
- September 27: Cognitive Factors in Human Computation
- September 29: Cognitive Factors in Human Computation (Brendan Rappazzo)
Week 7
Week 8
Week 9
Week 10
Week 11
Week 12
Week 13
Week 14
Week 15
Last update: 13 September 2016