Time and place TuTh 2:55pm-4:10pm, Phillips Hall 101
Instructor: Prof. Cristian Danescu-Niculescu-Mizil --- Office hours: Tu 4:30-5:00pm & Fr 4:30-5:00pm you need to make an appointment
PhD TAs: ,
Graduate TAs: ,
Undergrad TAs listed on Piaza
Office hours schedule listed on Piazza (Resources -> Staff)
Course homepage http://www.cs.cornell.edu/Courses/cs4300/2019sp/
Summary How to make sense of the vast amounts of information available online, and how to relate it and to the social context in which it appears? This course introduces basic tools for retrieving and analyzing unstructured textual infordia. Applications include information retrieval (with human feedback), sentiment analysis and social analysis of text. The coursework will include programming projects that play on the interaction between knowledge and social factors.
Programming proficiency: CS 2110 or equivalent and good Python skills.
Date | Lecture | Agenda | Assignments |
---|---|---|---|
Tu, Jan 22, 2019 |
#1 |
Intro: Dimensions of Information Systems Conversational Behavior and Social information Related material:Linguistic Coordination Toolkit NPR Story: Before The Internet, Librarians Would 'Answer Everything' — And Still Do Google duplex example and writeup in The Verge -- References:Cristian Danescu-Niculescu-Mizil, Lillian Lee, Bo Pang and Jon Kleinberg. Echoes of power: Language effects and power differences in social interaction. Cristian Danescu-Niculescu-Mizil, Michael Gamon and Susan Dumais. Mark my words! Linguistic style accommodation in social media. Proceedings of WWW, 2011. Kate G. Niederhoffer and James W. Pennebaker. Linguistic Style Matching in Social Interaction. Journal of Language and Social Psychology 2002 21: 337. Filip Radlinsky and Nick Craswell. A Theoretical Framework for Conversational Search. Proceedings of CHIIR 2017. |
Setup Quiz out (on CMS) Assignment 0 out (on CMS) |
Th, Jan 24, 2019 |
#2 |
Text similarity measures: Minimum Edit Distance Edit Distance worksheet (includes sketch of the Wagner Fisher algorithm we used in class) Related materialReadings:J&M Chapters 3.11 |
Assignment 1 out (on CMS) |
Tu, Jan 29, 2019 |
#3 |
Basic text processing concepts: Sentence Splitting, Word Tokenization, Types, Tokens Text similarity measures: Type Overlap, Jaccard similarity Classic (ad hoc) information retrieval systems In-class demo: Proto Information Retrieval System: IPython notebook and html Related material:Readings:J&M Chapters 3.8 and 23.1.1 |
|
Th, Jan 31, 2019 |
#4 |
Vector Space Model Dot product similarity, Cosine similarity, Geometric intuition Inverse document frequency (IDF) TF-IDF weighting In-class demo: (continued and updated) IPython notebook and html Readings:MRS Chapters 6.2, 6.3, 6.4.1 and 6.4.4 |
Assignment 2 out (on CMS) |
Tu, Feb 5, 2019 |
#5 |
Term document matrix Efficient retrieval Inverted Index Posting merge algorithm Boolean search In-class demo: (continued and updated) IPython notebook and html Related Material:Readings:MRS Chapter 1 |
|
Th, Feb 7, 2019 |
#6 |
Efficient cosine similarity scoring using the inverted index (algorithm) Fast cosine retrieval worksheet (includes sketch of the algorithm using the inverted index) Related Material:Inspiration for Assignment 3: QUOTUS project and interactive visualization Readings:MRS Chapter 6.3.3 |
Assignment 3 out (on CMS) |
Tu, Feb 13, 2019 |
#7 |
Efficient cosine similarity scoring using the inverted index (implementation) In-class demo: (continued and updated) IPython notebook and html Before optimizing retrieval with inverted indexes (one query on a collection of 40,000 reality TV utterances): After optimizing retrieval with inverted indexes (one query on a collection of 40,000 reality TV utterances): |
|
Th, Feb 14, 2019 |
#8 |
Evaluation of ranked retrieval systems: Precision@k, Precision-recall curve, Mean Average Precision Thinking about evaluation metrics worksheet In-class demo: IPython notebook and html Readings:MSR Chapter 8 |
Assignment 4 out (on CMS) |
Tu, Feb 20, 2019 |
#9 |
Evaluation of ranked retrieval systems: Discounted Cummulative Gain, Pooling, Annotation, K-statistic Readings:MSR Chapter 8 | |