An increasing number of domains are providing us with detailed trace data on human decisions, often made by experts with deep experience in the subject matter. This offers an opportunity to use prediction algorithms to ask several families of questions --- not only about how algorithms compare with expert-level human decision-making in specific domains, but also whether we can use algorithms to analyze the nature of the errors and potential biases of the human experts, to predict which instances will be hardest for these experts, and to explore some of the ways in which prediction algorithms can serve as supplements to human decision-making in different applications. 

We compare human and algorithmic decision-making in a context with important policy implications: judicial decisions on bail. By law, such decisions hinge on a judge’s prediction of what a defendant would do if released; it is thus a promising algorithmic application because it is a concrete prediction task for which there is a large volume of data available. Yet comparing algorithms to judges in this setting proves complicated: the data are themselves generated by prior judge decisions, and we only observe crime outcomes for released defendants, not for those whom the judges detained. We develop a set of techniques for dealing with these challenges, and explore a set of further issues, including questions of algorithmic fairness, and a set of analyses that focus on predicting judges' decisions as a way of gaining insight into their decision-making. 

This is joint work with Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 

Jon Kleinberg is a professor in both Computer Science and Information Science. His research focuses on issues at the interface of networks and information, with an emphasis on the social and information networks that underpin the Web and other on-line media. His work has been supported by an NSF Career Award, an ONR Young Investigator Award, a MacArthur Foundation Fellowship, a Packard Foundation Fellowship, a Sloan Foundation Fellowship, and grants from Google, Yahoo!, and the NSF. He is a member of the National Academy of Sciences, the National Academy of Engineering, and the American Academy of Arts and Sciences.