CCS'16 Reviewing Process — press ⟨space⟩ to advance

CCS'16 Reviewing Process

Shai Halevi
Christopher Kruegel
Andrew Myers

Problem: Scale

  • 831 submitted papers
  • 2.5K total reviews written
  • 141 program committee members (from 20 countries)
  • 360 external reviewers
  • ~17 papers reviewed per PC member
  • 137 (=16.5%) papers accepted (32 conditionally)
  • 60 days


How to assign reviews in 2 days?


  • 2 PC Chairs → 3 PC Chairs
  • Multiple rounds of reviewing (2)
  • Automatic review assignments

Automatic review assignments

  • Previous practice: reviewers express preferences on all papers, conference system tries to match reviewers to papers accordingly.
    Problem: not enough time to scan 831 papers, too much effort.
  • New experiment: use machine learning to automatically assign reviews (Toronto Paper Matching System (TPMS), run by Laurent Charlin at U. Toronto)
    1. Reviewers provide a link to their list of publications instead of preferences, publication PDFs gathered by crawling web sites.
    2. TPMS extracts text from all pubs, converts to “bag of words”
    3. Unsupervised machine learning algorithm generates preference scores based on similarity between documents.
    4. Assignments chosen to optimize preference scores while respecting conflicts.

T-SNE plot of all submissions

Papers shown in red independently labeled as “crypto” by Shai Halevi

Combined embedding

(cyan rectangles are reviewers)


Same technique used to create sessions and to assign papers to rebuttal committee (which minded paper discussions)


Assignments were somewhat better suited to expertise

Effort from PC

Similar or less work for most PC members


Somewhat lower diversity of topics


Generally positive!

Multiple rounds


Comments from PC poll

  • Not enough time in 2nd round to find external reviewers
  • Past papers don't capture current interest. Idea: bid (quickly) for a handful of papers but do the rest automatically.

Thank you!

  • To everyone who submitted papers
  • To the general chairs and other organizers
  • To reviewers (PC and external)
    Shweta Agrawal
    Gail-Joon Ahn
    Martin Albrecht
    Manos Antonakakis
    Frederik Armknecht
    Erman Ayday
    Michael Backes
    Davide Balzarotti
    Karthikeyan Bhargavan
    Alex Biryukov
    Marina Blanton
    Alexandra Boldyreva
    Herbert Bos
    Elie Bursztein
    Kevin Butler
    Juan Caballero
    Yinzhi Cao
    Srdjan Capkun
    David Cash
    Lorenzo Cavallaro
    Yan Chen
    Alessandro Chiesa
    Elisha Choe
    Omar Chowdhury
    Véronique Cortier
    Dana Dachman-Soled
    George Danezis
    Anupam Datta
    Alexander De Luca
    Rinku Dewri
    Adam Doupé
    Tudor Dumitras
    Stefan Dziembowski
    Manuel Egele
    William Enck
    Dario Fiore
    Michael Franz
    Matt Fredrikson
    Xinwen Fu
    Vinod Ganapathy
    Juan Garay
    Deepak Garg
    Cristiano Giuffrida
    Ian Goldberg
    Zhongshu Gu
    Amir Herzberg
    Viet Tung Hoang
    Thorsten Holz
    Amir Houmansadr
    Yan Huang
    Tibor Jager
    Abhishek Jain
    Limin Jia
    Hongxia Jin
    Brent Byunghoon Kang
    Chris Kanich
    Stefan Katzenbeisser
    Florian Kerschbaum
    Dmitry Khovratovich
    Taesoo Kim
    Engin Kirda
    Markulf Kohlweiss
    Vladimir Kolesnikov
    Ralf Kuesters
    Ranjit Kumaresan
    Andrea Lanzi
    Peeter Laud
    Wenke Lee
    Anja Lehmann
    Zhou Li
    Zhenkai Liang
    Benoît Libert
    Zhiqiang Lin
    Yao Liu
    Ben Livshits
    Long Lu
    Matteo Maffei
    Tal Malkin
    Mohammad Mannan
    Sarah Meiklejohn
    Prateek Mittal
    Ian Molloy
    Steven Murdoch
    Arvind Narayanan
    Nick Nikiforakis
    Hamed Okhravi
    Claudio Orlandi
    Xinming Ou
    Charalampos Papamanthou
    Bryan Parno
    Mathias Payer
    Roberto Perdisci
    Adrian Perrig
    Marco Pistoia
    Michalis Polychronakis
    Emmanuel Prouff
    Christina Pöpper
    Zhiyun Qian
    Kui Ren
    Konrad Rieck
    William Robertson
    Alejandro Russo
    Andrei Sabelfeld
    Ahmad-Reza Sadeghi
    Nitesh Saxena
    Jörg Schwenk
    abhi shelat
    Elaine Shi
    Tom Shrimpton
    Kapil Singh
    Sooel Son
    Douglas Stebila
    Gianluca Stringhini
    Thorsten Strufe
    Ed Suh
    Kun Sun
    Jakub Szefer
    Gang Tan
    Stefano Tessaro
    Mohit Tiwari
    Nikos Triandopoulos
    Mahesh Tripunitara
    XiaoFeng Wang
    Zhi Wang
    Hoeteck Wee
    Edgar Weippl
    Dongyan Xu
    Wenyuan Xu
    Daphne Yao
    Ting Yu
    David Zage
    Kehuan Zhang
    Xiangyu Zhang
    Yanchao Zhang
    Yinqian Zhang
    Sheng Zhong
    Haojin Zhu
  • Thanks for coming, see you next year!