POPL 2018 Program Chair Report—Andrew Myers

POPL 2018
PC Chair Report

Andrew Myers


  • 271 submitted papers
  • 66 (=24%) papers accepted (all conditionally)
  • 52 program committee members (from 15 countries)
  • 130 external reviewers
  • up to 16 papers reviewed per PC member
  • 1.1M words of review text written

Reviewing process: Things that did not change

  • “Lightweight” double-blind reviewing process
    • initial reviews written without authorship knowledge
    • authors revealed during discussion period
  • Reviewing broken into two rounds
    • Some submissions rejected after first round (2 reviews): 18%
  • Authors wrote rebuttals to reviews

Things that changed from 2017

  • Automatic assignment of reviews based on machine learning algorithm
  • No physical PC meeting—electronic discussion only
  • One large PC—no EPC or ERC
  • All papers accepted conditionally and shepherded (for PACM)

Automatic review assignments

The process

  • Previous practice: reviewers express preferences on all papers, conference system tries to match reviewers to papers accordingly.
    Problem: scanning almost 300 papers is a lot of work for the commitee.
  • Experiment for POPL: automatically assign reviews using Toronto Paper Matching System (TPMS), run by Laurent Charlin at U. Toronto
    1. Reviewers provide a link to their list of publications instead of preferences, publication PDFs gathered by crawling web sites.
    2. TPMS extracts text from all pubs, converts to “bag of words”
    3. Unsupervised machine learning algorithm generates preference scores based on similarity between documents.
    4. Assignments chosen to optimize preference scores while respecting conflicts.

T-SNE plot of all submissions

T-SNE plot of all reviewers

(different projection)


Assignments were about as well suited to expertise as with manual assignment. Reviewers rated themselves as “expert” on 44% of their reviews, the same percentage as in 2017.

Effort from PC

Similar or less work for most PC members


Diversity of topics mostly okay, possibly a little too high.


Range of opinions, skewing positive.

Online PC discussion

Rationale: Value of physical program committee meeting diminishes with increasing PC size while costs increase.

Effect on recruitment

A significant positive factor for many PC members


Not much effect on perceived engagement

Discussion committee

Assigning a PC member to push discussion along helped

Review quality

Review quality was perceived as about the same. Actually, 15% more review text was generated per paper than in 2017.

Overall reaction

Caveat: online poll evaluating online reviewing could have selection bias

Selected comments from PC (edited)

  • I thought the online discussion worked well, provided the PC members and discussion leads were proactive. The trouble is it's far too easy to agree to do lots of other stuff during the discussion period
  • I meet interesting people I haven't met before at PC meetings...in a way this is a reward for the hard work you put into it.
  • I strongly appreciate the opportunity to gather my thoughts, reread others reviews and explain myself in writing.
  • In-person discussion is more efficient because you get a lot more information about how PC members really feel about the paper from informal cues...perhaps schedule Skype calls for specific papers?
  • It was good that the reviewing process was in two stages, this forced me (and everyone else) to do and submit reviews in a timely manner and increased the quality of the reviews and discussion IMO.
  • The problem was the lack of an overall view of the submissions. With a normal PC meeting, everyone gets that view just by showing up
  • Online PC meetings worked pretty well for POPL... make it very very clear to PC members that they must be reactive and participate every single day of the discussion period.

Thank you!

  • Everyone who submitted papers
  • General chair Ranjit Jhala and other organizers
  • Andy Gordon (POPL'17 PC Chair) for great advice
  • All reviewers (PC and external)
    Thorsten Altenkirch
    Nada Amin
    Gilles Barthe
    Mark Batty
    Lars Birkedal
    Pavol Cerny
    James Cheney
    Koen Claessen
    William Cook
    Karl Crary
    Loris D'Antoni
    Mila Dalla Preda
    Isil Dillig
    Sophia Drossopoulou
    Constantin Enea
    Azadeh Farzan
    Xinyu Feng
    Atsushi Igarashi
    Laura Kovacs
    Neel Krishnaswami
    Sorin Lerner
    Xavier Leroy
    Conor McBride
    Mira Mezini
    Matthew Might
    Peter Müller
    Santosh Nagarakatte
    Aleksandar Nanevski
    Klaus Ostermann
    Ruzica Piskac
    Nadia Polikarpova
    Zvonimir Rakamaric
    Aseem Rastogi
    Grigore Rosu
    Andrey Rybalchenko
    Amr Sabry
    David Sands
    Davide Sangiorgi
    Peter Sewell
    Zhong Shao
    Yannis Smaragdakis
    Zhendong Su
    Éric Tanter
    Zachary Tatlock
    Tachio Terauchi
    Viktor Vafeiadis
    Philip Wadler
    David Walker
    Stephanie Weirich
    Jean Yang
    Danfeng Zhang
  • Thanks for attending!