initial reviews written without authorship knowledge
authors revealed during discussion period
Reviewing broken into two rounds
Some submissions rejected after first round (2 reviews): 18%
Authors wrote rebuttals to reviews
Things that changed from 2017
Automatic assignment of reviews based on machine learning algorithm
No physical PC meeting—electronic discussion only
One large PC—no EPC or ERC
All papers accepted conditionally and shepherded (for PACM)
Automatic review assignments
The process
Previous practice: reviewers express preferences on all papers,
conference system tries to match reviewers to papers accordingly. Problem: scanning almost 300 papers is a lot of work for the commitee.
Experiment for POPL: automatically assign reviews
using Toronto Paper Matching System (TPMS),
run by Laurent Charlin at U. Toronto
Reviewers provide a link to their list of publications instead
of preferences, publication PDFs gathered by crawling web sites.
TPMS extracts text from all pubs, converts to “bag of words”
Unsupervised machine learning algorithm generates preference scores
based on similarity between documents.
Assignments chosen to optimize preference scores while respecting
conflicts.
T-SNE plot of all submissions
T-SNE plot of all reviewers
(different projection)
Appropriateness
Assignments were about as well suited to expertise as with
manual assignment. Reviewers rated themselves as “expert”
on 44% of their reviews, the same percentage as in 2017.
Effort from PC
Similar or less work for most PC members
Diversity
Diversity of topics mostly okay, possibly a little too high.
Overall
Range of opinions, skewing positive.
Online PC discussion
Rationale: Value of physical program committee meeting
diminishes with increasing PC size while costs increase.
Effect on recruitment
A significant positive factor for many PC members
Engagement
Not much effect on perceived engagement
Discussion committee
Assigning a PC member to push discussion along helped
Review quality
Review quality was perceived as about the same. Actually,
15% more review text was generated per paper than in 2017.
Overall reaction
Caveat: online poll evaluating online reviewing could
have selection bias
Selected comments from PC (edited)
I thought the online discussion worked well, provided
the PC members and discussion leads were proactive. The trouble is
it's far too easy to agree to do lots of other stuff during the
discussion period
I meet interesting people I haven't met before at PC
meetings...in a way this is a reward for the hard work you put into it.
I strongly appreciate the opportunity to gather my thoughts, reread
others reviews and explain myself in writing.
In-person discussion is more efficient because you get a lot
more information about how PC members really feel about the paper
from informal cues...perhaps schedule Skype calls for specific
papers?
It was good that the reviewing process was in two stages, this
forced me (and everyone else) to do and submit reviews in a
timely manner and increased the quality of the reviews and
discussion IMO.
The problem was the lack of an overall view of the submissions.
With a normal PC meeting, everyone gets that view just by
showing up
Online PC meetings worked pretty well for POPL...
make it very very clear to PC members that they must be
reactive and participate every single day of the
discussion period.