Using the Review Normalizer with conference management systems


HotCRP provides the ability to download all review scores. Search on all papers and select all of them, then download Reviews/Scores. The columns “paper”, “reviewername”, and “OveMer” contain the needed data. Delete all other columns and rename these columns “Paper”, “Reviewer”, and “Score” respectively.

Alternatively, if you have direct access to the database, you can extract all the data needed by the Review Normalizer with some simple queries. Assuming that your data is stored in a MySQL database named ⟨database⟩, the following Unix commands will extract the data:

echo 'Paper,Reviewer,Score'
echo 'select paperId, email, overAllMerit from ContactInfo, PaperReview \
    where overAllMerit <> 0 and ContactInfo.contactId=PaperReview.contactId' | \
    mysql --user ⟨user⟩ --password ⟨password⟩ ⟨database⟩ -B | \
    tr '\t' , | tail -n +2 >! reviews1.csv

To extract the label CSV file, use the following commands:

echo 'Paper,Label'
echo "select paperId, outcome from Paper | \
    mysql --user ⟨user⟩ --password ⟨password⟩ ⟨database⟩ -B | \
    tr '\t' , | \
    tail -n +2 

If you don't have direct database access, you can still download via the regular HotCRP interface. Just select all submitted papers and download "Scores", which will give you a CSV file. Delete the "title", "decision", "email", and "RevExp" columns, and change the other column headers to match what RevNorm expects: "Paper", "Reviewer", and "Score".