Q: Why are you using double-blind reviewing (DBR)?
A: Our goal is to let PC members and other reviewers take an unbiased "first look" at each paper (wherever possible), without such involuntary reactions as "Barnaby; he writes a good paper" or "Who are these people? I have never heard of them".
Q: What does this mean for submitted papers?
A: Submitted papers must:
We do not want this DBR process to inhibit normal dissemination of scientific ideas: submitted papers may be posted to author web pages and the work can be discussed in seminars, workshops, and mailing lists as usual.
A: No. But we recommend that you do not use the same title for your POPL submission as any previous workshop submission.
A: No. The relationship between systems and authors changes over time, so there will be at least some doubt about authorship. Increasing this doubt by changing the system name would help with anonymity, but it would compromise the research process. In particular, changing the name requires explaining a lot about the system again because you can't just refer to the existing papers, which use the proper name. Not citing these papers runs the risk of the reviewers who know about the existing system thinking you are replicating earlier work. It is also confusing for the reviewers to read about the paper under Name X and then have the name be changed to Name Y.
A: Yes. Double-blind reviewing should not hinder the usual communication of results - see above. .
A: On the submission site there will be an option to submit supplementary material along with your main paper. This supplementary material need not be anonymized; it will only be revealed to reviewers after they have submitted their review of your paper (or a first draft thereof) and thereby learned your identity. Reviewers are under no obligation to look at this material. The submission itself is the object of review and so it should strive to convince the reader of at least the plausibility of reported results; supplemental material only serves to confirm, in more detail, the idea argued in the paper. Of course, reviewers are free to change their review upon viewing supplemental material (or for any other reason). For those authors who include supplementary material, they should mention it in the body of the paper to prevent it being overlooked. E.g., "The proof of Lemma 1 is included in the non-anonymous supplemental material submitted with this paper".
Q: Can I place supplementary material (anonymous or not) at a URL?
A: No. We encourage authors to provide any supplementary material that is required to support the claims made in the paper, such as detailed proofs, proof scripts, or experimental data, but making supplementary material available via URLs is often awkward: there is a potential loss of confidentiality (or less informed reviews, if the reviewer avoids downloading material just because of that); sometimes there are errors in the URLs; and some authors complete or update their supplementary material after the submission deadline, which is unfair to those that respect the deadline. Hence, for POPL 2014 we require supplementary material to be uploaded at submission time, as a single pdf or a tarball, not via a URL. It need not be anonymised, and so will be made available to reviewers only after they have submitted their first-draft review. If you would like to include a link to material that is intrinsically not uploadable (e.g. to a web interface of a running system), you can include that link in the uploaded supplementary material (and mention its existence in the paper). As usual, reviewers are under no obligation to look at the supplementary material.
A: Not for POPL 2014.
A: Using DBR does not change the principle that reviewers should not review papers with which they have a conflict of interest, even if they do not immediately know who the authors are. Quoting (with slight alteration) from the ACM SIGPLAN review policies document:
A conflict of interest is defined as a situation in which the reviewer can be viewed as being able to benefit personally in the process of reviewing a paper. For example, if a reviewer is considering a paper written by a member of his own group, a current student, his advisor, or a group that he is seen as being in close competition with, then the outcome of the review process can have direct benefit to the reviewer's own status. If a conflict of interest exists, the potential reviewer should decline to review the paper.As an author, you should list PC members and any others (since others may be asked for external reviews) which you believe have a conflict with you. While particular criteria for making this determination may vary, please apply the following guidelines, identifying a potential reviewer Bob as conflicted if
If a possible reviewer does not meet the above criteria, please do not identify him/her as conflicted. Doing so could be viewed as an attempt to prevent a qualified, but possibly skeptical reviewer from reviewing your paper. If you nevertheless believe that a reviewer who does not meet the above criteria is conflicted, you may identify the person and send a note to the PC Chair.
Q: What counts as ``explicit lobbying''?
A: The current version of the Principles of POPL states that "Authors should not, however, take [...] as a license to explicitly lobby the PC or likely reviewers on behalf of their work; doing so would be contrary to the intent of DBR.". For example, mass-mailing the PC a de-anonymised version of the paper would be contrary to the intent of DBR. On the other hand, giving a seminar at the institution of a PC member (if you would have done that naturally irrespective of whether the work is submitted to POPL) is fine, as is submitting a related talk proposal to a workshop. (Of course these may mean that some PC members will be able to guess the authors of more papers, making the blinding somewhat less effective. We accept that, to avoid DBR damaging the normal processes of the reearch community.)
A: If you feel that the authors' actions are principally aimed at ensuring that potential reviewers know their identity, you should contact the Program Chair. Otherwise you should not treat double-blind reviewing differently from regular blind reviewing. In particular, while you should generally refrain from seeking out information on the authors' identity, if you discover it accidentally this will not disqualify you as a reviewer.
Q: Can I deliberately de-blind a submission that I am reviewing?
A: This is left to your discretion: we expect that it will only rarely be required, but if you think it necessary to de-blind a submission in order to write a good review (e.g., if you think you have to see the supplementary material first), you can do so by submitting a placeholder review. We encourage (but do not require) you to write a first-draft review before doing this.
Q: How do we handle potential conflicts of interest since I cannot see the author names?
A: The conference review system will ask that you identify conflicts of interest when you get an account on the submission system. Please see the related question applied to authors to decide how to identify conflicts. Feel free to also identify additional authors whose papers you feel you could not review fairly for reasons other than those given (e.g., strong personal friendship).
Q: How will we identify expert external reviewers? How will the system of "guardians" work?
A: Each PC member must write reviews for all the papers assigned to him/her, but can suggest additional external reviewers to the paper guardians, as below.
We'll try to get at least one external review for each paper and at least two expert reviews for each paper in total. To manage this, each paper will be assigned a "guardian" from among the PC; based on the POPL 2013 numbers, each PC member will be guardian for around 9 papers. We will try to assign a PC member who bid strongly for the paper. The guardian will be responsible (in discussion with the PC chair as necessary) for ensuring that we seek appropriate expert external reviewers for each paper. To help with this:
Q: If I am assigned a paper for which I feel I am not an expert, how do I seek an outside review?
A: PC members should do their own reviews, not delegate them to someone else, but if you are aware of a good potential reviewer for a particular paper, you should let the designated guardian of that paper know.