POPL 2014: FAQ concerning lightweight double-blind reviewing

This FAQ is adapted, with thanks, from that produced by Mike Hicks for POPL 2012 (which includes links to background studies on reviewing). Note, however, that the POPL 2014 and 2012 processes are not identical.

Q: Why are you using double-blind reviewing (DBR)?

A: Our goal is to let PC members and other reviewers take an unbiased "first look" at each paper (wherever possible), without such involuntary reactions as "Barnaby; he writes a good paper" or "Who are these people? I have never heard of them".

Q: What does this mean for submitted papers?

A: Submitted papers must:

We are not trying to make it impossible for reviewers to discover the authors identities if they try: we intend this process to be cooperative, not adversarial. If a reviewer does discover an author's identity though a subtle clue or oversight the author will not be penalized. Moreover, nothing should be done in the name of anonymity that weakens the submission or makes the job of reviewing the paper more difficult (e.g., important background references should not be omitted or anonymized).

We do not want this DBR process to inhibit normal dissemination of scientific ideas: submitted papers may be posted to author web pages and the work can be discussed in seminars, workshops, and mailing lists as usual.

For authors

Q: I am submitting a paper that extends my own work that previously appeared at a workshop or conference. Should I anonymize any reference to that prior work?

A: No. But we recommend that you do not use the same title for your POPL submission as any previous workshop submission.

Q: I am building on my own past work on the WizWoz system. Do I need to rename this system in my paper for purposes of anonymity, so as to remove the implied connection between my authorship of past work on this system and my present submission?

A: No. The relationship between systems and authors changes over time, so there will be at least some doubt about authorship. Increasing this doubt by changing the system name would help with anonymity, but it would compromise the research process. In particular, changing the name requires explaining a lot about the system again because you can't just refer to the existing papers, which use the proper name. Not citing these papers runs the risk of the reviewers who know about the existing system thinking you are replicating earlier work. It is also confusing for the reviewers to read about the paper under Name X and then have the name be changed to Name Y.

Q: Am I allowed to post my (non-blinded) paper on my web page? Can I advertise the unblinded version of my paper on mailing lists or send it to colleagues? May I give a talk about my work while it is under review?

A: Yes. Double-blind reviewing should not hinder the usual communication of results - see above. .

Q: I would like to provide supplementary material for consideration, e.g., the code of my implementation or proofs of theorems. How do I do this?

A: On the submission site there will be an option to submit supplementary material along with your main paper. This supplementary material need not be anonymized; it will only be revealed to reviewers after they have submitted their review of your paper (or a first draft thereof) and thereby learned your identity. Reviewers are under no obligation to look at this material. The submission itself is the object of review and so it should strive to convince the reader of at least the plausibility of reported results; supplemental material only serves to confirm, in more detail, the idea argued in the paper. Of course, reviewers are free to change their review upon viewing supplemental material (or for any other reason). For those authors who include supplementary material, they should mention it in the body of the paper to prevent it being overlooked. E.g., "The proof of Lemma 1 is included in the non-anonymous supplemental material submitted with this paper".

Q: Can I place supplementary material (anonymous or not) at a URL?

A: No. We encourage authors to provide any supplementary material that is required to support the claims made in the paper, such as detailed proofs, proof scripts, or experimental data, but making supplementary material available via URLs is often awkward: there is a potential loss of confidentiality (or less informed reviews, if the reviewer avoids downloading material just because of that); sometimes there are errors in the URLs; and some authors complete or update their supplementary material after the submission deadline, which is unfair to those that respect the deadline. Hence, for POPL 2014 we require supplementary material to be uploaded at submission time, as a single pdf or a tarball, not via a URL. It need not be anonymised, and so will be made available to reviewers only after they have submitted their first-draft review. If you would like to include a link to material that is intrinsically not uploadable (e.g. to a web interface of a running system), you can include that link in the uploaded supplementary material (and mention its existence in the paper). As usual, reviewers are under no obligation to look at the supplementary material.

Q: Is there a way for me to submit anonymous supplemental material which could be considered by a reviewer before she submits her review (rather than potentially non-anonymous material that can only be viewed afterward) ?

A: Not for POPL 2014.

Q: Will the fact that POPL is double-blind have an impact on handling conflicts-of interest? When I am asked by the submission system to identify conflicts of interest, what criteria should I use?

A: Using DBR does not change the principle that reviewers should not review papers with which they have a conflict of interest, even if they do not immediately know who the authors are. Quoting (with slight alteration) from the ACM SIGPLAN review policies document:

A conflict of interest is defined as a situation in which the reviewer can be viewed as being able to benefit personally in the process of reviewing a paper. For example, if a reviewer is considering a paper written by a member of his own group, a current student, his advisor, or a group that he is seen as being in close competition with, then the outcome of the review process can have direct benefit to the reviewer's own status. If a conflict of interest exists, the potential reviewer should decline to review the paper.
As an author, you should list PC members and any others (since others may be asked for external reviews) which you believe have a conflict with you. While particular criteria for making this determination may vary, please apply the following guidelines, identifying a potential reviewer Bob as conflicted if Also please identify institutions with which you are affiliated; all employees or affiliates of these institutions will also be considered conflicted.

If a possible reviewer does not meet the above criteria, please do not identify him/her as conflicted. Doing so could be viewed as an attempt to prevent a qualified, but possibly skeptical reviewer from reviewing your paper. If you nevertheless believe that a reviewer who does not meet the above criteria is conflicted, you may identify the person and send a note to the PC Chair.

Q: What counts as ``explicit lobbying''?

A: The current version of the Principles of POPL states that "Authors should not, however, take [...] as a license to explicitly lobby the PC or likely reviewers on behalf of their work; doing so would be contrary to the intent of DBR.". For example, mass-mailing the PC a de-anonymised version of the paper would be contrary to the intent of DBR. On the other hand, giving a seminar at the institution of a PC member (if you would have done that naturally irrespective of whether the work is submitted to POPL) is fine, as is submitting a related talk proposal to a workshop. (Of course these may mean that some PC members will be able to guess the authors of more papers, making the blinding somewhat less effective. We accept that, to avoid DBR damaging the normal processes of the reearch community.)

For PC members and reviewers

Q: What should I do if I if I learn the authors' identity? What should I do if a prospective POPL author contacts me and asks to visit my institution?

A: If you feel that the authors' actions are principally aimed at ensuring that potential reviewers know their identity, you should contact the Program Chair. Otherwise you should not treat double-blind reviewing differently from regular blind reviewing. In particular, while you should generally refrain from seeking out information on the authors' identity, if you discover it accidentally this will not disqualify you as a reviewer.

Q: Can I deliberately de-blind a submission that I am reviewing?

A: This is left to your discretion: we expect that it will only rarely be required, but if you think it necessary to de-blind a submission in order to write a good review (e.g., if you think you have to see the supplementary material first), you can do so by submitting a placeholder review. We encourage (but do not require) you to write a first-draft review before doing this.

Q: How do we handle potential conflicts of interest since I cannot see the author names?

A: The conference review system will ask that you identify conflicts of interest when you get an account on the submission system. Please see the related question applied to authors to decide how to identify conflicts. Feel free to also identify additional authors whose papers you feel you could not review fairly for reasons other than those given (e.g., strong personal friendship).

Q: How will we identify expert external reviewers? How will the system of "guardians" work?

A: Each PC member must write reviews for all the papers assigned to him/her, but can suggest additional external reviewers to the paper guardians, as below.

We'll try to get at least one external review for each paper and at least two expert reviews for each paper in total. To manage this, each paper will be assigned a "guardian" from among the PC; based on the POPL 2013 numbers, each PC member will be guardian for around 9 papers. We will try to assign a PC member who bid strongly for the paper. The guardian will be responsible (in discussion with the PC chair as necessary) for ensuring that we seek appropriate expert external reviewers for each paper. To help with this:

To get external review requests underway as early as we can, but still avoid conflict problems, guardians may need to de-blind themselves for the papers they are guardianing by submitting a quick first-draft review (or just a placeholder) early on.

Q: If I am assigned a paper for which I feel I am not an expert, how do I seek an outside review?

A: PC members should do their own reviews, not delegate them to someone else, but if you are aware of a good potential reviewer for a particular paper, you should let the designated guardian of that paper know.