[4eyes] FW: CVPR 2017 : Area Chair Reviewing Guidelines

[Matthew Turk]

I thought it would be informative to share with the lab some of the information for CVPR Area Chairs and reviewers, to provide a sense of how the CVPR community treats the peer review process and some particular relevant issues (like anonymity and arXiv papers). These may change quite a bit over time, so consider this to be a snapshot from 2017.

The email below is for AC, but also see the link to reviewer guidelines: http://cvpr2017.thecvf.com/submission/main_conference/reviewer_guidelines

	Matthew

-----Original Message-----

Thank you again for agreeing to serve as Area Chairs for CVPR 2017. You have the awesome responsibility of enforcing the peer-review standards of the CVPR community, and in doing so you literally define the character of CVPR as a meeting. We are grateful for your service.

Please familiarize yourself with the guidelines we provided to your reviewers: http://cvpr2017.thecvf.com/submission/main_conference/reviewer_guidelines

In the following, we summarize the main issues with respect to anonymity and arXiv papers (excerpted from the guidelines), and provide some additional thoughts on novelty and conformity for your consideration.

Anonymity

•	The PAMI-TC arXiv policy makes it trivial for authors to reveal their identity to reviewers. Therefore, the CVPR double-blind policy is effectively optional; authors can reveal their identities with no penalty if they choose to do so. 
•	Considering this, it seems unnecessarily harsh to reject papers on the basis of a violation of double-blind policy. We encourage you to show leniency to authors who accidentally reveal their identity by citing their previous work inappropriately, publishing URLs in their paper, and so forth.
•	The presence of leaderboards and open competitions provides additional ways to reveal identity. Since performance in such competitions is an important piece of evidence in the review of a work, we strongly recommend that you not consider such revelations to be a  violation of policy.

arXiv

As voted on at the CVPR 2015 PAMI-TC meeting, arXiv papers are not considered prior work since they have not been peer reviewed. This has several consequences which are spelled out in the reviewer guidelines excerpted below. You are advised to overturn reviews which violate these guidelines:
•	It is Not OK for a reviewer to suggest rejection for not citing an arXiv paper or not being better than something on arXiv.
•	It is Not OK to accept a paper solely because it performs better than something on arXiv.
•	It is Not OK to reject a paper solely because it performs worse than something on arXiv.
•	It is Not OK to regard arXiv as a standard for the state of the art, because it is not reviewed. This applies *whoever* wrote the arXiv paper.
•	It is Not OK for a reviewer to reject a paper solely because another paper with a similar idea has already appeared on arXiv. If the reviewer is worried about plagiarism they should bring this up in confidential comments to the AC.
•	It is OK for a reviewer to suggest an author should acknowledge and be aware of something on arXiv.
•	It is OK for an author to decline to acknowledge something on arXiv (because it has not been reviewed and so may not be right).

Novelty and Conformity

Our community has achieved significant advances in performance in many problem domains by embracing data-driven methods, as exemplified by deep learning. At the same time, it is important for the long-term health of our community that we remain open to new ideas and reject any petty notions of conformity. We offer the following guidelines for your consideration:
•	Embrace papers that propose novel, brave concepts or approaches, even if they have not been tested on a large number of datasets. The extent to which a lack of evaluation is a weakness should be weighed against the innovativeness of the approach, as an excessive focus on evaluation can stifle the generation of new ideas.
•	Be open to papers that advance our theoretical understanding of the field or present highly-novel approaches, even if the performance of the new method lags the current state-of-the-art. An excessive focus on performance can prevent new ideas from being introduced quickly and can limit the breadth of our scientific dialog.
•	Paper rejection/acceptance decisions should not hinge on whether deep models or other popular methods are employed in a particular work, unless it is central to the question being asked. The scientific goals of a work, not conformity to an existing trend, should dictate whether an analysis technique must be used.