Following up on some of the material in Do we need to rethink peer reviewing? - #35 by mjl, I had a couple of questions / comments. And I’ll precede them by saying a big “thank you” to all meta-reviewers for NIME (!!!)
I’ve noticed two instances this year—one as an author, one as a reviewer—where the meta-reviewer has overridden reviews to submit an alternative recommendation. One was kind of egregious… every reviewer recommended accepting a paper and the meta-reviewer initially recommended “strong reject”. In the paper I was a co-author on, all the reviewers recommended some variation of “accept” (one high accept, two accepts, one maybe accept) and the meta-reviewer recommended “lean reject”, and left their own review that was somewhat disconnected from the other reviews. In both cases, the meta-reviewers provided thoughtful / helpful comments and suggestions, but they weren’t “meta” in any way.
So, I guess my first question is… what is the role of the meta-reviewer at NIME? Should they “only” be summarizing submitted reviews and making a recommendation based on them? Or should they also be providing an additional personal review that potentially overrides the opinions of the initial reviewers? Something in the middle? Just wondering what guidance meta-reviewers receive in this regard.
Second, the camera-ready submission instructions state:
"Please note that we expect all authors to address the revisions suggested by the meta-reviewer in the submission of the final camera-ready version.
When submitting your files, please also include a PDF document listing the changes made to your paper. This document should describe how comments and suggestions by the meta-reviewer have been addressed in the new version."
…and in the initial acceptance email:
Conditional acceptance means that the Program Committee will make the final decision on whether to accept the submission, after reviewing the changes that have been made.
Given that many of the recommended changes were unique to the meta-review (and in some case in contradiction to what reviewers stated) is it reasonable to selectively make changes based on the meta-review, as long as we discuss our justifications for making / not making changes in the rebuttal document? Or should we simply make every change the meta-reviewer requests? Finally, I just want to confirm that the Program Committee makes the final decision on conditional acceptance, not the meta-reviewer for the paper.
Thank you, and thanks again to all the reviewers / meta-reviewers who make NIME work!