Meta-reviewing / provisional accepts

1

Hi everyone,

Following up on some of the material in Do we need to rethink peer reviewing? - #35 by mjl, I had a couple of questions / comments. And I’ll precede them by saying a big “thank you” to all meta-reviewers for NIME (!!!)

I’ve noticed two instances this year—one as an author, one as a reviewer—where the meta-reviewer has overridden reviews to submit an alternative recommendation. One was kind of egregious… every reviewer recommended accepting a paper and the meta-reviewer initially recommended “strong reject”. In the paper I was a co-author on, all the reviewers recommended some variation of “accept” (one high accept, two accepts, one maybe accept) and the meta-reviewer recommended “lean reject”, and left their own review that was somewhat disconnected from the other reviews. In both cases, the meta-reviewers provided thoughtful / helpful comments and suggestions, but they weren’t “meta” in any way.

So, I guess my first question is… what is the role of the meta-reviewer at NIME? Should they “only” be summarizing submitted reviews and making a recommendation based on them? Or should they also be providing an additional personal review that potentially overrides the opinions of the initial reviewers? Something in the middle? Just wondering what guidance meta-reviewers receive in this regard.

Second, the camera-ready submission instructions state:

"Please note that we expect all authors to address the revisions suggested by the meta-reviewer in the submission of the final camera-ready version.

When submitting your files, please also include a PDF document listing the changes made to your paper. This document should describe how comments and suggestions by the meta-reviewer have been addressed in the new version."

…and in the initial acceptance email:

Conditional acceptance means that the Program Committee will make the final decision on whether to accept the submission, after reviewing the changes that have been made.

Given that many of the recommended changes were unique to the meta-review (and in some case in contradiction to what reviewers stated) is it reasonable to selectively make changes based on the meta-review, as long as we discuss our justifications for making / not making changes in the rebuttal document? Or should we simply make every change the meta-reviewer requests? Finally, I just want to confirm that the Program Committee makes the final decision on conditional acceptance, not the meta-reviewer for the paper.

Thank you, and thanks again to all the reviewers / meta-reviewers who make NIME work!

2 Likes
2

For anyone else wondering about similar questions on revisions, I received a helpful email from the program chairs saying to follow the recommendations of the meta-reviewer as closely as possible. Thanks for their quick response!

1 Like
3

Well this is all quite concerning.

My understanding was that meta’s would mainly summarise the submitted reviews and sometimes provide independent opinion, but primarily as a way to resolve issues with split reviews. Regarding guidance, not that much is given, definitely not written down permanently.

In some instances I as a meta have provided more review but that’s generally where the reviews are unacceptably short or something. Good practice would be to separate "meta review’ comments from “extra review”.

1 Like
4

Hi all,

Sorry for replying so late on that.
To comment on my personal experience, as a meta-reviewer went “against” and expanded the reviews, and I am probably the one who meta-reviewed your paper @charlieroberts .
Usually as a meta-reviewer, I generally tend to summarize the reviewers comments in instructions for the authors but for this NIME we received this email as instructions :
“Your meta-reviews should summarise and reflect on the comments from the individual reviewers, and include further comments based on your own assessment of the paper. The meta-review score doesn’t have to be a simple average of the reviewers’ scores, but if your judgment differs significantly from a consensus of reviewers then please explain why in your meta-review.”

Considering that in this case it was an interesting paper and that I saw potential improvements and that the reviews were very short, I felt it was important to add (many) comments and instructions. The recommendation might have been a bit strong though and it could have followed the reviewers’ scores more closely … I hope the comments were useful for the camera ready though.

On a more general notice I agree that the role of the meta-reviewer should be clarified for future editions of the conference, either as only a summary of reviewers or as an explicit reviewer + meta-reviewer role.

Best regards,

Florent

2 Likes
5

Hi Florent,

Yes, your comments were helpful and definitely improved the paper (which of course is the ultimate goal of all of this), so thank you!

The other paper that I was a reviewer on was also accepted, so all is well that ends well, or at least from my point of view :slight_smile: I do think a publicized clarification of the role of meta-reviewing would help the meta-reviewers as well as aid the authors in crafting their responses to meta-reviews.

Thanks again to everyone involved in the review process!

1 Like