Questionnaire for reviewers

Below we list a series of questions to help you conduct your review. You do not have to answer these questions. However, if you use them as a guide for your review report, they will help us to develop a review process that is comprehensive and at least partly consistent between reviewers.
We tested this questionnaire with a group of 10 reviewers. They all told us that it was a great tool to produce a complete and structured review report.

General questions

• Did you read the “guide for reviewers”? (see the Help menu of the thematic PCI or the dedicated blog post)
• Is the manuscript well written?
• Is the description of the rationale and methods clear and comprehensive?
• Are there flaws in the design of the research?
• Are there flaws in the analysis?
• Are there flaws in the interpretation of results?
• Do you have concerns about ethics or scientific misconduct?
• Did you detect a spin on the results, discussion or abstract? (a spin is a way of twisting the reporting of results such that the true nature and range of the findings are not faithfully represented, https://doi.org/10.1073/pnas.1710755115)
• Is something critical missing?

Evaluation of the various components of the article

Title/abstract/introduction

• Does the title clearly reflect the content of the article?
• Does the abstract present the supported findings of the study concerned and no other?
• Does the introduction clearly explain the motivation for the study?
• Is the research question/hypothesis/prediction clearly presented?
• Does the introduction build on relevant recent and past research performed in the field?

Materials and Methods

• Are the methods and analysis described in sufficient detail to allow replication by other researchers?
• Is the experimental plan consistent with the questions?
• Are the statistical analyses appropriate?
• Have you evaluated the statistical scripts and program codes?

Results

• Have you checked the raw data and their associated description?
• Have you run the data transformations and statistical analyses and checked that you get the same results?
• To the best of your ability, can you detect any obvious manipulation of data (e.g. removal)?
• Do the statistical results strongly support the conclusion (p< 10-3 or BF>20)?
• In the case of negative results, was a statistical power analysis (or an appropriate Bayesian analysis) performed?
• Did the authors conduct many experiments but retain only some of the results?

Discussion

• Do the interpretations of the analysis go too far?
• Are the conclusions adequately supported by the results?
• Does the discussion take into account relevant recent and past research performed in the field?
• Did the authors test many hypotheses but consider only a few in the discussion?

References

• Are all the references appropriate?
• Are the necessary references present?
• Do the references seem accurate?

Tables and figures

• Are the tables and figures clear and comprehensive?
• Are all the tables/figures useful?
• Are there too many/too few tables and figures?
• Do the tables and figures have suitable captions such that they can be understood without having to read the main text?

Peer review of the revised version of a preprint

• Did the authors correctly address all your comments?
• Do you disagree with the other reviewer(s) and agree with the authors’ explanation and defense of their original article?