Submit your Registered Report to Peer Community In Registered Reports for peer review

By Corina Logan and Dieter Lukas

 

In 2018, we were excited to implement the peer review of preregistrations at Peer Community In (PCI). PCI is a community-driven initiative to provide free and transparent assessment of research articles. The ease and clarity of their approach of publishing reviews and recommendations of articles that are deposited elsewhere appeared well-suited to us to implement a flexible approach to the peer review of preregistrations. Our innovation of making preregistrations more flexible at PCI was a huge success and evolved into the launch of PCI Registered Reports in 2021, where we joined forces with spearheaders of registered reports. Registered reports are when you write your introduction and study plan (hypotheses, methods, and analysis plan) before you collect the data or before you analyze the data (if the data are currently being collected or already exist). This registered report then undergoes a pre-study peer review and, if it is scientifically valid and meets the basic PCI RR criteria, it receives In Principle Acceptance (IPA). After the study is complete, it undergoes a second (post-study) peer review to review the results and discussion sections, check that you did what you said you were going to do and, if not, whether the deviations were pre-approved by the recommender and maintained the scientific validity of the research. We’re excited about this way of conducting verifiable research and we’re also always learning so if you have suggestions on how we can improve, please contact us.

Figure 1. A. The traditional way of conducting research where, after payment by readers and/or authors, readers are only able to see the final draft. B. Registered reports as they are often implemented. For both A and B, it is possible to publish the peer reviews alongside the final article at some journals, and it is the author’s choice whether to post a preprint. C. The solutions we’ve implemented with Registered Reports at PCI RR (see Table 1) allow anyone to verify the entire research process (study plan, peer reviews, and all revisions of the preprint, including the final version) and evaluate the quality of the research for themselves, at no cost to authors, institutions, or readers.

 

Why submit a registered report for pre-study peer review? We tend to think of it this way: this piece of research is going to undergo peer review at some point. Why not get it peer reviewed before we collect the data when we can actually change things? Having your study peer reviewed before conducting it can help you avoid several risks, which also saves you loads of time and resources by making the research scientifically valid before you’ve invested in actually conducting the research (Table 1). There is a great overview article on this by Chambers and Tzavella (2022).

 

Table 1. Some of the risks solved by registered reports.

Risk	Peer review of registered reports at PCI RR
HARKing (Hypothesizing After Results are Known)	My hypotheses and predictions are registered so it is clear which ideas were developed after data collection and analysis
P-hacking  (analyzing the data in as many ways possible until a significant p-value is found)	The analysis plan is registered and peer reviewed which means I would need to give valid scientific reasons for changing a method or analysis
Methods can’t answer the research questions	Peer reviewers help me by pointing out potential limitations of my registered methods and suggest alternatives
Have to add post-hoc predictions and hypotheses after analyses	I explore the whole logical space at the pre-study stage by providing alternative predictions
Only significant results are published	My results advance research and are published independent of statistical significance
Unclear exactly what data is needed until after data collection has started	I, my team, the recommender and reviewers, discuss and consider all variables and potential confounds before data collection (usually) and before analyses (always) begin
Fighting over authorship position	Authorship contributions are listed in the registered report and order is agreed upon initially and can also change as some people contribute more or less than expected

 

What’s the difference between registered reports at PCI Registered Reports and registered reports at a journal?

Both undergo pre- and post-study peer review in the same way, but we made registered reports (RRs) at PCI RR more flexible than they currently are at journals (Figure 2). The main feedback we hear from researchers is that they want a more flexible system because things always change along the way and they need to be able to control when data collection can start. So we made that happen at PCI RR (Table 2). 

Figure 2. Comparison of registered reports with non-registered reports at a traditional journal, registered reports at a traditional journal, and registered reports at PCI RR. Figure from PCI RR.

 

Table 2. Common fears researchers have about pre-study peer review and how the solutions PCI RR innovated addresses these fears.

Researcher fears	Peer review of preregistration at PCI
Registered reports are inflexible	I can revise my registered reports! I request permission for the change from the recommender (who so far have not needed to get the changes approved by reviewers, but can just approve it themselves) by saying what changed and why so everyone can see what happened and at what stage (e.g., pre-data collection, post-data collection, pre-analysis, all noted in the commit message at GitHub). The point is to make the process transparent so anyone can see what happened at every stage. As long as the changes result in keeping the research scientifically valid, it should be no problem to incorporate the change into the in principle acceptance
I have to wait until the registered report has passed peer review before collecting data	When you submit your registered report to PCI RR, you can say at the submission page when the data collection is going to start. If you don’t receive IPA before then, your registered report will drop in the Levels of Bias Taxonomy, but will still count as a registered report.
I can’t base a registered report on data that have already been collected	You can submit a registered report that uses secondary data – data that are already being (or were) collected for other hypotheses, but you make new hypotheses and register them, wait for the registered report to pass peer review at PCI RR, and then analyze the data as planned.
One registered report must result in one article	My research is long-term and I want my big ideas to be in one registered report to show how they fit together. As such, one registered report can result in multiple post-study articles and the process at PCI RR accommodates this (termed Programmatic Registered Report).
I can’t submit a reproducible manuscript	You can submit your registered report in as reproducible of a format as you wish (e.g., rmarkdown format). Just make sure there is a version that is easily readable for the recommender and reviewers. PCI RR stores a copy of the version that received IPA at the Open Science Framework repository, so it is clear what was agreed upon. I love writing my registered reports in rmarkdown because I have just one file that undergoes changes from the hypothesis stage to incorporating the final results and discussion – it’s a living document. There is a history that goes with it and anyone can view it at my GitHub repository. It’s also super easy to automatically turn an rmarkdown file into other file formats (e.g., PDF, Word, HTML). I’m not aware of many journals that allow one to submit reproducible manuscripts
I have to write almost the whole article at the registered report (Stage 1)	You don’t have to write the whole article for your Stage 1 registered report. PCI RR requires only an  abstract, introduction, and your hypothesis (if any), methods, and analysis plan. You will write your results and discussion in preparation for your post-study peer review.
What if I don’t like the journal my registered report is at by the time the study is complete?	The publishing landscape changes quickly and I’m not willing to commit to putting a registered report at a particular journal when it could take a few years to conduct the research. Committing to a journal so far in advance means the journal might not suit my strict publishing ethics by the time the article is finished. PCI RR isn’t a journal, it’s a (free and non-commercial) peer review service, therefore the changes it might undergo won’t conflict with my future publishing ethics
Journals will not accept my registered report because they consider it already published	If the final manuscript receives a recommendation at PCI RR, I don’t need to submit it to a journal because I can now point out that it underwent peer review. For example, cOAlition S organizations consider these manuscripts equivalent to those appearing in traditional journals. Although, lots of journals now automatically accept PCI RR’s recommended registered reports (see PCI RR-friendly journals).
Registered reports create extra work for me and for reviewers	Quite the opposite! From my perspective, I am simply frontloading part of the writing, and avoiding extra stress and hassle because I already know what I need to do to get the final recommendation. From the reviewer’s perspective, PCI RR serves as a one-stop place for this assessment. After a manuscript has been recommended, I as an author can choose if I want to have my manuscript automatically published in one of the many PCI RR-friendly journals. So I don’t have to wait and see whether a journal might make a subjective judgment about my manuscript.
I do exploratory research – what is there to include in a registered report?	Some of the research I do is exploratory and one thing I’ve noticed is that I’ve never gone exploring without a reason. All you have to do is write this reason down and that can be the basis for your registered report. If you also know something about the types of variables you want to look at or if you just have an idea of what they are, then you can add those too. PCI RR also offers registered reports for qualitative researchers – our authors, recommenders, and reviewers wrote a “How to” for this process here.

 

Learning how to submit my registered reports as version-controlled reproducible manuscripts to PCI

We’ve submitted several preregistrations to PCI Ecology and one registered report to PCI RR for pre-study peer review and, with the help of PCI founders, recommenders, reviewers, our research teams, and other researchers, we worked out a way to make these submissions easy for the recommenders and reviewers to read (using HTML files), while maintaining a verifiable research process at GitHub via our rmarkdown (Rmd) files. The process we’ve developed is just one of many ways it could work – other ways would need to be developed according to the needs of the authors – so we’ll share with you in detail how we do this so you can use as much or as little of this process as you like (note that you can just submit a Word or pdf file, as at journals, if you prefer). 

 

We write our manuscripts in rmarkdown, a free-to-use open source format, where we combine the manuscript text with the code for the analyses all in one document (saving tons of time in looking around for various bits of code in random folders with random file names). We then put the Rmd file at GitHub, which has the benefit of being fully version tracked with time and date stamps and track changes (all of this happens automatically and multiple people can edit at the same time). (Note: we connect RStudio with the GitHub repository using GitHub Desktop, which makes for easy synching.)

 

Table 3. We list a few of the main reasons why we developed this process of submitting reproducible manuscripts as registered reports to PCI RR.

 

 

 

 

 

Problem	Solution	Example / Code
Feedback from recommenders and reviewers indicated that it was not convenient to review .Rmd files at GitHub (in 2018 GitHub changed the appearance of these files such that you have to scroll very far to the right to read a whole paragraph on just one line)	I (Corina) use RStudio to export the Rmd files to HTML and upload the HTML files to my free website at GitHub Pages. This way reviewers can read the easy-to-read HTML files	Example HTML and Rmd files
Recommenders and reviewers were frustrated that they had to jump around to the various pieces of the registered report (e.g., scroll to Hypothesis 1 in the prediction section, the methods section, and the analysis section)	I remedied this by making a floating table of contents for the HTML file (this option is only available for the HTML version), which makes it easy to jump between sections	Example floating table of contents. Rmd CODE for the floating table of contents goes in the Header: Output:   html_document:     toc: true     toc_depth: 4     toc_float:       collapsed: false       code_folding: hide
Feedback from recommenders and reviewers indicated that having the R code visible was distracting	I added code folding to the HTML file, which makes the default to hide the R code, but if someone wants to see it, they can just click the “code” button and it will appear (and they can re-hide it again by clicking the same button, now called “hide”)	Example here Rmd CODE: see last line of code in the previous row: code_folding: hide
Recommenders and reviewers felt like there were too many links to other documents throughout the registered report (links to other registered reports, the protocols, and a separate figure)	The floating table of contents helps with this, as does including all figures as part of the HTML file (because they actually show up in this file, whereas they don’t in the Rmd file at GitHub). I also now have a Protocols or Open Materials section (listed in the floating table of contents) where I provide links to protocols all in one place	See an example

 

How to submit a registered report to PCI RR for peer review

Below, we share the nitty gritty details for submitting a version-tracked reproducible manuscript to PCI RR. Lots of these details should still be useful if you have chosen a different route for your registered report (e.g., by placing it at OSF). Before submitting, make sure the Rmd file has all of the most recent changes and is the version you want to submit. On your last commit for this file, in the GitHub commit note write “SUBMITTED TO PCI RR FOR PRE-STUDY PEER REVIEW”. 

1. Prepare your registered report according to the PCI RR Author Guidelines
2. Go to the PCI RR website, click “SUBMIT A REPORT”, read the instructions there, then click “SUBMIT YOUR REPORT”
3. Submission page 1: Fill in the details about your submission
   • If you want a double-blind peer review, check the box “I wish an anonymous submission”
   • Title: [insert title]
   • Authors: [how you want them to appear in the citation]
   • Full web address to most recent version of the manuscript: http://corinalogan.com/ManyIndividuals/mi1.html [replace html link with the correct link for your registered report]
   • Version: 2022-09-08 [the date on the html file is what distinguishes the versions that we submit]
   • Version of Record URL: https://github.com/ManyIndividuals/ManyIndividuals/blob/573b4b5802550f47f246fb8b71c8efc4f853445c/Files/rrs/mi1.Rmd [e.g., refer to the unique GitHub URL for the version you are submitting (go to the Rmd file, click History, scroll to the appropriate commit, then click on the icon for “View at this point in the history” of piece of paper with “<>” on it. It brings you to the Rmd file at this point in the history and then you can copy the URL from the URL bar]
   • Version of Record Unique Identifier: 573b4b5802550f47f246fb8b71c8efc4f853445c [e.g., refer to the unique GitHub ID for the version you are submitting (go to the Rmd file, click History, scroll to the appropriate commit, then click on the icon of the clipboard with the arrow pointing to the left and it automatically copies the ID for you]
   • Check the box “I wish to add a small picture”. Choose a professional looking picture
   • Picture: click “Choose file” to upload the picture you chose
   • Cover letter: [edit as needed, making sure to replace the Rmd and HTML URLs with the correct links, attribute photo credit to the person who took the photo, and state the planned data collection start date and that it would be ideal if we could get through the peer review process before then]
     Dear PCI RR Managing Board and Recommender,
     We thank you for the opportunity to submit our registered report Our version-tracked version of this registered report is available at [insert Rmd URL here]. Photo credit goes to Corina Logan (CC-BY 4.0). This research is part of a senior thesis at Arizona State University and we plan to begin data analyses in late September 2019. Therefore, we would greatly appreciate if it would be possible to complete the review process before then. Please let us know if you have any questions or need further information. Many thanks for your attention!
     All our best,
     [add co-author names]
   • Check the box “I am an author of the article and I am acting on behalf of all the authors”, but first make sure that all authors have seen the most recent version and that they are happy for you to submit it
   • Check the box “This report has not been published…”
   • Click “CONTINUE YOUR SUBMISSION”
4. Submission page 2: answer the questions about what type of registered report you are submitting (Stage 1 or 2, scheduled or regular review track, programmatic or regular, require all reviewers to sign or not, level of bias, planned data collection start date, etc.)
   • You can choose to suggest reviewers, but make sure you don’t have any conflicts of interest with them. You can also state which reviewers you oppose and the recommender will not invite them as reviewers.

1. Next, you will have the opportunity to suggest recommenders (editors) to handle your submission. You can also select recommenders who you do not want to handle your report, either because you have a conflict of interest with them or for any other reason.

• To figure out who you want to suggest, go to PCI RR > About > Recommenders and search by keywords that you type in and/or by thematic field (note: “Toggle thematic fields” unchecks all boxes). After you click “Search”, scroll down to see the results.
• Click on people’s names to see their profiles and get more information about whether they would be appropriate. If this information isn’t in their profile, search for their name on Google Scholar and read more about their work at their websites.
• Suggest at least 5 recommenders, but suggesting 10 is much better (consider balancing gender and whether they are based in a country that is over- or under-represented in your field).
• NOTE: if the search function at the PCI RR website stops letting you search for more recommenders, just click the DONE button and it will take you back to an area where you can navigate to search for more recommenders.

Resubmitting revisions to PCI RR at the pre-study peer review stage

When it comes time to RESUBMIT your work to PCI RR after it has been peer reviewed, draft the response at Google Docs, and include a letter to the recommender and reviewers at the top. In the letter, make sure it includes a link to the HTML version (the reviewers like to read the HTML versions) as well as a link to the version-tracked version of the document (e.g., the .Rmd file at GitHub). Here is some example text:

“Dear [insert Recommender and reviewer names],

We greatly appreciate the time you have taken to give us such useful feedback! 

Note that the version-tracked version of this registered report is in rmarkdown at GitHub: https://github.com/ManyIndividuals/ManyIndividuals/blob/573b4b5802550f47f246fb8b71c8efc4f853445c/Files/rrs/mi1.Rmd. In case you want to see the history of track changes for this document at GitHub, click the previous link and then click the “History” button on the right near the top. From there, you can scroll through our comments on what was changed for each save event and if you want to see exactly what was changed, click on the text that describes the change and it will show you the text that was replaced (in red) next to the new text (in green).

We think the revised version is much improved due to your generous feedback!

All our best,

[Insert author names]“

 

Tips for writing the response in a way that makes it easier for the recommender and reviewers:

• Number the comments and your responses so you can easily cross reference and so readers can orient themselves in the document (see example).
• Include quotations of the text you changed directly in the rebuttal document so readers don’t have to read the response and the manuscript, they can just read the response document to see what changed (see example).
• You can upload your response as a PDF file if you wish. Or you can copy and paste the response directly into the text box at PCI RR
• If you ended up changing the title of your registered report, remember to update it at the PCI RR website in response to the reviewer comments before you submit your response.

Congratulations! You received In Principle Acceptance at PCI RR! Now what?

Now you can format your registered report so it shows the PCI RR badge, which you can link to the Recommendation at PCI RR, and you can add the citation for the registered report, and the citation and link for the PCI RR Recommendation (see an example). The Rmd code for how to make all of this happen is in the example, but I also list it here:

<img width=”50%” src=”logoPCIRR.png”> #Logo available at https://peercommunityin.org/current-pcis/ 

Cite as: Logan CJ, Shaw R, Lukas D, McCune KB. 2022. How to succeed in human modified environments (http://corinalogan.com/ManyIndividuals/mi1.html) In principle acceptance by PCI Registered Reports of the version on 25 Aug 2022 https://github.com/ManyIndividuals/ManyIndividuals/blob/573b4b5802550f47f246fb8b71c8efc4f853445c/Files/rrs/mi1.Rmd.

<img width=”5%” src=”logoOpenAccess.png”> <img width=”5%” src=”logoOpenCode.png”> <img width=”5%” src=”logoOpenPeerReview.png”>

This registered report has been pre-study peer reviewed and received an In Principle Acceptance on 8 Sep 2022 by:

Chris Chambers (2022) The role of behavioural flexibility in promoting resilience to human environmental impacts. Peer Community in Registered Reports. https://rr.peercommunityin.org/articles/rec?id=200. Reviewers: Gloriana Chaverri, Vedrana Šlipogor, and Alizée Vernouillet

 

How to cite registered reports that have received an in principle acceptance

The Max Planck Society counts my registered reports as research outputs and these are included in our departmental evaluations. I list these as “in press” because this is the most analogous traditional term. They just spend more time in press than an article would that was submitted after the work was finished. I cite my registered reports that are in review and those that have passed pre-study peer review on my CV and in applications. At this stage, the Max Planck Institute for Evolutionary Anthropology librarian uploads it to their repository and lists them as “in press” and then it is automatically listed at Google Scholar.

Conducting your study

Make sure you get pre-approval from your recommender for any deviations from the registered report’s methods and analyses, and that you keep track of and justify them to ensure the research remains scientifically valid. We find it handy to do this in a section called “State of the Data”, which is at the top of each preregistration (see an example). We update the same Rmd file that we originally submitted for pre-study peer review so we never have to go looking for other versions of this document.

Before resubmitting to PCI RR for the post-study peer review, deposit your data in a repository. We like the Knowledge Network for Biocomplexity’s data repository because it is free and has great metadata requirements that makes your data findable and useable.

Perhaps at this point you might be wondering whether it would be better to have this one registered report be more than one finished article. Maybe you want each hypothesis to be its own article as a stand alone piece. In this case, you can write multiple post-study articles based on the one registered report. You can keep the pieces of the introduction and methods that are relevant to that particular article and delete the other pieces. Your registered report will move from having an in principle acceptance to a full acceptance only after all pieces of the registered report have passed post-study peer review.

Because the research is now complete, it is eligible to be a preprint and you could opt for putting a PDF copy of the Rmd file at a preprint server rather than having the HTML version as the easy-to-read option. We prefer the EcoEvoRxiv preprint server for our field of research because it allows you to add a link to the DOI of the journal version (if we put our article in a journal) and they are working with PCI RR to also include a link to the recommendation to show that it is peer reviewed.

Resubmitting to PCI RR: Post-study peer review stage

You finished your study and analyzed the results and revised your registered report into its final draft! In addition to the clean Stage 2 manuscript, make sure you have a separate document where ALL changes from since the approved Stage 1 are tracked because you will need to include the link to the tracked changes version when you submit your Stage 2 to PCI RR.

Remember to integrate the deviations that inevitably occurred during your study into the manuscript. We like to keep this in a separate section. Several reviewers suggested that the best format for the final method section is one that lists specifically what you did to obtain the results you are presenting. So anything that was originally planned but dropped could go in the deviation section.

Now you resubmit it to PCI RR as a new submission, choosing in the drop down menus at the submission pages that it is a Stage 2. Suggest the same recommender that handled your submission before because they will be familiar with the work and can more easily assess any changes that need to be addressed. In the cover letter, highlight where in the submission you explain any pre-approved deviations from your registered report.

Resources

• Initial Evidence of Research Quality of Registered Reports Compared to the Traditional Publishing Model
• The past, present, and future of Registered Reports
• An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
• Open science challenges, benefits and tips in early career and beyond
• A primer for choosing, designing and evaluating registered reports for qualitative methods
• We regularly post tips for making the process more efficient at the PCI RR Mastodon account: @pcirr@spore.social

 

Originally published 15 December 2020, updated 26 April 2023