Submit your preregistration to Peer Community In for peer review

In 2018, we were excited to implement the peer review of preregistrations at Peer Community In (PCI). PCI is a community-driven initiative to provide free and transparent assessment of research articles. The ease and clarity of their approach of publishing reviews and recommendations of articles that are deposited elsewhere appeared well-suited to us to implement a flexible approach to the peer review of preregistrations. Preregistrations are when you write your study plan (hypotheses, methods, and analysis plan) before you collect the data. The new feature we offer at PCI is the ability to submit your preregistrations for pre-study peer review. After the study is complete, it undergoes a second (post-study) peer review to double check that you did what you said you were going to do and, if not, whether the deviations maintained the scientific validity of the research. We’re excited about this way of conducting verifiable research and we’re also always learning so if you have suggestions on how we can improve, please contact us.

Figure 1. A. The traditional way of conducting research where, after payment by readers and/or authors, readers are only able to see the final draft. B. Registered reports as they are often implemented. For both A and B, it is possible to publish the peer reviews alongside the final article at some journals, and it is the author’s choice whether to post a preprint. C. The solutions we’ve implemented with preregistrations at PCI (see Table 1) allow anyone to verify the entire research process (study plan, peer reviews, and all revisions of the preprint, including the final version) and evaluate the quality of the research for themselves, at no cost to authors or readers.

Why submit a preregistration for pre-study peer review? We tend to think of it this way: this piece of research is going to undergo peer review at some point. Why not get it peer reviewed before we collect the data when we can actually change things? Having your study pre-study peer reviewed can help you avoid several risks, which also saves you loads of time and resources by making the research scientifically valid before you’ve invested in actually conducting the research (Table 1). There is a great overview article on all of this by Nosek and colleagues (2018).

Table 1. Some of the risks solved by the track for the peer review of preregistrations at PCI.

Risk	Peer review of preregistration at PCI
HARKing (Hypothesizing After Results are Known)	My hypotheses and predictions are pre-registered so it is clear which ideas were developed after data collection and analysis
P-hacking (analyzing the data in as many ways as possible until a significant p-value is found)	The analysis plan is preregistered and peer reviewed which means I would need to give valid scientific reasons for changing a method or analysis
Methods can’t answer the research questions	Peer reviewers help me by pointing out potential limitations of my preregistered methods and suggest alternatives
Have to add post-hoc predictions and hypotheses after analyses	I explore the whole logical space at the preregistration stage by providing alternative predictions
Only significant results are published	My results advance research independent of significance
Unclear exactly what data is needed until after data collection has started	I, my team, and the reviewers, discuss and consider all variables and potential confounds before data collection (usually) and before analyses (always) begin
Fighting over authorship position	Authorship contributions are listed in the preregistration and order is agreed upon initially and can also change as some people contribute more or less than expected

What’s the difference between the peer-review of preregistrations at PCI and Registered Reports at a journal?

Both undergo pre- and post-study peer review in the same way, but we made preregistrations at PCI more flexible than Registered Reports (RRs) currently are. The main thing we hear from researchers is that they want a more flexible system because things always change along the way and they need to be able to control when data collection can start. So we made that happen at PCI.

Table 2. Common fears researchers have about pre-study peer review and how the solutions at PCI’s track for the peer review of preregistrations addresses these fears.

Researcher fears	Peer review of preregistration at PCI
Preregistrations are inflexible	I revise my preregistrations all the time! I say what changed and why so everyone can see what happened and at what stage (e.g., pre-data collection, post-data collection, pre-analysis, all noted in the commit message at GitHub). The point is to make the process transparent so anyone can see what happened at every stage. As long as the changes are result in keeping the research scientifically valid, it should be no problem to pass the post-study peer review
I have to wait until the preregistration has passed peer review before collecting data	When you submit your preregistration to PCI, you can say in the cover letter when the data collection is going to start and ask if it would be possible to finish the peer review before then. At this point, they have a heads up and if data collection starts when it is in the middle of peer review, it won’t be a surprise to anyone. PCI Ecology has a pretty quick turn around (1-4 months) so I have at least received the first round of reviews and revised before collecting the data for most of the preregistrations I have submitted.
I can’t base a preregistration on data that have already been collected	You can submit a preregistration that uses secondary data – data that are already being (or were) collected for other hypotheses, but you make new hypotheses and preregister them, wait for the preregistration to pass peer review at PCI, and then analyze the data.
One preregistration must result in one article	My research is long-term and I want to preregister my big ideas and how they fit together. As such, one preregistration might result in multiple post-study articles and the process at PCI accommodates this.
I can’t submit a reproducible manuscript	You can submit your preregistration in as reproducible of a format as you wish (e.g., rmarkdown format). Just make sure there is a version that is easily readable for the Recommender and reviewers. I love writing my preregistrations in rmarkdown because I have just one file that undergoes changes from the hypothesis stage to incorporating the final results and discussion – it’s a living document. There is a history that goes with it and anyone can view it at my GitHub repository. It’s also super easy to automatically turn an rmarkdown file into other file formats (e.g., PDF, Word, etc.). I’m not aware of many journals that allow one to submit reproducible manuscripts
I have to write almost the whole article at the preregistration stage	You don’t have to write the whole article for your preregistration. PCI requires only an outline: write an abstract, list your hypothesis, methods, and analysis plan. You can write your intro and flesh out all of the details later when you’re preparing it for post-study peer review. Giving these details in the outline is enough for researchers who are not associated with my project to understand what I’m testing and why
What if I don’t like the journal my preregistration is at by the time the study is complete?	The publishing landscape changes quickly and I’m not willing to commit to putting a Registered Report at a particular journal when it could take a few years to conduct the research. Committing to a journal so far in advance means the journal might not suit my strict publishing ethics by the time the article is finished. PCI isn’t a journal, it’s a (free) peer review service, therefore the changes it might undergo won’t conflict with my future publishing ethics
I do exploratory research – what is there to preregister?	Some of the research I do is exploratory (see an example) and one thing I’ve noticed is that I’ve never gone exploring without a reason. All you have to do is just write this reason down and that can be your preregistration. If you also know something about the types of variables you want to look at or if you just have an idea of what they are, then you can add those too.

What does your preregistration need to be submittable to PCI? It needs to be publicly available online (just like the preprints that PCI reviews), version controlled (so there is a time and date stamp whenever you update it because the reviewers and Recommender will have to track the changes over the course of the study), and have a DOI (digital object identifier). One way of doing this is to upload your file to OSF as a preregistration or to your institutional repository, making it publicly available, where it will get a DOI that you can submit to PCI. OSF is version tracked in the sense that it records when the files were re-uploaded, but not what changes were made. You’ll need to check with your institutional repository to see whether they do version tracking. We wanted a better version tracking system for our preregistrations (which also accommodates multiple authors making changes at the same time), so we opted for a different route: version-controlled reproducible manuscripts.

Learning how to submit my preregistrations as version-controlled reproducible manuscripts to PCI

We’ve now submitted 10 preregistrations to PCI Ecology for pre-study peer review and, with the help of PCI founders, Recommenders, reviewers, my research team, and other researchers, we worked out a way to make these submissions easy for the Recommenders and reviewers to read (using HTML files), while maintaining a verifiable research process at GitHub via our rmarkdown (Rmd) files. The process we’ve developed is just one of many ways it could work – other ways would need to be developed according to the needs of the authors – so we’ll share with you in detail how we do this so you can use as much or as little of this process as you like.

We write our manuscripts in rmarkdown, a free-to-use open source format, where we combine the manuscript text with the code for the analyses all in one document (saving tons of time in looking around for various bits of code in random folders with random file names). We then put the Rmd file at GitHub, which has the benefit of being fully version tracked with time and date stamps and track changes (all of this happens automatically and multiple people can edit at the same time). (Note: we connect RStudio with the GitHub repository using GitHub Desktop, which makes for easy synching.)

Table 3. We list a few of the main reasons why we developed this process of submitting reproducible manuscripts as preregistrations to PCI.

Problem	Solution	Example / Code
Feedback from Recommenders and reviewers indicated that it was not convenient to review .Rmd files at GitHub (in 2018 GitHub changed the appearance of these files such that you have to scroll very far to the right to read a whole paragraph on just one line)	I use RStudio to export the Rmd files to HTML and upload the HTML files to my free website at GitHub Pages. This way reviewers can read the easy-to-read HTML files	Example HTML and Rmd files
Recommenders and reviewers were frustrated that they had to jump around to the various pieces of the preregistration (e.g., scroll to Hypothesis 1 in the prediction section, the methods section, and the analysis section)	I remedied this by making a floating table of contents for the HTML file (this option is only available for the HTML version), which makes it super easy to jump between sections	Example floating table of contents. Rmd CODE: for the floating table of contents goes in the Header: Output: html_document: toc: true toc_depth: 4 toc_float: collapsed: false code_folding: hide
Feedback from Recommenders and reviewers indicated that having the R code visible was distracting	I added code folding to the HTML file, which makes the default to hide the R code, but if someone wants to see it, they can just click the “code” button and it will appear (and they can re-hide it again by clicking the same button, now called “hide”)	Example here Rmd CODE: see last line of code in the previous row: code_folding: hide
The DOI box at the PCI submission website was tricky because I keep all of my preregistrations in one GitHub repository, but I can’t get a DOI for each file in the repo, I can only get a DOI for the whole repo (and the easiest way to do this is to connect it with an OSF component). So entering your OSF DOI in this box will bring someone to your entire GitHub repository and then Recommenders and reviewers don’t know which file they are supposed to review	I now enter the HTML link in the DOI box (e.g., http://corinalogan.com/Preregistrations/g_flexmanip.html). This still meets PCI’s need to have version tracking on the file (that’s the point of the DOI) because in the HTML file and in the cover letter at the submission page, I list the version-tracked version (Rmd) of the file at GitHub	Rmd CODE: ensure the reader can navigate to the Rmd file: ***Click [here](https://github.com/corinalogan/grackles/blob/master/README.md) to navigate to the version-tracked reproducible manuscript (.Rmd file)***
Recommenders and reviewers felt like there were too many links to other documents throughout the preregistration (links to other preregistrations, the protocols, and a separate figure)	The floating table of contents really helps with this, as does including all figures as part of the HTML file (because they actually show up in this file, whereas they don’t in the Rmd file at GitHub). I also now have an Open Materials section (listed in the floating table of contents) where I provide links to protocols all in one place	See an example

How to submit a preregistration to PCI for peer review

Below, we share the nitty gritty details for submitting a version-tracked reproducible manuscript to PCI. But lots of these details should still be useful if you have chosen a different route for your preregistration (e.g., by placing it at OSF). Before submitting, make sure the Rmd file has all of the most recent changes and is the version you want to submit. On your last commit for this file, in the GitHub commit note write “SUBMITTED TO PCI ECOLOGY FOR PRE-STUDY PEER REVIEW”.

1. Go to the PCI Ecology website, click “SUBMIT A PREPRINT”, and read the instructions there, including clicking on the extra instructions for preregistrations at the bottom, then click “SUBMIT YOUR PREPRINT”
2. Fill in the details about your submission
   • If you want a double-blind peer review, check the box “I wish an anonymous submission”
   • Title: [insert title]
   • Authors: [how you want them to appear in the citation]
   • DOI (or URL): http://corinalogan.com/Preregistrations/gdispersal.html [replace html link with the correct link for your preregistration. You can ignore the fact that this isn’t a DOI because the reviewers just want to know which file they need to review]
   • Version: [e.g., refer to the unique GitHub ID for the version you are submitting (go to the Rmd file, click History, scroll to the appropriate commit, then click on the icon of the clipboard with the arrow pointing to the left and it automatically copies the ID for you, which will look something like this: 2e0d4b74dcf49eadefcfa4711ad52cc05af824e8 ]
   • Check the box “I wish to add a small picture”. Choose a professional looking picture
   • Picture: click “Choose file” to upload the picture you chose
   • Abstract: Before the text of your abstract, you might want to alert the managing board and potential editors that this is a preregistration by adding “This is a PREREGISTRATION submitted for pre-study peer review. Our planned data collection START DATE is [insert month year], therefore it would be ideal if the peer review process could be completed before then.”
     [insert abstract]
   • Thematic fields: always check Preregistration field. Then check other fields that can additionally apply to your research
   • Keywords: birds, great-tailed grackle, [etc.]
   • Cover letter: [edit as needed, making sure to replace the Rmd and HTML URLs with the correct links, attribute photo credit to the person who took the photo, and state the planned data collection start date and that it would be ideal if we could get through the peer review process before then]Dear PCI Managing Board and Recommender,
     We thank you for the opportunity to submit a PREREGISTRATION for pre-study peer review (for more information about preregistrations, please see this handy article: https://www.pnas.org/content/115/11/2600 and also the instructions at PCI Ecology: https://ecology.peercommunityin.org/about/help_practical#To%20submit%20a%20preregistration%20of%20a%20study). Our version-tracked version of this preregistration is available at [insert Rmd URL here and add \ so it works in markdown – see note below]. Photo credit goes to Corina Logan (CC-BY 4.0). This research is part of a senior thesis at Arizona State University and we plan to begin data analyses in late September 2019. Therefore, we would greatly appreciate if it would be possible to complete the review process before then. Please let us know if you have any questions or need further information. Many thanks for your attention!
     All our best,
     [add co-author names]
   • Check the box “I am an author of the article and I am acting on behalf of all the authors”, but first make sure that all authors have seen the most recent version and that they are happy for you to submit it
   • Check the box “This preprint has not been published…”
   • Click “COMPLETE YOUR SUBMISSION”

NOTE: in all boxes EXCEPT the DOI box, if a URL has an underscore (“_”) in it, it will show up as broken at the PCI website because they use markdown. Therefore, add a backslash (“\”) just before the underscore in the URL to make it show up properly at their website (e.g., http://corinalogan.com/Preregistrations/g\_sociallearning.html).

3. Next, you will have the opportunity to suggest Recommenders (editors) to handle your submission.

• To figure out who you want to suggest, go to PCI Ecology > About > Recommenders and search by key words that you type in and/or by thematic field (note: “Toggle thematic fields” unchecks all boxes!). After you click “Search”, scroll down to see the results.
• Click on people’s names to see their profiles and get more information about whether they would be appropriate (e.g., lots of people who work on biological invasions study plants, not animals). If this information isn’t in their profile, search for their name on Google Scholar and read more about their work at their websites.
• Suggest at least 5 Recommenders, but suggesting 10 is much better (consider balancing gender and whether they are based in a country that is over- or under-privileged).
• NOTE: if the search function at the PCI Ecology website stops letting you search for more Recommenders, just click the DONE button and it will take you back to an area where you can navigate to search for more Recommenders.

Resubmitting revisions to PCI at the pre-study peer review stage

When it comes time to RESUBMIT your work to PCI after it has been peer reviewed, draft the rebuttal at Google Docs, and include a letter to the Recommender and reviewers at the top. In the letter, make sure it includes a link to the HTML version (the reviewers like to read the HTML versions) as well as a link to the version-tracked version of the document (e.g., the .Rmd file at GitHub). Here is some example text:

“Dear [insert Recommender and reviewer names],

We greatly appreciate the time you have taken to give us such useful feedback! We are very thankful for your willingness to participate in the peer review of preregistrations, and we are happy to have the opportunity to revise and resubmit.

We revised our preregistration and associated files at http://corinalogan.com/Preregistrations/g_sociallearning.html, and we responded to your comments (which we numbered for clarity) below.

Note that the version-tracked version of this preregistration is in rmarkdown at GitHub: https://github.com/corinalogan/grackles/blob/master/Files/Preregistrations/g_sociallearning.Rmd. In case you want to see the history of track changes for this document at GitHub, click the previous link and then click the “History” button on the right near the top. From there, you can scroll through our comments on what was changed for each save event and if you want to see exactly what was changed, click on the text that describes the change and it will show you the text that was replaced (in red) next to the new text (in green).

We think the revised version is much improved due to your generous feedback!

All our best,

[Insert author names]“

Tips for writing the rebuttal in a way that makes it easier for the editor and reviewers:

• Number the comments and your responses so you can easily cross reference and so readers can orient themselves in the document (see example).
• Include quotations of the text you changed directly in the rebuttal document so readers don’t have to read the rebuttal and the manuscript, they can just read the rebuttal to see what changed (see example).
• You can upload your rebuttal as a PDF file if you wish. Or you can copy and paste the rebuttal directly into the text box at PCI, which will ultimately show up in markdown format. Therefore, if you want the rebuttal to be formatted in a particular way, use this markdown cheatsheet to make that happen. One additional formatting command that is not mentioned in the cheatsheet but might be useful: in case you want to separate text/paragraphs by more than a single line, write “ ” at the start of a new line followed by pressing enter to create a blank line after it.
• If you ended up changing the title of your preregistration, remember to update it at the PCI website it in response to the reviewer comments before you submit your rebuttal.

Congratulations! You received In Principle Acceptance at PCI! Now what?

Now the PCI managers will ask you to format your preregistration so it shows the PCI logo, the citation for the preregistration, and the citation and link for the PCI Recommendation (see an example). The Rmd code for how to make all of this happen is in the example, but I also list it here:

<img width=”50%” src=”logoPCIecology.png”> #Logo available at https://peercommunityin.org/movies-and-posters/

**Cite as:** Logan CJ, Rowney C, Bergeron L, Seitz B, Blaisdell A, Johnson-Ulrich Z, Folsom M, McCune K. 2019. Is behavioral flexibility manipulatable and, if so, does it improve flexibility and problem solving in a new context? (http://corinalogan.com/Preregistrations/g_flexmanip.html) In principle acceptance by PCI Ecology of the version on 26 Mar 2019 https://github.com/corinalogan/grackles/blob/master/Files/Preregistrations/g_flexmanip.Rmd.

<img width=”5%” src=”logoOpenAccess.png”> <img width=”5%” src=”logoOpenCode.png”> <img width=”5%” src=”logoOpenPeerReview.png”>

**This preregistration has been pre-study peer reviewed and received an In Principle Recommendation by:**

Aurélie Coulon (2019) Can context changes improve behavioral flexibility? Towards a better understanding of species adaptability to environmental changes. *Peer Community in Ecology*, 100019. 10.24072/pci.ecology.100019

– Reviewers: Maxime Dahirel and Andrea Griffin

How to cite your preregistrations that have received an in principle acceptance

The Max Planck Society counts my preregistrations that have passed pre-study peer review as research outputs and these are included in our departmental evaluations. I list these as “in press” because this is the most analogous traditional term. They just spend more time in press than an article would that was submitted to a journal after the work was finished. I cite my preregistrations that are in review and those that have passed pre-study peer review on my CV and in applications. At this stage, the Max Planck Institute for Evolutionary Anthropology librarian uploads it to their repository and lists them as “in press” and then it is automatically listed at Google Scholar.

Once they pass post-study peer review, the citation will change to:

Logan, C. J., MacPherson M, Rowney, C., Bergeron, L., Seitz, B., Blaisdell, A., Folsom M, Johnson-Ulrich, Z., & McCune, K. (2019) Is behavioral flexibility manipulatable and, if so, does it improve flexibility and problem solving in a new context? (http://corinalogan.com/Preregistrations/g_flexmanip.html) Peer Community In Ecology, 100019 (peer review history https://ecology.peercommunityin.org/public/rec?id=17&reviews=True).

Conducting your study

Make sure you keep track of any deviations from the preregistered methods and analyses and that you justify them to ensure the research remains scientifically valid. We find it handy to do this in a section called “State of the Data”, which is at the top of each preregistration (see an example). We update the same Rmd file that we originally submitted for pre-study peer review so we never have to go looking for other versions of this document.

Before resubmitting to PCI for the post-study peer review, deposit your data in a repository. We like the Knowledge Network for Biocomplexity’s data repository because it is free and has great metadata requirements that make your database findable.

We haven’t yet gotten to the post-study stage with our peer reviewed preregistrations at PCI Ecology, but Dieter has a great idea about how to turn a preregistration into a final manuscript without having to rewrite everything: above the preregistration, add the background, results, and discussion using a short journal article format. The article would have the following structure:

1. Title and authors
2. PCI badge with the citations to the preregistration with in principle acceptance and the review history
3. an updated abstract which states your findings
4. a short introduction that presents the relevant background information and states your hypotheses (but you don’t necessarily need all of the details about the various predictions and their alternatives)
5. results where you briefly state the method used
6. discussion
7. the modified preregistration, starting with the state of the data section listing any deviations, the detailed hypotheses and predictions, and the methods and analysis plan, including any code

Perhaps at this point you might be wondering whether it would be better to have this one preregistration be more than one finished article. Maybe you want each hypothesis to be its own article as a stand alone piece. In this case, you could write new text for points 3-6 above (you could also change the title, and potentially some authors are only affiliated with certain hypotheses), while keeping the relevant pieces of the preregistration (point 7 above). Your preregistration will move from having an in principle acceptance to a full acceptance only after all pieces of the preregistration have passed post-study peer review.

Because the research is now complete, it is eligible to be a preprint and you could opt for putting a PDF copy of the Rmd file at a preprint server rather than having the HTML version as the easy-to-read option.

Resubmitting to PCI: post-study peer review stage

You finished your study and analyzed the results and revised your preregistration into its final draft! Now you resubmit it it to PCI as a new submission and write in the cover letter that it is the post-study submission for the preregistration that passed its pre-study peer review. Suggest the same Recommender that handled your submission before because they will be familiar with the work and can more easily assess any changes that need to be addressed. In the cover letter, highlight where in the submission you explain any potential deviations from your preregistration.