I’ve noticed that the peer review process is fairly opaque – I have received little guidance or feedback on my process, and I think that everyone has varying standards and ethos with little scrutiny.
To that end, I detailed my peer review process in a blogpost: https://williamngiam.github.io/blog/my_peer_review_process. The blogpost can be used as a guide or resource to familiarize oneself with the peer review process and perhaps inform their first attempts, but moreso, I’m looking for criticism and scrutiny as to improve my own process and start discussions around the standards and tools in the process.
Thanks for sharing your post, I enjoyed reading about your process for doing peer-reviews. I agree there aren’t many resources available for learning about peer review and there are even fewer first-hand accounts of individual’s processes (I assume that this gets passed on when people chat informally about peer review with colleagues).
I was impressed by how carefully you described checking the research material supporting a manuscript (preregistration dates, sample sizes in datasets, reproducing statistical results and recovering parameters; described in sections 4 and 5). The reviews I have done rarely had enough supporting material available to do this, but even when it has been available, I mostly focused my review on what was contained within the manuscript itself (inc. supporting material), and I sometimes checked up on key references if I wasn’t familiar with them.
In terms of accepting reviews, I now only accept reviews from invertebrate vision science (my former field) if the abstract looks very exciting or very relevant to my previous publications. However, I have changed fields and would be more open to reviewing articles from my new field (physical virology, although I haven’t yet published in this) in future. As for ratios, I have 14 peer reviews to 6 first, 2 senior author publications. I have reviewed for some publishers with business models I no longer support, but I probably would not do so in future. This is partially based on taking the Gold OA Free Our Knowledge pledge, it hasn’t activated yet, but does help me to be aware of what I’m supporting (or not supporting) by peer reviewing for a journal.
The structured description of your reviewing process actually reminded me of a checklist, I know these are sometimes used for author submission (reporting checklists) but I don’t recall ever seeing them in relation to peer reviews (although they are mentioned briefly in this meta-summary). I tend to use a fairly ad hoc workflow when reviewing and often find myself jumping back and forth between the sections you’ve described (I assume this is common but have never really asked anybody about it). I can imagine having a checklist would make the review more thorough and consistent, and probably also reduce the time I spend on it. I wonder if creating an open science peer review checklist could be a nice activity for a hackathon sometime. What do you think?