Peer review is the foundation of our whole system of recognizing and sharing scholarship. At the same time, it consists of unpaid labor that is rarely rewarded or recognized, though some efforts are now being made. Its burden also falls disproportionately on the conscientious who are more likely to respond to requests and to do a decent job as reviewers. Those who ignore invitations to review or do such a poor job that they will not be asked again can free ride on the system. We need better ways to incentivize scholars to do their fair share of peer review. Publons, a website that tracks peer reviews and works with both journals and individual scholars, promises to help with this problem by tracking and analyzing the work that referees do. More journals, including Apeiron and British Journal for the History of Philosophy, have begun using this service or encouraging their referees to do so.
Is this service helpful? Should the philosophy community be using it more? I’ve been using Publons for the last three years and thought I’d share my impression of the pros and cons. From an individual perspective, it’s a useful way to have all my refereeing work documented in one place without requiring additional time and energy on my part. For partnered journals I don’t have to do anything and for those that aren’t partnered I just need to forward the email acknowledging that my report has been received and wait a few days (in my experience, it takes 3-6 business days to process submitted reviews). I then have a complete record of the work I’ve done to use for my university’s review and promotion process, which require documenting all of my service. I find this much easier than hunting through all my emails. Publons also tracks whether you’re re-reviewing the same piece or doing something new. While its analytic metrics probably aren’t that accurate or well-calibrated for philosophy, they may be instrumentally useful. A university wide committee may be more impressed to hear that I’m in the 94th percentage for all peer reviewers than that I’ve done 11 peer reviews in the past year. I’m also very goal oriented, so I like the reward I get just from seeing that my review has been processed and counted. It reinforces the pleasure of completing the process and seeing that I’ve done my part.
Reviewing is also much more suited to quantitative measures than philosophical work itself. I sympathize with worries about judging research on quantity instead of quality. This does seem to give researchers the wrong incentives. Perhaps limiting how many publications a tenure committee looks at might lead to more careful and bigger picture thought. However, in the case of refereeing the main considerations are whether the review good enough and delivered in a timely fashion. We would all like to have thorough consideration of our work, but I, at least, will settle for decent refereeing that doesn’t take too long. Beyond the low (but not always met) bar of actually reading and thinking about the paper you’re reviewing and avoiding serious misunderstandings, it’s hard to judge the quality of refereeing and I don’t think we need too much qualitative assessment. We want to make sure referees do well enough in their reports and consideration (and avoid using those who fail to do this), but then the main thing is not to take too long. A system like Publons which tracks how many reviews people do and how long they take does give us data that tracks reasonably well with the features that matter in reviewing. It would also be good to build up community norms about how many peer reviews are reasonably expected (taking into account the nature of someone’s position, teaching load etc.). That would allow us to better recognize those who are doing a lot and put pressure on those who aren’t doing enough. I think Publons helps to do this.
My one main worry is the uncertainty about what their future plans are and what they might do with all this data. Sites such as academia.edu were useful platforms for sharing research at first but now are too focused on finding ways to monetize our work and our interest (for a small yearly fee, I can find out which famous researchers are reading my work!). With that caveat, I’m gladly continuing to use Publons and would encourage others to do so, at least under current conditions.