What history tells us about diversity in the peer review process

With this year’s Peer Review Week focusing on diversity, there has been a lot of discussion of the changes that could or should be made to ensure that peer review is not being done by people who all think the same, or who all share the same implicit biases. Our historical data has some rather striking things to say about the effectiveness of certain kinds of intervention (On why diversity matters to peer review, see ‘Then and now’).
We have been able to count the number of women who were invited to act as reviewers of papers submitted to the Royal Society from the 1920s onwards. We can compare these figures with those for women authors, and women Fellows of the Society.
The number of women fellows steadily increased after 1945 (when women were first admitted to the Royal Society), and continued – very slowly – climbing. It had only reached 3.5% in the 1980s, and was still only 8% in 2017.
The participation rates of women as authors and as reviewers do not follow the same trend.

Women had already been involved in authoring papers before 1945; and there was no necessary link between being in the fellowship and authoring a paper (although, until 1990, you did need to know someone in the fellowship to be able to submit a paper). The dip in submissions from women authors around World War 2 reflects trends observed elsewhere about a masculinisation of technical disciplines around and after the war years. I have no insight into the decline between 1975 to 1985, which is doubly intriguing for coming at a time when the Society was managing to attract a growing number of submissions in the biological sciences (in which more women were active). To have only 4% of submitted papers having a female author in the 1980s does seem surprising. Things have since improved: the Society’s 2017 figure is that 33% of authors of (published) papers were women.
What about reviewers? In the 1940s, only fellows could review, so the number of women involved was obviously limited. But in 1969, the Society lifted that restriction. One might predict that enabling editors to choose reviewers from the entire pool of scientific researchers, not just those who were fellows of the Society, would lead to a higher proportion of women reviewers. So it is rather surprising to see that FEWER women acted as reviewers in 1975 and 1985 than in the years before the opening up of reviewing (and that this is true even though there were also more women in the fellowship by this time).
This raises questions about how reviewers were actually being chosen. It certainly raises suspicions about the power of social/personal networks within the scientific community, and the extent to which the networks of women scientists connected with those of the senior male scientists in the Royal Society. In other words, giving editors more freedom to choose reviewers did not encourage them to choose more diverse reviewers.
In 2017, the Royal Society reports that 30% of its reviewers were women (close to the level of women authors, as one would surely hope). This is a striking change from 1985, not least because we are not aware of any formal change in procedures or policy regarding the choice of reviewers since 1969. This suggests that it is not rule changes that matter, but changes in practices, probably forced by the strongly stated collective ambition (at institutional level) to improve diversity.

What has changed?

Unfortunately, without the editorial data from 1990 to 2017, we cannot be quite sure when the change occurred…
Since 1990, each journal has had an editorial board, which expands the personal networks on which the editor can draw when seeking referees.
And since 2015, the Society has become very actively concerned about diversity. As well as appointing a Diversity Officer, this has involved annual reporting of diversity statistics in all areas of its operations, including publishing.
And before we celebrate too much, we should note that the modern figures do not tell us how much reviewing each of the women reviewers actually did. Our historical data suggest that the early women fellows (in the 1950s/60s) were much like the male fellows: some did lots of reviewing, while others did none. But we also know that the most active reviewers – the ones who reviewed the greatest proportion of papers – were all male. We do not know what that pattern looks like now. In other words, we may now have more women doing refereeing, but what proportion of all refereeing are they doing?
We need more data!

Postscript: Where do the data come from and how do we attribute gender?

The historical data have been published in tabular form in Rostvik & Fyfe (2018); the graph was created for this post. The 2017 figures come from the Royal Society’s Diversity Data Report.
Note that the historical data are for ‘number of papers authored/reviewed by women’, and are based on a full sample of that year’s papers (including papers that were not ultimately published); gender identity has been attributed by us (see below).
The 2017 figures are based on the responses received from a survey to authors/reviewers that year: it is a partial and probably self-selecting sample; the gender identities are self-reported. The 2017 figures are of the proportion of reviewers/authors who were women. Feel free to comment on how comparable the two sets of figures are – but it’s what we’ve got…
The number of female fellows comes from the Society’s list of fellows, which includes dates of election and of death, and has a field for gender.
The historical data on authors, reviewers and communicators come from the ‘Register of Papers’ ledgers (in the RS archives), which contain an entry for every paper submitted to the Society since the 1850s. We have taken annual samples every decade (i.e. every entry for each sample year has been entered into a database). For those years where our sample year yielded a very low number of women, we manually checked the years on either side, just in case that year was atypical (it wasn’t).
We realise that gender diversity is certainly not the only sort of diversity that we should be concerned with when thinking about peer review. But historians can’t send out surveys to deceased reviewers, asking about their gender identity, ethnicity, or disability. When all we have to work on is a name (and sometimes, only initials), there is not that much we can do about identifying ethnicity, socio-economic class or disability. Attributing gender is not entirely reliable either, but we can at least have a stab at it.
In this respect, it is helpful that the Royal Society had a historical tendency to make gender visible in the editorial process: female authors routinely got ‘Miss’ or ‘Mrs’ in front of their initials (sometimes, even if they had a PhD), and/or had their first names written out in full. Male authors were identified only by initials, unless they were ‘Sir Neville’ or ‘Sir James’. This gives us a way to identify gender. The resulting figures may be under-estimates, since we know that, by the 1970s, if not earlier, there was a handful of younger women who carefully hid behind initials when building their scientific careers.
The Register of Papers was discontinued in its paper form in 1990, which is why our historical data stops there. The Royal Society did not start publishing diversity statistics for authors/reviewers until 2017. The data that would shed light on the intervening period are in electronic editorial management systems; and we have not (yet) got access to them.

Further reading

Full research paper on women in peer review at the Royal Society: Rostvik & Fyfe (2018) in Open Library of the Humanities

Overview of the above research: Fyfe & Rostvik (2018) in Nature

For background on the origins of the Royal Society’s use of refereeing (peer review), see Moxham & Fyfe (2017/2018) in Historical Journal

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.