More submissions, more rejections: the Royal Society Journals since the 1950s

I looked at the numbers of submissions to the Royal Society journals in an earlier post. Here, we look at the relationship between the number of submissions, the rejection rate and the sustainability of peer review.

Graph 1: Number of articles submitted vs published in all the Royal Society research journals, 1952-2018

Graph 1 clearly shows two phases: a period when the Royal Society’s Transactions and Proceedings received a modest number of submissions, and published most of them; and a period when its journals received and published much more, but also rejected much more.

The transition point is in the gap in the data (for an explanation, see below): in 1990, significant changes were made to the organisation and scope of the Royal Society’s journals. Among them, and almost silently, the traditional requirement that papers could only be communicated via a fellow was finally dropped, and direct submissions became the norm. It meant that the Society’s journals were now open to a much wider pool of authors: no longer limited to those personal or professional networks intersected with those of one of the Society’s mostly UK-based fellows. The number of submissions rose; and would rise further in the early twenty-first century when the Society made a concerted effort to reach a global pool of authors.

The requirement for communication via a fellow had been experimentally (and briefly) removed in the mid-1970s. At that time, the editorial office reported that the result was more articles, but few worth publishing, and this was taken as evidence that the communication requirement was not a significant obstacle to the submission of publishable papers; and remained a valuable way of saving editorial effort by filtering out unpublishable papers.

The post-1990 experience suggests that the Society did manage to find significantly more papers worth publishing, but it is interesting that the biggest growth in submissions came in the early 2000s, rather than immediately after the removal of ‘communication’, i.e. it came after the Society made more effort at author-marketing. This suggests that, if you want to attract more papers from a wider diversity of people, simply removing an obstacle is not enough: you have to actively reach out to potential new groups of authors. By 2010, over 60% of authors in Royal Society journals were from outside the UK; it had only been 10% in 1950.

Increased submissions means an increase of editorial and reviewing work, to assess, select and improve the papers for publication. We have written elsewhere (Fyfe et al, 2020) about the challenges that faced the Royal Society’s editorial and review system in the early twentieth century, when the Society relied entirely on its own fellowship for reviewers. The problem of a limited pool of reviewers trying to cope with an expanding pool of submissions had been partly solved in 1969 when the Society began asking non-fellows to act as reviewers.

Graph 1 shows that the early-mid twentieth-century strains on peer review pale in comparison to those of the post-1990 period. It is also clear that the ‘cost/benefit’ ratio of editorial and reviewing work (in time and money) must be considerably higher in an era of high rejection rates, than it had been in the pre-1990 era.

Graph 2: the effective rejection rate for Royal Society research journals, 1952-2018

Graph 2 shows the ‘effective rejection rate’: it includes all articles not published, though that may include articles withdrawn by their authors, or never resubmitted after revisions. Historically, the Royal Society’s rejection rate had been only about 10-15%, thanks to the filtering-out performed by the requirement that papers be communicated via a fellow. The post-1990 rejection rates certainly mark a different phase.

I remain intrigued, however, that rejection-rate graph is not as clearly divided into two phases as Graph 1: the rejection rate had already climbed to around 20% by the 1980s. Closer inspection suggests that this was due to changes in the Society’s A-side (i.e. physical sciences): in those fields, submission rates had been slowly declining since the 1950s, but from the mid-1970s, the rejection rate became higher than for the biological sciences. 

Incidentally, the notion that a high rejection rate could be seen as a proxy for quality was definitely not present at the Royal Society before 1990, and nor was it apparent during the 1990s, when editorial staff regarded the newly increased rejection rate of 30% as a real worry, and a threat to the sustainability of the Society’s editorial and review processes.

Where do the data come from?

The data for 1952-1984 come from the Society’s annual reports: key performance indicators were regularly published in the Year Book of the Royal Society, and then (in the 1980s) in the Society’s Annual Report. The indicators included the number of papers submitted, and the number rejected. Earlier data on submissions survive, but not (easily) for rejections. The submissions/rejections in this first batch are for four journals: Philosophical Transactions series A and B; and Proceedings series A and B.

The data for 1994 onwards come from the Society’s current electronic database courtesy of Publishing Director Stuart Taylor. Re-submissions have been excluded for the period after 2000. The submissions/rejections in this batch relate only to the journals then defined as ‘research journals’. Thus, both Transactions are excluded (because they now carry invitation-only thematic review issues); but the new research journals (such as Open Biology and Interface are included).

The gap in the data arises from the fact that the Royal Society stopped publishing details of its number of rejections in the mid-1980s; and the electronic archive only goes back to the mid-1990s. The missing data probably survives in paper form, somewhere in the archive.

For more details on the earlier part of the story, see Fyfe, A., Squazzoni, F., Torny, D., & Dondio, P. (2020). Managing the Growth of Peer Review at the Royal Society Journals, 1865-1965. Science, Technology, & Human Values45(3), 405–429. https://doi.org/10.1177/0162243919862868

Quality in peer review: a view through the lens of time

The Royal Society has been asking for expert advice on papers submitted for publication since the 1830s, and quality (or something like it) has always been one of the elements under consideration. Here, I investigate how the definition of ‘quality in peer review’ has changed over time.

Continue reading “Quality in peer review: a view through the lens of time”

What was the function of early peer review?

What does peer review do? Every academic author nowadays is used to the process of receiving reports on their submitted manuscripts from independent experts consulted by the editor of the journal. Refereeing undoubtedly delays the publication of research, but it is widely believed to add significant value as a means of accrediting ‘proper’ research and researchers.

Amidst all the current discussions of the future of academic publishing, there lurks a strangely ahistorical view of the academic journal. It is not uncommon to hear that the peer-reviewed research journal has been at the heart of the scientific (and, by implication, scholarly) enterprise since the beginnings of modern science. But our research reveals that there is nothing natural, inevitable or timeless about the way academic research is published.

Continue reading “What was the function of early peer review?”

Then and now – exploring diversity in peer review at the Royal Society

This piece on the history of peer review at the Royal Society and the problem of unconscious bias originally appeared on the RS Publishing Blog, 10 Sept. 2018, as part of Peer Review Week 2018. 

Peer review cannot be done by everyone. It can only be done by people who share certain levels of training and subject-expertise, and have a shared sense of what rigorous experimentation, observation and analysis should look like. That shared expertise and understanding is what should enable alert peer reviewers to reject shoddy experimental methods, flawed analysis and plans for perpetual motion machines.

But as we have increasingly come to realise, any group of people with shared characteristics may display unconscious bias against outsiders, whether that means women, ethnic minorities, or those with unusual methods. While peer review should exclude poor science, it should not exclude good research on the basis of the individual traits or institutional affiliation of the researchers, nor should it dismiss innovative approaches to old problems.

However, it seems socio-cultural and intellectual criteria have often been mixed together in the peer review process, and history can help us to understand why.

Continue reading “Then and now – exploring diversity in peer review at the Royal Society”

What history tells us about diversity in the peer review process

With this year’s Peer Review Week focusing on diversity, there has been a lot of discussion of the changes that could or should be made to ensure that peer review is not being done by people who all think the same, or who all share the same implicit biases. Our historical data has some rather striking things to say about the effectiveness of certain kinds of intervention (On why diversity matters to peer review, see ‘Then and now’).
We have been able to count the number of women who were invited to act as reviewers of papers submitted to the Royal Society from the 1920s onwards. We can compare these figures with those for women authors, and women Fellows of the Society.
The number of women fellows steadily increased after 1945 (when women were first admitted to the Royal Society), and continued – very slowly – climbing. It had only reached 3.5% in the 1980s, and was still only 8% in 2017.
The participation rates of women as authors and as reviewers do not follow the same trend.

Continue reading “What history tells us about diversity in the peer review process”

1936: LNG Filon on the importance of journal reputation

“Research qualifications are now more and more insisted upon for appointments to academic and other posts, and appointing bodies have often no means of discriminating between important and trivial research, except the particular medium of publication. The publications of the Society have always been recognized as of exceptionally high standard, and special significance has been attached to papers published in them. Should such discrimination between publications become obsolete or even weakened, a spate of trivial papers may easily outweigh, in the minds of lay persons, a few really valuable contributions, with results ultimately detrimental to the best interests of Science.”

So wrote mathematician (and fellow of the Royal Society) Louis Filon, in the summer of 1936.

Continue reading “1936: LNG Filon on the importance of journal reputation”

Who are the referees?

As a rule, referees had to be Fellows. Only in 1990 (earlier?) did the Society officially allow non-Fellows to referee papers. [If there are earlier examples of non-Fellows could add here.] This meant that there was a finite number of available referees. The number of Fellows changed over time but in the mid nineteenth century, after reforms to reduce the size of the Fellowship, the number of Fellows was around 400 to 500. In reality, a very small proportion of the Fellowship was involved in refereeing. Those who were most active were typically past, current or future Council members. In other words, fellows who were active in one area of the Society’s service tended to be active in other areas too. During his Secretaryship, for example, George Gabriel Stokes was the most active referee, followed by his co-secretary William Sharpey. The Society’s Fellowship in the nineteenth century, when the number of practitioners in science in Britain was growing, represented only an elite group of the scientific community. At the same time, the majority of the papers submitted were from Fellows. Refereeing was thus about judging one’s peers, rather than making judgements on outsiders. Overtime this changed, and increasingly in the early twentieth century more non-Fellows submitted papers to the Society. Refereeing was still conducted by Fellows only but now many authors were not known personally to the Society’s Officers and Council. The Fellowship had grown, and now more Fellows were involved in refereeing: in the 1880s, 8% of the Fellowship were involved in refereeing; this rose to 30% by the 1930s. The workload, however, became even more uneven. The top most active 20% of referees produced just under 40% of referee reports in the nineteenth century, while the same % group in the mid-twentieth century produced 50% of referee reports. Getting a paper through the refereeing process at the Society signified acceptance by representatives of an elite national learned body of judges

1925: Printed referee report forms (used since the 1890s)

In the 1890s, the Royal Society had introduced a set of 7 questions for referees, in the hope of structuring the reports (which were sometimes extremely long-winded!). These were originally hand-written into the covering letter, but were quickly turned into a printed standardised report form, sent to each referee with the manuscript to be evaluated. Referees were encouraged to return their reports within 14 days – a deadline that was routinely breached.

By the early twentieth century, these report forms included clear instructions for referees, including advising them of the confidentiality attached to the papers referred to them (see image). It was routine for the author’s name to be written on the form: refereeing was single-blind, not double-blind. At this time referees were always Fellows of the Society (and their names and reports were kept confidential), but the majority of papers came from those outside the Fellowship.

The report forms made it possible for a referee to present an extremely succinct report, as was the case with Professor H. Lamb’s report on this 1925 paper by ‘Mrs. H. Ayrton’. Hertha Ayrton’s work in electrical engineering had previously been published with and exibited to the Society, but her status as a married woman had prevented the Royal Society accepting a fellowship nomination certificate in her name in 1902. (Her husband was also a well-known electrical engineer, and Fellow of the Royal Society, William Ayrton.)

The printed forms were also an attempt to standardize the refereeing process, or to at least advise referees on how to write an effective report. The Society never officially instructed referees until this date; referees were automatically expected to know how to write a report. Guidance on this continued to develop. By 1926, ‘Instructions to Referees’ was part of the Society’s Standing Orders.

Source: Box RR, 1925-1926, Royal Society Archives, London.

Printed referee report form (used since the 1890s)

In the 1890s, the Royal Society had introduced a set of 7 questions for referees, in the hope of structuring the reports (which were sometimes extremely long-winded!). These were originally hand-written into the covering letter, but were quickly turned into a printed standardised report form, sent to each referee with the manuscript to be evaluated. Referees were encouraged to return their reports within 14 days – a deadline that was routinely breached.

By the early twentieth century, these report forms included clear instructions for referees, including advising them of the confidentiality attached to the papers referred to them (see image). It was routine for the author’s name to be written on the form: refereeing was single-blind, not double-blind. At this time referees were always Fellows of the Society (and their names and reports were kept confidential), but the majority of papers came from those outside the Fellowship.

The report forms made it possible for a referee to present an extremely succinct report, as was the case with Professor H. Lamb’s report on this 1925 paper by ‘Mrs. H. Ayrton’. Hertha Ayrton’s work in electrical engineering had previously been published with and exibited to the Society, but her status as a married woman had prevented the Royal Society accepting a fellowship nomination certificate in her name in 1902. (Her husband was also a well-known electrical engineer, and Fellow of the Royal Society, William Ayrton.)

The printed forms were also an attempt to standardize the refereeing process, or to at least advise referees on how to write an effective report. The Society never officially instructed referees until this date; referees were automatically expected to know how to write a report. Guidance on this continued to develop. By 1926, ‘Instructions to Referees’ was part of the Society’s Standing Orders.

 

1925-1926: Standardised letter to referees
1925-1926: Report on Ayrton
Ayrton report by H. Lamb

1951: Rosalind Franklin at the Royal Society

Rosalind Elsie Franklin (1920-1958) is becoming more known for her contribution to X-ray crystallography and the discovery of the DNA double helix. Brenda Maddox’ excellent biography, ‘The Dark Lady of DNA’ is key reading for anyone interested in Franklin, women in science, or the DNA-discovery saga. But because Franklin was never made an FRS, her times at the Royal Society have often been overlooked. In fact, she both visited, spoke and published with the Royal Society, as this example of a 1950s referee report shows. Note the questions referees are asked, and that the referees in question are JD Bernal and Dorothy Hodgkin, both huge names in the field by 1951. Franklin’s paper was well received by both, as you can see, and published by the Royal Society. Today, a photograph of a young, smiling Franklin hangs to the right of the main staircase when you walk into the Royal Society. Despite her lack of FRS status, her work was recognized by the Society in the fifties; – and today through the Rosalind Franklin award and lecture.

JD Bernal writes in support as a referee. Click for a larger version.
Dorothy Hodgkin writes in support as a referee. Click for a larger version.