Designing an Open Peer Review Process for Open Access Guides

The LIBER Citizen Science Working Group is embarking on the design of an open peer review process for the guidebook series being published on the topic of citizen science for research libraries. The LIBER working group in collaboration with COPIM is looking for input and feedback on the design of the open peer review workflow. COPIM is supporting the working group by contributing its experience and knowledge of open access book publishing, with respect to collaborative post-publication input, community peer review processes, and reuse. The first section of the guide Citizen Science Skilling for Library Staff, Researchers, and the Public has already been published with three more sections to follow.

About open peer review

Open peer review (OPR) is a part of an ongoing revaluation of peer review for the humanities and social sciences, which has been given greater focus because of the growth of the open science agenda across academia (Ochsner et al., 2020, Wolfram et al., 2020). The paper Toward Open Research is a call for a dialog on open research as opposed to open science as ‘in practice the humanities still trail behind the sciences in open research’ (Longley Arthur & Hearn, 2021) and it is this context that we are giving careful consideration to OPR design in relation to the challenges in this specific context of creating learning materials from community contributions such as a guide book series.

The push to reform peer review is symptomatic of the unease voiced across many scholarly disciplines and from different stakeholders of academe from, research libraries, and funders, to intergovernmental bodies such as UNESCO with its open science recommendations (UNESCO, 2021). This unease can be seen in how peer review has come to be used as part of an arbitrary form of metrics and evaluation and as a yardstick of academic quality (as just two examples). The DORA Declaration that addresses a model for a more fair system of research assessment is one such expression and context, calling to ‘eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations’ (DORA, 2012).

The COPIM report Promoting and Nurturing Interactions with Open Access Books: Strategies for Publishers and Authors – Part 1: Interaction in Context (Adema et al., 2021) highlights a wide number of questions related to the potential value of OPR and the debates that have informed how peer review in the humanities and social science is being reshaped. Below is an outline of a small selection of considerations from the report pertinent to the design of an OPR process for guides:

• ‘How can new forms of peer review further contribute to this co-production of knowledge?’ – Review practices in the humanities differ from STEM disciplines or academia in general where the focus is evaluation, assessment, quality control. Alongside this in the humanities there has been ‘a “process of collaborative improvement of research”’, as Knöchelmann calls it, instead of ‘being used predominantly as a gatekeeping practice’ (Knöchelmann, 2019).

• ‘Rethink how we conduct quality evaluation within scholarship’ – “open peer review (OPR) is an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the ethos of Open Science, including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process” (Ross-Hellauer, 2017).

• Community and conversation – “deepening the relationship between the text and its audience” (Fitzpatrick, 2012). In this sense for Fitzpatrick open peer review of long-form text can help build a community around a publication in a way that starts to elide the difference between author, reviewer, and reader.

• Drawbacks of OPR – Reviewing in the open inevitably brings along new tensions and biases — ranging from self-censorship, reinforcing hierarchies, or gender bias. Other issues around reach and being able to create a community around a publication, which can in turn end up adding additional workload on already thinly stretched labour time and resources.

The question for a guide publication is how can OPR be beneficial for the uptake of a set of practices to promote citizen science. Open access guides are predominantly free of charge, seek reuse, and look to be practically effective in the uptake of working practices or other types of explicit knowledge that can be codified and transferred (Nonaka, 1994). In the case of Citizen Science for Research Libraries our target audience are research library staff who will support research projects that make use of citizen science processes, for example of crowdsourcing, co-creation, and other forms of participatory learning. A reuse use case in this context would be one focused on content swaps and remixing with partner publications such as done by the SciStarter’s The Library & Community Guide to Citizen Science (Cavalier et al., 2020), which is a publication targeted at public libraries.

How does OPR work?

OPR is about making the scholarly review process open, but the apparent simplicity of just saying it is open is deceptive as doesn't expose the tacit and implicit knowledge hidden in the process and its workings. The how to implement OPR and its workflow design is the key to making it meaningful or useful for the reader, author, and reviewer.

Here are a handful of examples where very different OPR workflows have been put in place:

• It can be about complete transparency for example the publisher eLife has the process open from the get-go, including the editor decision letter (Sai Srinivas Panapakkam 2022);

• The Collective Wisdom Handbook: Perspectives on Crowdsourcing in Cultural Heritage - community review version. This publication has used sprints for the authoring process and then for review made an open invitation using the PubPub platform, stating ‘We are were open for review!’ (The Collective Wisdom Handbook, 2021)';

• Another approach to the process can be about 'post publishing review';

• It can be about community engagement for example with The Turing Way on the topic of data science (The Turing Way Community, 2020);

• Its aim could be nurturing a review culture – the Public Philosophy Journal have created a detailed web of support, ‘Their review process involves supporting both the publications as they go through their development and the people involved in the formative peer review process. They do this by setting up “review teams” which “develop an inclusive, supportive space in which ideas are explored and refined collaboratively”. For them this practice of formative review and publishing is a way to create a publics and to support collegiality and academic service work’ (Adema et al., 2021),1 and finally;

• ORCID Peer Review is tracking and displaying a reviewers activity so that the review contributions are attached to a scholars ORCID PID. Plaudit is one such service that uses the ORCID and DOI technical connection to surface reviews.

Designing the guide OPR process

Currently Citizen Science for Research Libraries – A Guide has a closed peer review process which we are looking to make open. The guide is made by volunteers of the working group, contributors, and partners, and collaborators from a variety of international communities.

The guide is made up of four individual sections, each covering one topic — skilling, infrastructure, open science practice, and programme development. Each section is about eight thousand words long, with approximately twelve articles with as many contributors. The initial releases are intended to be the start for further editions, for new sections to be added, and to be transformed into other learning formats and platforms.

Below are the initial consideration being taken into account for the OPR design process. In addition COPIM have produced a set of guidelines as part of their report on Promoting and Nurturing Interactions with Open Access Books. The report sub-section is called Part 3: Recommendations, Guidelines, and Best Practices (Adema et al., 2021) and lays out a series of considerations that will feed into the design process.

What do we want from the OPR process?

1. We need to actively design a process that will beneficial and will aid with: content ideas and quality of editorial work; reuse of content; design of the delivery format to support targeted learning outcomes; improve enhanced publishing technical quality (these are the 'checklist' of omissions mentioned earlier, using: PIDs, interoperable formats, attribution, etc.);

2. Potentially create a more formulaic review process for reviewers. Have some tick box replies; as specific questions; ways to solicit creative feedback;

3. Start small and build up, and create something reviewers will feel comfortable and excited about using';

4. Make sure that reviewers are given a clear structure for reviewing and given clear ways to get support. Demonstrate how the review will be made open;

5. Communicate that post-publishing feedback is solicited on content and presentation;

6. Feedback is sought for the editorial and publishing processes and technical implementations — especially thinking about reuse in other training programmes or platforms;

7. How to make all stage of the process open? This is while being sensitive to reviewers, authors, designers and contributors, while being productive for the publication and the citizen science community;

8. Some parts of the publication will be made using sprints or dashes. These are time-boxed authoring sessions. How can we add OPR to sprints and dashes?

Learn more about the publication

You can find out more details about the publication and the OPR design on GitHub. It is not anticipated that the OPR workflow can be released until after the Summer of 2022 as contributions have already come in and retrospectively applying a new peer review process isn't a workable scenario. We would introduce the new OPR process for new additional sections and editions, or to assist with the guide’s transformation to other learning product formats, such as webinars, or connections to other skilling provisions such as BESPOC which is a single point of contact service for citizen science and libraries being developed in the LIBER working group.

The following steps are anticipated for the OPR design process:

• Initial stakeholder and community consultation;

• publish a working paper for further community consultation;

• put in place a version 1.0 OPR process for new guide sections, revisions for Autum 2022; and

• evaluate new OPR process , report on progress, consult community, and release a version 1.1.

If you would like to make contributions or check-in on the result the working process see our discussion forum on GitHub. We will come out with a draft set of recommendations as a working paper from June 2022 to share with colleagues at the LIBER Annual Conference in July 2022.

Header image: Fragment of the Citizen Science for Research Libraries, CC BY 4.0. Image based on: European Space Agency (ESA), ASAR global monitoring Mode of the Antarctic, https://www.esa-photolibrary.com/ESA/media/20377. The material is ESA copyright and is supplied to you free of charge on the following terms and conditions https://www.esa-photolibrary.com/ESA/info2.do | About Envisat https://earth.esa.int/eogateway/missions/envisat/descriptionmissions/envisat/description