Emerging policies to better recognize preprints and open scholarship

Research funding organizations play an important role in setting the tone for what is valued in research assessment through the projects they fund and the outputs they assign value to. Similarly, academic institutions signal what they value through how they assess researchers for hiring, promotion, and tenure. An increasing number of research funding organizations and academic institutions have codified open scholarship into their research assessment policies and practices. Examples include Wellcome, the Chan Zuckerberg Initiative, the Ministry of Business, Innovation & Employment of Aotearoa New Zealand, the University of Zurich and the Open University.

This shift is accompanied by policies that recognize preprints as evidence of research activity (e.g. NIH, Japan Science and Technology Agency, Wellcome, EMBO, and some UKRI Councils). Some funders are now formally recognizing peer-reviewed preprints at the same level of journal articles, such as EMBO and many of the cOAlition S funders. A preprint is a scholarly manuscript that the authors upload to a public server but has not (yet) been accepted by a journal (it is usually the version submitted to a journal if the authors do decide to take it further for journal publication). It can be accessed without charge and, depending on the preprint server, is screened and typically posted within a couple of days, making it available to be read and commented on. Because preprints offer a means outside of journals to share research results, they have the potential to support responsible assessment by decoupling journal prestige from assumptions on the quality of research findings. Preprints also enable sharing of a range of outputs and results that may not be attractive to a journal (for example, research that is technically sound but has a limited scope, or null/negative findings). Because they are also usually free to post, preprints can also help reduce author-facing cost-associated barriers often associated with traditional open access publications, although these services are typically therefore reliant on ongoing grant funding to maintain their sustainability.

One of the most recent examples of a substantial policy change is the Bill and Melinda Gates Foundation’s March 2024 announcement of its upcoming Open Access Policy in 2025. The 2025 policy will introduce two changes for grantees: grantees will have to share preprints of their research, and the Gates Foundation will stop paying article processing charges (APCs). As with many shifts towards open access policies, these changes were motivated by several factors, including the Gates Foundation’s desire to provide journal-agnostic avenues for research assessment and to empower their grantee authors to share different versions of their work openly on preprint servers and without the costs of APCs. Reducing the costs associated with publishing research and increasing readers’ accessibility to research via preprint servers also supports more equitable access to research products.

The Gates Foundation used ten years of existing data to inform its decision to refine its Open Access Policy, has engaged actively with community dialogue, and has made it clear that this is a first step on a longer path to better evaluate research on its own merits and increase accessibility to research. Notably, the Gates Foundation took this step after taking into account the existing shortcomings of current open access models that rely on APCs that effectively limit global access to research outputs. Given that the Gates Foundation is “the wealthiest major research funder to specifically mandate the use of preprints,” this approach is groundbreaking in its emphasis on preprints and its shift away from spending on APCs. It has also placed a spotlight on these issues and catalyzed discourse around trust in preprints. Policy changes like this indicate a willingness among research funders to take steps toward change and move away from recognizably flawed processes. This is an important step, since flawed processes are often retained because fixing them or adopting new processes is perceived as too high effort and too high risk (also known as the status quo bias).

Overcoming the status quo and tackling new challenges

Overcoming the status quo bias is difficult, but not impossible. Indeed, a common concern around changing research assessment processes to include new ways of sharing knowledge is taking a leap into the unknown. Because these new policies are on the leading edge of change, there are gaps in our knowledge around their effects on research culture and assessment. For example, will assessors penalize researchers who include preprints in their CVs or will research culture shift what it values?

Another key question centers on how preprints will impact our concept of traditional manuscript peer review processes. Traditionally, journals select a panel of peer reviewers, ideally field experts, who voluntarily review manuscript submissions for free. These detailed reviews inform an editor’s decision on whether to publish a manuscript. Generally, preprints are only lightly checked before being made public, after which anyone can read and comment on them and provide peer feedback. One common concern is that preprints are only subject to light checking before being made public, though it is important to note that issues with rigor and reproducibility exist within current peer-review publication systems. Preprint peer feedback holds the potential for positive change, opening up the opportunity for authors to receive a wide range of community input and making it easier to spot issues early.

One step to foster trust in preprints will be to create a shared understanding of what preprint “review” is. What qualifies as review in the context of a preprint was recently defined via expert consensus as “A specific type of preprint feedback that has: Discussion of the rigor and validity of the research. Reviewer competing interests declared and/or checked. Reviewer identity disclosed and/or verified, for example, by an editor or service coordinator, or ORCID login.” Additionally, there are a growing number of preprint review services available, for example VeriXiv (created through a partnership between the Gates Foundation and F1000), Peer Community In and Review Commons who have all created infrastructure and pipelines to verify preprints and facilitate structured and invited expert peer review of preprints, post publication. They provide journal-independent assessment of the preprint, typically using various forms of open peer review practices, making it more transparent, fostering accountability and enabling reviewers to be rewarded for their contributions to the field. However, some have raised concerns about whether greater transparency increases risk of retaliation, particularly for early career researchers, although recent evidence suggests that more research is needed to determine if repercussions occur.

Questions like these are legitimate and highlight the value of the organizations that are actively seeking to answer them, like the Research on Research Institute, which studies the results of research policy reform using the same scholarly rigor that reform efforts are trying to foster in the academic ecosystem. Organizations like ASAPbio are working to address concerns around the agility of preprint servers to correct or retract preprints and to support rigorous and transparent preprint peer review processes.

In the meantime, fear of unintended consequences is not reason enough to avoid trying to improve research incentives and the culture associated with it. The changes that research funders are implementing to recognize and incentivize open scholarship practices are on the leading edge of reform efforts, pushing research culture forward in new ways that aim to address existing burdens caused by APCs and journal prestige. As with all policies that aim to shift assumptions around what can and should be valued in research, gaps in knowledge will need to be filled through iteration, open dialogue with groups that new policies will impact, and careful study of how new policies change research culture.

Responsible research assessment and open scholarship are interconnected

Responsible research assessment: An umbrella term for “approaches to assessment which incentivise, reflect and reward the plural characteristics of high-quality research, in support of diverse and inclusive research cultures.” –RoRI Working Paper No.3

As well as progress in the reform of research assessment, a further fundamental change in the research ecosystem over the past decade has been the emergence of open scholarship (also known as open science or open research)¹. The UNESCO 2021 Recommendation on Open Science outlined a consensus definition of open science that comprises open scientific knowledge (including open access to research publications), open dialogues with other knowledge systems, open engagement of societal actors, and open science infrastructures. It is an inclusive movement to “make multilingual scientific knowledge openly available, accessible and reusable for everyone, to increase scientific collaborations and sharing of information for the benefits of science and society, and to open the processes of scientific knowledge creation, evaluation and communication to societal actors beyond the traditional scientific community.” This definition captures the broad nature of open scholarship: it is both a movement to change how scholarly knowledge is shared, and to address global inequities in scholarly culture itself.

DORA (Declaration On Research Assessment) is a global non-profit initiative that actively works with the scholarly community to support responsible research assessment for hiring, promotion, tenure, and funding decisions. DORA is part of a global movement that aims to reform research assessment equitably including expanding the definition of what gets assessed, and changing the way the assessment takes place. Reducing emphasis on flawed proxy measures of quality such as the Impact Factor or h-index, broadening the type of work that is rewarded, and challenging assumptions about quality and excellence are critical facets of the movement towards responsible research assessment. However, these core concepts do not exist in a vacuum (see Venn diagram by Hatch, Barbour and Curry).

The concepts of research assessment reform, open scholarship, and equality and inclusion cannot be treated separately.

Responsible research assessment is intricately linked with open scholarship, and also with equity and inclusion initiatives. Biases and assumptions about research quality can determine who is assessed and how they are assessed, and decoupling research products from journal prestige is an important step to address these biases.

Greater transparency and accessibility enables the recognition of a broader range of scholarly outputs (including datasets, protocols, and software). In alignment with the aims of open scholarship, DORA seeks to address the “publish or perish” culture by recognizing and rewarding transparency, rigor, and reproducibility. Ultimately, enabling and rewarding rigor and transparency serve to foster trust both within academia and with the broader public.

The intersection between these movements is apparent in many policies and practices being adopted at academic institutions around the world. Reformscape, a database of research assessment reform at academic institutions, contains over twenty examples of how institutions are incorporating aspects of open scholarship into their hiring, promotion, and tenure practices. Many of DORA’s in-depth case studies of institutional research assessment reform include mention of the institution codifying open scholarship into their practices.

Fostering public trust through open scholarship requires a systems approach

A critical part of research culture is trust. There are many ways to build trust. Traditional peer reviewed journals emphasize building trust by invoking expert opinion. Open scholarship emphasizes building trust through transparency: making all stages of the research process, from conception to data collection to analysis and review visible to all.

The two approaches are not mutually exclusive. Peer reviewed journals have created policies to promote openness, and several forms of open scholarship have sought ways to solicit expert opinions. However, peer review has been used as the main signal of trustworthiness for so long that it can be difficult to convince researchers that having an article pass peer review isn’t necessarily a definitive signal to say that the article is trustworthy. Consequently, many have not been convinced about the value of sharing research ahead of peer review, and have been concerned that removing that vetting would open the gates to flawed research and increase the risk of misinformation.

In practice, a couple of studies (here and here) have suggested that the distinctions between peer reviewed journal articles and preprints can be minimal. Preprints have gained increased acceptance from researchers who are posting more preprints, and from reporters who are writing more stories based on preprints (although more work needs to be done to ensure that disclaimers about the lack of peer review are always added).

Understanding what scholarly communication can be viewed as trustworthy was, is, and always will be a complex task for both experts and non-experts alike. Experts are expected to gain this through advanced training and experience. Non-experts might benefit from increased media literacy, a subject that is taught to less than half of US high school students.

Call to Action

Reforming research assessment requires new policies and practices that embrace diverse scholarly outputs, reduce the emphasis on journal prestige as an implicit indicator of research quality, and evaluate research based on its intrinsic value. As we expand the type of work that we recognize to include preprints and other “non-traditional” outputs, we can foster trust in these new outputs by 1) recognizing and rewarding transparency, rigor, and high-quality review, and 2) by developing resources to foster and support responsible preprint review. For example, a growing number of bibliographic indexers are starting to index preprints (e.g., Europe PMC, PubMed Central) and there are several efforts to index and link reviews to preprints (e.g. Sciety, COAR Notify). There are also a number of efforts underway to develop a range of consistent trust signals and markers. Alongside these efforts lies the crucial task of consistently educating and communicating about innovations in publishing and open scholarship practices to cultivate public trust and literacy.

Change on this scale is not immediate, nor should it be. DORA has long advocated for the strategy of iteratively fine tuning policies over time using data and input from their target communities. As more and more research funders and institutions test new ways of rewarding researchers for their open scholarship practices, it is important to seize the opportunity for careful review, refinement, and fostering an open dialogue on what works and what doesn’t.

Zen Faulkes is DORA’s Program Director
Haley Hazlett is DORA’s Program Manager

Acknowledgements: The co-authors would like to thank DORA co-Chairs, Ginny Barbour and Kelly Cobey, and DORA Vice-Chair, Rebecca Lawrence for their editorial input on the piece.

¹ The term “open scholarship” will be used throughout this piece for consistency. Different organizations also use the terms “open research” and “open science” to describe broad policies that encourage openness, though often all three terms generally have a holistic focus on fostering a culture of openness, transparency, and accessibility. DORA uses “open scholarship” to better encapsulate all scholarly disciplines.

The post Encouraging innovation in open scholarship while fostering trust: A responsible research assessment perspective appeared first on DORA.

Announcements

Reformscape in Full Swing

Since its release in January, Reformscape has been serving the research community in providing 230 documents encouraging openness & transparency. New documents and information continue to be added and are always publicly available. Upgrades have also been made to make the platform more user-friendly. You can now search for responsible research assessment resources based on type, making it easy to find:

• Action plans
• Policies to reform hiring, promotion or tenure
• Outcomes of new policies
  

New guidance released

DORA’s opposition to the overuse of the Journal Impact Factor is well known, but the original declaration did not specifically address other forms of indicators that are sometimes used as proxy measures of quality in research assessment. In a new guidance document, we examine the potential problems of not only the Journal Impact Factor, but the h-index, altmetrics, and various other citation measures. None of these indicators are without problems, but the guidance provides five principles to help reduce some of the concerns.

This guidance is available on the DORA website and Zenodo. For questions about this guidance, email info@sfdora.org.

New Narrative CV Report

DORA is pleased to announce a new report on the implementation and monitoring of narrative CVs for grant funding. This report was created in collaboration with FORGEN CoP, Science Foundation Ireland, the Swiss National Science Foundation, UK Research and Innovation, and the University of Bristol, Elizabeth Blackwell Institute for Health Research. The report summarizes takeaways and recommended actions from a joint workshop held in February 2022 on identifying shared objectives for and monitoring the effectiveness of narrative CVs for grant evaluation. More than 180 people from over 30 countries and 50 funding organizations participated.

Read the report

New report on improving pre-award processes

The processes that take place before research is submitted for funding (pre-award processes) serve as important scaffolding to support equitable and transparent research assessment. This report summarizes the key recommendations from DORA’s Funder Discussion Group symposia and workshops to improve pre-award processes, which were held in collaboration with the Elizabeth Blackwell Institute for Health Research (EBI) at the University of Bristol and the MoreBrains Cooperative.

Read the report

Building on this work, we are pleased to also announce that DORA, EBI, and MoreBrains are continuing their collaboration and are developing a new project to look at how three of the recommendations could be implemented. In May 2024, we will host two workshops that will bring DORA’s Funder Discussion Groups together with research administrators and managers to generate tools and guidance that address practical implementation of these recommendations.

DORA seeks new steering committee member from Asia

DORA is looking for Steering Committee members based in Asia. To be considered for this position, please complete this self-nomination form by May 31, 2024.

Approaching 25,000 signatories

There are nearly 25,000 individuals and organizations that have recognized a need to promote responsible research assessment. Every day the number of signatories for DORA increases and we anticipate reaching 25,000 signatories by summer. Signing the San Francisco Declaration on Research Assessment signifies a dedication to the principle that scientists should be evaluated on their individual achievements and quality of their work rather than on journal-based metrics, such as Journal Impact Factor. We are excited to reach the 25,000 mark and share this milestone with the research community!

The post DORA Newsletter May 2024 appeared first on DORA.

During this quarters’ initiatives meeting, our very own Haley Hazlett, Program Manager, presented about Reformscape, a searchable collection of assessment criteria and standards from academic institutions. Reformscape is a tool developed by the TARA team at DORA and has been molded and improved over the past two and a half years by including community discussions and user testing. Reformscape is meant to inform institutions on how reform occurs by providing information on reform policies, processes, action plans, announcements etc. for multiple disciplines and levels of career. The tool has been active since January, 2024, and will continue to grow and expand in information, materials and geographics.

During the round table discussion we heard updates from multiple organizations including the following:

• HELIOS Open organized a workshop in which 50 senior leaders (presidents, VPs, provosts etc.) joined in Miami to discuss modernizing tenure and promotion procedures.
• The Southern African Regional Universities Association (SARUA) is creating a community of practice for responsible research assessment at academic institutions in South Africa mentioning their interest in the participation of experts to help drive the discussion.
• The Natural Sciences and Engineering Research Council of Canada (NSERC) has published guidelines on research assessment, developing an evaluation framework for research, training and mentoring.

Suggested Reading List:

• Project TARA
  • https://sfdora.org/project-tara/

• Reformscape link
  • https://sfdora.org/reformscape/
• Reformscape FAQ
  • https://sfdora.org/reformscape/reformscape-faq/
• Reformscape Glossary
  • https://sfdora.org/reformscape/glossary/

• HELIOS Open
  • https://www.heliosopen.org/testimonials-1
• NSERC
  • https://www.nserc-crsng.gc.ca/NSERC-CRSNG/Policies-Politiques/assessment_of_contributions-evaluation_des_contributions_eng.asp 
• Transforming Evidence Funders Network
  • https://transforming-evidence.org/projects/transforming-evidence-network

• Global Research Council
  • https://globalresearchcouncil.org/about/responsible-research-assessment-working-group/

Casey Donahoe is DORA’s Policy Associate

The post DORA Initiatives Meeting: Reformscape appeared first on DORA.

January 30, 2024 to February 4, 2024

The Reformscape press release was picked up by Research Information and Information Today.

The Reformscape launch was mentioned in the CARL-ABRC newsletter (Canadian Association of Research Libraries or Association des bibliothèques de recherche du Canada) and the GraspOS newsletter this week.

We thank the contributor who added Reformscape to DORA’s Wikipedia page.

Brian Owens wrote an article about Reformscape for the journal Nature: How to make academic hiring fair: database lists innovative policies. (Originally free to read, now paywalled.) This was also shared in the Nature Daily Briefing for January 31, 2024.

We also want to thank the many people who liked and shared our posts on social media during this first week that Reformscape was available to everyone!

February 2024

Our colleagues in CLACSO shared news about Reformscape’s launch (in Spanish).

March 2024

Social Science Space (S3) picked up the Reformscape press release.

April 2024

Illinois Tech featured the contributions of Project TARA’s Ruth Schmidt to Reformscape.

The post Reformscape in the news appeared first on DORA.

For years, the traditional way in which academic careers are assessed has been criticized as being unfair, biased and unfit for purpose. An excessive focus on narrow criteria and publication metrics, with an overreliance on journal impact factors and quantity of output rather than the quality and diversity of research, has left talented people overlooked and held back progress in diversity, equity and inclusion.

Free to use through a user-friendly online portal, Reformscape is a rich, organized dataset bringing together hundreds of real-life examples showing how universities and other academic institutions around the world are bringing in fairer, more responsible and more informative approaches to academic career assessment. Reformscape is populated with policies, action plans and other documents from more than 200 institutions from all over the world, together with trends and expertly curated insights.

Administrators, faculty and others in the academic community can explore Reformscape for ideas and inspiration around how to implement new approaches to career assessment and progression in their own institution, and also share their own policies and plans with the wider world. 

For example:

• Read institutional profiles describing detailed summaries of their actions and progress. 
• Access source materials to dig into the details of announcements, action plans, policies and published practices. 
• Search and filter examples by location, career stage, discipline and date.  
• Read expertly curated insights and commentary by the DORA team drawn straight from the data.
• View graphs and visualizations showing trends in responsible academic career assessment over time and by country.
• Share institutional profiles and insights with your networks to spark conversations and celebrate progress. 
• Keep up to date with new materials added over time by using meta tags as well as the latest insights shared by the DORA team.  

Reformscape has been developed as part of Project TARA (Tools to Advance Research Assessment) – a collaboration between Sarah de Rijcke, Alex Rushforth and Marta Sienkiewicz at the Centre for Science and Technology Studies (CWTS) at Leiden University, Netherlands, Ruth Schmidt at the Institute of Design at Illinois Tech, Stephen Curry at Imperial College London, and Anna Hatch, Haley Hazlett and Zen Faulkes at DORA. The project was co-created with members of the academic community and is supported by Arcadia, a charitable foundation that works to protect nature, preserve cultural heritage and promote open access to knowledge.

Professor Stephen Curry is the former chair of DORA who worked on the Project TARA team. He said, “DORA is very much a community effort to discover, develop and share solutions to the knotty problems of research assessment and this approach is very much at the heart of our new Reformscape tool. We are immensely proud of what we’ve achieved and excited to see how it will be used to foster the uptake of fairer and more robust career assessment in institutions across the world.”

Sarah de Rijcke, Professor of Science, Technology and Innovation Studies at Leiden University, said, “Reformscape is a significant step in evolving the way we assess academic careers. Its launch is an encouraging development towards more equitable and inclusive academic evaluations. I’m hopeful about the potential impact of this tool, especially in expanding our perspective on academic achievements beyond conventional measures. I’m keen to see how it grows to include more institutions and diverse career stages, offering a more nuanced view of academic merit.”

Notes and resources:

• Reformscape is free to use and can be accessed online at http://reformscape.org.
• A short sharable video introducing Reformscape is available on Youtube: https://youtu.be/IklEMsuTleA 
• Logos for DORA, Arcadia and Reformscape and screengrabs demonstrating the resource are available here: https://drive.google.com/drive/u/0/folders/1ph4c5dKY4_-Bf4CpbHdaWkEv7GaaejtO 

About DORA

The San Francisco Declaration on Research Assessment (DORA) is a worldwide initiative to improve how scholarly research is evaluated, starting at the American Society for Cell Biology 2012 meeting in San Francisco. DORA covers all scholarly disciplines and all key stakeholders including funders, publishers, professional societies, institutions, and researchers. We encourage all individuals and organizations who are interested in developing and promoting best practice in the assessment of scholarly research to sign DORA. Find out more: https://sfdora.org 

About Arcadia

Arcadia is a charitable foundation that works to protect nature, preserve cultural heritage and promote open access to knowledge. All of Arcadia’s awards are granted on the condition that any materials produced are made available for free online. Find out more: http://www.arcadiafund.org.uk  

Media contact:

Zen Faulkes, DORA Program Director
reformscape@sfdora.org

– 30 –

The post Announcing Reformscape: a new online tool to explore responsible academic career assessment and drive positive change appeared first on DORA.

The Open University is an interesting example of how changes in assessment can contribute to a broader strategic goal. It also offers inspiration for how to create a participatory change process.

Between 2012-2015, the Open University took action to strengthen their knowledge exchange with society. One of the solutions they devised was to introduce a new knowledge exchange-focused career track for academic faculty. Assessment criteria for promotions were revised to include, for instance, contributions to business, community, policy, practice or product/service development; leadership in knowledge exchange; and contributions from knowledge exchange to the university’s teaching and learning. 

This change was informed by action research conducted with staff at all levels of the organization. Involving staff revealed that they had a relatively narrow understanding of ‘engaged research’ and the ‘communities’ with whom they could exchange knowledge. Therefore, the change process focused on formulating broader definitions to deepen future engagement and capture their breadth in promotions, in line with the strategic knowledge-exchange mission of the University.

The follow-up surveys and evaluations of the program were published in a peer-reviewed article to promote both internal and community-wide learning.

Curious to know more? Read Open University’s background material in which they documented their process and results.

The post Stimulating engaged research through changes in the promotions process appeared first on DORA.

Utrecht University (UU) and Utrecht University Medical Centre (UMC) have been steadily developing policies and practices aimed to reward a variety of academic activities in assessments.

Their most recent tool that captured our attention is a set of conversational guidelines. This tool helps to translate high-level commitments to improving assessments into new practices on team and individual levels. It equips leaders and colleagues with better ways to reflect on their achievements and goals.

This tool speaks to the fact that individual academics are embedded in various teams and should be assessed in specific team contexts. They were designed to support more tailored ways of working and evaluating in different parts of the University.  

The guidelines can be used to organize structured discussions on themes of team spirit and leadership; team impact; mapping collective activities, competences, and qualities. They also include questions and approaches useful for discussing individual development in annual reviews or for stimulating ad-hoc reflections and feedback.

Curious to know more? Check out the background material that details this and other initiatives undertaken by Utrecht University and Utrecht University Medical Center.

The post A guide to good conversations appeared first on DORA.