Research Ideas and Outcomes : Editorial
PDF
Editorial
Open science in practice: 300 published research ideas and outcomes illustrate how RIO Journal facilitates engagement with the research process
expand article infoDaniel Mietchen, Lyubomir Penev§,|, Teodor Georgiev, Boriana Ovcharova, Iva Kostadinova
‡ School of Data Science, University of Virginia, Charlottesville, United States of America
§ Pensoft Publishers & Bulgarian Academy of Sciences, Sofia, Bulgaria
| Institute of Biodiversity & Ecosystem Research - Bulgarian Academy of Sciences and Pensoft Publishers, Sofia, Bulgaria
¶ Pensoft Publishers, Sofia, Bulgaria
Open Access

Abstract

Since Research Ideas and Outcomes was launched in late 2015, it has stimulated experimentation around the publication of and engagement with research processes, especially those with a strong open science component. Here, we zoom in on the first 300 RIO articles that have been published and elucidate how they relate to the different stages and variants of the research cycle, how they help address societal challenges and what forms of engagement have evolved around these resources, most of which have a nature and scope that would prevent them from entering the scholarly record via more traditional journals. Building on these observations, we describe some changes we recently introduced in the policies and peer review process at RIO to further facilitate engagement with the research process, including the establishment of an article collections feature that allows us to bring together research ideas and outcomes from within one research cycle or across multiple ones, irrespective of where they have been published.

Introduction

RIO’s mission is to publish the research process, to facilitate engagement with both the process and its outcomes and to highlight how the research published this way relates to societal challenges (Mietchen et al. 2015). When it was launched, such framing was very unusual for a scholarly journal. Hence, RIO was designed with flexibility in mind, so as to allow for its technical and policy parameters to be adjusted and adapted on an ongoing basis.

Since then, the research landscape has been evolving and the importance of a wider sharing of the substantial and diverse bits and pieces underlying the various research processes is receiving broader attention. While there are some steady developments in this direction, much of the observable progress was triggered by disruptive events, such as the Ebola and Zika epidemics and now the ongoing COVID-19 pandemic. This resonates with the observation, in a review of Hurricane Katrina, that "Open data matters most when the stakes are high". At RIO, we agree, but think that open data and open processes matter even when the stakes are not high or not (yet) known to be high.

We also observe some concerning trends, from increased “openwashing” to the continued “consolidation” of the publishing industry, where so-called “transformative agreements”, “read and publish” and other formats of large corporate deals between traditional publishers and well-funded research consortia eat up resources that could have instead been used to actually improve the research landscape. At RIO, we will continue to emphasise innovation for the benefit of the research ecosystem, rather than just a few individual players.

In this editorial, we explore some of the key developments in RIO over the last few years, how they relate to societal challenges and how RIO can continue to stimulate experimentation in this space by launching exciting new features and opportunities for researchers, projects, institutions, funders and readers.

Overview of research published in RIO so far

There are various ways in which the content of RIO can be grouped. Here, we will look primarily at two of RIO’s key unique features: distribution by the various article types - which roughly correspond to different stages of the research cycle - as well as at the Sustainable Development Goals, to which RIO articles are mapped routinely. On that basis, we will highlight various ways in which RIO readers engage with RIO content and explore how these publications can serve as a resource for research projects.

Article types

As of 31 March 2021, RIO had published 300 articles. Of these, 32 (i.e. 11%) were traditional publication types from the end of a research cycle as published in most scholarly journals, while 132 (44%) were from early stages that rarely get published elsewhere and 136 (45%) from intermediate stages of the research cycle whose coverage in the scholarly record has traditionally been patchy (Table 1).

Table 1.

Distribution of the first 300 RIO articles by research cycle stages: early in the research cycle (132), intermediate outcomes (136) and final outcomes (32). The “Grant proposals” row groups together all article types for grant proposals, including the two generic ones (Grant Proposal and Small Grant Proposal) and six funder-specific ones (for COST, Horizon 2020, NSF, NIH, FWF and DFG). The “Other” row groups all article types with just one example in RIO so far: two early types (Software Management Plan and PostDoc Project Plan) and six intermediate ones (Correspondence, Editorial, PhD Thesis, Monitoring Schema, Research Poster and Short Communication).

Article type

Count

Example

Stage in the research cycle

Grant Proposals

76

Tracking Invasive Alien Species (TrIAS): Building a data-driven framework to inform policy (Vanderhoeven et al. 2017)

early

Project Reports

38

Support Your Data: A Research Data Management Guide for Researchers (Borghi et al. 2018)

intermediate

Research Ideas

33

Mental synthesis involves the synchronization of independent neuronal ensembles (Vyshedskiy and Dunn 2015)

early

Workshop Reports

33

The London Workshop on the Biogeography and Connectivity of the Clarion-Clipperton Zone (Glover et al. 2016)

intermediate

Research Articles

23

Neurobiological mechanisms for nonverbal IQ tests: implications for instruction of nonverbal children with autism (Vyshedskiy et al. 2017)

final

Data Management Plans

16

Data Management Plan for a Biotechnology and Biological Sciences Research Council (BBSRC) Tools and Resources Development Fund (TRDF) Grant (Gatto 2017)

early

Policy Briefs

13

Community engagement: The ‘last mile’ challenge for European research e-infrastructures (Koureas et al. 2016)

intermediate

Case Studies

12

Case Study: Indigenous Knowledge and Data Sharing (Neylon 2017)

intermediate

Review Articles

9

Hazards and disasters in the geological and geomorphological record: a key to understanding past and future hazards and disasters (Tilley et al. 2019)

final

Methods

7

Methods & Proposal for Metadata Guiding Principles for Scholarly Communications (Kaiser et al. 2020)

intermediate

Commentaries

5

Genetic Testing for Type 2 Diabetes in High-Risk Children: the Case for Primordial Prevention (Wessel and Marrero 2017)

intermediate

PhD Project Plans

5

Physics of Laser in Contemporary Visual Arts: the research protocol (Ahmedien 2016)

early

Single-media Publications

4

EU BON’s contributions towards meeting Aichi Biodiversity Target 19 (Despot-Belmonte et al. 2017)

intermediate

Software Descriptions

4

PyLogFinder: A Python Program for Graphical Geophysical Log Selection (Amosu and Mahmood 2018)

intermediate

Research Presentations

3

Online direct import of specimen records into manuscripts and automatic creation of data papers from biological databases (Senderov et al. 2016)

intermediate

R Packages

3

Novel pedagogical tool for simultaneous learning of plane geometry and R programming (Briz-Redón and Serrano-Aroca 2018)

intermediate

Conference Abstracts

2

Induced pluripotent stem (iPS) cells and somatic cardiac regeneration — An exploratory bioinformatic analysis (Chen and Yu 2016)

intermediate

Data Papers

2

Groundwater quality dataset of Semarang area, Indonesia (Irawan et al. 2018)

intermediate

Forum Papers

2

Copyright and the Use of Images as Biodiversity Data (Egloff et al. 2017)

intermediate

Guidelines

2

Foundational Practices of Research Data Management (Briney et al. 2020)

intermediate

Other

8

Benefits and costs of aphid phenological bet-hedging strategies (Joschinski 2016)

2 early, 6 intermediate

Examples by funder

Research ideas can be conceived, developed and published without a specific funder in mind, but the implementation of such ideas typically requires some level of funding. Conversely, research funders have a mission to support research in various ways. While funders could, in principle, peruse public collections of research ideas to identify researchers, research infrastructure or research projects to fund, the predominant way of allocating research funds is for a funder to review a set of grant proposals submitted to them and to select some of them for funding.

Accordingly, early stages of the research cycle are particularly sensitive to priorities and policies of research funders. With that in mind, Suppl. material 1 provides a set of examples of RIO articles describing research associated with specific funders and indicates their funding status.

The TableTable 1 shows that these examples cover national funders like the German Research Foundation (DFG, for example, Altenhöner et al. 2020), the Environmental Protection Agency of Ireland (Kelly-Quinn et al. 2019) and the National Science Foundation of the United States (NSF, for example, Fisher and Nading 2016), but also multi-national ones like Horizon 2020 (e.g. Eisenhauer et al. 2019), the European Cooperation in Science and Technology (COST, for example, Datry et al. 2017) or the European Environment Agency (e.g. Moldovan et al. 2019), as well as private ones, like the Gordon and Betty Moore Foundation (White 2016), the Shuttleworth Foundation (Agosti 2016) or the Alfred P. Sloan Foundation (Rasberry et al. 2019). Many other examples are of funders operating at other scales, be it specific to a given institution (e.g. Luleå University of Technology, as per Ek et al. (2017)) or demographic (e.g. the Hungarian diaspora, as per Tóth (2016)). Some examples are from funding schemes involving multiple funders: for instance, several submissions to the Open Science Prize - a collaboration between Wellcome, the United States National Institutes of Health and the Howard Hughes Medical Institute (cf. Kittrie et al. (2017)) - have been published in RIO (e.g. Wojnarski and Hanken Kurtz (2016)).

A good number of examples in the Table are about global development funded out of Europe and North America, including through Canada’s International Development Research Center (IDRC, for example, Neylon and Chan (2016); see also the dedicated RIO collection), the Swiss Programme for Research on Global Issues for Development (r4d, for example, Véron et al. (2018)) and the Norwegian Agency for International Cooperation and Quality Enhancement in Higher Education (Diku, for example, Prylutskyi et al. (2019)). In contrast, only a small number of funders from outside Europe and North America are represented in RIO articles (e.g. King Abdulaziz University, as per Astek (2019) or the National Research Foundation, South African Institute of Aquatic Biodiversity, as per Stefanoudis et al. (2020)).

Sustainable Development Goals

Besides opening up the research process, RIO emphasises the connection between research and societal challenges, in particular, by mapping its articles to the United Nations’ Sustainable Development Goals (SDGs, see Griggs et al. (2013) and Lu et al. (2015)).

While papers published in RIO have addressed each of the 17 SDGs, some goals remain quite under-represented: five of the goals have 50 or more RIO articles to them, six of the goals have 10 or less (Table 2). Some possible reasons for this could be that research funding and interest might be unevenly distributed across SDGs or scholarly communication practices might differ between communities addressing different SDGs or the popularity of RIO or open science more generally might vary amongst them.

Table 2.

Overview of the Sustainable Development Goals, with examples of RIO articles that have been mapped to them and the type of these articles.

No. of articles

SDG

Example article

Article type

111

SDG9: Industry, innovation & infrastructure

Gentoo Linux for Neuroscience - a replicable, flexible, scalable, rolling-release environment that provides direct access to development software (Ioanas et al. 2017)

Software Management Plan

92

SDG15: Life on land

ConservePlants: An integrated approach to conservation of threatened plants for the 21st Century (Fišer et al. 2021)

Grant Proposal

91

SDG3: Good health & well-being

Data Management Plan: Opening access to economic data to prevent tobacco related diseases in Africa (Woolfrey 2017)

Data Management Plan

59

SDG14: Life below water

World Register of marine Cave Species (WoRCS): a new Thematic Species Database for marine and anchialine cave biodiversity (Gerovasileiou et al. 2016)

Project Report

51

SDG4: Quality education

Building and hacking open source hardware (Monachino et al. 2018)

Workshop Report

44

SDG17: Partnerships for the goals

Widening the circle of care: An arts-based, participatory dialogue with stakeholders on cancer care for First Nations, Inuit, and Métis peoples in Ontario, Canada (Hammond 2016)

Small Grant Proposal

28

SDG10: Reduced inequalities

Eliminating disparities and implicit bias in health care delivery by utilizing a hub-and-spoke model (Joseph 2018)

Grant Proposal

25

SDG13: Climate action

The shadow of the future and the shadow of the past : Studying the impact of climate change on human behaviour (Vollan 2019)

Grant Proposal

22

SDG11: Sustainable cities and communities

A Political Ecology of Value: A Cohort-Based Ethnography of the Environmental Turn in Nicaraguan Urban Social Policy (Fisher and Nading 2016)

Data Management Plan

15

SDG16: Peace and justice strong institutions

Exploring the opportunities and challenges of implementing open research strategies within development institutions (Neylon and Chan 2016)

Grant Proposal

12

SDG6: Clean water & sanitation

Monitoring and risk assessment for groundwater sources in rural communities of Romania (GROUNDWATERISK) (Moldovan et al. 2019)

Grant Proposal

10

SDG8: Decent work & economic growth

Case Study: Strengthening the Economic Committee of the National Assembly in Vietnam (Neylon 2017)

Case Study

10

SDG12: Responsible consumption & production

The London Workshop on the Biogeography and Connectivity of the Clarion-Clipperton Zone (Glover et al. 2016)

Workshop Report

4

SDG2: Zero hunger

Supercritical carbon dioxide pasteurization to reduce the activity of muscle protease and its impact on physicochemical properties of Nile tilapia (Sugiharto et al. 2020)

Research Article

4

SDG5: Gender equality

Rotatory role-playing and role-models to enhance the research integrity culture (Prieß-Buchheit et al. 2020)

Grant Proposal

2

SDG7: Affordable & clean energy

Challenges in Swedish hydropower – politics, economics and rights (Ek et al. 2017)

Grant Proposal

1

SDG1: No poverty

Social processes in post-crisis municipal solid waste management innovations: A proposal for research and knowledge exchange in South Asia (Véron et al. 2018)

Grant Proposal

We are experimenting with a more granular mapping between RIO articles and the SDGs and have thus introduced the possibility for articles published on SDG14 (Life below water) to also indicate which of the ten Targets under SDG14 they help to address. This way, six of these Targets have RIO articles associated with them, of which Target 14.a (Increase scientific knowledge, develop research capacities and transfer marine technology) with 17 articles (e.g. Bingham et al. (2017)), Target 14.2 (by 2020, Sustainably manage and protect marine and coastal ecosystems) with nine9 articles (e.g. Datry et al. (2017)) and Target 14.c (Enhance the conservation and sustainable use of oceans and their resources) with five articles (e.g. Drazen et al. (2019)) are the most popular.

While RIO uses English as its default language, we are aware that this is a barrier for some to engage with its content. Since all RIO content is openly licensed, anyone is free to translate any part of it into any languages they are interested in and this we encourage. We also support the publication of multilingual content, as long as an English version is available. For instance, Increasing understanding of alien species through citizen science (Alien-CSI) (Roy et al. 2018) is a Grant Proposal with its abstract available in 30 languages. Two other Grant Proposals - Conservation of saproxylic beetles in the Carpathians (Mirea et al. 2021) and A transnational cooperation for sustainable use and management of non-native trees in urban, peri-urban and forest ecosystems in the Alpine region (ALPTREES) (Lapin et al. 2020) - complement their English abstracts with one or four additional translations, respectively.

Examples by affiliation

Most authors of RIO publications are based at universities and research institutions in Europe and North America, particularly in Germany, the United States and the United Kingdom. However, author affiliations in RIO publications are spread across over 70 countries and well beyond classical research institutions, which is a good basis for enhancing societal impact of research globally.

Such affiliations include a range of organizations that work in the field of sustainable development, for instance the Ministry of Environment, Energy and Climate Change of the Seychelles (Deep reef ecosystems of the Western Indian Ocean: addressing the great unknown (Stefanoudis et al. 2020)), the municipalities of Maribor in Slovenia and of Trento in Italy (A transnational cooperation for sustainable use and management of non-native trees in urban, peri-urban and forest ecosystems in the Alpine region (ALPTREES) (Lapin et al. 2020)), the Department of Forests, Ministry of Agriculture, Rural Development and Environment of Cyprus (An approach for the mass propagation of Cupressus sempervirens L. (Cupressaceae), for quality propagule production (Pericleous and Eliades 2020)), the Agency for Assessment and Application of Technology in Indonesia (Supercritical carbon dioxide pasteurization to reduce the activity of muscle protease and its impact on physicochemical properties of Nile tilapia (Sugiharto et al. 2020)), the non-governmental organization "Zan wa Zamin" (Woman and Land) in Tajikistan (BioDATA - Biodiversity Data for Internationalisation in Higher Education (Prylutskyi et al. 2019)), the Institute for European Environmental Policy (The use of biodiversity data in spatial planning and impact assessment in Europe (Underwood et al. 2018), The use of biodiversity data in rural development programming (Underwood and Grace 2017)) or the European Commission and the Swedish Environmental Protection Agency (Citizen Science and Open Data: a model for Invasive Alien Species in Europe (Cardoso et al. 2017)).

Topics covered

RIO is open to submissions from all research fields and has published research from behavioural sciences (Vollan 2019) to neurosciences (Keshavan et al. 2017), sports sciences (Hausken et al. 2018), digital humanities (Altenhöner et al. 2020), nanophysics (Susi 2015), agricultural policy (Ziv et al. 2020), education (Cole 2018), medicine (Cirillo 2020), earth sciences (Kissling et al. 2017), copyright law (Egloff et al. 2017), open hardware (Monachino et al. 2018), software engineering (Wagner 2015), cave biology (Gerovasileiou et al. 2016), robotics (Hardy et al. 2020) and beyond (e.g. Lahti et al. (2019)). However, the distribution of RIO content across fields is not uniform (cf. Suppl. material 2 for a detailed breakdown). In this section, we provide some examples of themes that span across multiple RIO articles.

Improving scholarly workflows

This area is core to RIO’s mission and its support of open scholarship. Contributions to it include explorations of what the Scholarly Commons might look like (Kramer et al. 2016, a Workshop Report), how the Citation Typing Ontology can be leveraged as a discovery tool (Philipson 2016, a Research Idea), how to leverage statistical tools to detect data fabrication (Hartgerink et al. 2016, a Small Grant Proposal), whether videos could help in improving grant peer review (Doran et al. 2017, a Small Grant Proposal), technical aspects of preprint services in the life sciences (Chodacki et al. 2017, a Workshop Report), how to enhance computational reproducibility of research (Ioanas et al. 2017, a Software Management Plan), how reference lists could better facilitate understanding of scholarly publications (Sarja 2017, a Correspondence), best practices for sharing scholarly metadata (Kaiser et al. 2020, a Methods article) or how to assign credit for contributions throughout the research lifecycle (Haak et al. 2020, a Policy Brief).

Data management

Data management is central to contemporary research, open or not and, thus, important to RIO already from the perspective of improving scholarly practices, as discussed in the previous paragraph. Our emphasis on engagement with the research process and the various steps along the research cycle raise the importance of data management higher still, since some aspects of data management are important at every step.

It is thus not surprising that a full-text search for “data management” in RIO articles currently yields 210 results, i.e. 70% of the 300 articles. By comparison, a similar full-text search on PubMed Central yields 45915 hits for “data management” amongst 2671350 articles published since 17 December 2015 (the date when RIO published its first articles) - this is a rate of 1.7%.

This importance of data management to RIO manifests itself in various ways: besides the general importance of data management for all article types, there is a dedicated article type for Data Management Plans and several RIO collections have a strong focus on data management (e.g. Public Data Management Plans created with the DMPTool, Exploring the opportunities and challenges of implementing open research strategies within development institutions or Building the European Biodiversity Observation Network (EU BON) Project Outcomes).

Furthermore, data management plays an increasing role in Grant Proposals on any subject (see, for example, Work Package 1 in Vanderhoeven et al. (2017)), and it is in the focus of a growing number of projects, especially infrastructural ones (e.g. Simms et al. (2016), Steinbeck et al. (2020) or Altenhöner et al. (2020)). It is also the subject of project reports (e.g. Borghi et al. (2018)), workshops and, thus, workshop reports (e.g. Simms et al. (2017), Van Tuyl and Whitmire (2018) or Petersen et al. (2020)), guidelines (e.g. Briney et al. (2020)) and review articles (e.g. Neylon (2017)).

Citizen science

Open science and citizen science have originated largely independently from each other, one emphasizing issues of reproducibility and transparency of research workflows, the other stimulating contributions to research workflows from outside the classical research ecosystem. Over time, they have evolved to interact in various fashions, and RIO is one of the contexts in which they do so.

Examples include Grant Proposals (Klein 2016, Roy et al. 2018, Kelly-Quinn et al. 2019, Fišer et al. 2021), Workshop Reports (Vohland et al. 2016, Cardoso et al. 2017, Sturm et al. 2018), Policy Briefs (Runnel et al. 2016), Case Studies (Stehle et al. 2020), Commentaries (Schade et al. 2019) or Research Articles (Hardisty et al. 2020).

Another context in which open science and citizen science meet is provided by Wikimedia projects (Mietchen 2019) and Wikidata in particular, which features prominently in a number of RIO articles, especially Grant Proposals (Mietchen et al. 2015, Martone et al. 2016, Agosti 2016, Rasberry et al. 2019, Steinbeck et al. 2020, Altenhöner et al. 2020, Rasberry and Mietchen 2021), but also hackathon reports (Sachs et al. 2019).

Biodiversity research

RIO’s founders all have a background in biodiversity research and many of the technical aspects of RIO’s publishing workflows have been prototyped with its sister journal, the Biodiversity Data Journal, which was launched two years earlier (Smith et al. 2013).

On that basis, biodiversity research is very visible in RIO all along the research cycle, especially via RIO collections for biodiversity-focused research projects like ICEDIG and SYNTHESYS+ (both about the digitization and interoperability of natural history collections) or EUBON (integration of biodiversity data across ground-based and remote sensing modalities).

Examples from outside these collections include Grant Proposals (Franz et al. 2016, Datry et al. 2017, Mariani 2018, Prylutskyi et al. 2019, Müller et al. 2020, Cavender-Bares et al. 2021), Research Ideas (Page 2016, Minelli et al. 2018, Samuel et al. 2020), Data Management Plans (Lange Canhos 2017), Workshop Reports (Glover et al. 2016, Sturm et al. 2018, Seltmann et al. 2018), Methods (Marek 2017), Project Reports (Gerovasileiou et al. 2016, Faulwetter et al. 2016, Avila-Poveda 2020), Policy Briefs (Koureas et al. 2016, Underwood et al. 2018, Borsch et al. 2020, Kröger et al. 2021), Case Studies (Rivera-Vega and Mikó 2017), Guidelines (Penev et al. 2017), Review Articles (Tilley et al. 2019) or Research Articles (Mounce et al. 2017, Sancho-Chavarria et al. 2018, Bowser et al. 2019).

SARS-CoV-2 and COVID-19

The ongoing COVID-19 pandemic has triggered a notable shift in the wider research landscape towards more open and more rapid sharing, thereby improving the alignment with RIO. Examples of RIO publications on the virus, the disease or the pandemic can be found in a dedicated collection, which includes Research Ideas (Senapati et al. 2020, Sanal and Dubey 2020) and Grant Proposals (Mietchen and Li 2020), as well as Research Articles (Padilla-Sanchez 2020, Padilla-Sanchez 2021) and Review Articles (Alibek and Tskhay 2020).

Engagement with RIO content

A very basic form of engagement with a journal’s content is to access, browse, explore and read that content. Further forms of engagement may involve bookmarking, sharing, reviewing, annotating, commenting on, building on, reproducing, using or reusing the content. Lin and Fenner (2013) discussed various mechanisms of engagement with research articles and distinguished between articles being viewed, saved, discussed, recommended or cited. In this section, we explore these potential routes of engagement, focusing on RIO content, but expanding the scope beyond Research Articles and beyond those five basic categories of engagement.

All RIO articles and all of their components - for example, individual figures, tables or supplementary materials - are available via both the RIO website and Zenodo. Traffic via Zenodo can be substantial, for example, the Zenodo copy of the Software Management Plan “Gentoo Linux for Neuroscience - a replicable, flexible, scalable, rolling-release environment that provides direct access to development software” (Ioanas et al. 2017) currently has 1522 unique views and 1619 total views, as well as 39 unique downloads and 41 total downloads. Nonetheless, we focus here on traffic via the RIO website, which is provided via each article's Metrics tab.

Viewing

Journal-level views

Traffic at the journal level over a given time period depends on a number of criteria, including the amount of its content, the degree to which the content matches interest in the group of potential readers during that timeframe, as well as findability and accessibility.

For RIO, journal-level traffic data has been in the order of just above 100,000 unique page views in 2020, at an average annual growth rate of about 20,000, for which 2021 is on track (cf. Fig. 1). This sums up to about 340,000 unique page views, in total, until now. Of these, about 100,000 (26%) are due to traditional publications from the final stage of the research cycle (Research Articles and Review Articles), while 74% of RIO’s traffic is due to articles about early and intermediate steps. RIO tracks views to the HTML, XML and PDF versions of its articles and distinguishes between unique views and total views, with the latter including repeat visits. The unique HTML, PDF and XML views represent about 58%, 37% and 5% of the unique views, respectively.

Figure 1.  

Aggregate annual traffic to RIO articles between 2015 and March 2021 by file format.

Article-level views

For each article, RIO records the number of first-time (unique) and repeat visits (as identified via cookies) to the RIO website, as well as total views (i.e. the sum of unique and repeat visits) and makes these data available via an article’s Metrics tab. A list of the ten articles with the highest number of unique views as of 30 March 2021 is given in Table 3. Repeat visits are a basic form of engagement with the content, so the total number of views per article is also indicated in Table 3, as well as the ratio between both numbers (Total versus Unique ratio, τ).

Table 3.

Articles with the most unique views as of 30 March 2021, along with information about total views, the ratio between total and unique views (τ) and the article type.

Rank

Unique views

Total views

Ratio τ of

Total views/ Unique views

Article

Article type

1

14684

23803

1.6

The influence of religion on science: the case of the idea of predestination in biospeleology (Romero Jr. 2016)

Research Article

2

10802

18250

1.7

A review of biodiversity-related issues and challenges in megadiverse Indonesia and other Southeast Asian countries (von Rintelen et al. 2017)

Review Article

3

10423

23624

2.3

Strategies and guidelines for scholarly publishing of biodiversity data (Penev et al. 2017)

Guidelines

4

9258

16128

1.7

Vertical-Horizontal Regulated Soilless Farming via Advanced Hydroponics for Domestic Food Production in Doha, Qatar (Abdullah 2016)

Methods

5

8997

17404

1.9

Support Your Data: A Research Data Management Guide for Researchers (Borghi et al. 2018)

Project Report

6

8565

14156

1.7

The value of statistical tools to detect data fabrication (Hartgerink et al. 2016)

Small Grant Proposal

7

8182

33908

4.1

DNAqua-Net: Developing new genetic tools for bioassessment and monitoring of aquatic ecosystems in Europe (Leese et al. 2016)

Grant Proposal

8

8149

14726

1.8

Language evolution to revolution: the leap from rich-vocabulary non-recursive communication system to recursive language 70,000 years ago was associated with acquisition of a novel component of imagination, called Prefrontal Synthesis, enabled by a mutation that slowed down the prefrontal cortex maturation simultaneously in two or more children – the Romulus and Remus hypothesis (Vyshedskiy 2019)

Research Article

9

8004

16473

2.1

Enabling Open Science: Wikidata for Research (Wiki4R) (Mietchen et al. 2015b)

H2020 Grant Proposal

10

7410

17985

2.4

Controlling the taxonomic variable: Taxonomic concept resolution for a southeastern United States herbarium portal (Franz et al. 2016)

NSF Grant Proposal

Inspecting the data in Table 3 from the perspective of the research stages defined for Table 1 gives rise to a number of observations. First, publication types representing the early (ranks 6, 7, 9 and, 10), late (1, 2 and 8) and intermediate stages (3, 4 and 5) of the research cycle as per Table 1 are all represented. Second, the three late-stage entries in this list of ten is more than would be expected, based on the 11% prevalence of late-stage publications in the overall RIO corpus, but very close to the 29% unique visits going to late-stage articles, as discussed along Fig. 1. Third, τ is highest for early-stage entries in the Table (2.6 altogether), lowest for late-stage entries (1.7) and intermediate for intermediate-stage entries (2.0), so a possible research question could be what the relationship is between different stages of the research cycle and measures of engagement (like τ) with outcomes from those stages. Two more data points on τ for intermediate outcomes: a Workshop Report (Cunha et al. 2020) currently has 21181 total views versus 1171 unique views, i.e. τ = 18 and a Policy Brief (Borsch et al. 2020) has 20229 total views versus 3013 unique views, i.e. a τ = 6.7.

Sub-article-level views

Beyond per-article views, RIO also tracks sub-article view stats, which are likewise made accessible via an article’s Metrics tab. Suppl. materials 3, 4, 5, 9

contain aggregate view stats for figures, subfigures, tables and supplementary materials across the current RIO corpus.

The most viewed figure - Figure 1 of Maumet and Nichols (2017) (a Project Report from a hackathon) - contains sample code. The second most viewed figure - Figure 1 of Bingham et al. (2017) - contains a “map of the global and European biodiversity informatics landscape” and the third one - from Maumet and Nichols (2017) again, this time their Fig. 2 - contains screenshots of example results. Figure views can also be expressed relative to article views: one in four viewers of Senapati et al. (2020) (a Research Idea) specifically viewed its Figure 1, which is a graphical summary.

Many articles in RIO and elsewhere contain plates that combine several original images into one composite figure. RIO provides authors with the possibility to keep the original images individually accessible, which facilitates reuse. As of 31 March 2021, 28 of 300 RIO articles had composite figures that were set up this way.

The top 12 most viewed subfigures are all from just two articles, Marek (2017) (a Methods article that has five figures in total, all composite, with a total of 17 sub-figures) and Vyshedskiy et al. (2017) (a Research Article that has 12 figures in total, of which five are composite, with a total of 12 sub-figures). The first subfigure not from these two - Fig. 6a of Egloff et al. (2017) - is interesting in that it, in turn, depicts a plate composed of 34 illustrations. The 20th-most viewed subfigure has 86% of the views of the most viewed subfigure. This distribution can be interpreted as suggesting that if there are subfigures, then readers tend to go through them more systematically than selectively. The ratio of article views versus subfigure views, which is between 10.5 and 27.3 for all of these 20 most downloaded subfigures, is consistent with this interpretation.

The most viewed table - Table 3 of Chodacki et al. (2017) (a Workshop Report) contains “Principles and recommendations for preprint technology development”. The second most viewed table is the collection of quotes listed in our opening editorial, and the third most viewed table is Table 1 of Koureas et al. (2016) (a Grant Proposal), summarizing the comments that the proposal received during peer review.

Supplementary files receive less attention than main-text ones: six main-text figures have more views than the most viewed supplementary file, an Excel spreadsheet from von Rintelen et al. (2017) containing “Biodiversity-related data and economic information for the 10 ASEAN member states”. Likewise, the top 20 most viewed main-text figures have more views than the second most viewed supplementary file, a Data Sharing Agreement from Egloff et al. (2016).

Saving

Materials published in RIO can be saved in various ways. For instance, an article can be included in a RIO collection, the article or any of its components can be downloaded from RIO or Zenodo, the URL of an article or any of its components can be bookmarked, or the metadata of any of these can be included in reference managers. In the following section, we will concentrate on downloads via the RIO website.

Article-level downloads

All RIO articles are available in HTML, PDF and JATS XML formats and they include figures - mostly in JPEG or PNG formats - as well as tables in CSV format. Some figures are composite figures. Figures and tables are optional and articles can further be complemented by supplementary materials that may come in any format, with common ones being spreadsheet (CSV, TSV, XLS, XLSX, ODS) and word processing formats (DOC, DOCX, TXT, ODT), as well as presentation formats (PPT, PPTX, ODP) and PDF again. Note that RIO encourages the deposition of data and other such materials in suitable repositories, as well as citation of such deposits.

Although RIO is designed for its content to be explored online, PDF downloads of its articles remain popular, clocking in at 10000 for the Review Article A review of biodiversity-related issues and challenges in megadiverse Indonesia and other Southeast Asian countries (von Rintelen et al. 2017), 4000 for the Grant Proposal DNAqua-Net: Developing new genetic tools for bioassessment and monitoring of aquatic ecosystems in Europe (Leese et al. 2016), 3000 for the Small Grant Proposal The value of statistical tools to detect data fabrication (Hartgerink et al. 2016) or 1000 for the 164-page Research Article Conceptual design blueprint for the DiSSCo digitization infrastructure - DELIVERABLE D8.1 (Hardisty et al. 2020). There are also articles for which HTML views dominate, for example, the Workshop Report Foresight Workshop on Advances in Ocean Biological Observations: a sustained system for deep-ocean meroplankton (Cunha et al. 2020) with only 913 PDF downloads (i.e. ca. 4%) out of 21186 visits.

Sub-article-level downloads

Beyond whole-article traffic, more granular content types also experience significant download activity across the research cycle, for example, Fig. 2 of the Review Article (von Rintelen et al. (2017), which lays out the transition from traditional to contemporary pathways for knowledge sharing in Indonesia (2000 downloads) or Fig. 2 (a photograph illustrating the atmosphere at the event) of the Workshop Report Citizen Science and Open Data: a model for Invasive Alien Species in Europe (Cardoso et al. 2017) (600 downloads) or Fig. 1 (graphical abstract) and Table 1 (list of expressed proteins) of the Research Idea Chimeric spider silk production in microalgae: a modular bionanomateria (Molino et al. 2016) (both at about 450 downloads) or Table 8 (which defines abbreviations) of the Grant Proposal NFDI4Chem - Towards a National Research Data Infrastructure for Chemistry in Germany (Steinbeck et al. 2020) (100 downloads).

It is interesting to note that the most viewed and the most downloaded files typically differ. Similarly, the most viewed table received 375 total views, while dozens of tables have had more downloads than that, which indicates that users prefer to view tables offline (this preference is not news to us, but provided one of the major reasons why all tables in RIO and its sister journals can be downloaded as spreadsheets).

Suppl. material 7 provides an overview of the most downloaded subfigures. Of the two articles leading the subfigure view stats (Suppl. material 4), one of them - Marek (2017) - also has two of its subfigures leading the subfigure download stats, while the other article has none of its subfigures in the top 20 for downloads. However, similar to the observations about views, the 20th-most downloaded subfigure has 84% of the downloads of the most downloaded subfigure, which again suggests that readers approach subfigures more systematically than selectively. This idea receives further support from the ratio of article downloads versus subfigure downloads, which is between 3.4 and 7.8 for all of these 20 most downloaded subfigures.

Just as for views, supplementary files receive less attention than main-text ones in terms of downloads: the top 20 most downloaded main-text figures (Suppl. material 6) all have more downloads than the most downloaded supplementary file (cf. Suppl. material 10). This is Supplementary Material 2 of Cole (2018) (a PhD thesis) - a PDF containing a list of studies that were included in the meta-analysis as well as of “near-miss” studies that were “potentially relevant [..] but ultimately excluded” and whose description also states “Future researchers might find this list especially valuable.” The second most downloaded supplementary file - Supplementary Material 1 of Criscuolo (2019) (a Research Article) - is an Excel spreadsheet with genomic data on 187 genera. The third most downloaded supplementary file - Supp. 5 of Jay et al. (2017) (a Workshop Report) - is the PDF of a lightning talk given at the workshop.

Some figures - for example, Figure 6 (a PERT diagram) of Smith et al. (2019) (a Grant Proposal) and Figure 2 of Padilla-Sanchez (2020) (a Research Article) - were downloaded more frequently than their respective articles. For most figures, the ratio is inverse. The most downloaded tables (Suppl. material 8) all relate to project management, including a budget in Supplementary Table 1 of Neittaanmäki et al. (2016) (a Grant Proposal). Overall, project management-related materials like PERT and Gantt charts are very popular in terms of downloads, so we discuss them separately.

Resources for projects

With more of the research process becoming visible through publications at all stages of the research cycle, these publications become resources in and of themselves. While not all aspects of research (e.g. the development and provision of infrastructure) can be usefully cast in terms of projects, some key elements of project management are relevant across disciplines. Their traffic statistics indicate that sharing them is of particular interest to RIO readers, so we highlight some examples here, thereby complementing the project-related examples given earlier, especially in the section on data management above.

For instance, PERT charts explain the relationships between different components of a project and examples can be found in Fig. 1 of Vanderhoeven et al. (2017), in Fig. 2 of Roy et al. (2018), in Fig. 2 of Cuenca-Garcia et al. (2018), in Fig. 6 of Smith et al. (2019), in Fig. 1 of Mattsson et al. (2020) and in Fig. 7 of Altenhöner et al. (2020), while Fig. 3 of Cunha et al. (2020) is a very engaging infographic providing a similar overview of relationships between different actors across an entire field of research (deep-ocean meroplankton observations).

Gantt charts visualize the timeline of activities within a project and examples are in Fig. 6 of Franz et al. (2016), in Fig. 2 of Valatin et al. (2017) and in Fig. 4 of Kissling et al. (2017), as well as in Fig. 1 of Mariani (2018), in Fig. 1 of Roy et al. (2018) and in Supplementary Fig. 5 of Astek (2019).

SWOT analyses highlight strengths and weaknesses of, as well as opportunities for and threats to a project, and examples can be found in Fig. 11 of Hardisty et al. (2020) and in Table 2 of Altenhöner et al. (2020).

Risk assessments zoom in on potential threats, estimate their likelihood, assess their potential effects on the course of the project and outline mitigation measures. Examples can be found in Table 12 of Mietchen et al. (2015b) and in Table 4 of Mariani (2018), as well as in Tables 2-7 of Steinbeck et al. (2020).

Another common element of project planning and management is an overview of the project’s governance structure and examples for that include Fig. 4 of Steinbeck et al. (2020) and Fig. 5 of Mietchen et al. (2015b).

Budgets are a key element of project planning and management too and examples can be found in Supplementary Table 1 of Neittaanmäki et al. (2016), as well as in Table 1 of Smith et al. (2016), in Table 2 of ElSabry (2017), in Table 13 of Mietchen et al. (2015b) or in Table 2 of Astek (2019). While it is common to describe budgets in tabular form, there are also a good number of examples that do it in verbal form in a dedicated section, as can be seen, for instance, in Wagner (2015) or in Smith et al. (2019).

For a further key element of research projects - data management plans - RIO has a dedicated collection with examples including the management of legacy data (Nichols and Stolze 2016) or data from a PhD thesis (Pannell 2016) and subjects ranging from hydrology (Fey and Anderson 2016) to proteomics (Gatto 2017) to health economics (Woolfrey 2017) and sociology (Wael 2017).

Review reports of Grant Proposals are even more rarely available than Proposals themselves, but if they are, then that does receive attention, as is the case with the reviews for the Grant Proposal Heteroatom quantum corrals and nanoplasmonics in graphene (HeQuCoG) (Susi 2015) (200 downloads of Supplementary File 1 that contains the reviewer reports) or the Small Grant Proposal Widening the circle of care: An arts-based, participatory dialogue with stakeholders on cancer care for First Nations, Inuit, and Métis peoples in Ontario, Canada (Hammond 2016) (80 downloads of Supplementary File 1 with the reviewer reports). If the reviewer reports cannot be shared, then their content can be summarized, as done in Table 1 of Koureas et al. (2016).

Discussing

At RIO, one way to initiate discussion of an article is to invite others into the ARPHA drafting environment (Penev et al. 2017), where they can be given the rights to review, comment on or edit the draft before submission. This can then be complemented by post-publication reviews and annotations (see Changes in RIO workflows). The nature of the publications in RIO also brings to light new layers of discussion around individual steps and outcomes of the research process. In this section, we thus provide some examples from different settings.

The Research Idea An oral live attenuated vaccine strategy against Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2/2019-nCoV) (Sanal and Dubey 2020) has received three pre-submission peer reviews (Perumal 2020, Alfieri 2021 and Panicker 2021) providing constructive feedback from multiple perspectives, in addition to further context to the idea presented. External feedback at this stage - when the research has not been performed and can thus still be influenced - is very important, yet rarely possible in the current research ecosystem. Another Research Idea received two post-publication reviews outlining problems with the idea presented (for one of these reviews, see Susi 2018). The Research Idea A Million Brains in the Cloud (Klein and Ghosh 2016), on the other hand, was included in an Open Science MOOC and received 29 annotations via Hypothes.is, with which RIO is integrated.

Grant Proposals are typically reviewed by funding agencies, yet the reviewers are generally anonymous and, thus, cannot be easily contacted for permission to publish the reviews along with the proposal. In some cases, though, this can be worked out. For instance, the Grant Proposal Heteroatom quantum corrals and nanoplasmonics in graphene (HeQuCoG) (Susi 2015) has been reviewed by the Austrian Science Fund (FWF) and it has been published along with the two reviews that provided the rationale for the proposal to be funded. Likewise, the Grant Proposal Widening the circle of care: An arts-based, participatory dialogue with stakeholders on cancer care for First Nations, Inuit, and Métis peoples in Ontario, Canada (Hammond 2016) has been reviewed by the Canadian Institutes of Health Research (CIHR) and its three reviews are available alongside the Proposal as well. Another Grant Proposal - Injury-Free Children and Adolescents: Towards Better Practice in Swedish Football (FIT project) (Hausken et al. 2018) - has received an endorsing pre-publication peer review, as well as two post-publication ones, one of which (Spencer-Cavaliere 2021) is rather detailed and explores how different disciplinary perspectives on the matter could be usefully integrated.

The Commentary Open comments on the Task Force SIRS report: Scholarly Infrastructures for Research Software (EOSC Executive Board, EOSCArchitecture) (Gomez-Diaz and Recio 2021) provides a set of comments on a publication by the European Commission (Directorate-General for Research and Innovation (European Commission) 2020) and it has itself received a post-publication review (Duque 2021) which comments on the scope of the definition of “research software” as used in the report.

The Review Article A review of biodiversity-related issues and challenges in megadiverse Indonesia and other Southeast Asian countries (von Rintelen et al. 2017) has received two pre-submission peer reviews (Balke 2021, Freitag 2021) highlighting the value of the article beyond its immediate focus, including biodiversity outside Indonesia and in the context of global open science. It also received a post-publication comment by a reader from Indonesia, who highlighted the value of the article in the local context (Irawan 2021).

Beyond those interactions taking place directly on the RIO website or via the ARPHA Writing Tool, discussion happens, of course, on many other channels, including social media where the community regularly tags us (thank you!) in discussions on open science matters, be they on unconventional publication types or peer review practices.

Recommending

In Lin and Fenner (2013), this is defined as “Activity of a user formally endorsing the research article (via a platform such as an online recommendations channel).” We currently do not have systematic engagement with any such platform, but we would be interested in exploring options to do this in an open and cross-disciplinary fashion.

Citing

The act of citing a resource from a research publication is meant to indicate the flow of information, so as to allow others to trace it back when trying to build on something reported previously, to question or reframe it or otherwise engage with it. This has given rise to a whole industry of citation-based research evaluation that is hard to ignore in any research context, but much of what RIO publishes does not fit neatly into the current research evaluation landscape.

In this section, we focus instead on the original information flow aspects of citations and consider two kinds of flows - within a research cycle and across fields - and illustrate them with RIO examples.

Flow of information within a research cycle - even if it is slow as in the case of “sleeping beauties'' (Philipson 2016) - implies some form of continuity. An example is a Project Report (Peterson 2017) - resulting from the Neurohackweek 2016 - that followed up on a previously-published Methods paper (Madan 2016), which provides a guide to generating a print-ready 3D model of brain structures. On another occasion, a Data Management Plan (Neylon 2017) followed the successful Grant Proposal that was published the previous year (Neylon and Chan 2016). Similarly, a Research Presentation (Senderov et al. 2016) - part of a PhD project aiming to build an Open Biodiversity Knowledge Management System (OBKMS) - can be traced back to the PhD Project Plan (Senderov and Penev 2016).

Within scientific projects, the progress of the research process - and even workshops - can also be conveniently tracked through a series of Workshop Reports, as in the case of the European Biodiversity Observation Network (EU BON), whose series of four stakeholder events were timely communicated one by one in Wetzel et al. (2016), Vohland et al. (2016a), Vohland et al. (2016b) and Wetzel et al. (2017).

Research published in RIO has naturally been reused and built upon in other scholarly sources. For instance, an article investigating the role of religion in attitudes and responses to the climate crisis in Nigeria (Nche 2020) cites a Research Article from RIO about the influence of religion on science (Romero Jr. 2016).

Interim and early outputs available from RIO have also served as stepping stones in unrelated studies published elsewhere. A paper outlining a framework for a reference database for images of marine taxa (Howell et al. 2019) noted that manual processing of images “forms the current bottleneck in image-based ecological sampling”, citing a RIO report (Schoening et al. 2017) on a workshop that had explored avenues for automation. A 2020 Report by the European Commission’s Joint Research Centre (JRC) (Joint Research Centre 2020) addressed the discoveries made possible with the help of citizen science over a period of 5 years and referenced a Workshop Report about “defining principles for mobile apps and platforms development in citizen science” (Sturm et al. 2018), as well as a RIO commentary (Schade et al. 2019) on the potential of combining low-cost air quality sensors with citizen science.

Besides continuity, RIO is interested in cross-fertilization between different parts of the research ecosystem. A RIO Research Article laying out a hypothesis about the neural basis of imagination (Vyshedskiy 2019) has been discussed from perspectives as diverse as connectomics (Changeux et al. 2021), marketing (Tanaka 2020) and linguistic philosophy (Lobo 2021). A Research Idea about a knowledge graph for biodiversity research (Page 2016) has helped inspire work on a similar knowledge graph for hydrogen research (Lin et al. 2020). Last but not least, the RIO article DNAqua-Net: Developing new genetic tools for bioassessment and monitoring of aquatic ecosystems in Europe (Leese et al. 2016) is likely one of the most cited Grant Proposals, with about 100 citations to date from various fields, including invasion biology (Rusch et al. 2020) and marine biotechnology (Rotter et al. 2021).

When aggregating data at the journal level, other perspectives on citations emerge. For instance, Fig. 2 illustrates that a non-RIO article on the Darwin Core standard for biodiversity data (Wieczorek et al. 2012) is cited from multiple RIO articles, for example, an NSF Grant Proposal (Franz et al. 2016), a Research Idea (Page 2016), a Project Report (Smirnova et al. 2016), a Policy Brief (Runnel et al. 2016), a Guidelines article (Penev et al. 2017) and a Forum Paper (Egloff et al. 2017).

Figure 2.  

Partial citation graph around the Darwin Core paper (Wieczorek et al. 2012), filtered for citations involving RIO papers. From Wikidata.

Other engagement mechanisms

Engagement with RIO content can go beyond the five mechanisms outlined by Lin and Fenner (2013). Here, we provide some examples. Some projects (e.g. Hartgerink and George (2015) or Prieß-Buchheit et al. (2020)) have mentioned directly in their proposal that they plan to publish the proposal in RIO, which is a strong indication that they are serious about the dissemination of the outcomes resulting from the project and probably a useful strategy at a time when funders are increasingly evaluating a prospective project’s impact, based on its communication strategy.

RIO content keeps finding new uses: for instance, some job ads have begun to link to the proposals that triggered the grants providing the funding for the advertised positions (example based on Steinbeck et al. 2020), while project websites have put the link to their grant proposal into the footer of their website, right next to information about the funder (for example, see website for the project described by Cavender-Bares et al. (2021)).

When authors add a reference to a RIO manuscript while drafting it through the ARPHA platform, the metadata of that reference is served to them via RefindIt. If it is not in there yet, authors should use the feature for entering the bibliographic information into the manuscript manually.

The Scholarly Publishing and Academic Resources Coalition found a unique way to engage with RIO by honouring it with its June 2016 SPARC Innovator Award (details).

In the HTML version of its articles, RIO provides a number of tabs, including article-level and citation metrics of the kind reported above, as well as ways to search for more publications by any of the authors (example).

Another form of engagement are updates. Authors can import any of their published RIO articles into the ARPHA drafting environment in order to create updates. For example, a Grant Proposal has been updated with the reviewer reports and information about the funding outcome (Hammond 2016) or a Data Management Plan with information about the datasets being stored (Pannell 2016).

Yet other ways to engage with RIO content are provided by collections.

RIO collections

As research progresses through the various steps within research cycles, collections provide a way to bundle the resulting outputs together, either within the same research cycle or across different ones (Table 4). Examples exist for projects, events and subject areas.

Table 4.

Overview of RIO collections, ordered by total views, with the last three rows representing totals across all collections, as well as averages per collection and per article in collections (traffic data as of 8 April 2021).

Collection title

Collection URL

Collection editor

Total PDF pages

Total views

Unique views

Number of articles

Building the European Biodiversity Observation Network (EU BON) Project Outcomes

https://riojournal.com/topical_collection/87/

Florian Wetzel, Lyubomir Penev

398

149832

66592

17

Exploring the opportunities and challenges of implementing open research strategies within development institutions: A project of the International Development Research Center

https://riojournal.com/topical_collection/69/

Cameron Neylon, Leslie Chan

168

89945

42967

17

Brainhack 2016 Project Reports

https://riojournal.com/topical_collection/71/

Jörg Pfannmöller, Cameron Craddock, Pierre Bellec, Daniel Margulies, Nolan Nichols

82

82344

42944

15

Public Data Management Plans created with the DMPTool

https://riojournal.com/topical_collection/67/

Jennifer McWhorter, Jennifer Pannell, Josh Fisher, Jeri Fey

73

65650

32085

9

Open Science

https://riojournal.com/topical_collection/73/

Leo Lahti

94

41732

20734

10

DNAqua-Net

https://riojournal.com/topical_collection/70/

Florian Leese

33

38522

10526

2

ICEDIG Project Outcomes

https://riojournal.com/topical_collection/84/

Laurence Livermore, Anne Koivunen, Kari Lahti and Leif Schulman

463

26735

11544

14

Open Biodiversity Knowledge Management System, PhD Project

https://riojournal.com/topical_collection/81/

Viktor Senderov

44

22480

11056

3

Observations, prevention and impact of COVID-19

https://riojournal.com/topical_collection/88/

Victor Padilla-Sanchez

47

15703

9084

6

Metadata 2020 Project Outputs

https://riojournal.com/topical_collection/86/

Laura Paglione, Ginny Hendricks

59

11971

6251

4

SYNTHESYS+ Project Outcomes

https://riojournal.com/topical_collection/83/

Laurence Livermore, Vince Smith, Katherine Dixey

125

11069

5369

4

Path2Integrity Project Outcomes

https://riojournal.com/topical_collection/85/

Julia Prieß-Buchheit, Oliver Claas, Iliyana Demirova, Agnieszka Dwojak-Matras, Lisa Häberlein, Belén López, Katharina Miller, Christiane Stock

66

5945

2588

3

Selected papers of the SAVE-SD 2016 workshop on “Semantics, Analytics, Visualisation: Enhancing Scholarly Data”

https://riojournal.com/topical_collection/68/

Silvio Peroni, Alejandra Gonzalez-Beltran, Francesco Osborne

12

5737

2450

1

COST Action SAGA

https://riojournal.com/topical_collection/82/

Carmen Cuenca-García

25

5267

2324

1

OpenCon 2016: Empowering the Next Generation to Advance Open Access, Open Education and Open Data

https://riojournal.com/topical_collection/72/

Joe McArthur

6

5103

2535

1

ReNature: Promoting research excellence in nature-based solutions for innovation, sustainable economic growth and human well-being in Malta

https://riojournal.com/topical_collection/89/

Anna Sapundzhieva, Mario Balzan

31

4247

2009

3

Political Psychology

https://riojournal.com/topical_collection/79/

Vincent Weidlich, Scott Bottorff, Jimmy Gustafsson, Kristian Åström, Gerhardt Fritzsche

15

371

284

1

At the time of writing, these 17 collections collectively contained a total of 111 articles (i.e. 37% of all RIO articles) that had received about 580000 total page views (67% of all RIO page views), of which about 270000 were unique views (57% of all unique views). Per collection, there was an average of 6.5 articles, with about 34000 total views and 16000 unique views. Per article in a collection, this translates to 5200 total page views and 2400 unique views, as compared to the RIO averages of 2900 total views and 1600 unique views. Inclusion in a collection thus correlates with higher traffic and, while the reasons for this are not entirely clear, we are keeping a watchful eye on collections as we further develop RIO’s sociotechnical framework.

RIO as a knowledge hub emphasizing teams, projects, communities, curation and collaboration

Over the last few years, the idea that the research cycle is worth sharing as a continuum, rather than as a scattering of standalone and supposedly final outcomes, has gained increased traction amongst researchers, research funders, research participants, research reusers and others who are involved in the development and cultivation of the research ecosystem (e.g. Burgelman et al. (2019)).

In particular, the pandemics caused by the Ebola, Zika and SARS-CoV-2 viruses have acted as catalysts for changes in scholarly communication practices worldwide that emphasize the early sharing of results, along with a more comprehensive sharing of associated data, code and other materials (e.g. Fraser et al. (2021)). We wholeheartedly welcome these developments and keep them in mind when adapting, refining or otherwise developing RIO’s workflows.

Key ingredients of these workflows are the various article types - especially those still widely considered unconventional - as well as the societal challenges addressed by the underlying research and the various and continuously evolving forms of community engagement around that.

For instance, websites about research projects usually link prominently to publications that resulted from the project. If such publications come out when the project is winding down or has ended, the potential for engagement by others is limited. If, on the other hand, these publications come out during earlier phases in the project’s life cycle (e.g. grant proposals, data management plans, early reports), this gives current and potential collaborators or students, as well as journalists and the public, detailed insights into the project’s activities, which provide an excellent basis for meaningful engagement and collaboration.

For an example, see this call for proposals by the SYNTHESYS+ project, which links prominently to a report they published in RIO about a previous such call for proposals (Hardy et al. 2020). Such continuity is facilitated by the project having a dedicated RIO collection, where that report can be found in the context of related outcomes.

Changes in RIO workflows

Taking into account our experience with RIO over the last five years and the insights and trends outlined above, we have made a number of changes to the way RIO operates. This section discusses the two main ones, which concern the scope of collections and the organization of peer review and briefly looks into a third - article types.

Collection management

With the evolving range of uses of RIO materials in mind, permanent article collections in RIO have recently been upgraded such that they can not only show content published in RIO, but also metadata of materials published elsewhere, all in a consistent design that can be configured by the collection editors (cf. Fig. 3).

Figure 3.  

A collection in RIO - for example for a topic, an event or a project - may include a diverse range of both traditional and unconventional research outputs, as well as links to publications from elsewhere (for details, see What can I publish on the RIO website).

This way, RIO collections can combine elements of traditional journal publishing (where the journal only publishes materials submitted to it) with elements of overlay journals (which pick some or all of their content from materials previously published elsewhere). This arrangement is not only interesting for projects, but also for events, organizations and communities centred around a specific topic or methodology. Apart from flexibility in terms of the source of the materials, RIO collections have also become more flexible in terms of the type of files they can accept: while articles published in RIO are natively available in the minable JATS XML, from which semantically enriched HTML and PDF versions can be generated, the inclusion of files published elsewhere into a RIO collection does not require those files to be available in XML or HTML (it can, in fact, be as easy as entering a DOI, which will then be used to fetch the relevant metadata). Thanks to the integration of the journal with the general-purpose open-access repository Zenodo, all items in a collection are automatically archived and indexed there, which further facilitates dissemination and citation.

In an example of a project collection, the EU-funded ICEDIG (Innovation and Consolidation for Large Scale Digitisation of Natural Heritage), led by several major natural history institutions, including the Natural History Museum of London, Naturalis Biodiversity Center, the French National Museum of Natural History and Helsinki University, brought together Policy Briefs, Project Reports, Research Articles and Review Papers, in order to provide a detailed overview of their own research continuum. As a result, future researchers and various stakeholders can easily piece together the key components within the project, in order to learn from, recreate or even build on the experience of ICEDIG.

Peer review

Operating with a wide range of publication types and outputs originating from different fields, RIO has made use of several separate peer review paths to accommodate the specificity of frequent-use cases aiming at strengthening the role of the community in both pre-and post-publication peer review. While it only makes sense that contributions, such as Research Articles and Review Articles, are subject to pre-publication review, the situation is different for several non-conventional research outputs, such as Grant Proposals, Workshop or Project Reports, Policy Briefs, PhD Theses or more traditional ones, such as Conference Materials, as these have often already been assessed by a relevant institution, funder, scientific conference or another legitimate organisation before submission to a journal.

We have now streamlined these workflows by more clearly specifying the two main types of manuscripts with regard to the peer-review process (see How it works section of the journal’s website for more detail):

  1. Research outcomes for which pre-publication peer review is required (for example, Research Ideas, Data Papers, Software Descriptions, Methods, Research and Review Articles) and
  2. Research outcomes that do not require pre-publication peer review and can be published upon a public author statement describing the quality checks and review the manuscript has passed before submission (e.g. various project deliverables); such author statements are published together with the article and are then also available via an article’s Review tab - see Raes et al. (2020) for an example.

Regardless of whether or not a submission warrants mandatory peer review or not, each manuscript is subject to editorial evaluation, in order to ensure that the content is sound and meets RIO’s standards. In addition to that, all published articles in RIO can be subject to voluntary post-publication peer review. All reviews in RIO have always been signed and published alongside the reviewed article (under the Review tab) and with their own DOI - see von Rintelen et al. (2017), for example.

For articles submitted for inclusion in RIO collections, the collection editors decide on the mode of peer review, taking guidance from RIO’s default policies.

What we have changed in the default policies is that RIO editors will not organize pre-publication peer reviews anymore. For manuscript types requiring pre-publication peer review, authors are requested to suggest suitable reviewers and these reviewers will be invited automatically by the RIO editorial management system, rather than at the discretion of RIO editors. Authors of such manuscripts can also choose to have their manuscript published as is, at present, on ARPHA Preprints, subject to editorial screening. At any time of the peer review process, however, the RIO editors will be able to invite additional reviewers independently.

Once the manuscript has received at least two positive reviews pre-publication (and fewer negative than positive ones) or an endorsement from an editor, the manuscript will be accepted for publication, unless editorial evaluation of the manuscript in the context of its reviews finds a mismatch with RIO policies (e.g. in terms of data availability).

For article types where pre-publication peer review is not mandatory, RIO will not offer pre-publication peer review anymore. Instead, unless editorial evaluation indicates otherwise, we will publish the manuscript as is and encourage post-publication reviews and comments. This is in line with broader trends to preprints in multiple disciplines and with “publish, then review” approaches being adopted by other journals as well (e.g. Eisen et al. (2020)), which also helps reduce the schism between the materials that researchers are asked to review versus those they are reading, based on their own activities and interests.

Article types

The article types already published in RIO show a great variety, but there are still more elements of the research process that could be shared more openly and RIO tries to facilitate that. We are thus exploring a range of potential new article types: a Registered Report could lay out the methodology for data yet to be acquired, an Ethics Management Plan article could outline the ethical approval process for a study, a Consent Form article could provide a blank version of a consent form used in a study, a Research Question article could zoom in on a single research question, while an Open Questions article could lay out a set of open questions in the context of a particular subject area and a Hypothesis article could be describing a single hypothesis, a Definition article a single definition, a Nanopublication article just a single factoid, a Call for Proposals article could invite funding proposals for a funding line or session proposals for an event, a Job Ad article could provide details for an open position and so on.

On the other hand, keeping information about these various - and often non-standard - article types in a structured format is not simple, so we are keeping an eye on how this could be streamlined further. This means that we are exploring mechanisms by which more generic article types - for example, for Grant Proposal articles - can be more readily adapted to specific use cases, for example, different funders or funding lines, in a way that is as much aligned with JATS best practices as possible.

Conclusion

The experience of these first 300 articles in RIO has demonstrated the multiple interconnected layers of actual or potential engagement through which these publications help enrich their respective research processes and lay a good foundation for reuse in research, education, sustainable development and beyond. Taken together, these articles cover a lot of ground and their aggregation highlights gaps and opportunities. Some of these have been addressed by new policies, some others need further attention. While this article is focused on the interaction between RIO and the research cycle in its many shades and forms, we are planning follow-ups that will situate RIO's efforts around publishing more of the research processes - and facilitating engagement with it - in the broader context of the evolving research landscape at large.

Acknowledgements

We would like to thank all authors, reviewers and editors of RIO manuscripts so far for providing the content on which this article is based and for their contribution to making the research landscape more open and collaborative. We would also like to thank the Advisory Board for guidance on strategic matters and Ivo Grigorov and Caterina Bergami for help with the pilot on granular tagging of articles pertaining to SDG14, Life under water.

References

Supplementary materials

Suppl. material 1: Examples by funder 
Authors:  Daniel Mietchen, Teodor Georgiev
Data type:  text/CSV
Brief description: 

Selection of RIO publications (as of 8 April 2021) arranged by funder, roughly sorted by whether their geographic scope is primarily international (top) or national (bottom).

Suppl. material 2: Topics covered 
Authors:  Daniel Mietchen, Teodor Georgiev
Data type:  text/CSV
Brief description: 

Topics covered in RIO as of 8 April 2021.

Suppl. material 3: Most viewed figures in RIO. 
Authors:  Daniel Mietchen, Teodor Georgiev
Data type:  text/CSV
Brief description: 

Most viewed figures in RIO as of 8 April 2021.

Suppl. material 4: Most viewed subfigures in RIO. 
Authors:  Daniel Mietchen, Teodor Georgiev
Data type:  text/CSV
Brief description: 

Most viewed subfigures in RIO as of 8 April 2021.

Suppl. material 5: Most viewed tables in RIO. 
Authors:  Daniel Mietchen, Teodor Georgiev
Data type:  text/CSV
Brief description: 

Most viewed tables in RIO as of 8 April 2021.

Suppl. material 6: Most downloaded figures in RIO. 
Authors:  Daniel Mietchen, Teodor Georgiev
Data type:  text/CSV
Brief description: 

Most downloaded figures in RIO as of 8 April 2021.

Suppl. material 7: Most downloaded subfigures in RIO. 
Authors:  Daniel Mietchen, Teodor Georgiev
Data type:  text/CSV
Brief description: 

Most downloaded subfigures in RIO as of 8 April 2021.

Suppl. material 8: Most downloaded tables in RIO. 
Authors:  Daniel Mietchen, Teodor Georgiev
Data type:  text/CSV
Brief description: 

Most downloaded tables in RIO as of 8 April 2021.

Suppl. material 9: Most viewed supplementary materials in RIO. 
Authors:  Daniel Mietchen, Teodor Georgiev
Data type:  text/CSV
Brief description: 

Most viewed supplementary materials in RIO as of 8 April 2021.

Suppl. material 10: Most downloaded supplementary materials in RIO. 
Authors:  Daniel Mietchen, Teodor Georgiev
Data type:  text/CSV
Brief description: 

Most downloaded supplementary materials in RIO as of 8 April 2021.