Open science in practice: 300 published research ideas and outcomes illustrate how RIO Journal facilitates engagement with the research process

Since Research Ideas and Outcomes was launched in late 2015, it has stimulated experimentation around the publication of and engagement with research processes, especially those with a strong open science component. Here, we zoom in on the first 300 RIO articles that have been published and elucidate how they relate to the different stages and variants of the research cycle, how they help address societal challenges and what forms of engagement have evolved around these resources, most of which have a nature and scope that would prevent them from entering the scholarly record via more traditional journals. Building on these observations, we describe some changes we recently introduced in the policies and peer review process at RIO to further facilitate engagement with the research process, including the establishment of an article collections feature that allows us to bring together research ideas and outcomes from within one research cycle or across multiple ones, irrespective of where they have been published. ‡ §,| ¶ ¶ ¶ © Mietchen D et al. This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.


Introduction
RIO's mission is to publish the research process, to facilitate engagement with both the process and its outcomes and to highlight how the research published this way relates to societal challenges (Mietchen et al. 2015). When it was launched, such framing was very unusual for a scholarly journal. Hence, RIO was designed with flexibility in mind, so as to allow for its technical and policy parameters to be adjusted and adapted on an ongoing basis.
Since then, the research landscape has been evolving and the importance of a wider sharing of the substantial and diverse bits and pieces underlying the various research processes is receiving broader attention. While there are some steady developments in this direction, much of the observable progress was triggered by disruptive events, such as the Ebola and Zika epidemics and now the ongoing COVID-19 pandemic. This resonates with the observation, in a review of Hurricane Katrina, that "Open data matters most when the stakes are high". At RIO, we agree, but think that open data and open processes matter even when the stakes are not high or not (yet) known to be high.
We also observe some concerning trends, from increased "openwashing" to the continued "consolidation" of the publishing industry, where so-called "transformative agreements", "read and publish" and other formats of large corporate deals between traditional publishers and well-funded research consortia eat up resources that could have instead been used to actually improve the research landscape. At RIO, we will continue to emphasise innovation for the benefit of the research ecosystem, rather than just a few individual players.
In this editorial, we explore some of the key developments in RIO over the last few years, how they relate to societal challenges and how RIO can continue to stimulate experimentation in this space by launching exciting new features and opportunities for researchers, projects, institutions, funders and readers.

Overview of research published in RIO so far
There are various ways in which the content of RIO can be grouped. Here, we will look primarily at two of RIO's key unique features: distribution by the various article typeswhich roughly correspond to different stages of the research cycle -as well as at the Sustainable Development Goals, to which RIO articles are mapped routinely. On that basis, we will highlight various ways in which RIO readers engage with RIO content and explore how these publications can serve as a resource for research projects.

Article types
As of 31 March 2021, RIO had published 300 articles. Of these, 32 (i.e. 11%) were traditional publication types from the end of a research cycle as published in most scholarly journals, while 132 (44%) were from early stages that rarely get published elsewhere and 136 (45%) from intermediate stages of the research cycle whose coverage in the scholarly record has traditionally been patchy (Table 1). Distribution of the first 300 RIO articles by research cycle stages: early in the research cycle (132), intermediate outcomes (136) and final outcomes (32). The "Grant proposals" row groups together all article types for grant proposals, including the two generic ones (Grant Proposal and Small Grant Proposal) and six funder-specific ones (for COST, Horizon 2020, NSF, NIH, FWF and DFG). The "Other" row groups all article types with just one example in RIO (2020)).

Sustainable Development Goals
Besides opening up the research process, RIO emphasises the connection between research and societal challenges, in particular, by mapping its articles to the United Nations' Sustainable Development Goals (SDGs, see Griggs et al. (2013) and Lu et al. (2015)).
While papers published in RIO have addressed each of the 17 SDGs, some goals remain quite under-represented: five of the goals have 50 or more RIO articles to them, six of the goals have 10 or less (  While RIO uses English as its default language, we are aware that this is a barrier for some to engage with its content. Since all RIO content is openly licensed, anyone is free to translate any part of it into any languages they are interested in and this we encourage. We also support the publication of multilingual content, as long as an English version is available.

Data management
Data management is central to contemporary research, open or not and, thus, important to RIO already from the perspective of improving scholarly practices, as discussed in the previous paragraph. Our emphasis on engagement with the research process and the various steps along the research cycle raise the importance of data management higher still, since some aspects of data management are important at every step.
It is thus not surprising that a full-text search for "data management" in RIO articles currently yields 210 results, i.e. 70% of the 300 articles. Furthermore, data management plays an increasing role in Grant Proposals on any subject (see, for example, Work Package 1 in Vanderhoeven et al. (2017)), and it is in the focus of a growing number of projects, especially infrastructural ones (e.g. , Steinbeck et al. (2020) or Altenhöner et al. (2020)). It is also the subject of project reports (e.g. Borghi et al. (2018)), workshops and, thus, workshop reports (e.g. , Van Tuyl and Whitmire (2018) or Petersen et al. (2020)), guidelines (e.g. Briney et al. (2020)) and review articles (e.g. Neylon (2017)).

Citizen science
Open science and citizen science have originated largely independently from each other, one emphasizing issues of reproducibility and transparency of research workflows, the other stimulating contributions to research workflows from outside the classical research ecosystem. Over time, they have evolved to interact in various fashions, and RIO is one of the contexts in which they do so.

SARS-CoV-2 and COVID-19
The ongoing COVID-19 pandemic has triggered a notable shift in the wider research landscape towards more open and more rapid sharing, thereby improving the alignment with RIO. Examples of RIO publications on the virus, the disease or the pandemic can be found in a dedicated collection, which includes Research Ideas (Senapati et al. 2020, Sanal andDubey 2020) and Grant Proposals (Mietchen and Li 2020), as well as Research Articles (Padilla-Sanchez 2020, Padilla-Sanchez 2021) and Review Articles (Alibek and Tskhay 2020).

Engagement with RIO content
A very basic form of engagement with a journal's content is to access, browse, explore and read that content. Further forms of engagement may involve bookmarking, sharing, reviewing, annotating, commenting on, building on, reproducing, using or reusing the content. Lin and Fenner (2013) discussed various mechanisms of engagement with research articles and distinguished between articles being viewed, saved, discussed, recommended or cited. In this section, we explore these potential routes of engagement, focusing on RIO content, but expanding the scope beyond Research Articles and beyond those five basic categories of engagement.
All RIO articles and all of their components -for example, individual figures, tables or supplementary materials -are available via both the RIO website and Zenodo. Traffic via Zenodo can be substantial, for example, the Zenodo copy of the Software Management Plan "Gentoo Linux for Neuroscience -a replicable, flexible, scalable, rolling-release environment that provides direct access to development software" (Ioanas et al. 2017) currently has 1522 unique views and 1619 total views, as well as 39 unique downloads and 41 total downloads. Nonetheless, we focus here on traffic via the RIO website, which is provided via each article's Metrics tab.

Journal-level views
Traffic at the journal level over a given time period depends on a number of criteria, including the amount of its content, the degree to which the content matches interest in the group of potential readers during that timeframe, as well as findability and accessibility.
For RIO, journal-level traffic data has been in the order of just above 100,000 unique page views in 2020, at an average annual growth rate of about 20,000, for which 2021 is on track (cf. Fig. 1). This sums up to about 340,000 unique page views, in total, until now. Of these, about 100,000 (26%) are due to traditional publications from the final stage of the research cycle (Research Articles and Review Articles), while 74% of RIO's traffic is due to articles about early and intermediate steps. RIO tracks views to the HTML, XML and PDF versions of its articles and distinguishes between unique views and total views, with the latter including repeat visits. The unique HTML, PDF and XML views represent about 58%, 37% and 5% of the unique views, respectively.  Table 3.

Rank Unique views
Articles with the most unique views as of 30 March 2021, along with information about total views, the ratio between total and unique views (τ) and the article type.

Article-level views
For each article, RIO records the number of first-time (unique) and repeat visits (as identified via cookies) to the RIO website, as well as total views (i.e. the sum of unique and repeat visits) and makes these data available via an article's Metrics tab. A list of the ten articles with the highest number of unique views as of 30 March 2021 is given in Table 3. Repeat visits are a basic form of engagement with the content, so the total number of views per article is also indicated in Table 3, as well as the ratio between both numbers (Total versus Unique ratio, τ).
Inspecting the data in Table 3 from the perspective of the research stages defined for Table  1 gives rise to a number of observations. First, publication types representing the early (ranks 6, 7, 9 and, 10), late (1, 2 and 8) and intermediate stages (3, 4 and 5) of the research cycle as per Table 1 are all represented. Second, the three late-stage entries in this list of ten is more than would be expected, based on the 11% prevalence of late-stage publications in the overall RIO corpus, but very close to the 29% unique visits going to latestage articles, as discussed along Fig. 1. Third, τ is highest for early-stage entries in the  The most viewed table -

Saving
Materials published in RIO can be saved in various ways. For instance, an article can be included in a RIO collection, the article or any of its components can be downloaded from RIO or Zenodo, the URL of an article or any of its components can be bookmarked, or the metadata of any of these can be included in reference managers. In the following section, we will concentrate on downloads via the RIO website.

Sub-article-level downloads
Beyond whole-article traffic, more granular content types also experience significant download activity across the research cycle, for example, Fig. 2  It is interesting to note that the most viewed and the most downloaded files typically differ. Similarly, the most viewed table received 375 total views, while dozens of tables have had more downloads than that, which indicates that users prefer to view tables offline (this preference is not news to us, but provided one of the major reasons why all tables in RIO and its sister journals can be downloaded as spreadsheets).
Suppl. material 7 provides an overview of the most downloaded subfigures. Of the two articles leading the subfigure view stats (Suppl. material 4), one of them -Marek (2017)also has two of its subfigures leading the subfigure download stats, while the other article has none of its subfigures in the top 20 for downloads. However, similar to the observations about views, the 20th-most downloaded subfigure has 84% of the downloads of the most downloaded subfigure, which again suggests that readers approach subfigures more systematically than selectively. This idea receives further support from the ratio of article downloads versus subfigure downloads, which is between 3.4 and 7.8 for all of these 20 most downloaded subfigures.
Just as for views, supplementary files receive less attention than main-text ones in terms of downloads: the top 20 most downloaded main-text figures (Suppl. material 6) all have more downloads than the most downloaded supplementary file (cf. Suppl. material 10). This is Supplementary Material 2 of Cole (2018) (a PhD thesis) -a PDF containing a list of studies that were included in the meta-analysis as well as of "near-miss" studies that were "potentially relevant [..] but ultimately excluded" and whose description also states "Future researchers might find this list especially valuable.

Resources for projects
With more of the research process becoming visible through publications at all stages of the research cycle, these publications become resources in and of themselves. While not all aspects of research (e.g. the development and provision of infrastructure) can be usefully cast in terms of projects, some key elements of project management are relevant across disciplines. Their traffic statistics indicate that sharing them is of particular interest to RIO readers, so we highlight some examples here, thereby complementing the projectrelated examples given earlier, especially in the section on data management above.
For instance, PERT charts explain the relationships between different components of a project and examples can be found in Fig. 1  SWOT analyses highlight strengths and weaknesses of, as well as opportunities for and threats to a project, and examples can be found in Fig. 11 of Hardisty et al. (2020) and in Table 2 of Altenhöner et al. (2020).
Risk assessments zoom in on potential threats, estimate their likelihood, assess their potential effects on the course of the project and outline mitigation measures. Examples can be found in Table 12 of Mietchen et al. (2015b) and in Table 4 of Mariani (2018), as well as in Tables 2-7  Budgets are a key element of project planning and management too and examples can be found in Supplementary Table 1 of Neittaanmäki et al. (2016), as well as in Table 1 of , in Table 2 of ElSabry (2017), in Table 13 of Mietchen et al. (2015b) or in  Table 1 of .

Discussing
At RIO, one way to initiate discussion of an article is to invite others into the ARPHA drafting environment , where they can be given the rights to review, comment on or edit the draft before submission. This can then be complemented by postpublication reviews and annotations (see Changes in RIO workflows). The nature of the publications in RIO also brings to light new layers of discussion around individual steps and outcomes of the research process. In this section, we thus provide some examples from different settings.

The Research Idea An oral live attenuated vaccine strategy against Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2/2019-nCoV) (Sanal and Dubey 2020)
has received three pre-submission peer reviews (Perumal 2020, Alfieri 2021 and Panicker 2021) providing constructive feedback from multiple perspectives, in addition to further context to the idea presented. External feedback at this stage -when the research has not been performed and can thus still be influenced -is very important, yet rarely possible in the current research ecosystem. Another Research Idea received two post-publication reviews outlining problems with the idea presented (for one of these reviews, see Susi 2018 Beyond those interactions taking place directly on the RIO website or via the ARPHA Writing Tool, discussion happens, of course, on many other channels, including social media where the community regularly tags us (thank you!) in discussions on open science matters, be they on unconventional publication types or peer review practices.

Recommending
In Lin and Fenner (2013), this is defined as "Activity of a user formally endorsing the research article (via a platform such as an online recommendations channel)." We currently do not have systematic engagement with any such platform, but we would be interested in exploring options to do this in an open and cross-disciplinary fashion.

Citing
The act of citing a resource from a research publication is meant to indicate the flow of information, so as to allow others to trace it back when trying to build on something reported previously, to question or reframe it or otherwise engage with it. This has given rise to a whole industry of citation-based research evaluation that is hard to ignore in any research context, but much of what RIO publishes does not fit neatly into the current research evaluation landscape.
In this section, we focus instead on the original information flow aspects of citations and consider two kinds of flows -within a research cycle and across fields -and illustrate them with RIO examples.

Other engagement mechanisms
Engagement with RIO content can go beyond the five mechanisms outlined by Lin and Fenner (2013). Here, we provide some examples. Some projects (e.g. Hartgerink and George (2015) or Prieß-Buchheit et al. (2020)) have mentioned directly in their proposal that they plan to publish the proposal in RIO, which is a strong indication that they are serious about the dissemination of the outcomes resulting from the project and probably a useful strategy at a time when funders are increasingly evaluating a prospective project's impact, based on its communication strategy.
RIO content keeps finding new uses: for instance, some job ads have begun to link to the proposals that triggered the grants providing the funding for the advertised positions (exam ple based on Steinbeck et al. 2020), while project websites have put the link to their grant proposal into the footer of their website, right next to information about the funder (for example, see website for the project described by Cavender-Bares et al. (2021)).
When authors add a reference to a RIO manuscript while drafting it through the ARPHA platform, the metadata of that reference is served to them via RefindIt. If it is not in there yet, authors should use the feature for entering the bibliographic information into the manuscript manually.
The Scholarly Publishing and Academic Resources Coalition found a unique way to engage with RIO by honouring it with its June 2016 SPARC Innovator Award (details).
In the HTML version of its articles, RIO provides a number of tabs, including article-level and citation metrics of the kind reported above, as well as ways to search for more publications by any of the authors (example). Yet other ways to engage with RIO content are provided by collections.

RIO collections
As research progresses through the various steps within research cycles, collections provide a way to bundle the resulting outputs together, either within the same research cycle or across different ones (Table 4). Examples exist for projects, events and subject areas.  RIO as a knowledge hub emphasizing teams, projects,

communities, curation and collaboration
Over the last few years, the idea that the research cycle is worth sharing as a continuum, rather than as a scattering of standalone and supposedly final outcomes, has gained increased traction amongst researchers, research funders, research participants, research reusers and others who are involved in the development and cultivation of the research ecosystem (e.g. Burgelman et al. (2019)).
In particular, the pandemics caused by the Ebola, Zika and SARS-CoV-2 viruses have acted as catalysts for changes in scholarly communication practices worldwide that emphasize the early sharing of results, along with a more comprehensive sharing of associated data, code and other materials (e.g. Fraser et al. (2021)). We wholeheartedly welcome these developments and keep them in mind when adapting, refining or otherwise developing RIO's workflows.
Key ingredients of these workflows are the various article types -especially those still widely considered unconventional -as well as the societal challenges addressed by the underlying research and the various and continuously evolving forms of community engagement around that.
For instance, websites about research projects usually link prominently to publications that resulted from the project. If such publications come out when the project is winding down or has ended, the potential for engagement by others is limited. If, on the other hand, these publications come out during earlier phases in the project's life cycle (e.g. grant proposals, data management plans, early reports), this gives current and potential collaborators or students, as well as journalists and the public, detailed insights into the project's activities, which provide an excellent basis for meaningful engagement and collaboration.
For an example, see this call for proposals by the SYNTHESYS+ project, which links prominently to a report they published in RIO about a previous such call for proposals (Hardy et al. 2020). Such continuity is facilitated by the project having a dedicated RIO collection, where that report can be found in the context of related outcomes.

Changes in RIO workflows
Taking into account our experience with RIO over the last five years and the insights and trends outlined above, we have made a number of changes to the way RIO operates. This section discusses the two main ones, which concern the scope of collections and the organization of peer review and briefly looks into a third -article types.

Collection management
With the evolving range of uses of RIO materials in mind, permanent article collections in RIO have recently been upgraded such that they can not only show content published in RIO, but also metadata of materials published elsewhere, all in a consistent design that can be configured by the collection editors (cf. Fig. 3).

Figure 3.
A collection in RIO -for example for a topic, an event or a project -may include a diverse range of both traditional and unconventional research outputs, as well as links to publications from elsewhere (for details, see What can I publish on the RIO website).
This way, RIO collections can combine elements of traditional journal publishing (where the journal only publishes materials submitted to it) with elements of overlay journals (which pick some or all of their content from materials previously published elsewhere). This arrangement is not only interesting for projects, but also for events, organizations and communities centred around a specific topic or methodology. Apart from flexibility in terms of the source of the materials, RIO collections have also become more flexible in terms of the type of files they can accept: while articles published in RIO are natively available in the minable JATS XML, from which semantically enriched HTML and PDF versions can be generated, the inclusion of files published elsewhere into a RIO collection does not require those files to be available in XML or HTML (it can, in fact, be as easy as entering a DOI, which will then be used to fetch the relevant metadata). Thanks to the integration of the journal with the general-purpose open-access repository Zenodo, all items in a collection are automatically archived and indexed there, which further facilitates dissemination and citation.
In an example of a project collection, the EU-funded ICEDIG (Innovation and Consolidation for Large Scale Digitisation of Natural Heritage), led by several major natural history institutions, including the Natural History Museum of London, Naturalis Biodiversity Center, the French National Museum of Natural History and Helsinki University, brought together Policy Briefs, Project Reports, Research Articles and Review Papers, in order to provide a detailed overview of their own research continuum. As a result, future researchers and various stakeholders can easily piece together the key components within the project, in order to learn from, recreate or even build on the experience of ICEDIG.

Peer review
Operating with a wide range of publication types and outputs originating from different fields, RIO has made use of several separate peer review paths to accommodate the specificity of frequent-use cases aiming at strengthening the role of the community in both pre-and post-publication peer review. While it only makes sense that contributions, such as Research Articles and Review Articles, are subject to pre-publication review, the situation is different for several non-conventional research outputs, such as Grant Proposals, Workshop or Project Reports, Policy Briefs, PhD Theses or more traditional ones, such as Conference Materials, as these have often already been assessed by a relevant institution, funder, scientific conference or another legitimate organisation before submission to a journal.
We have now streamlined these workflows by more clearly specifying the two main types of manuscripts with regard to the peer-review process (see How it works section of the journal's website for more detail): 1.
Research outcomes for which pre-publication peer review is required (for example, Research Ideas, Data Papers, Software Descriptions, Methods, Research and Review Articles) and 2.
Research outcomes that do not require pre-publication peer review and can be published upon a public author statement describing the quality checks and review the manuscript has passed before submission (e.g. various project deliverables); such author statements are published together with the article and are then also available via an article's Review tab -see Raes et al. (2020) for an example.
Regardless of whether or not a submission warrants mandatory peer review or not, each manuscript is subject to editorial evaluation, in order to ensure that the content is sound and meets RIO's standards. In addition to that, all published articles in RIO can be subject to voluntary post-publication peer review. All reviews in RIO have always been signed and published alongside the reviewed article (under the Review tab) and with their own DOIsee von Rintelen et al. (2017), for example.
For articles submitted for inclusion in RIO collections, the collection editors decide on the mode of peer review, taking guidance from RIO's default policies.
What we have changed in the default policies is that RIO editors will not organize prepublication peer reviews anymore. For manuscript types requiring pre-publication peer review, authors are requested to suggest suitable reviewers and these reviewers will be invited automatically by the RIO editorial management system, rather than at the discretion of RIO editors. Authors of such manuscripts can also choose to have their manuscript published as is, at present, on ARPHA Preprints, subject to editorial screening. At any time of the peer review process, however, the RIO editors will be able to invite additional reviewers independently.
Once the manuscript has received at least two positive reviews pre-publication (and fewer negative than positive ones) or an endorsement from an editor, the manuscript will be accepted for publication, unless editorial evaluation of the manuscript in the context of its reviews finds a mismatch with RIO policies (e.g. in terms of data availability).
For article types where pre-publication peer review is not mandatory, RIO will not offer prepublication peer review anymore. Instead, unless editorial evaluation indicates otherwise, we will publish the manuscript as is and encourage post-publication reviews and comments. This is in line with broader trends to preprints in multiple disciplines and with "publish, then review" approaches being adopted by other journals as well (e.g. Eisen et al. (2020)), which also helps reduce the schism between the materials that researchers are asked to review versus those they are reading, based on their own activities and interests.

Article types
The article types already published in RIO show a great variety, but there are still more elements of the research process that could be shared more openly and RIO tries to facilitate that. We are thus exploring a range of potential new article types: a Registered Report could lay out the methodology for data yet to be acquired, an Ethics Management Plan article could outline the ethical approval process for a study, a Consent Form article could provide a blank version of a consent form used in a study, a Research Question article could zoom in on a single research question, while an Open Questions article could lay out a set of open questions in the context of a particular subject area and a Hypothesis article could be describing a single hypothesis, a Definition article a single definition, a Nanopublication article just a single factoid, a Call for Proposals article could invite funding proposals for a funding line or session proposals for an event, a Job Ad article could provide details for an open position and so on.
On the other hand, keeping information about these various -and often non-standardarticle types in a structured format is not simple, so we are keeping an eye on how this could be streamlined further. This means that we are exploring mechanisms by which more generic article types -for example, for Grant Proposal articles -can be more readily adapted to specific use cases, for example, different funders or funding lines, in a way that is as much aligned with JATS best practices as possible.

Conclusion
The experience of these first 300 articles in RIO has demonstrated the multiple interconnected layers of actual or potential engagement through which these publications help enrich their respective research processes and lay a good foundation for reuse in research, education, sustainable development and beyond. Taken together, these articles cover a lot of ground and their aggregation highlights gaps and opportunities. Some of these have been addressed by new policies, some others need further attention. While this article is focused on the interaction between RIO and the research cycle in its many shades and forms, we are planning follow-ups that will situate RIO's efforts around publishing more of the research processes -and facilitating engagement with it -in the broader context of the evolving research landscape at large. Mietchen D et al