Open infrastructure governance, sustainability, and insurance principles (Bilder et al. 2015) are critical for building trust in new lines of credit by ensuring the transparency and availability of data that supports research claims.  The FAIR principles (Wilkinson et al. 2016) build upon these principles and focus on the ability of machines to automatically find and use research data, and support its reuse by individuals. 

In addition to the technical and services component, we also need to ensure that research communities are integral to the change process.  There are many examples of community engagement in this space: development of the CRediT taxonomy (Allen et al. 2019), the DORA Initiative, adoption of ORCID at the national level (e.g., Simons 2015), MetaData 2020 working groups, and organizations such as the Research Data Alliance, to mention but a few.  It is not necessary, nor is it advisable,  to have one organization solely responsible for driving new credit models. Coordination efforts across stakeholder groups to spur the iterative development of the expanded credit model are an essential design element.

While promoting trust in findability, FAIR principles do not fully meet the credit needs of researchers and communities.  This is illustrated, for example, by the general lack of source- and person-credit fields in many data repositories (e.g., see Krznarich 2019).  Source metadata is particularly important for Indigenous Peoples, who must be able to assert control over the application and use of Indigenous data and Indigenous knowledge for collective benefit (United Nations General Assembly 2007). 

To address these needs, the CARE principles (Global Indigenous Data Alliance 2019) have been developed. CARE principles reflect the crucial role of people and purpose in building community trust and participation in the new research credit economy and provide a template for participation by other communities such as research facilities (ORCID 2017) and collection curators.  The Tribal Knowledge and Biocultural labels developed by Local Contexts coupled with personas developed in the Metadata2020 project and the Educopia Values and Principles Checklist (Skinner and Lippincott 2020) provide additional bridges between researchers, data, creators, communities, and curators.

Assurance standards are a component of trust building.  FAIR and CARE get at findability and appropriate use.  We also need transparency in the design principles of the new credit economy.  To instantiate trust, we need to know more than a node or edge on a graph.   ORCID has done some work in this area, examining how assertions (connections between an ORCID ID, a work item, and the organization(s) hosting/funding/resourcing that work) are made into the ORCID registry and classifying assurance standards based on source transparency and traceability (Peters 2018).  

In addition, stakeholders need to be involved in developing the metrics of contributions, sharing and collaborating, and in the analysis of the data.  Data models, inputs, pre-processing steps, attribution, and de-identification methods must be transparent, while also respecting privacy (Lane et al. 2014). Credit units applied to diverse project goals and disciplines must be normalized if they are to serve in assessing researchers’ subsequent funding and career advancement. This is no easy task, but there are examples of successful measurement frameworks (e.g., Basner et al. 2013).  Critically, at these early stages of reinventing the research process, stakeholders can use their sticks (policy mandates) and carrots (resources and rewards) and must ensure that researchers are integral partners in the measurement system, helping to design and test tools and platforms that capture open and collaborative behaviors. 

Bringing together governance, ethics, and provenance, we can develop transparent and trusted methods to track use of collaboration technologies and then drive adoption of a new credit economy, becoming part of the assessment frameworks being used by funders and tenure and promotion committees to be truly effective. We are already part-way there:  CRediT roles, persistent identifiers, CARE and FAIR principles are already in use and, if used in concert, provide an effective means to tie together components of the research lifecycle and measure, at least in the first iteration, what is working and what is not.  

An open identifier infrastructure has been developing over the last 20 years (Haak et al. 2012) that is providing the underlayment for the credit revolution.  Infrastructure services have enabled clear identification of the people and, increasingly, organizations involved in driving research, as well as the papers, datasets, and resources associated with research activities.  Embedding persistent identifiers into standard research workflows is making it possible to not only identify but also connect components within and across the research lifecycle (e.g., Fenner 2020).  From these connections graphs (Fenner and Aryani 2020) can be derived, showing associations between research activity components.  Grouping activities using project identifiers adds context for evaluating research impact (Haak et al. 2018) and innovation drivers (Glennon et al. 2018), as well as for tracking efforts to improve research processes we propose in this article.

Before research is communicated publicly its component parts need to be logged, shared with trusted colleagues, and stored in a way that makes them discoverable and persistent. There are many repositories, tools, and sites for sharing and storing datasets and other research outputs and, while use of these has been slowly increasing, most outputs remain scattered across various local drives or isolated cloud storage. Researchers don’t typically use the available reliable third party repositories for data, code, and other research outputs such as protocols and resources. While some funders and institutions have policies on open data (and other outputs), many have voiced frustration that it is difficult to track compliance. There aren’t clear pathways to using these third party repositories and no way to check and monitor their use by either funders or institutions.

Because there is no persistent record of when and where datasets and other outputs have been shared or reused by others, credit cannot be given for all of the work done by researchers and more nuanced measures of impact are not possible. These outputs are generally not tied to preprints, journal articles, or future funding proposals so are not contributing to the complete communication of scholarship, the reproducibility of the work, or the reputation of those who worked hard to produce them. If the code used to analyze a dataset is not shared alongside the dataset, for example, that analysis cannot be verified and the person who designed the software is not given credit for the work. 

The Research Output Management System (ROMS), a project initiated by Stratos and undertaken by Aligning Science Across Parkinson’s (ASAP), is a demonstration of our proposed design principles: extending the attribution of credit throughout the research lifecycle, with services for storing, preserving, and monitoring research outputs.  As a living, dynamic tool with automation built in, the ROMS operationalizes connections between interrelated open source components to support a living representation of research workflows, beginning at the start of a funded project and carrying through to publication and beyond. Through the use of identifiers and open APIs, information sharing and metrics collection can be semi-automated, and sharing permissions managed as a component of the project. 

The ROMS is currently being built by ASAP as an open source tool that can be adopted by others, including funders and institutions. Because it logs all research outputs, with persistent identifiers and accurate metadata, it can serve multiple functions, including helping researchers share their work in a consistent, discoverable, and minable way and offering funders insight into the full impact of their funding programs. 

Journals can drive research reproducibility with tools such as Stencila, a platform for embedding live code and datasets to a manuscript.  Used to create eLife’s recently announced Executable Research Article (ERA), preprints and journal articles can be ‘born reproducible’ with authors demonstrating how their data and code work through the preprint or article itself. Recent scale implementation of executable research articles (Tsang and Maciocci 2020) demonstrates the feasibility of this concept.  Beyond linking related resources to a published article, ERA functionality allows authors to easily incorporate datasets, code, and protocols into their manuscript, without possessing coding knowledge.

The Facilitated Living Review (FLR) is a process developed by Rapid Science to encourage and enable researchers from different disciplines to encounter each other’s ideas and latest findings in a setting not unlike a journal club.  The FLR incorporates a curatorial service that interprets these insights and findings as they relate to the latest topically relevant published evidence.  

This process addresses impediments to collaborative and open research.  First, there are few opportunities in team-based initiatives for researchers to gather and informally discuss their work, as occurs at conferences or departmental journal clubs when new evidence is published.  Second, early, incremental, and null findings are rarely posted openly to the research community because of time constraints, lack of context, fear of being wrong or being scooped, and the absence of incentives/rewards. And, finally, Incremental findings are generally not subjected to peer review or oversight by peers, and yet entire projects and subsequent publications are built upon them

The FLR addresses each of these. The process is managed by an Editorial Facilitator (EF), a subject matter expert with editing expertise, who writes and maintains the review, updating it continually when a report of new evidence is published. Team members who are expert on that topic are called in to debate/annotate/revise the positioning of the evidence, based on how their work supports or challenges the findings. Incremental findings such as a dataset or null results, organically peer reviewed by the team members, can be cited in the FLR and shared external to the team simultaneously. 

The FLR attributes credit to team members who were involved in producing the review and to investigators whose early findings are incorporated in the review. This leverages an existing credit standard – that of citation – with new reward metrics for the collaborative behaviors leading up to open dissemination of the FLR.  Once shared, ongoing feedback keeps the FLR alive, incorporates new findings, and informs the subsequent versions, and shown in Fig. 2.

Figure 2.  

Collaborative Workflow of an Incremental Dataset in a Consortium Setting (click to view enlarged slide in Present mode). The workflow of an investigator’s incremental dataset is shown as it is incorporated into the Facilitated Living Review (FLR), moving along a continuum from closed to open review. (1) After ideation and hypothesis formation by the team, early experimentation creates an incremental dataset.  (2) The dataset is shared and discussed with the Project X research consortium, then iterated.  (3) The v.3 dataset is shared with the consortium for further analysis,  positioning, and citation in the context of the latest published evidence in the FLR.  (4) The FLR with its cited, organically peer reviewed dataset is posted on a preprint server under the authorship of the FLR consortium. (5) Feedback from the broader community may lead to further iteration of the dataset at the discretion of the project team in the next round of the FLR revisions.

Change requires actively rewarding the behaviors that we want to see. Attribution extended to the full range of outputs across the research lifecycle must lead to more granular tracking of activities and outputs, large and small. Sophistication in digital technology and design, combined with increasing familiarity and use of these technologies by researchers, makes it possible to track participants’ contributions both qualitatively and quantitatively. Activities such as sharing findings and insights can be logged for each individual, as can peer reviewing, replicating, co-authoring, data curation and analysis, and posting incremental and negative results to open access repositories and journals.  

Application of these principles is demonstrated in the ResCognito platform, which utilizes an extended attribution label taxonomy, persistent identifiers, and associated open digital infrastructure, to uniquely identify researchers and their contributions and enable community acknowledgement and curation of contributions.  The platform can also incorporate checklists, which can help researchers better understand and act on research citizenship standards. 

Taking this a step farther is the C-Score, a combined metric proposed by Rapid Science that captures collaborative activities on a project platform and aggregates them into a composite score.  Weighting of activities can be determined by the team, the project sponsor, and/or other relevant stakeholders. For instance, collaborative activities may include sharing results widely; robust discussions; reviewing work;  starting and moderating special-topic groups; or downloading, liking, bookmarking and other social media actions, with quantification levels determined in the project team plan.  In the course of the project, a team member could access their accumulating score directly and within the context of group contributions through leaderboards.

The C-score is not intended to serve as a measure of the team’s output or impact – that is, it would not displace quantitative or qualitative measures of disseminated results.  Rather, it offers funders and other adjudicators of large projects a transparent means of assessing research citizenship in equal measure with output.  Accordingly, it will be critical for research stakeholders to define and prioritize activities to be scored at project launch, and signal which behaviors are highly regarded, prioritized, and factored into future rounds of funding.  The National Institutes of Health have been experimenting with such “defined up-front” metrics models for large team science projects with some success (Basner et al. 2013).

Given that the publish-or-perish mentality and the accompanying reward system of publishing metrics are  responsible for intense competition and lack of sharing research results – why introduce yet another metric?  Is competing to collaborate a solution to a problem or will it amplify the drawbacks that currently plague the scientific enterprise?  Does the accompanying transparency of the collaboration metric system described above improve the validity of the investigator’s work?  Why not simply track and archive contributions without introducing yet another form of competition via a metric?

Identifying goals for team development and project success at the outset leads to a new paradigm for scientific investigation: competing to contribute to the team goal rather than restricting aims to individual or lab goals. This approach amplifies collaboration, advancing the team’s objectives while building reputation based on one’s effectiveness in sharing data and insights. Thinking of competition in this context turns it from a negative to a positive. 

This paradigm converts “research excellence” from a zero-sum game based on one gold standard metric, to an inclusive game that fosters diversity, participation, and sharing. The best competitors respect their opponents because they perceive at some level that cooperation and competitiveness are not zero-sum operations. For example, sports such as baseball and soccer are popular because of the duality of competition and cooperation occurring among participants.  Rewards, or performance evaluations, are based not only on individual output but on how well that person improved their group’s performance.  Robert Merton described “competitive cooperation” in 1942, referring to scientists as “compeers” (Merton 1942), emphasizing that the interplay of these conflicting modes of interaction “in pursuit of knowledge and other rewards” can elicit highly effective results (Nickelsen and Krämer 2016).

There are many examples of successful programs that involve teams competing to solve scientific problems, such as those sponsored by the XPrize Foundation and Sage Bionetworks’ DREAM challenges (Boutros et al. 2014).  Similarly, a workshop on rescuing biomedical research highlighted how “competition strengthens research, but hypercompetition weakens it” (Kimble et al. 2015).  Competition can be used not only to spur innovation but also to promote collaboration, rendering the traditional meaning of competition obsolete.