Objective 1 - CONCEPTUALISE: Create the conceptual & evidential framework for the project

Key activities

• Establish the state-of-play by defining reproducibility across diverse epistemic contexts & inventorying tools/practices & evidence of uptake & efficacy
• Identify gaps in current knowledge & key tools/practices
• Design the TIER2 framework for assessing reproducibility impact pathways

Key results

• Scoping report on state-of-the-art regarding evidence/uptake of reproducibility interventions
• Detailed inventory of reproducibility tools/practices and their current uptake
• Framework for assessing reproducibility impact pathways across epistemic contexts

Relation to call

“Creating an open knowledge base of results, methodologies & interventions on the drivers and consequences of reproducibility for the R&I system; and to fill the main gaps in such knowledge”

Objective 2 – DESIGN: Co-create interventions for improved reproducibility across contexts

Key activities

• Engage co-creation communities to assess user-needs across communities of researchers (social, life, computer sciences), funders, publishers
• Define priorities for action to increase the reproducibility of research & appraise/validate the applicability of current tools for domain-specific use-cases
• Plan development/adaptation of reproducibility tools & practices
• Co-design pilot experiments & their evaluation frameworks
• Minimise bias in design by ensuring any relevant EDI issues (e.g., sex/gender) are addressed in methodologies, target groups & assessment measures

Key results

• Detailed scenario-analysis setting priorities for action to optimise reproducibility practices across five contexts (social, life & computer sciences, as well as funders & publishers)
• Detailed methodological plans for development & piloting of existing tools/practices relating to 5 different contexts (soc, life, comp sci + funders, publishers)

Relation to call

“[F]ill the main gaps in … knowledge”; “Find … solutions and best-practices to increase the reproducibility of research, including through the more systematic integration of sex and gender as variables”

Objective 3 – IMPLEMENT: Drive change through community-driven stakeholder development & piloting of new interventions & tools for reproducibility

Key activities

• Develop or adapt reproducibility tools & practices for new contexts
• Conduct pilot experiments to implement new reproducibility tools & practices across the whole research lifecycle for funders, publishers & scientists
• Investigate & mainstream concrete solutions & best-practices to fill main gaps in knowledge & increase the reproducibility of research

Key results

• Eight pilot activities successfully implemented with researchers, publishers & funders
  • New tools & practices for researchers: Reproducibility Checklists; Reproducibility Management Planning tools; tools for reproducible workflows
  • New tools & practices for publishers: Workflows for review of data/code; tools & standards for “threaded” publications; dashboard for monitoring of policies linked to reproducibility
  • New tools & practices for funders: Reproducibility Promotion Plans; funder extension of tool for Reproducibility Management Plans; Reproducibility monitoring dashboard

Relation to call

“Develop, validate, pilot and deploy practices and practical tools for funders, publishers and scientists”; “fill the main gaps in … knowledge”; “experiment and mainstream concrete solutions and best-practices to increase the reproducibility of research”

Objective 4 – ASSESS: Synthesise findings from across the project using impact pathway logics & econometric analysis to validate the framework

Key activities

• Assess pilot activities using impact pathway analysis and conduct econometric analysis
• Synthesise findings from across project to discover overall gains/savings from reproducibility, including long-term impacts on trust, integrity & efficiency
• Validate the TIER2 framework on reproducibility impact pathways

Key results

• Comparative assessment of pilots and synthesised findings on reproducibility gains & savings
• Validated framework for assessing reproducibility impact pathways

Relation to call

“Determine how increased reproducibility generates gains and savings in the R&I process and improve overall performance - alongside the demonstrated positive effects on their quality, integrity and trust-worthiness”

Objective 5 – RECOMMEND/REFLECT: Co-create a cohesive roadmap for future developments

Key activities

• Co-create a roadmap for future reproducibility, including policy recommendations/guidelines (for researchers, funders, institutions, & publishers) & reform of reward/recognition structures
• Targeted upstream stakeholder engagement to maximise uptake/impact of recommendations
• Auto-ethnographic self-reflection on experiences at the forefront of reproducibility in TIER2

Key results

• Validated practical policy & implementation recommendations/guidelines for research funders, institutions, policy-makers, publishers & researchers in Europe & beyond
• Priorities to reform reward & recognition structures
• Four policy briefs directed to funders, institutions, policy-makers, & publishers
• Auto-ethnographic self-reflection report on TIER2’s experiences at the forefront of reproducibility

Relation to call

“Assist further policy development, based on scoping work by the Commission. While solutions should be applicable to Europe, attention should be paid to reproducibility in global science.”; “It is expected that the funded action(s) will adhere to best practices in open science and reproducibility (e.g., re-use existing results, fully document the research process), and provide a final reflection based on their own experience at the forefront of reproducibility”

Objective 6 – NETWORK/EMPOWER: Equip researchers, funders, publishers & others with the skills, connections & resources to exploit state-of-the-art guidance, tools & services

Key activities

• Create Reproducibility Hub, an open knowledge hub of training, skills, awareness resources for all stakeholders to learn and share regarding reproducibility
• Build capacity of Reproducibility Networks (incl. creation of 3 new RNs in Widening Countries)
• Disseminate & valorise project findings, including training & awareness events
• Ensure interoperability of project tools/results with recognised standards, esp. EOSC
• Align national, regional & international stakeholders and initiatives to harness network effects to boost capacity for increased reproducibility

Key results

• Reproducibility Hub, a sustainable open (minimum CC BY) knowledge base of results, methodologies & interventions on the drivers & consequences of reproducibility for the R&I system hosted at the Embassy of Good Science, including functionalities for researcher checklists, targeted guidance on best tools/practices, and community-building
• Minimum 3 new national Reproducibility Networks created in Widening Participation countries
• Reproducibility Conference & min. 6 “Reprohacks” co-located with discipline-specific events

Relation to call

“Creating an open knowledge base of results, methodologies and interventions on the drivers and consequences of reproducibility for the R&I system; and to fill the main gaps in such knowledge”; “promote uptake, greater collaboration, and increased alignment of the activities of stakeholders - scientific and technical communities, publishers and funders among others - to increase reproducibility”

Reproducibility Hub (incl. checklists)

TRL*4 (start/end): 0 to 9

Lead: VUmc

Training, skills and information resource for researchers, publishers and funders. Building on: Embassy of Good Science (VUmc/TRL 9)

Reproducibility Management Plan tool

TRL (start/end): 2 to 5

Lead: OpenAIRE

New concept to extend Data Management Plans to enable reproducible research. Building on: Argos DMP Tool (OpenAIRE/TRL 8), ROAL (ARC/TRL5), FAIRsharing (UOXF/TRL9), ResearchGraph (OpenAIRE/TRL9)

Reproducible research workflow tools

TRL (start/end): 6 to 8

Lead: ARC

Containerised workflows to facilitate reproducibility and data/code reuse in social (lead: GESIS), life (lead: UOXF), and computer sciences (lead: ARC). Building on: SCHeMa (ARC, FLEMING/TRL8), Methods Hub (GESIS/TRL3), FAIRsharing, ROAL, OpenAIRE ResearchGraph

Data/code review workflow

TRL (start/end): 5 to 8

Lead: KNOW

Streamlined publisher workflows for review of data/code to facilitate publishing checks in soc, life and comp sci. Building on: F1000 platform (F1000/TRL9), CODECHECK (TRL3), SCHeMa, ROAL, OpenAIRE ResearchGraph

Standards for threaded publications

TRL (start/end): 3 to 7

Lead: KNOW

New common standards and best practice guidelines for to enable links between connected research outputs - and associated meta-data descriptors (e.g., Grant information; author and contributor details). Building on: F1000 platform, OpenAIRE ResearchGraph, FAIRsharing, Docmaps (KNOW/TRL6)

Funder Reproducibility Plan instrument

TRL (start/end): 0 to 9

Lead: KNOW

New tool to assist funders create a holistic plan to increase reproducibility of their results. Building on: RiPP (AU, VUmc/TRL8)

Reproducibility monitoring dashboard

TRL (start/end): 4 to 7

Lead: ARC

Dashboard for funders to check levels of FAIRness and re-use of research objects from funded research. Building on: ROAL, FAIRsharing, Argos, OpenAIRE ResearchGraph

TIER2 Key Output

Lever(s) for change

Enables

KO1. Conceptual framework

Evidence

Structured understanding, clarity of aims /means

Results: TIER2 will create a new framework for assessing reproducibility impact pathways across epistemic contexts, with synthesised findings on gains & savings possible via increased reproducibility. This will consolidate evidence on the state-of-the-art regarding evidence/uptake of reproducibility interventions, & provides inventory of reproducibility tools/practices across contexts. This enhanced theoretical/evidential basis will enable shared understanding and orientation on best practices to increase reproducibility.

KO2. Innovative tools & practices

Tools

Concrete solutions (policy-, technical- & practice-based), empowerment, innovation

Results: After TIER2, eight ground-breaking new tools have been successfully implemented & piloted activities successfully implemented. These concrete, innovative solutions for at the levels of policy, technology & practice will empower our key stakeholders to take action.

• For researchers: Reproducibility Checklists; Reproducibility Management Planning tools; tools for reproducible workflows (software containerisation, workflow description, experiment packaging).
• For publishers: Workflows for review of data/code; tools & standards for “threaded” publications; dashboard for monitoring of policies linked to reproducibility.
• For funders: Reproducibility Promotion Plans; funder extension of tool for Reproducibility Management Plans; Reproducibility monitoring dashboard.

KO3. Increased capacity

Skills/Communities

Collaboration, alignment, joint action, skills

Results: TIER2 will empower individuals and networks to boost capacity for the long-term, including via

1. Reproducibility Hub, a sustainable open knowledge base of results, methodologies & interventions on the drivers & consequences of reproducibility for the R&I system hosted at the Embassy of Good Science;
2. Over 1000 researchers, publishers & funders engaged to increase skills via outreach & co-creation events (Reprohacks, training events, webinars, final conference) & Reproducibility Hub functionalities (e.g., Checklist tool, training modules);
3. Empowered networks, with at least 3 new national Reproducibility Networks created in Widening Participation countries & all RNs linked with Open Science, Research Integrity, research infrastructure (EOSC), publisher/funder networks & co-creation communities for increased capacity & multiplier effects in EU & beyond.

The increased alignment and collaboration possible through these linkages, as well as increased capacity for action, will spur reproducibility across all actors.

KO4. Policy roadmap

Incentives/policy

Direction, momentum, sustainability, inclusivity

Results: TIER2 will finally result in a consolidated stakeholder roadmap on priorities for future reproducibility, including reform of reward & recognition structures. This will include practical policy & implementation recommendations/guidelines/briefs for research funders, institutions, policy-makers, publishers & researchers in Europe & beyond. By the end of the project 30 funders, institutions & networks will have endorsed the recommendations. This vision for future action will provide direction and momentum to unite stakeholders in sustainable efforts to address issues of reproducibility for the long-term.

Expected Outcome 1: “Structured understanding of the underlying drivers, of concrete and effective interventions - funding, community-based, technical and policy - to increase reproducibility of the results of R&I; & of their benefits”

KO1 will bring new understanding of the impact of epistemic diversity on the meanings & implications of reproducibility. Mapping the evidence to this framework will result in structured understanding of key reproducibility impact pathways (which interventions work in which domains, with which gains & savings). A broader understanding of reproducibility across epistemic contexts will ensure a more holistic model to implementing & incentivising reproducibility initiatives at multiple levels including bottom-up & top-down approaches. Made available via the Reproducibility Hub (KO3), this analysis will then be a key resource for stakeholders interested in action on reproducibility. This will help stakeholders such as researchers, publishers, funders, & research administrators to prioritise tools & approaches for implementation (e.g., via incorporation into our Researcher Checklist & Reproducibility Promotion Plan (RePP) tools, KO2). By the project’s end, our Researcher Checklist will have been pilot-tested with 50 users, & used by thousands. 3 funders will have implemented RePPs building on our framework, with wider roll-out in the ensuing years (estimated 10 funders have RePPs 3 years post-project, with institutional RePPs a growing theme). In addition, we expect our Reproducibility Impact Pathways analysis to also become a key resource for meta-researchers concerned with monitoring reproducibility (to be monitored via citation analysis).

Expected Outcome 2: “Effective solutions, policy-, technical- and practice-based, to increase the reproducibility of R&I results in funding programmes, communities and dissemination”

KO2 will make a range of new tools & practices available for researchers, publishers & funders. For researchers, our Researcher Checklist empowers & guides to raise awareness & skills; our Reproducibility Management Plan piloting will be the start of a new age of simplified Research Data Management & project reporting; new containerised workflow tools, extended into new areas of social & computer science, will simplify reproducible research pipelines. For publishers, new tools to enable review of data & code will raise review standards & hence the reproducibility of published research; new common standards for threaded publications will enable such incremental publishing to become the norm, combating publication bias & speeding dissemination; & the suitability of new models of publishing (publish then review, registered reports) will be better understood. For funders, Reproducibility Promotion Plans will become a standard tool for reproducibility strategy; Reproducibility Management Plan standards will enable more reproducible funded research & simplify reporting; & our Reproducibility Dashboard will enable easy monitoring of uptake of reproducibility-related practices. By the end of TIER2, these tools will have been piloted with >1000 users, with many in production & available via the Reproducibility Hub & EOSC (KO3), hence to be used by thousands more in the years after TIER2 ends.

Outcome 3: “Greater collaboration, alignment of practices and joint action by stakeholders to increase reproducibility, including but not limited to training, specialised careers and guidelines for best practice.”

KO3 will bring together major building blocks of reproducibility, most prominently the Reproducibility Networks (empowered through expansion) & the EOSC, along with Research Infrastructures like ELIXIR, as well as networks of publishers & funders. TIER2 acts as the facilitatory link between researchers, RNs & other key networks (EOSC & FAIR data ecosystem) as well as funders & publishers through co-creation events, which build lasting networks through engagement to develop mutual solutions in reproducibility. Growing & networking the RNs will allow for greater alignment of activities across Europe. They serve as a platform to share lessons learned at different institutions & collectively, the considerations for implementing tools & practices in different geographical locations & in research ecosystems with varying socio-economic bases. Training modules on the Reproducibility Hub & our Reprohack events will raise skills amongst all stakeholders (>500 participants by project end). Communities engaged in co-creation to design, develop & pilot our tools (KO2) will also be brought together in mutual learning exercises. Through co-creation, “ownership” of tools created incentivizes individual players (researchers, publishers, funders) to advocate for tools within their community, & share best practices in use. Finally, our recommendations & policy instruments (KO4) will set out a powerful, co-created roadmap for future reproducibility in the year beyond the project, including a plan to sustain & empower networks.

Expected Impact 1: “Increased proportion of reproducible results from publicly funded R&I”

Through its Key Impact Pathways methodology & increased awareness of the meanings, gains & savings of reproducibility across epistemic contexts (KO1), the new tools/practices created (KO2), empowered through increased capacity (KO3) & sustained & organised into the future by our policy roadmap (KO4), TIER2 will play a major role in optimising levels of reproducibility in the R&I system. Our mix of tools (KO2) includes pragmatic instruments whose implementation will have immediate effects, as well as more blue-sky thinking (Reproducibility Management Plans) which will be the seeds of change for years to come. Not only will our tools enable this increase, but also its monitoring & assessment (Reproducibility Dashboard, KO2).

Expected Impact 2: “Increased re-use of scientific results by research and innovation”

By bringing our tools into the EOSC (KO3), we also link to the FAIR ecosystem that will enable re-use for the longer term. Increased confidence in the reliability & credibility of research results will further spur re-use, also by industry & other societal actors. Our capacity-building (KO3) & policy actions (KO4) will assist in aligning & motivating this re-use far beyond the end of TIER2. Over the long-term, this will mean that actions to foster reproducibility are linked, with network effects upon impact, to the wider discussion of re-use of results currently framed around FAIR data.

Expected Impact 3: “Greater quality of the scientific production”

Increased reproducibility will come with greater quality of research results. Quality here should be taken in a broad sense, however: more reproducible results will result in less waste, in healthier research communities, & in more efficient & effective research endeavours. Giving the right tools (KO2) & recommendations (KO4) to organisations, funders & publishers, as well as the means to understand what quality means in relation to reproducibility across epistemic contexts (KO1) will hence help reform & enhance the EU R&I system, improve access to excellence & deepen the ERA. This will ultimately result in a more open & inclusive research & innovation system, with increased trust in research outcomes, greater quality of scientific production & stronger translation of R&I results into the economy, as well as more inclusive research environments for researchers.

Key Output 1. Conceptual framework

Scale: TIER2’s conceptual framework has the potential to expand understanding of reproducibility gains & savings across epistemic contexts. Doing so will enable realistic mapping of impact pathways of reproducibility interventions across disciplines & regions, empowering effective decision-making for RPOs, funders & publishers. This framework also consolidates state-of-the-art evidence/advice on existing tools & practices that will be used by more than 1000 researchers during TIER2 & many thousands more afterwards.

Significance: These two contributions enable significant efficiency- & quality-gains through informed future policy. The results provided in this task will be applicable to a large range of research, with our case studies/pilot activities covering three key domains (computer, life, & social sciences). Results will be applicable to researchers in these domains, as well as publishers & funders on a global scale.

Key barrier: Framework too theoretical & not applicable to real-life contexts

Mitigation: TIER2 uses co-creation with relevant stakeholders & evidence accumulated throughout the project to inform development of the framework & ensure real-world relevance/impact.

Key Output 2. Innovative Tools & practices for diverse contexts

Scale: TIER2 will develop 8 new tools & practices for researchers, funders, institutions, & publishers, which will be applicable to key disciplinary areas & ready for extension to many more after the project. The impact of these tools over the long-term will be immeasurable. We expect, for example. our Reproducibility Promotion Plans for funders to become a de facto standard for priority planning, & our workflows for review of data/code will be instrumental in ensuring & validating the reproducibility & re-usability of those objects.

Significance: Our tools target all elements of the research lifecycle from project planning to assessment & hence are of relevance to many thousands of researchers, funders & publishers. Closely aligning these tools with the Embassy of Good science & EOSC will result in fast & efficient uptake.

Key barrier: Tools/practices not fit for purpose

Mitigation: We have selected this provisional list of tools as priorities for development based on state-of-the-art evidence (sec. 1.2). They will be refined, developed & piloted through co-creation with users, & a solid framework for assessment will be used to evaluate & showcase efficacy. Targeted exploitation & dissemination measures, including via our Reproducibility Hub & EOSC will ensure uptake.

Key Output 3. Increased Capacity

Scale: National Reproducibility Networks are powerful networks which reach directly to institutions & across Europe & the globe. Growing aligning this with infrastructural investments like EOSC (250m EUR invested 2018-2020*14), & international networks of funders, publishers & RPOs, mean TIER2 will have powerful multiplier effects in building capacity. Our training & skills resources, hosted for the long-term by the sustainable Embassy of Good Science (over 2500 monthly visitors) will develop further awareness amongst individuals & bring them into this community, meaning that TIER2 will potentially affect the reproducibility practices of hundreds of thousands in the years after the project.

Significance: Under this objective, practical policy & implementation recommendations/guidelines for research funders, institutions, policy-makers, publishers & researchers in Europe & beyond will be developed. The recommendations & guidelines will reach a minimum of 1000 researchers, we aim for adoption of our policy recommendations in at least 4 funders, & 2 publishers through the dissemination activities outlined under section 2.2. This will lead to increased awareness among those stakeholders, & to concrete changes, jump-started by the pilot implementations. We expect pilot activities to be taken up well by stakeholders, given the co-created approach taken to the pilots. We therefore further expect 4 funders & 2 main publishers to continue using the tools & practices developed throughout the project, maximising impact via their own reach.

Key barrier: Increased capacity not carried forward once project-funding & TIER2 activities are over

Mitigation: By seeking to embed our gains in the interlinking of the RNs & EOSC, which both will only grow in importance in the coming years, we are certain our capacity-gains will be sustainable for the long-term, also enabled via our teaming with funder-groups like Science Europe & EVIR (to foster Reproducibility Promotion Plans) as well as publisher-groups like STM Association.

Key Output 4. Policy roadmap (guidelines & recommendations)

Scale: Key themes in TIER2 are epistemic diversity, the need to discern Impact Pathways to optimise reproducibility, & collaboration for network effects. Carrying these into the future will be heavily influenced by the success of our policy recommendations. If done well, they will bind stakeholders in collaborative action to optimise reproducibility in the right ways for the right contexts. This will ensure the prioritisation of reproducibility as a policy priority for funders, publishers, RPOs & others, having the potential to sustain & amplify TIER2’s gains for the long-term.

Significance: We started by noting a conservative calculation that enabling a 1% increase in Horizon Europe would result in savings of 100m EUR. Extrapolating such figures to the global R&I system shows the massive potential significance of TIER2, if momentum & direction is sustained in the correct way beyond the project. Our policy recommendations, co-created with stakeholders, will ensure this happens.

Key barrier: Low acceptance of policy recommendations amongst major stakeholders

Mitigation measure: TIER2 is community-based from the start. Through co-developing our framework & tools with stakeholders, to targeted capacity-building actions, & finally co-creating the recommendations with stakeholders, we are assured the recommendations target the reproducible future stakeholders wish to see. An effective & dedicated exploitation & dissemination plan, in which major stakeholders not already involved in recommendation co-creation are kept informed & engaged, will ensure maximal impact upon release & into the future.

Stakeholder

Channels

Researchers

Social sciences: Links via GESIS (+SSHOC, CESSDA), AU, and via key scholarly societies for target methods; Life sciences: Links via UOXF (+EOSC-Life, ELIXIR), Charite, FLEMING, key scholarly societies; Computer science: Strong links via ARC, KNOW to IEEE, ACM, EUHubs4Data; additional outreach via promotional campaign (esp. for participation in co-creation activities) and presence at disciplinary conferences (Reprohacks)

Funders (RFOs)

Strong links via OpenAIRE, VUmc, to: HERA, EViR, Science Europe, Open Research Funders Group; ARC link to ARCHIMEDES (AI funding); additional outreach via promotional campaign (esp. for participation in co-creation activities)

Publishers

Strong links via OpenAIRE, UOXF, PENSOFT, KNOW, AB (Catriona Maccallum) to: OASPA, STM Assoc, COPE; additional outreach via promotional campaign (esp. for participation in co-creation activities)

Reproducibility Networks

Strong links via Charite, UOXF, Advisory Board, and collected Letters of Interest to RNs in Germany, UK, Slovakia, Brazil (RN forthcoming), Switzerland, Sweden, Norway, Finland, Italy; targeted outreach via Task 1.4 (strategic alignment)

Research Infrastructures (EOSC)

Strong links via OpenAIRE, UOXF, GESIS, FLEMING, KNOW to: EOSC Association; EOSC clusters (EOSC-Life, SSHOC); EOSC-Future; RDA; ELIXIR, CESSDA, ESFRI & ERIC Forum; targeted outreach via Task 1.4 (strategic alignment)

Research Performing Organisations (RPOs)

Strong links via OpenAIRE & others to: EUA, LERU, CAESAR, AURORA, YERUN; targeted outreach via Task 1.4 (strategic alignment)

Research Integrity officers/trainers

Strong links via VUmc, AU to: EARMA, ERION, ENRIO, building on existing close relationships via EnTIRE, SOPs4RI, POIESIS, NRIN, VIRT2UE, iRECs; targeted outreach via Task 1.4 (strategic alignment)

Scholarly/learned societies

Strong links via OpenAIRE and all partners to scholarly societies in a range of disciplines

Libraries & library organisations

Strong links via OpenAIRE to LIBER, EIFL, SPARC; targeted outreach via Task 1.4 (strategic alignment)

Research administrators

Links to iNorms, EARMA

Civil Society Organisations/General public

General media (via press releases), blogs & online media, online courses, newsletter, videos, special interest groups

Industry/SMEs

Industry events, industry media, EC events

Key Exploitable Results

Target audiences

Exploitation route

KO1. Reproducibility impact pathways framework

A framework for assessing reproducibility impact pathways across epistemic contexts, with synthesised findings on gains & savings.

Primary: Research funders, Research publishers, RPOs, Scholarly/learned societies, Scientists & researchers

Secondary: University administrators, Civil Society Organisations, Knowledge-transfer offices

Enable systematic assessment of the impact of reproducibility interventions across contexts to optimize action; enable assessment indicators; provide ideas/inspiration to stakeholders for actions on reproducibility

KO2. Innovative tools & practices

Eight ground-breaking new tools for researchers, funders & publishers.

Primary: Scientists & researchers, Research funders, Research publishers

Secondary: Scholarly/learned societies, Research communities, Libraries & library organisations, University administrators, Civil Society Organisations, industry

Showcase tools via Reproducibility Hub (Embassy of Good Science), EOSC & RN websites; disseminate widely in respective researcher, funder & publisher communities; highlight good practice; TIER2 partners will take up relevant tools within own practices

KO3. Network & increased capacity Sustainable & aligned networks, online training/skills resources.

Primary: Scientists & researchers, Research funders, Research publishers, RPOs, Scholarly/learned societies

Secondary: Research communities, Libraries & library organisations, University administrators, Civil Society Organisations, industry

Enable multiplier effects through linkage of networks (RNs linked with Open Science, Research Integrity, research infrastructure (EOSC), publisher/funder networks); provide knowledge & training both online & in-person; RI training among researchers

KO4. Policy roadmap

Consolidated stakeholder roadmap on priorities for future reproducibility, including reform of reward & recognition structures.

Primary: Research funders, Research publishers, RPOs, Scholarly/learned societies

Secondary: Scientists & researchers, Research communities, Libraries & library organisations, University administrators, Civil Society Organisations, industry

Create practical policy & implementation recommendations/guidelines/briefs for research funders, institutions, policy-makers, publishers & researchers in Europe & beyond. By the end of the project 30 funders, institutions & networks will have endorsed the recommendations.

SPECIFIC NEEDS

D & E & C MEASURES

EXPECTED RESULTS

Discussions on reproducibility driven by a narrow range of disciplines. Common under­standing of pathways to maximise reproducibility across epistemic approaches is highly needed

Need sufficient capacity (network, training, skills & trustworthy policy) for reproducibility

Reproducibility of results relies on tools & practices, which need to be further developed to increase usability & applicability across research contexts

Communication & Dissemination

Co-creation activities

Capacity building

Engage with experts

Exploit partner networks

Engage networks of publishers, funders & RPOs

Disseminate to scientific outlets

Social media engagement

Project dedicated website

Exploitation

Promote repro. promotion plans

Promote recommendations

Promote tools & methodologies across scientific communities

Reproducibility Hub / EOSC

Available EC tools

KO1. Reproducibility impact pathways framework: A framework for assessing reproducibility impact pathways across epistemic contexts, with synthesised findings on gains & savings

KO2. Innovative tools & practices: Eight ground-breaking new tools for researchers, funders & publishers

KO3. Network & increased capacity:

Sustainable & aligned networks, online training/skills resources

KO4. Policy roadmap:

Consolidated stakeholder roadmap on priorities for future reproducibility, including reform of reward & recognition structures

TARGET GROUPS

OUTCOMES

IMPACTS

Researchers (soc, life, comp sci)

Publishers

Funders

Reproducibility Networks

EOSC & Research Infrastructures

Research Performing Organisations (RPOs)

Research Integrity officers / trainers,

Scholarly/learned societies

Libraries & library organisations

Research administrators

Civil Society Organisations / General public

Industry/SMEs

Reproducibility Hub & Impact Pathways analysis providing structured understanding of the drivers, benefits and effective interventions to increase reproducibility of the results of R&I

Eight new tools providing new, effective solutions (policy, technical & practice), to increase the reproducibility of R&I results in funding, publishing, and communities (soc, life, comp sci)

Boosted capacity through greater collaboration, training, alignment of practices & joint action by stakeholders (e.g., 3 new RNs, increased alignment with EOSC)

Trust, Integrity, Efficiency

Scientific

Increased proportion of reproducible results from publicly funded R&I

Greater quality of scientific production

More open and inclusive R&I

Increased capacity for open science

Greater research integrity

Societal

Higher reliability of research results underlying policy making processes

Increased trust in science and R&I outcomes

Economic

Increased re-use of scientific results by research and innovation

Stronger translation of R&I results into the economy

WP No.

1

Lead beneficiary

KNOW

Start month

1

End month

36

WP title

Coordination and Management

Objectives

This work package aims to oversee & coordinate the financial, & day-to-day activities of TIER2 to ensure the achievement of the project’s overall aims & objectives. Specifically, it will coordinate all activities in the project & ensure that partners work together effectively & facilitate the scientific, research & impact success of the project through careful monitoring of progress, management of resources, open communication, & coordination of activities within the time & budget constraints of the project. Additionally, it ensures the project meets its contractual commitments & predicts, mitigates & overcomes project-related risks. WP1 develops plans to ensure the project activities meet the highest ethical, legal, integrity & quality assurance standards. It provides efficient operational management support including administrative & financial planning, reporting to the EC, & management of project legal aspects including project-related contracts & IPR, & management of day-to-day operational & technical progress. Finally, it sets up an international advisory board.

Description of work

Task 1.1 Scientific and administrative coordination (M1-36; KNOW [lead], ARC, VUmc)

Scientific & administrative coordination tasks include the following:

1. Coordinate the Grant Agreement, Consortium Agreement, & signature procedures;
2. Develop, disseminate & monitor adherence to a Project Management, Research Integrity Plan, a Quality Assurance Plan, Data Management Plan, & a Publications Agreement;
3. Coordinate ethics deliverables;
4. Support communication & collaboration between WPs, & the alignment & integration of work between them;
5. Implement & maintain internal reporting & monitoring procedures (including internal review procedures, gender distribution in the consortium composition, & sex, gender & diversity considerations in the project activities);
6. Monitor & support the completion on schedule of deliverables & milestones & take corrective action in the case of delays;
7. Mediate in cases of internal disputes;
8. Organise the General Assembly, Executive Board & Advisory Board meetings
9. Handle daily project correspondence & requests from partners & external bodies;
10. Produce periodic & final reports for the EC.

Task 1.2 Financial Coordination (M1-36; KNOW [lead])

Financial coordination tasks include:

1. Calculate & distribute EC payments to project beneficiaries according to the Consortium Agreement;
2. Liaise with finance departments to monitor contracts;
3. Maintain financial records;
4. Coordinate financial statements submission by all project partners;
5. Provide financial overviews for the periodic & final reports for the EC.

Task 1.3. TIER2 Open and reproducible research practices (M1-36; ARC [lead], KNOW, AU)

Throughout the project, TIER2 will itself adhere to radical reproducibility & transparency. This task will ensure best practices, including adherence to Horizon Europe requirements on Research Data Management & Open Science, & take an auto-ethnographic approach to study our own project as a case study. It will ensure TIER2 adheres to state-of-the-art practices for reproducibility (e.g., pre-registration, reporting of negative results, sharing of protocols, methods, code, data). It will reflect upon the challenges, costs & benefits resulting from such reproducibility & transparency approaches in the context of international, multidisciplinary project consortia. This will lead to a final self-reflection report & feed into the synthesis of results obtained from the empirical work in WPs 3-5.

Task 1.4. Strategic alignment activities (M1-M36; OpenAIRE [lead], KNOW, Charite)

A unique selling point for TIER2 is our position at the nexus of various existing networks & initiatives (Reproducibility Networks, EOSC, other infrastructures, funder/publisher networks) whose effective alignment will bring powerful network effects. As two projects are foreseen to be funded via this HORIZON-WIDERA-2022-ERA-01-41 call, we also identify effective collaboration with the other project to be funded as a priority to avoid overlaps & duplications. This task ensures this alignment, coordinated via OpenAIRE & its network of actors in more than 30 countries covering the EC & beyond.

Deliverables

D1.1 Project handbook (including management, research integrity & quality assurance plans) (KNOW, M4)

D1.2 Data Management Plan (ARC, M6)

D1.3 Data Management Plan (Update) (ARC, M18)

D1.4 Autoethnographic reflections on implementing radical reproducibility in the TIER2 project (AU, M36)

WP No.

2

Lead beneficiary

PENSOFT

Start month

1

End month

36

WP title

Communities, Communication and Dissemination

Objectives

This work package builds the TIER2 network to coordinate community engagement, co-creation activities, fluid outreach/dissemination & hence build capacity for skills & knowledge amongst all actors. This includes building communities & networks to foster & strengthen communication with all key stakeholders in reproducibility & increase awareness/skills for reproducibility tools & practices across disciplines. The WP will: Create a stakeholder map & engagement plan; Co-ordinate the co-creation communities for WPs 3, 4 & 5; Create the Reproducibility Hub to showcase state-of-the-art knowledge, tools & practices for reproducibility; Ensure effective communication (within the project) & dissemination of outcomes (beyond the project).

Description of work

Task 2.1. Stakeholder mapping (M1-M3; KNOW [lead], OpenAIRE, Charite)

As a first task in TIER, we conduct in-depth stakeholder analysis to identify the key players & stakeholders for engagement in co-creation, communication, dissemination & exploitation (cf. provisional analysis section 2.2.2). Using the 5Rs framework, we will analyse each stakeholder’s potential impact & influence within the project (incl. “push-pull” factors) to identify key stakeholders, structures of decision-making, where relationships can/should be strengthened, & resources required. This process will explore/identify key channels/methods/platforms for engagement with each group. The Stakeholder Map will be a key resource underpinning all co-creation & communication/dissemination activities, help structure Task 3.2 (evidence mapping), & underpin classification of resources on the Reproducibility Hub (Task 2.3).

Task 2.2. Community development and coordination of co-creation activities (M1-M36; Charite [lead], KNOW, VUmc, PENSOFT, OpenAIRE, UOXF)

Co-creation is an essential element of TIER2, with co-design & piloting activities with communities of researchers (soc, life, comp sci), publishers & funders, forming the backbone of the project. To ensure successful community-management & effective coordination of activities across WPs3-5, we hence dedicate a specific task. Building heavily on our existing links to the Reproducibility Networks, EOSC (incl. EOSC-Life & SSHOC) & publisher/funder fora, we use platforms/ strategies outlined in D2.1 “Stakeholder Communication & Engagement Plan” to offer co-creation communities a unique value proposition underpinned by support activities to facilitate engaged participation (as outlined in sec 1.2.2.2). Within each case study domain, we will identify groups of reproducibility “champions” to spread awareness of initiatives & tools. We will hold virtual brainstorming events or “BarCamps” to co-create whitepapers on topics such as needs-gap analyses, barriers & enabler assessments, & virtual “co-working” events or “hackathons” to promote & improve reproducibility tools developed & piloted in WPs4/5. To build the network of Reproducibility Networks, we will hold an open call (deadline M9, eligible only to Widening Participation countries), where consortia of 3 or more institutions within a county will be able to apply for one of 3 awards of 5k EUR to fund a first meeting to establish an RN in their country.

Task 2.3. Development of the Reproducibility Hub (M10-M36; VUmc [lead], KNOW, GESIS, OpenAIRE)

This task will develop the Reproducibility Hub as a sub-site of The Embassy of Good Science (hosted by task leader VUmc). The Reproducibility Hub will iteratively combine content from across the project as a knowledge-base of reproducibility tools & best practices for TIER2 stakeholders (researchers, funders, publishers & beyond). The Hub, a wiki-based web-based platform, will use interactive modes of presentation, combining curated checklists, community engagement functionalities, five introductory training modules & in-depth articles on key topics & training materials. FAIRsharing & OpenAIRE data will power interactive graphs that visualise the landscape of reporting standards & best practices (for data, metadata & software), & their relations, as well as their use (by the EOSC clusters) & their adoption by data policies (by funders & publishers). The Hub’s relevance beyond the project’s lifetime will be ensured by its curation within The Embassy, whose governing foundation is already committed to its long-term sustainability. To this end, activities will include workshops hosted at domain-specific conferences to boost dissemination to stakeholders & to provide training on how to contribute to the Hub's content. We will leverage network effects offered by communities from OpenAIRE, FAIRsharing (including the newly launched EOSC-Future/RDA FAIRsharing Community Curation Programme) & the Reproducibility Networks to initiate community-led processes to update content beyond the project’s runtime. In developing the Reproducibility Hub & engaging the wider community to ensure its success, we will strive to leverage synergies with any other EC projects, especially the second project funded under this call.

Task 2.4. Project dissemination and communication (M1-M36; PENSOFT [lead], KNOW, ARC, VUmc, AU, OpenAIRE, Charite)

This task will manage all TIER2 dissemination & communication activities. Building on the provisional dissemination & communication plan described in Section 2.2, this task will ensure the project is optimally promoted to targeted audiences & to the most important stakeholders across Europe. The revised dissemination plan will outline a strategic communication campaign including a social media strategy, which will be updated periodically (D2.1) to ensure targeted communication strategies for the different project phases. Specific activities include the development of a dissemination toolkit, including a professional corporate identity, communication & dissemination materials as well as templates to be used for presentations, reports, meetings & internal documents, within the first six months of the project. This includes a public project website, which will showcase the project’s activities & results both to the scientific/policy community as well as to the public at large. It will coordinate regular dissemination of project progress & results via press releases, news entries & blogs. Further dissemination materials will be created for the purpose of communicating project activities & results (e.g., posters, stickers & other materials for events). As part of the communication plan, publication guidelines will be developed that regulate authorship & specify measures to ensure transparency of methods & reproducibility of project results. This task will also provide organisational support for TIER2’s final project cross-stakeholder conference. The conference will provide a forum to showcase TIER2 results, share reflections on the team’s practices in terms of reproducible research, & present the recommendations from Task 3.3.

Deliverables:

D2.1 Stakeholder Communication & Engagement Plan (PENSOFT, M6)

D2.2 Stakeholder Communication & Engagement Plan (First Update) (PENSOFT, M24)

D2.3 Reproducibility Hub (Resource/website) (VUmc, M36)

D2.4 Stakeholder Communication & Engagement Plan (Second Update) (PENSOFT, M36)

D2.5 Policy Briefing 1 (KNOW, M12)

D2.6 Policy Briefing 2 (KNOW, M36)

Milestones:

MS2.1 Reproducibility Hub (Beta version) (M15)

MS2.2 Final self-reflection report on co-creation processes (M34)

MS2.3 Final conference (M36)

MS2.4 Website and Logo (M2)

WP No.

3

Lead beneficiary

KNOW

Start month

1

End month

36

WP title

Concept, evidence, synthesis and recommendations

Objectives

WP3 provides the methodological & conceptual framework for TIER2. As a first task in the project as a whole, we build on the work of Leonelli (2018), Leonelli (2021) to create a conceptual framework for assessing impact pathways to increase reproducibility across epistemic contexts. We then use this framework to map evidence, including drivers/barriers & an inventory of tools/practices, regarding reproducibility across epistemic contexts. This concept & evidence forms the main knowledge base upon which the later design, development & piloting activities (WP4/5) build. In a final stage, this WP synthesizes findings from these latter activities to systematize knowledge on the reproducibility impact pathways (incl. gains & savings) possible across epistemic contexts. These reflections then inform final recommendations & a roadmap for future reproducibility co-created with relevant stakeholders (including researchers, publishers & funders).

Description of work

Task 3.1. Conceptual framework for reproducibility across contexts (M1-M8; AU [lead], KNOW)

Conceptions, roles & pertinence of reproducibility are closely linked to different principal modes of knowledge production within & across research fields, so-called ‘epistemic contexts’. Using desk research & 5 focus-groups with co-creation communities, we will map out an exhaustive set of epistemic, methodological, social, legal, technological & cultural factors which shape conditions for reproducibility. This task will provide the theoretical framework for TIER2 in the form of a scoping report centred around a matrix that maps the contextual factors to various dimensions of ‘reproducibility’, as well as pertinent framework conditions (e.g., political, ethical, social, legal) that may affect the uptake of reproducibility practices. This framework enables analyses of diverse conceptions, roles & barriers of reproducibility & permits identification of relevant & targeted tools, irrespective of fields.

Task 3.2 Evidence-base and inventory of reproducibility tools and practices (M1-M12; KNOW [lead], AU, Charite, UOXF)

Next, we consolidate & valorise current knowledge on practices & tools for reproducibility by systematically scoping, critically appraising, & synthesising the literature using the Scoping Review (PRISMA-SCR), an Impact Pathway assessment methodology outlined above (sections 1.2.1 & 1.2.2, obj. 1). This will systematically search academic databases & grey literature (including EC project outputs, policy documents, & tool registries) to inventory tools & practices, & synthesise current knowledge on their efficacy across epistemic contexts & stakeholder categories. In addition, we will use FAIRsharing to collect & visualize the reporting standards & best practices within the EOSC science clusters (in particular EOSC-Life & SSHOC, for the life & social science use case, respectively) for both data & software. FAIRsharing will also provide a snapshot of the data policies by publishers & funders, focusing on their requirements, or the lack of, for data reporting & reproducibility. Synthesizing & presenting current knowledge in this way will create a centralised resource which enables all stakeholders (including project partners) to orient themselves easily to the state-of-the-art in this area. This content will then be integrated into the Reproducibility Hub (Task 2.3).

Task 3.3. Synthesis and recommendations (M13-M36; KNOW [lead], AU, VUmc)

In the later phase of the project, this task will bring the project full circle by synthesising results obtained from the empirical studies in WPs4/5, based on the framework established in Tasks 3.1/3.2. First, we incorporate results from our pilots & survey co-creation communities (including questions on costs) to reflect upon how reproducibility generates gains & savings & thereby improves overall performance in the R&I process (including demonstrated positive effects on R&I quality, integrity & trust-worthiness). These results are then validated qualitatively through 3 interactive workshops with our main stakeholder groups (using co-creative approaches, in collaboration with WP4). Based on the synthesis, the task will then use a modified Delphi methodology to work with stakeholders (including members of the co-creation communities) to create recommendations & policy guidelines for practices & joint action by researchers, funders, publishers & institutions to increase reproducibility, including but not limited to training, specialised careers, potential role in assessment criteria & guidelines for best practice. While solutions will be primarily applicable to Europe, attention will be paid to reproducibility in global science. Final recommendations will hence be aligned with user needs, & mapped on the existing practices used by our three research domains & for funders & publishers. The resulting synthesis of knowledge on reproducibility impact pathways & targeted recommendations for action will provide an overall roadmap for future developments beyond the project.

Deliverables:

D3.1 Reproducibility Impact Pathways: State-of-play on methods, tools, practices to increase reproducibility across diverse epistemic contexts (combines findings from T3.1/3.2) (KNOW, M12)

D3.2 Validated key impact pathways for reproducibility, including recommendations (KNOW, M36)

Milestones:

MS3.1 Conceptual framework for reproducibility across contexts (M8)

MS3.2 Interim synthesis of findings on reproducibility gains and savings (M28)

WP No.

4

Lead beneficiary

VUmc

Start month

3

End month

30

WP title

Community-driven design and piloting of reproducibility tools and practices

Objectives

This WP uses future studies, interviews, focus groups & co-creation methods to engage the communities curated in Task 2.2 to steer & assist the development and/or adaptation of the reproducibility tools & practices in WP5. With a focus on the comparative analysis of reproducibility across different epistemological contexts, including barriers & drivers to reproducibility across full research lifecycle, we identify the gaps of knowledge for reproducibility across contexts, & design case-study pilots for the implementation & assessment of new tools. Main objectives:

• Use an innovative backcasting, co-design methodology to work with communities to sketch & select optimal scenarios for future reproducibility across contexts, & identify key areas for development to achieve these scenarios.
• Plan & design interventions that can help important stakeholders see what the costs & benefits are for reproducibility across epistemic contexts for researchers, funders & publishers.
• Prepare & run pilot scenarios for tools & practices developed in WP5 to showcase, verify, & evaluate them across methodological contexts

Description of work

Task 4.1. Future studies to identify priorities from the stakeholder community to predict future of reproducibility and identify actionable steps (M3-M9; VUmc [lead], KNOW, AU)

Profound knowledge on potential reproducibility issues that are faced by researchers from different communities is essential to build a framework, create tools & make potential gains & costs visible. This task will first scope the ideal futures of reproducibility for 5 different stakeholders (soc, life, comp sci researchers, plus funders & publishers) in online scenario workshops. Using provisional results from the scoping work in Task 3.1 & 3.2, future scenarios will be depicted through 5 scenario workshops with backcasting & transcend methods. Based on these goals & ideas of the future, we will then work backwards to identify the key steps needed to reach these scenarios. In a next stage, these ideal scenarios will then be used to work with communities (3 cross-stakeholder focus groups & 14 interviews) to further identify drivers & barriers to realising these futures. Bottom-up coding of transcriptions will be conducted in Nvivo, synthesised & reported back to WP3 to further develop the framework (Deliverable 3.1), & carried forward to inform Task 4.2.

Task 4.2. Requirements procurement and design (M8-M30; GESIS [lead], KNOW, ARC, VUmc, AU, UOXF, FLEMING)

Building upon the priority action areas identified in T4.1, & leveraging the scoping work (inventory of tools & practices) from Task 3.2 & the epistemic contexts identified in Task 3.1, this task will work together with pilot community members to specify new interventions to increase reproducibility across all phases of the research lifecycle from ideation to assessment for different methodologies and epistemic contexts. For each case, a detailed plan of action will be created & the technical & social requirements mapped for development & implementation in WP5. New contexts for existing tools as well as the further development of tools in existing contexts will be considered. In the context of the pilots, we need to apply tools/practices to particular communities/scenarios. These communities have their own special needs & particularities & we here take them into consideration. While we have indicated priority tools for development based on our initial scoping work for this proposal (see section 1.2.2.2), in the spirit of co-creation we base the exact direction of this development on the input of our communities. We hence remain flexible in considering other tools/practices for adaptation or development. Hence this task does essential work in assessing community needs & steering accordingly the development efforts in WP5.

Task 4.3 Pilots preparation activities (M13-M19; ARC [lead], VUmc)

Piloting of new tools & practices is of great importance in TIER2 for validating, evaluating, & improving the set of tools & practices to facilitate the reproducibility of scientific results. Consequently, we implement a set of preparation activities to ensure successful planning & implementation. Although the pilot partners have contributed to the co-design of the tools & practices, various factors can influence their involvement in the piloting & evaluation activities having a large impact on the respective case studies. These factors are expected to be stakeholder-related (e.g., familiarity with technologies, software stacks), inherent to the project work (e.g., problematic/non-expected operation of components), or even unforeseen situations. Our goal is to plan ahead & implement all necessary steps to anticipate & mitigate any possible setbacks & problems. The task will also pre-register all methods to be used in pilot implementation, & design an evaluation framework for the assessment of WP5’s tools considering different epistemic contexts & specific stakeholder groups. In collaboration with the stakeholders, we will compile a list of performance indicators that can help us in unifying the assessment of the tools per research type & stakeholder group. During the proposal phase we have identified key elements of the pilots (listed in section 1.2.2.3 above), which will be validated, fine-grained & attuned to stakeholder requirements during the project, per the co-design approach we propose.

Task 4.4 Pilot implementation and assessment (M20-M30; VUmc [lead], KNOW, ARC, AU, PENSOFT, GESIS, UOXF, FLEMING)

This task involves execution of the pilot actions & the respective evaluation activities. The pilot actions involve use cases that showcase & apply in practice reproducibility-related tools & practices for researchers, publishers, & funders (details on the respective actions in section 1.2.2.3) involving all stakeholder communities (Social sciences, Life sciences, Computer Science researchers, plus funders & publishers). The stakeholders will also work towards the evaluation of the tools & practices according to the workflows designed in T4.3 (details on assessment methodology: section 1.2.2.4) to get insights into what works & what is not working for different epistemic diversities & what is needed to be able to successfully implement them in the research community, for funders & for publishers.

Deliverables:

D4.1 Future reproducibility across epistemic contexts: Report on future studies/backcasting outcomes (VUmc, M9)

D4.2 Pilot implementation and assessment plans (GESIS, M18)

D4.3 Pilot implementation reflection report including assessment of efficacy & recommendations for future developments (VUmc, M30)

Milestones:

MS4.1: Pre-registration of protocol for future studies Task 4.1 (M4)

MS4.2: Pre-registration of methods for pilot implementation/assessment (M18)

MS4.3: Update briefing reports on pilot implementation progress via Reproducibility Hub (M26)

WP No.

5

Lead beneficiary

ARC

Start month

9

End month

34

WP title

Development of tools and practices for communities

Objectives

This work package will develop, extend and/or adapt practical reproducibility-related tools (incl. software, checklists, standards, workflows, monitoring dashboards & policy creation guides) for funders, publishers, & researchers, to be used in the WP4 pilot activities. An agile, continuous, co-creation design & development approach (in constant dialogue with Tasks 4.2-4.4) will be used to ensure that the respective needs & requirements of pilot users are met.

NOTE: The tools earmarked for development here are selected on the basis of the priorities identified in our scoping work for this proposal. Given our co-creative approaches, however, the exact direction of development will be oriented to user-needs discovered through the project implementation. We recognise the list of activities below is highly ambitious. However, we point out that TIER2 development partners have been selected for the tools & initiatives they already bring to the project (many already in production – see section 1.2.2.2), whose effective integration will deliver results at scale & cost.

Description of work

Task 5.1 Practical tools and practices for researchers (M9-M34; ARC [lead], KNOW, GESIS, OpenAIRE, UOXF)

This task implements, customises, & deploys practical tools & practices for researchers that promote & facilitate reproducibility of scientific results during planning, data collection & results analysis phases of the research lifecycle, based on the scoping & co-creation activities in WPs3/4. Planned activities (to be further refined through the WP4 co-creation design processes) are:

• 5.1.1) Reproducibility Checklist: Building upon the scoping work from WP3 we will create an interactive Reproducibility Checklist to guide researchers in best reproducibility practices for their epistemic context. Hosted via the Reproducibility Hub (WP2), the Checklist will also include well-established, discipline specific & domain agnostic recommendations & standards to report digital objects, listed in & tracked in FAIRsharing via community contributions.
• 5.1.2) Extending DMPs to RMPs: Effective Data Management Plans (DMPs) are already a very important element of ensuring reproducible workflows, & actionable DMP tools are increasing this usefulness, but TIER2 believes the potential is even greater. This activity will extend DMPs into a new concept (“Reproducibility Management Plans” - RMPs), that will include additional reproducibility-related metadata for the reported research. To this end, Argos machine-actionable DMP tool will be extended to enable qualified references*15 across research resources & outputs for the whole lifecycle, connecting to FAIRsharing for the community standards, assisting researchers to self-assess & ensure the reproducibility of their research.
• 5.1.3) Reproducible Workflows: We will next leverage software containerisation technologies, workflow description languages (e.g., CWL), & experiment packaging specifications (e.g. RO-crate) to create a detailed methodology to ensure the reproducibility of computational workflows in different epistemic contexts. Our focus will be on adapting SCHeMa, an open-source tool of this type already successfully used within life sciences, to investigate its use in different epistemic contexts (e.g., extending to social sciences for survey data & computer science for Machine Learning).

Task 5.2 Practical tools and practices for publishers (M9-M34; UOXF [lead], KNOW, ARC, GESIS, OpenAIRE)

This task will involve all activities for the implementation, customization, validation & deployment of the set of practical tools & practices that promote, facilitate, monitor & assess reproducibility of research outputs, taking into consideration the special needs & requirements of publishers.

• 5.2.1) Workflows to review research datasets & code: Various mechanisms have been proposed as workflows for checking of data (Wilkinson et al. 2019) & code (Nüst and Eglen 2021), while existing publishing schemes & venues (e.g., Octopus, F1000) have processes in place for advanced screening and/or review of datasets & software regarding reproducibility. To further this work, TIER2 will develop streamlined workflows for review of data & code (at the manuscript submission time), including scoping of a common system of ‘stamps’ or validity marks to indicate that work has been checked, validated and/or reused. We also plan to exploit ARC’s Reproducibility Assessment Toolkit (ROAL) in order to spot & highlight research artefacts (e.g., datasets, software, models) stated in the manuscript, indicating potential missing metadata elements that should accompany & enrich research output descriptions
• 5.2.2) Threaded publications: Incremental publishing models (e.g., Octopus, ResearchEquals) where work is published in smaller units (e.g., protocols, methods, data, code) & then linked (or “threaded”) together at the end to form a cohesive whole are seen as potential routes to greater reproducibility, as publication bias is avoided through early, chronologic & continuous reporting of work & greater scrutiny is enabled through early-sharing of work (Hartgerink and van Zelst 2018). This concept involves basically linking published outputs arising at different stages of a research project or programme together to help ensure that insights & usable outputs are not missing from the research system, thus supporting reproducibility & minimising research waste. Greater uptake of such models amongst publishers requires development to establish shared technical standards for linking entities (i.e., research objects, as well as grants, funders, individuals), as well as to establish who should create & share this information & how, enabling the provenance of research artefacts. TIER2 will do this development work, including investigating potential protocols including RO-Crate, Nanopublications, Docmaps & others, & implement pilot activities.
• 5.2.3) Registered reports/open publishing models for new contexts: From our scoping work (sec. 1.2.1.3), extension of models for publishing which minimise publication bias and maximise transparency are a key priority for publishers. TIER2 Associated Partner and their parent company Taylor & Francis have been key players in introducing such models. This task will further support two lines of action:

1. extension of the F1000 transparent publishing model to new contexts – especially the recently launched Routledge Open Research (for humanities/soc sci) Open Research Europe (covering all Horizon research), especially definition of reporting standards/guidelines for qualitative research;
2. extension of T&F’s workflow for registered reports to new disciplines (e.g., education) to investigate the efficacy (or not) of this model for new epistemic contexts. In addition, with leading conferences, we will also investigate potential for registered reports in Computer Science publishing.

Task 5.3. Practical tools and practices for funders (M9-M34; ARC [lead], KNOW, ARC, VUmc, GESIS, OpenAIRE, UOXF)

This task will implement, customise, & deploy practical tools & practices that enable & support funding institutions in prioritising & tracking reproducibility within their funded projects. Building upon existing production-ready tools from TIER2 partners, this task will develop:

• 5.3.1) Reproducibility Promotion Plans: TIER2 will produce practical advice for funders on how to create a plan to boost the reproducibility of their funded-results. Modelled on the “Research Integrity Promotion Plan (RIPP)” created by VUmc/AU in SOPS4RI, we will work with funders & other stakeholders to develop a tool that assists research funding (and performing) organisations to create “Reproducibility Promotion Plans” (RePPs), including considerations for reforming incentives to foster reproducible practices. The plan will describe how a specific funder will ensure, foster & promote reproducible research practices, avoid detrimental practices, & handle misconduct assisting funders to form their own RePPs to take disciplinary, organisational & national differences into account.
• 5.3.2) Funder extension of RMPs. This activity involves the extension of machine actionable RMP tools developed in T4.1.2 to offer evaluation & reporting functionalities for the officers in funding organisations. Such tools will automate the identification & classification of research artefacts produced during the project & indicate potential enrichment of DMPs with relevant research output descriptions, leveraging the reporting standards in FAIRsharing to also assist in assessing levels of FAIRness.
• 5.3.3) Reproducibility monitoring dashboard: Building upon algorithms created by ARC for the DG-RTD Study “Assessment of reproducibility of research results in EU Framework Programmes”, as well as the OpenAIRE Knowledge Graph & FAIRsharing registry, this activity involves the development of tools that enable funding agencies in tracking & monitoring reusability of research artefacts (datasets, software, tools/systems, etc.) produced within projects of interest, across different programmes, topics & disciplines. An automatically generated report will be generated facilitating assessment & quantification of the impact of policies for data-sharing, code-sharing, etc. We will also scope requirements for making such a dashboard available to publishers.

Deliverables:

D5.1: Reproducibility toolset (tools & practices) for researchers (ARC, M34)

D5.2: Reproducibility toolset (tools & practices) for publishers (KNOW, M34)

D5.3: Reproducibility toolset (tools & practices) for funders (ARC, M34)

Milestones:

M5.1: TIER2 researcher reproducibility toolset first release (M22)

M5.2: TIER2 publisher reproducibility toolset first release (M22)

M5.3: TIER2 funder reproducibility toolset first release (M22)

Know-Center GmbH (KNOW, Austria): KNOW’s Open & Reproducible Research Group (ORRG) is an interdisciplinary team of information scientists & sociologists with key expertise in Open Science implementation & policy (e.g., coordinating ON-MERRIT), assessing reproducibility (DG RTD Reproducibility Study), & enabling transparency in research cultures (OpenUP). As a leading European Research Center for Data-driven Business & Artificial Intelligence, KNOW also has substantial domain expertise in computational sciences, as well as social sciences competences. In TIER2, Know-Center coordinates, leads work to consolidate the evidence-base & synthesise project results (WP3), & contributes to work on piloting (WP4) & developing (WP5) tools & practices.

Athena Research Center (ARC, Greece): ARC has high-level expertise in R&I monitoring (e.g., DG RTD Reproducibility Study), evaluation of Open Science policies & development of e-Infrastructures for scientific content. A leading European AI research institution, ARC has substantial expertise on reproducibility across Computer Science (esp. technologies to assist computational reproducibility). ARC also brings a funder perspective as its “ARCHIMEDES” initiative is an emblematic initiative for Greece funded with over 21m EUR for cascading funds. In TIER2, ARC contributes to the design of the pilots & leads the WP on the implementation of tools & practices (WP 5) that will be demonstrated & evaluated in the context of the pilots (incl. funder piloting via ARCHIMEDES).

Stichting VUmc (VUmc, Netherlands): Stichting VUmc (part of AmsterdamUMC) is a university medical centre with a very active research ethics/integrity department. VUmc has led several Horizon Europe funded projects, including EnTIRe, the founding project of the Embassy of Good Science (now hosted at VUmc). VUmc is initiator of the project Replication in the Humanities, a 4-year project that investigates to what extent replication should be a crucial element in research in the humanities. VUmc also led the use of co-creative methodologies in SOPs4RI to create and implement Research Integrity guidelines at research institutions. These co-creative methodologies will be used on several occasions in TIER2, where VUmc will lead development of the Reproducibility Hub (WP 2), and lead on the design and assessments of pilot activities (WP 4).

Aarhus University (AU, Denmark): The Danish Centre for Studies in Research & Research Policy (CFA) at Aarhus University, is a transdisciplinary research centre with extensive expertise in the areas of science & technology studies, meta-research & research, innovation, & university policy. Expertise on fostering Research Integrity (SOPs4RI), & will coordinate POIESIS (starting 2022) studying connections between public trust in Science, Research Integrity & Open Science. In TIER2 AU will lead on the conceptual framework & contribute to inventorising tools & practices (WP3).

Pensoft Publishers (PENSOFT, Bulgaria): Pensoft is an SME with long-standing expertise in academic, open access book & journal publishing, software development, web design, project dissemination & science communication. PENSOFT has substantial expertise as dissemination partner (over 40 projects). Pensoft is also a renowned Open Science publisher. In TIER2, PENSOFT will be responsible for successful dissemination & communication (WP2), and contribute publisher context in piloting of Reproducibility Management Plans.

Gesis Leibniz Institute for Social Sciences (GESIS, Germany): GESIS is the largest European infrastructure institute for social sciences, providing data, services, infrastructure & training for all stages of research projects. GESIS brings user-communities and expertise for TIER2’s social science pilots. In TIER2, GESIS will contribute to the design (WP4) & development (WP 5) of reproducibility tools & practices, in particular for the social sciences.

OpenAIRE (OpenAIRE, Greece): A key EU e-Infrastructure for Open Science and cornerstone of the EOSC. OpenAIRE links national & EC infraspheres and has active links to publishers and funders. OpenAIRE has an extensive training portfolio, and will act as a capacity-building multiplier via its 47 members in 30 counties to disseminate. In TIER2, OpenAIRE leads alignment activities with EOSC (WP1). contribute its infrastructure for the development/implementation of new tools/practices (WP5), & contributes to stakeholder engagement activities (WP2).

Charité – Universitätsmedizin Berlin (Charite, Germany): Charité is one of the largest university hospitals in Europe. Their QUEST (Quality-Ethics-Open Science-Translation) centre examines how institutions can improve the rigour, reproducibility & transparency of biomedical research. As a founding member of the German Reproducibility Network, and with considerable experience in community engagement/training activities, Charite will lead on stakeholder/co-creation activities, especially bringing together the European reproducibility networks (WP2).

University of Oxford (UOXF, UK): The Data Readiness Group, at the Oxford e-Research Centre, in the Department of Engineering Science has rich expertise in creating elements that underpin reproducibility (esp. in Life Sciences), as part of the development & implementation of standards & policies via the FAIRsharing network, ELIXIR activities & EOSC-Life. UOXF has excellent links to the UK Reproducibility Network (funded partner) and publishers/funders (via related RDA Interest Groups) In TIER2, UOXF join as Associated Partner, bringing their own in-kind resources (funded via UKRI) to contribute to the Reproducibility Hub (WP 2), & to the development and piloting of tools/practices for researchers, funders and especially publishers (WP 5).

Biomedical Research Center Fleming (FLEMING, Greece): Fleming is a high-profile non-profit research organisation focusing on scientific & technological excellence, training & innovation in biomedical sciences. In TIER2, FLEMING will contribute to pilot preparation, implementation & assessment for the life sciences (WP4).

Repro-ducibility, RI, OS

R&I policy

Pilot domain expertise

Infra­­­structure / tools

Repro­ducibility networks

EOSC

Dissem., training, network­ing

Publisher view

Funder view

KNOW

x

x

CS

x

x

x

ARC

x

x

CS

x

x

x

VUmc

x

SS

x

AU

x

x

SS

PENSOFT

x

x

GESIS

SS

x

x

OpenAIRE

x

x

x

x

x

x

x

Charite

x

LS

x

x

UOXF

x

LS

x

x

x

x

FLEMING

LS

x

Advisory Board

x

x

x (all)

x

x

x

x

x

x