Research Ideas and Outcomes : Grant Proposal
Grant Proposal
TIER2: enhancing Trust, Integrity and Efficiency in Research through next-level Reproducibility
expand article infoTony Ross-Hellauer, Thomas Klebel, Alexandra Bannach-Brown§, Serge P.J.M. Horbach|, Hajira Jabeen, Natalia Manola#, Teodor Metodiev¤, Haris Papageorgiou«, Martin Reczko», Susanna-Assunta Sansone˄, Jesper Schneider|, Joeri Tijdink˅, Thanasis Vergoulis«
‡ Know-Center GmbH Research Center for Data-Driven Business & Big Data Analytics, Graz, Austria
§ Charite - Universitaetsmedizin Berlin, Berlin, Germany
| Aarhus University, Aarhus, Denmark
¶ GESIS-Leibniz-Institut Für Sozialwissenschaften, Mannheim, Germany
# OpenAIRE AMKE, Athens, Greece
¤ Pensoft Publishers, Sofia, Bulgaria
« Athena - Athena Research & Innovation Center In Information Communication & Knowledge Technologies, Athens, Greece
» Biomedical Sciences Research Center Alexander Fleming, Vari, Greece
˄ Oxford e-Research Centre, Department of Engineering Science, University of Oxford, Oxford, United Kingdom
˅ Stichting VUmc, Amsterdam, Netherlands
Open Access


Lack of reproducibility of research results has become a major theme in recent years. As we emerge from the COVID-19 pandemic, economic pressures and exposed consequences of lack of societal trust in science make addressing reproducibility of urgent importance. TIER2 is a new international project funded by the European Commission under their Horizon Europe programme. Covering three broad research areas (social, life and computer sciences) and two cross-disciplinary stakeholder groups (research publishers and funders) to systematically investigate reproducibility across contexts, TIER2 will significantly boost knowledge on reproducibility, create tools, engage communities, implement interventions and policy across different contexts to increase re-use and overall quality of research results in the European Research Area and global R&I, and consequently increase trust, integrity and efficiency in research.


Open Science, Reproducibility, Research quality, Epistemic diversity, Tools and practices, Policy intervention, EOSC, Reproducibility Networks, Community engagement

List of participants

Detailed list of all paricipants is available in Table 1.

Table 1.

List of particiapants.


Participant organisation name

Principal investigator


1 (Co­ordinator)

Know-Center GmbH Research Center for Data-Driven Business & Big Data Analytics (KNOW)

Tony Ross-Hellauer (m)



Athena - Athena Research & Innovation Center In Information Communication & Knowledge Technologies (ARC)

Thanasis Vergoulis (m)



Stichting VUmc (VUmc)

Joeri Tijdink (m)



Aarhus Universitet (AU)

Jesper Schneider (m)



Pensoft Publishing (PENSOFT)

Lyubomir Penev (m)



GESIS-Leibniz-Institut Für Sozialwissenschaften EV (GESIS)

Hajira Jabeen (f)




Natalia Manola (f)



Charite - Universitätsmedizin Berlin (Charite)

Alexandra Bannach-Brown (f)



The Chancellor Masters & Scholars of The University of Oxford (UOXF) (Associated Partner)

Susanna-Assunta Sansone (f)



Biomedical Sciences Research Center Alexander Fleming (FLEMING)

Martin Reczko (m)



TIER2 is a new project funded by the European Commission under their Horizon Europe programme (call HORIZON-WIDERA-2022-ERA-01-41 - Increasing the reproducibility of scientific results*1), running from January 2023 until December 2025. (Fig. 1)

Figure 1.  

TIER 2 Logo

Lack of reproducibility of research results has become a major theme in recent years. As we emerge from the COVID-19 pandemic, economic pressures (increasing scrutiny of research funding) and exposed consequences of lack of societal trust in science make addressing reproducibility of urgent importance. TIER2 does so by selecting 3 broad research areas (social, life and computer sciences) and 2 cross-disciplinary stakeholder groups (research publishers and funders) to systematically investigate reproducibility across contexts. The project starts by thoroughly examining the epistemological, social and technical factors (epistemic diversity) which shape the meanings and implications of reproducibility across contexts. Next, we build a state-of-the-art evidence-base on existing reproducibility interventions, tools and practices, identifying key knowledge gaps. Then TIER2 will use (co-creation) techniques of scenario-planning, backcasting and user-centred design to select, prioritise, design/adapt and implement new tools/practices to enhance reproducibility across contexts. Alignment activities ensure tools are EOSC-interoperable, & capacity-building actions with communities (i.e., Reproducibility Networks) will facilitate awareness, skills and community-uptake. Systematic assessment of the efficacy of interventions across contexts will synthesise knowledge on reproducibility gains and savings. A final roadmap for future reproducibility, including policy recommendations is co-created with stakeholders. Thus, TIER2 will significantly boost knowledge on reproducibility, create tools, engage communities, implement interventions and policy across different contexts to increase re-use and overall quality of research results in the European Research Area and global R&I, and consequently increase trust, integrity and efficiency in research.

We here present the TIER2 project “Description of Action”, which programmatically guides project activities. This text represents our initial project proposal as submitted to the EC, with only slight modifications (e.g., streamlining and clarification of some deliverables and milestones, removal of some administrative information for readability).

As TIER2’s success will depend on community engagement we make this information public not only as part of our commitment to principles of open and reproducible research, but also to inform the community of our plans and invite interested parties to get involved.

1 Excellence

This section describes: (a) the rationale and objectives of TIER2, including how the project aims to go beyond the state-of-the-art (subsection 1.1), and (b) our overall methodology for the project (subsection 1.2).

1.1 Objectives and ambition

The European Research Area (ERA) is an ambitious effort to create a “single, borderless market for research, innovation and technology across the EU”. Launched in 2000 and “revitalised” in 2018, the current ERA Policy Agenda sets out 20 concrete actions for 2022-2024, including to “Enable Open Science, including through the European Open Science Cloud (EOSC),” “Upgrade EU guidance for a better knowledge valorisation”, “Strengthen research infrastructures” and “Build-up research and innovation ecosystems to improve excellence”. Enabling increased reproducibility*2 of research results directly contributes to, and builds upon, these aims.

Reproducibility is often claimed as a central principle of the scientific method (Popper 2002). Although definitions vary, at its highest level, (as per the text of this call) reproducibility refers to the possibility for the scientific community to obtain the same results as the originators of a specific finding. Recent years have seen perceptions of a “reproducibility crisis” grow in a variety of disciplines, especially behavioural and medical sciences. Famously, a 2016 survey by Nature reported that over half of respondents perceived “a ‘significant 'crisis' of reproducibility” (Baker 2016). Systematic replication studies and scoping reviews of prevalence of reproducibility-related factors in published studies confirm that an issue exists, although differing across disciplines (Freedman et al. 2015; Open Science Collaboration 2015). Lack of transparency (e.g., poor reporting/publishing of methods/data/code/analysis), lack of reproduction/replication studies, publication bias towards reporting of positive results, and growing awareness of questionable research practices have been identified as key problems (Atmanspacher and Maasen 2016). Poor levels of reproducibility are seen as serious threats to scientific self-correction, efficiency of research processes and societal trust in research results.

As the world emerges from the COVID-19 pandemic, economic pressures (and hence increased scrutiny of research funding) and the exposed consequences of lack of societal trust in science make addressing such issues of urgent importance. Doing so will reduce inefficiencies, avoid repetition, maximise return on investment, prevent mistakes, and speed innovation to bring trust, integrity and efficiency to the ERA and global Research and Innovation (R&I) system in general. To set the scale of potential savings, according to Freedman et al. (2015), more than half of results in preclinical medicine were not reproducible. This will of course vary by discipline, but even if we assume only a fifth of overall results are irreproducible, this would still equate to almost 20bn EURs of the 95bn to be spent in Horizon Europe.

Increasing reproducibility is a multifaceted challenge, however. At the practical level, there is an urgent need for capacity-building to improve infrastructure/services, skills, communities, incentives and policies to enable and encourage reproducibility-maximising practices. At the theoretical level, moreover, three key information gaps currently hinder progress:

  1. Limited clarity on meanings, limits and implications of reproducibility across modes of knowledge production;
  2. Limited understanding of optimal reproducibility impact pathways to maximise gains and minimise costs in reproducibility reform;
  3. No coherent roadmap for implementing policies and practices to optimise reproducibility across the whole R&I system.

An influential EC scoping report Reproducibility of scientific results in the EU (DG-RTD 2020) shows the EC’s commitment to this issue and offers many possible actions. These suggestions have been refined through a further key EC study, Assessing the reproducibility of research results in EU Framework Programmes.*3 This study, whose central authors include key members of TIER2, adds crucial new context to understanding reproducibility across EU research. Building on this work (detailed below, section 1.2.1), the TIER2 consortium proposes an innovative programme of activities founded upon four key principles:

  1. Reproducibility is an opportunity, not a crisis;
  2. Epistemic diversity (variation across modes of knowledge production and socio-technical contexts) must be centred;
  3. Evidence must be systematised for informed policy across contexts;
  4. Action must be targeted holistically to boost capacity at all levels.

TIER2 will centre epistemic diversity by selecting three broad research areas (social, life and computer sciences) and two cross-disciplinary stakeholder groups (research publishers and funders) to systematically investigate reproducibility across contexts. In tandem with curated co-creation communities of these groups, we will design, implement and assess systematic interventions addressing key levers of change (tools, skills, communities, incentives, and polices). The project will start by thoroughly examining the epistemological, social and technical factors (epistemic diversity) which shape the meanings and implications of reproducibility across contexts (epistemic contexts). Next, we will build a state-of-the-art evidence-base on extent and efficacy of existing reproducibility interventions and practices, as well as an inventory of relevant tools, identifying key gaps in current knowledge. Then TIER2 will use (co-creation) techniques of scenario-planning, backcasting and user-centred design to select, prioritise, design/adapt and implement new tools to enhance reproducibility across contexts. Alignment activities will ensure tools are EOSC-interoperable, and capacity-building actions with communities (i.e., Reproducibility Networks) will facilitate awareness, skills and community-uptake. Systematic assessment of the efficacy of interventions across contexts will enable a synthesis of knowledge regarding reproducibility gains and savings. This will inform a final roadmap for future reproducibility, including policy recommendations co-created by stakeholders. Through these activities, TIER2 will significantly boost knowledge on reproducibility, create tools, engage communities, implement interventions and policy across different contexts to increase re-use and overall quality of research results in the ERA and beyond, and consequently increase trust, integrity and efficiency in research.

TIER2's core objectives, along with their relative key activities and forseen results, are outlined in Table 2 below.

Table 2.

TIER2's core objectives.

Objective 1 - CONCEPTUALISE: Create the conceptual & evidential framework for the project

Key activities

  • Establish the state-of-play by defining reproducibility across diverse epistemic contexts & inventorying tools/practices & evidence of uptake & efficacy
  • Identify gaps in current knowledge & key tools/practices
  • Design the TIER2 framework for assessing reproducibility impact pathways

Key results

  • Scoping report on state-of-the-art regarding evidence/uptake of reproducibility interventions
  • Detailed inventory of reproducibility tools/practices and their current uptake
  • Framework for assessing reproducibility impact pathways across epistemic contexts

Relation to call

“Creating an open knowledge base of results, methodologies & interventions on the drivers and consequences of reproducibility for the R&I system; and to fill the main gaps in such knowledge”

Objective 2 – DESIGN: Co-create interventions for improved reproducibility across contexts

Key activities

  • Engage co-creation communities to assess user-needs across communities of researchers (social, life, computer sciences), funders, publishers
  • Define priorities for action to increase the reproducibility of research & appraise/validate the applicability of current tools for domain-specific use-cases
  • Plan development/adaptation of reproducibility tools & practices
  • Co-design pilot experiments & their evaluation frameworks
  • Minimise bias in design by ensuring any relevant EDI issues (e.g., sex/gender) are addressed in methodologies, target groups & assessment measures

Key results

  • Detailed scenario-analysis setting priorities for action to optimise reproducibility practices across five contexts (social, life & computer sciences, as well as funders & publishers)
  • Detailed methodological plans for development & piloting of existing tools/practices relating to 5 different contexts (soc, life, comp sci + funders, publishers)

Relation to call

“[F]ill the main gaps in … knowledge”; “Find … solutions and best-practices to increase the reproducibility of research, including through the more systematic integration of sex and gender as variables”

Objective 3 – IMPLEMENT: Drive change through community-driven stakeholder development & piloting of new interventions & tools for reproducibility

Key activities

  • Develop or adapt reproducibility tools & practices for new contexts
  • Conduct pilot experiments to implement new reproducibility tools & practices across the whole research lifecycle for funders, publishers & scientists
  • Investigate & mainstream concrete solutions & best-practices to fill main gaps in knowledge & increase the reproducibility of research

Key results

  • Eight pilot activities successfully implemented with researchers, publishers & funders
    • New tools & practices for researchers: Reproducibility Checklists; Reproducibility Management Planning tools; tools for reproducible workflows
    • New tools & practices for publishers: Workflows for review of data/code; tools & standards for “threaded” publications; dashboard for monitoring of policies linked to reproducibility
    • New tools & practices for funders: Reproducibility Promotion Plans; funder extension of tool for Reproducibility Management Plans; Reproducibility monitoring dashboard

Relation to call

“Develop, validate, pilot and deploy practices and practical tools for funders, publishers and scientists”; “fill the main gaps in … knowledge”; “experiment and mainstream concrete solutions and best-practices to increase the reproducibility of research”

Objective 4 – ASSESS: Synthesise findings from across the project using impact pathway logics & econometric analysis to validate the framework

Key activities

  • Assess pilot activities using impact pathway analysis and conduct econometric analysis
  • Synthesise findings from across project to discover overall gains/savings from reproducibility, including long-term impacts on trust, integrity & efficiency
  • Validate the TIER2 framework on reproducibility impact pathways

Key results

  • Comparative assessment of pilots and synthesised findings on reproducibility gains & savings
  • Validated framework for assessing reproducibility impact pathways

Relation to call

“Determine how increased reproducibility generates gains and savings in the R&I process and improve overall performance - alongside the demonstrated positive effects on their quality, integrity and trust-worthiness”

Objective 5 – RECOMMEND/REFLECT: Co-create a cohesive roadmap for future developments

Key activities

  • Co-create a roadmap for future reproducibility, including policy recommendations/guidelines (for researchers, funders, institutions, & publishers) & reform of reward/recognition structures
  • Targeted upstream stakeholder engagement to maximise uptake/impact of recommendations
  • Auto-ethnographic self-reflection on experiences at the forefront of reproducibility in TIER2

Key results

  • Validated practical policy & implementation recommendations/guidelines for research funders, institutions, policy-makers, publishers & researchers in Europe & beyond
  • Priorities to reform reward & recognition structures
  • Four policy briefs directed to funders, institutions, policy-makers, & publishers
  • Auto-ethnographic self-reflection report on TIER2’s experiences at the forefront of reproducibility

Relation to call

“Assist further policy development, based on scoping work by the Commission. While solutions should be applicable to Europe, attention should be paid to reproducibility in global science.”; “It is expected that the funded action(s) will adhere to best practices in open science and reproducibility (e.g., re-use existing results, fully document the research process), and provide a final reflection based on their own experience at the forefront of reproducibility”

Objective 6 – NETWORK/EMPOWER: Equip researchers, funders, publishers & others with the skills, connections & resources to exploit state-of-the-art guidance, tools & services

Key activities

  • Create Reproducibility Hub, an open knowledge hub of training, skills, awareness resources for all stakeholders to learn and share regarding reproducibility
  • Build capacity of Reproducibility Networks (incl. creation of 3 new RNs in Widening Countries)
  • Disseminate & valorise project findings, including training & awareness events
  • Ensure interoperability of project tools/results with recognised standards, esp. EOSC
  • Align national, regional & international stakeholders and initiatives to harness network effects to boost capacity for increased reproducibility

Key results

  • Reproducibility Hub, a sustainable open (minimum CC BY) knowledge base of results, methodologies & interventions on the drivers & consequences of reproducibility for the R&I system hosted at the Embassy of Good Science, including functionalities for researcher checklists, targeted guidance on best tools/practices, and community-building
  • Minimum 3 new national Reproducibility Networks created in Widening Participation countries
  • Reproducibility Conference & min. 6 “Reprohacks” co-located with discipline-specific events

Relation to call

“Creating an open knowledge base of results, methodologies and interventions on the drivers and consequences of reproducibility for the R&I system; and to fill the main gaps in such knowledge”; “promote uptake, greater collaboration, and increased alignment of the activities of stakeholders - scientific and technical communities, publishers and funders among others - to increase reproducibility”

The ambition of TIER2 is to increase trust, integrity, & efficiency in research through next-level reproducibility tools, practices & policies across diverse epistemic contexts. We will achieve this by:

  • Taking stock of existing knowledge/evidence, and clarifying the meanings/implications of reproducibility across epistemic contexts.
  • Building capacity and innovating new EOSC-native tools and practices for funders, publishers and researchers through community-led pilots addressing infrastructure/services, skills, communities, incentives and policies.
  • Enumerating gains and savings to build a common understanding and roadmap for promoting and monitoring reproducibility impact pathways across epistemic contexts.

Thereby, TIER2 will enhance reproducibility beyond the state-of-the-art in diverse ways:

  • Deepened understanding of reproducibility across modes of knowledge production: epistemic diversity is at the core of TIER2. Simply put, profound differences in the aims and methods of knowledge production have deep implications for the types of reproducibility that can/should be expected. Thus far, the conversation on reproducibility has been dominated by a relatively narrow section of this spectrum (e.g., behavioural and clinical sciences). TIER2, through its commitment to first understanding these differences and then studying them in depth via a cross-disciplinary case-study approach will greatly expand this understanding.
  • Network & capacity-building: TIER2 will harness network effects by connecting existing networks of researchers, funders and publishers. TIER2’s activities will be a keystone in furthering development of reproducibility communities-of-practice, especially by linking the national Reproducibility Networks (grassroots local networks embedded at individual institutions, already present in the UK, Finland, Germany, Italy, Norway, Portugal, Slovakia, Sweden and Switzerland) to higher-level networks of funders (e.g., Science Europe, RDA Research Funders IG, Open Research Funders Group), publishers associations (e.g., STM Association, OASPA, EASE) and infrastructures (e.g., EOSC, ESFRIs, OpenAIRE). These actions will sustainably spur collective action amongst major stakeholders. (For full list of stakeholders, see section 2.1).
  • Vision & roadmap for future reproducibility: TIER2 will work with communities of researchers, funders and publishers to envisage future optimal conditions for reproducibility in their contexts and use back-casting to prioritise what types of tools and innovations are necessary for this future of reproducibility. All pilot activities will be implemented to enable maximum assessment of efficacy, and synthesise all findings into our framework of epistemic diversity. In a final stage, we will use innovative methods of co-creation (modified Delphi method) to collaboratively construct a practical roadmap for future actions amongst researchers, funders, and institutions.
  • Next-level EOSC-native reproducibility tools & practices: EOSC aims at seamless access across infrastructures to support a 'Web of FAIR Data and services' for science in Europe. FAIR (Findable, Accessible, Interoperable and Reusable) research objects are a cornerstone of reproducibility (Wilkinson et al. 2016). TIER2 will create tools and practices designed to be interoperable with EOSC from the start, building on existing key elements of the EOSC ecosystem (e.g., OpenAIRE, and FAIR-enabling components connected to EOSC science clusters, such as FAIRsharing). TIER2 will seamlessly enrich EOSC’s range of value-added services to increase reproducibility and reuse. Table 3 below summarises the provisional range of tools to be potentially developed in TIER2.
Table 3.

New tools/practices resulting from TIER2.

Reproducibility Hub (incl. checklists)

TRL*4 (start/end): 0 to 9

Lead: VUmc

Training, skills and information resource for researchers, publishers and funders. Building on: Embassy of Good Science (VUmc/TRL 9)

Reproducibility Management Plan tool

TRL (start/end): 2 to 5

Lead: OpenAIRE

New concept to extend Data Management Plans to enable reproducible research. Building on: Argos DMP Tool (OpenAIRE/TRL 8), ROAL (ARC/TRL5), FAIRsharing (UOXF/TRL9), ResearchGraph (OpenAIRE/TRL9)

Reproducible research workflow tools

TRL (start/end): 6 to 8

Lead: ARC

Containerised workflows to facilitate reproducibility and data/code reuse in social (lead: GESIS), life (lead: UOXF), and computer sciences (lead: ARC). Building on: SCHeMa (ARC, FLEMING/TRL8), Methods Hub (GESIS/TRL3), FAIRsharing, ROAL, OpenAIRE ResearchGraph

Data/code review workflow

TRL (start/end): 5 to 8

Lead: KNOW

Streamlined publisher workflows for review of data/code to facilitate publishing checks in soc, life and comp sci. Building on: F1000 platform (F1000/TRL9), CODECHECK (TRL3), SCHeMa, ROAL, OpenAIRE ResearchGraph

Standards for threaded publications

TRL (start/end): 3 to 7

Lead: KNOW

New common standards and best practice guidelines for to enable links between connected research outputs - and associated meta-data descriptors (e.g., Grant information; author and contributor details). Building on: F1000 platform, OpenAIRE ResearchGraph, FAIRsharing, Docmaps (KNOW/TRL6)

Funder Reproducibility Plan instrument

TRL (start/end): 0 to 9

Lead: KNOW

New tool to assist funders create a holistic plan to increase reproducibility of their results. Building on: RiPP (AU, VUmc/TRL8)

Reproducibility monitoring dashboard

TRL (start/end): 4 to 7

Lead: ARC

Dashboard for funders to check levels of FAIRness and re-use of research objects from funded research. Building on: ROAL, FAIRsharing, Argos, OpenAIRE ResearchGraph

1.2 Methodology

Here, we present the overall methodology for TIER2, including key concepts, methodologies and project open and reproducible research practices.

1.2.1 Overall concept

In 2020 the EC commissioned a major review “Assessing the reproducibility of research results in EU Framework Programmes”.*5 The study aimed to understand, test and monitor the progress of reproducibility over time and across programmes through qualitative and quantitative assessment of outputs (proposals, DMPs, publications, datasets) from 1000 EC projects for prevalence of reproducibility-related factors; survey of EC researchers, journal editors, funders; co-created recommendations with stakeholders (researchers, funders, publishers). Two of the three partners (KNOW, ARC) were involved in conducting this work are core partners in TIER2, and its major findings deeply inform our proposal, including:

  • The “crisis” narrative is unreflective of current reality. Our survey of H2020 beneficiaries found that only around a fifth perceived a significant crisis (much lower than the 2016 Nature survey). We believe this is possibly reflective of two phenomena: (1) the conversation on reproducibility has entered a new phase, (2) our sample, selected randomly from Principal Investigators of H2020 projects, possibly reflects a wider disciplinary scope than the Nature survey which overemphasised certain disciplines (e.g., biology comprised almost half).
  • Understanding of the notion of “reproducibility” & the attitude towards reproducibility varies significantly by field. Epistemic and social factors must be taken into account in any policy actions designed to increase reproducibility. Qualitatively and quantitatively assessing documents and outputs (proposals, DMPs, publications, datasets) from 1000 EC projects, we found that practices associated with increased reproducibility (FAIR data, software/materials sharing, reporting standards) apply very differently across fields.
  • Interventions to improve reproducibility are currently often targeted broadly, while much of the evidence emerges from distinct fields (esp. medicine/health, psychology). Work to understand issues of reproducibility have been predominantly led by select disciplines, especially psychology and clinical medicine (Cobey et al. 2022). However, these disciplines are only part of the funding landscape. More work is needed to systematise knowledge of which interventions are appropriate in which contexts to determine the impact pathways whereby interventions result in increased reproducibility (and the extent to which this is desirable in different contexts).
  • Cultural factors (pressure to publish & lack of incentives), followed by training & lack of infrastructure, are perceived by researchers, journal editors & funders as the greatest barriers to reproducibility. Joined-up approaches which work across all levels from technical and skills, to norms and incentives are required. In addition, researchers, publishers, funders and others (including infrastructure providers) all support the principles of reproducibility, yet the landscape of joint action is currently very diffuse. Increasing reproducibility (as a distinct strategic goal), for example, is not yet a high priority for journals or funders. Many diverse initiatives exist whose alignment would have powerful multiplier effects.

Building upon these findings, TIER2 proposes a programme of activities based on four key principles. We believe the reproducibility agenda must enter a new phase. If phase one is typified by the “crisis narrative”, narrow focus on specific fields, and piecemeal initiatives with limited alignment of strategic action across stakeholders and elements of research, phase two (TIER2) must be founded in the following:

  1. Reproducibility is an opportunity not a crisis. Our finding that far fewer perceive reproducibility a “significant crisis” confirms recent calls to treat enhanced awareness of reproducibility as an opportunity rather than a crisis (Munafò et al. 2022). Thus reframing the debate will enable us to move beyond hyperbole to more considered analysis of which solutions work in which circumstances across the research enterprise.
  2. Epistemic diversity must be centred. Recent work by TIER2 Advisor Leonelli (2018) adds nuance to our understanding of reproducibility by highlighting the importance of “epistemic diversity” in shaping consequences for reproducibility across research contexts. Factors including degrees of control over environments, reliance on inferential statistics, precision of research aims, and reliance on interpretation, as well as technical, social and cultural factors, all have deep implications for the meanings, implications and even applicability of concepts of reproducibility across epistemic contexts. These factors must be better understood across research contexts to inform thorough analysis of potential gains and savings across R&I.
  3. Evidence must be systematised for informed policy across contexts. Acknowledging that reproducibility has very different meanings and consequences across epistemic, social, and technical contexts epistemic contexts, it is essential that any analysis of gains and savings be rooted in an understanding of how intervention impact pathways vary according to these contexts. Not all impacts will be positive, and trade-offs/unintended consequences are to be expected. For example, Vazire (2018) suggests that although increased reproducibility may raise productivity in general, productivity may be reduced in some subfields. At the same time, persistent structural inequalities and biases, as well as mechanisms of cumulative advantage within research, may mean that uncritical policies for transparency and openness may have unintended negative consequences which compound inequities (Ross-Hellauer 2022, Ross-Hellauer et al. 2022). We hence fully agree with the EC reproducibility scoping report that we must “[d]evelop policies that support communities at different levels of maturity, not only advanced disciplines or countries” (DG-RTD 2020). (Fig. 2)
  4. Action must be targeted holistically to boost capacity at all levels. Fostering a culture of maximal reproducibility will require concerted action across levels of research cultures: creating tools (infrastructures and services) to make it possible, equipping researchers and others with skills to make it easy, networking communities to make reproducible practices the norm, revising incentives to make it rewarding, and implementing policies (where helpful) to make it necessary. Much work is already in place across all these elements. Linking and building on such initiatives is an essential task..
Figure 2.  

Levers for research culture change, adapted from (Nosek 2019).

Crucial concepts for TIER2’s approach to implementing these principles are the concepts of key impact pathways, epistemic diversity and our case-study approach. Key impact pathways

TIER2 will bring together theory and evidence to design a framework that defines gaps and prioritises new approaches, based upon the Key Impact Pathways (KIPs) methodology currently being operationalised to monitor R&I impact in Horizon Europe, and that TIER2 partners KNOW and ARC will employ in the “PathOS: Open Science Impact Pathways” project commencing Sept 2022.*6 TIER2 will aim to identify and document reproducibility impact pathways, i.e., the possible paths that connect input to output, outcome and impact (including linkages of causal mechanisms and drivers/barriers) to elucidate the routes to increased reproducibility across epistemic contexts and quantify resultant gains and savings (Fig. 3).

Figure 3.  

TIER2’s provisional schema of key elements to map Reproducibility Impact Pathways.

Impact pathway methods are grounded in theory-based approaches (Brousselle and Buregeya 2018), responding to the need not only to estimate and measure net effects of an intervention, but to provide explanations why and how impacts occur. Such approaches are particularly helpful for the production of policy-oriented recommendations, highlighting the conditions to be ensured for a given public initiative or intervention to be successful. To construct and understand these pathways we need to:

  • identify key elements of pathways (input-output-outcome-impact) under the organisational prism of needs and objectives,
  • describe how they are linked and work together,
  • develop metrics for impact, and
  • measure and test on selected cases.

Charting the impact pathways requires that we describe the sequence of input-output-outcome-impact relations that show non-linear linkages and the steps from resources to more long-term impacts. It also entails developing narratives describing causal chains, including the effects of possible enabling factors and barriers. Starting from the key inputs of factors in research culture change, for each of our target domains (comp, life, soc sci), we will trace and prioritise the activities required to produce desired outputs, outcomes and impacts. From the description of the rationales and mechanisms linking the elements of reproducibility impact pathways across epistemic contexts, Fig. 3 will be further developed and validated in its components/linkages. Epistemic diversity

Reproducibility impact pathways will vary greatly due to epistemic diversity. Although, as already stated, concerns regarding reproducibility have been most vocally stated from a relatively narrow range of disciplines, they are increasingly addressed in other areas as well (Stagge et al. 2019, Powers and Hampton 2018, Ioannidis et al. 2017, Camerer et al. 2018), However, much more work is needed to understand reproducibility issues and their pertinence both within and across disciplines. For example, empirical science domains each have different conventions and cultures regarding the role of chance, the level of certainty required for making published claims, and the adopted criteria for “proof”. Indeed, reproducibility cannot be thought of as an all-embracing epistemic scientific principle or equivalent term for quality and soundness of research findings. As Sabina Leonelli has argued (Leonelli 2018), probing issues of reproducibility requires systematic accounting of “epistemic diversity” and especially the ways in which features of the research environment in types of knowledge production have profound consequences for reproducibility. In Leonelli’s conception, factors to be examined include: (assumed) level of control on the research environment, degree to which statistics are used for inference, precision of research aims (e.g., exploratory = expected low reproducibility, clinical trials = expected high reproducibility), and the degree to which researcher judgement is a factor.

Reproducibility is not only affected by epistemic and methodological factors, however. Auer et al. (2021) argue that in the life sciences, multiple factors influence levels of reproducibility, including technical (e.g., contaminated materials, natural variability, batch effects), study design and statistics (e.g., design flaws, questionable practices like HARKing and P-hacking, selective reporting), as well as human (lack of detail on methods, lack of data-sharing, mistakes, publication bias) and external (insufficient reward structures, hypercompetition, fraud). Leonelli goes further and identifies many other factors, which she terms “sources of epistemic diversity”: Material (target objects, materials); Conceptual; Methodological (standards, methods); Infrastructural (funding, infras, ICTs, mobility); Socio-cultural (reward structures, legal/ethical factors. Geopolitics, language, values/goals, researcher demographics); Institutional (career stage/power dynamics, administrative support mechanisms, norms of research fields. intellectual property requirements (Leonelli 2018). In sum, research domains differ in their epistemic assumptions, methodological constraints, specific contextual and technical challenges, and sensitivity to cognitive and behavioural biases, which all influence reproducibility. Understanding this diversity is a core aim of TIER2 Case-study approach

Acknowledging that there is substantial epistemic diversity across research fields, with the need to better understand the relevance, meanings and implications of reproducibility across them, TIER2 hence proposes a comparative methodology which looks at reproducibility across such contexts with as much breadth and depth as possible (given the resources). We hence select three broad research domains (social sciences, life sciences, and computer sciences) for research(er) contexts, as well as two contexts which cut across disciplines (publishers and funders). The Horizon Dashboard indicates that our target domains (computer, life, social sciences) accounted for well over half of H2020 funding. Each of these broad communities is at different stages of recognition of reproducibility as a theme, and face different issues. Analysing them comparatively will enable greater understanding of such variation. We next outline the key issues facing each of these groups to contextualise TIER2’s priorities and methods outlined later (Table 4).

Table 4.

Key reproducibility issues in TIER2's selected five cases.

Case 1: Reproducibility issues in social sciences

Social sciences can be conceptualised as a very heterogeneous field, encompassing diverse epistemological and methodological approaches, working with various kinds of data including opinion polls, voting records, surveys, self-reported perceptions, behaviours, beliefs or attitudes, social network data, government statistics and indices, GIS data measuring human activity, and various forms of qualitative data, such as interview transcripts, field notes, and observational protocols. Reflecting this diversity, recognition of reproducibility as an issue greatly varies. While parts of psychology have been a dominant part of the ‘reproducibility crisis’ discussion since the start (Open Science Collaboration 2015; Camerer et al. 2018), domains traditionally relying on observational methodologies, such as sociology or economics have more recently begun to address such issues (Freese and Peterson 2017; Christensen and Miguel 2017). A key building block in tackling the consequences of this epistemic diversity in TIER2 will be the Data Documentation Initiative *7 (DDI), a collection of widely-used standards for describing data produced by surveys and other observational methods, already used by SSHOC (the EOSC social sciences and humanities cluster) of which TIER2 member GESIS is a key partner. Actions to facilitate reproducibility should hence be tailored to the specific epistemic contexts they address. Weston et al. (2019) suggest various approaches to make secondary data analyses more reliable, such as conducting sensitivity analyses, multiverse analysis or specification curve analysis. To address issues of quantitative approaches in the social sciences in general, Engzell and Rohrer (2020) suggest relying on researchers’ tacit knowledge to identify common forms of malpractice in their field, which could be taken as a springboard to develop new tools and guidelines. Proposed actions to target reproducibility among qualitative methods such as ethnography or grounded theory are scarce, due to fundamentally different conceptions of how knowledge is and should be produced. Through our engagement with GESIS’ user communities, TIER2 will be able to investigate these issues across a wide range of contexts, with particular interest on reproducibility in survey data.

Case 2: Reproducibility issues in life sciences

Life science is a very heterogeneous field with a number of disciplines, study and technology types. The data community in biology and medicine, however, is probably the most active one in creating data and metadata standards to support the reuse (including reproducibility) and sharing of the information, supported by strong international data mandates (e.g., 1996 Bermuda for genomics data, 2009 Toronto agreement for omics/clinical data). Much of the understanding of scientific transparency stems from the experience in bioinformatics, where the focus has been on information (incl. datasets, code, models and software) that is harmonised with respect to structure, format and annotation. Nowadays there are over 900 standards in the life sciences. Since the early 2000s grass-roots initiatives and standards organisation have worked to create:

  • minimal reporting requirements (outlining the necessary and sufficient information vital for contextualising and understanding of data and other digital objects;
  • terminologies (ranging from dictionaries to ontologies, provide definitions and unambiguous identification for concepts and objects, and statistical results, e.g. Stato; and
  • models and formats (defining the structure and relationship of information for a conceptual model and include transmission formats to facilitate the exchange of data between different systems, including workflows, e.g. Common Workflow Language, and containerised methods, e.g. RO-Crate.

Standardisation to enable FAIR data, which underpins reproducibility, is at the core of ELIXIR (European Life Sciences infrastructure), and part of its Interoperability Platform activities (Harrow et al. 2021), also via resources like the RDMkit and the FAIR Cookbook, and more broadly of the EOSC-Life cluster’s mission. Despite the many initiatives, the uptake of experimental data and metadata standards has been slow and uneven, and further compromised if standards are not promptly implemented by repositories and tools to become ‘positively’ invisible to the researchers (Sansone et al. 2019). Recently, a survey of 251 researchers, comprising authors who have published in eLIFE journal showed how life scientists view the issues around reproducibility in journal articles and that even “there is an apparent disconnection between the development of tools addressing reproducibility and their use by the wider scientific and publishing communities who might benefit from them” (Samota and Davey 2021). TIER2 will draw up the work and experience of the EOSC-Life large collaborative network of communities, as well as other relevant national and international life science initiatives connected to ELIXIR (e.g., Australian BioCommons, Global BioData Coalition, NIH Data Science, and the new RDA Life Science WG based on the expanded RDA ELIXIR IG) to address these issues.

Case 3: Reproducibility issues in computer science

Computer science research involves a large degree of determinism (high precision of goals, high dependence on statistics, total control on environment). In experimental work, high specification (far higher than in lab or other types of experimental research) of methods is theoretically possible since each computational action is logged (Peng 2011). Yet this is often not the case due to data, software, and execution environment (aka digital laboratory) not being shared in consistent, persistent and structured ways (incl. issues with lack of infrastructure; complex and deprecated dependencies; non-open source software; discrepancies between fields; access to computational resources; lack of training and unintentional bias in data analysis; selective reporting). Lack of data used for training in, e.g., Machine learning (ML), is a key problem as results are reliant on training datasets, yet data is often proprietary or not made available. Selective reporting whereby “people tend to report on the edge cases when they get ‘really lucky’ in one run” (Cacho and Taghva 2020) is a known issue. Algorithms themselves are often closed (Hutson 2018). In addition, the generalisability of results to real-life cases beyond the training data is often in question. Algorithms trained on datasets skewed to specific populations or situations may not perform as well beyond those limited environments. ML is even known to repeat structural and cognitive biases (such as racism) present in society (Garcia 2016). Recognition of issues is growing, with “Reproducibility Tracks” being introduced at many major Computer Science conferences (e.g., SIGIR, ECIR, ISWC) to incentivise submission reproduction/replication of existing work, and conferences (e.g., AAAI) include a strict reproducibility checklist for all submissions. Workflow tools like Docker and high-level description frameworks like RO-Crate are increasingly enabling easy pipelines for packaging and sharing of data, software, and execution environment but much remains to be done. TIER2’s priorities will be to continue this work, enabling workflow tools to enable reproducible computational workflows, as well as investigating the cultural aspects of reproducible practice in Computer Science.

Case 4: Reproducibility issues in research publishing

In line with growing concerns about the quality and credibility of research and publishing processes, ensuring the reliability of published research has become increasingly important to publishers and journal editors over the past decade. Several approaches to incentivise or improve the reproducibility of the published record have been proposed. These include both pre- and post-submission measures, such as journal reporting guidelines, improved peer review practices and data availability requirements. However, recent studies have shown that, while improving, the number of journals explicitly demanding or enabling data sharing practices is still limited (Malički et al. 2021), the effectiveness of journal reporting guidelines is mixed (Hepkema et al. 2021), and peer review endeavours to filter irreproducible research face multiple challenges (Horbach and Halffman 2018). This goes for both traditional journal outlets, as well as for novel preprint platforms (Malicki et al. 2020). Hence, efforts to foster reproducibility at the stage of publication or peer review are still lagging. In part, this is due to a lack in understanding of differences across and within disciplinary publishing cultures, to which TIER2 will specifically contribute. More specifically, a recent article in Science Editor (Kousta et al. 2019) proposes three priority interventions to foster reproducibility through journal efforts: checklists for transparent reporting, peer review of data and code, and new models of publishing (e.g., registered reports) to avoid publication bias. In TIER2 we address the reporting requirements, as well as peer review and new models of publishing digital objects, both by studying them and by drafting templates for their implementation, in consultation with relevant stakeholders in the publishing community.

Case 5: Reproducibility issues in research funding

Research funders have similar motivations to journal editors. Where research is open and reproducible research, it helps maximise potential impact and return on investment (ROI). Funders of scientific research are well positioned to guide scientific discoveries by enabling and incentivising the most rigorous and transparent methods. There are numerous recommendations on how funders should act in order to increase reproducibility (Broman et al. 2017, Center for Open Science 2022). However, in practice, beyond actions in support of FAIR data, funder actions to centre reproducibility as an explicit theme are arguably still at an early stage. Funder priorities identified via our EC study were mechanisms to centre such actions at a policy-level, tools to enhance data management planning and reporting, and tools to monitor uptake of reproducible practices amongst funded projects. TIER2 will aim to address these issues particularly.

1.2.2 Detailed methodology

The TIER2 overall methodology, following six stages according to our stated objectives, is illustrated in Fig. 4. Each stage incorporates checks and balances between all activities to ensure alignment. We next detail each step.

Figure 4.  

TIER2 methodological steps. Methods for Objective 1: CONCEPTUALISE (conceptual/evidential framework)

Realised through WP3:

  • Task 3.1. Conceptual framework for reproducibility across contexts (M1-M8)
  • Task 3.2. Evidence-base and inventory of reproducibility tools and practices (M1-M12)

Our first major task within TIER2 will be to consolidate knowledge to date relating to reproducibility across epistemic contexts. Using desk research and three online focus-groups with co-creation communities, we will map out factors influencing epistemic diversity across our case-study domains (life, social and computer sciences). Systematically elucidating pertinent epistemic and methodological factors for reproducibility across these contexts will provide the initial theoretical framework for TIER2. The output will be a scoping report centred around a matrix that maps different epistemological aims and methods to various dimensions of ‘reproducibility’, as well as pertinent framework conditions (e.g., political, ethical, social, legal) that may affect the uptake of reproducibility practices. This framework will enable analyses of diverse conceptions, roles and barriers of reproducibility and permits identification of relevant and targeted tools, irrespective of fields.

Next, we will use a PRISMA-SCR ‘Scoping Review’ methodology to scope literature to date to answer the question: “What tools and practices are suggested to improve reproducibility across these epistemic contexts, and what evidence exists regarding their efficacy?”. We will systematically search for key terms across academic databases (e.g., Scopus, Web of Science, OpenALEX, OpenAIRE) as well for grey literature including policy reports (via and searches of stakeholder/project websites and funder databases of outputs (e.g., EC CORDIS). Methods will be pre-registered in advance (via Open Science Framework), including search protocols and data-charting strategies. In addition to evidence on efficacy of reproducibility interventions, we will also compile an exhaustive list of tools and practices, classified according to the elements of research lifecycle and epistemic contexts to which they pertain. Finally, we will collect and visualise key reporting standards and best practices in use within the EOSC science clusters relevant to our case-study domains, especially EOSC-Life (life sci) and SSHOC (soc sci).

These strands will be synthesised using the in a Deliverable report “D1.1 Reproducibility Impact Pathways: State-of-play on methods, tools, practices to increase reproducibility across diverse epistemic contexts”, which uses the Key Impact Pathways approached outlined above to identify key areas for intervention and tools/practices upon which to build in the project. Synthesising and presenting current knowledge in this way will create a centralised resource which enables all stakeholders (including project partners) to orient themselves easily to the state-of-the-art. In addition to providing the general theoretical and evidential basis for assessing gains/savings of reproducibility in the project, this content will also be used within the Reproducibility Hub (knowledge resource created in WP2) and the Researcher Reproducibility Checklist and Reproducibility Management Plan development activities (WP5). Methods for Objective 2: DESIGN (co-creation of interventions)

Realised through WPs 2, 4:

  • Task 2.1. Stakeholder mapping (M1-M3)
  • Task 2.2. Community development and coordination of co-creation activities (M1-M36)
  • Task 4.1. Future studies to identify priorities from the stakeholder community to predict future of reproducibility and identify actionable steps (M3-M9)
  • Task 4.2. Requirements procurement and design (M8-M30)

Co-creation, defined as “a transparent process of value creation in ongoing, productive collaboration with, and supported by all relevant parties, with end-users playing a central role" (Pieters and Jansen 2017), will run throughout TIER2 to ensure the most appropriate, sustainable and synergistic solutions to reproducibility challenges. Building on principles of dialogue, access, reflexivity and transparency (Prahalad and Ramaswamy 2004), continuous iterative loops of stakeholder participation will accompany every step of TIER2’s research and innovation activities. A dedicated task in WP2 will oversee co-creation community engagement, with five dedicated sub-communities (covering computer, life and social sciences, as well as publishers and funders) of stakeholders convened for each TIER2 case. Incentivising community engagement is key. A call for co-creation community participants accompanied by a promotional video will make clear the benefits for communities - offering access to cutting-edge tools, networking opportunities, training/skills development and involvement in events to secure active participation. Regular online community meetings will facilitate active involvement and learning within and across community groups.

For the design phase, TIER2 will build on partner experience in successful co-creative methodologies in the SOPs4RI and ON-MERRIT projects to conduct five (co-creative) online scenario workshops with relevant stakeholders (8-10 stakeholders per workshop). In these workshops, we will work with stakeholders to:

  1. envision the ideal future scenario regarding reproducibility for each stakeholder group;
  2. use “backcasting” techniques (Inayatullah 2008) to identify the major building blocks needed to enable these future scenarios. In a next round, we will use an online survey to
  3. prioritise what interventions for development and piloting are desirable and needed within subsequent TIER2 activities (further design in WP4 and development in WP5).

With these results, we can determine what contexts need more co-creation design in order to better equip researchers, funders and publishers with tools that promote reproducibility. Field notes and transcribed recordings will be used to create draft reports of the outcomes of each workshop, which will then be shared with participants for their further feedback. These priorities will then be carried forward to the use-case definition and design phase. Here, working closely with the developers from WP5, stakeholders will be engaged to define central use-cases for the envisioned tools/practices to be developed. For each case, a “design thinking” canvas-approach based on the JISC innovation canvas tool*8 will be used, with a structured design methodology centred on eight aspects (stakeholders, challenges, state-of-play, vision, opportunity, impact, management, and solution) to create detailed plans of action which build on knowledge from the previous tasks to map technical and social requirements for development activities in WP5 and subsequent piloting in WP4. Tools to be designed will build, in particular, upon the following software/platform assets brought by the TIER2 consortium (see Table 5 below).

Table 5.

Existing TIER2 consortium software/platform assets.


TRL: 9

Partner: OpenAIRE

Argos is an open, extensible, and configurable machine-actionable tool developed to facilitate Research Data Management activities concerning the implementation of Data Management Plans (DMPs).

Embassy of Good Science

TRL: 9

Partner: VUmc

The Embassy is a wiki-based platform for the research community to share experiences and insights about research integrity and ethics, continuously contributing to the development of good science.


TRL: 9

Partner: UOXF

FAIRsharing is a curated registry, tool and service for data and metadata standards, inter-related to repositories and data policies. It hosts 1600 standards, 1900 databases, and 160 policies.


TRL: 9

Partner: GESIS

da|ra is a DOI registration service in Germany for social science and economic data, in cooperation with DataCite. It has more than 699k DOI registrations, and ~4m DOI resolutions per year.


TRL: 7

Partner: GESIS

gesisDataSearch enables searching for research datasets based on a periodically crawled index. It has more than 100,000 Datasets with approximately 600 new users per month.

GESIS Search

TRL: 9

Partner: GESIS

Integrated search across different Social Science data collections at GESIS. It has 6,500 datasets, 13,000 survey variables, 400 measure instruments, and 107,000 publications.


TRL: 2

Partner: GESIS

MethodHub builds on containerisation technologies and digital lab notebooks to facilitate social science data analysis and its reproducibility, to find, learn, and experiment with computational methods.

OpenAIRE Research Graph

TRL: 9

Partner: OpenAIRE

OpenAIRE Research Graph aggregates research data properties (metadata, links) for funders, organisations, researchers, research communities and publishers to interlink information.


TRL: 5

Partner: ARC

ROAL (ReprOducibility Assessment tooLkit) streamlines assessment of reproducibility by automating the identification of datasets, their classification in terms of re-use, and the extraction of metadata.


TRL: 8

Partner: ARC

SCHeMa facilitates reproducibility of computational experiments on heterogeneous clusters, exploiting containerization, experiment packaging, workflow description languages, and Jupyter notebooks. Methods for Objective 3: IMPLEMENT (community-driven development & piloting)

Realised through WPs 2, 4, 5:

  • Task 2.2. Community development and coordination of co-creation activities (M1-M36)
  • Task 5.1. Practical tools and practices for researchers (M9-M34)
  • Task 5.2. Practical tools and practices for publishers (M9-M34)
  • Task 5.3. Practical tools and practices for funders (M9-M34)
  • Task 4.3. Pilots preparation activities (M13-M19)
  • Task 4.4. Pilot implementation and assessment (M20-M30)

In the tool development activities in WP5, we will use an agile early/rapid prototyping methodology with short development cycles (release early/often), ensure constant evaluation from domain experts, regular cycles of user-feedback, flexible experimentation with various ideas and directions, and increased collaborative activities. Design sketches and static mock-ups will be produced rapidly to test new concepts and receive critical feedback. Working prototypes with limited functionality (e.g., baseline algorithms, smaller data) will be rapidly developed to convey a more realistic impression of their operation. After feedback, the prototypes will be iteratively improved, until the design is gradually finalised, and the underlying algorithms are configured, customised and optimised. In their development, all tools will aim to be optimised for integration into the European Open Science Cloud, building especially on our existing EOSC links to enrich EOSC’s range of value-added services to increase reproducibility and reuse. These tools will then be piloted in WP4 (Tasks 4.3, 4.4). Table 6 below outlines provisional aims, instruments and expected outcomes for the Pilots, although we point out we expect these to evolve in line with the co-creation activities outlined in the methods for the prior objective.

Table 6.

TIER2 Provisional list of TIER2 pilot activities (to be refined/adapted in line with co-creation methods).

Pilot 1. Reproducibility Checklist (for researchers in social, life, computer sciences) [lead: KNOW]

Aim: To customise & evaluate the Reproducibility Checklist created in T5.1.1 for researchers across case-study domains.

Activities: Conduct user-testing & surveys for the use, refinement, & enrichment of the Reproducibility Checklist by stakeholders in

  1. Social sciences (GESIS via CESSDA/SSHOC);
  2. Life sciences (UOXF via ELIXIR);
  3. Computer science (KNOW/ARC via IEEE). Build a connection between the Reproducibility Hub and FAIRsharing

Instruments: Various metadata standards for disciplines & digital objects via FAIRsharing; tool inventory (T3.2) & advice on skills (T2.2) included in the Reproducibility Hub (WP3); GESIS MethodsHub, da|ra

Expected outcomes: Detailed user-testing & feedback (survey) from stakeholders; Production-level Reproducibility Checklist tool, available via Reproducibility Hub and connected to FAIRsharing; Reproducibility Checklist registered in FAIRsharing, as a new ‘reporting guideline’, and DOI assigned.

KPIs: At least 24 participants in the surveys; At least TRL8 for the Reproducibility Checklist tool

Pilot 2. Reproducibility management plans (RMPs) (for researchers in social, life, computer sciences, & funders) [lead: OpenAIRE]

Aim: Customise & evaluate prototype “Reproducibility Management Plan” functionalities in case-study domains.

Activities: User-testing/surveys for definition, core functionalities & initial prototyping of RMP tool amongst:

  1. researchers: active Horizon Europe projects;
  2. funders: minimum three funders

Instruments: Argos DMP tool (see T5.1.2 & 5.1.3); PENSOFT publishing platforms; FAIRsharing; GESIS MethodsHub, Go-inter

Expected outcomes: Core RMP functionalities pilot tested

KPIs: At least 15 HE projects in soc, life, comp sci participating in the tests; Min 3 funders pilot test; At least TRL5 for the RMP tool

Pilot 3. Reproducible workflows (for researchers in social, life, computer sciences) [lead: ARC]

Aim: Customise & evaluate tools/practices for reproducible workflows in 3 epistemic contexts.

Activities: (1) Adapt & extend SCHeMa protocol to further facilitate data/code reproducibility in life sci (e.g., scRNA-seq analysis workflow provided by FLEMING); (2) Extend SCHeMa to

  • Social sciences, incorporate & test aspects of SCHeMa into GESIS MethodHub;
  • Computer sciences, adapt SCHeMa for Machine Learning use-cases, incl. support of ReproZip for computational experiment packaging. Final online validation workshop, users from all domains will share experiences & mutual learning

Instruments: SCHeMa (see T5.1-activity 5.1.3); RO-Crates; FAIRsharing; GESIS Search; GESIS MethodsHub; standards (e.g., DOME recommend­ations for life science; DDI for social science); RDMKit; FAIR Cookbook; (ReproZip for computational experiment packaging)

Expected outcomes: Use-cases implemented to extend SCHeMa containerisation workflows; Consoli­dated knowledge on applicability of SCHeMa for new domains; Mutual learning across research communities

KPIs: Min 3 use cases for SCHeMa expansion; SCHeMa min TRL6 for new research domains; Min 25 participants across epistemic contexts (mutual learning activities)

Pilot 4. Workflows to review research datasets & code (for publishers) [lead: UOXF]

Aim: (1) Implement & test workflows for data/code, including advanced screening & review of datasets & software regarding reproducibility; (2) Scope common system of ‘stamps’ or validity marks to indicate that work has been checked, validated and/or reuse

Activities: (1) user-testing & survey with reviewer-editor pairs; (2) cross-stakeholder focus groups to scope essential features of “validity marks”; (3) piloting workflows to assess reproducibility (based on containerisation & experiment packaging technologies) & review method/model generalisability in CS/AI conferences

Instruments: Workflows/ tools developed in T5.2.1 (building on ARC’s ROAL toolkit; containerisation elements (RO-Crate, Docker, SCHeMa); FAIRsharing; GESIS Data search (soc sci use-case)

Expected outcomes: New workflows for review of data/code; Approaches piloted with cross stakeholder fora (publishers, researchers); Scoping report on requirements for common system of validity marks; Testing generalisability workflows in CS/AI conferences

KPIs: Min 30 reviewer-editor pairs participate in user-testing/surveys; Min 3 cross-stakeholder focus groups; 1 scoping report for ‘stamps’ or validity marks

Pilot 5. Threaded (linked) publications (for publishers, plus researchers in social, life, computer sciences) [lead: KNOW]

Aim: Investigate how models of threaded, linked related publications can support reproducibility in case-study domains (social, life, computer science)

Activities: Focus groups & user-testing with researchers using current platforms (e.g., Octopus, ResearchEquals) to iterate required common models, standards, drivers & barriers to mainstreaming of threaded publication models

Instruments: Octopus, ResearchEquals, F1000 platform, plus other potential tools/standards for incorporation (RO-Crates, Nanopublications, Doc­maps, GESIS MethodHub & Knowledge Graph)

Expected outcomes: Enhanced under­standing of common models, stand­ards, drivers & barriers for threaded publication models

KPIs: 1 focus group to be implemented; Min two threaded publications platforms tested; At least 24 participants in the surveys

Pilot 6. New models of publishing & review - focusing on open & transparency & mandatory data deposition & availability (for publishers, plus researchers in social sciences & humanities) [lead: KNOW]

Aim: Investigate how: (1) the F1000 model (publish then review, with maximally open data & review, & (2) models for registered reports, transfer to humanities & social science research contexts

Activities: Study how (1) SSH users of Open Research Europe platform & the new Routledge Open Research (both based on F1000 platform) experience this new publishing model; (2) user-testing & data analysis of Taylor & Francis SSH research published as registered reports

Instruments: Open Research Europe & Routledge-branded platform for SSH research (both based on F1000 Platform); Taylor & Francis journals piloting registered reports workflows

Expected outcomes: New publication workflows for SSH research, including new guidelines on “data” deposition; Enhanced understanding of potential role of registered reports in combating publication bias in SSH

KPIs: At least 20 users participating to the user-testing; At least 10 registered reports published in the context of testing

Pilot 7. Reproducibility Promotion Plans (for funders) [lead: VUmc]

Aim: Produce practical advice for funders on how to create a plan to boost the reproducibility of their funded-results

Activities: Pilot the RPP tool with at least two funders (one will be ARC’s ARCHIMEDES) to create a “Reproducibility Promotion Plan” & examine the issues involved in creating & implementing such a plan.

Instruments: “Reproducibility Promotion Plan” tool developed in T5.3.1

Expected outcomes: Validated “Reproducibility Promotion Plan” tool made available to the wider funder community (via Reproducibility Hub)

KPIs: At least 2 funders to create RPPs; At least 1 RPP to be tested by each funder

Pilot 8. Reproducibility monitoring dashboard (for funders) [lead: ARC]

Aim: Development & testing of tools to enable funding agencies in tracking & monitoring reusability of research artefacts (datasets, software, tools/systems, etc) created in funded projects

Activities: Implement dashboard & conduct user-testing with funder communities

Instruments: ROAL; FAIRsharing, OpenAIRE Knowledge Graph

Expected outcomes: Validated prototype Funder Reproducibility monitoring dashboard

KPIs: At least 8 funder representatives perform user testing, incl. ARC’s ARCHIMEDES scheme for AI research Methods for Objective 4: ASSESS (evaluation of pilots & findings synthesis)

Realised through WPs 2, 3, 4:

  • Task 2.2. Community development and coordination of co-creation activities (M1-M36)
  • Task 4.4. Pilot implementation and assessment (M20-M30)
  • Task 3.3. Synthesis and recommendations (M13-M36)

A systematic framework for assessment of pilot activities will be developed alongside the design activities (Objective 3) and implemented according to the steps to impact pathway evaluation proposed by (Douthwaite et al. 2003):

  1. What would success look like (what would intended outcomes be)?
  2. What factors influence achieving each outcome?
  3. Which factors can the project influence
  4. Which can it not?
  5. Which factors, in which ways, will be targeted for change to bring about desired outcomes?
  6. What performance information should be collected (including to assess the ways in which reproducibility brings gains or savings)?
  7. How can this information be obtained?

Following (Douthwaite 2007), each pilot activity will be asked in the planning stages to create such an impact plan with measurable KPIs. In collaboration with the stakeholders, we will compile a list of performance indicators that can help us in unifying the assessment of the tools per epistemic context and stakeholder group. Indicators will be developed to cover key aspects of the tools’ impact such as their practicability (ease of use) and efficacy (proven record of increased reproducibility, assessed where possible via randomised controlled trials). This will enable a consistent logic describing, ex-ante, how pilot activities are hypothesised to contribute to increased reproducibility (via which outputs, outcomes, impacts), and enable effective ex-post impact assessment once the pilots have concluded. Throughout these processes, close attention will be paid to issues of potential bias amongst pilot populations (e.g., gender, region) that may affect generalisability of findings. This common basis for evaluation of pilot activities will hence enable learning across the piece and elucidate the reproducibility interventions from which real impact emerges. This knowledge will then be subsumed in a next step of synthesis.Collecting information on assessment of pilots in this structure will facilitate the synthesis of these results into overall knowledge on impact pathways for improving reproducibility across epistemic contexts (which inputs and activities lead to which outputs, outcomes and impacts). Hence, the impact pathways identified via Tasks 1.1 and 1.2 will be either validated, questioned or expanded based on this new evidence produced within the project. With participating and linked stakeholders we will validate and revise the TIER2 framework and other key outputs to result in a synthesis of project results which determines how increased reproducibility generates gains and savings in the R&I process and improves overall performance - alongside the demonstrated positive effects on their quality, integrity and trust-worthiness. We will integrate these findings to extend and validate our model of Key Impact Pathways for reproducibility derived from Tasks 3.1 and 3.2 which will also result in a framework for analysis of costs, tangible and non-tangible savings through reproducibility. Methods for Objective 5: RECOMMEND/REFLECT (roadmap & policy guidelines/recommendations)

Realised through WPs 2, 3:

  • Task 2.2. Community development and coordination of co-creation activities (M1-M36)
  • Task 1.3. Open and reproducible research practices (M1-36)
  • Task 3.3. Synthesis and recommendations (M13-M36)
  • Task 2.4. Project dissemination and communication (M1-M36)

Next, these synthesis activities will inform the creation of policy briefs, guidelines and recommendations for funders, publishers, research institutions and researchers. Firstly, we will reflect upon the challenges, costs and benefits resulting from reproducibility and transparency approaches in the context of TIER2, leading to a final “autoethnographic” self-reflection report that feeds into the synthesis of results obtained from the empirical work in WPs 3-5. As detailed below (sec. on Open Science and Research Data Management) TIER2 aims to be the change we seek by fostering a maximally open, transparent and reproducible approach to the implementation of Horizon Europe projects (beginning from right now, by making our proposal publicly available at the time of submission*11), and we will use techniques of continuous auto-ethnography during the project to collect and synthesise our collective experiences in this endeavour (e.g., with making our qualitative data maximally FAIR, pre-registering and publishing full protocols for all scoping and pilot activities, using the containerised workflow tools we create to making our data, code, software and algorithms open source and easily replicable).

We will then conduct co-creation workshops with individuals selected via purposive sampling from five stakeholder-categories (researchers, funders/RFOs, institutions/RPOs, infrastructures, and umbrella bodies/networks, including those who participated in the approaches implemented) to iteratively create recommendations and policy guidelines for practices and joint action by stakeholders for future training priorities. This will use a co-creative, modified Delphi methodology that combines three rounds of anonymised survey with four online consensus-seeking meetings to work with 30 community members. This methodology has recently been used by the respective Task 3.3. Leader (KNOW) to create the ON-MERRIT recommendations (Cole et al. 2022). While solutions will be applicable to Europe, attention will be paid to training in global science. This will result in an overall roadmap for future developments beyond the project. Success will be measured by the degree of consensus achieved across our co-creation community for each specific recommendation (target minimum 90% consensus for inclusion of each recommendation). In addition to an overall recommendations document, from these activities, key policy briefs addressing specific domains (life sci, comp sci, soc sci) and stakeholders (funders, publishers, institutions) will be distilled. All recommendations and suggested policy interventions will be devised with the involvement and critical feedback of key stakeholders, and validated via a range of channels, including a final TIER2 stakeholder workshop in the final stages of the project. The final aim will be to produce practical guidelines, checklists and policy briefings for EU, national, and institutional policy-makers and other stakeholders.

In parallel, we will engage in liaison activities to ensure maximum impact of the recommendations. A dissemination and communications strategy (Milestones) will guide this process, whereby the co-creation community will be engaged to disseminate feedback within networks, and key stakeholders will be engaged. We will work with major umbrella bodies (e.g., Science Europe, EUA), as well as funders, research institutions, infrastructures (e.g., ESFRIs clusters, EOSC), and other networks (e.g., Reproducibility Networks, RDA groups). With the support of relevant national and supranational organisations (e.g., EUA, Science Europe) we will ensure that policy briefs are distributed to the senior management of RPOs, the research governance teams of RPOs, and other relevant organisations (e.g., EARME, ENRIO). As a final validation step, we will seek endorsement of this vision from all the stakeholder groups mentioned. Success will be measured via the number of endorsing organisations, with a target KPI of at least 25 stakeholder endorsements. Methods for Objective 6: NETWORK/EMPOWER (capacity-building on skills, connections, resources)

Realised through WPs 1, 2:

  • Task 1.4. Strategic alignment activities (M1-M36)
  • Task 2.2. Community development and coordination of co-creation activities (M1-M36)
  • Task 2.3. Development of the Reproducibility Hub (M10-M36)
  • Task 2.4. Project dissemination and communication (M1-M36)

In order to equip researchers, funders, publishers and others with the skills, connections and resources to exploit state-of-the-art guidance, tools and services, we will undertake capacity-building activities throughout the project. Three clusters of activities will particularly add capacity to address reproducibility issues in Europe and beyond.

(a) Training & awareness: In addition to the learning and networking multiplication-effects for stakeholders engaged in our many co-creation activities described above, TIER2 will undertake a range of activities to boost reproducibility skills and awareness. We will host events including a final Reproducibility Conference (Month 34) and a minimum six “Reprohack” events (co-located with discipline-specific conferences) where researchers are engaged to reproduce the work of others.*12 In addition to the functionalities mentioned above, we will create five training modules to lead researchers from our target disciplines, plus funders and publishers, through basic reproducibility issues and guide them to advanced content available on the Hub and elsewhere. Training modules will be created building on the consortium’s rich experience in previous EC projects to create training materials for Research Integrity and Open Science, e.g., EnTIRE, VIRT2UE, SOPs4RI, FOSTER, OpenUP, and OpenAIRE.

(b) Reproducibility Hub: Firstly, all evidence, tools and resources created via the activities described above will be collected and made easily available via our Reproducibility Hub, a sustainable open (minimum CC BY) knowledge base of results, methodologies and interventions on the drivers and consequences of reproducibility for the R&I system hosted at the Embassy of Good Science. The Embassy, hosted by TIER2 partner VUmc, is already a central resource for researchers and others (recently specifically mentioned in many EC funding calls), whose stated goal “to promote research integrity among all those involved in research” aligns with aims of reproducibility (some resources on the subject are already available). The Embassy’s existing functionalities for resources, training materials, and community-building already offer the core functions needed for the Reproducibility Hub, which will be created as an Embassy sub-site, hosting content created across TIER2. Users will themselves be able to add or update content via the Embassy’s Wiki functionalities. Particular highlights will be training modules and checklists, inventories of reproducibility tools and practices for specific research fields (life, social, and computer sciences) and stakeholders (publishers, funders, institutions), guidance on best practice, and interactive elements that allow users to exchange good practice and tools. Created iteratively throughout the project as results become available, this resource will fill the main gaps in current knowledge and provide a platform for exchange regarding reproducibility.

(c) Aligning & empowering networks: A core principle of TIER2 is that action to increase reproducibility must be targeted holistically to boost capacity at all levels. Fortunately, much action is already underway. The last ten years has seen huge investment in infrastructure and services to enable Open Science and Research Integrity. TIER2 includes key Open Science players, most prominently OpenAIRE, the European Open Science infrastructure present with partners in more than 30 countries. Partners AU and VUmc, meanwhile, are key players in Research Integrity with strong links to relevant organisations (e.g., ENRIO). VUmc is host to the Embassy of Good Science, a key RI training and community-building resource. Meanwhile, networks of publishers and funders consider the same questions. Of particular importance to TIER2’s success in networking and empowering communities are:

  • Reproducibility Networks (RNs): Building from a model pioneered in the UK (UKRN), RNs now exist in many countries internationally. These national networks of lead researchers who spearhead activities within institutions, especially support grassroots and early-career activities (e.g., ReproducibiliTea journal clubs). ERNs coordinate sharing of best practices and organisation of joint training initiatives. Local networks include grassroots groups of researchers and institutions, and are supported by external stakeholders such as funders, publishers, and other scholarly organisations (incl. FAIRsharing). TIER2 will empower the RNs (including by fostering creation of three new RNs in “Widening Participation” countries) by funding establishment meetings (3 awards of 5k EUR made via an open call for networks of >3 institutions within a Widening country who wish to establish an RN). TIER2 will build upon the RNs’ existing advances through our excellent links (see section TIER2 partners Charite and UOXF are core partners in the German and UK RNs, and we have secured Letters of Interest confirming intent to collaborate from RNs in Australia, Brazil, Finland, Germany, Italy, Norway, Slovakia, Sweden, Switzerland, and UK.
  • Research infrastructures (incl. EOSC): The European Open Science Cloud is an EC-funded initiative to develop a “Web of FAIR Data and Services” and “provide European researchers, innovators, companies and citizens with a federated and open multi-disciplinary environment where they can publish, find and reuse data, tools and services for research, innovation and educational purposes.” Increasingly, national infrastructural investments like the German national research data infrastructure (NFDI) are also being designed explicitly to link to the EOSC. Tools and practices are EOSC-native (our term), when they are created to be interoperable and embedded in the EOSC from the outset and linked via relevant EOSC components and community-led infrastructures (e.g., eInfrastructures like TIER2 partner OpenAIRE, ESFRIs like CESSDA or ELIXIR (where GESIS, UOXF and FLEMING are key national nodes), and clusters such as EOSC-Life and SSHOC, where many of our partners also take part. As key players in this landscape, we are uniquely placed to ensure our tools are EOSC-interoperable, and to bring the many EOSC activities on standards, training, and interoperability into contact with the RNs and other networks.

Networking and linking such initiatives has the potential to play an out-sized role in increasing reproducibility. The TIER2 consortium’s close existing connections to such networks means we are uniquely placed to facilitate and strengthen these connections. By aligning such efforts, TIER2 will greatly add to synergies and network-effects between them to boost capacity at national and international levels. Relevant projects/initiatives and TIER2 links.

  • BERD@NFDI (GESIS) Integrated analysis platform, Infrastructure. Initiative to build a platform for collecting, processing, analysing & preserving business, economic & related data, with a focus on unstructured (big) data such as video, image, audio, text or mobile data.
  • Data 4Impact (ARC; 2017-2019) Assessment, Infrastructure, Impact. Developed new indicators for assessing research & innovation performance based on a data-driven approach.

  • EnTIRe (VUmc; 2017-2021) Platform for dissemination, Stakeholder hub, Research Integrity. Created the wiki-platform “Embassy of Good Science” to share knowledge on fostering Research Integrity.

  • EOSC-Life (UOXF; 2019-2023) Data sharing, Data & metadata standards, Infrastructure. Coordinated by ELIXIR, this cluster brings together 13 ESFRI Research infrastructures in the Health & Food domain to create an open collaborative digital space for life science in the EOSC.

  • EpistemicProgress in Humanities (VUmc; 2020-2023) Reproducibility, Replication. Studies reproducibility in the humanities, conducting theoretical, conceptual & empirical work.

  • IntelComp (ARC, OpenAIRE; 2021-) Science, Technology & Innovation policy, InfrastructureDevelops big data/AI tools to model & assist Science Technology & Innovation (STI) policy making.

  • NFDI4Datascience (GESIS; 2021-) Reproducibility, Research data lifecycle, Infrastructure. Initiative to support all steps of interdisciplinary research data lifecycles, including collecting/creating, processing, analysing, publishing, archiving, & reusing resources in Data Science & Artificial Intelligence.

  • ON-MERRIT (KNOW; 2019-2022) Open Science, Responsible research & innovation, Equity. Investigated dynamics of equity & inclusion in open & responsible research, including reproducibility.

  • OpenUP (KNOW, ARC; 2016-2019) Open Science, Open peer review, science communication. Built a framework for the review-disseminate-assess phases of the research life cycle to support & promote Open Science, focussing on Open Peer Review, innovative dissemination, & research impact measurement.

  • PathOS (ARC, KNOW, OpenAIRE; 2022-2025) Open Science for science, economy & society, Reusability. New EC HE project (starts Sep 2022). Quantifies the Key Impact Pathways of Open Science relating to the research system & its interrelations with economic & societal actors.

  • POIESIS (AU; 2022-2025) Research integrity, Open Science, Public trust in science. New EC HE project (starts Sep 2022). Studies the connections between public trust in Science, research integrity & open science practices.

  • RTD/2020/SC/010 - Reproducibility (ARC, KNOW; 2021-2022) Reproducibility. Tender study to assess the reproducibility of research results in EU Framework Programmes for Research.

  • SARI MOAP RTD/2019/SC/021 (ARC; 2020-202) Open Science, Impact, Open Access. Tender study, commissioned by DG RTD, on "Monitoring the Open Access Policy of Horizon 2020”.

  • SARS (ARC; 2020-2022) Horizon Europe Key Impact Pathways & related indicators, Open Science. Study to support the monitoring & evaluation of the Framework Programme for research & innovation along Key Impact Pathways.

  • SSHOC (GESIS; 2019-2022) Data sharing, Confidential data, EOSC. Coordinated by CESSDA, this cluster unites 51 organisations in developing the social sciences & humanities area of EOSC, encouraging secure environments for sharing & using sensitive/confidential data.

  • SOPs4RI (AU, VUmc; 2019-2022) Research Integrity, RE, guideline development. Fostering research Integrity in institutions (Research performing institutions & research funders).

  • VIRT2UE (VUmc; 2018-2021) Research Integrity. Developed a sustainable train-the-trainer programme for tailored ERI teaching across Europe, focusing on understanding & upholding the principles of the European Code of Conduct for Research Integrity.

  • UKRN UKRI-funded project (UOXF; 2019-2024) Reproducibility, community engagement. Peer-led consortium of 18 UK universities & several partners to drive uptake of open research practices for reproducible research, connected to a network of RNs. Other methodological building blocks


To bring reproducibility into its next phase, TIER2 convenes a unique constellation of experts in reproducibility issues in targeted research fields, processes of research culture change, capacity-building & knowledge infrastructures to propose a holistic, pragmatic methodology that critically addresses all these elements simultaneously. We include: Domain experts for the target communities (computer science [ARC, KNOW], life science [UOXF, Charite, FLEMING], social science [GESIS, AU, VUmc]); Experts in science and technology studies, research ethics and meta-research with profound understanding of political and social aspects of research culture reform (on e.g., skills, incentives, policies: KNOW, AU, VUmc); experts in socio-economic impact assessment (AU, KNOW); data science (ARC); standards for digital objects across disciplines (UOXF); Core technical expertise in infrastructure and services, including established links to national and EC infraspheres (especially EOSC - OpenAIRE, UOXF, GESIS, ARC). As outlined in the methodology above, integration of the interdisciplinary expertise present in TIER2 will happen at the junctions between objectives, when moving from concept and design to implementation and assessment, as well in particular in Objective 5 (recommend/reflect), where results will be integrated and synthesised, by taking in perspectives across disciplines. In addition, TIER2 will engage in continuous co-creation dialogue with researchers in many different scientific fields, bringing even broader perspectives to bear in the design, implementation and assessment of our activities.

Gender dimension & EDI (equity, diversity, inclusion) factors

Gender and diversity considerations will be reflected in all stages of TIER2: in the research design, methods, analyses, interpretation, dissemination and creation of guidelines and recommendations. As outlined in the provisional schema of elements to map reproducibility impact pathways (Fig. 3), socio-cultural factors such as ethnicity, gender, sexual orientation and socio-economic status are important context factors in considering a culture change towards increased reproducibility. Nosek (2019) Strategy for culture change in reproducibility relies on infrastructure and user experience to base further sustainable normative and rewarding practices. This is not realised, for example, if infrastructure is less accessible to researchers belonging to certain groups, or if how it is experienced differs substantially. We will therefore strive to have even gender balance in our recruitment for participants for community engagement, focus groups, and community co-creation activities, considering further socio-economic factors when applicable. This will ensure the voice of all is heard to increase reproducibility and understand the behaviours and needs across all segments of stakeholders. All surveys and subsequent analyses will record gender and further socio-cultural aspects (e.g., ethnicity, sexuality), to better understand, for example, how events and hackathons can be adapted to better suit the needs of all. The success of stakeholder engagement and community building also relies on understanding the needs and behaviours of those from a wide range of EU countries, therefore geographical location of stakeholders will play a strong role in our community building activities. Analysis of both the gender and location of stakeholders, as well as intersectionality of group membership, will give insight into how norms and relations between groups may play a role as barriers or enablers to increased reproducibility and how these factors contribute to effective implementation of tools. The tools developed and trialled in TIER2 include the use of algorithms. We will address the methods and data used to build tools to confirm there is no inherent gender or other discrimination biases in algorithms. All digital communications will adhere to Web Content Accessibility Guidelines (WCAG) to verify our materials can be accessed and utilised by people with disabilities. Furthermore, TIER2 will use the SAGER guidelines (Heidari et al. 2016) for gender- and sex-sensitive reporting and communication in science.

Open Science practices

TIER2 convenes experts in open and reproducible research practices. TIER2 partners are committed to Open Science as proven by their involvement, leadership and expertise in Open Science for the past decade. TIER2 will build on this expertise to follow current best practices to ensure Open Science embedded in all aspects of the project:

  • Project proposal made public immediately upon submission*13
  • Policies for early results sharing, quality processes for full documentation of methodologies and tools (incl. standards for data and use well-established ontologies) produced in WPs4/5, and for pre-registration of all pilots and interventions will be defined in the project handbook (D1.1) and monitored for adherence (Task1.3)
  • Data Management Plan developed and regularly updated (min. M6, M18) according to Horizon Europe template, using the OpenAIRE Argos service
  • OpenAIRE Guidelines used when publishing results to ensure all artefacts are linked for discovery; our tools will be based on open source technologies, and all code will be shared according to the FAIR4RS Principles
  • All possible data will be shared according to FAIR principles – where some information needs protection or anonymization, e.g. surveys, interviews, the metadata will be made “as open as possible, as closed as necessary”, with qualitative data suitably anonymised to meet ethical and data protection requirements, and shared via qualitative data repositories (TIER2 partners GESIS are world-leaders in such techniques)
  • Upstream engagement and participatory approaches will be used throughout TIER2, fostering open collaboration by actively enrolling users in a co-design and evaluation processes for co-creation with major stakeholders throughout
  • All publications pre-printed and made Open Access (with preference for venues practicing open peer review where possible). For OA, we will at a minimum publish in Green OA (self-archiving) deposited in an OpenAIRE compliant repository (e.g., Zenodo) at the time of submission (pre-prints) and update with Author Manuscript (post-print), or in Open Research Europe (ORE) which offers an open peer review process. Budget of 3000 EUR has been allocated per partner for publishing in APC gold OA journals.

Research Data Management

Through our expertise (see Table 19, section 3.3) on Research Data Management (RDM) and FAIRification of research objects, TIER2 will enable state-of-the-art practices to ensure effective management, preservation and sharing of all the research objects we create:

  • FAIR research objects: TIER2 will create a wide range of methodologies, tools and data. Input data will be derived through a range of databases (mainly open), desk research and interviews. Once the indicators produced in WP3 have been validated by the experts from the case studies, TIER2 partners Athena RC, CWTS, CNRS will collect and package all components used in the development of an indicator and publish in open repositories such as Zenodo, using well-established open type licenses (e.g., CC, GNU, GPL, MIT). Special attention will be given to output data that is derived from proprietary /commercial databases, where we ensure the inclusion of only aggregate/statistical data. The same will apply to evidence from interviews and surveys that, in compliance to privacy policies, will be processed in an anonymous way and made public only in aggregated form. Finally, any training material developed in WP5 will be published according to the emerging EOSC QA practices in the OpenAIRE Learning Portal.
  • Standards: TIER2 will use existing standards such as OpenAIRE Guidelines, DataCite, DDI to publish the research objects, adapting them to cover specific needs of the development of indicators. The landscape report in WP1 will be produced using the PRISMA framework, while interviews/survey results will be fully anonymised and published using appropriate repositories. TIER2 will additionally ensure that all metadata complies to RISIS, the emerging EOSC Interoperability Framework.
  • Reproducibility/Ethics: To ensure transparency, all algorithms and training sets developed in WP2/3 will be documented presenting assumptions, biases and limitations. Code will be published via GitHub-Zenodo integration and will be linked to publications and data.

Curation & storage/preservation: Even though TIER2 has foreseen actions for coordinated data management and FAIR publishing with allocated effort in WP1 led by Know-Center, partners will use their own facilities for storing intermediate data (from databases, surveys, interviews) with well-established and agreed upon procedures. GitHub will be used for sharing code. The estimated cost for effective RDM is in the order of 80K, spread across WP activities.

2 Impact

This section outlines how TIER2 aims to contribute to the outcomes and impacts described in the EC work programme, the likely scale and significance of this contribution, and our measures to maximise these impacts.

2.1 Project’s pathways towards impact

As the world emerges from the COVID-19 pandemic, economic pressures (and hence increased scrutiny of research funding) and the exposed consequences of lack of societal trust in science are of urgent importance. Increased reproducibility of results in the European Research Area and beyond will reduce inefficiencies, avoid repetition, maximise return on investment, prevent mistakes, and speed innovation to bring trust, integrity and efficiency to the ERA and global Research & Innovation (R&I) system in general. To set the scale of potential savings, according to Freedman et al. (2015), more than half of results in preclinical medicine were not reproducible. This will of course vary by discipline, but even if we assume only a fifth of overall results are irreproducible, this would still equate to 16bn EURs of the 80bn spent in Horizon 2020. The Horizon Dashboard indicates that our target domains (computer, life, social sciences) accounted for well over half of H2020 funding. With a conservative estimate of 20% irreproducible research, we can say that of the €95.5 Horizon Europe budget, approx. 10bn would be irreproducible research from our target domains. Hence, if TIER2 can achieve even a very modest 1% improvement in levels of reproducibility, this would still equate to savings of 100m EUR in Horizon Europe (5% would mean 500m saved). Such numbers are substantial, and do not take into account the wider effects on other domains to which our tools/practices will be applied post-project, or their effects on the global and EU-national R&I systems. Nor does it account for the wider systemic impacts of increased societal trust in research which would speed uptake of innovation, or the efficiency-savings accumulated to research not duplicated or wasted building on false results (Fig. 5).

Figure 5.  

TIER2's pathways to impact.

TIER2 sets out an ambitious yet pragmatic programme of activities founded in four fundamental principles:

  1. Reproducibility is as an opportunity not a crisis;
  2. Epistemic diversity (variation across modes of knowledge production and socio-technical contexts) must be centred;
  3. Evidence must be systematised for informed policy across contexts;
  4. Action must be targeted holistically to boost capacity at all levels.

To enact these principles, in TIER2 stakeholder communities of researchers, publishers and funders will positively inform

  • a new structured understanding of the nature of reproducibility (concrete interventions, drivers, barriers, gains and savings);
  • creation and implementation of effective new solutions at all levels (from technical to policy) to increase the reproducibility of R&I results;
  • empowered communities and networks whose new linkages and shared vision have powerful network effects that enable alignment and joint action on training, skills, infrastructure and more.

This will be done through innovative, applied research performed, providing tangible and feasible policy options. By the conclusion of the TIER2 project, Key Outputs will have been achieved which will already be translating to outcomes (mid-term benefits achieved during, or max two years after, the project). In turn, these outcomes will result in longer-term impacts (3+ years after project end). In this section we describe these Key Outputs and follow the causal chains (pathways) whereby we are certain they result in maximal short, mid and long-term impact (see Table 7 below).

Table 7.

TIER2 Key Outputs.

TIER2 Key Output

Lever(s) for change


KO1. Conceptual framework


Structured understanding, clarity of aims /means

Results: TIER2 will create a new framework for assessing reproducibility impact pathways across epistemic contexts, with synthesised findings on gains & savings possible via increased reproducibility. This will consolidate evidence on the state-of-the-art regarding evidence/uptake of reproducibility interventions, & provides inventory of reproducibility tools/practices across contexts. This enhanced theoretical/evidential basis will enable shared understanding and orientation on best practices to increase reproducibility.

KO2. Innovative tools & practices


Concrete solutions (policy-, technical- & practice-based), empowerment, innovation

Results: After TIER2, eight ground-breaking new tools have been successfully implemented & piloted activities successfully implemented. These concrete, innovative solutions for at the levels of policy, technology & practice will empower our key stakeholders to take action.

  • For researchers: Reproducibility Checklists; Reproducibility Management Planning tools; tools for reproducible workflows (software containerisation, workflow description, experiment packaging).
  • For publishers: Workflows for review of data/code; tools & standards for “threaded” publications; dashboard for monitoring of policies linked to reproducibility.
  • For funders: Reproducibility Promotion Plans; funder extension of tool for Reproducibility Management Plans; Reproducibility monitoring dashboard.

KO3. Increased capacity


Collaboration, alignment, joint action, skills

Results: TIER2 will empower individuals and networks to boost capacity for the long-term, including via

  1. Reproducibility Hub, a sustainable open knowledge base of results, methodologies & interventions on the drivers & consequences of reproducibility for the R&I system hosted at the Embassy of Good Science;
  2. Over 1000 researchers, publishers & funders engaged to increase skills via outreach & co-creation events (Reprohacks, training events, webinars, final conference) & Reproducibility Hub functionalities (e.g., Checklist tool, training modules);
  3. Empowered networks, with at least 3 new national Reproducibility Networks created in Widening Participation countries & all RNs linked with Open Science, Research Integrity, research infrastructure (EOSC), publisher/funder networks & co-creation communities for increased capacity & multiplier effects in EU & beyond.

The increased alignment and collaboration possible through these linkages, as well as increased capacity for action, will spur reproducibility across all actors.

KO4. Policy roadmap


Direction, momentum, sustainability, inclusivity

Results: TIER2 will finally result in a consolidated stakeholder roadmap on priorities for future reproducibility, including reform of reward & recognition structures. This will include practical policy & implementation recommendations/guidelines/briefs for research funders, institutions, policy-makers, publishers & researchers in Europe & beyond. By the end of the project 30 funders, institutions & networks will have endorsed the recommendations. This vision for future action will provide direction and momentum to unite stakeholders in sustainable efforts to address issues of reproducibility for the long-term.

2.1.1 TIER2 contributions to outcomes

In turn, these Key Outputs will directly lead to a host of short- to medium-term Outcomes which significantly address the main concerns in reproducibility. Fuelled by the dissemination and exploitation measures described below (Sec 2.2), by the end of (or max 3 years after) our project, the contributions outlined below in Table 8 will be implemented:

Table 8.

TIER2 contributions to outcomes.

Expected Outcome 1: “Structured understanding of the underlying drivers, of concrete and effective interventions - funding, community-based, technical and policy - to increase reproducibility of the results of R&I; & of their benefits

KO1 will bring new understanding of the impact of epistemic diversity on the meanings & implications of reproducibility. Mapping the evidence to this framework will result in structured understanding of key reproducibility impact pathways (which interventions work in which domains, with which gains & savings). A broader understanding of reproducibility across epistemic contexts will ensure a more holistic model to implementing & incentivising reproducibility initiatives at multiple levels including bottom-up & top-down approaches. Made available via the Reproducibility Hub (KO3), this analysis will then be a key resource for stakeholders interested in action on reproducibility. This will help stakeholders such as researchers, publishers, funders, & research administrators to prioritise tools & approaches for implementation (e.g., via incorporation into our Researcher Checklist & Reproducibility Promotion Plan (RePP) tools, KO2). By the project’s end, our Researcher Checklist will have been pilot-tested with 50 users, & used by thousands. 3 funders will have implemented RePPs building on our framework, with wider roll-out in the ensuing years (estimated 10 funders have RePPs 3 years post-project, with institutional RePPs a growing theme). In addition, we expect our Reproducibility Impact Pathways analysis to also become a key resource for meta-researchers concerned with monitoring reproducibility (to be monitored via citation analysis).

Expected Outcome 2: “Effective solutions, policy-, technical- and practice-based, to increase the reproducibility of R&I results in funding programmes, communities and dissemination

KO2 will make a range of new tools & practices available for researchers, publishers & funders. For researchers, our Researcher Checklist empowers & guides to raise awareness & skills; our Reproducibility Management Plan piloting will be the start of a new age of simplified Research Data Management & project reporting; new containerised workflow tools, extended into new areas of social & computer science, will simplify reproducible research pipelines. For publishers, new tools to enable review of data & code will raise review standards & hence the reproducibility of published research; new common standards for threaded publications will enable such incremental publishing to become the norm, combating publication bias & speeding dissemination; & the suitability of new models of publishing (publish then review, registered reports) will be better understood. For funders, Reproducibility Promotion Plans will become a standard tool for reproducibility strategy; Reproducibility Management Plan standards will enable more reproducible funded research & simplify reporting; & our Reproducibility Dashboard will enable easy monitoring of uptake of reproducibility-related practices. By the end of TIER2, these tools will have been piloted with >1000 users, with many in production & available via the Reproducibility Hub & EOSC (KO3), hence to be used by thousands more in the years after TIER2 ends.

Outcome 3: “Greater collaboration, alignment of practices and joint action by stakeholders to increase reproducibility, including but not limited to training, specialised careers and guidelines for best practice.

KO3 will bring together major building blocks of reproducibility, most prominently the Reproducibility Networks (empowered through expansion) & the EOSC, along with Research Infrastructures like ELIXIR, as well as networks of publishers & funders. TIER2 acts as the facilitatory link between researchers, RNs & other key networks (EOSC & FAIR data ecosystem) as well as funders & publishers through co-creation events, which build lasting networks through engagement to develop mutual solutions in reproducibility. Growing & networking the RNs will allow for greater alignment of activities across Europe. They serve as a platform to share lessons learned at different institutions & collectively, the considerations for implementing tools & practices in different geographical locations & in research ecosystems with varying socio-economic bases. Training modules on the Reproducibility Hub & our Reprohack events will raise skills amongst all stakeholders (>500 participants by project end). Communities engaged in co-creation to design, develop & pilot our tools (KO2) will also be brought together in mutual learning exercises. Through co-creation, “ownership” of tools created incentivizes individual players (researchers, publishers, funders) to advocate for tools within their community, & share best practices in use. Finally, our recommendations & policy instruments (KO4) will set out a powerful, co-created roadmap for future reproducibility in the year beyond the project, including a plan to sustain & empower networks.

2.1.2 TIER2 Contributions to impact

These outcomes will then lead to a multiplicity of longer-term impacts, corresponding to the impacts listed in the Destination programme of the EU. TIER2 will help reform and enhance the EU R&I system, improve access to excellence, deepen the ERA, modernise the higher education sector, increase interconnection of knowledge ecosystems, as well as addressing biases to make research careers more attractive and foster gender equality. At a societal level, higher reliability of research results underlying policy making processes will raise trust in science and R&I outcomes. Economically, greater quality of scientific production will lead to increased re-use of scientific results by research and innovation, and hence stronger translation of R&I results into the economy. Table 9 below details the outstanding contribution that TIER2, via the Outputs and Outcomes detailed above, will lead to sustainable change that addresses the expected impacts listed in this call:

Table 9.

TIER2 Contributions to impact.

Expected Impact 1: “Increased proportion of reproducible results from publicly funded R&I

Through its Key Impact Pathways methodology & increased awareness of the meanings, gains & savings of reproducibility across epistemic contexts (KO1), the new tools/practices created (KO2), empowered through increased capacity (KO3) & sustained & organised into the future by our policy roadmap (KO4), TIER2 will play a major role in optimising levels of reproducibility in the R&I system. Our mix of tools (KO2) includes pragmatic instruments whose implementation will have immediate effects, as well as more blue-sky thinking (Reproducibility Management Plans) which will be the seeds of change for years to come. Not only will our tools enable this increase, but also its monitoring & assessment (Reproducibility Dashboard, KO2).

Expected Impact 2: “Increased re-use of scientific results by research and innovation

By bringing our tools into the EOSC (KO3), we also link to the FAIR ecosystem that will enable re-use for the longer term. Increased confidence in the reliability & credibility of research results will further spur re-use, also by industry & other societal actors. Our capacity-building (KO3) & policy actions (KO4) will assist in aligning & motivating this re-use far beyond the end of TIER2. Over the long-term, this will mean that actions to foster reproducibility are linked, with network effects upon impact, to the wider discussion of re-use of results currently framed around FAIR data.

Expected Impact 3: “Greater quality of the scientific production

Increased reproducibility will come with greater quality of research results. Quality here should be taken in a broad sense, however: more reproducible results will result in less waste, in healthier research communities, & in more efficient & effective research endeavours. Giving the right tools (KO2) & recommendations (KO4) to organisations, funders & publishers, as well as the means to understand what quality means in relation to reproducibility across epistemic contexts (KO1) will hence help reform & enhance the EU R&I system, improve access to excellence & deepen the ERA. This will ultimately result in a more open & inclusive research & innovation system, with increased trust in research outcomes, greater quality of scientific production & stronger translation of R&I results into the economy, as well as more inclusive research environments for researchers.

2.1.3 Scale & significance, barriers & mitigation measures for impact of TIER2 Key Outputs

To realistically frame the certainty with which we expect our Key Outputs to lead to these Outcomes and Impacts, we next (Table 10) detail how we (1) perceive their scale (how widespread) and significance (how important); (2) the key barriers to this Impact and the mitigation measures we have in place to minimise them.

Table 10.

Scale & significance, barriers & mitigation measures for impact of TIER2 Key Outputs.

Key Output 1. Conceptual framework

Scale: TIER2’s conceptual framework has the potential to expand understanding of reproducibility gains & savings across epistemic contexts. Doing so will enable realistic mapping of impact pathways of reproducibility interventions across disciplines & regions, empowering effective decision-making for RPOs, funders & publishers. This framework also consolidates state-of-the-art evidence/advice on existing tools & practices that will be used by more than 1000 researchers during TIER2 & many thousands more afterwards.

Significance: These two contributions enable significant efficiency- & quality-gains through informed future policy. The results provided in this task will be applicable to a large range of research, with our case studies/pilot activities covering three key domains (computer, life, & social sciences). Results will be applicable to researchers in these domains, as well as publishers & funders on a global scale.

Key barrier: Framework too theoretical & not applicable to real-life contexts

Mitigation: TIER2 uses co-creation with relevant stakeholders & evidence accumulated throughout the project to inform development of the framework & ensure real-world relevance/impact.

Key Output 2. Innovative Tools & practices for diverse contexts

Scale: TIER2 will develop 8 new tools & practices for researchers, funders, institutions, & publishers, which will be applicable to key disciplinary areas & ready for extension to many more after the project. The impact of these tools over the long-term will be immeasurable. We expect, for example. our Reproducibility Promotion Plans for funders to become a de facto standard for priority planning, & our workflows for review of data/code will be instrumental in ensuring & validating the reproducibility & re-usability of those objects.

Significance: Our tools target all elements of the research lifecycle from project planning to assessment & hence are of relevance to many thousands of researchers, funders & publishers. Closely aligning these tools with the Embassy of Good science & EOSC will result in fast & efficient uptake.

Key barrier: Tools/practices not fit for purpose

Mitigation: We have selected this provisional list of tools as priorities for development based on state-of-the-art evidence (sec. 1.2). They will be refined, developed & piloted through co-creation with users, & a solid framework for assessment will be used to evaluate & showcase efficacy. Targeted exploitation & dissemination measures, including via our Reproducibility Hub & EOSC will ensure uptake.

Key Output 3. Increased Capacity

Scale: National Reproducibility Networks are powerful networks which reach directly to institutions & across Europe & the globe. Growing aligning this with infrastructural investments like EOSC (250m EUR invested 2018-2020*14), & international networks of funders, publishers & RPOs, mean TIER2 will have powerful multiplier effects in building capacity. Our training & skills resources, hosted for the long-term by the sustainable Embassy of Good Science (over 2500 monthly visitors) will develop further awareness amongst individuals & bring them into this community, meaning that TIER2 will potentially affect the reproducibility practices of hundreds of thousands in the years after the project.

Significance: Under this objective, practical policy & implementation recommendations/guidelines for research funders, institutions, policy-makers, publishers & researchers in Europe & beyond will be developed. The recommendations & guidelines will reach a minimum of 1000 researchers, we aim for adoption of our policy recommendations in at least 4 funders, & 2 publishers through the dissemination activities outlined under section 2.2. This will lead to increased awareness among those stakeholders, & to concrete changes, jump-started by the pilot implementations. We expect pilot activities to be taken up well by stakeholders, given the co-created approach taken to the pilots. We therefore further expect 4 funders & 2 main publishers to continue using the tools & practices developed throughout the project, maximising impact via their own reach.

Key barrier: Increased capacity not carried forward once project-funding & TIER2 activities are over

Mitigation: By seeking to embed our gains in the interlinking of the RNs & EOSC, which both will only grow in importance in the coming years, we are certain our capacity-gains will be sustainable for the long-term, also enabled via our teaming with funder-groups like Science Europe & EVIR (to foster Reproducibility Promotion Plans) as well as publisher-groups like STM Association.

Key Output 4. Policy roadmap (guidelines & recommendations)

Scale: Key themes in TIER2 are epistemic diversity, the need to discern Impact Pathways to optimise reproducibility, & collaboration for network effects. Carrying these into the future will be heavily influenced by the success of our policy recommendations. If done well, they will bind stakeholders in collaborative action to optimise reproducibility in the right ways for the right contexts. This will ensure the prioritisation of reproducibility as a policy priority for funders, publishers, RPOs & others, having the potential to sustain & amplify TIER2’s gains for the long-term.

Significance: We started by noting a conservative calculation that enabling a 1% increase in Horizon Europe would result in savings of 100m EUR. Extrapolating such figures to the global R&I system shows the massive potential significance of TIER2, if momentum & direction is sustained in the correct way beyond the project. Our policy recommendations, co-created with stakeholders, will ensure this happens.

Key barrier: Low acceptance of policy recommendations amongst major stakeholders

Mitigation measure: TIER2 is community-based from the start. Through co-developing our framework & tools with stakeholders, to targeted capacity-building actions, & finally co-creating the recommendations with stakeholders, we are assured the recommendations target the reproducible future stakeholders wish to see. An effective & dedicated exploitation & dissemination plan, in which major stakeholders not already involved in recommendation co-creation are kept informed & engaged, will ensure maximal impact upon release & into the future.

2.2 Measures to maximise impact - Dissemination, exploitation & communication

Effective communication and dissemination processes are crucial for the success of TIER2. Mobilising a strong communication team from PENSOFT, in tandem with OpenAIRE (with outreach to 30+ countries), TIER2 brings together the leaders of excellent international consortia and networks, including the leaders of a large number of European projects on Open Science, Research Infrastructures and Research Integrity. The consortium includes members from diverse disciplines and regions. More importantly, via our established networks, we have direct access to important global networks of researchers, publishers and funders. These previous relations and connections will facilitate the building of a network of stakeholders co-creating, validating and spreading TIER2 output. TIER2’s Scientific Advisory Board further expands our links via the international networks of established experts. Table 11 provides an overview of TIER2’s stakeholder groups. These groups will all play a key role in the project: engaging in co-creation activities; co-organising events; taking up our tools, practices, and training; and amplifying/disseminating our messages.

Table 11.

TIER2 key stakeholders.




Social sciences: Links via GESIS (+SSHOC, CESSDA), AU, and via key scholarly societies for target methods; Life sciences: Links via UOXF (+EOSC-Life, ELIXIR), Charite, FLEMING, key scholarly societies; Computer science: Strong links via ARC, KNOW to IEEE, ACM, EUHubs4Data; additional outreach via promotional campaign (esp. for participation in co-creation activities) and presence at disciplinary conferences (Reprohacks)

Funders (RFOs)

Strong links via OpenAIRE, VUmc, to: HERA, EViR, Science Europe, Open Research Funders Group; ARC link to ARCHIMEDES (AI funding); additional outreach via promotional campaign (esp. for participation in co-creation activities)


Strong links via OpenAIRE, UOXF, PENSOFT, KNOW, AB (Catriona Maccallum) to: OASPA, STM Assoc, COPE; additional outreach via promotional campaign (esp. for participation in co-creation activities)

Reproducibility Networks

Strong links via Charite, UOXF, Advisory Board, and collected Letters of Interest to RNs in Germany, UK, Slovakia, Brazil (RN forthcoming), Switzerland, Sweden, Norway, Finland, Italy; targeted outreach via Task 1.4 (strategic alignment)

Research Infrastructures (EOSC)

Strong links via OpenAIRE, UOXF, GESIS, FLEMING, KNOW to: EOSC Association; EOSC clusters (EOSC-Life, SSHOC); EOSC-Future; RDA; ELIXIR, CESSDA, ESFRI & ERIC Forum; targeted outreach via Task 1.4 (strategic alignment)

Research Performing Organisations (RPOs)

Strong links via OpenAIRE & others to: EUA, LERU, CAESAR, AURORA, YERUN; targeted outreach via Task 1.4 (strategic alignment)

Research Integrity officers/trainers

Strong links via VUmc, AU to: EARMA, ERION, ENRIO, building on existing close relationships via EnTIRE, SOPs4RI, POIESIS, NRIN, VIRT2UE, iRECs; targeted outreach via Task 1.4 (strategic alignment)

Scholarly/learned societies

Strong links via OpenAIRE and all partners to scholarly societies in a range of disciplines

Libraries & library organisations

Strong links via OpenAIRE to LIBER, EIFL, SPARC; targeted outreach via Task 1.4 (strategic alignment)

Research administrators

Links to iNorms, EARMA

Civil Society Organisations/General public

General media (via press releases), blogs & online media, online courses, newsletter, videos, special interest groups


Industry events, industry media, EC events

2.2.1 Dissemination & communication plan

Purposefully designed dissemination, exploitation, and communication (DEC) activities are key components for maximising the impact of TIER2. The project’s DEC activities will be streamlined in a Communication Plan Stakeholder Communication and Engagement Plan (D2.1). The plan will serve as a management tool for defining how the project’s progress and results are shared with stakeholders and target audiences. Two updates of the plan (in M24 and M36) will include any necessary modification and adapt appropriately to project progress and new circumstances, including feedback received by co-creation activities. The engagement plan will identify target audiences (‘who’), project research activities and outputs (‘what’), tools and channels employed (‘how’), and the envisioned timeline for implementation of these activities (‘when’). All activities have the objective to maximise the short-, medium- and long-term impact of TIER2 results. In addition, the plan will contain: TIER2 visual identity (logo and graphical layouts guidelines, templates); Communication/dissemination targets, clear distinction between key end users, broad outreach, and general public; Illustration of materials to be produced, communication formats, and online and social media channels which will be used according to the target categories; Indication of how co-creation will be supported by other dissemination channels and tools in WP2; Indication of how external events to be targeted by the project’s dissemination will be selected; Assigning of roles and responsibilities of individual participants in project dissemination; Schedule for the implementation of dissemination and communication activities; Identification of all project outputs and anticipated outcomes, with measurable indicators.

To maximise impacts, TIER2 will actively target its dissemination and communication activities at audiences and stakeholders as defined above. TIER2 will tailor various uni- and bi-directional dissemination channels to the needs of the target stakeholders and audiences, eliciting expertise, knowledge, and perceptions from stakeholders as part of the project’s co-creation and engagement activities. A preliminary mapping of dissemination tools and channels, their target audiences, and relevant Key Performance Indicators (KPIs) is shown in section 2.2.2 and will be refined in D2.1. To effectively disseminate TIER2 results, specific dissemination tools and channels will be established by the project and existing channels used by the consortium participants will be fully exploited. Furthermore, consortium participants will use their existing networks and develop new links, presenting TIER2 and its results at relevant events at local, national, and international scales.

For TIER2 internal communication, mailing lists and communication/document exchange tools, such as MS Teams, will be used as part of a web-enabled communications and learning platform, to create a chat-like environment that simultaneously eases communication and streamlines information and access to documents into relevant channels to alleviate workload and stimulate fruitful and focused discussions. It will provide a place where the TIER2 project team and collaborators can communicate, share documents and work together. Regular consortium and WP calls will ensure timely communication on all issues.

2.2.2 TIER2 dissemination & communication channels

Branding: We will develop a brand identity for the TIER2 website, deliverables, oral presentation & poster templates. TIER2’s brand identity will consist of logo, colour set & typography to be utilised in all types of communication activities. KPIs: 6 templates for TIER2 documents, reports, presentations, posters & infographics by M6.

Reproducibility Hub, via Embassy of Good Science: Since TIER2 partner VUmc is the host of the Embassy, we will assure the dissemination of all outputs on this wiki-platform. We will also use the newly developed stakeholder hub to engage a large stakeholder community. KPI: >1000 monthly visits by M36.

Website: The TIER2 website will be launched to provide up-to-date information on the project, partners, progress, goals, events & outputs (e.g., deliverables & peer reviewed publications) & create regular news updates for the general public. KPI: >1000 monthly visits by M36.

Social media: Social media are instrumental in reaching all relevant stakeholders. TIER2 will utilise LinkedIn & Twitter to communicate TIER2’s progress, both for the stakeholders, but also for the general public. KPIs: >1000 Twitter followers (>500 total Tweets); >500 LinkedIn followers (>60 posts) by M36

Podcasts, video tutorials, infographics: In year 1, develop a video for project visual presentation & promotion, updated with results on year 3. Present via podcasts interviews & stories. KPI: 2 project videos by M36

Co-creation workshops & events: Organisation of: 22 co-creation workshops (minimum 5 in-person) creation of tools/practices (design, testing, piloting, assessment), impact pathways methodology & final recommendations; 6 webinars on TIER2 themes; 6 Reprohack events at domain-specific conferences; one final conference with relevant stakeholder representatives, in order to disseminate the major outcomes of the project. KPI: >1000 researchers, funders & publishers engaged via TIER2 events.

External conferences/workshops: TIER2 consortium members will participate in conferences & interact with experts in the field of RI, RRI & OS, & exchange experiences with relevant stakeholders. TIER2 participants will actively participate in other relevant running EU-funded project (H2020 & HE) workshops/events & cluster events organised by the EC. KPI: >30 conference/workshop participations by M36

Public outreach events: TIER2 partners will participate in open lectures in science museums, participate in Researcher’s Night events & in Science Communication events. KPI: >10 events where TIER2 is represented by M36

Press releases: Press releases, targeting newspapers with national circulation written in national languages will boost the project’s communication of the latest findings nationally & internationally scale. KPI: >6 PRs by M36

Policy briefs: TIER2 will create policy briefs for funders, publishers, RPOs & other stakeholders, especially highlighting our Impact Pathways framework & co-created recommendations (D3.1/3.2). Target: >5 policy briefs by M36

Scientific publications: Publications in leading peer reviewed journals related to reproducibility, Research Integrity, research policy, & scholarly infrastructure. KPI: >15 publications by M36

2.2.3 Exploitation plan

To maximise exposure of project results and their potential for exploitation, TIER2 will take advantage of the EC’s Horizon Results Platform. This platform will serve as a bridge towards researchers and other stakeholders, giving access to the project’s main and prioritised results with a high potential value (Key Exploitable Results). In addition, the project will consider the Horizon Results Booster for dissemination and exploitation of results so that the added value of the KER is amplified through publication in RIO journal, to enhance findability, accessibility, interoperability, and reusability (FAIR), and secure distribution beyond the project’s website. Research outputs described in Table 12 will be used as a basis for a complete Key Exploitable Results table to be included in the Stakeholder Communication and Engagement Plan (D2.1), and which will be updated throughout the project. Table 12 provides an overview of TIER2’s Key Exploitable Results (KERs), as well as their pathways for exploitation.

Table 12.

Key Exploitable Results & appropriate dissemination measures.

Key Exploitable Results

Target audiences

Exploitation route

KO1. Reproducibility impact pathways framework

A framework for assessing reproducibility impact pathways across epistemic contexts, with synthesised findings on gains & savings.

Primary: Research funders, Research publishers, RPOs, Scholarly/learned societies, Scientists & researchers

Secondary: University administrators, Civil Society Organisations, Knowledge-transfer offices

Enable systematic assessment of the impact of reproducibility interventions across contexts to optimize action; enable assessment indicators; provide ideas/inspiration to stakeholders for actions on reproducibility

KO2. Innovative tools & practices

Eight ground-breaking new tools for researchers, funders & publishers.

Primary: Scientists & researchers, Research funders, Research publishers

Secondary: Scholarly/learned societies, Research communities, Libraries & library organisations, University administrators, Civil Society Organisations, industry

Showcase tools via Reproducibility Hub (Embassy of Good Science), EOSC & RN websites; disseminate widely in respective researcher, funder & publisher communities; highlight good practice; TIER2 partners will take up relevant tools within own practices

KO3. Network & increased capacity Sustainable & aligned networks, online training/skills resources.

Primary: Scientists & researchers, Research funders, Research publishers, RPOs, Scholarly/learned societies

Secondary: Research communities, Libraries & library organisations, University administrators, Civil Society Organisations, industry

Enable multiplier effects through linkage of networks (RNs linked with Open Science, Research Integrity, research infrastructure (EOSC), publisher/funder networks); provide knowledge & training both online & in-person; RI training among researchers

KO4. Policy roadmap

Consolidated stakeholder roadmap on priorities for future reproducibility, including reform of reward & recognition structures.

Primary: Research funders, Research publishers, RPOs, Scholarly/learned societies

Secondary: Scientists & researchers, Research communities, Libraries & library organisations, University administrators, Civil Society Organisations, industry

Create practical policy & implementation recommendations/guidelines/briefs for research funders, institutions, policy-makers, publishers & researchers in Europe & beyond. By the end of the project 30 funders, institutions & networks will have endorsed the recommendations.

2.2.4 IPR management

TIER2 will develop its Intellectual Property framework (IP) in connection with its Data Management Plan (DMP). Management of intellectual property and foreseen protection measures Knowledge generated will be managed in compliance with the Consortium Agreement (CA), which will be signed at the beginning of the project. The CA will address background and foreground knowledge, ownership, protected third party components, and protection, use, and dissemination of results and access rights. The principles are:

  1. Background information and knowledge contributed to the project by each participant will be listed in the CA. When included in the work plan, access to background information will be provided royalty-free to other participants for the implementation of the project’s tasks;
  2. Results shall be owned by the participant who generated them. Each participant will be responsible for ensuring fulfilment of their obligations under the GA regarding results by planning with any third parties that could claim rights to them;
  3. Whenever results have been produced jointly by two or more participants, the ownership of the results will be shared among the participants who carried out the work. The terms of joint ownership, protection, and share of ownership, and costs for possible protection will be agreed upon in writing via a joint ownership agreement;
  4. Each participant will be responsible for examining possibilities to protect results that may be commercially or industrially exploited. When deciding on protection, the participant must consider its own legitimate interests and the interests of the other participants. Participants will ensure that adequate steps towards protection are taken prior to DEC activities, preventing unapproved public disclosure of results, models, tools, and data;
  5. Access rights to results will be granted on a royalty-free basis for further research and commercial exploitation. Results and outputs generated by TIER2 can be used freely upon acknowledgment of TIER2’s ownership of these.

2.3 Summary

In Table 13 you can find a summary of key elements of TIER2's impact pathways and of the measures we will take to maximise this impact.

Table 13.





Discussions on reproducibility driven by a narrow range of disciplines. Common under­standing of pathways to maximise reproducibility across epistemic approaches is highly needed

Need sufficient capacity (network, training, skills & trustworthy policy) for reproducibility

Reproducibility of results relies on tools & practices, which need to be further developed to increase usability & applicability across research contexts

Communication & Dissemination

Co-creation activities

Capacity building

Engage with experts

Exploit partner networks

Engage networks of publishers, funders & RPOs

Disseminate to scientific outlets

Social media engagement

Project dedicated website


Promote repro. promotion plans

Promote recommendations

Promote tools & methodologies across scientific communities

Reproducibility Hub / EOSC

Available EC tools

KO1. Reproducibility impact pathways framework: A framework for assessing reproducibility impact pathways across epistemic contexts, with synthesised findings on gains & savings

KO2. Innovative tools & practices: Eight ground-breaking new tools for researchers, funders & publishers

KO3. Network & increased capacity:

Sustainable & aligned networks, online training/skills resources

KO4. Policy roadmap:

Consolidated stakeholder roadmap on priorities for future reproducibility, including reform of reward & recognition structures




Researchers (soc, life, comp sci)



Reproducibility Networks

EOSC & Research Infrastructures

Research Performing Organisations (RPOs)

Research Integrity officers / trainers,

Scholarly/learned societies

Libraries & library organisations

Research administrators

Civil Society Organisations / General public


Reproducibility Hub & Impact Pathways analysis providing structured understanding of the drivers, benefits and effective interventions to increase reproducibility of the results of R&I

Eight new tools providing new, effective solutions (policy, technical & practice), to increase the reproducibility of R&I results in funding, publishing, and communities (soc, life, comp sci)

Boosted capacity through greater collaboration, training, alignment of practices & joint action by stakeholders (e.g., 3 new RNs, increased alignment with EOSC)

Trust, Integrity, Efficiency


Increased proportion of reproducible results from publicly funded R&I

Greater quality of scientific production

More open and inclusive R&I

Increased capacity for open science

Greater research integrity


Higher reliability of research results underlying policy making processes

Increased trust in science and R&I outcomes


Increased re-use of scientific results by research and innovation

Stronger translation of R&I results into the economy

3 Quality & efficiency of the implementation

This section provides a detailed description of the TIER2 work plan, including Work Packages, Deliverables, Milestones and key dates.

3.1 Work plan & resources

TIER2 is structured in five Work Packages (WPs, see Figs 6, 7) designed to achieve the objectives described in section 1.1. The WPs are nested such that the next builds on, and delivers outputs back to, each preceding WP. WP1 manages project implementation processes. WP2 oversees building of co-creation communities and project dissemination. WP3 builds the theoretical backbone of the project and translates its findings into actionable recommendation beyond the project. WP4 co-designs new tools and practices with stakeholders, and then oversees the piloting and assessment of these new tools. WP5 builds the tools designed by WP4. Each package has checkpoints (described inside the WP tables) to ensure proper alignment.

Figure 6.  


Figure 7.  

TIER2 Gantt chart

Management: WP1 will enable a smooth operation and delivery of TIER2 by setting a management structure comprised of the following teams: the Executive Board (EB) with WP leaders as members, held monthly chaired by KNOW; the General Assembly (GA) with one representative from each partner, meeting at M1 M12 and M24, chaired by the Project Coordinator (PC); a Quality Assurance team run by the PC and assisted by ARC setting the procedures, the structure and the quality indicators to monitor and ensure a seamless and high-quality delivery of TIER2 and its results; an exploitation team led by PENSOFT and OpenAIRE (WP2); an international Advisory Board providing guidance & alignment.

3.2 Work package descriptions

In Table 14 you can find the description of Work Package 1.

Table 14.

Work package descriptions 1.

WP No.


Lead beneficiary


Start month


End month


WP title

Coordination and Management


This work package aims to oversee & coordinate the financial, & day-to-day activities of TIER2 to ensure the achievement of the project’s overall aims & objectives. Specifically, it will coordinate all activities in the project & ensure that partners work together effectively & facilitate the scientific, research & impact success of the project through careful monitoring of progress, management of resources, open communication, & coordination of activities within the time & budget constraints of the project. Additionally, it ensures the project meets its contractual commitments & predicts, mitigates & overcomes project-related risks. WP1 develops plans to ensure the project activities meet the highest ethical, legal, integrity & quality assurance standards. It provides efficient operational management support including administrative & financial planning, reporting to the EC, & management of project legal aspects including project-related contracts & IPR, & management of day-to-day operational & technical progress. Finally, it sets up an international advisory board.

Description of work

Task 1.1 Scientific and administrative coordination (M1-36; KNOW [lead], ARC, VUmc)

Scientific & administrative coordination tasks include the following:

  1. Coordinate the Grant Agreement, Consortium Agreement, & signature procedures;
  2. Develop, disseminate & monitor adherence to a Project Management, Research Integrity Plan, a Quality Assurance Plan, Data Management Plan, & a Publications Agreement;
  3. Coordinate ethics deliverables;
  4. Support communication & collaboration between WPs, & the alignment & integration of work between them;
  5. Implement & maintain internal reporting & monitoring procedures (including internal review procedures, gender distribution in the consortium composition, & sex, gender & diversity considerations in the project activities);
  6. Monitor & support the completion on schedule of deliverables & milestones & take corrective action in the case of delays;
  7. Mediate in cases of internal disputes;
  8. Organise the General Assembly, Executive Board & Advisory Board meetings
  9. Handle daily project correspondence & requests from partners & external bodies;
  10. Produce periodic & final reports for the EC.

Task 1.2 Financial Coordination (M1-36; KNOW [lead])

Financial coordination tasks include:

  1. Calculate & distribute EC payments to project beneficiaries according to the Consortium Agreement;
  2. Liaise with finance departments to monitor contracts;
  3. Maintain financial records;
  4. Coordinate financial statements submission by all project partners;
  5. Provide financial overviews for the periodic & final reports for the EC.

Task 1.3. TIER2 Open and reproducible research practices (M1-36; ARC [lead], KNOW, AU)

Throughout the project, TIER2 will itself adhere to radical reproducibility & transparency. This task will ensure best practices, including adherence to Horizon Europe requirements on Research Data Management & Open Science, & take an auto-ethnographic approach to study our own project as a case study. It will ensure TIER2 adheres to state-of-the-art practices for reproducibility (e.g., pre-registration, reporting of negative results, sharing of protocols, methods, code, data). It will reflect upon the challenges, costs & benefits resulting from such reproducibility & transparency approaches in the context of international, multidisciplinary project consortia. This will lead to a final self-reflection report & feed into the synthesis of results obtained from the empirical work in WPs 3-5.

Task 1.4. Strategic alignment activities (M1-M36; OpenAIRE [lead], KNOW, Charite)

A unique selling point for TIER2 is our position at the nexus of various existing networks & initiatives (Reproducibility Networks, EOSC, other infrastructures, funder/publisher networks) whose effective alignment will bring powerful network effects. As two projects are foreseen to be funded via this HORIZON-WIDERA-2022-ERA-01-41 call, we also identify effective collaboration with the other project to be funded as a priority to avoid overlaps & duplications. This task ensures this alignment, coordinated via OpenAIRE & its network of actors in more than 30 countries covering the EC & beyond.


D1.1 Project handbook (including management, research integrity & quality assurance plans) (KNOW, M4)

D1.2 Data Management Plan (ARC, M6)

D1.3 Data Management Plan (Update) (ARC, M18)

D1.4 Autoethnographic reflections on implementing radical reproducibility in the TIER2 project (AU, M36)

In Table 15 you can find the description of Work Package 2.

Table 15.

Work package descriptions 2.

WP No.


Lead beneficiary


Start month


End month


WP title

Communities, Communication and Dissemination


This work package builds the TIER2 network to coordinate community engagement, co-creation activities, fluid outreach/dissemination & hence build capacity for skills & knowledge amongst all actors. This includes building communities & networks to foster & strengthen communication with all key stakeholders in reproducibility & increase awareness/skills for reproducibility tools & practices across disciplines. The WP will: Create a stakeholder map & engagement plan; Co-ordinate the co-creation communities for WPs 3, 4 & 5; Create the Reproducibility Hub to showcase state-of-the-art knowledge, tools & practices for reproducibility; Ensure effective communication (within the project) & dissemination of outcomes (beyond the project).

Description of work

Task 2.1. Stakeholder mapping (M1-M3; KNOW [lead], OpenAIRE, Charite)

As a first task in TIER, we conduct in-depth stakeholder analysis to identify the key players & stakeholders for engagement in co-creation, communication, dissemination & exploitation (cf. provisional analysis section 2.2.2). Using the 5Rs framework, we will analyse each stakeholder’s potential impact & influence within the project (incl. “push-pull” factors) to identify key stakeholders, structures of decision-making, where relationships can/should be strengthened, & resources required. This process will explore/identify key channels/methods/platforms for engagement with each group. The Stakeholder Map will be a key resource underpinning all co-creation & communication/dissemination activities, help structure Task 3.2 (evidence mapping), & underpin classification of resources on the Reproducibility Hub (Task 2.3).

Task 2.2. Community development and coordination of co-creation activities (M1-M36; Charite [lead], KNOW, VUmc, PENSOFT, OpenAIRE, UOXF)

Co-creation is an essential element of TIER2, with co-design & piloting activities with communities of researchers (soc, life, comp sci), publishers & funders, forming the backbone of the project. To ensure successful community-management & effective coordination of activities across WPs3-5, we hence dedicate a specific task. Building heavily on our existing links to the Reproducibility Networks, EOSC (incl. EOSC-Life & SSHOC) & publisher/funder fora, we use platforms/ strategies outlined in D2.1 “Stakeholder Communication & Engagement Plan” to offer co-creation communities a unique value proposition underpinned by support activities to facilitate engaged participation (as outlined in sec Within each case study domain, we will identify groups of reproducibility “champions” to spread awareness of initiatives & tools. We will hold virtual brainstorming events or “BarCamps” to co-create whitepapers on topics such as needs-gap analyses, barriers & enabler assessments, & virtual “co-working” events or “hackathons” to promote & improve reproducibility tools developed & piloted in WPs4/5. To build the network of Reproducibility Networks, we will hold an open call (deadline M9, eligible only to Widening Participation countries), where consortia of 3 or more institutions within a county will be able to apply for one of 3 awards of 5k EUR to fund a first meeting to establish an RN in their country.

Task 2.3. Development of the Reproducibility Hub (M10-M36; VUmc [lead], KNOW, GESIS, OpenAIRE)

This task will develop the Reproducibility Hub as a sub-site of The Embassy of Good Science (hosted by task leader VUmc). The Reproducibility Hub will iteratively combine content from across the project as a knowledge-base of reproducibility tools & best practices for TIER2 stakeholders (researchers, funders, publishers & beyond). The Hub, a wiki-based web-based platform, will use interactive modes of presentation, combining curated checklists, community engagement functionalities, five introductory training modules & in-depth articles on key topics & training materials. FAIRsharing & OpenAIRE data will power interactive graphs that visualise the landscape of reporting standards & best practices (for data, metadata & software), & their relations, as well as their use (by the EOSC clusters) & their adoption by data policies (by funders & publishers). The Hub’s relevance beyond the project’s lifetime will be ensured by its curation within The Embassy, whose governing foundation is already committed to its long-term sustainability. To this end, activities will include workshops hosted at domain-specific conferences to boost dissemination to stakeholders & to provide training on how to contribute to the Hub's content. We will leverage network effects offered by communities from OpenAIRE, FAIRsharing (including the newly launched EOSC-Future/RDA FAIRsharing Community Curation Programme) & the Reproducibility Networks to initiate community-led processes to update content beyond the project’s runtime. In developing the Reproducibility Hub & engaging the wider community to ensure its success, we will strive to leverage synergies with any other EC projects, especially the second project funded under this call.

Task 2.4. Project dissemination and communication (M1-M36; PENSOFT [lead], KNOW, ARC, VUmc, AU, OpenAIRE, Charite)

This task will manage all TIER2 dissemination & communication activities. Building on the provisional dissemination & communication plan described in Section 2.2, this task will ensure the project is optimally promoted to targeted audiences & to the most important stakeholders across Europe. The revised dissemination plan will outline a strategic communication campaign including a social media strategy, which will be updated periodically (D2.1) to ensure targeted communication strategies for the different project phases. Specific activities include the development of a dissemination toolkit, including a professional corporate identity, communication & dissemination materials as well as templates to be used for presentations, reports, meetings & internal documents, within the first six months of the project. This includes a public project website, which will showcase the project’s activities & results both to the scientific/policy community as well as to the public at large. It will coordinate regular dissemination of project progress & results via press releases, news entries & blogs. Further dissemination materials will be created for the purpose of communicating project activities & results (e.g., posters, stickers & other materials for events). As part of the communication plan, publication guidelines will be developed that regulate authorship & specify measures to ensure transparency of methods & reproducibility of project results. This task will also provide organisational support for TIER2’s final project cross-stakeholder conference. The conference will provide a forum to showcase TIER2 results, share reflections on the team’s practices in terms of reproducible research, & present the recommendations from Task 3.3.


D2.1 Stakeholder Communication & Engagement Plan (PENSOFT, M6)

D2.2 Stakeholder Communication & Engagement Plan (First Update) (PENSOFT, M24)

D2.3 Reproducibility Hub (Resource/website) (VUmc, M36)

D2.4 Stakeholder Communication & Engagement Plan (Second Update) (PENSOFT, M36)

D2.5 Policy Briefing 1 (KNOW, M12)

D2.6 Policy Briefing 2 (KNOW, M36)


MS2.1 Reproducibility Hub (Beta version) (M15)

MS2.2 Final self-reflection report on co-creation processes (M34)

MS2.3 Final conference (M36)

MS2.4 Website and Logo (M2)

In Table 16 you can find the description of Work Package 3.

Table 16.

Work package descriptions 3.

WP No.


Lead beneficiary


Start month


End month


WP title

Concept, evidence, synthesis and recommendations


WP3 provides the methodological & conceptual framework for TIER2. As a first task in the project as a whole, we build on the work of Leonelli (2018), Leonelli (2021) to create a conceptual framework for assessing impact pathways to increase reproducibility across epistemic contexts. We then use this framework to map evidence, including drivers/barriers & an inventory of tools/practices, regarding reproducibility across epistemic contexts. This concept & evidence forms the main knowledge base upon which the later design, development & piloting activities (WP4/5) build. In a final stage, this WP synthesizes findings from these latter activities to systematize knowledge on the reproducibility impact pathways (incl. gains & savings) possible across epistemic contexts. These reflections then inform final recommendations & a roadmap for future reproducibility co-created with relevant stakeholders (including researchers, publishers & funders).

Description of work

Task 3.1. Conceptual framework for reproducibility across contexts (M1-M8; AU [lead], KNOW)

Conceptions, roles & pertinence of reproducibility are closely linked to different principal modes of knowledge production within & across research fields, so-called ‘epistemic contexts’. Using desk research & 5 focus-groups with co-creation communities, we will map out an exhaustive set of epistemic, methodological, social, legal, technological & cultural factors which shape conditions for reproducibility. This task will provide the theoretical framework for TIER2 in the form of a scoping report centred around a matrix that maps the contextual factors to various dimensions of ‘reproducibility’, as well as pertinent framework conditions (e.g., political, ethical, social, legal) that may affect the uptake of reproducibility practices. This framework enables analyses of diverse conceptions, roles & barriers of reproducibility & permits identification of relevant & targeted tools, irrespective of fields.

Task 3.2 Evidence-base and inventory of reproducibility tools and practices (M1-M12; KNOW [lead], AU, Charite, UOXF)

Next, we consolidate & valorise current knowledge on practices & tools for reproducibility by systematically scoping, critically appraising, & synthesising the literature using the Scoping Review (PRISMA-SCR), an Impact Pathway assessment methodology outlined above (sections 1.2.1 & 1.2.2, obj. 1). This will systematically search academic databases & grey literature (including EC project outputs, policy documents, & tool registries) to inventory tools & practices, & synthesise current knowledge on their efficacy across epistemic contexts & stakeholder categories. In addition, we will use FAIRsharing to collect & visualize the reporting standards & best practices within the EOSC science clusters (in particular EOSC-Life & SSHOC, for the life & social science use case, respectively) for both data & software. FAIRsharing will also provide a snapshot of the data policies by publishers & funders, focusing on their requirements, or the lack of, for data reporting & reproducibility. Synthesizing & presenting current knowledge in this way will create a centralised resource which enables all stakeholders (including project partners) to orient themselves easily to the state-of-the-art in this area. This content will then be integrated into the Reproducibility Hub (Task 2.3).

Task 3.3. Synthesis and recommendations (M13-M36; KNOW [lead], AU, VUmc)

In the later phase of the project, this task will bring the project full circle by synthesising results obtained from the empirical studies in WPs4/5, based on the framework established in Tasks 3.1/3.2. First, we incorporate results from our pilots & survey co-creation communities (including questions on costs) to reflect upon how reproducibility generates gains & savings & thereby improves overall performance in the R&I process (including demonstrated positive effects on R&I quality, integrity & trust-worthiness). These results are then validated qualitatively through 3 interactive workshops with our main stakeholder groups (using co-creative approaches, in collaboration with WP4). Based on the synthesis, the task will then use a modified Delphi methodology to work with stakeholders (including members of the co-creation communities) to create recommendations & policy guidelines for practices & joint action by researchers, funders, publishers & institutions to increase reproducibility, including but not limited to training, specialised careers, potential role in assessment criteria & guidelines for best practice. While solutions will be primarily applicable to Europe, attention will be paid to reproducibility in global science. Final recommendations will hence be aligned with user needs, & mapped on the existing practices used by our three research domains & for funders & publishers. The resulting synthesis of knowledge on reproducibility impact pathways & targeted recommendations for action will provide an overall roadmap for future developments beyond the project.


D3.1 Reproducibility Impact Pathways: State-of-play on methods, tools, practices to increase reproducibility across diverse epistemic contexts (combines findings from T3.1/3.2) (KNOW, M12)

D3.2 Validated key impact pathways for reproducibility, including recommendations (KNOW, M36)


MS3.1 Conceptual framework for reproducibility across contexts (M8)

MS3.2 Interim synthesis of findings on reproducibility gains and savings (M28)

In Table 17 you can find the description of Work Package 4.

Table 17.

Work package description 4.

WP No.


Lead beneficiary


Start month


End month


WP title

Community-driven design and piloting of reproducibility tools and practices


This WP uses future studies, interviews, focus groups & co-creation methods to engage the communities curated in Task 2.2 to steer & assist the development and/or adaptation of the reproducibility tools & practices in WP5. With a focus on the comparative analysis of reproducibility across different epistemological contexts, including barriers & drivers to reproducibility across full research lifecycle, we identify the gaps of knowledge for reproducibility across contexts, & design case-study pilots for the implementation & assessment of new tools. Main objectives:

  • Use an innovative backcasting, co-design methodology to work with communities to sketch & select optimal scenarios for future reproducibility across contexts, & identify key areas for development to achieve these scenarios.
  • Plan & design interventions that can help important stakeholders see what the costs & benefits are for reproducibility across epistemic contexts for researchers, funders & publishers.
  • Prepare & run pilot scenarios for tools & practices developed in WP5 to showcase, verify, & evaluate them across methodological contexts

Description of work

Task 4.1. Future studies to identify priorities from the stakeholder community to predict future of reproducibility and identify actionable steps (M3-M9; VUmc [lead], KNOW, AU)

Profound knowledge on potential reproducibility issues that are faced by researchers from different communities is essential to build a framework, create tools & make potential gains & costs visible. This task will first scope the ideal futures of reproducibility for 5 different stakeholders (soc, life, comp sci researchers, plus funders & publishers) in online scenario workshops. Using provisional results from the scoping work in Task 3.1 & 3.2, future scenarios will be depicted through 5 scenario workshops with backcasting & transcend methods. Based on these goals & ideas of the future, we will then work backwards to identify the key steps needed to reach these scenarios. In a next stage, these ideal scenarios will then be used to work with communities (3 cross-stakeholder focus groups & 14 interviews) to further identify drivers & barriers to realising these futures. Bottom-up coding of transcriptions will be conducted in Nvivo, synthesised & reported back to WP3 to further develop the framework (Deliverable 3.1), & carried forward to inform Task 4.2.

Task 4.2. Requirements procurement and design (M8-M30; GESIS [lead], KNOW, ARC, VUmc, AU, UOXF, FLEMING)

Building upon the priority action areas identified in T4.1, & leveraging the scoping work (inventory of tools & practices) from Task 3.2 & the epistemic contexts identified in Task 3.1, this task will work together with pilot community members to specify new interventions to increase reproducibility across all phases of the research lifecycle from ideation to assessment for different methodologies and epistemic contexts. For each case, a detailed plan of action will be created & the technical & social requirements mapped for development & implementation in WP5. New contexts for existing tools as well as the further development of tools in existing contexts will be considered. In the context of the pilots, we need to apply tools/practices to particular communities/scenarios. These communities have their own special needs & particularities & we here take them into consideration. While we have indicated priority tools for development based on our initial scoping work for this proposal (see section, in the spirit of co-creation we base the exact direction of this development on the input of our communities. We hence remain flexible in considering other tools/practices for adaptation or development. Hence this task does essential work in assessing community needs & steering accordingly the development efforts in WP5.

Task 4.3 Pilots preparation activities (M13-M19; ARC [lead], VUmc)

Piloting of new tools & practices is of great importance in TIER2 for validating, evaluating, & improving the set of tools & practices to facilitate the reproducibility of scientific results. Consequently, we implement a set of preparation activities to ensure successful planning & implementation. Although the pilot partners have contributed to the co-design of the tools & practices, various factors can influence their involvement in the piloting & evaluation activities having a large impact on the respective case studies. These factors are expected to be stakeholder-related (e.g., familiarity with technologies, software stacks), inherent to the project work (e.g., problematic/non-expected operation of components), or even unforeseen situations. Our goal is to plan ahead & implement all necessary steps to anticipate & mitigate any possible setbacks & problems. The task will also pre-register all methods to be used in pilot implementation, & design an evaluation framework for the assessment of WP5’s tools considering different epistemic contexts & specific stakeholder groups. In collaboration with the stakeholders, we will compile a list of performance indicators that can help us in unifying the assessment of the tools per research type & stakeholder group. During the proposal phase we have identified key elements of the pilots (listed in section above), which will be validated, fine-grained & attuned to stakeholder requirements during the project, per the co-design approach we propose.

Task 4.4 Pilot implementation and assessment (M20-M30; VUmc [lead], KNOW, ARC, AU, PENSOFT, GESIS, UOXF, FLEMING)

This task involves execution of the pilot actions & the respective evaluation activities. The pilot actions involve use cases that showcase & apply in practice reproducibility-related tools & practices for researchers, publishers, & funders (details on the respective actions in section involving all stakeholder communities (Social sciences, Life sciences, Computer Science researchers, plus funders & publishers). The stakeholders will also work towards the evaluation of the tools & practices according to the workflows designed in T4.3 (details on assessment methodology: section to get insights into what works & what is not working for different epistemic diversities & what is needed to be able to successfully implement them in the research community, for funders & for publishers.


D4.1 Future reproducibility across epistemic contexts: Report on future studies/backcasting outcomes (VUmc, M9)

D4.2 Pilot implementation and assessment plans (GESIS, M18)

D4.3 Pilot implementation reflection report including assessment of efficacy & recommendations for future developments (VUmc, M30)


MS4.1: Pre-registration of protocol for future studies Task 4.1 (M4)

MS4.2: Pre-registration of methods for pilot implementation/assessment (M18)

MS4.3: Update briefing reports on pilot implementation progress via Reproducibility Hub (M26)

In Table 18 you can find the description of Work Package 5.

Table 18.

Work package description 5.

WP No.


Lead beneficiary


Start month


End month


WP title

Development of tools and practices for communities


This work package will develop, extend and/or adapt practical reproducibility-related tools (incl. software, checklists, standards, workflows, monitoring dashboards & policy creation guides) for funders, publishers, & researchers, to be used in the WP4 pilot activities. An agile, continuous, co-creation design & development approach (in constant dialogue with Tasks 4.2-4.4) will be used to ensure that the respective needs & requirements of pilot users are met.

NOTE: The tools earmarked for development here are selected on the basis of the priorities identified in our scoping work for this proposal. Given our co-creative approaches, however, the exact direction of development will be oriented to user-needs discovered through the project implementation. We recognise the list of activities below is highly ambitious. However, we point out that TIER2 development partners have been selected for the tools & initiatives they already bring to the project (many already in production – see section, whose effective integration will deliver results at scale & cost.

Description of work

Task 5.1 Practical tools and practices for researchers (M9-M34; ARC [lead], KNOW, GESIS, OpenAIRE, UOXF)

This task implements, customises, & deploys practical tools & practices for researchers that promote & facilitate reproducibility of scientific results during planning, data collection & results analysis phases of the research lifecycle, based on the scoping & co-creation activities in WPs3/4. Planned activities (to be further refined through the WP4 co-creation design processes) are:

  • 5.1.1) Reproducibility Checklist: Building upon the scoping work from WP3 we will create an interactive Reproducibility Checklist to guide researchers in best reproducibility practices for their epistemic context. Hosted via the Reproducibility Hub (WP2), the Checklist will also include well-established, discipline specific & domain agnostic recommendations & standards to report digital objects, listed in & tracked in FAIRsharing via community contributions.
  • 5.1.2) Extending DMPs to RMPs: Effective Data Management Plans (DMPs) are already a very important element of ensuring reproducible workflows, & actionable DMP tools are increasing this usefulness, but TIER2 believes the potential is even greater. This activity will extend DMPs into a new concept (“Reproducibility Management Plans” - RMPs), that will include additional reproducibility-related metadata for the reported research. To this end, Argos machine-actionable DMP tool will be extended to enable qualified references*15 across research resources & outputs for the whole lifecycle, connecting to FAIRsharing for the community standards, assisting researchers to self-assess & ensure the reproducibility of their research.
  • 5.1.3) Reproducible Workflows: We will next leverage software containerisation technologies, workflow description languages (e.g., CWL), & experiment packaging specifications (e.g. RO-crate) to create a detailed methodology to ensure the reproducibility of computational workflows in different epistemic contexts. Our focus will be on adapting SCHeMa, an open-source tool of this type already successfully used within life sciences, to investigate its use in different epistemic contexts (e.g., extending to social sciences for survey data & computer science for Machine Learning).

Task 5.2 Practical tools and practices for publishers (M9-M34; UOXF [lead], KNOW, ARC, GESIS, OpenAIRE)

This task will involve all activities for the implementation, customization, validation & deployment of the set of practical tools & practices that promote, facilitate, monitor & assess reproducibility of research outputs, taking into consideration the special needs & requirements of publishers.

  • 5.2.1) Workflows to review research datasets & code: Various mechanisms have been proposed as workflows for checking of data (Wilkinson et al. 2019) & code (Nüst and Eglen 2021), while existing publishing schemes & venues (e.g., Octopus, F1000) have processes in place for advanced screening and/or review of datasets & software regarding reproducibility. To further this work, TIER2 will develop streamlined workflows for review of data & code (at the manuscript submission time), including scoping of a common system of ‘stamps’ or validity marks to indicate that work has been checked, validated and/or reused. We also plan to exploit ARC’s Reproducibility Assessment Toolkit (ROAL) in order to spot & highlight research artefacts (e.g., datasets, software, models) stated in the manuscript, indicating potential missing metadata elements that should accompany & enrich research output descriptions
  • 5.2.2) Threaded publications: Incremental publishing models (e.g., Octopus, ResearchEquals) where work is published in smaller units (e.g., protocols, methods, data, code) & then linked (or “threaded”) together at the end to form a cohesive whole are seen as potential routes to greater reproducibility, as publication bias is avoided through early, chronologic & continuous reporting of work & greater scrutiny is enabled through early-sharing of work (Hartgerink and van Zelst 2018). This concept involves basically linking published outputs arising at different stages of a research project or programme together to help ensure that insights & usable outputs are not missing from the research system, thus supporting reproducibility & minimising research waste. Greater uptake of such models amongst publishers requires development to establish shared technical standards for linking entities (i.e., research objects, as well as grants, funders, individuals), as well as to establish who should create & share this information & how, enabling the provenance of research artefacts. TIER2 will do this development work, including investigating potential protocols including RO-Crate, Nanopublications, Docmaps & others, & implement pilot activities.
  • 5.2.3) Registered reports/open publishing models for new contexts: From our scoping work (sec., extension of models for publishing which minimise publication bias and maximise transparency are a key priority for publishers. TIER2 Associated Partner and their parent company Taylor & Francis have been key players in introducing such models. This task will further support two lines of action:
  1. extension of the F1000 transparent publishing model to new contexts – especially the recently launched Routledge Open Research (for humanities/soc sci) Open Research Europe (covering all Horizon research), especially definition of reporting standards/guidelines for qualitative research;
  2. extension of T&F’s workflow for registered reports to new disciplines (e.g., education) to investigate the efficacy (or not) of this model for new epistemic contexts. In addition, with leading conferences, we will also investigate potential for registered reports in Computer Science publishing.

Task 5.3. Practical tools and practices for funders (M9-M34; ARC [lead], KNOW, ARC, VUmc, GESIS, OpenAIRE, UOXF)

This task will implement, customise, & deploy practical tools & practices that enable & support funding institutions in prioritising & tracking reproducibility within their funded projects. Building upon existing production-ready tools from TIER2 partners, this task will develop:

  • 5.3.1) Reproducibility Promotion Plans: TIER2 will produce practical advice for funders on how to create a plan to boost the reproducibility of their funded-results. Modelled on the “Research Integrity Promotion Plan (RIPP)” created by VUmc/AU in SOPS4RI, we will work with funders & other stakeholders to develop a tool that assists research funding (and performing) organisations to create “Reproducibility Promotion Plans” (RePPs), including considerations for reforming incentives to foster reproducible practices. The plan will describe how a specific funder will ensure, foster & promote reproducible research practices, avoid detrimental practices, & handle misconduct assisting funders to form their own RePPs to take disciplinary, organisational & national differences into account.
  • 5.3.2) Funder extension of RMPs. This activity involves the extension of machine actionable RMP tools developed in T4.1.2 to offer evaluation & reporting functionalities for the officers in funding organisations. Such tools will automate the identification & classification of research artefacts produced during the project & indicate potential enrichment of DMPs with relevant research output descriptions, leveraging the reporting standards in FAIRsharing to also assist in assessing levels of FAIRness.
  • 5.3.3) Reproducibility monitoring dashboard: Building upon algorithms created by ARC for the DG-RTD Study “Assessment of reproducibility of research results in EU Framework Programmes”, as well as the OpenAIRE Knowledge Graph & FAIRsharing registry, this activity involves the development of tools that enable funding agencies in tracking & monitoring reusability of research artefacts (datasets, software, tools/systems, etc.) produced within projects of interest, across different programmes, topics & disciplines. An automatically generated report will be generated facilitating assessment & quantification of the impact of policies for data-sharing, code-sharing, etc. We will also scope requirements for making such a dashboard available to publishers.


D5.1: Reproducibility toolset (tools & practices) for researchers (ARC, M34)

D5.2: Reproducibility toolset (tools & practices) for publishers (KNOW, M34)

D5.3: Reproducibility toolset (tools & practices) for funders (ARC, M34)


M5.1: TIER2 researcher reproducibility toolset first release (M22)

M5.2: TIER2 publisher reproducibility toolset first release (M22)

M5.3: TIER2 funder reproducibility toolset first release (M22)

3.3 Capacity of participants and consortium as a whole

TIER2 brings a unique combination of skills and perspectives, consisting of eleven partners from seven countries, well spread across Europe. The partner organisations all bring a wealth of expertise and experience regarding the target communities (computer science, life science, social science), research culture reform, technical aspects of infrastructure and services, as well as expertise in social sciences and humanities, and gender aspects of R&I. The partners have collaborated with success in numerous previous projects together. The inter-disciplinary and complementary nature of our mix of knowledge and networks will allow TIER2 to excel in delivering success in our ambitious activities (see Table 19).

Table 19.

TIER2 Consortium experience and expertise.

Know-Center GmbH (KNOW, Austria): KNOW’s Open & Reproducible Research Group (ORRG) is an interdisciplinary team of information scientists & sociologists with key expertise in Open Science implementation & policy (e.g., coordinating ON-MERRIT), assessing reproducibility (DG RTD Reproducibility Study), & enabling transparency in research cultures (OpenUP). As a leading European Research Center for Data-driven Business & Artificial Intelligence, KNOW also has substantial domain expertise in computational sciences, as well as social sciences competences. In TIER2, Know-Center coordinates, leads work to consolidate the evidence-base & synthesise project results (WP3), & contributes to work on piloting (WP4) & developing (WP5) tools & practices.

Athena Research Center (ARC, Greece): ARC has high-level expertise in R&I monitoring (e.g., DG RTD Reproducibility Study), evaluation of Open Science policies & development of e-Infrastructures for scientific content. A leading European AI research institution, ARC has substantial expertise on reproducibility across Computer Science (esp. technologies to assist computational reproducibility). ARC also brings a funder perspective as its “ARCHIMEDES” initiative is an emblematic initiative for Greece funded with over 21m EUR for cascading funds. In TIER2, ARC contributes to the design of the pilots & leads the WP on the implementation of tools & practices (WP 5) that will be demonstrated & evaluated in the context of the pilots (incl. funder piloting via ARCHIMEDES).

Stichting VUmc (VUmc, Netherlands): Stichting VUmc (part of AmsterdamUMC) is a university medical centre with a very active research ethics/integrity department. VUmc has led several Horizon Europe funded projects, including EnTIRe, the founding project of the Embassy of Good Science (now hosted at VUmc). VUmc is initiator of the project Replication in the Humanities, a 4-year project that investigates to what extent replication should be a crucial element in research in the humanities. VUmc also led the use of co-creative methodologies in SOPs4RI to create and implement Research Integrity guidelines at research institutions. These co-creative methodologies will be used on several occasions in TIER2, where VUmc will lead development of the Reproducibility Hub (WP 2), and lead on the design and assessments of pilot activities (WP 4).

Aarhus University (AU, Denmark): The Danish Centre for Studies in Research & Research Policy (CFA) at Aarhus University, is a transdisciplinary research centre with extensive expertise in the areas of science & technology studies, meta-research & research, innovation, & university policy. Expertise on fostering Research Integrity (SOPs4RI), & will coordinate POIESIS (starting 2022) studying connections between public trust in Science, Research Integrity & Open Science. In TIER2 AU will lead on the conceptual framework & contribute to inventorising tools & practices (WP3).

Pensoft Publishers (PENSOFT, Bulgaria): Pensoft is an SME with long-standing expertise in academic, open access book & journal publishing, software development, web design, project dissemination & science communication. PENSOFT has substantial expertise as dissemination partner (over 40 projects). Pensoft is also a renowned Open Science publisher. In TIER2, PENSOFT will be responsible for successful dissemination & communication (WP2), and contribute publisher context in piloting of Reproducibility Management Plans.

Gesis Leibniz Institute for Social Sciences (GESIS, Germany): GESIS is the largest European infrastructure institute for social sciences, providing data, services, infrastructure & training for all stages of research projects. GESIS brings user-communities and expertise for TIER2’s social science pilots. In TIER2, GESIS will contribute to the design (WP4) & development (WP 5) of reproducibility tools & practices, in particular for the social sciences.

OpenAIRE (OpenAIRE, Greece): A key EU e-Infrastructure for Open Science and cornerstone of the EOSC. OpenAIRE links national & EC infraspheres and has active links to publishers and funders. OpenAIRE has an extensive training portfolio, and will act as a capacity-building multiplier via its 47 members in 30 counties to disseminate. In TIER2, OpenAIRE leads alignment activities with EOSC (WP1). contribute its infrastructure for the development/implementation of new tools/practices (WP5), & contributes to stakeholder engagement activities (WP2).

Charité – Universitätsmedizin Berlin (Charite, Germany): Charité is one of the largest university hospitals in Europe. Their QUEST (Quality-Ethics-Open Science-Translation) centre examines how institutions can improve the rigour, reproducibility & transparency of biomedical research. As a founding member of the German Reproducibility Network, and with considerable experience in community engagement/training activities, Charite will lead on stakeholder/co-creation activities, especially bringing together the European reproducibility networks (WP2).

University of Oxford (UOXF, UK): The Data Readiness Group, at the Oxford e-Research Centre, in the Department of Engineering Science has rich expertise in creating elements that underpin reproducibility (esp. in Life Sciences), as part of the development & implementation of standards & policies via the FAIRsharing network, ELIXIR activities & EOSC-Life. UOXF has excellent links to the UK Reproducibility Network (funded partner) and publishers/funders (via related RDA Interest Groups) In TIER2, UOXF join as Associated Partner, bringing their own in-kind resources (funded via UKRI) to contribute to the Reproducibility Hub (WP 2), & to the development and piloting of tools/practices for researchers, funders and especially publishers (WP 5).

Biomedical Research Center Fleming (FLEMING, Greece): Fleming is a high-profile non-profit research organisation focusing on scientific & technological excellence, training & innovation in biomedical sciences. In TIER2, FLEMING will contribute to pilot preparation, implementation & assessment for the life sciences (WP4).

These competences and networks are enhanced and supported by TIER2’s international Advisory Board, external stakeholders with a demonstrated record/interest in reproducibility and Open Science, including expertise in our target research areas, as well and research publishing and funding. The board will advise the project in strategic matters and will provide advice for the high-level dissemination and outreach strategy of the project. The Advisory Board will also be essential for the verification and finalisation of the project’s core results and recommendations.

In addition to the range of perspectives and depth of expertise/experience reflected here, within our consortium and Advisory Board, we also have excellent gender balance (5 of 11 partner PIs and 4 of 8 AB members are female) and excellent international reach across three continents, with exceptional links to further EC Widening Participation countries and world regions. To summarise, Table 20 below gives an overview of how the expertise assembled via TIER2 covers all essential elements needed to assure the success of our project.

Table 20.

Overview of areas of TIER2 expertise to address key challenges.

Repro-ducibility, RI, OS

R&I policy

Pilot domain expertise

Infra­­­structure / tools

Repro­ducibility networks


Dissem., training, network­ing

Publisher view

Funder view





















































Advisory Board



x (all)







4 Ethics

Ethical dimension of the objectives, methodology and likely impact

TIER2 aims to enhance reproducibility through design, implementation and assessment of systematic interventions addressing key levers of change (tools, skills, communities, incentives, and polices). Our objectives align strongly with broad ethical goals of community-engagement, reform of research cultures to address problematic practices, and increasing trust in research. Activities to achieve the objectives will include both quantitative and qualitative studies, as well as stakeholder engagement via online and live meetings, which will be recorded and analysed. Trainings and training pathways will be evaluated with participants, using surveys and interviews. The activities to achieve these objectives will include the involvement of different stakeholders reproducibility in WPs 2,3,4,5. Only adult participants will be included and will not belong to vulnerable populations. Personal data will be collected from project activities described above and will not include sensitive personal data, such as health and genetic data, or political and religious opinions. The project will also involve non-EU countries (UK and potentially one or more Associated Country). The activities in these countries will be the same as in EU countries involved in the project. Benefit-sharing is planned for any lower middle-income countries that will take part in the project (their involvement will be specifically with the aim of boosting reproducibility capacity in these countries). We do not foresee negative impacts of the project activities with regard to political or adverse financial consequences, misuse or stigmatisation of specific populations. Protection of rights, well-being, safety, privacy and confidentiality of the participants in project activities will be ensured by the coordination and management of the TIER2 governing body in WP1, coordinated by KNOW. Some elements of our research may involve elements of AI, especially in the context of our development of tools to support reproducible researcher workflows and publisher tools for review of data/code. At any step of these activities we will adhere strictly to best-practice guidelines in the design and testing of these systems to ensure there are no ethical concerns related to human rights and values, especially to clearly identify and address any biases contained within training datasets which may negatively affect performance for any populations. The TIER2 governance structures will provide oversight of all project activities, and will include a General Assembly of all partners, meeting in-person during consortium meetings and virtually when required; a steering committee, comprising WP leads and meeting virtually at least 4 times a year; and an external Advisory Board. The TIER2 core leadership team will provide operational, administrative and financial support in all project activities.

Compliance with ethical principles and relevant legislations

The work in the TIER2 project will be led by relevant EU/national legal and ethical requirements of the country or countries where the tasks are to be carried out. The legal and ethics framework includes: The Universal Declaration of Human Rights, The European Convention on Human Rights, The Charter of Fundamental Rights of the European Union, and the EU General Data Protection Regulation. The European Code of Conduct for Research Integrity will be the main ethical and integrity reference followed in the project. As the activities raising ethics issues are related to the involvement of human participants in social sciences/humanities research, TIER2 takes full awareness of all ethical issues addressed in the guides “Ethics in Social Science and Humanities” and “Ethics and Data Protection”, published in 2021 by panels of experts at the request of the European Commission. For the activities in any lower middle-income countries, TIER2 will follow the principles outlined in the Global Code of Conduct for Research in Resource-poor Settings. Research involving human participants - the consortium will ensure compliance with the applicable international conventions, EU and national legal provisions in relation to the conduct of studies involving human participants in social science/ humanities research, data protection and confidentiality. All necessary ethical approvals and authorisations will be obtained prior to the start of the research activities. The consortium is aware of differing practices of approvals for social sciences/humanities research and expects that in some countries ethics approvals are not required or is waived for that type of research. Ethics compliance oversight is dealt with in WP1 (Task 1.1). Protection of rights, well-being, safety, privacy and confidentiality of the participants in project activities will be ensured by the coordination and management of the TIER2 governing body in WP1, coordinated by KNOW. The TIER2 governance structures will provide oversight of all project activities. A Consortium Agreement will describe the rights and obligations of partners and a succinct governance structure, including a General Assembly (all consortium members), the Executive Board (consisting of one representative of each partner organisation), and an external, international Advisory Board. The TIER2 core leadership team will provide operational, administrative and financial support in all project activities. Data from non-EU partners of the consortium will be imported into the EU, in compliance with the laws of the country in which the data was collected. At the moment, EU has made adequacy decision that UK has adequate level of data protection, so that the flow of research data is possible without any further safeguards. The consortium will adopt technical and organisational security measures to prevent unauthorised access to personal data collected during the project, adequately train the research staff and publish the search results in such a way as to do not allow their improper use. Involvement of non-EU countries – the consortium will perform activities and collect data from three Widening Participation countries (to be decided via open call – it hence may be that one or more are Associated Countries). Research and data collection will be performed in the same conditions as in EU countries except if justified by local requirements that do not contradict the spirit of ethical guidelines. All study activities will be submitted to the local legal and ethical committees in compliance with relevant requirements. The consortium confirms that the research to be held in non-EU countries would be allowed in all EU Member states. Some elements of our research may involve elements of AI, especially in the context of our development of tools to support reproducible researcher workflows and publisher tools for review of data/code. At any step of these activities we will adhere strictly to best-practice guidelines (e.g., The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems). Ethically aligned design: A vision for prioritizing human well-being with autonomous and intelligent systems) in the design and testing of these systems to ensure there are no ethical concerns related to human rights and values, especially to clearly identify and address any biases contained within training datasets which may negatively affect performance for any populations.

Funding program

Funded by the European Union (grant agreement No 101094817). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.



Reproducibility: Definitions of reproducibility (and replicability, etc.) vary widely. In this proposal, reproducibility is used in the broad sense to refer to the possibility for the scientific community to obtain the same results as the originators of a specific finding. Where more exact language is intended, we use the terms: Computational reproducibility: Same finding obtained using same dataset, experimental setup and methodology; Replicability: Same finding obtained from the same or similar analytical method on different datasets. Can be further delineated: Direct replication: Same analytical method, different data, same result; Conceptual replication: Different analytical method, different data, same result.


Contract RTD/2020/SC/010, 2020-2022, budget 480k EUR. Final report:


Technological Readiness Level is “a type of measurement system used to assess the maturity level of a particular technology”, cf.


EC DG-RTD. 2022. Assessing the reproducibility of research results in EU Framework Programmes for Research


Project recently funded (Horizon Europe, 2m EUR) under call topic HORIZON-WIDERA-2021-ERA-01-40


MethodHub is under development by Gesis and will be TRL4 by TIER2’s anticipated kick-off


ROAL is under development by ARC and will be TRL5 by TIER2’s anticipated kick-off


Proposal originally made public upon submission at


TIER2 coordinator Tony Ross-Hellauer was co-organiser of the original Reprohack events, hosted as OpenCon satellite events (Berlin 2016, London 2017). See:


Proposal made public at


From the Horizon Europe “Data Management Plan Template”: A qualified reference is a cross-reference that explains its intent. For example, X is regulator of Y is a much more qualified reference than X is associated with Y, or X see also Y. The goal therefore is to create as many meaningful links as possible between (meta)data resources to enrich the contextual knowledge about the data. (Source:

login to comment