Corresponding author: Tony Ross-Hellauer (
Academic editor:
Lack of reproducibility of research results has become a major theme in recent years. As we emerge from the COVID-19 pandemic, economic pressures and exposed consequences of lack of societal trust in science make addressing reproducibility of urgent importance. TIER2 is a new international project funded by the European Commission under their Horizon Europe programme. Covering three broad research areas (social, life and computer sciences) and two cross-disciplinary stakeholder groups (research publishers and funders) to systematically investigate reproducibility across contexts, TIER2 will significantly boost knowledge on reproducibility, create tools, engage communities, implement interventions and policy across different contexts to increase re-use and overall quality of research results in the European Research Area and global R&I, and consequently increase trust, integrity and efficiency in research.
Funded by the European Union (grant agreement No 101094817). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
Detailed list of all paricipants is available in Table
TIER2 is a new project funded by the European Commission under their Horizon Europe programme (call HORIZON-WIDERA-2022-ERA-01-41 - Increasing the reproducibility of scientific results*
Lack of reproducibility of research results has become a major theme in recent years. As we emerge from the COVID-19 pandemic, economic pressures (increasing scrutiny of research funding) and exposed consequences of lack of societal trust in science make addressing reproducibility of urgent importance. TIER2 does so by selecting 3 broad research areas (social, life and computer sciences) and 2 cross-disciplinary stakeholder groups (research publishers and funders) to systematically investigate reproducibility across contexts. The project starts by thoroughly examining the epistemological, social and technical factors (epistemic diversity) which shape the meanings and implications of reproducibility across contexts. Next, we build a state-of-the-art evidence-base on existing reproducibility interventions, tools and practices, identifying key knowledge gaps. Then TIER2 will use (co-creation) techniques of scenario-planning, backcasting and user-centred design to select, prioritise, design/adapt and implement new tools/practices to enhance reproducibility across contexts. Alignment activities ensure tools are EOSC-interoperable, & capacity-building actions with communities (i.e., Reproducibility Networks) will facilitate awareness, skills and community-uptake. Systematic assessment of the efficacy of interventions across contexts will synthesise knowledge on reproducibility gains and savings. A final roadmap for future reproducibility, including policy recommendations is co-created with stakeholders. Thus, TIER2 will significantly boost knowledge on reproducibility, create tools, engage communities, implement interventions and policy across different contexts to increase re-use and overall quality of research results in the European Research Area and global R&I, and consequently increase trust, integrity and efficiency in research.
We here present the TIER2 project “Description of Action”, which programmatically guides project activities. This text represents our initial project proposal as submitted to the EC, with only slight modifications (e.g., streamlining and clarification of some deliverables and milestones, removal of some administrative information for readability).
As TIER2’s success will depend on community engagement we make this information public not only as part of our commitment to principles of open and reproducible research, but also to inform the community of our plans and invite interested parties to get involved.
This section describes: (a) the rationale and objectives of TIER2, including how the project aims to go beyond the state-of-the-art (subsection 1.1), and (b) our overall methodology for the project (subsection 1.2).
The European Research Area (
Reproducibility is often claimed as a central principle of the scientific method (
As the world emerges from the COVID-19 pandemic, economic pressures (and hence increased scrutiny of research funding) and the exposed consequences of lack of societal trust in science make addressing such issues of urgent importance. Doing so will reduce inefficiencies, avoid repetition, maximise return on investment, prevent mistakes, and speed innovation to bring
Increasing reproducibility is a multifaceted challenge, however. At the practical level, there is an urgent need for capacity-building to improve infrastructure/services, skills, communities, incentives and policies to enable and encourage reproducibility-maximising practices. At the theoretical level, moreover, three key information gaps currently hinder progress:
Limited clarity on meanings, limits and implications of reproducibility across modes of knowledge production; Limited understanding of optimal reproducibility impact pathways to maximise gains and minimise costs in reproducibility reform; No coherent roadmap for implementing policies and practices to optimise reproducibility across the whole R&I system.
An influential EC scoping report
TIER2 will centre epistemic diversity by selecting three broad research areas (social, life and computer sciences) and two cross-disciplinary stakeholder groups (research publishers and funders) to systematically investigate reproducibility across contexts. In tandem with curated co-creation communities of these groups, we will design, implement and assess systematic interventions addressing key levers of change (tools, skills, communities, incentives, and polices). The project will start by thoroughly examining the epistemological, social and technical factors (epistemic diversity) which shape the meanings and implications of reproducibility across contexts (epistemic contexts). Next, we will build a state-of-the-art evidence-base on extent and efficacy of existing reproducibility interventions and practices, as well as an inventory of relevant tools, identifying key gaps in current knowledge. Then TIER2 will use (co-creation) techniques of scenario-planning, backcasting and user-centred design to select, prioritise, design/adapt and implement new tools to enhance reproducibility across contexts. Alignment activities will ensure tools are EOSC-interoperable, and capacity-building actions with communities (i.e., Reproducibility Networks) will facilitate awareness, skills and community-uptake. Systematic assessment of the efficacy of interventions across contexts will enable a synthesis of knowledge regarding reproducibility gains and savings. This will inform a final roadmap for future reproducibility, including policy recommendations co-created by stakeholders.
TIER2's core objectives, along with their relative key activities and forseen results, are outlined in Table
Taking stock of existing knowledge/evidence, and clarifying the meanings/implications of reproducibility across epistemic contexts. Building capacity and innovating new EOSC-native tools and practices for funders, publishers and researchers through community-led pilots addressing infrastructure/services, skills, communities, incentives and policies. Enumerating gains and savings to build a common understanding and roadmap for promoting and monitoring reproducibility impact pathways across epistemic contexts.
Thereby, TIER2 will enhance reproducibility beyond the state-of-the-art in diverse ways:
Here, we present the overall methodology for TIER2, including key concepts, methodologies and project open and reproducible research practices.
In 2020 the EC commissioned a major review “Assessing the reproducibility of research results in EU Framework Programmes”.*
Building upon these findings, TIER2 proposes a programme of activities based on four key principles. We believe the reproducibility agenda must enter a new phase. If phase one is typified by the “crisis narrative”, narrow focus on specific fields, and piecemeal initiatives with limited alignment of strategic action across stakeholders and elements of research, phase two (TIER2) must be founded in the following:
Crucial concepts for TIER2’s approach to implementing these principles are the concepts of key impact pathways, epistemic diversity and our case-study approach.
TIER2 will bring together theory and evidence to design a framework that defines gaps and prioritises new approaches, based upon the
Impact pathway methods are grounded in theory-based approaches (
identify key elements of pathways (input-output-outcome-impact) under the organisational prism of needs and objectives, describe how they are linked and work together, develop metrics for impact, and measure and test on selected cases.
Charting the impact pathways requires that we describe the sequence of input-output-outcome-impact relations that show non-linear linkages and the steps from resources to more long-term impacts. It also entails developing narratives describing causal chains, including the effects of possible enabling factors and barriers. Starting from the key inputs of factors in research culture change, for each of our target domains (comp, life, soc sci), we will trace and prioritise the activities required to produce desired outputs, outcomes and impacts. From the description of the rationales and mechanisms linking the elements of reproducibility impact pathways across epistemic contexts, Fig.
Reproducibility impact pathways will vary greatly due to epistemic diversity. Although, as already stated, concerns regarding reproducibility have been most vocally stated from a relatively narrow range of disciplines, they are increasingly addressed in other areas as well (
Reproducibility is not only affected by epistemic and methodological factors, however.
Acknowledging that there is substantial epistemic diversity across research fields, with the need to better understand the relevance, meanings and implications of reproducibility across them, TIER2 hence proposes a comparative methodology which looks at reproducibility across such contexts with as much breadth and depth as possible (given the resources). We hence select three broad research domains (social sciences, life sciences, and computer sciences) for research(er) contexts, as well as two contexts which cut across disciplines (publishers and funders). The
The TIER2 overall methodology, following six stages according to our stated objectives, is illustrated in Fig.
Our first major task within TIER2 will be to consolidate knowledge to date relating to reproducibility across epistemic contexts. Using desk research and three online focus-groups with co-creation communities, we will map out factors influencing epistemic diversity across our case-study domains (life, social and computer sciences). Systematically elucidating pertinent epistemic and methodological factors for reproducibility across these contexts will provide the initial theoretical framework for TIER2. The output will be a scoping report centred around a matrix that maps different epistemological aims and methods to various dimensions of ‘reproducibility’, as well as pertinent framework conditions (e.g., political, ethical, social, legal) that may affect the uptake of reproducibility practices. This framework will enable analyses of diverse conceptions, roles and barriers of reproducibility and permits identification of relevant and targeted tools, irrespective of fields.
Next, we will use a PRISMA-SCR ‘Scoping Review’ methodology to scope literature to date to answer the question: “What tools and practices are suggested to improve reproducibility across these epistemic contexts, and what evidence exists regarding their efficacy?”. We will systematically search for key terms across academic databases (e.g., Scopus, Web of Science, OpenALEX, OpenAIRE) as well for grey literature including policy reports (via Overton.io) and searches of stakeholder/project websites and funder databases of outputs (e.g., EC CORDIS). Methods will be pre-registered in advance (via Open Science Framework), including search protocols and data-charting strategies. In addition to evidence on efficacy of reproducibility interventions, we will also compile an exhaustive list of tools and practices, classified according to the elements of research lifecycle and epistemic contexts to which they pertain. Finally, we will collect and visualise key reporting standards and best practices in use within the EOSC science clusters relevant to our case-study domains, especially EOSC-Life (life sci) and SSHOC (soc sci).
These strands will be synthesised using the in a Deliverable report “D1.1 Reproducibility Impact Pathways: State-of-play on methods, tools, practices to increase reproducibility across diverse epistemic contexts”, which uses the Key Impact Pathways approached outlined above to identify key areas for intervention and tools/practices upon which to build in the project. Synthesising and presenting current knowledge in this way will create a centralised resource which enables all stakeholders (including project partners) to orient themselves easily to the state-of-the-art. In addition to providing the general theoretical and evidential basis for assessing gains/savings of reproducibility in the project, this content will also be used within the Reproducibility Hub (knowledge resource created in WP2) and the Researcher Reproducibility Checklist and Reproducibility Management Plan development activities (WP5).
Co-creation, defined as “a transparent process of value creation in ongoing, productive collaboration with, and supported by all relevant parties, with end-users playing a central role" (
For the design phase, TIER2 will build on partner experience in successful co-creative methodologies in the SOPs4RI and ON-MERRIT projects to conduct five (co-creative) online scenario workshops with relevant stakeholders (8-10 stakeholders per workshop). In these workshops, we will work with stakeholders to:
envision the ideal future scenario regarding reproducibility for each stakeholder group; use “backcasting” techniques ( prioritise what interventions for development and piloting are desirable and needed within subsequent TIER2 activities (further design in WP4 and development in WP5).
With these results, we can determine what contexts need more co-creation design in order to better equip researchers, funders and publishers with tools that promote reproducibility. Field notes and transcribed recordings will be used to create draft reports of the outcomes of each workshop, which will then be shared with participants for their further feedback. These priorities will then be carried forward to the use-case definition and design phase. Here, working closely with the developers from WP5, stakeholders will be engaged to define central use-cases for the envisioned tools/practices to be developed. For each case, a “design thinking” canvas-approach based on the JISC innovation canvas tool*
In the tool development activities in WP5, we will use an agile early/rapid prototyping methodology with short development cycles (release early/often), ensure constant evaluation from domain experts, regular cycles of user-feedback, flexible experimentation with various ideas and directions, and increased collaborative activities. Design sketches and static mock-ups will be produced rapidly to test new concepts and receive critical feedback. Working prototypes with limited functionality (e.g., baseline algorithms, smaller data) will be rapidly developed to convey a more realistic impression of their operation. After feedback, the prototypes will be iteratively improved, until the design is gradually finalised, and the underlying algorithms are configured, customised and optimised. In their development, all tools will aim to be optimised for integration into the European Open Science Cloud, building especially on our existing EOSC links to enrich EOSC’s range of value-added services to increase reproducibility and reuse. These tools will then be piloted in WP4 (Tasks 4.3, 4.4). Table
A systematic framework for assessment of pilot activities will be developed alongside the design activities (Objective 3) and implemented according to the steps to impact pathway evaluation proposed by (
What would success look like (what would intended outcomes be)? What factors influence achieving each outcome? Which factors can the project influence Which can it not? Which factors, in which ways, will be targeted for change to bring about desired outcomes? What performance information should be collected (including to assess the ways in which reproducibility brings gains or savings)? How can this information be obtained?
Following (
Next, these synthesis activities will inform the creation of policy briefs, guidelines and recommendations for funders, publishers, research institutions and researchers. Firstly, we will reflect upon the challenges, costs and benefits resulting from reproducibility and transparency approaches in the context of TIER2, leading to a final “autoethnographic” self-reflection report that feeds into the synthesis of results obtained from the empirical work in WPs 3-5. As detailed below (sec. 1.2.2.7 on Open Science and Research Data Management) TIER2 aims to be the change we seek by fostering a maximally open, transparent and reproducible approach to the implementation of Horizon Europe projects (beginning from right now, by making our proposal publicly available at the time of submission*
We will then conduct co-creation workshops with individuals selected via purposive sampling from five stakeholder-categories (researchers, funders/RFOs, institutions/RPOs, infrastructures, and umbrella bodies/networks, including those who participated in the approaches implemented) to iteratively create recommendations and policy guidelines for practices and joint action by stakeholders for future training priorities. This will use a co-creative, modified Delphi methodology that combines three rounds of anonymised survey with four online consensus-seeking meetings to work with 30 community members. This methodology has recently been used by the respective Task 3.3. Leader (KNOW) to create the ON-MERRIT recommendations (
In parallel, we will engage in liaison activities to ensure maximum impact of the recommendations. A dissemination and communications strategy (Milestones) will guide this process, whereby the co-creation community will be engaged to disseminate feedback within networks, and key stakeholders will be engaged. We will work with major umbrella bodies (e.g.,
In order to equip researchers, funders, publishers and others with the skills, connections and resources to exploit state-of-the-art guidance, tools and services, we will undertake capacity-building activities throughout the project. Three clusters of activities will particularly add capacity to address reproducibility issues in Europe and beyond.
Networking and linking such initiatives has the potential to play an out-sized role in increasing reproducibility. The TIER2 consortium’s close existing connections to such networks means we are uniquely placed to facilitate and strengthen these connections. By aligning such efforts, TIER2 will greatly add to synergies and network-effects between them to boost capacity at national and international levels.
To bring reproducibility into its next phase, TIER2 convenes a unique constellation of experts in reproducibility issues in targeted research fields, processes of research culture change, capacity-building & knowledge infrastructures to propose a holistic, pragmatic methodology that critically addresses all these elements simultaneously. We include: Domain experts for the target communities (computer science [ARC, KNOW], life science [UOXF, Charite, FLEMING], social science [GESIS, AU, VUmc]); Experts in science and technology studies, research ethics and meta-research with profound understanding of political and social aspects of research culture reform (on e.g., skills, incentives, policies: KNOW, AU, VUmc); experts in socio-economic impact assessment (AU, KNOW); data science (ARC); standards for digital objects across disciplines (UOXF); Core technical expertise in infrastructure and services, including established links to national and EC infraspheres (especially EOSC - OpenAIRE, UOXF, GESIS, ARC). As outlined in the methodology above, integration of the interdisciplinary expertise present in TIER2 will happen at the junctions between objectives, when moving from concept and design to implementation and assessment, as well in particular in Objective 5 (recommend/reflect), where results will be integrated and synthesised, by taking in perspectives across disciplines. In addition, TIER2 will engage in continuous co-creation dialogue with researchers in many different scientific fields, bringing even broader perspectives to bear in the design, implementation and assessment of our activities.
Gender and diversity considerations will be reflected in all stages of TIER2: in the research design, methods, analyses, interpretation, dissemination and creation of guidelines and recommendations. As outlined in the provisional schema of elements to map reproducibility impact pathways (Fig.
TIER2 convenes experts in open and reproducible research practices. TIER2 partners are committed to Open Science as proven by their involvement, leadership and expertise in Open Science for the past decade. TIER2 will build on this expertise to follow current best practices to ensure Open Science embedded in all aspects of the project:
Through our expertise (see Table 19, section 3.3) on Research Data Management (RDM) and FAIRification of research objects, TIER2 will enable state-of-the-art practices to ensure effective management, preservation and sharing of all the research objects we create:
This section outlines how TIER2 aims to contribute to the outcomes and impacts described in the EC work programme, the likely scale and significance of this contribution, and our measures to maximise these impacts.
As the world emerges from the COVID-19 pandemic, economic pressures (and hence increased scrutiny of research funding) and the exposed consequences of lack of societal trust in science are of urgent importance. Increased reproducibility of results in the European Research Area and beyond will reduce inefficiencies, avoid repetition, maximise return on investment, prevent mistakes, and speed innovation to bring trust, integrity and efficiency to the ERA and global Research & Innovation (R&I) system in general. To set the scale of potential savings, according to
TIER2 sets out an ambitious yet pragmatic programme of activities founded in four fundamental principles:
Reproducibility is as an opportunity not a crisis; Epistemic diversity (variation across modes of knowledge production and socio-technical contexts) must be centred; Evidence must be systematised for informed policy across contexts; Action must be targeted holistically to boost capacity at all levels.
To enact these principles, in TIER2 stakeholder communities of researchers, publishers and funders will positively inform
a new structured understanding of the nature of reproducibility (concrete interventions, drivers, barriers, gains and savings); creation and implementation of effective new solutions at all levels (from technical to policy) to increase the reproducibility of R&I results; empowered communities and networks whose new linkages and shared vision have powerful network effects that enable alignment and joint action on training, skills, infrastructure and more.
This will be done through innovative, applied research performed, providing tangible and feasible policy options. By the conclusion of the TIER2 project, Key Outputs will have been achieved which will already be translating to outcomes (mid-term benefits achieved during, or max two years after, the project). In turn, these outcomes will result in longer-term impacts (3+ years after project end). In this section we describe these Key Outputs and follow the causal chains (pathways) whereby we are certain they result in maximal short, mid and long-term impact (see Table
In turn, these Key Outputs will directly lead to a host of short- to medium-term Outcomes which significantly address the main concerns in reproducibility. Fuelled by the dissemination and exploitation measures described below (Sec 2.2), by the end of (or max 3 years after) our project, the contributions outlined below in Table
These outcomes will then lead to a multiplicity of longer-term impacts, corresponding to the impacts listed in the Destination programme of the EU. TIER2 will help reform and enhance the EU R&I system, improve access to excellence, deepen the ERA, modernise the higher education sector, increase interconnection of knowledge ecosystems, as well as addressing biases to make research careers more attractive and foster gender equality. At a societal level, higher reliability of research results underlying policy making processes will raise trust in science and R&I outcomes. Economically, greater quality of scientific production will lead to increased re-use of scientific results by research and innovation, and hence stronger translation of R&I results into the economy. Table
To realistically frame the certainty with which we expect our Key Outputs to lead to these Outcomes and Impacts, we next (Table
Effective communication and dissemination processes are crucial for the success of TIER2. Mobilising a strong communication team from PENSOFT, in tandem with OpenAIRE (with outreach to 30+ countries), TIER2 brings together the leaders of excellent international consortia and networks, including the leaders of a large number of European projects on Open Science, Research Infrastructures and Research Integrity. The consortium includes members from diverse disciplines and regions. More importantly, via our established networks, we have direct access to important global networks of researchers, publishers and funders. These previous relations and connections will facilitate the building of a network of stakeholders co-creating, validating and spreading TIER2 output. TIER2’s Scientific Advisory Board further expands our links via the international networks of established experts. Table
Purposefully designed dissemination, exploitation, and communication (DEC) activities are key components for maximising the impact of TIER2. The project’s DEC activities will be streamlined in a Communication Plan Stakeholder Communication and Engagement Plan (D2.1). The plan will serve as a management tool for defining how the project’s progress and results are shared with stakeholders and target audiences. Two updates of the plan (in M24 and M36) will include any necessary modification and adapt appropriately to project progress and new circumstances, including feedback received by co-creation activities. The engagement plan will identify target audiences (‘who’), project research activities and outputs (‘what’), tools and channels employed (‘how’), and the envisioned timeline for implementation of these activities (‘when’). All activities have the objective to maximise the short-, medium- and long-term impact of TIER2 results. In addition, the plan will contain: TIER2 visual identity (logo and graphical layouts guidelines, templates); Communication/dissemination targets, clear distinction between key end users, broad outreach, and general public; Illustration of materials to be produced, communication formats, and online and social media channels which will be used according to the target categories; Indication of how co-creation will be supported by other dissemination channels and tools in WP2; Indication of how external events to be targeted by the project’s dissemination will be selected; Assigning of roles and responsibilities of individual participants in project dissemination; Schedule for the implementation of dissemination and communication activities; Identification of all project outputs and anticipated outcomes, with measurable indicators.
To maximise impacts, TIER2 will actively target its dissemination and communication activities at audiences and stakeholders as defined above. TIER2 will tailor various uni- and bi-directional dissemination channels to the needs of the target stakeholders and audiences, eliciting expertise, knowledge, and perceptions from stakeholders as part of the project’s co-creation and engagement activities. A preliminary mapping of dissemination tools and channels, their target audiences, and relevant Key Performance Indicators (KPIs) is shown in section 2.2.2 and will be refined in D2.1. To effectively disseminate TIER2 results, specific dissemination tools and channels will be established by the project and existing channels used by the consortium participants will be fully exploited. Furthermore, consortium participants will use their existing networks and develop new links, presenting TIER2 and its results at relevant events at local, national, and international scales.
For TIER2 internal communication, mailing lists and communication/document exchange tools, such as MS Teams, will be used as part of a web-enabled communications and learning platform, to create a chat-like environment that simultaneously eases communication and streamlines information and access to documents into relevant channels to alleviate workload and stimulate fruitful and focused discussions. It will provide a place where the TIER2 project team and collaborators can communicate, share documents and work together. Regular consortium and WP calls will ensure timely communication on all issues.
To maximise exposure of project results and their potential for exploitation, TIER2 will take advantage of the EC’s Horizon Results Platform. This platform will serve as a bridge towards researchers and other stakeholders, giving access to the project’s main and prioritised results with a high potential value (Key Exploitable Results). In addition, the project will consider the Horizon Results Booster for dissemination and exploitation of results so that the added value of the KER is amplified through publication in RIO journal, to enhance findability, accessibility, interoperability, and reusability (FAIR), and secure distribution beyond the project’s website. Research outputs described in Table
TIER2 will develop its Intellectual Property framework (IP) in connection with its Data Management Plan (DMP). Management of intellectual property and foreseen protection measures Knowledge generated will be managed in compliance with the Consortium Agreement (CA), which will be signed at the beginning of the project. The CA will address background and foreground knowledge, ownership, protected third party components, and protection, use, and dissemination of results and access rights. The principles are:
Background information and knowledge contributed to the project by each participant will be listed in the CA. When included in the work plan, access to background information will be provided royalty-free to other participants for the implementation of the project’s tasks; Results shall be owned by the participant who generated them. Each participant will be responsible for ensuring fulfilment of their obligations under the GA regarding results by planning with any third parties that could claim rights to them; Whenever results have been produced jointly by two or more participants, the ownership of the results will be shared among the participants who carried out the work. The terms of joint ownership, protection, and share of ownership, and costs for possible protection will be agreed upon in writing via a joint ownership agreement; Each participant will be responsible for examining possibilities to protect results that may be commercially or industrially exploited. When deciding on protection, the participant must consider its own legitimate interests and the interests of the other participants. Participants will ensure that adequate steps towards protection are taken prior to DEC activities, preventing unapproved public disclosure of results, models, tools, and data; Access rights to results will be granted on a royalty-free basis for further research and commercial exploitation. Results and outputs generated by TIER2 can be used freely upon acknowledgment of TIER2’s ownership of these.
In Table
This section provides a detailed description of the TIER2 work plan, including Work Packages, Deliverables, Milestones and key dates.
TIER2 is structured in five Work Packages (WPs, see Figs
In Table
In Table
In Table
In Table
In Table
TIER2 brings a unique combination of skills and perspectives, consisting of eleven partners from seven countries, well spread across Europe. The partner organisations all bring a wealth of expertise and experience regarding the target communities (computer science, life science, social science), research culture reform, technical aspects of infrastructure and services, as well as expertise in social sciences and humanities, and gender aspects of R&I. The partners have collaborated with success in numerous previous projects together. The inter-disciplinary and complementary nature of our mix of knowledge and networks will allow TIER2 to excel in delivering success in our ambitious activities (see Table
These competences and networks are enhanced and supported by TIER2’s international
In addition to the range of perspectives and depth of expertise/experience reflected here, within our consortium and Advisory Board, we also have excellent gender balance (5 of 11 partner PIs and 4 of 8 AB members are female) and excellent international reach across three continents, with exceptional links to further EC Widening Participation countries and world regions. To summarise, Table
TIER2 aims to enhance reproducibility through design, implementation and assessment of systematic interventions addressing key levers of change (tools, skills, communities, incentives, and polices). Our objectives align strongly with broad ethical goals of community-engagement, reform of research cultures to address problematic practices, and increasing trust in research. Activities to achieve the objectives will include both quantitative and qualitative studies, as well as stakeholder engagement via online and live meetings, which will be recorded and analysed. Trainings and training pathways will be evaluated with participants, using surveys and interviews. The activities to achieve these objectives will include the involvement of different stakeholders reproducibility in WPs 2,3,4,5. Only adult participants will be included and will not belong to vulnerable populations. Personal data will be collected from project activities described above and will not include sensitive personal data, such as health and genetic data, or political and religious opinions. The project will also involve non-EU countries (UK and potentially one or more Associated Country). The activities in these countries will be the same as in EU countries involved in the project. Benefit-sharing is planned for any lower middle-income countries that will take part in the project (their involvement will be specifically with the aim of boosting reproducibility capacity in these countries). We do not foresee negative impacts of the project activities with regard to political or adverse financial consequences, misuse or stigmatisation of specific populations. Protection of rights, well-being, safety, privacy and confidentiality of the participants in project activities will be ensured by the coordination and management of the TIER2 governing body in WP1, coordinated by KNOW. Some elements of our research may involve elements of AI, especially in the context of our development of tools to support reproducible researcher workflows and publisher tools for review of data/code. At any step of these activities we will adhere strictly to best-practice guidelines in the design and testing of these systems to ensure there are no ethical concerns related to human rights and values, especially to clearly identify and address any biases contained within training datasets which may negatively affect performance for any populations. The TIER2 governance structures will provide oversight of all project activities, and will include a General Assembly of all partners, meeting in-person during consortium meetings and virtually when required; a steering committee, comprising WP leads and meeting virtually at least 4 times a year; and an external Advisory Board. The TIER2 core leadership team will provide operational, administrative and financial support in all project activities.
The work in the TIER2 project will be led by relevant EU/national legal and ethical requirements of the country or countries where the tasks are to be carried out. The legal and ethics framework includes: The Universal Declaration of Human Rights, The European Convention on Human Rights, The Charter of Fundamental Rights of the European Union, and the EU General Data Protection Regulation. The European Code of Conduct for Research Integrity will be the main ethical and integrity reference followed in the project. As the activities raising ethics issues are related to the involvement of human participants in social sciences/humanities research, TIER2 takes full awareness of all ethical issues addressed in the guides “Ethics in Social Science and Humanities” and “Ethics and Data Protection”, published in 2021 by panels of experts at the request of the European Commission. For the activities in any lower middle-income countries, TIER2 will follow the principles outlined in the Global Code of Conduct for Research in Resource-poor Settings. Research involving human participants - the consortium will ensure compliance with the applicable international conventions, EU and national legal provisions in relation to the conduct of studies involving human participants in social science/ humanities research, data protection and confidentiality. All necessary ethical approvals and authorisations will be obtained prior to the start of the research activities. The consortium is aware of differing practices of approvals for social sciences/humanities research and expects that in some countries ethics approvals are not required or is waived for that type of research. Ethics compliance oversight is dealt with in WP1 (Task 1.1). Protection of rights, well-being, safety, privacy and confidentiality of the participants in project activities will be ensured by the coordination and management of the TIER2 governing body in WP1, coordinated by KNOW. The TIER2 governance structures will provide oversight of all project activities. A Consortium Agreement will describe the rights and obligations of partners and a succinct governance structure, including a General Assembly (all consortium members), the Executive Board (consisting of one representative of each partner organisation), and an external, international Advisory Board. The TIER2 core leadership team will provide operational, administrative and financial support in all project activities. Data from non-EU partners of the consortium will be imported into the EU, in compliance with the laws of the country in which the data was collected. At the moment, EU has made adequacy decision that UK has adequate level of data protection, so that the flow of research data is possible without any further safeguards. The consortium will adopt technical and organisational security measures to prevent unauthorised access to personal data collected during the project, adequately train the research staff and publish the search results in such a way as to do not allow their improper use. Involvement of non-EU countries – the consortium will perform activities and collect data from three Widening Participation countries (to be decided via open call – it hence may be that one or more are Associated Countries). Research and data collection will be performed in the same conditions as in EU countries except if justified by local requirements that do not contradict the spirit of ethical guidelines. All study activities will be submitted to the local legal and ethical committees in compliance with relevant requirements. The consortium confirms that the research to be held in non-EU countries would be allowed in all EU Member states. Some elements of our research may involve elements of AI, especially in the context of our development of tools to support reproducible researcher workflows and publisher tools for review of data/code. At any step of these activities we will adhere strictly to best-practice guidelines (e.g.,
Contract RTD/2020/SC/010, 2020-2022, budget 480k EUR. Final report:
Technological Readiness Level is “a type of measurement system used to assess the maturity level of a particular technology”, cf.
EC DG-RTD. 2022. Assessing the reproducibility of research results in EU Framework Programmes for Research
Project recently funded (Horizon Europe, 2m EUR) under call topic HORIZON-WIDERA-2021-ERA-01-40
MethodHub is under development by Gesis and will be TRL4 by TIER2’s anticipated kick-off
ROAL is under development by ARC and will be TRL5 by TIER2’s anticipated kick-off
Proposal originally made public upon submission at
TIER2 coordinator Tony Ross-Hellauer was co-organiser of the original Reprohack events, hosted as OpenCon satellite events (Berlin 2016, London 2017). See:
Proposal made public at
From the Horizon Europe “Data Management Plan Template”: A qualified reference is a cross-reference that explains its intent. For example, X is regulator of Y is a much more qualified reference than X is associated with Y, or X see also Y. The goal therefore is to create as many meaningful links as possible between (meta)data resources to enrich the contextual knowledge about the data. (Source:
TIER 2 Logo
Levers for research culture change, adapted from (
TIER2’s provisional schema of key elements to map Reproducibility Impact Pathways.
TIER2 methodological steps.
TIER2's pathways to impact.
TIER2 PERT Chart.
TIER2 Gantt chart
List of particiapants.
|
|
|
|
1 (Coordinator) | Know-Center GmbH Research Center for Data-Driven Business & Big Data Analytics (KNOW) | Tony Ross-Hellauer (m) | AT |
2 | Athena - Athena Research & Innovation Center In Information Communication & Knowledge Technologies (ARC) | Thanasis Vergoulis (m) | GR |
3 | Stichting VUmc (VUmc) | Joeri Tijdink (m) | NL |
4 | Aarhus Universitet (AU) | Jesper Schneider (m) | DK |
5 | Pensoft Publishing (PENSOFT) | Lyubomir Penev (m) | BG |
6 | GESIS-Leibniz-Institut Für Sozialwissenschaften EV (GESIS) | Hajira Jabeen (f) | DE |
7 | OpenAIRE AMKE (OpenAIRE) | Natalia Manola (f) | GR |
8 | Charite - Universitätsmedizin Berlin (Charite) | Alexandra Bannach-Brown (f) | DE |
9 | The Chancellor Masters & Scholars of The University of Oxford (UOXF) ( |
Susanna-Assunta Sansone (f) | UK |
10 | Biomedical Sciences Research Center Alexander Fleming (FLEMING) | Martin Reczko (m) | GR |
TIER2's core objectives.
|
|
|
Establish the state-of-play by defining reproducibility across diverse epistemic contexts & inventorying tools/practices & evidence of uptake & efficacy Identify gaps in current knowledge & key tools/practices Design the TIER2 framework for assessing reproducibility impact pathways |
|
Scoping report on state-of-the-art regarding evidence/uptake of reproducibility interventions Detailed inventory of reproducibility tools/practices and their current uptake Framework for assessing reproducibility impact pathways across epistemic contexts |
|
“Creating an open knowledge base of results, methodologies & interventions on the drivers and consequences of reproducibility for the R&I system; and to fill the main gaps in such knowledge” |
|
|
|
Engage co-creation communities to assess user-needs across communities of researchers (social, life, computer sciences), funders, publishers Define priorities for action to increase the reproducibility of research & appraise/validate the applicability of current tools for domain-specific use-cases Plan development/adaptation of reproducibility tools & practices Co-design pilot experiments & their evaluation frameworks Minimise bias in design by ensuring any relevant EDI issues (e.g., sex/gender) are addressed in methodologies, target groups & assessment measures |
|
Detailed scenario-analysis setting priorities for action to optimise reproducibility practices across five contexts (social, life & computer sciences, as well as funders & publishers) Detailed methodological plans for development & piloting of existing tools/practices relating to 5 different contexts (soc, life, comp sci + funders, publishers) |
|
“[F]ill the main gaps in … knowledge”; “Find … solutions and best-practices to increase the reproducibility of research, including through the more systematic integration of sex and gender as variables” |
|
|
|
Develop or adapt reproducibility tools & practices for new contexts Conduct pilot experiments to implement new reproducibility tools & practices across the whole research lifecycle for funders, publishers & scientists Investigate & mainstream concrete solutions & best-practices to fill main gaps in knowledge & increase the reproducibility of research |
|
Eight pilot activities successfully implemented with researchers, publishers & funders New tools & practices for researchers: Reproducibility Checklists; Reproducibility Management Planning tools; tools for reproducible workflows New tools & practices for publishers: Workflows for review of data/code; tools & standards for “threaded” publications; dashboard for monitoring of policies linked to reproducibility New tools & practices for funders: Reproducibility Promotion Plans; funder extension of tool for Reproducibility Management Plans; Reproducibility monitoring dashboard |
|
“Develop, validate, pilot and deploy practices and practical tools for funders, publishers and scientists”; “fill the main gaps in … knowledge”; “experiment and mainstream concrete solutions and best-practices to increase the reproducibility of research” |
|
|
|
Assess pilot activities using impact pathway analysis and conduct econometric analysis Synthesise findings from across project to discover overall gains/savings from reproducibility, including long-term impacts on trust, integrity & efficiency Validate the TIER2 framework on reproducibility impact pathways |
|
Comparative assessment of pilots and synthesised findings on reproducibility gains & savings Validated framework for assessing reproducibility impact pathways |
|
“Determine how increased reproducibility generates gains and savings in the R&I process and improve overall performance - alongside the demonstrated positive effects on their quality, integrity and trust-worthiness” |
|
|
|
Co-create a roadmap for future reproducibility, including policy recommendations/guidelines (for researchers, funders, institutions, & publishers) & reform of reward/recognition structures Targeted upstream stakeholder engagement to maximise uptake/impact of recommendations Auto-ethnographic self-reflection on experiences at the forefront of reproducibility in TIER2 |
|
Validated practical policy & implementation recommendations/guidelines for research funders, institutions, policy-makers, publishers & researchers in Europe & beyond Priorities to reform reward & recognition structures Four policy briefs directed to funders, institutions, policy-makers, & publishers Auto-ethnographic self-reflection report on TIER2’s experiences at the forefront of reproducibility |
|
“Assist further policy development, based on scoping work by the Commission. While solutions should be applicable to Europe, attention should be paid to reproducibility in global science.”; “It is expected that the funded action(s) will adhere to best practices in open science and reproducibility (e.g., re-use existing results, fully document the research process), and provide a final reflection based on their own experience at the forefront of reproducibility” |
|
|
|
Create Build capacity of Reproducibility Networks (incl. creation of 3 new RNs in Widening Countries) Disseminate & valorise project findings, including training & awareness events Ensure interoperability of project tools/results with recognised standards, esp. EOSC Align national, regional & international stakeholders and initiatives to harness network effects to boost capacity for increased reproducibility |
|
Minimum 3 new national Reproducibility Networks created in Widening Participation countries Reproducibility Conference & min. 6 “Reprohacks” co-located with discipline-specific events |
|
“Creating an open knowledge base of results, methodologies and interventions on the drivers and consequences of reproducibility for the R&I system; and to fill the main gaps in such knowledge”; “promote uptake, greater collaboration, and increased alignment of the activities of stakeholders - scientific and technical communities, publishers and funders among others - to increase reproducibility” |
New tools/practices resulting from TIER2.
|
||
Training, skills and information resource for researchers, publishers and funders. |
||
|
||
New concept to extend Data Management Plans to enable reproducible research. |
||
|
||
Containerised workflows to facilitate reproducibility and data/code reuse in social (lead: GESIS), life (lead: UOXF), and computer sciences (lead: ARC). |
||
|
||
Streamlined publisher workflows for review of data/code to facilitate publishing checks in soc, life and comp sci. |
||
|
||
New common standards and best practice guidelines for to enable links between connected research outputs - and associated meta-data descriptors (e.g., Grant information; author and contributor details). |
||
|
||
New tool to assist funders create a holistic plan to increase reproducibility of their results. |
||
|
||
Dashboard for funders to check levels of FAIRness and re-use of research objects from funded research. |
Key reproducibility issues in TIER2's selected five cases.
|
Social sciences can be conceptualised as a very heterogeneous field, encompassing diverse epistemological and methodological approaches, working with various kinds of data including opinion polls, voting records, surveys, self-reported perceptions, behaviours, beliefs or attitudes, social network data, government statistics and indices, GIS data measuring human activity, and various forms of qualitative data, such as interview transcripts, field notes, and observational protocols. Reflecting this diversity, recognition of reproducibility as an issue greatly varies. While parts of psychology have been a dominant part of the ‘reproducibility crisis’ discussion since the start ( |
|
Life science is a very heterogeneous field with a number of disciplines, study and technology types. The data community in biology and medicine, however, is probably the most active one in creating data and metadata standards to support the reuse (including reproducibility) and sharing of the information, supported by strong international data mandates (e.g., 1996 Bermuda for genomics data, 2009 Toronto agreement for omics/clinical data). Much of the understanding of scientific transparency stems from the experience in bioinformatics, where the focus has been on information (incl. datasets, code, models and software) that is harmonised with respect to structure, format and annotation. Nowadays there are over minimal reporting requirements (outlining the necessary and sufficient information vital for contextualising and understanding of data and other digital objects; terminologies (ranging from dictionaries to ontologies, provide definitions and unambiguous identification for concepts and objects, and statistical results, e.g. models and formats (defining the structure and relationship of information for a conceptual model and include transmission formats to facilitate the exchange of data between different systems, including workflows, e.g. |
|
Computer science research involves a large degree of determinism (high precision of goals, high dependence on statistics, total control on environment). In experimental work, high specification (far higher than in lab or other types of experimental research) of methods is theoretically possible since each computational action is logged ( |
|
In line with growing concerns about the quality and credibility of research and publishing processes, ensuring the reliability of published research has become increasingly important to publishers and journal editors over the past decade. Several approaches to incentivise or improve the reproducibility of the published record have been proposed. These include both pre- and post-submission measures, such as journal reporting guidelines, improved peer review practices and data availability requirements. However, recent studies have shown that, while improving, the number of journals explicitly demanding or enabling data sharing practices is still limited ( |
|
Research funders have similar motivations to journal editors. Where research is open and reproducible research, it helps maximise potential impact and return on investment (ROI). Funders of scientific research are well positioned to guide scientific discoveries by enabling and incentivising the most rigorous and transparent methods. There are numerous recommendations on how funders should act in order to increase reproducibility ( |
Existing TIER2 consortium software/platform assets.
|
||
Argos is an open, extensible, and configurable machine-actionable tool developed to facilitate Research Data Management activities concerning the implementation of Data Management Plans (DMPs). | ||
|
||
The Embassy is a wiki-based platform for the research community to share experiences and insights about research integrity and ethics, continuously contributing to the development of good science. | ||
|
||
FAIRsharing is a curated registry, tool and service for data and metadata standards, inter-related to repositories and data policies. It hosts 1600 standards, 1900 databases, and 160 policies. | ||
|
|
|
da|ra is a DOI registration service in Germany for social science and economic data, in cooperation with DataCite. It has more than 699k DOI registrations, and ~4m DOI resolutions per year. | ||
|
|
|
gesisDataSearch enables searching for research datasets based on a periodically crawled index. It has more than 100,000 Datasets with approximately 600 new users per month. | ||
|
||
Integrated search across different Social Science data collections at GESIS. It has 6,500 datasets, 13,000 survey variables, 400 measure instruments, and 107,000 publications. | ||
|
||
MethodHub builds on containerisation technologies and digital lab notebooks to facilitate social science data analysis and its reproducibility, to find, learn, and experiment with computational methods. | ||
|
|
|
OpenAIRE Research Graph aggregates research data properties (metadata, links) for funders, organisations, researchers, research communities and publishers to interlink information. | ||
|
||
ROAL (ReprOducibility Assessment tooLkit) streamlines assessment of reproducibility by automating the identification of datasets, their classification in terms of re-use, and the extraction of metadata. | ||
|
||
SCHeMa facilitates reproducibility of computational experiments on heterogeneous clusters, exploiting containerization, experiment packaging, workflow description languages, and Jupyter notebooks. |
TIER2 Provisional list of TIER2 pilot activities (to be refined/adapted in line with co-creation methods).
researchers: active Horizon Europe projects; funders: minimum three funders |
||
S |
||
TIER2 Key Outputs.
|
|
|
|
|
|
|
|
|
|
|
|
Reproducibility Hub, a sustainable open knowledge base of results, methodologies & interventions on the drivers & consequences of reproducibility for the R&I system hosted at the Embassy of Good Science; Over 1000 researchers, publishers & funders engaged to increase skills via outreach & co-creation events (Reprohacks, training events, webinars, final conference) & Reproducibility Hub functionalities (e.g., Checklist tool, training modules); Empowered networks, with at least 3 new national Reproducibility Networks created in Widening Participation countries & all RNs linked with Open Science, Research Integrity, research infrastructure (EOSC), publisher/funder networks & co-creation communities for increased capacity & multiplier effects in EU & beyond. |
||
|
|
|
TIER2 contributions to outcomes.
TIER2 Contributions to impact.
Through its Key Impact Pathways methodology & increased awareness of the meanings, gains & savings of reproducibility across epistemic contexts ( |
By bringing our tools into the EOSC ( |
Increased reproducibility will come with greater quality of research results. Quality here should be taken in a broad sense, however: more reproducible results will result in less waste, in healthier research communities, & in more efficient & effective research endeavours. Giving the right tools ( |
Scale & significance, barriers & mitigation measures for impact of TIER2 Key Outputs.
|
|
|
|
TIER2 key stakeholders.
|
|
Researchers | |
Funders (RFOs) | Strong links via OpenAIRE, VUmc, to: |
Publishers | Strong links via OpenAIRE, UOXF, PENSOFT, KNOW, AB (Catriona Maccallum) to: |
Reproducibility Networks | Strong links via Charite, UOXF, Advisory Board, and collected Letters of Interest to RNs in |
Research Infrastructures (EOSC) | Strong links via OpenAIRE, UOXF, GESIS, FLEMING, KNOW to: |
Research Performing Organisations (RPOs) | Strong links via OpenAIRE & others to: |
Research Integrity officers/trainers | Strong links via VUmc, AU to: |
Scholarly/learned societies | Strong links via OpenAIRE and all partners to scholarly societies in a range of disciplines |
Libraries & library organisations | Strong links via OpenAIRE to |
Research administrators | Links to |
Civil Society Organisations/General public | General media (via press releases), blogs & online media, online courses, newsletter, videos, special interest groups |
Industry/SMEs | Industry events, industry media, EC events |
Key Exploitable Results & appropriate dissemination measures.
|
|
|
Enable systematic assessment of the impact of reproducibility interventions across contexts to optimize action; enable assessment indicators; provide ideas/inspiration to stakeholders for actions on reproducibility | ||
Showcase tools via Reproducibility Hub (Embassy of Good Science), EOSC & RN websites; disseminate widely in respective researcher, funder & publisher communities; highlight good practice; TIER2 partners will take up relevant tools within own practices | ||
Enable multiplier effects through linkage of networks (RNs linked with Open Science, Research Integrity, research infrastructure (EOSC), publisher/funder networks); provide knowledge & training both online & in-person; RI training among researchers | ||
Create practical policy & implementation recommendations/guidelines/briefs for research funders, institutions, policy-makers, publishers & researchers in Europe & beyond. By the end of the project 30 funders, institutions & networks will have endorsed the recommendations. |
Summary.
|
|
|
Discussions on reproducibility driven by a |
||
|
|
|
Work package descriptions 1.
WP No. | 1 | Lead beneficiary | KNOW | Start month | 1 | End month | 36 |
WP title |
|
||||||
Coordinate the Grant Agreement, Consortium Agreement, & signature procedures; Develop, disseminate & monitor adherence to a Project Management, Research Integrity Plan, a Quality Assurance Plan, Data Management Plan, & a Publications Agreement; Coordinate ethics deliverables; Support communication & collaboration between WPs, & the alignment & integration of work between them; Implement & maintain internal reporting & monitoring procedures (including internal review procedures, gender distribution in the consortium composition, & sex, gender & diversity considerations in the project activities); Monitor & support the completion on schedule of deliverables & milestones & take corrective action in the case of delays; Mediate in cases of internal disputes; Organise the General Assembly, Executive Board & Advisory Board meetings Handle daily project correspondence & requests from partners & external bodies; Produce periodic & final reports for the EC. Calculate & distribute EC payments to project beneficiaries according to the Consortium Agreement; Liaise with finance departments to monitor contracts; Maintain financial records; Coordinate financial statements submission by all project partners; Provide financial overviews for the periodic & final reports for the EC. |
|||||||
Work package descriptions 2.
WP No. | 2 | Lead beneficiary | PENSOFT | Start month | 1 | End month | 36 |
WP title |
|
||||||
Work package descriptions 3.
WP No. | 3 | Lead beneficiary | KNOW | Start month | 1 | End month | 36 |
WP title |
|
||||||
Work package description 4.
WP No. | 4 | Lead beneficiary | VUmc | Start month | 3 | End month | 30 |
WP title |
|
||||||
Use an innovative backcasting, co-design methodology to work with communities to sketch & select optimal scenarios for future reproducibility across contexts, & identify key areas for development to achieve these scenarios. Plan & design interventions that can help important stakeholders see what the costs & benefits are for reproducibility across epistemic contexts for researchers, funders & publishers. Prepare & run pilot scenarios for tools & practices developed in WP5 to showcase, verify, & evaluate them across methodological contexts |
|||||||
Work package description 5.
WP No. | 5 | Lead beneficiary | ARC | Start month | 9 | End month | 34 |
WP title |
|
||||||
5.1.1) 5.1.2) 5.1.3) 5.2.1) 5.2.2) 5.2.3) extension of the F1000 transparent publishing model to new contexts – especially the recently launched Routledge Open Research (for humanities/soc sci) Open Research Europe (covering all Horizon research), especially definition of reporting standards/guidelines for qualitative research; extension of T&F’s workflow for registered reports to new disciplines (e.g., education) to investigate the efficacy (or not) of this model for new epistemic contexts. In addition, with leading conferences, we will also investigate potential for registered reports in Computer Science publishing. 5.3.1) 5.3.2) 5.3.3) |
|||||||
TIER2 Consortium experience and expertise.
Overview of areas of TIER2 expertise to address key challenges.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|||
|
|
|
|
||||||
|
|
|
|
||||||
|
|
|
|||||||
|
|
|
|
||||||
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|||||
|
|
|
|
|
|
|
|||
|
|
|
|||||||
|
|
|
|
|
|
|
|
|
|