DESCRIPTION OF THE STATE-OF-THE-ART

The open science paradigm, digitalisation, interdisciplinarity and internationalisation have all significantly changed the research process, collaboration, dissemination and impact of scholarly work in the 21^st century. Science is not only publication-based anymore. A variety of new types of research output has emerged in recent decades: datasets, software, workflows and specific article types, such as data papers, reproducibility papers, code papers, executable papers etc. Despite the recognition that open science involves a wide range of practices and considerations, such as open data, open reviews, open workflows and open education (Guy and Ploeger 2015), many policies focus on regulating and encouraging open access to publications alone (Leonelli et al. 2015, The PASTEUR4OA consortium 2016). ROARMAP is an international, searchable registry with more than 1,000 catalogued open access mandates and policies, primarily related to open access publications. With a few notable exceptions, rewarding other open science practices is still in the recommendation stage, rather than the implementation stage. For instance, the Open Science Career Assessment Matrix (OS-CAM), which recognises the diversity of open science outputs and practices, has yet to be implemented. Moreover, communication with society has changed and the involvement of citizens in team science, as well as the interdisciplinarity of research teams, are encouraged. This a topic has already been discussed in the community at various workshops, such as the two-day “Digital Transformation in Scholarly Communication” workshop organised by Science Europe in November 2019. There is a demand to demonstrate the impact of research beyond academia and to democratise this impact by giving greater voice and vote to under-represented groups (gender, ethnicity, disability, geographic etc.) in determining it (Sugimoto 2016, Memisevic et al. 2017).

Therefore, communication and dissemination channels have changed and citations of textual publications are no longer the only measure of impact. New metrics, which may include Web 2.0 channels, can provide information for research impact assessment (Cronin et al. 1998). Open science, social media metrics and alternative metrics represent the groundwork and components for the new concept – Scientometrics 2.0. In 2001, Cronin presciently predicted that, as a more diverse publishing environment emerges, bibliometricians will have a much broader array of objects and artefacts to feed their accounts and analyses (Cronin 2001). Scientometrics 2.0 embraces the move to new platforms for scholarly practice as a source for this “broader array” and presents an exciting opportunity to improve our image of scholarship.

However, the majority of metrics are still citation-based, which is a slow process (due to the time required for articles to accumulate citations) and a biased one (as larger fields tend to accumulate more citations than specialised ones, regardless of quality). Therefore, this narrow focus is not suited to the increasingly fast and broad Web 2.0-based scholarly world. While using citations for other types of research output, such as to data and software, is increasingly encouraged, the infrastructure and standards for this approach are not yet well developed. The Next Generation Metrics report (Wilsdon et al. 2017) recommends using citations for all research outputs and highlights that the academic assessment indicators can only reach their full potential if they are underpinned by an open and interoperable academic data infrastructure, based on persistent identifiers (PIDs). Lowenberg et al. (2019) proposed a vision for the development of open metrics, i.e. metrics that are open and can be used, shared and built upon freely by anyone. This vision also includes recommendations for metrics to be inclusive (i.e. applicable to all disciplines and to consider all data types), transparent and multi-dimensional (e.g. by not conflating multiple dimensions into a single number). The Snowball Metrics initiative (Clements et al. 2017) similarly calls for more open metrics, where calculations and recipes are made publicly available. The Make Data Count project (Cousijn et al. 2019) addresses the significant social as well as technical barriers to the widespread development of data metrics.

In recognition of the fact that non-publication research outputs (e.g. data, software and other outputs) are not literature and that citations may not adequately capture the range of uses for these outputs, alternative metrics (altmetrics), such as social network activities, views, downloads etc. are being investigated as a way to demonstrate the attention received by a range of outputs and audiences (Fecher and Friesike 2013). Similar to citation databases, there are different data sources, ranging from individual repositories to aggregators, which can define and use numerous metrics for research outputs. The authors of “Open Data Metrics: Lighting the Fire” (Lowenberg et al. 2019) suggest differentiating data usage at different levels, for example, different levels of views (of metadata, images etc.) and downloads (file or dataset level). They warn that, without standardisation of counting views and downloads, any definition of these used by various stakeholders can be arbitrary. While counts are not metrics per se, they can be standardised in order to create a metric. The COUNTER Code for Research Data is a promising way to standardise data usage counts, but, at the moment, there are many different ways of counting, which are being implemented. Although there are calls for individual repositories to report their usage metrics to centralised data brokers, such as DataCite, this rarely occurs (Konkiel 2020). Transforming these counts into a measure of data reuse is a thorny issue and raises many questions: What type of use? Scholarly? Societal? By whom and for which purposes? Are there different levels of views (of metadata, images etc.) and downloads (file or dataset level)? Standardisation of counting views and downloads is needed.

Social media activity can be used to measure attention by using indicators such as shares, likes, tweets, comments and downloads (Sugimoto 2016). Altmetrics aggregators, such as PlumX and Altmetric, collect diverse alternative metrics from various sources. Alternative metrics offer many potential benefits for assessing new research formats and outputs, as well as new types of usage. Altmetrics can assess interdisciplinary research and the impact of scientific results on the society as a whole, because they can include the views of all citizens, not just other scholars (Wilsdon et al. 2017, Ravenscroft et al. 2017).

Although social media metrics and other altmetrics (e.g. tweets and downloads) offer an alternative to publication-based metrics, they are rarely incorporated into assessments of researchers and the science of science. This may be due to known challenges, such as data quality (e.g. accuracy, coverage, comprehensiveness), the heterogeneity of data sources and application programming interfaces (APIs) and potential manipulation (e.g. metrics gaming). Altmetrics are mostly tied to provider platforms and their regulations (e.g. X/Twitter's rules for data sharing); the quality of the underlying data; and their potential to be manipulated by bots. There is also a lack of consistency and accuracy, amongst others (Haustein 2016). Data collected by altmetrics aggregators are also biased towards the parent companies’ data sources (e.g. Altmetric towards Digital Science / Dimensions and PlumX towards Elsevier). Unfortunately, aggregators do not have meaningful disciplinary benchmarks at the moment (Konkiel 2020). Fenner (2013) identifies some additional questions that need to be addressed before altmetrics can be used for assessment, such as whether altmetrics can be applied across disciplines and over time to measure research; what they measure (e.g. impact or popularity); and whether their application in assessment can produce undesired incentives. Furthermore, some analysts argue that major social academic networking sites have not yet proven their essential worth (Van Noorden 2014) and that they should not “count toward scholarly reputation” (Jamali et al. 2015).

Responsible research assessment guidelines (e.g. the San Francisco Declaration on Research Assessment, the Leiden Manifesto for research metrics, the Hong Kong Principles for assessing researchers) (Cagan 2013, Hicks et al. 2015, Moher et al. 2020) converge on the themes of diversity of research activities and outputs, transparency of systems and indicators and reflexivity about indicators and practices. They also highlight the need for robust, comparable data and open and Findable, Accessible, Interoperable and Reusable (FAIR) assessment infrastructure, which can be used to calculate metrics to use in providing information for (and not completely driving) a variety of assessments. Moreover, these topics are discussed at some conferences, forums and in projects by bibliometricians: https://lisbconference.wordpress.com/, https://bric-conference.ca/, https://www.nihlibrary.nih.gov/services/bibliometrics/bibSymp20/agenda, https://blog.tib.eu/2020/06/02/a-public-archive-for-the-videos-of-the-altmetrics-conferences-and-workshops/, https://thebibliomagician.wordpress.com/, https://www.uantwerpen.be/en/conferences/30th-annual-international-conference-on-science-and-technology-indicators/ and https://avointiede.fi/en/networks/eosc-co-creation.

APPROACH TO THE CHALLENGE AND PROGRESS BEYOND THE STATE-OF-THE-ART

To obtain a comprehensive picture of the state of development and uptake of Scientometrics 2.0, as well as the challenges to its application and wider adoption, the expertise of the COST Action members from different backgrounds will be used to create a directory of relevant resources and to design a survey, disseminate it and analyse the resulting data. Additionally, a review of the state-of-the-art will be conducted and a roadmap will be created to overcome the challenges of implementing Scientometrics 2.0. The roadmap will include a vision for the scholarly community and a set of recommendations to achieve it. The Action participants will study certain technologies and software ecosystems around them. Besides strengths and perspectives of technologies and software systems, their limitations should be analysed as well. The plan will strike a balance between feasibility and usefulness. This means that all the challenges of Scientometrics 2.0, such as the heterogeneity of data sources and data reliability (i.e. data quality and gaming) will be analysed in depth and realistic solutions for overcoming these challenges will be suggested. Similarly, any suggested solutions must not undermine the main benefits of using Web 2.0 to study the impact of research and must take into account the specific characteristics of various scientific disciplines. These characteristics include how certain scientific fields generate knowledge, what kind of scientific and societal interactions characterise these fields and what are the patterns of dissemination in these fields. Therefore, the Action will adopt an innovative approach to address these challenges by combining the following four perspectives:

1. A technology-driven perspective
   • by analysing what the technology and the current research in the area can contribute.
2. A domain-driven perspective
   • by analysing the specific practices and scientific activities of the Action participants with different scientific backgrounds.
3. A region-driven perspective
   • by analysing the specific practices and scientific activities of the Action participants coming from various parts of Europe and
   • by examining the need and possibility for aligning policies across Europe to increase its global competitiveness.
4. A purpose-driven perspective
   • by analysing general cultural changes in science, in publishing research outputs and communication over the last decade and
   • by analysing the different purposes for which academic metrics are used, such as:
     • individual and group academic assessment, the science of science (trends, patterns etc.) and the study of correlation between policies and scientific results.

To address the identified challenges, the Action developed its strategy, based on the following three fundamental pillars:

1. Standardisation:
   • studying techniques and novel methods for the standardisation of formats, APIs and query languages to facilitate the use of metrics data sources and the application of artificial intelligence techniques
   • studying machine-actionable language notations for expressing metrics.
2. Quality of academic metrics data sources:
   • studying techniques for representing data quality and provenance and for making verifiable claims
   • studying techniques for detecting metrics gaming
   • assessing comprehensiveness of metrics data sources.
3. FAIRness of academic metrics:
   • building a register of available metrics data sources and academic metrics
   • adapting the FAIR principles for the description of metrics.

Addressing the challenges requires a complete solution that exploits synergies between several disciplines. This is typically not achievable within funding frameworks other than COST, due to the specific focus and numerous areas of expertise usually required of applications. To maximise knowledge sharing amongst participants and the establishment of lasting professional relationships, brainstorming, networking sessions and team-building activities will be organised. Meetings will be organised, where participants will be grouped together and asked to produce a 'tangible' outcome of their work or to discuss a topic in a short amount of time. These efforts are necessary for the rapid development of business applications that bridge the gap between the requirements for the application of Scientometrics 2.0 and the available software solutions.

ADDED VALUE IN RELATION TO EXISTING EFFORTS AT EUROPEAN AND/OR INTERNATIONAL LEVEL

The research ideas proposed by this Action have only been partially addressed by other initiatives and research projects. Many initiatives and declarations suggest that approaches to the science of science and the assessment of scientific results and impact should change in line with cultural shifts amongst scientists and their activities and outputs:

• The San Francisco Declaration on Research Assessment, known as DORA, facilitates the development of responsible research and researcher assessment policies and practices (Cagan 2013);
• The Leiden Manifesto for research metrics defines best practices in metrics-based research assessment and suggests that efforts for evaluators and applicants should be decreased, which is possible to achieve by developing appropriate e-infrastructure (Hicks et al. 2015);
• The Metric Tide report discusses and elaborates criteria of responsible metrics from the perspective of the UK's Research Excellence Framework (REF) (Thelwall et al. 2015);
• The Hong Kong Principles focus on the need to drive research improvement through ensuring that researchers are explicitly recognised and rewarded for behaviour that strengthens research integrity (Moher et al. 2020);
• The INORMS Research Evaluation Working Group was established to consider how best to ensure that research evaluation is meaningful, responsible and effective (Gadd et al. 2021);
• FORCE11 is an international coalition of researchers, librarians, publishers and research funders working to reform or enhance the research publishing and communication systems (Martone 2015);
• The COST Action ENRESSH is an example of a European effort to propose best practices in the field of social sciences and humanities (SSH) research evaluation (Kulczycki et al. 2018);
• The Latin American Forum for Research Assessment (FOLEC) is an example of a regional space for debate and exchange on the meanings, policies and practices of research evaluation (Beigel 2020).

Moreover, there are some more practically orientated solutions related to the Action theme, which are outlined below:

• Scimeter (https://scimeter.org/) allows everyone to create their own metric to capture what they personally consider the best possible way of quantifying scientific impact of arXiv preprints;
• Dimensions (https://www.dimensions.ai/) maps the entire research lifecycle, meaning users can follow research from funding through research output to impact. Users are encouraged to use the broad range of connected data available in the Dimensions database to develop the next generation of useful metrics;
• Digital Science and Elsevier developed Altmetric (https://www.altmetric.com/) and PlumX Analytics (https://plumanalytics.com/) tools for tracking where published research is mentioned online, i.e. for measuring the influence of scientific research outputs. Both tools offer embeddable widgets/badges to display metrics on a website, as well as APIs for fetching all metrics values assigned to a research output;
• The European Research Infrastructure for Science, Technology and Innovation Policy Studies (RISIS) (https://www.risis2.eu/) is a Horizon 2020 project aiming at building data and services infrastructure supporting the development of a new generation of analyses and metrics. The RISIS-KNOWMAK tool, which is a result of this project, enables the analysis and download of a number of relevant integrated metrics on knowledge production in Europe (https://www.risis2.eu/risis-indicators/).

Therefore, the need to change the techniques and metrics used to measure impact of research, due to development of Web 2.0 and the cultural changes amongst researchers is recognised. Unfortunately, some of the challenges associated with implementing this change have yet to be resolved. Therefore, the theme of the Action is still not yet fully addressed by other European and global projects and initiatives. However, the theme is recognised as important within the community. Some of the above-mentioned initiatives are goal-orientated without analysing the challenges or the feasibility of the recommendations. Moreover, some of the referenced projects and initiatives are discipline- or country-specific. The Action will analyse the challenges from the following four perspectives: technology-driven, domain-driven, region-driven and purpose-driven. For all these perspectives, knowledge and expertise in informetrics, information science, computer science and social sciences are necessary, as well as representatives of different scientific fields and parts of Europe. Moreover, metrics data providers and policy-makers could contribute to a better understanding of, and response to, the challenges. There is no 'one size fits all' solution when it comes to measuring the impact of research in different scientific disciplines, parts of Europe and for different purposes. Therefore, networking is essential for addressing challenges concerning changing techniques and metrics for measuring the impact of research at the European level. The COST Action programme is the ideal tool for bringing together the expertise and skills from different scientific fields and European regions and for engaging in research projects in challenging areas. The Action will facilitate bringing multidisciplinary people together and help establish a common ground for communication, which is pivotal for working on the identified challenges.

SECURING THE CRITICAL MASS AND EXPERTISE

The achievement of the RCOs and CBOs (see Sections 'Research Coordination Objectives' and 'Capacity-building Objectives') requires the following knowledge and expertise from several disciplines at the intersection of informetrics, information science, computer science and social sciences:

• expertise in diversified, but complementary fields, as well as with different research skills, ranging from theoretical to practical, to address the multifaceted nature of the challenges;
• different countries and parts of Europe are considered, taking into account the specificity of scientific practices and activities by different regions and disciplines;
• different goals
  • researchers, who are aiming to start new research projects and identify new challenges;
  • early career researchers with the goal of consolidating their skills;
  • trainers, who want to find new materials and scenarios for their activities;
  • researchers from commercial metrics data providers can collect the feedback on what their infrastructure should offer;
  • research software engineers can identify the gaps and needs for the implementation of new e-infrastructure and
  • policy-makers, who want to learn about the barriers and possibilities of applying metrics based on Web 2.0 in research impact assessment.

The network of the Action proposers consists of 27 Members belonging to organisations in 22 countries. The network includes representatives from the following five stakeholder categories:

• Researchers will bring expertise in theoretical and practical aspects of informetrics, information science, computer science and social sciences;
• Research software engineers will bring expertise in technical and practical aspects of building and maintaining research e-infrastructures, APIs and GUI;
• Librarians will bring expertise in information science and cataloguing scientific outputs and activities;
• Representatives of the metrics data providers will share their expertise on existing challenges and how they can be addressed. They will also discuss what is needed to overcome these challenges and when this could be achieved;
• Policy-makers will contribute their expertise in group and individual assessment of research outputs and activities. They will also share their experience of creating undesired incentives when metrics are introduced to provide information for research impact assessments.

The Network already includes a large number of participants, but further activities are planned to involve and reach a critical mass of active experts, which will ensure sustainability of the network beyond the Action (see the Section 'Capacity-building Objectives').

SCIENTIFIC, TECHNOLOGICAL AND/OR SOCIOECONOMIC IMPACTS (INCLUDING POTENTIAL INNOVATIONS AND/OR BREAKTHROUGHS)

The Action will contribute to the transparency, reliability and reusability of metrics. Moreover, it will foster metrics that consider the various Web 2.0 channels used by scholars to communicate with the wider public. The Action will shed new light on the science of science and analyse the impact of research. Moreover, assessments informed by such metrics could lead to the hiring of better candidates and the selection of better project proposals. Therefore, they could have a long-term impact on the research process, benefitting society and the economy as a whole. Additionally, the use of domain-specific language for the machine-actionable description of academic metrics, as well as the standardisation of APIs of metrics data providers, could make the assessment process more transparent and simpler, thus saving time. Furthermore, it will facilitate the integration of existing science of science tools and the development of new ones, as well as the application of artificial intelligence techniques. Additionally, the existence of metrics based on Web 2.0 tools may encourage researchers to make their research results more transparent and change their communication practices. This could generate greater interest in academia within the wider society. Ultimately, the scientific and technological results of the Action, such as the domain-specific language for machine-actionable description of academic metrics, techniques for dealing with data quality and metrics gaming, can be adopted and applied in other domains (see Tables 1, 2).

KNOWLEDGE CREATION, TRANSFER OF KNOWLEDGE AND CAREER DEVELOPMENT

The Management Committee (MC) members will implement the plan for knowledge creation, transfer and career development, supported by the Core Group Members of the Action. These members will play an active role in WGs, acting as scientific and training coordinators. They will be responsible for organising and facilitating the interdisciplinary cooperation between researchers, identifying and creating opportunities for holistic research approaches and developing career opportunities and outcomes within and beyond the WGs. New knowledge will be created through collaboration between the Action participants with diverse backgrounds and expertise, inlcuding different research skills, ranging from theoretical to practical, different methodologies and experience in various fields. This knowledge will be transferred internally to other Action members through presentations at regular WG meetings and externally to non-Action members through three training schools and two open conferences organised as a part of the Action, as well as through the dissemination and exploitation channels described in Section 'Plan for dissemination and/or exploitation and dialogue with the general public or policy'.

ECIs will be divided into small interdisciplinary teams, with a senior scientific mentor assigned to each group. The mentors will help the ECIs to organise their activities within the Action and to recognise opportunities for networking and career development, both within and outside the COST Action. Furthermore, senior scientific mentors will recommend ECIs from their groups for the “Fellow of ENFAIRAM” award, which will recognise the most active ECIs. The criteria for assessing Short Term Scientific Mission applications, which will be defined at the beginning of the Action, will prioritise collaboration between ECIs and senior researchers, as well as between participants with different levels of knowledge and from different regions. Besides the creation of knowledge, these STSMs will faciliate the knowledge transfer between STSMs participants (i.e. hosts and visitors), especially from senior researchers in EU countries to ECIs in ITCs. This knowledge transfer, as well as the networks established and the scientific results gained, will support the ECIs in their career development.

The Action will focus on organising large, open meetings, where invited participants can have opportunities to benefit from networking opportunities, including with participants from outside the Action. Particular attention will be given to the themes addressed at these meetings, which must balance the interests of academics and commercial metrics data providers.

The web repository of academic metrics is open to the addition of new metrics and discussion. This will enable knowledge tranfser beyond the Action. The Action members will ensure sustainability (i.e. by hosting and moderating of the repository for at least five years after the Action).

DESCRIPTION OF WORKING GROUPS, TASKS AND ACTIVITIES

The Action participants will be included in three Working Groups (WG 1–3), reflecting the three main pillars described above (see Section 'Approach to the Challenge and Progress beyond the State-of-the-Art') and contributing to the research coordination objectives stated in the Section 'Research Coordination Objectives'. An additional fourth group will be dedicated to the coordination and dissemination of the Action’s activities and outcomes. Therefore, WG4 will contribute to the capacity building objectives stated in Section 'Capacity-building Objectives' and ensure a maximum visibility of the Action.

[WG1] Standardisation of academic metrics and its data sources

Task 1.1 (preparation) Analysis of the state of the art

Activity 1.1.1 Identifying available metrics usage, metrics data sources, pros and cons of their APIs and supported formats: designing a characterisation matrix; collecting and characterising available web resources (documentation, APIs, papers, initiatives etc.); designing a survey; disseminating a survey; analysing the survey results and resources.

Activity 1.1.2 Identifying available techniques and novel methods for standardisation of formats, APIs and query languages for simple use of metrics data sources: designing a characterisation matrix; collecting and characterisating available web resources (documentation, papers, blogs, initiatives etc.); analysing the resources.

Activity 1.1.3 Identifying machine-actionable and domain-specific language notations, which may be used for expressing metrics: designing a characterisation matrix; collecting and characterising available web resources (some techniques from legal informatics and computational law might be adopted for building the language for computational metrics); analysing the resources.

Task 1.2 (implementation) Standardisation of the metrics data sources APIs

Activity 1.2.1 Analysing research e-infrastructures relations with metrics data sources.

Activity 1.2.2 Specifying MDS API (metrics data sources’ standard API): endpoints, functionalities, input and output format.

Activity 1.2.3 Specifying AcaMet domain-specific language for expressing how metrics should be automatically calculated, based on metrics data sources: selection of language notations for building a domain-specific language for expressing metrics; definition of AcaMet domain-specific language.

Task 1.3 (exploitation) Building a roadmap for machine-actionable academic metrics

Activity 1.3.1 Developing a vision for an architecture of a distributed global research e-infrastructure and its relation with metrics data sources, based on the standard API.

Activity 1.3.2 Building a roadmap for interoperability of metrics data sources and platforms for the science of science and academic assessment.

Activity 1.3.3 Making examples of representing metrics in the AcaMet language.

[WG2] Quality of metrics data sources

Task 2.1 (preparation) Analysis of the state of the art

Activity 2.1.1 Identifying available techniques and practices for representation of data quality and provenance and assigning verifiable claims: designing characterisation matrix; characterisation of available web resources (i.e. documentation, papers, project reports, W3C standards, initiatives etc.); designing a survey; dissemination of a survey; analysing of the survey results and characterised resources.

Activity 2.1.2 Identifying available techniques and novel methods for detecting data gaming which might be applied in detecting academic metrics gaming: designing characterisation matrix; characterisation of available web resources (the available techniques in other domains, such as gaming in search engine relevance scores, should be analysed and adopted for the metrics data domain); analysing of the resources.

Activity 2.1.3 Identifying available techniques and methodologies for comparing coverage and comprehensiveness of databases and indexes: designing characterisation matrix; characterisation of available web resources (the available techniques for assessment of coverage and comprehensiveness in other domains, such as scholarly outputs databases and search engine indexes, should be analysed and adopted for the metrics data domain); analysing of the resources.

Task 2.2 (implementation) Development of frameworks for assessing reliability of metrics data sources

Activity 2.2.1 Creating an evaluation framework with criteria for the data quality in the metrics data source (e.g. provenance tracking, data verification process status, verifiable claims).

Activity 2.2.2 Creating a framework for assessment of metrics data providers databases’ coverage and comprehensiveness.

Activity 2.2.3 Developing recommendations for detecting and dealing with gaming of academic metrics

Task 2.3 (exploitation) Case studies for assessing reliability of metrics data sources

Activity 2.3.1 Validation of the developed framework by assessing the data quality of a few popular metrics data providers databases.

Activity 2.3.2 Validation of the developed framework by assessing comprehensiveness of a few popular metrics data provider databases.

[WG3] FAIRness of academic metrics

Task 3.1 (preparation) Specification of a web register of metrics data sources and metrics

Activity 3.1.1 Use case analysis.

Activity 3.1.2 Specifying a set of functionalities.

Activity 3.1.3 Analysing free and customised solutions, which may be used for the development of the web register.

Task 3.2 (implementation) Development of a web register of FAIR metrics (ENFAIRAM)

Activity 3.2.1 Integration, customisation and deployment of a selected set of solutions.

Activity 3.2.2 Prescribing and documenting processes and conventions for adoption of FAIR principles in cataloguing of metrics in the repository.

Activity 3.2.3 Prescribing and documenting processes and rules for moderating discussions in the web register.

Task 3.3 (exploitation) Filling in the web register

Activity 3.3.1 Cataloguing existing metrics and data sources.

Activity 3.3.2 Moderating discussions. This activity will be continued for at least five years after the end of the Action.

Activity 3.3.3 Defining new academic metrics: identifying gaps for missing sources or metrics for some research output or activity, taking into account different practices by numerous scientific disciplines; introducing new metrics based on Web 2.0 data sources (academic metrics 2.0); cataloguing new metrics.

[WG4] Coordination and dissemination

Task 4.1 Enhancement of participation in the Action

Activity 4.1.1 Defining strategies for improving the active participation of the members.

Activity 4.1.2 Assigning senior researchers as mentors to ECIs and organising process of assigning “Fellow of ENFAIRAM” status awarded to the most active ECIs.

Activity 4.1.3 Expanding the initial Action Network taking into account the balance in terms of stakeholders, gender and level of experience.

Task 4.2 Coordination and monitoring the Action

Activity 4.2.1 Organisation of meetings/conferences/training schools: defining topics, agendas, involvement of the Action members in the meetings and training schools, defining modalities, which will enhance knowledge transfer and support goals of the Action, coordination with local organisers.

Activity 4.2.2 Management of STSMs and ITC Conference Grants: defining the criteria to rank the applications for STSMs and ITC Conference Grants, to issue the calls and to propose to the MC the applications to be funded.

Activity 4.2.3 Coordination of quality assurance process for the Action outputs and deliverables.

Activity 4.2.4 Monitoring the participation of the Action Members: fostering balance in terms of stakeholders, gender and level of experience, encourage involvement of ITCs and ECIs, encourage collaboration and joint funding applications, as well as submission of joint research articles.

Task 4.3 Dissemination

Activity 4.3.1 Creation a mailing list of national and European stakeholders, who may be interested in the Action.

Activity 4.3.2 Building annual communication plans, which should be approved by the MC.

Activity 4.3.3 Designing and moderating the Action website and its content.

Activity 4.3.4 Designing and disseminating the Action news via various channels (leaflets, newsletters, Twitter, news etc.).

Activity 4.3.5 Coordinating the translation of the most important Action deliverables into other European languages.

New initiatives relevant to research assessment reform have emerged, offering contemporary frameworks and infrastructures that were not yet mature or widely recognised in 2021. The Global Research Assessment Platform Open Science project (GraspOS, https://graspos.eu/) is a European initiative developing an open, federated infrastructure to support next-generation research assessment aligned with open science principles (Xenou et al. 2025). GraspOS aims to provide data, tools, services and guidance that enable broader and more responsible evaluation practices beyond traditional metrics. Notably, work under GraspOS includes frameworks for comprehensive researcher profiles and landscape analyses of research assessment practices, reflecting policy reform priorities such as those articulated by the Coalition for Advancing Research Assessment (CoARA, https://coara.eu/) (Xenou et al. 2025).

The Future of Research Evaluation is a global initiative that stimulates global reform of research assessment by moving beyond narrow, metrics-driven evaluation towards more inclusive, transparent and context-sensitive approaches (de Rijcke et al. 2023). It seeks to broaden what counts as research quality and contribution, encourage collective action across stakeholders, share and scale good practices and support experimentation and learning, particularly in response to open science, interdisciplinarity and emerging technologies such as AI (de Rijcke et al. 2023).

The CoARA initiative is a coalition of research organisations, funders and assessment authorities committed to reforming research assessment. Building on the Agreement on Reforming Research Assessment, CoARA promotes more holistic and qualitative evaluation methods supported by the responsible use of quantitative indicators. It hosts working groups (e.g. on responsible metrics) and produces conceptual frameworks aimed at replacing proprietary, narrow metrics with open, transparent and community-governed systems (Xenou et al. 2025).

The social media landscape for academic exchange has also changed since 2021. Although Twitter (now X) has historically been a central platform for scholarly communication and dissemination, recent trends show a notable shift amongst researchers towards alternative platforms (Quelle et al. 2025). For example, one study using a unique dataset of engineering PhD holders shows that professional networking via LinkedIn is strongly shaped by sector and mobility, with higher uptake and larger networks amongst those in industry and those with international collaboration or migration experience, underscoring both the potential and limitations of social media data for understanding academic exchange (Baruffaldi et al. 2017). Similarly, research on scholarly posting practices indicates that the newer platform Bluesky now generates higher engagement (e.g. likes, reposts, replies) for research-related content than X, with many scientists reporting greater depth of academic discussion there (Quelle and Bovet 2025).

Complementary work has analysed the migration of academic users from Twitter to Bluesky, indicating substantive movement of scholarly networks and engagement patterns (Quelle et al. 2025). Such evidence suggests that assumptions made in 2021 about Twitter’s centrality to scientific networking may no longer hold and that Bluesky and similar platforms (e.g. Mastodon) should be factored into studies of scholarly communication and impact.

A substantive body of research has matured regarding the value and limitations of academic networking sites, such as ResearchGate, Academia.edu and other platforms, in the context of research assessment. For example, empirical studies have examined altmetric and bibliometric correlations on Academic Social Networking Tools (ASNTs), showing positive relationships between altmetric indicators on platforms like ResearchGate and traditional bibliometric measures, suggesting that online visibility can relate meaningfully to scholarly impact (Adriaanse et al. 2024).

Additionally, literature reviews of ResearchGate and Academia.edu highlight their roles as socio-technical systems supporting scholarly communication, reputation building and alternative indicators, while also calling for more nuanced research into their functions and implications for assessment (Manca 2018). These studies contribute to a stronger evidence base on how academic networking sites can complement traditional evaluation practices, an evolution from earlier discussions focused primarily on Twitter-based altmetrics (e.g. studies on tweeting and citations) (Adriaanse et al. 2024).