The conventional evaluation of the impact of scientific publications is carried out using metrics such as the Journal Impact Factor or the h-index. The representation of high or low impact using a simple indicator has been a controversial issue for some time and causes dissatisfaction among researchers. The use of metrics such as the Journal Impact Factor (JIF) for research evaluation is problematic because it is calculated on the basis of licensed data from Clarivate Web of Science and can hardly be reproduced or varied by the end user (Herb 2016, PLoS Medicine Editors 2006). However, in addition to such proprietary metrics (including Elsevier Scopus, Google Scholar, and ResearchGate), data is increasingly available under free licenses, including those from CrossRef Event Data, Wikidata, and Microsoft Academic Search (Das 2015, p. 67).

Within the ROSI project new assessments and visualizations of freely licensed conventional and alternative metrics (altmetrics) will be developed and their effect on the target groups research and science administration will be investigated. Not only metrics are undergoing a change, the evaluation of the influence of a publication by measuring its impact in the exclusively scientific field itself is also being called into question (Bornmann and Haunschild 2016). The project is oriented towards "Leiden Manifesto for Research Metrics" (Hicks et al. 2015), which criticizes the state-of-the-art of research evaluation and sets out ten principles for improving the current practice. The ROSI project therefore also addresses the heterogeneity of research products beyond the classical journal article (research data, research software, and blog postings etc.), the roles of researchers and contributors with regard to these different products, as well as different references depending on product type and role (e.g., reviews, re-use in the case of research data or software, or derived works).

Within the framework of the project, information should be collected from freely accessible sources such as those mentioned above on the basis of personal, document or organisation identifiers. This is usually done via interfaces explicitly provided for this purpose, e.g. the CrossRef Event Data API. The information collected in this way can be displayed at different aggregation levels and thus enable evaluations at the three levels named: person, document or organization. In addition to conventional publication types such as books or articles, the definition of the term document also includes other object types such as scientific software and research data, which will become increasingly important in the scientific discourse (Peters et al. 2016, Smith et al. 2016). For this purpose, a reference implementation for the community-based open source research information system VIVO will be created, which combines various metrics with data from open sources such as CrossRef Event Data API, or Wikidata (cf. Wilsdon et al. 2015).

The project will make use of qualitative methods. We will analyze the current use of indicators, evaluate the requirements and needs of the scientific community, and examine their repercussions. In order to determine the requirements of the target groups, interviews with scientists from different disciplines are conducted in a first step in order to evaluate different views on scientometric indicators. The knowledge gained is used for an initial draft of the reference implementation. In the further process, the perception of this reference implementation is evaluated in focus groups and the implementation is adapted. In these focus groups, the selection of individual indicators, the method of aggregation, and different ways of visualization are discussed. Diffenrent focus groups are addressed to achieve a domain-specific representation of the scientometric indicators. In this way, the handling of new object types such as software or research data can also be evaluated. The reference implementation is adapted to their needs through feedback from the target groups in an iterative process. Through free licensing of the initial data and the software created in the project and the detailed documentation of the results, the knowledge gained and also the implementation is going to be easily transferable to research-related platforms.

The objectives of the project are firstly the evaluation of the scientometric requirements and concerns of the target groups and secondly the implementation of a reference implementation of a toolset which reflects the results of the study and which enables a license-free, flexibly adaptable analysis of the output of researchers, contributors and organisations. This includes the user-friendly integration of bibliometrics into researcher profiles according to the needs of scientists.

The DFG-funded project Linked Open Citation Database (LOC-DB) of the Hochschule der Medien (HdM) Stuttgart and the Christian-Albrechts-Universität zu Kiel deals with the development of practical tools and processes in the field of linked data technologies for the participation of libraries in an open, distributed infrastructure for the indexing of citations.

The Initiative for Open Citations (I4OC) is an association of scientific publishers, scientists and various organisations in the field of "Openness". It tries to increase the availability of freely licensed data. Founding members include Datacite, OpenCitations, PLoS and Wikimedia.

At the Centre for Science and Technology Studies at Leiden University, the Science and Evaluation Studies research group analyses the impact of various research evaluation practices. One goal of the group is to develop a deeper empirical understanding of the effects of evaluation practices on academic knowledge production. With its research results, it aims to contribute to a responsible handling of research results and metrics.

The NISO Alternative Assessment Metrics (Altmetrics) Initiative tried to identify different perspectives and requirements for dealing with new metrics in the scientific field. For this purpose, the existing metrics were documented and examined with regard to data quality problems.

VIVO is a community-based open source software for displaying researcher profiles and research output on the Net. VIVO offers storage, editing, search, browsing and visualization of scientific activities. VIVO supports visualized evaluations and thus the presentation of scientific outputs, and relies on linked data, and open standards. The ontologies (especially ontologies C4O and CiTO for mapping bibliometric information) are used to describe scientific persons, projects, and publications. In addition, the integration of an Altmetrics widget is possible.

The ROSI project aims to bring together the findings of the projects described here in the field of scientometric indicators and to evaluate the implications for the people involved in research output in order to derive practices that take into account the interests of scientists. It will be shown how the free availability of publication and bibliometric data from different sources as well as methods (in the form of the VIVO software, its ontologies, and extensions) makes it possible to represent research impact in a very flexible and with a wide variety of configurations. On this basis, a broader and systematic comparison can be made with the requirements of the scientists themselves.

The work programme is divided into 5 work packages and will run for twenty-four month (Fig. 1). Work packages 1 to 3 mainly deal with quantitative methods: selection of data sources for scientometric information, data aggregation and integration, and development of reference implementation based on VIVO. In work package 4 qualitative procedures are carried out, whereby the use and the evaluation standards of scientometric indicators are considered and the effects on the scientific community are evaluated by questioning experts. In work package 5, a handbook on how to deal with this data will be prepared on the basis of the findings and the project results published.

Figure 1.  

PM1 and PM2: Person months of the first or second project member; Q: quarter.

The TIB is responsible for project management and public relations.

Work package 1: Data sources for scientometric information

Work package 2: Data aggregation and integration

Work package 3: Development of the reference implementation

Work package 4: Practical use of indicators, assessment standards and repercussions

Work package 5: Project documentation and preparation for reuse

The project will produce various research data. These are primarily the data from the individual interviews and the focus group surveys. Research data is published in accordance with the TIB Research Data Policy. Data is anonymized and stored in the most open, standardized formats possible and, as far as possible, also published.

The results of this project can be widely re-used and applied for different purposes.

First, the VIVO-based reference implementation is available for free reuse. The reference implementation can be re-used in the context of research information systems, research profile services or discovery systems. Another technical exploitation of the project results is the re-use of the ontologies created or expanded in these or similar services.

The project handbook (see WP 5) on the handling of scientometric data will document the knowledge gained and processed for subsequent use and provides a guideline for the handling of scientometric data and indicators. Among other things, the TIB itself offers a comprehensive range of advice for publishing researchers on Open Access, rights and opportunities for authors, into which the handouts will also flow.

Another core element to reuse the scientific data of this project is the publication in peer-reviewed journals. Knowledge transfer is also realized via project communication (public relations) and multipliers. These are going to be selected from the expert surveys and the VIVO community. The TIB is an active member of the VIVO community and organizes VIVO workshops and participates in conferences and in technical communication on VIVO in general. Other starting points include events such as the annual conference of research advisers in Germany, at which the TIB had already held a heavily attended workshop in 2015, and the workshop as part of the LOC DB project (see invitation in the Letter of Intent of the Mannheim University Library).

The DINI AG FIS is also an important hub for the introduction of research information systems in Germany.

The transfer of scientific results also opens up perspectives for follow-up projects.

This project is funded by the German Federal Ministry of Education and Research (BMBF) for 24 months.

Quantitative Wissenschaftsforschung

Referenzimplementierung für offene szientometrische Indikatoren

Technische Informationsbibliothek (TIB) – German National Library of Science and Technology

The authors declare that they have no conflict of interest.