Natural Science Collections (NSCs) hold biological, geological and palaeobiological specimens that are containers of rich data (e.g. geochemical, taxonomic, molecular). They are viable references to past and present biodiversity and geodiversity from across the globe and contain unique specimens that can no longer be collected in the field or are extinct. However, currently, European NSCs are scattered geographically and there is insufficient awareness of the extent of their content due to limited metadata access and a lack of harmonisation in data standards which reduces interoperability and meaningful interpretation. This impedes the collective usage of such valuable data in science, policy and technology for helping to understand and mitigate large scale societal and environmental challenges (e.g. biodiversity loss caused by climate change and habitat destruction). The Distributed System of Scientific Collections Research Infrastructure (DiSSCo) aims to address this impediment by digitally unifying all European collections via a massive digitisation effort under common curation, policies and data standards to ensure that all data are Findable, Accessible, Interoperable and Reusable (FAIR) (www.dissco.eu). This is a gradual process and a service in the form of a Data Dashboard is needed to facilitate this large scale effort. A data dashboard is an information management tool that visually tracks, analyses and displays key performance indicators (KPI), metrics and key data points, for example, to monitor health and development of an organisation and/or specific processes. They often aggregate and reduce voluminous or complex data into a series of summary statistics, sometimes in real time and in a visually appealing way.

This paper focuses on the design and construction of a pilot Collections Digitisation Dashboard (CDD) developed under the SYNTHESYS+ project (a DiSSCo-linked project, funded by the Horizon 2020 Framework Programme of the European Union, www.synthesys.info). It provides an abridged version of the formal project deliverable report (D2.2 'Joint dashboard of collections assessment tools'). The pilot CDD aims to provide a dynamic window for stakeholders to discover the contents, coverage and strength of NSCs held in European institutions, both digitised and undigitised. In addition, it will help track current digitisation progress, to aid in the prioritisation of digitisation and to support high-level decision-making. The system needs the capability to be embedded into other DiSSCo e-services, a major one being the European Loans and Visits System (ELViS, https://elvis.dissco.eu/welcome). ELViS will be a one-stop-shop that will enable global users (e.g. Scientific researchers, end-user communities etc.) to request loans, virtual access (digitisation on demand) and physical access to collections held in DiSSCo institutions. The CDD will provide support by providing access to information about the physical and digital holdings of NSCs in order to support users in planning visits and arranging loans of material (Islam et al. 2021). In addition, it will also track usage of collections accessed via ELViS (i.e. number of visits and loans, digitisation status etc.).

The work described herein builds upon a previous 'proof of concept' design, which was developed as part of the DiSSCo-linked project ICEDIG (Innovation and consolidation for large scale digitisation of natural heritage) (www.icedig.eu), funded by the Horizon 2020 Framework Programme of the European Union. The ICEDIG project focused on developing the blueprint of DiSSCo. Moreover, work under ICEDIG demonstrated the potential of a collections dashboard, through the development of a set of user stories, the evaluation of technical solutions and the creation of a Collections Classification Scheme. The Collections Classification Scheme uses high-level categorisation to describe physical natural science collections and was developed via a gap analysis of existing data standards (van Egmond et al. 2019).

The aim of the SYNTHESYS+ project is to produce an accessible and integrated European resource for users of natural science collections and facilities. Moreover, it is focused on providing the technical foundation for DiSSCo. The work towards the design and construction of the CDD, described herein, was conducted over 14 months (July 2019 - September 2020) and was led by the Consortium of European Taxonomic Facilities (CETAF, https://cetaf.org) with the support and collaborative effort of nine official partner institutions: London Natural History Museum (NHM), Museum für Naturkunde Berlin (MfN), Hungarian Natural History Museum - Budapest (HNHM), National Museum of Natural History - Paris (MNHN), National Museum of Natural Sciences - Madrid (MNCN), Naturalis Biodiversity Center - Leiden (Naturalis), Royal Belgian Institute of Natural Sciences – Brussels (RBINS), University of Copenhagen (UCPH) and Meise Botanic Garden (MBG). There were also valuable contributions from two other SYNTHESYS+ partners who were not officially involved in the task: Natural History Museum Wien - Vienna (NHMW) and the Royal Museum of Central Africa - Tervuren (RMCA).

In this section, each of the main dimensions of the Collections Classification Scheme are presented and described in terms of purpose, categories they contain, hierarchical level and enhancements that have been made.

The first step towards acquiring data for the CDD was to define the level of data granularity and aggregation required. This refers to the extent to which the institutional collections should be quantified according to the different dimensions of the classification scheme (i.e. Taxonomy, Geographic Region, Storage and Chronostratigraphy) and how the dimensions might need to be combined to support users’ needs. It required the consideration of data utility against the effort needed to generate and maintain data by institutions. The decision had to consider a balance between two extremes:

• Collection objects quantified by the four dimensions independently: This is the simplest and requires least effort to contribute the data to the CDD, but it provides low utility of data because questions can only be answered within the individual classification dimensions. For example, users can see how many objects are from South America or how many objects are fungi, but not how many fungi are from South America.
• Collection objects quantified using a combination of all the dimensions in one single breakdown: This would allow users to answer any question related to any combination of the classification schemes used, thus generating a high level of data utility (e.g. how many fluid preseserved fungi specimens are from South America). However, in this case, the amount of effort required would not be feasible for many (if any) institutes, especially within the timeframe of the task, as they would have had to complete up to 50,000 object counts in addition to digitisation level assessments and confidence indicators.

In order to find a middle ground between these two extremes of granularity and aggregation, user stories were analysed to see what dimension combinations were essential. From this analysis, the only combination of dimension that appeared to be useful and achievable was that of the ‘Geographic Region’ and ‘Taxonomy’ dimensions, since ‘Geographic Region’ has a relatively small number of categories compared to the other dimensions. However, it was also agreed that, while collecting object counts for each combination of ‘Geographic Region’ and ‘Taxonomy’ should be feasible, asking for MIDS level assessments in addition to those would be an unrealistic expectation.

Although no other dimensions were combined in their entirety, the highest level of the ‘Taxonomy’ hierarchy (‘Discipline’) was incorporated into each of the dimension breakdowns. The dimension ‘Discipline’ consists of just nine disciplines (‘Zoology invertebrates’, ‘Botany’, ‘Geology’ etc.), so did not greatly increase the amount of data that needed to be contributed. However, it provides a top layer of classification that is common across all breakdowns, which is important for aggregation within the dashboard and for basic interoperability with collections data in other platforms, such as the CETAF Registry of Collections and GBIF Collections Catalogue. A summary of the four breakdown schemes is shown in Table 6.

Two types of metrics were captured for each breakdown of collections: a count or estimate of the number of physical objects and a measure of the completeness of digital records representing those objects.

The data model underlying the dashboard was designed with close reference to the data standard and model under development by the TDWG Collection Descriptions Data Standard Task Group (Latima Core) (Woodburn et al. 2022). This is intended to become the global standard for Natural Science collection descriptions data and so its early adoption for the CDD promotes future interoperability of CDD data with other collection descriptions datasets, such as the CETAF Registry of Collections, ELViS and GBIF Collections Catalogue. The TDWG data model is also being designed to provide for the structured, quantitative collection data that support the dynamic reporting and visualisation offered by the CDD. A simplified representation of the TDWG data model is shown in Fig. 1.

Figure 1.  

A simplified conceptual view of the TDWG Collections Description data model.

In the CDD back-end database, the collection classification scheme is represented as dimensions in the TDWG Collection Description Standard data model. The construct is used to differentiate between the multiple breakdowns of each institutional collection according to the different dimensions. This prevents the same object from being counted more than once in any of the dashboard visualisations.

Representing the NSCs as object groups attached to an institution (rather than a fixed hierarchy of institutions, collections and subcollections) means that metrics can be dynamically aggregated and visualised across institutions and also within (and to a degree, across) dimensional hierarchies like Taxonomy and Geographic region.

For the purposes of the pilot CDD, the data model was implemented as a MySQL relational database and the complete data model is shown in Fig. 2.

Figure 2.  

CDD relational data model.

For the pilot dashboard, a Google Sheet survey was considered to be the best tool to test the feasibility for the nine partners to collate data using the Collection Classification Scheme. Data were extracted from the completed surveys through a semi-automated process, involving downloading the individual Google sheets as Excel spreadsheets and using VBA code to generate the SQL queries needed to insert the data into the database in the correct format and structure. The relatively small number of pilot institutions made this a more appropriate method within the timeframe of the task. However, if the pilot framework is scaled up to a much larger number of institutions or more regular updates of the data, then methods for further automation should be explored. Options for this might include more extensive, robust scripting (using Python, for example) to extract and validate data in the survey sheets and directly interact with the database to insert and update data. Alternatively, an ETL (Extract, Transform and Load) tool, such as Pentaho Data Integration, could be employed to achieve similar ends via a more automated workflow.

Data validation was carried out both in the source spreadsheets, to ensure there would be no data integrity issues in loading into the database and after each load to ensure that it had been successfully executed. While many common data quality issues were avoided by adding data validation to cells in the survey sheets, some were still encountered in the returned surveys. The most common issue was missing or partial data for object counts and MIDS assessments for one or more dimensions, reflecting the challenge for institutions in generating and collating this information within the allotted timeframe. Wherever possible, these gaps were handled and the data loaded, but in cases where the integrity of the database or the dashboard might be compromised, then some data were excluded until the issues could be resolved with the contributing institution.

The design of the CDD was completed in the following phases:

The trade-off between gathering structured high quality data about an NSC, versus the institutional effort involved in provisioning the source data, has long been the primary barrier to delivering an overview of global natural science collections. Additional factors, including the absence of associated data standards, the fact that the data are often held by multiple individuals within an organisation (if held at all), the need to provide regular updates of the data and the absence of any technical agreements on how to provide the data, all compound to create a complex and potentially insurmountable challenge for a goal that, from the outset, seemed to be a simple problem.

Several regional or thematic efforts have been established to solve this problem. Index Herbariorum (IH) is the directory of information on the world’s herbaria (addresses, contacts, specialities, size etc.). It is a well-managed resource and highly regarded as a tool by the botanical community. No full equivalent exists globally for other natural history collections, although national/regional infrastructures, such as the Atlas of living Australia (ALA) collections pages, the iDigBio US Collections List and the CETAF Institution profiles serve similar roles. GBIF has recently integrated the Global Registry of Scientific Collections (GRSciColl) into its registry as a framework that can be extended with richer information curated by collections communities.

The GBIF Online international consultation (April 2022) examined the issues associated with the use, information, technology and governance of a global collections catalogue (Hobern et al. 2020). Recommendations from this consultation included that each institution should have primary responsibility and control for information on its collections. However, it may be appropriate to delegate full or partial responsibility for ensuring quality and standardisation of collections descriptions to thematic, regional or national communities that have qualified data curators and/or automated quality checks in place. In this regard, communities, such as Index Herbariorum, ALA , iDigBio and CETAF, play an important role supporting collections and promoting standards-based practices.

Within a European context, CETAF is presently in the final stages of redeveloping the CETAF Institution Profiles as the CETAF Registry of Collections. This has the potential to provide a more automated approach to the provision of data to the CDD and increase the reliability and accuracy of the sourced information by engaging directly with the CETAF community. The intention is that this registry will be a data entry and management interface for the DiSSCo European Collection Objects Index (ECOI), an internal service for the cataloguing and curation of digital objects by the DiSSCo hub. More specifically, the CDD will hold high-level information about European NSC institutions, including different institutional features apart from collections, such as facilities and laboratories. Navigation is both ‘human readable’ via a user-friendly interface and ‘machine readable’ to facilitate data exchange and harvesting of data. In this regard, the DiSSCo ECOI has the potential to become a stable source for information for European collections feeding into the GBIF Collections Catalogue.

TDWG Collection Descriptions Data Standard

The pilot CDD data model has been designed in alignment with the development of the TDWG Collection Description data standard and model (Latima core) (Woodburn et al. 2022). This early adoption means that CDD data should be interoperable with other key platforms that are intending to adopt the standard, including the CETAF Registry of Collections and DiSSCo ECOI.

As the CDD and Collection Classification Scheme was being developed in parallel with the TDWG standard and to some extent moving at different speeds, there was a degree of divergence at the point where the CDD database needed to be finalised. Work will continue to make sure that outstanding CDD requirements are incorporated into the CD standard and that the CDD database continues to conform to the standard as it develops.

Common hierarchies and vocabularies

While the use of a common data standard provides a base layer of technical interoperability between different collections datasets, data are made truly comparable by the use of common hierarchies and vocabularies (such as the classification schemes used for the CDD). Greater harmonisation of these across initiatives is a longer term challenge, especially for those already well established, but there is potential for incremental gains in this area. For example, it has been agreed that the nine categories specified in the ‘Discipline’ layer of the CDD ‘Taxonomy’ hierarchy will be harmonised across the CDD, CETAF Registry of Collections and DiSSCo ECOI (including the European Loans and Visits System (ELViS), providing a common top layer of collections breakdown across these platforms.

Global persistent identifiers

A core requirement for interoperability between datasets is the use of globally unique persistent identifiers (PIDs) to identify collections and their subsets. This is a framework that needs to be addressed at the global community level, rather than duplicated by individual platforms and conversations are progressing on this topic within and between DiSSCo, CETAF, GBIF and other contributors. For the CDD database, a temporary solution using GUIDs (Globally Unique Identifiers) has been used with the intention of adopting a wider community PIDS framework for collections when it is available.

There was significant effort towards trying to develop a comprehensive and representative Collection Classification Scheme for Natural Science Collections. Nevertheless, time constraints on the input from some disciplines mean that further alterations/expansion may be required in certain areas. For example, future work should include the development of a schema and identifiers for living collections.

After wider dissemination of the Collections Classification Scheme to the CETAF Earth Science Group, we received useful feedback about the classification of minerals and meteorites. Due to the late stage of receipt, this feedback could not be incorporated into the CDD, but we have included the recommendation that mineralogy, as an independent discipline, should not be classed within geology, since minerals have their own complex classifications and are often curated separately from geology collections. Suggested categories within the Discipline Mineralogy are ‘Minerals’ and ‘Gems’. The proposed storage type categories for ‘Mineralogy' are 'cut/polished gems’, ‘Powder in vials’, ‘Radioactive’, ‘Humidity controlled containers’ and ‘Asbestos form in Perspex boxes’. The renaming of the taxonomic categories for the discipline ‘Extraterrestrial’ was also recommended. The recommended new categories are as follows: 'Terrestrial finds/falls’, ‘Terrestrial Impacta’ and ‘Sample returns’. For geology, it was suggested that the category ‘Petrology’ would be better replaced by ‘rocks’; and loose sediment replaced by sediment. Along with other improvements in the collections classification scheme, the future development of an age classification for Anthropology collections was also a mentioned prospect.

Living collections (notably the outdoor and indoor botanical collections) could also be added into future developments of a DiSSCo Dashboard that should go then beyond the “Digitisation” scheme and model that the current CDD pilot proposes. The same applies to a certain extent to some collections hosted by zoos and aquaria. However, neither one of those collections fall under a global digitisation endeavour and the development should rather focus on establishing interoperability standards and sharable flows of information, whenever possible.

Some of the feedback and requirements received from partners during the agile build and design of the CDD could not be incorporated in the pilot version. This is due to limits on the data collected and MS power BI’s licensing model that controls the publishing and implementation mechanisms used for the CDD. The feedback includes such requests as embedding the institutional logos; the need to see the rate of progress in digitisation; the ability to export the underlying data for analyses; and the ability for institutions to embed, in their own website, a pre-filtered view of the overview page (e.g. pre-filtered to their institution). These requests should be considered in further work of enhancing the CDD, especially in helping to explore alternative software solutions to construct the dashboard. A solution that has more features and functions, especially with regard to configurability and fine-grained control of dashboard functionality may be needed to meet these requirements. Further work will be conducted under CETAF and the construction phase of DiSSCo (2024 - 2025).