Building on previous research on detecting and annotating the plant organs from digitised herbarium scans (Younis et al. 2018, Younis et al. 2020b), we present now an extended approach involving an advanced segmentation technique to facilitate detailed quantitative analysis of morphological traits. The aim of the segmentation method is to subdivide images into objects or regions. Fundamentally, there are two types of segmentation: semantic and instance segmentation (Long et al. 2015, He et al. 2017). Semantic segmentation labels pixels with a class without distinguishing them into individual objects, whereas the instance segmentation labels each pixel and differentiates the individual objects of each class. Instance segmentation, used in the present context, enables the model to detect the plant organs and segment each organ individually. This is particularly beneficial for distinguishing closely-positioned or overlapping specimen organs, such as leaves, stems, fruits, seeds and flowers. Within the framework of this study, Mask R-CNN and YOLO11 were chosen as instance segmentation models. These models are based, as further explained below, on different architecture approaches: Mask R-CNN is a two-stage model, while YOLO11 is single stage. In the initial stage of the Mask R-CCN classification, it identifies regions of interest in an image and, in the second stage, it utilises local features around these proposed regions to determine contained objects. On the other hand, single-stage detectors, like YOLO, employ a fully convolutional approach to gridded regions of an image for the simultaneous prediction of bounding boxes and segmentation in a single pass (Redmon et al. 2016).

As in the aforementioned previous studies (Younis et al. 2020b), the dataset used for training plant organ segmentation consists of 652 images of herbarium scans from the Muséum national d’Histoire naturelle's (MNHN) vascular plant collection (Le Bras et al. 2017, Younis et al. 2020a). These images are openly accessible through the Global Biodiversity Information Facility (GBIF) portal (MNHN and Chagnoux 2020).

Out of 652 images, 497 images were used for training and 155 images were used for testing. As shown in Table 1, the images were annotated into six different categories: leaf, stem, flower, fruit, root and seed. These images were previously annotated for object detection in plant organs. The training subset has 15486 annotations and the testing subset has 4137 annotations.

For the instance segmentation task, the annotations from the previous study were further refined to generate detailed pixel-level masks for each plant organ using the Segment Anything Model (SAM, Kirillov et al. (2023)). The files containing the annotations were parsed to extract the bounding box information, which was then passed to the SAM model along with the corresponding original images.

The SAM model iteratively segmented the plant organs for each bounding box and the resulting segmentations were combined for each image. Next, the masks were used to prepare instance segmentation annotations by uniquely identifying and labelling each organ type across all images (Fig. 1). This approach facilitates the reuse of the existing dataset and annotations were produced suitable for MASK R-CNN training. Additionally, the dataset was reformatted to comply with YOLO11's input specifications and allows precise segmentation of individual features in the scans and supports more detailed morphological analysis.

The model was further independently trained to detect the scale in the digitised herbarium sheets and to calculate the surface area of plant organs in the digital herbarium sheet. The training dataset consists of 163 annotated images. Amongst these, 32 images are sourced from the Senckenberg herbarium dataset (Younis et al. 2020a), 24 images from the vascular plants collection at the Herbarium of the Muséum national d'Histoire naturelle, Paris (MNHN and Chagnoux 2020) and an additional 107 images from the GBIF to increase variability in the dataset for the training for scale detection. Of the total, 124 images are used for training and 39 for testing. A dataset containing the subset of images included from the Senckenberg Herbarium is available (Rajendran et al. 2025).

The Mask R-CNN model (He et al. 2018) is trained using PyTorch (Paszke et al. 2019b) to perform instance segmentation on digitised herbarium scans. Out of 652 images, 497 annotated images were used for training and 155 images were used for testing. Data transformations, such as random horizontal and vertical flips, colour jitter, scaling and rotation, were applied during training to introduce variability to improve the model’s generalisation.

The Mask R-CNN model used a ResNet-50 backbone to balance feature extraction quality and computational efficiency for the segmentation of high-resolution herbarium images. Training strategies, such as early stopping, were implemented by comparing mean Average Precision (mAP) of the current epoch and previous epoch, involving a patience threshold to avoid overfitting (Ying 2019). Additionally, the model was customised with a modified Region Proposal Network (RPN) anchor generator and adjusted Region Of Interest (ROI) heads to improve performance on the dataset (He et al. 2017).

The model performance was evaluated with metrics, such as precision and recall, for object detection and segmentation. To test the model generalisation, inference was conducted on the images from the Frankfurt Senckenberg Herbarium dataset (Younis et al. 2020a).

The YOLO 11x-seg (Jocher et al. 2023) segmentation model was trained with the same dataset used for MASK R-CNN training (Jocher et al. 2023). The default input image size for the YOLO model training is 640 × 640 pixels, but in this study, a resolution of 1064 × 1064 pixels was used, since it was the largest resolution that could fit into the GPU memory during training. The training strategies, such as early stopping of training, were applied in the same way as in the previous case.

The measurement of surface area of plant organs is an essential component of morphometric analysis in biodiversity studies (Hodač et al. 2024). To calculate the surface area of a plant organ in the herbarium sheet, the scales need to be detected and the surface area in pixels needs to be calculated and converted into absolute values, which are units of measurement such as square millimetres or square centimetres. This is accomplished through a combination of deep learning techniques for scale detection and Optical Character Recognition (OCR) for extraction of numerical values.

The Mask R-CNN and YOLO11 models were again employed for the detection of scales in the herbarium images. The surface area of plant organs was subsequently calculated. We used 167 images, of them 124 images for training and 43 for testing.

Once the scale was detected by the model, Tesseract OCR was employed to extract the numerical values from the scale present in the images (Smith 2007).

Tesseract (Smith 2007) is a widely used optical character recognition tool to identify text in the images. Image preprocessing techniques, such as rotation of scales to correct orientation, greyscale conversion and bilateral filtering, are applied to enhance the text visibility. Following the preprocessing, the Tesseract OCR was utilised on the scales to extract the numerical values from the scale. The digits with a confidence level above 75% and in the range of numbers from 0 to 10 are filtered and sorted to identify the consecutive sequences. The pixel distances between adjacent digits are calculated and averaged with the detected numbers to determine the number of pixels corresponding to 1 cm. In cases where the consecutive digits are not detected, the closest pair of digits is used to estimate the pixel distance. The character recognition accuracy is computed by comparing the detected digits with a ground truth reference. Once the pixel distances between consecutive digits on the scale have been calculated, the next step is to establish the conversion factor that relates the relative pixel measurements to absolute measurement in centimetres. The pixel-to-centimetre conversion factor is computed as follows:

\documentclass[12pt]{standalone} \usepackage{varwidth} \usepackage[utf8x]{inputenc} \usepackage[T1]{fontenc} \usepackage{lmodern} \usepackage{amsmath, amssymb, graphics, setspace} \newcommand{\mathsym}[1]{{}} \newcommand{\unicode}[1]{{}} \newcounter{mathematicapage} \begin{document} \begin{varwidth}{50in} \begin{equation*} \text{one cm in pixels} = \dfrac{\text{Pixel distance between digits A and B}}{\text{Difference between digit B and digit A}} \end{equation*} \end{varwidth} \end{document}

where

Pixel Distance Between Digit A and Digit B is the pixel distance between the detected digits;

With the pixel-to-centimetre conversion factor established, the next step is to calculate the surface area of the detected plant organs. The plant organ surface area is calculated as follows:\documentclass[12pt]{standalone} \usepackage{varwidth} \usepackage[utf8x]{inputenc} \usepackage[T1]{fontenc} \usepackage{lmodern} \usepackage{amsmath, amssymb, graphics, setspace} \newcommand{\mathsym}[1]{{}} \newcommand{\unicode}[1]{{}} \newcounter{mathematicapage} \begin{document} \begin{varwidth}{50in} \begin{equation*} \end{equation*} \end{varwidth} \end{document}

\documentclass[12pt]{standalone} \usepackage{varwidth} \usepackage[utf8x]{inputenc} \usepackage[T1]{fontenc} \usepackage{lmodern} \usepackage{amsmath, amssymb, graphics, setspace} \newcommand{\mathsym}[1]{{}} \newcommand{\unicode}[1]{{}} \newcounter{mathematicapage} \begin{document} \begin{varwidth}{50in} \begin{equation*} \text{Plant organ surface area } (\text{cm}^2) = \frac{\text{The sum of pixels in the segmented organs}}{(\text{Calculated pixels present in 1cm})^2} \end{equation*} \end{varwidth} \end{document}

where

The sum of pixels in the segmented organs refers to the total number of pixels in the segmented organ regions;

This additional functionality of calculating surface area supports detailed morphometric analysis of herbarium specimens.

Both models for plant organ segmentation and surface area calculation are then hosted as a single Machine Annotation Service (MAS) to streamline the process of extracting and analysing the herbarium data within the DiSSCo platform. Fig. 2 further illustrates the process flow and communication between the DiSSCo Digital Specimen architecture and the plant organ segmentation MAS.

In DSArch, a service request for a MAS is requested through the user front-end DiSSCover (https://sandbox.dissco.tech) on the digital media. This request adds a message in DiSSCover’s Message Broker, which triggers scheduling of a MAS. The corresponding MLaaS APIs are hosted via Uvicorn (Trylesinski and Christie 2019), a high-performance asynchronous python web server designed for efficient handling of API requests on a remote virtual machine (VM) at Senckenberg. Within the scope of our specific use case, the API receives image URLs as the input from the Message Broker through the HTTP POST requests. The API then adds the message to the processing queue, which forwards the message to a web socket service, hosted on a server for performing plant organ segmentation.

Subsequently, the model performs plant organ segmentation and scale detection on the image and sends the output information comprising bounding box coordinates, class labels, confidence scores and area in pixels. Both the pixel-to-centimetre conversion ratio and an area calculation in cm² are returned to the Uvicorn API, which relays the results to the DiSSCo infrastructure. These processed data are then structured into an annotation event complying to DiSSCo's open Digital Specimen (openDS, Addink and Hardisty (2020)) specification. The message is published back to DiSSCo’s Core architecture by placing it on the Message Broker, making it in this way available for downstream applications.

Fig. 3 provides a detailed visualisation of the annotations which allows users to explore the extracted information by hovering over specific segments. This includes details, such as the type of organ, confidence score, segmented polygon coordinates, area in pixels, pixel-to-centimetre conversion ratios and calculated areas in square centimetres for each segmented region. Fig. 3 demonstrates the integration of MAS within the DiSSCover platform highlighting its ability to generate enriched and standardised annotations in compliance with openDS .

Mask R-CNN was employed on 203 herbarium images from the Senckenberg collection (Younis et al. 2020a). Fig. 4 shows an example of object detection inference performed by Mask RCNN on a herbarium sheet.

When performing the inference on the Senckenberg images, the most recognised organs were leaves, stems and flowers. The identified regions of interest were then passed on to the segmentation process. In the segmentation process, the model generates segmentation masks for each identified organ and provides pixel-level masks as shown in Fig. 4.

Table 2: Statistics of the MASK R-CNN inference on 203 images. The original annotated herbarium sheet has the total plant organ count of 6693 where only 4022 organs were detected with a confidence level of 50%. The model did not infer any root or seed due to the strong bias in the training data.

Fig. 5 represents the prediction recall curve of all plant organ detections and segmentations at an intersection of unit (IoU) thresholds of 0.5. The precision for the detection of leaves and stems is better than for flowers and fruits. The model effectively fails to predict roots and seeds.

Overall, the results suggest that the Mask R-CNN model moderately performs in identifying and segmenting the plant organs, achieving an average mean Average Precision (mAP) score of 0.211 for detection and 0.215 for segmentation. However, further addition of new images in the dataset could improve its performance of organ detection and segmentation.

The YOLO11 model was used with the same dataset of 203 herbarium images which was used for inference with Mask R-CNN. The model identified the distinct plant organs, such as leaves, stems, flowers, roots and seeds. In accordance with a similar study (Sapkota et al. 2024), we found that the YOLO11 model performs better than Mask R-CNN; YOLO11 was able to identify all the organs as shown in Fig. 6.

Table 3 below shows the statistics of YOLO11 inference on 203 images. The model was able to detect and segment all plant organs and, compared to Mask R-CNN, YOLO11 prediction count is higher.

Fig. 7 represents the prediction recall curve of all detected and segmented plant organs at IoU thresholds of 0.5. The YOLO11 model demonstrates improved performance compared to Mask R-CNN, achieving an average mean Average Precision (mAP) score of 0.373 for detection and 0.353 for segmentation.

In addition to the plant organ detection, the YOLO11 model integrates a scale detection component that successfully identifies the scales in 98% of the Senckenberg images. Fig. 8a shows an example of the scale detection.

Once the scale is detected, the model further uses the pixel-to-centimetre ratio to calculate the surface area for the plant organs in absolute values (cm²). However, variations in scale standards across different organisations necessitate further training to improve scale detection.

Fig. 8b shows detected plant organs and scale with bounding boxes, class labels and confidence scores. Additionally, Fig. 8b illustrates the segmentation mask alongside the organ surface area calculations.

Suppl. material 1 provides a qualitative impression of the applicability of the method to other types of digitised specimen, in which scale detection was applied to images from Senckenberg's marine collections. The system successfully detected scale markers, recognised textual characters and identified morphological structures resembling stems and flowers (Suppl. material 1). These findings demonstrate the potential for extending our approach to applications beyond herbarium specimen analysis. However, successful detection and extraction of morphological traits requires retraining on the corresponding taxonomic groups.

Table 4 represents the statistics of scale detection, text recognition and character accuracy with Mask R-CNN and YOLO11 on 203 images. Although the pipeline of text recognition is the same for both models, there is difference in optical character accuracy due to the number of detected digits. The resulting bounding coordinates have small pixel variation due to the internal architecture of the models. The YOLO11 uses the single shot object detection where the images are passed one time. Mask R-CNN, on the other hand, uses two-stage object detection where the first pass proposes the region of interest and the second pass refines this prediction. The architecture difference impacts the resulting bounding box coordinates and results in a small offset pixel difference on pixel calculation between the models. This slight variation in bounding box calculation influences the OCR performance, as even minimal pixel shifts can affect character segmentation and thereby following recognition accuracy.

The output data then sent to DSArch includes the bounding box of detected organs, class, confidence score, area in pixels, pixel length of one centimetre, area in cm² and polygon data.