Ahead of the curve: three approaches to mass digitisation of vials with a focus on label data capture

There has been little research on novel approaches to digitising liquid-preserved natural history specimens stored in jars or vials. This paper discusses and analyses three different prototypes for high-throughput digitisation using cheap, readily available components. This paper has been written for other digitisation teams or curators who want to trial or improve upon these new digitisation approaches in liquid preserved collections


Introduction
A large proportion of natural history collections are stored in jars or vials, for example a recent inventory of the Natural History Museum, London (NHM) collections estimates over 17 million specimens are stored in 1.8 million curatorial units (Wilson et al. 2018). The collection is currently being digitised and made available via the NHM Data Portal (Scott et al. 2019). The NHM collection spans over 400 years of collecting, leading to variation in the size, shape, sealing and labelling of jars and vials, all of which increase the complexity of developing an efficient digitisation solution. While there is currently no alternative to specimen handling if standardised specimen images are required (e.g. dorsal, lateral and ventral views), recent advances focussing on the label data associated with dry pinned insects have significantly increased the rate at which label information and overview images of the specimen can be captured .
Previously published workflows for imaging wet specimens and their labels have removed the specimens from their container to be laid out for imaging in a petri dish (DeWalt et al. 2018) or a 3D printed box (Mendez et al. 2018). The average time for these two approaches has been reported as 2.78 minutes per vial with a single operator (DeWalt et al. 2018) or 8 minutes per vial when using two operators (Mendez et al. 2018), equating to a rate of 2.78 -16 minutes per vial or alternatively 3.75 -21.6 vials per person hour, excluding data transcription. At these published rates the NHM insect spirit-preserved collection, estimated at 700,000 vials, would take between 26 to 150 person years to image and catalogue. Furthermore if the primary goal is to create an inventory of the collection and record the data associated with these vials, rather than images of the specimens themselves, this might be achieved more rapidly by imaging the vials while reducing the risk of damage to the specimens by minimising their handling.
Recent advances in multi-camera imaging, focussed on label data extraction from pinned insects ) enables the digitisation of over 1000 pinned specimens per person day. We propose that a similar multi-image, multi-angle, data-focused approach can be used to speed up imaging and data capture from uncatalogued wet collections. We present three tested prototype approaches: (1) multiple mirrors used to replicate the multiple angles captured by ALICE (Price et al. 2018) with one camera; (2) a commercially available rotating turntable; and (3) slit-scan imaging using a custom built rotating turntable. All three solutions are described and discussed in the context of collections digitisation with particular focus on benefits, limitations and further improvements.

MALICE: Mirror Angled Label Image Capture and Extraction
The MALICE setup is an iteration of the ALICE system described in . The aim of MALICE is to capture multiple angled images with a single overhead camera using mirrors that enable the reconstruction of a label using post-processing. This composite, reconstructed label makes data extraction (e.g. transcription and OCR) easier. mirror tilt to accomdate different vial sizes. Mirror tilt is achieved using a friction joint (Fig.  1e). Once an optimal mirror angle is chosen, the setup can be built out of any suitable material. For testing the MALICE base plate was attached to a Kaizer RS10 copy stand with re-usable adhesive putty (Blu Tack) during imaging. Images were taken using a Canon 5DSR and Canon 24-70mm lens, controlled with EOS Utilities v.3.

Workflow
Imaging: A vial is removed from the jar, placed in the centre of the MALICE setup, with a unique identifier (UID) label, encoded as a barcode, placed horizontally next to it. A single image is taken from above,capturing five views of the vial in the mirror images. The vial is then opened andthe barcode label is inserted. The vial is then re-sealed and placed back in the original jar.
The five vial reflections were cropped out of the image using Adobe Photoshop (CC 2019).

3.
Cropped images were manually deskewed, rotated and reflected using Photoshop creating a composite image.

VILE: Vial Imaging and Label Extraction
Successfully taking multiple images by rotating an object in front of a fixed camera is a straightforward approach, but requires some automation to make the approach scalable for

Hardware
Images were taken using a Nikon D5300 camera. Specimen vials were placed on a Stackshot rotary table with controller (https://www.cognisys-inc.com/store/rotary-table.html) (Fig. 2) Both the camera and rotary table were controlled using Helicon Remote v.3.9.1.

Workflow
Imaging: A vial is removed from the jar, placed in the centre of the rotating stage, with a UID label, encoded as a barcode placed vertically on a stationary stage within the imaging frame. Images are taken at six different angles corresponding to 0, 72, 144, 216, 288 and 360 degrees. The vial is then opened, the UID label inserted, the vial re-sealed and placed back in the original jar. The arbitrary degree between the images reflects a mistaken configuration of the stackshot turntable resulting in the duplication of the initial view with that of 360 degrees, which was subsequently deleted. Future imaging should utilise 0, 60, 120, 180, 240, 300 degrees to maximise label views while removing duplication.
Processing: Image processing followed these steps:

ReVILE: Revolving Vial Imaging and Label Extraction
The technique applied in the ReVILE setup is based on the 19th century concept of slitscanning and primarily used today for panoramic photography. This concept involves a slit moving across the photographic medium (film or digital sensor), thereby exposing only a small section of the photographic medium to light at any given time. The implication of the temporal element of the slit scan means that a slit-scan image is composed of several snapshots in time. By moving the subject of the photograph instead of the slit, the process is inverted, reproducing a 3D object on a 2D plane. The imaging concept of ReVILE is no different from VILE in that an object is rotated in front of a stationary camera. However, generating an image by extracting vertical lines from the camera video and aligning these to create a composite image requires a more precise camera-object alignment and a rotation speed that is determined by the diameter of the object.

Hardware
Camera, turntable, controller and lighting were built into a formex casing. The setup presented uses a Canon 5DsR camera with a 100mm Canon macro lens for capturing video and still images. The turntable is custom-built using 2mm aluminum, and polystyrene sheets. Movement of the turntable is provided by a 5v stepper motor (28BYJ-48 5V) controlled by a drive chip (ULN2003) and an Arduino UNO. Lighting was provided by 3 COB 48-SMD LED panels for front lighting and a Neewer 40004082 light panel for backlighting ( Fig. 3a-e). To ensure camera, turntable and lightshield (Fig. 3f) alignment as well as XYZ adjustment the camera, the turntable and light shield were mounted on a Small Rig rail system.

Workflow
Imaging: A vial is placed in the centre of the turntable and filmed over a 720 degree rotation. Rotating the vial multiple times ensures that all possible angles are covered. The setup is calibrated by taking a single frame of a scale at the centre of the turntable and using image software to estimate the width (in pixels) of a single millimeter. The number of frames n required can then be calculated as such: Where d is the vial diameter (in mm), l is the focal length (in mm), x is the size of the frame (in pixels) across the x axis of the vial (i.e. width if landscape, height if portrait), s is the size of the sensor (in mm) across that same x dimension, and w is the width of 1mm in pixels (obtained during the calibration step).
The duration of a single rotation can be calculated by dividing this figure by the stream's framerate: This figure is largely a helpful estimate and high levels of accuracy are not necessary; it is preferable to err on the side of longer rotations to ensure that detail is not missed.
In the example dataset, videos were captured at approximately 60 frames per second, at a focal length of 100mm and a resolution of 1280x720 pixels (in portrait orientation). The size of the sensor on the Canon 5DSR is 36x24mm. The width of 1mm on the calibration frame was approximately 21 pixels.
For a 10mm vial: and For a 20mm vial: and Processing: ReVILE is capable of producing 3 separate image outputs outlined in the results section. The primary output is the composite rolled out image. The workflow of the f 1 rollout photography output is outlined below and the potential for the other two outputs -a composite image of the vial at different angles (equivalent to the VILE output) and an interactive digital-only output that allows for user rotation of the imaged vial -are examined in the discussion.
Custom software was developed to control the camera and turntable as well as process the camera's outputs to produce the rolled out image of the object. This software is available at https://github.com/NaturalHistoryMuseum/revile.
Once the object has been securely positioned on the turntable in front of the camera, the software follows the following steps to produce the rolled out image of the object: 1.
Initialise the video capture process -this is done first as it takes a couple of seconds to begin capturing, but step 2 should generally begin first; 2.
Start rotating the turntable for the specified number of seconds; 3.
Iterate through the stream or video frames, extracting the middle column or row (depending on camera orientation) of pixels from each frame; 4.
Concatenate the extracted lines of pixels from each frame together to build the image; 5.
Crop the image so that it shows only one complete rotation, using feature matching to locate similar columns of pixels; and 7.
Write two JPEG images to disk: the complete, "raw" output, and the cropped version.
Using the hardware setup pictured in Fig. 3, the middle row of pixels is extracted from each frame as the camera is positioned on its side. The concatenated image is rotated through 270 degrees to turn it the right way up before being written to disk.
The speed of the turntable's rotation is determined by the diameter of the object being processed. Wider objects, such as a jar, require a slower rotation while thinner objects, such as a vial, can be rotated faster. The software provides an estimate function through its command line interface (CLI) which can be used to guess the rotation duration for the object given its diameter in mm, number of pixels per mm as seen by the camera (by default, 21) and the source frame rate (either the stream frame rate or the video frame rate).

Data resources
Example media from the MALICE, VILE and ReVILE setups can be downloaded here:

Results
The workflow for each of the three approaches is outlined in Fig. 4. All setups provide an output aimed at enabling the extraction of data from labels (Figs 5, 6, 7). Image timing estimates varied from 21-123 seconds per vial, or 29-171 vials per person per hour, and are summarised in Table 1, including the number of vials imaged during the tests and an extrapolated estimate of the time required to digitise a collection of 5000 vials.    Ahead of the curve: three approaches to mass digitisation of vials with ...

Conclusions
The three setups described provide means of digitising wet collections at a considerable speed and with a variation of outputs that are useful. We propose that a ReVILE approach is a more versatile solution for wet collection digitisation. While MALICE is arguably the fastest approach the setup is not ameanable to the large variation of vials available in collections not only due to mirror size requirements, but also the limits to the depth of field required by the camera. Using a combination of the VILE and ReVILE approach which have the same setup but differ in the image capture and processing it is possible to create the output of both setups as is suggested in Fig. 4. While there is no doubt that images of the specimens themselves is often of use, for example condition checking and potentially identification (depending on the taxon group) these images are often much slower to capture and require specimen handling which may result in damage. The imaging setups presented provide multiple solutions for a "first pass", especially for collections which lack a basic inventory. A summary of the strengths and weaknesses of these methods is provided in Table 2.   Table 2.
A summary of the pros and cons of the three systems presented.
Ahead of the curve: three approaches to mass digitisation of vials with ...