Research Ideas and Outcomes : Research Article
Print
Research Article
Ahead of the curve: three approaches to mass digitisation of vials with a focus on label data capture
expand article infoSteen Dupont, Josh Humphries, Alice J Butcher, Edward Baker‡,§, Laura Balcells, Benjamin W Price
‡ Natural History Museum, London, United Kingdom
§ University of York, York, United Kingdom
Open Access

Abstract

There has been little research on novel approaches to digitising liquid-preserved natural history specimens stored in jars or vials. This paper discusses and analyses three different prototypes for high-throughput digitisation using cheap, readily available components. This paper has been written for other digitisation teams or curators who want to trial or improve upon these new digitisation approaches in liquid preserved collections.

Keywords

natural history collections, imaging, vials, jars, liquid preserved collections, spirit collections, wet specimens, digitisation

Introduction

A large proportion of natural history collections are stored in jars or vials, for example a recent inventory of the Natural History Museum, London (NHM) collections estimates over 17 million specimens are stored in 1.8 million curatorial units (Wilson et al. 2018). The collection is currently being digitised and made available via the NHM Data Portal (Scott et al. 2019). The NHM collection spans over 400 years of collecting, leading to variation in the size, shape, sealing and labelling of jars and vials, all of which increase the complexity of developing an efficient digitisation solution. While there is currently no alternative to specimen handling if standardised specimen images are required (e.g. dorsal, lateral and ventral views), recent advances focussing on the label data associated with dry pinned insects have significantly increased the rate at which label information and overview images of the specimen can be captured (Price et al. 2018).

Previously published workflows for imaging wet specimens and their labels have removed the specimens from their container to be laid out for imaging in a petri dish (DeWalt et al. 2018) or a 3D printed box (Mendez et al. 2018). The average time for these two approaches has been reported as 2.78 minutes per vial with a single operator (DeWalt et al. 2018) or 8 minutes per vial when using two operators (Mendez et al. 2018), equating to a rate of 2.78 - 16 minutes per vial or alternatively 3.75 - 21.6 vials per person hour, excluding data transcription. At these published rates the NHM insect spirit-preserved collection, estimated at 700,000 vials, would take between 26 to 150 person years to image and catalogue. Furthermore if the primary goal is to create an inventory of the collection and record the data associated with these vials, rather than images of the specimens themselves, this might be achieved more rapidly by imaging the vials while reducing the risk of damage to the specimens by minimising their handling.

Recent advances in multi-camera imaging, focussed on label data extraction from pinned insects (Price et al. 2018) enables the digitisation of over 1000 pinned specimens per person day. We propose that a similar multi-image, multi-angle, data-focused approach can be used to speed up imaging and data capture from uncatalogued wet collections. We present three tested prototype approaches: (1) multiple mirrors used to replicate the multiple angles captured by ALICE (Price et al. 2018) with one camera; (2) a commercially available rotating turntable; and (3) slit-scan imaging using a custom built rotating turntable. All three solutions are described and discussed in the context of collections digitisation with particular focus on benefits, limitations and further improvements.

MALICE: Mirror Angled Label Image Capture and Extraction

The MALICE setup is an iteration of the ALICE system described in Price et al. (2018). The aim of MALICE is to capture multiple angled images with a single overhead camera using mirrors that enable the reconstruction of a label using post-processing. This composite, reconstructed label makes data extraction (e.g. transcription and OCR) easier.

Hardware

Built with materials including LEGO©, 5 acrylic mirrors, a base and specimen holder inset cut from Formex (Fig. 1a-d). Materials used were chosen to ensure easy and cost effective replication. As a prototype MALICE is built to be configurable with easy adjustment of mirror tilt to accomdate different vial sizes. Mirror tilt is achieved using a friction joint (Fig. 1e). Once an optimal mirror angle is chosen, the setup can be built out of any suitable material. For testing the MALICE base plate was attached to a Kaizer RS10 copy stand with re-usable adhesive putty (Blu Tack) during imaging. Images were taken using a Canon 5DSR and Canon 24-70mm lens, controlled with EOS Utilities v.3.

Figure 1.  

MALICE vial setup showing the acrylic mirrors (a) LEGO mirror tilt arms (b), formex base (c) and LEGO friction joint (d)

Workflow

Imaging: A vial is removed from the jar, placed in the centre of the MALICE setup, with a unique identifier (UID) label, encoded as a barcode, placed horizontally next to it. A single image is taken from above,capturing five views of the vial in the mirror images. The vial is then opened andthe barcode label is inserted. The vial is then re-sealed and placed back in the original jar.

Processing:

  1. Images were renamed with Gouda, using the command: "decode_barcodes.exe -a rename --avoid-collisions libdmtx "path to files"".
  2. The five vial reflections were cropped out of the image using Adobe Photoshop (CC 2019).
  3. Cropped images were manually deskewed, rotated and reflected using Photoshop creating a composite image.

VILE: Vial Imaging and Label Extraction

Successfully taking multiple images by rotating an object in front of a fixed camera is a straightforward approach, but requires some automation to make the approach scalable for mass digitisation. VILE uses an off-the-shelf rotary table and controller positioned in front of a consumer camera, controlled using a laptop.

Hardware

Images were taken using a Nikon D5300 camera. Specimen vials were placed on a Stackshot rotary table with controller (https://www.cognisys-inc.com/store/rotary-table.html) (Fig. 2) Both the camera and rotary table were controlled using Helicon Remote v.3.9.1.

Figure 2.  

VILE setup showing the camera (a), Stackshot rotary table (b)

Workflow

Imaging: A vial is removed from the jar, placed in the centre of the rotating stage, with a UID label, encoded as a barcode placed vertically on a stationary stage within the imaging frame. Images are taken at six different angles corresponding to 0, 72, 144, 216, 288 and 360 degrees. The vial is then opened, the UID label inserted, the vial re-sealed and placed back in the original jar. The arbitrary degree between the images reflects a mistaken configuration of the stackshot turntable resulting in the duplication of the initial view with that of 360 degrees, which was subsequently deleted. Future imaging should utilise 0, 60, 120, 180, 240, 300 degrees to maximise label views while removing duplication.

Processing: Image processing followed these steps:

  1. Images were renamed with Gouda, using the command: "decode_barcodes.exe -a rename --avoid-collisions libdmtx "path to files"".
  2. Images were cropped, minimizing extra space, with XNConvert v1.76, saving the output images as JPG at 90% quality.
  3. Stitch 5 images together using mageMagick 7.0.7 ( https://github.com/ImageMagick ) with a recursive script based on the image name (e.g. “magick 013670425-4.jpg 013670425-3.jpg 013670425-2.jpg 013670425-1.jpg 013670425.jpg +append 013670425_combined.jpg”)

ReVILE: Revolving Vial Imaging and Label Extraction

The technique applied in the ReVILE setup is based on the 19th century concept of slit-scanning and primarily used today for panoramic photography. This concept involves a slit moving across the photographic medium (film or digital sensor), thereby exposing only a small section of the photographic medium to light at any given time. The implication of the temporal element of the slit scan means that a slit-scan image is composed of several snapshots in time. By moving the subject of the photograph instead of the slit, the process is inverted, reproducing a 3D object on a 2D plane. The imaging concept of ReVILE is no different from VILE in that an object is rotated in front of a stationary camera. However, generating an image by extracting vertical lines from the camera video and aligning these to create a composite image requires a more precise camera-object alignment and a rotation speed that is determined by the diameter of the object.

Hardware

Camera, turntable, controller and lighting were built into a formex casing. The setup presented uses a Canon 5DsR camera with a 100mm Canon macro lens for capturing video and still images. The turntable is custom-built using 2mm aluminum, and polystyrene sheets. Movement of the turntable is provided by a 5v stepper motor (28BYJ-48 5V) controlled by a drive chip (ULN2003) and an Arduino UNO. Lighting was provided by 3 COB 48-SMD LED panels for front lighting and a Neewer 40004082 light panel for backlighting (Fig. 3a-e). To ensure camera, turntable and lightshield (Fig. 3f) alignment as well as XYZ adjustment the camera, the turntable and light shield were mounted on a Small Rig rail system.

Figure 3.  

A lateral image of ReVILE with the side panel removed to show the camera (a), turntable (b), stepper motor (c), Arduino Uno and motor driver controllers (d), front light panels (e), back light panel (f), light shield (g) and position of vial (arrow).

Workflow

Imaging: A vial is placed in the centre of the turntable and filmed over a 720 degree rotation. Rotating the vial multiple times ensures that all possible angles are covered. The setup is calibrated by taking a single frame of a scale at the centre of the turntable and using image software to estimate the width (in pixels) of a single millimeter. The number of frames \(n_f\) required can then be calculated as such:

\(n_f = {2\pi dlxw_1 \over 2lx-dsw_1}\)

Where \(d\) is the vial diameter (in mm), \(l\) is the focal length (in mm), \(x\) is the size of the frame (in pixels) across the x axis of the vial (i.e. width if landscape, height if portrait), \(s\) is the size of the sensor (in mm) across that same x dimension, and \(w_1\) is the width of 1mm in pixels (obtained during the calibration step).

The duration of a single rotation can be calculated by dividing this figure by the stream's framerate: \(t = {n_f \over r}\)

This figure is largely a helpful estimate and high levels of accuracy are not necessary; it is preferable to err on the side of longer rotations to ensure that detail is not missed.

In the example dataset, videos were captured at approximately 60 frames per second, at a focal length of 100mm and a resolution of 1280x720 pixels (in portrait orientation). The size of the sensor on the Canon 5DSR is 36x24mm. The width of 1mm on the calibration frame was approximately 21 pixels.

For a 10mm vial: \(n_f = {2\pi \times 10 \times 100 \times 720 \times 21 \over 2 \times 100 \times 720 - 10 \times 24 \times 21 } = 684\) and \(t = {684 \over 60} = 11.4\)

For a 20mm vial: \(n_f = {2\pi \times 20 \times 100 \times 720 \times 21 \over 2 \times 100 \times 720 - 20 \times 24 \times 21 } = 1419\) and \(t = { 1419 \over 60} = 23.7\)

Processing: ReVILE is capable of producing 3 separate image outputs outlined in the results section. The primary output is the composite rolled out image. The workflow of the rollout photography output is outlined below and the potential for the other two outputs - a composite image of the vial at different angles (equivalent to the VILE output) and an interactive digital-only output that allows for user rotation of the imaged vial - are examined in the discussion.

Custom software was developed to control the camera and turntable as well as process the camera’s outputs to produce the rolled out image of the object. This software is available at https://github.com/NaturalHistoryMuseum/revile.

Once the object has been securely positioned on the turntable in front of the camera, the software follows the following steps to produce the rolled out image of the object:

  1. Initialise the video capture process - this is done first as it takes a couple of seconds to begin capturing, but step 2 should generally begin first;
  2. Start rotating the turntable for the specified number of seconds;
  3. Iterate through the stream or video frames, extracting the middle column or row (depending on camera orientation) of pixels from each frame;
  4. Concatenate the extracted lines of pixels from each frame together to build the image;
  5. Undertake any postprocessing steps, e.g. rotating the output;
  6. Crop the image so that it shows only one complete rotation, using feature matching to locate similar columns of pixels; and
  7. Write two JPEG images to disk: the complete, "raw" output, and the cropped version.

Using the hardware setup pictured in Fig. 3, the middle row of pixels is extracted from each frame as the camera is positioned on its side. The concatenated image is rotated through 270 degrees to turn it the right way up before being written to disk.

The speed of the turntable’s rotation is determined by the diameter of the object being processed. Wider objects, such as a jar, require a slower rotation while thinner objects, such as a vial, can be rotated faster. The software provides an estimate function through its command line interface (CLI) which can be used to guess the rotation duration for the object given its diameter in mm, number of pixels per mm as seen by the camera (by default, 21) and the source frame rate (either the stream frame rate or the video frame rate).

Data resources

Example media from the MALICE, VILE and ReVILE setups can be downloaded here:

MALICE:https://doi.org/10.5281/zenodo.3497060

VILE: https://doi.org/10.5281/zenodo.3462015

ReVILE: https://doi.org/10.5281/zenodo.3727644

The software for ReVILE can be found here:

https://doi.org/10.5281/zenodo.3733440

Results

The workflow for each of the three approaches is outlined in Fig. 4. All setups provide an output aimed at enabling the extraction of data from labels (Figs 5, 6, 7). Image timing estimates varied from 21-123 seconds per vial, or 29-171 vials per person per hour, and are summarised in Table 1, including the number of vials imaged during the tests and an extrapolated estimate of the time required to digitise a collection of 5000 vials.

Results of the imaging tests completed for the three setups including test metrics and an estimate of the imaging time required for a collection of 5000 vials for reference. Note the estimates for MALICE and ReVILE exclude any recuration and rehousing.

Number of vials imaged Total imaging time (minutes) Еstimated vials/ person hour Estimated imaging time in person days (5.5 hrs) for a vial collection of n = 5000 with databasing
MALICE

42 (only imaged)

15

168

23.9

48 (with databasing) 75 38
VILE 2349 (with databasing) Not recorded 30-40 26.5
ReVILE

21 (only imaged)

18

70

31*

(with databasing) 29*

*assumed increase in imaging time of 72 seconds per vile when databasing based on timings from MALICE, time also includes image processing.

Figure 4.  

Illustrative workflow for the three vial digitisation approaches MALICE, VILE and ReVILE.

Figure 5.  

MALICE: Image output of MALICE including original output image (a) and the final processed image (b).

Figure 6.  

VILE: Image output of VILE after cropping and stitching five individual images together.

Figure 7.  

ReVILE: Two Rollout photography outputs of ReVILE. Left to right: a frame from the video (rotated 90° clockwise) showing the vial itself; the uncropped rollout image, covering more than one full rotation; the cropped rollout image, showing only one 360° rotation; the cropped rollout image, "shifted" across (by transferring a manually-defined block of pixel columns from the left side of the image to the right) to show the complete label

Conclusions

The three setups described provide means of digitising wet collections at a considerable speed and with a variation of outputs that are useful. We propose that a ReVILE approach is a more versatile solution for wet collection digitisation. While MALICE is arguably the fastest approach the setup is not ameanable to the large variation of vials available in collections not only due to mirror size requirements, but also the limits to the depth of field required by the camera. Using a combination of the VILE and ReVILE approach which have the same setup but differ in the image capture and processing it is possible to create the output of both setups as is suggested in Fig. 4. While there is no doubt that images of the specimens themselves is often of use, for example condition checking and potentially identification (depending on the taxon group) these images are often much slower to capture and require specimen handling which may result in damage. The imaging setups presented provide multiple solutions for a "first pass", especially for collections which lack a basic inventory. A summary of the strengths and weaknesses of these methods is provided in Table 2.

A summary of the pros and cons of the three systems presented.

Pros Cons
MALICE
  • Fastest imaging setup
  • Commercial software available
  • Only camera control needed for imaging
  • Custom built hardware
  • Requires complex image processing
  • Vial size restricted by setup and mirror size
  • Views restricted to number of mirrors
  • Curved labels might not be fully visible in a single vial image
VILE
  • Commercial hardware and software available
  • Both camera and turntable tethered and operated through a single user interface
  • Imaging speed currently restricted to 15 degrees/sec
  • Curved labels might not be fully visible in a single vial image
ReVILE
  • Multiple image output options
  • Can give full 360 view of imaged vials
  • Output is a flattened image that allows for direct image processing on labels
  • Custom built hardware
  • Requires complex image processing
  • Straight labels in vials can be skewed due to the curvature correction
  • When using video capture mode this generates a large amount of temporary data before processing
  • Centering of the vial on the turntable critical for end result
  • Rotation speed must be estimated correctly for each object to produce good results

Acknowledgements

We would like to thank Laura Sivess and Lyndall Pereira da Conceicoa for discussion around the VILE setup. Zoe Hughes and Jennifer White for testing the ReVILE setup and Vladimir Blagoderov for valuable discussion and suggestions for the MALICE design. In addition we would like to thank Laurence Livermore for constructive comments on the manuscript.

Funding program

Funding for staff time and materials to create ReVILE and MALICE was provided by the Natural History Museum Digital Collections Programme and ICEDIG.

ICEDIG was funded under H2020-EU.1.4.1.1. - Developing new world-class research infrastructures.

Grant title

ICEDIG – “Innovation and consolidation for large scale digitisation of natural heritage”, Grant Agreement No. 777483.

Author contributions

All authors contributed to devising the project. Software was developed by Alice Butcher with additional contributions by Josh Humphries. All authors contributed to writing the manuscript.

References