For the purposes of this report, ‘robotics’ and ‘robot’ refer to physical (mechanical) constructions with electronic and software components, designed to replicate certain human actions with a degree of autonomy.

Robotics is a form of automation - technologies that reduce the need for human intervention in a vast range of processes. Factory or warehouse automation often uses machines to perform highly repetitive and standardised tasks, sometimes boringly simple, sometimes highly complex, with processes for identifying and handling exceptions e.g. by alerting a human operative. Due to their use of sensors to interact with physical objects, and often to being reprogrammable, robots are often associated with greater versatility than other automated machines. For this report, we use ‘automated warehousing’ and ‘automation’ to refer to the range of technologies that allow mass processing or the management of a large environment, for instance retrieval of collections objects from a warehouse (often using mechanical components that do not resemble humans at all); and ‘robotics’ to refer to more ‘human’ processes such as the handling of an individual specimen (e.g. by a robot ‘arm’); however, these are not hard and fast distinctions - for example a robot arm is usually capable of more than human movements e.g. greater rotation, and robots may be part of the automation solutions in a bulk processing environment.

In addition to robotics and automated warehousing, this report touches briefly on assistive technologies, particularly those such as ‘exosuits’ that are designed to assist with, and reduce risk from, handling or lifting large, heavy and awkward objects.

The terms robotics, robot and automation are also now frequently applied to process automation using software, without any mechanical or physical components. Software automation is outside the scope of this paper, as it is addressed in related reports including the ICEDIG report on automated text digitisation (Owen et al. 2020), and the SYNTHESYS+ Specimen Data Refinery (Walton et al. 2020). Software automation is already in use across many aspects of digitisation, including image processing, batch quality control, barcode detection and image segmentation (Allan et al. 2019, Hudson et al. 2015).

At this time, to the best knowledge of the authors no natural history collections has implemented a fully automated solution from storage/retrieval to imaging (or other uses) and back to storage, nor has such a solution been described or analysed. By developing independent components, however, which can be connected in the future and build on existing practice, progress can be made towards an end-to-end solution (see Fig. 1 and Table 1 below).

Figure 1.  

Possible end-to-end process of storing and imaging natural history objects for automation.

At its simplest, automation can refer to storage that is controlled electronically - for example where we have remote control of environmental conditions such as temperature and humidity that could be described as automation. The BBC archive in Perivale uses automated compactors in their master film store - this is a cold store for flammable cellulose nitrate film, and automation allows for the position of the compactors to be standardised every night to ensure equal cooling throughout*1. This solution is not used in their other archive vaults, however, because these are subject to greater use and there is a high risk of system problems and breakdown if dust or debris get into the runners.

Automated warehousing with automated or partly-automated retrieval is more complex - this requires every relevant object to be identifiable, ideally with the physical location in a database. This can be at the object level, or per systematic category (based on taxonomy, geography and other used categories in NH collections) at either shelf or shelving unit level. Machine readable barcodes (or other solutions such as radio-frequency identification - RFIDs) at object and/or storage container level would increase the effectiveness of such a system.

Due to the great variety of natural history collections, multiple automated imaging systems are likely to be needed. In this case imaging automation of natural history collections needs to be split along more than taxonomic lines alone. In assessing digitisation costs for outsourcing, the Smithsonian’s National Museum of Natural History considers approximate object size; volume/dimensionality (is it flat or not); quantity of objects to be processed; and degree of complexity in the handling required e.g. whether objects can come straight out of storage to imaging, or require e.g. conservation or curatorial interventions*2. These are all likely to be equally relevant criteria in considering the possible application of automation to the workflow stages above, as is preservation type.

"[...] it’s worth remembering that nothing stumps a robot quite like a bag of oranges. They just can’t deal with it. The bag moves in too many weird ways, there are no obvious bits to grab hold of, and if you squeeze too hard you end up with orange juice instead." - Vincent 2018

Industrial uses of robots and automation highlight a number of key strengths and limitations that apply across sectors (The Manufacturer 2018, Faya 2013, Vincent 2018). Robots are good at repetitive tasks - while they may wear out, they can typically carry out repetitive tasks numerous times, at high speed, without breaks or risk of injury. Many of the steps in Table 1 involve repetitive elements, such as taking containers in and out or pressing a button to take an image.

In relation to handling of collections, however, the quote above is highly relevant. Natural history collections can be a challenge for robots because there is very little that is standardised between collections (within or between institutes) and even within one type of collection. The specimens are fragile, don’t necessarily stay put in one place on a flat surface, and can also be soft and limp.

For example, the picking of pinned insects by the pin is relatively easy for humans, but a roboticised solution is not yet available which can ensure the integrity of the specimen. In this example with pinned insects in a drawer, the robot needs to identify a specimen, locate its pin, grasp it at a suitable point (clear of specimen and labels) with the correct amount of pressure, lift and move carefully. Then a suitable location for placing it needs to be identified (empty, suitable surface) and place it with the pin still vertical with the correct amount of pressure. Experts from Eindhoven University of Technology commented that identification of a single specimen is still a challenge, as long as the specimens aren’t regularly spaced out in a grid - overlap of specimens would be an issue for robotic identification (e.g. butterfly wings). Subsequent manipulations, such as removal of labels or placing in a specific orientation would be even harder.

Because these challenges exist in other industries, however, progress in robotic development is swift and robotic handling is increasingly becoming more sophisticated, for instance using computer vision with sensors as input, and a variety of physical handling solutions (see ‘grippers’ below).

Another factor to be addressed in relation to specimen handling is stakeholders’ opinion. Institute management and collections staff need to support the notion of automation and robotisation of imaging and storage workflows, a substantial change. Systems will have to be trialled and demonstrated before risks are likely to be seen as acceptable. However, the head of a major R&D institute developing an automated 3D digitisation system for heritage and industry has noticed that many collection holders are becoming more comfortable with the idea: for example digitisation technologies such as conveyor belts are now commonplace (see case study 3.4).

While direct robotic handling of natural history specimens may not currently offer an acceptable balance of risks and costs against benefits, there is greater potential in relation to indirect handling, i.e. handling storage containers/units such as drawers or jars. At present, standardisation of storage is often limited, though not as limited as standardisation of specimens. Historic collections and object variety often mean a wide variety of storage, e.g. jars of all sizes, shapes, sealants etc. There may also be limitations such as jammed drawers and uneven navigation routes (although these also impact human workers). An alternative is to increase standardisation with robotic handling in mind - this involves higher cost and effort before adoption of automation, but may be suitable when collections moves and rehousing are planned.

Hardware for automation is developing rapidly. Some specific examples of relevant technologies are included in detail in the case studies below. Some of the most relevant current hardware solutions include robotic ‘arms’ and grippers; conveyor belts; and exosuits. The cost, precision, reliability and ease of implementation of these hardware components is key to the success of automation. In general, the abilities of this hardware continue to increase (e.g. precision of handling) and their costs to fall.

Automated solutions similar to those used in commercial warehousing can be used for the large-scale handling and retrieval of natural history objects in storage, improving retrieval times from storage, space efficiency, and allowing for e.g. low-oxygen environments which are beneficial for conservation of some collections but can be dangerous to humans. Library collections are close to natural history collections: they serve to make object based research available; most of the objects spend most of their lifetime in storage but need to be retrievable in ways that are often not predictable; and object condition depends on controlled climate (temperature, humidity, oxygen levels, and pest control). Library books have somewhat fewer variables than natural history collections do, although similarly for automation they rely on standardised storage, with e.g. books in crates, boxes or ‘bins’; and newspapers or other larger formats secured on pallets.

In recent years, some large libraries, on a national level or at large universities, are building advanced warehouses with storage robots. Some examples are the Glucksman Library at the University of Limerick, Macquarie University Library, North Carolina State University Library, University of Missouri Kansas City Library and University of Chicago Library; these systems are developed by Dematic. In most cases, the books are stored in standardised steel bins, which the robot can manipulate when a book request is made through the catalogue. The physical storage location of the book is recorded in the database. The organisation of the objects can remain as it was before (e.g. by topic, author, year, language), but this is not necessary - in the event of complete database malfunction with no backup data available all books would be untraceable, but with a backup protocol in place this should never occur. After the request is made, the robot retrieves the bin in which the item is located. The whole bin is then made accessible to a librarian, who takes out the specific requested item and processes it for pick up by the requester. The rest of the bin is returned to storage. Return is very much the same: either a random bin or the dedicated bin is retrieved by the robot for the librarian to put the book into, after which the bin is automatically reshelved.

Usually the parts of the collections with highest use are still available on the shelf for the readers and only the “reserve collections” or less used collections are stored in the automated warehouse.

The British Library started their project for a new building and automated storage system in 2004-2006. A second building and automated storage system with shelves going 30m high went live in 2015 for the newspaper collection. The company that built the systems for them is TGW Group. After trial and error, they discovered that, in the case of newspapers, bundling and securing a set of papers with a canvas belt was the simplest solution for librarian and robot. In the first automated warehouse there were initial problems with box size and distortion of plastic boxes - these issues have now been resolved.

Items are accessed and requested through relevant systems. A user, e.g. in the reading room in London, can request an item through the digital catalogue. The request is then sent to the system in Boston Spa (Yorkshire, 280km north of London), where a robot retrieves the relevant container, transporting it to workers who take out the specific requested item. The rest of the box is returned by the robot to its place, while the requested item is set on a trolley for transport. Trolleys are loaded into lorries with daily (or more) deliveries to the reading room in London where the user has access. After use, items are sent back by the same route; the whole outgoing process can be done within 24 hours of the request being made.

While the storage area has limited access for humans (given height and restricted oxygen environment), books remain organised by subject and author in the boxes, although the robotic system keeps more frequently-requested boxes closer to the picking station. Maintenance of the hardware is not done by library staff: like cleaning services, it is completely outsourced to specialised engineers on call. Support is reduced out of hours (e.g. overnight) owing to cost - systems are still used at these times but the functions used are reduced to those where the BL operators can usually resolve any issues themselves. The newspaper warehouses contain 70km of storage. To make this work every item needs to have its own identifier and physical storage needs to be completely in sync with the database and software; this initially proved to be a challenge. Over the years, no objects have been dropped, however there have been a few cases of objects or bins becoming stuck. After the initial phase, they now experience very little issues. With the number of overdue outstanding request as indicator, there are almost never problems.

The workflow of the users has changed very little because these items were already only available on request, with 24 hour processing time. The work of the librarians has changed: to keep up with the demand they work in shifts and due to automation the work can be done with less training. The newspaper collections were not in the best condition before the move to the new automated warehouse; now their condition is stable. Because the storage area is off-limits to humans, a much lower oxygen and temperature level can be maintained (oxygen comparable to Mount Everest) which is the main source of deterioration of the material. The low oxygen level also reduces the risk of fire. Another major factor is cost: in London the cost of a meter of shelving space was about 40GBP, in Boston Spa 4GBP. This is also due to the use of automation to enable storage at a much greater height than would be usable by humans without assistive technologies, so that each meter of floor space accomodates more items.

Advantages of this solution are space efficiency, lower housing cost from off-site storage, improved specimen conditions due to climate control, and reduced fire hazards. A major part of the current workflow of natural history curators and researchers is to browse the collections, going through boxes and shelves, enabling associative specimen selection. This can be a specimen sitting next to the initial target, from the same location, same collector, same label format etc. An automated warehouse would not allow this workflow physically. However, this can be achieved through software solutions. Imaging per shelf can substitute initial shelf/box browsing; image recognition of labels, transcribed metadata such as collector and location and data linkage allows for queries and serendipitous discovery across a range of variables. A tiled view with associated material of the specific request based on taxonomy, location, collector would also recreate this workflow.

Most current library warehousing robots still require human picking from bins/containers. This offers a useful lesson for natural history collections in relation to standardising storage units where possible - while the variety of items and storage types is greater than in libraries, the volume of many collections could still allow for extensive standardisation of e.g. drawer or jar size, shape and other features. The key benefits of automation at scale for specimen handling involve the complete refitting or new development of storage spaces, so can only be considered e.g. when moving collections at scale to new or refurbished facilities, however pilots such as the use of a small robot to fetch collections of one type within one storage area may be possible. There is a potential to collaborate with other collection-based institutes: either other natural history museums in the larger region, cultural history collections (museums, archaeological departments) and libraries. This would reduce the cost of building new facilities and the transportation to the main site.

While use of robots in collections storage and retrieval is currently rare, robots and automation have been and continue to be rapidly adopted and developed for e-commerce logistics and order picking. Companies such as DHL, Amazon, large grocery stores (e.g. Ocado in UK) are adapting their warehousing process to extensive automation. The process known as order (or item) picking is closest to the needs of natural history collections.

One variant is where the entire cabinet/shelving system containing the requested object is picked up by a transporter robot, and transported to a human for object picking (for example at Amazon fulfillment centres). An alternative is a grid system with crates or racks of goods below and autonomous robots on top, navigating the grid to pick up an object or packet e.g. using a suction cup, storing the item in the cart and dropping several objects at a packing station for further shipping, as used by Ocado (Vincent 2018).

These examples demonstrate the massive scale at which roboticised warehousing operates, but also how humans and robots complement one another, including for many object handling tasks. These users of automation are also able to accept, and factor into costing, a certain amount of damage or wastage, while collections cannot accept any margin for this as objects are not replaceable.

A report by DHL (Bonkenburg 2016), examining the potential for robotics and automation in logistics, raised concerns that any ‘logistics robot would need to handle a wide array of different parts in an infinite number of combinations. It would help if the robot could see, move, and react to its environment.’ This echoes some of the barriers we believe may exist in our sector and mention in this report - however automation is now increasingly widespread and vital to the logistics industry so this is a useful lesson in how quickly limitations are being overcome.

Please note that a full description of these solutions can be found in the ICEDIG report on digitisation of herbarium sheets (Guiraud et al. 2019).

Herbarium collections provide the best examples of automated mass digitisation incorporating both physical and software components, as there are currently two of these solutions by Digitarium (now Bioshare by Sertifer Consulting Ltd.) (Digitarium 2017, Bioshare Digitization 2017) and Picturae (Picturae 2017). These systems still require handling by human operators to load and unload, to apply barcodes, and to detect special specimens such as types and specimens requiring restoration. Specimens are loaded onto a conveyor belt system, which automatically moves a series of specimens through an imaging station. At the end of the line a human operator unloads the specimens. Image adjustments (including colour correction, potential sharpening, cropping, etc.) and file naming through barcode detection are done automatically. After quality assessment the batch of specimens is ready to be returned to storage. To achieve the highest speed it is important that as few as possible human actions are required: any positioning and adjustment of labels slows the process down. These systems have been tested as being 4 to 5 times faster than one at a time scanning of herbarium sheets - however a key barrier to achieving these imaging speeds is the ability to have specimens ready for this process in sufficient volume (e.g. pre-curation). This issue is associated with storage optimisation, which in some instances could greatly increase the speed of retrieval and replacement.

The human component in this process is the loading and unloading of specimens and detecting special cases. Sometimes a specimen has associated material in an envelope, which can be opened into trays to be imaged as well. It is essential that any loose material and labels stay together. The fragility and special attention needed for loose material currently mean that human involvement cannot be fully replaced by automation.

The conveyor belt component of these solutions could be used to automate digitisation workflows of other collections. It has indeed already been applied to pinned insects (Tegelberg et al. 2014, Tegelberg et al. 2017). Herbarium sheets are more standardised and closer to two -dimensional (flat), making it easier to automate digitisation. Issues such as depth of field and positioning of an object taking into consideration stability, standardised anatomical, and diagnostic views are greater when objects have greater depth. Still, if only label images are needed (e.g. for transcription purposes, either by humans or through machine vision), or lower resolution specimen images are sufficient, then the current herbarium digitisation conveyor systems may be adapted for objects such as open boxes of bones, molluscs and other dry material. There may also be scope to image labels on jars from liquid collections, although this may come with increased risks to manage, if using electronics in proximity to liquids, especially flammable liquids.

Another important aspect of these workflows is that image correction, such as cropping, straightening and colour calibration, can be done automatically because there are a number variables such as shape, camera position and settings which are controlled by software to an appropriate level in this workflow.

In general, conveyor belts can be an efficiency boosting component when certain conditions are met. First, it takes a certain amount of time, effort and space to set up a system with a conveyor belt. The quantity of material needs to be large enough to balance the set up time/cost. Secondly, the diversity of the material needs to be within a set range. For example, this includes dimensions, views to be captured, acceptable levels of vibration and similar parameters. Further, the handling needs to be able to be broken up into independent actions at certain stations of the workflow. By simplifying the handling into separate repetitive actions (and reducing the number of actions where possible) the speed of the whole process can be improved. Finally, all the necessary preparations to standardise and prepare the collection for automated digitisation need to match the speed of the conveyor belt. This is the key barrier to optimising the benefits of automation through these systems. The experience of many digitisation projects (in both cultural heritage and natural history) shows that the amount of preparation work and the impact on the collections department is often underestimated. This is especially relevant for digitisation projects procuring a conveyor belt system and/or service through outsourcing, where it is not economic to stop and start the process.

The Fraunhofer Institute for Computer Graphics Research IGD (Darmstadt, Germany) is a research institute focussing on various fields in applied science, which has a Competence Center Cultural Heritage Digitization, including the research lab CultLab3D. They have developed multiple mass-digitisation 3D scanning solutions, that have been used for industry (e.g scanning of shoes) and cultural heritage (e.g. globe digitisation at Friedrich Schiller University) (Santos et al. 2017a, Santos et al. 2017b, Ritz et al. 2018). The developers understand the limitations of working with heritage objects and the need of curators for high quality and calibrated, repeatable results. For this case study, conversations were held with Pedro Santos, head of the Competence Center, as well as attendance of an open day with demonstrations & discussion at the Getty Research Institute in Los Angeles (USA) on 2019-07-15. A report of the open day is given in Appendix 1 of Walsum (2020).

The first solution from CultLab3D is a conveyor belt system equipped with cameras on two half-arcs with ring lights and glass turntables to allow imaging of the object’s underside (Fig. 7). Multiple objects can be loaded onto the conveyor belt, so that the imaging system doesn’t have downtime while a human operator loads the next object.

Figure 7.  

The CultLab3D conveyor belt and automated imaging system (© CultLab3D).

The newer solutions, named CultArm3D, are mounted on robot arms and can be either photogrammetry, laser or structured light based (Fig. 8. This allows flexibility in choosing the right solution based on cost, resolution, object surface type and colour accuracy requirements. The system can work with standard DSLR cameras, as well as professional high-resolution Phase One cameras. With the current camera and lens combination, objects with dimensions between 40×40×60cm (max 20kg) can be digitised, with a maximum resolution of 20 microns. When different lenses and a robot arm with longer reach are used, smaller and larger objects can be captured. Highly controlled focus stacking for small objects is an option. These robot arm systems can also be attached to the end of the conveyor belt system when insufficient data is detected in specific areas after initial alignment, to target these areas only.

Figure 8.  

The CultArm 3D automated imaging system (© CultLab3D).

Figure 9.  

Manual Handling risk assessment template extract.

The CultArm3D system is fitted with a glass turntable so that of many objects the underside can be sufficiently captured. Two positions have been pre-programmed to minimise refraction from the glass as well as to ensure that the arm does not hit the turntable. This does mean that objects with complex undersides (or objects with no definable undersides, which includes many non-man-made objects) will still need to be repositioned to properly capture them from all sides. Another scenario is objects that will not sit sufficiently stable by themselves and require some support. As long as there is sufficient contrast between the supporting material and the object, this can be removed automatically from images.

The capturing system is not currently compatible with Mac. The computer specifications for capture are very low: it can run on a lightweight laptop. However, the requirements for the processing stage are much greater and can be a bottleneck. A typical capture results in hundreds of images.

The safety of the object and of humans in the vicinity needs to be assured to make this robotic system viable. To do this, multiple layers of security measures are present. The robotic arm (Universal Robotics UR10) is designed to be safe for human workers in its vicinity. Due to its inbuilt collision detection, contact such as with a human causes the arm to freeze. The same happens during a power failure, where some other robotic arms might unlock their joints so that gravity pulls the components down. The arm is programmed with predefined zones that it avoids, this includes the turntable and some other areas. Finally, based on an initial scan, a cylindrical space is defined that the arm cannot enter, giving a safe margin around the object.

After calibration and besides loading objects, the capturing process does not need a human operator. The object is initially photographed against a lit background so that only the silhouette is captured, which some software can use to improve photogrammetric algorithms. The robot arm is an off-the-shelf product which Fraunhofer can upgrade at a later point, including on existing systems. While Fraunhofer’s code should work with a new model robotic arm made by the same manufacturer from the same product line, only minimal code changes would be needed for a different arm, as long as these have the same degrees of freedom. Focus stacking processing, and detection and discarding of unsharp areas in the photos are done through scripts. To optimise the quality of the output, the pipeline generates depth maps to be able to ignore poor data (out of focus) in the images. The processing pipeline is not locked to a specific photogrammetric software, and can use either various commercial packages, open source software or software developed by Fraunhofer, as long as it is scriptable. Fraunhofer has developed software to optimise the 3D output for various presentation goals including web optimisation (this includes UV unwrapping, retopology, normal maps, publication in web specific format).

Fraunhofer’s 3D scanning robot has been designed to fully automate the capture process, as well as capturing extra data to be able to fully automate the 3D processing workflow. Some human interaction may still be required during post-processing, especially for the more complex objects. Compared to the herbarium imaging solution, the system deals with added complexity because of the 3D nature of the objects and many more processing steps that the raw captures and generated models need to go through.

Fraunhofer’s solutions are advanced in optimisation and automation, but adoption by collections is limited due to cost and floor space requirements of the original CultLab3D conveyor system. In some instances also, technologies like these can be a rather ‘gold plated’ solution for natural history collections - they are perhaps more suited to outstanding objects or particular object types than to typical ‘mass’ digitisation of thousands of natural history specimens. The components, both hardware and photography/photogrammetry software, are off-the-shelf solutions, so Fraunhofer’s goal is to make use of the development of more affordable options. These solutions are informative for ICEDIG’s automation recommendations because of the flexibility of components, while also being fully integrated. The original system used a conveyor belt to minimise downtime during loading, but constraints around object-loading and floor space mean this may become obsolete.

The use of remotely operable microscopy was proposed by Wheeler et al. (2012) to engage taxon experts in targeting and annotating information about type specimens, as part of a 4-step approach to improving nomenclatural benchmarking. This approach built on ‘telemedicine’ or ‘telepathology’ practice, which uses technology to facilitate remote sample examination and diagnosis using either digitised images of e.g. microscope preparations; or cameras, including mobile phone cameras, attached to microscopes to transmit images over the internet.

For this initiative, a system was developed known as ROBOT(E) (Remotely Operable Benchmarker Of Types, first used in Entomology) (Fig. 10). ROBOT(E) was designed to allow taxonomists to examine, manipulate, and digitally photograph type specimens through a Web connection. Three such instruments were set up in the major insect collections in Washington, London, and Paris. The goal was to minimise costs and maximise reliability and simplicity by using as much off-the-shelf technology as feasible, including a Canon 7D camera that allowed auto-focus and through-the-sensor high resolution viewing. The specimen holder was designed to allow pinned specimens, held in a tight bundle of fine acrylic cable, to be rotated 360 degrees and ‘rolled’ 180 degrees to reveal the ventral surface. Software was written to control the ROBOT(E) using mouse, arrow and button commands to move the system’s motors and e.g. control focusing. Images were stored in a temporary folder from which they could be downloaded to any target folder.

Figure 10.  

The ROBOT(E) remotely operable digital imaging system from Wheeler et al. 2012 (©Erik Holsinger, Arizona State University).

Figure 11.  

ROBOCASE with robot arm and automated retrieval designed for closer inspection of items in an exhibition (© Bruns).

This system appears not to have been well-used in practice when the pilot was deployed. The authors noted that ‘This is not a general solution to type accessibility or a substitute for creating e-types. We propose a broader strategy of which this direct connection of expert and type is merely one component.’ - in practice, digitisation in 2D or 3D is often the best method to satisfy the needs of remote access to specimens and their data, provided sufficient views are taken to satisfy diagnostic and research uses in most cases. Remote solutions require collaboration between human operators at both ‘ends’, e.g. to place material into the system, but the nature of tasks as each end may not align readily e.g. processes are needed to alert an operator that the remote examination is complete, and this timing may not fit well with other tasks they are performing. A strong internet connection is of course also required for remote examination - this may now be more readily available in more locations but is not universal. Overall, it seems likely that technological solutions for remote specimen examination are more likely to be useful in research and collaboration involving small numbers of specimens, or with a time-critical driver such as medical diagnosis, than in the mass handling or digitisation of collections where digitisation can usually provide high quality remote access in a repeatable, low-cost way.

The Molecular Collections facility (MCf) at NHM London has a Hamilton Microlab Star Robotic Liquid Handling Workstation, which was purchased and installed in 2011 to function as part of the NHM's molecular pipeline (Fig. 12).

Figure 12.  

The Hamilton Microlab STAR automated liquid handling workstation (© Hamilton).

The robot performs pre-PCR (polymerase chain reaction) sample bulk analysis and processing (sample reformatting, quality control/assurance, DNA/RNA extractions and PCR reagent set up) prior to post-PCR processing (PCR, next gen sequencing) on its 'sister'/same model Microstar robot in the neighbouring sequencing facility. Workflow formats, consumables, scripts, service and maintenance contracts, including technical and apps support, are shared between the two robots for cost efficient high throughput molecular pipeline sample processing and workflow development.

The robot was first used in 2011⁠–⁠2012 to reformat and assess the quality of legacy molecular collections, e.g. transferring DNAs in solution into 2D barcoded cryo-vials for modern biobanking in the NHM's new cryofacility (-80°C ultracold and -196°C liquid nitrogen freezers). Since 2012 the robot has been used to perform bulk DNA extractions, mainly for automating researchers' preferred DNA extraction kit methods e.g. Qiagen's DNeasy kit. Additional robotic workstation deck accessories have been purchased since the robot's original installation, including integrated automatic 2D/1D cryo-vial and rack barcode readers, magnetic bead and vacuum extraction units. Further accessories can be added as required, plus 3rd party equipment integration, with interchangeable deck set up for flexibility and future proofing.

As described, one of the limiting factors in automated specimen handling is the absence of proven solutions to pick up natural history objects and place them in a specific way in a specific place, without damage. One of the key solutions to this is increase standardisation of containers or supports. The Robocase, developed by exhibition designer Bruns in collaboration with software maker Kiss the Frog (both from the Netherlands), is of interest for this reason (Fig. 11). It is designed specifically for handling museum objects on a standardised base ‘plate’, for display. The robot arm sits on rails that can move on two axes, and the arm itself has multiple degrees of freedom. The robot arm takes an object on a base from a shelf or drawer in a specifically designed cabinet, then transports the object to the viewer, usually placing the object on a turntable so that it can be viewed from all angles. This displaying behaviour depends on the object and the computer code behind it. The idea is that a visitor can use a screen to choose an item to view up close, then the robot retrieves, displays and returns the object. It is possible to have multiple robot arms in one Robocase. The whole process has been designed with reliability and safety of the objects in mind.

To achieve this, Bruns undertook extensive research on robot arms for museum objects. Initially, their idea was to develop a system that could handle the objects directly. The robot arm had to be able to work in slow and steady movements with high repeatability. Also, for this purpose a range of grippers were investigated and tested. Due to the variety of objects and the desire to also interact with cabinet doors and drawers, it proved too hard to find a reliable flexible gripper for handling the objects directly. Instead, a design was made of a hook end effector, combined with low pedestals/plates on which the objects were placed. These pedestals/bases have a slot in the side, which the hook slides into so that the pedestal with object can be moved. This hook was also designed to be able to interact with the cabinet doors and drawers.

This means that this robot arm does not directly handle the object. Also, the system does not have sensors to locate and precisely pick up objects, this is done by storing the location programmatically. For natural history collections the lesson here is that it is easier to get a robot to interact with standardised containers, than it is to develop a system that can handle a wide range of specimens. This solution also highlights the potential for robotics in engaging displays that respond to viewer interests, potentially offering access to a greater range of objects (including e.g. those that cannot be left out exposed to too much light) in a relatively small space. The contents in the Robocase can also be changed and updated regularly, within the relevant size constraints.

While this report makes a range of general points about automation technologies; their potential applications and limitations in our sector; and some case study examples; in practice any individual institution will need to consider their own specific context and challenges; and the costs and benefits of specific investments and solutions in relation to these. This section offers some key areas to consider.

McKinsey & Company identify five general factors relevant to the take up and success of automation in any sector (Chui et al. 2017). These are (1) technical feasibility; (2) costs of hardware & software; (3) labour supply & demand (cost & availability of human resources in relation to automation costs, and costs/availability of skilled resources to implement automation); (4) benefit beyond labour substitution (i.e. reasons to automate besides replacing human resources); and (5) the acceptability or otherwise of automation in the relevant sector (social, legal etc). These five factors are a useful framework for considering automation in the natural collections sector, both in general and in terms of the return on investment in specific cases. The five factors can be taken alongside three key stages for any particular institution and case - defining your problem or opportunity and your aims; understanding solution options; and understanding the business case including Return on Investment.

Technical feasibility

As is demonstrated in the variety of case studies in this document, many existing technologies have demonstrable or potential uses in the natural history collections sector. A more detailed mapping of existing technologies to workflows, including gap analysis, is recommended as one of the next steps for DiSSCo below. This further work is likely to help institutes hoping to understand both the opportunities that may be available to them, and the solutions and suppliers that may be relevant in their context. Discussion with suppliers, whether as part of a formal procurement process or as part of a research alliance for example, will be key to a more detailed and up to date understanding in relation to a particular challenge or set of requirements.

Costs - hardware, software and resource (labour)

It is of course critical to any investment case to understand the balance of costs and benefits/returns over time. Costs of some automation hardware such as robotic arms are falling, however costs will need to be looked at in detail in relation to any particular project and the most relevant technology solutions. Costs will include immediate purchase, but also maintenance over time, depreciation, and any ultimate costs of disposal. It remains the case that automation may come with relatively high ‘up front’ costs, for example, a robot arm may have a higher cost than employing a human operator over the first year; but this balance may shift when factors including working time, risk of injury, recruitment/replacement costs and so on are taken into account, and where costs are looked at over a longer period. It is also necessary to understand end to end workflows here, and any limitations such as human capacity to provide relevant material to the robotic solution - this may be a limiting factor for instance in relation to digitisation, which may prevent automation operating at maximum speed and efficiency. Even where the costs of automation outweigh the benefits in relation to a particular context and set of aims, it is worth bearing in mind that this may change - particularly where institutions are undertaking major changes to collections such as moves or mass digitisation it is worth revisiting the business case regularly, and trying to retain flexibility to future automation options.

Benefits - labour substitution and beyond

Automation is often perceived as a time and/or cost-saving technology, and this may indeed be key - for example institutes considering new storage facilities may aim to minimise their future resource costs by examining the scope for automated retrieval of storage units. There are, however, a variety of other aims to which automation may be relevant - for example managing key operational risks (health and safety; security); increasing speed and quality/consistency e.g. in performing digitisation; and ‘freeing up’ increasingly scarce human resources to focus on higher value, more rewarding activities that may increase wellbeing. Use of automation may be part of a wider business case, e.g. it may be that cost savings of new storage per square foot can only be realised if storage at height can be used with automated retrieval. It is important to fully establish the criteria against which you are assessing automated or other solutions, taking a rounded view of benefits over time.

Acceptability

Although, as shown in this paper, instances of physical automation are relatively widespread in collections, these instances may not be recognised widely (e.g. conveyor belts may not be talked about in terms of robotics or automation). Further pilots, as suggested below, and communication of their successes and lessons learned across the sector, are likely to be vital to building wider acceptance of the possibilities. Similarly, such pilots and communications are needed to build awareness in the industry of the needs and challenges in our sector, and to highlight this as an area of opportunity that is open to the possibility of automation. If pilots are isolated, and not communicated, it is likely to be a very long time before there is any systematic acceptance of automation within the sector, or any deeper engagement with potential suppliers.

Return on Investment

Overall return on investment (RoI) is the ratio of net returns to the cost of any investment (measured over a defined period of time). Usually, net returns are expressed in the form of profit, however in social investment for example monetary value is given to other forms of benefit or return. To understand the RoI of any investment in automation, the benefits set out above need to be calculable in financial terms. This is relatively straightforward for substitution benefits (i.e. relative savings compared to human time/cost), but can be more complex for wider benefits. Institutions may need to consider, for example, cost savings as a result of fewer worker days lost to injury; or the benefits to research of having access to greater volumes of digital specimen data more quickly. DiSSCo may be able to provide templates and methods to help understand the money value of benefits consistently.

A key benefit of robots in industry is that they can manage many operations, particularly repetitive tasks and those involving objects or environments that pose risks to humans, with much lower chance of injury or damage than human operators. To understand a risk-based approach to assessing the benefits of automation for natural history collections, the Natural History Museum London (NHM) analysed their Health and Safety risk management system - the detailed outcomes and methodology are at Appendix 2.

Across discussion with the NHM Health and Safety team, and analysis of relevant risk categories and risk assessments, key risk areas fell under the broad categories of manual handling; broader environmental factors; and working alone. There is scope for automation to mitigate risks across these areas, although there are also some risk categories where the risk to robots is equivalent to or higher in severity than the risk to humans. In all cases, the usefulness of automation to mitigate risks depends on the desired outcomes (e.g. acceptable risk level) and on cost-benefit analysis relevant to the particular circumstances.

As discussed in the Case Studies section, it proved difficult to engage industry in this ICEDIG task in order to examine case studies; understand industry (including SME) needs, and develop specifications for research & development around robotics in relation to natural history collections.

The engineer that Picturae worked with during the development of their herbarium conveyor belt was one of the few who was available for extended contact. He indicated that during the development of the herbarium conveyor belt, a key factor was understanding the strong requirement for safe transport and automated recognition of the specimens, which for example meant that vibration was minimised. Clear requirements are central to effective solutions tailored to our sector, across smaller or larger industrial partners and subject matter experts. Industry can’t be expected to come up with solutions for problems that are specific for natural (and cultural) history collections - only through clear requests, and demarcated problems that need to be solved, can they attempt to contribute. One of the issues is that the term ‘biodiversity collections’ or similar is largely unknown to this sector, and is very unlikely to be recognised spontaneously as a field of opportunity.

A combination of clear requirements and the right financial framework and incentives is likely to be critical. A previous example during which natural history collections similarly failed to gain sufficient engagement from technologists was the Beyond the Box competition, by the American Institute of Biological Sciences and the National Science Foundation (2014-2015). This offered an award of $1 million (USD) to the person or team who created a technology that increased the speed and accuracy of digitisation of a drawer of insect specimens and their associated data (National Science Foundation 2014). After the deadline, no submissions were received which met the requirements, resulting in no award made. While this competition was backed with a large sum of prize money to attract competitors, this fund was specifically to further develop and to market the solution, putting off some not-for-profit entrants, while the wider conditions and rules of the competition were quite prescriptive. When seeking technical collaborators it is important to allow means for partners to become familiar with the requirements and limitations of specific natural history collections. This competition laid out strict levels of achievement and a set of abstracted test material (PDF, with dimension and location details of drawers, unit trays and specimens); giving a limited range of flexibility for the potential contestants. Because roboticised solutions for most heritage collections are high in complexity, more interaction is needed to establish a suitable solution. In any future competition, a better outcome may be achieved by defining the outcome and requirements in principle, rather than the approach. It may be helpful to consider examples of competitions in other sectors, such as the Amazon Picking Challenge.

Many businesses in the automation industry have a large enough market in various sectors (manufacture, logistics, food production). In comparison, the natural history and heritage sector is relatively small and complex. Developers need an incentive to focus on the heritage collection market, which often translates into making funds available for development (subject to relevant procurement regulations). Funds may come directly from institutions, or from e.g. government innovation funding in some countries. The right combination of funding and clarity of requirements and challenges is key to future work with the robotics and automation industry.

It has not yet been decided whether DiSSCo will own or operate centralised digitsation facilities, however it has been recommended (Hardisty et al. 2020 - also quoted below) that DiSSCo should consider Centres of Excellence in particular services, offering structured leadership and best practice, and 'should consider building an experimental facility (DiSSCo Centre of Excellence) for an ‘out-of-town’ fully automated, industrial scale specimen storage and digitization facility'.

Prior to such a facility being possible, further work and pilot projects are needed to better understand the technologies available and to explore and demonstrate their successful application to natural history collections. This is recognised in the recommendation that DiSSCo should 'work towards a future automated end-to-end digitization solution, development should focus on independent components (including storage and retrieval, transport, object picking, and imaging) which can be connected in the future' and develop a series of pilot projects.

In the first instance, the analysis in this paper should be expanded to a formal identification of relevant workflow stages; whether solutions are already marketed that could address these; and where there are specific challenges and gaps. Pilot projects can then be targetted to test solutions in practice, and to develop solutions to fill gaps. It's likely that this will require a 'mixed economy' approach, including some or all of:

• identification and incentivisation of consortium members who may have opportunities to pilot physical automation approaches
• innovation competitions targetting R&D challenges
• partnering with industry to bid for innovation funding and work bilaterally on R&D
• customer-supplier projects to test particular technologies

As set out above, the right combination of clear requirements/problem statements, coupled with incentives for subject matter experts to engage, will be critical. DiSSCo coordination of potentially disparate projects, perhaps through a Centre of Excellence, can support the process of making specifications and requirements clear, and can provide a sufficiently substantial platform to interest commercial suppliers. DiSSCo should also assure project alignment to workflow steps, sharing the successes and lessons learned from demonstrator projects to inform the wider community. Templates and/or consistent methodologies for valuing benefits in financial terms would also support understanding of return on investment in automation, and also in other kinds of investment such as wider investment in digitisation - DiSSCo may be able to access the support of economists to develop this area, which may not be available to individual institutions.

These steps could then lead towards the creation of a fully automated DiSSCO facility if desired.

As part of assessing the potential for robotics, automated warehousing technologies and assistive technologies in relation to storing, handling and digitising natural history collections, we investigated key categories of risk in the Natural History Museum, London’s (NHM) risk management system. Sections i - iii below cover discussion, findings and suggestions for how these could be applied. The remainder of the Appendix is a methodological account of the system review.

Across discussion with the Museum's Health and Safety team, and analysis of relevant risk categories and risk assessments, key risk areas fell under the broad categories of manual handling; broader environmental factors; and working alone. There is scope for automation to mitigate risks across these areas, although there are also some risk categories where the risk to robots is equivalent to or higher in severity than the risk to humans. In all cases, the usefulness of automation to mitigate risks depends on the desired outcomes (e.g. acceptable risk level) and on cost-benefit analysis relevant to the particular circumstances.