Research Ideas and Outcomes : Research Article
Print
Research Article
Research and development in robotics with potential to automate handling of biological collections
expand article infoHelen Hardy, Myriam van Walsum§, Laurence Livermore, Stephanie Walton
‡ The Natural History Museum, London, United Kingdom
§ Naturalis Biodiversity Center, Leiden, Netherlands
Open Access

Abstract

This report investigates the current state of physical (mechanical) robotics, automated warehousing approaches and assistive technologies in relation to the storage, handling and processing (particularly digitisation) of natural history collections.

Robotics can sound futuristic, however we provide case studies that show many and growing examples of physical automation in the natural history and cultural heritage sectors, including barcodes and conveyor belts for digitisation; robots that handle multiple vials for molecular and genetic work; robots for use in in display or exhibition contexts; and automated warehousing of library collections. We provide a non-exhaustive example of an end to end workflow of storage, retrieval and processing and discuss aspects of the tools and challenges relevant to these stages. The Distributed System of Scientific Collections (DiSSCo), a new Research Infrastructure for natural science collections, should build on this, leading a future programme of pilots that develop understanding of independent stages, and can be connected to make progress towards end-to-end solutions.

Robots, or automated systems, excel at repetitive tasks, and are developing rapidly to be able to handle more complex object types, at lower cost. High volume, high variety of objects, and considerations such as fragility are not unique to the natural history sector - they apply for example to major retail operations - however natural history collections do offer some of the more extreme examples of these challenges, and in particular are not replaceable. Increased consistency of storage units is likely to be a critical factor in enabling automated handling in future, as well as looking at automation possibilities when new collections storage spaces are developed and built. Engagement with industry and subject matter experts has been patchy and again we recommend that DiSSCo help to ensure a joined up engagement with the right incentives in place, and with clear communication of requirements and challenges for shared R&D.

When examining return on investment for particular automation, collections-holding institutions need to consider not only time and cost of automation compared to human labour, but wider factors including: health and safety such as physical environment and repetitve strain injury; security; quality and consistency of outputs; degree of criticality in response times (e.g. if digitising on demand); effective use of spaces; and freeing up staff to conduct other tasks.

Purely software-based automation is outside the scope of this report, but is also in increasing use and has enormous potential, for example to transform the extraction of label and specimen data at scale from images. The challenges of managing and digitising collections at scale under DiSSCo are likely to require a combination of hardware and software automation approaches.

Keywords

Robotics, Automation, Warehousing, Digitisation, Natural History Collections, Natural Science Collections, Research Infrastructure, Health and Safety

1. Introduction

Global natural history collections are estimated to contain 2 to 3 billion specimens worldwide and 1 to 2 billion in Europe, of which 3% (at best) were accessible through GBIF in 2010 (Ariño 2010). Rough calculations were made of time and cost for imaging and databasing the global natural history collections, resulting in the staggering numbers of €150 billion and 1,500 person years (Blagoderov et al. 2012). Manual handling of specimens is the largest contributor to the high cost and time. Although many workflows are becoming more efficient as they are refined, imaging and databasing at sufficient speed and scale remain an enormous hurdle for the sector. Introducing both software and robotic automation has potential to decrease processing time and costs, and alleviate problems with regard to health and safety in both digitisation and the wider storage, retrieval and handling of specimens. The goal for this report was to produce requirements and recommendations in relation to physical robotics and automation that can support practical consideration and further research and development (R&D) and pilots. This has proven difficult owing to lack of industry engagement and the low level of automation maturity in our sector to date, however this report offers discussion and suggestions for collections considering the use of robotics and automation.

The topic of robotics and automated warehousing is very broad. While we need to look forward to relevant developments in the foreseeable future, we need to bear in mind that these can be hard to predict - perhaps the most important element going forward will be to remain flexible and open to these opportunities as they continue to develop and as their costs continue to fall.

This paper was written as Deliverable 3.8 (Walsum 2020) of the ICEDIG Project. This enhanced final version has been revised to associate the case studies with workflow stages; add to the section about investment analysis; and clarify the next steps for the Distributed System of Scientific Collections project (DiSSCo).

1.1 Definitions and scope

For the purposes of this report, ‘robotics’ and ‘robot’ refer to physical (mechanical) constructions with electronic and software components, designed to replicate certain human actions with a degree of autonomy.

Robotics is a form of automation - technologies that reduce the need for human intervention in a vast range of processes. Factory or warehouse automation often uses machines to perform highly repetitive and standardised tasks, sometimes boringly simple, sometimes highly complex, with processes for identifying and handling exceptions e.g. by alerting a human operative. Due to their use of sensors to interact with physical objects, and often to being reprogrammable, robots are often associated with greater versatility than other automated machines. For this report, we use ‘automated warehousing’ and ‘automation’ to refer to the range of technologies that allow mass processing or the management of a large environment, for instance retrieval of collections objects from a warehouse (often using mechanical components that do not resemble humans at all); and ‘robotics’ to refer to more ‘human’ processes such as the handling of an individual specimen (e.g. by a robot ‘arm’); however, these are not hard and fast distinctions - for example a robot arm is usually capable of more than human movements e.g. greater rotation, and robots may be part of the automation solutions in a bulk processing environment.

In addition to robotics and automated warehousing, this report touches briefly on assistive technologies, particularly those such as ‘exosuits’ that are designed to assist with, and reduce risk from, handling or lifting large, heavy and awkward objects.

The terms robotics, robot and automation are also now frequently applied to process automation using software, without any mechanical or physical components. Software automation is outside the scope of this paper, as it is addressed in related reports including the ICEDIG report on automated text digitisation (Owen et al. 2020), and the SYNTHESYS+ Specimen Data Refinery (Walton et al. 2020). Software automation is already in use across many aspects of digitisation, including image processing, batch quality control, barcode detection and image segmentation (Allan et al. 2019, Hudson et al. 2015).

2. Landscape analysis

This section provides a general discussion of the strengths and limitations of robotics and automation in relation to natural history collections handling, storage and digitisation; a summary of key hardware; and discussion of how collections might assess potential solutions and the business case for using automation in their particular circumstances.

2.1 Process components for retrieval, digitisation and imaging

At this time, to the best knowledge of the authors no natural history collections has implemented a fully automated solution from storage/retrieval to imaging (or other uses) and back to storage, nor has such a solution been described or analysed. By developing independent components, however, which can be connected in the future and build on existing practice, progress can be made towards an end-to-end solution (see Fig. 1 and Table 1 below).

Table 1.

Possible components of an automated system for storing and imaging natural history collections, and indicative limitations to be solved - this is illustrative not exhaustive and it is likely that there are more potential solutions and limitations. Components correspond to Fig. 1.

Step Potential automation solution Dependencies/limitations
1. Locate in storage Physical location known in database Ideally every object needs to be in database, to programme physical retrieval
2. Retrieve from storage location

1. Individual object picking robot

2. Retrieval of whole drawer or shelf for subsequent individual object picking by a person at a central location

Ability to operate the shelving units, and accommodate relevant objects.
3. Transport Mini self-driving robot Needs to be safe for humans and collections. Needs to navigate existing space.
4. Pre-imaging processing Object handling robot Handling of specimens and labels.
5. Positioning of object for imaging Object handling robot Handling of specimens and labels.
6. Positioning of camera for imaging Automated imaging station Safety of objects
7. Post-imaging processing Object handling robot Handling of specimens and labels
8. Return to storage Mini self-driving robot See steps 2. retrieval and 3. transport
9. Return to storage location

1. Object picking robot

2. Return whole shelf/drawer for human handling at a central point

Ability to operate the shelving units and accommodate relevant objects
Figure 1.  

Possible end-to-end process of storing and imaging natural history objects for automation.

At its simplest, automation can refer to storage that is controlled electronically - for example where we have remote control of environmental conditions such as temperature and humidity that could be described as automation. The BBC archive in Perivale uses automated compactors in their master film store - this is a cold store for flammable cellulose nitrate film, and automation allows for the position of the compactors to be standardised every night to ensure equal cooling throughout*1. This solution is not used in their other archive vaults, however, because these are subject to greater use and there is a high risk of system problems and breakdown if dust or debris get into the runners.

Automated warehousing with automated or partly-automated retrieval is more complex - this requires every relevant object to be identifiable, ideally with the physical location in a database. This can be at the object level, or per systematic category (based on taxonomy, geography and other used categories in NH collections) at either shelf or shelving unit level. Machine readable barcodes (or other solutions such as radio-frequency identification - RFIDs) at object and/or storage container level would increase the effectiveness of such a system.

Due to the great variety of natural history collections, multiple automated imaging systems are likely to be needed. In this case imaging automation of natural history collections needs to be split along more than taxonomic lines alone. In assessing digitisation costs for outsourcing, the Smithsonian’s National Museum of Natural History considers approximate object size; volume/dimensionality (is it flat or not); quantity of objects to be processed; and degree of complexity in the handling required e.g. whether objects can come straight out of storage to imaging, or require e.g. conservation or curatorial interventions*2. These are all likely to be equally relevant criteria in considering the possible application of automation to the workflow stages above, as is preservation type.

2.2 Robotic object handling - overview

"[...] it’s worth remembering that nothing stumps a robot quite like a bag of oranges. They just can’t deal with it. The bag moves in too many weird ways, there are no obvious bits to grab hold of, and if you squeeze too hard you end up with orange juice instead." - Vincent 2018

Industrial uses of robots and automation highlight a number of key strengths and limitations that apply across sectors (The Manufacturer 2018, Faya 2013, Vincent 2018). Robots are good at repetitive tasks - while they may wear out, they can typically carry out repetitive tasks numerous times, at high speed, without breaks or risk of injury. Many of the steps in Table 1 involve repetitive elements, such as taking containers in and out or pressing a button to take an image.

In relation to handling of collections, however, the quote above is highly relevant. Natural history collections can be a challenge for robots because there is very little that is standardised between collections (within or between institutes) and even within one type of collection. The specimens are fragile, don’t necessarily stay put in one place on a flat surface, and can also be soft and limp.

For example, the picking of pinned insects by the pin is relatively easy for humans, but a roboticised solution is not yet available which can ensure the integrity of the specimen. In this example with pinned insects in a drawer, the robot needs to identify a specimen, locate its pin, grasp it at a suitable point (clear of specimen and labels) with the correct amount of pressure, lift and move carefully. Then a suitable location for placing it needs to be identified (empty, suitable surface) and place it with the pin still vertical with the correct amount of pressure. Experts from Eindhoven University of Technology commented that identification of a single specimen is still a challenge, as long as the specimens aren’t regularly spaced out in a grid - overlap of specimens would be an issue for robotic identification (e.g. butterfly wings). Subsequent manipulations, such as removal of labels or placing in a specific orientation would be even harder.

Because these challenges exist in other industries, however, progress in robotic development is swift and robotic handling is increasingly becoming more sophisticated, for instance using computer vision with sensors as input, and a variety of physical handling solutions (see ‘grippers’ below).

Another factor to be addressed in relation to specimen handling is stakeholders’ opinion. Institute management and collections staff need to support the notion of automation and robotisation of imaging and storage workflows, a substantial change. Systems will have to be trialled and demonstrated before risks are likely to be seen as acceptable. However, the head of a major R&D institute developing an automated 3D digitisation system for heritage and industry has noticed that many collection holders are becoming more comfortable with the idea: for example digitisation technologies such as conveyor belts are now commonplace (see case study 3.4).

While direct robotic handling of natural history specimens may not currently offer an acceptable balance of risks and costs against benefits, there is greater potential in relation to indirect handling, i.e. handling storage containers/units such as drawers or jars. At present, standardisation of storage is often limited, though not as limited as standardisation of specimens. Historic collections and object variety often mean a wide variety of storage, e.g. jars of all sizes, shapes, sealants etc. There may also be limitations such as jammed drawers and uneven navigation routes (although these also impact human workers). An alternative is to increase standardisation with robotic handling in mind - this involves higher cost and effort before adoption of automation, but may be suitable when collections moves and rehousing are planned.

2.3 Key hardware

Hardware for automation is developing rapidly. Some specific examples of relevant technologies are included in detail in the case studies below. Some of the most relevant current hardware solutions include robotic ‘arms’ and grippers; conveyor belts; and exosuits. The cost, precision, reliability and ease of implementation of these hardware components is key to the success of automation. In general, the abilities of this hardware continue to increase (e.g. precision of handling) and their costs to fall.

Robot arms

Robot arms are currently available off-the-shelf at various performance levels and prices (Fig. 2). For handling small to medium specimens or storage units, payload is of lesser importance. Precision, safety features, degrees of freedom and ease of programmatic control are probably the key elements. However, full account should be taken of the expected load e.g. a professional camera with macro lens, macro extension tubes and lights, as well as the required reach (depending on the size of the specimen the arm needs to move around).

Figure 2.  

Example of robot arm with a camera. The object is placed on a surface or turntable, and the robot does not physically interact with the object (© CultLab3D).

Robot arm options and features

A relatively new trend in robotics is the development of collaborative robots, or co-bots. Co-bots, which are designed to work safely with humans, are becoming more commonplace in industrial manufacturing applications. The top four major industrial robot manufacturers, (ABB, Bosch, Fanuc and Kuka) are following start-up companies such as Rethink Robots and Universal Robots in development of co-bots. Collaborative robots use safety-rated sensors, allowing an operator to work in the same space as the robot and share tasks without fear of injury. As prices for these safety-rated sensors drop, the trend is reflected in the cost of co-bots.

The price of industrial robots has dropped more than 25 percent since 2014, and is expected to drop an additional 22 percent by 2025. Today, an industrial robotic arm can cost anywhere from $25,000 to $400,000. When looking at the cost of an industrial robot system, other peripherals like controllers, a teach pendant, end of arm tooling (EOAT) and software must be considered as well. Once these application-specific peripherals are added, total system costs could double. In some cases, purchasing a used or refurbished robot or system can reduce the price as much as 50 percent. Robots for schools, universities and other non-industrial applications can be found for $1,000 or less, but these robot arms are not suited for industrial applications.

The largest factors involved in determining cost are robot size (reach), number of axes, application, end of arm tooling (EOAT) and safety components. In general, the bigger the reach and the larger the payload, the more the robot will cost. However, application-specific peripherals and safety components, such as collision sensors and safety cages, also contribute to the price of an industrial robot system.” - Thayer 2017

Safety for objects and humans

Traditional robot arms often are surrounded by shields on multiple sides, to ensure the safety of humans in the vicinity. If human loading and unloading is needed, this is often achieved by having the “safety cage” on three sides and a sensor (e.g. light screen or motion sensor) on the fourth side. Further, they can be equipped with collision sensors. Collaborative robots are equipped with safety features such as collision detection or force feedback. When unexpected force is detected, the arm stops operating. Specific models will freeze their position at this point, others will release all joints causing the arm to “collapse”. Other safety features include a programmatic bounding box for operation to define where the arm is not allowed to come and emergency switches. Collaborative robots are also safer due to the lower speeds at which they move.

Grippers

A robot arm can be equipped with a range of end effectors (also called end of arm tooling, EOAT) of which grippers are a likely tool for handling specimens, containers or equipment such as cameras. In industry spray guns and power tools are commonly used, as well as various specially designed end effectors. Grippers can for example be two opposable “fingers”, vacuum and suction pads, and computer-controlled magnets (Fig. 3). Replicating the dexterity and flexibility of human hands is a known challenge, especially when multiple ‘hands’ work together. To pick up specimens, grippers either need to sense / be programmed for factors such as the weight, shape and slipperiness of objects, to apply the correct amount of force and prevent slippage and crushing, or need to deploy types of grip that by nature are less likely to cause damage. The former offers greater precision, however the latter appears to be more straightforward to achieve.

Figure 3.  

Robot arm with vacuum gripper, potentially suitable for lifting containers of NH specimens (© Universal Robots).

The field of soft robotics is potentially of great interest in this context, as it deals with developing robots that are soft, flexible and compliant, as opposed to rigid. This allows the grippers to interact with objects that are pliable and of varying shapes. Part of the research is in fact mimicry, copying nature for industrial purposes. An example of mimicry in soft robotics is mGrip designed by Soft Robotics Inc (Fig. 4). This gripper can interact with objects of high shape variability without need for computer vision systems or pre-programming (at least for grasping, less for sensing the context of objects around it), through controlled pressure of the air in the tentacles of the gripper, they form around the object and grasp it. Even where gripping can be achieved without issue, however, release may still cause issues unless this can also be achieved with great precision (i.e. objects are put precisely on a surface, not 'dropped' slightly from above or pushed down).

Figure 4.  

mGrip, the soft robotic modular tooling system - potentially suitable for picking up pressure-sensitive and small objects (© Soft Robotics).

Another specific type of gripper capable of dealing with a high variety of object shapes involves a flexible membrane filled with granular material, called the granular jamming gripper, developed by Empire Robotics under the name VERSABALL (Fig. 5) (Amend et al. 2016). This specific design allowed the gripper to form gently around the object (unjammed, fluid-like state), after which the air is evacuated causing the granular material to become jammed in that specific shape, gripping the object softly. Release of the object is achieved by releasing pressurised air into the gripper. However, in 2016 Empire Robotics closed its doors due to not having been able to fully commercialise the product (Saunders 2017).

Figure 5.  

VERRSABALL ‘jamming’ gripper capable of pressure-sensitivity and picking up irregularly shaped objects (© Empire Robotics.

Exosuits

Exoskeleton technology (Fig. 6) comes in various forms, targeting various movements, body parts and actions. Some are powered, for the use of sensors and actuators, while passive exoskeletons can provide weight redistribution and shock dampening. These and other assistive technologies have often been developed for medical or military uses, but are coming to be used more in industry to support human operatives and avoid injuries such as back injuries and repetitive strain injuries. We are not aware of any current uses of these technologies in our sector, however they may be relevant to some of the Health and Safety risk management elements discussed above, particularly if warehouse-type environments become more common for collections storage. In effect, they are an extension of the principles of current Personal Protective Equipment such as masks and helmets, and of technologies such as forklift trucks that help with lifting.

Figure 6.  

An example of a biomechanical exoskeleton (© Yuichiro C. Katsumoto / Wikimedia Commons / CC-BY 2.0).

3. Case studies

The authors identified case studies in consultation with ICEDIG project members and with industry and other subject matter experts. Various businesses in the relevant industry sectors were approached, as well as technical universities and their spin-off companies (more than 50 potential contacts in total, selected in relation to their expertise). Unfortunately, very little response was received. This in itself is valuable and an opportunity to learn.

The tasks of digitisation, handling and warehousing were the main focus for contact selection. For example, some of the companies approached had experience in automated handling of fragile, organically-shaped objects in the agricultural sector (eggs, flowers, live chicks). Among companies approached were both users and developers, with the intent to gain diverse feedback. In practice, however, many of those approached by phone, email or contact forms either did not reply or did not identify the correct expert to respond. For example, a specialist robotics department of a technical university, linking research and industry, was contacted for collaboration but contact with an expert could not be established.

Steps to address industry collaboration are addressed in the Discussion, and it is likely that appropriate funding models will be key. The case studies below do cover a number of commercial products, discussed with their suppliers, a well as a general view of some e-commerce warehousing solutions drawn from desk-based research.

In relation to Fig. 1 and Table 1 above, case studies 1 and 2 look at storage and transport (including object retrieval); case studies 3, 4 and 5 consider different types of imaging; and case studies 6 and 7 look at additional processes outside the simplified workflow in the figure.

3.1 Automated Warehousing at the British Library

Automated solutions similar to those used in commercial warehousing can be used for the large-scale handling and retrieval of natural history objects in storage, improving retrieval times from storage, space efficiency, and allowing for e.g. low-oxygen environments which are beneficial for conservation of some collections but can be dangerous to humans. Library collections are close to natural history collections: they serve to make object based research available; most of the objects spend most of their lifetime in storage but need to be retrievable in ways that are often not predictable; and object condition depends on controlled climate (temperature, humidity, oxygen levels, and pest control). Library books have somewhat fewer variables than natural history collections do, although similarly for automation they rely on standardised storage, with e.g. books in crates, boxes or ‘bins’; and newspapers or other larger formats secured on pallets.

In recent years, some large libraries, on a national level or at large universities, are building advanced warehouses with storage robots. Some examples are the Glucksman Library at the University of Limerick, Macquarie University Library, North Carolina State University Library, University of Missouri Kansas City Library and University of Chicago Library; these systems are developed by Dematic. In most cases, the books are stored in standardised steel bins, which the robot can manipulate when a book request is made through the catalogue. The physical storage location of the book is recorded in the database. The organisation of the objects can remain as it was before (e.g. by topic, author, year, language), but this is not necessary - in the event of complete database malfunction with no backup data available all books would be untraceable, but with a backup protocol in place this should never occur. After the request is made, the robot retrieves the bin in which the item is located. The whole bin is then made accessible to a librarian, who takes out the specific requested item and processes it for pick up by the requester. The rest of the bin is returned to storage. Return is very much the same: either a random bin or the dedicated bin is retrieved by the robot for the librarian to put the book into, after which the bin is automatically reshelved.

Usually the parts of the collections with highest use are still available on the shelf for the readers and only the “reserve collections” or less used collections are stored in the automated warehouse.

The British Library started their project for a new building and automated storage system in 2004-2006. A second building and automated storage system with shelves going 30m high went live in 2015 for the newspaper collection. The company that built the systems for them is TGW Group. After trial and error, they discovered that, in the case of newspapers, bundling and securing a set of papers with a canvas belt was the simplest solution for librarian and robot. In the first automated warehouse there were initial problems with box size and distortion of plastic boxes - these issues have now been resolved.

Items are accessed and requested through relevant systems. A user, e.g. in the reading room in London, can request an item through the digital catalogue. The request is then sent to the system in Boston Spa (Yorkshire, 280km north of London), where a robot retrieves the relevant container, transporting it to workers who take out the specific requested item. The rest of the box is returned by the robot to its place, while the requested item is set on a trolley for transport. Trolleys are loaded into lorries with daily (or more) deliveries to the reading room in London where the user has access. After use, items are sent back by the same route; the whole outgoing process can be done within 24 hours of the request being made.

While the storage area has limited access for humans (given height and restricted oxygen environment), books remain organised by subject and author in the boxes, although the robotic system keeps more frequently-requested boxes closer to the picking station. Maintenance of the hardware is not done by library staff: like cleaning services, it is completely outsourced to specialised engineers on call. Support is reduced out of hours (e.g. overnight) owing to cost - systems are still used at these times but the functions used are reduced to those where the BL operators can usually resolve any issues themselves. The newspaper warehouses contain 70km of storage. To make this work every item needs to have its own identifier and physical storage needs to be completely in sync with the database and software; this initially proved to be a challenge. Over the years, no objects have been dropped, however there have been a few cases of objects or bins becoming stuck. After the initial phase, they now experience very little issues. With the number of overdue outstanding request as indicator, there are almost never problems.

The workflow of the users has changed very little because these items were already only available on request, with 24 hour processing time. The work of the librarians has changed: to keep up with the demand they work in shifts and due to automation the work can be done with less training. The newspaper collections were not in the best condition before the move to the new automated warehouse; now their condition is stable. Because the storage area is off-limits to humans, a much lower oxygen and temperature level can be maintained (oxygen comparable to Mount Everest) which is the main source of deterioration of the material. The low oxygen level also reduces the risk of fire. Another major factor is cost: in London the cost of a meter of shelving space was about 40GBP, in Boston Spa 4GBP. This is also due to the use of automation to enable storage at a much greater height than would be usable by humans without assistive technologies, so that each meter of floor space accomodates more items.

Advantages of this solution are space efficiency, lower housing cost from off-site storage, improved specimen conditions due to climate control, and reduced fire hazards. A major part of the current workflow of natural history curators and researchers is to browse the collections, going through boxes and shelves, enabling associative specimen selection. This can be a specimen sitting next to the initial target, from the same location, same collector, same label format etc. An automated warehouse would not allow this workflow physically. However, this can be achieved through software solutions. Imaging per shelf can substitute initial shelf/box browsing; image recognition of labels, transcribed metadata such as collector and location and data linkage allows for queries and serendipitous discovery across a range of variables. A tiled view with associated material of the specific request based on taxonomy, location, collector would also recreate this workflow.

Most current library warehousing robots still require human picking from bins/containers. This offers a useful lesson for natural history collections in relation to standardising storage units where possible - while the variety of items and storage types is greater than in libraries, the volume of many collections could still allow for extensive standardisation of e.g. drawer or jar size, shape and other features. The key benefits of automation at scale for specimen handling involve the complete refitting or new development of storage spaces, so can only be considered e.g. when moving collections at scale to new or refurbished facilities, however pilots such as the use of a small robot to fetch collections of one type within one storage area may be possible. There is a potential to collaborate with other collection-based institutes: either other natural history museums in the larger region, cultural history collections (museums, archaeological departments) and libraries. This would reduce the cost of building new facilities and the transportation to the main site.

3.2 E-commerce warehousing solutions

While use of robots in collections storage and retrieval is currently rare, robots and automation have been and continue to be rapidly adopted and developed for e-commerce logistics and order picking. Companies such as DHL, Amazon, large grocery stores (e.g. Ocado in UK) are adapting their warehousing process to extensive automation. The process known as order (or item) picking is closest to the needs of natural history collections.

One variant is where the entire cabinet/shelving system containing the requested object is picked up by a transporter robot, and transported to a human for object picking (for example at Amazon fulfillment centres). An alternative is a grid system with crates or racks of goods below and autonomous robots on top, navigating the grid to pick up an object or packet e.g. using a suction cup, storing the item in the cart and dropping several objects at a packing station for further shipping, as used by Ocado (Vincent 2018).

These examples demonstrate the massive scale at which roboticised warehousing operates, but also how humans and robots complement one another, including for many object handling tasks. These users of automation are also able to accept, and factor into costing, a certain amount of damage or wastage, while collections cannot accept any margin for this as objects are not replaceable.

A report by DHL (Bonkenburg 2016), examining the potential for robotics and automation in logistics, raised concerns that any ‘logistics robot would need to handle a wide array of different parts in an infinite number of combinations. It would help if the robot could see, move, and react to its environment.’ This echoes some of the barriers we believe may exist in our sector and mention in this report - however automation is now increasingly widespread and vital to the logistics industry so this is a useful lesson in how quickly limitations are being overcome.

3.3 Automation of herbarium sheet digitisation

Please note that a full description of these solutions can be found in the ICEDIG report on digitisation of herbarium sheets (Guiraud et al. 2019).

Herbarium collections provide the best examples of automated mass digitisation incorporating both physical and software components, as there are currently two of these solutions by Digitarium (now Bioshare by Sertifer Consulting Ltd.) (Digitarium 2017, Bioshare Digitization 2017) and Picturae (Picturae 2017). These systems still require handling by human operators to load and unload, to apply barcodes, and to detect special specimens such as types and specimens requiring restoration. Specimens are loaded onto a conveyor belt system, which automatically moves a series of specimens through an imaging station. At the end of the line a human operator unloads the specimens. Image adjustments (including colour correction, potential sharpening, cropping, etc.) and file naming through barcode detection are done automatically. After quality assessment the batch of specimens is ready to be returned to storage. To achieve the highest speed it is important that as few as possible human actions are required: any positioning and adjustment of labels slows the process down. These systems have been tested as being 4 to 5 times faster than one at a time scanning of herbarium sheets - however a key barrier to achieving these imaging speeds is the ability to have specimens ready for this process in sufficient volume (e.g. pre-curation). This issue is associated with storage optimisation, which in some instances could greatly increase the speed of retrieval and replacement.

The human component in this process is the loading and unloading of specimens and detecting special cases. Sometimes a specimen has associated material in an envelope, which can be opened into trays to be imaged as well. It is essential that any loose material and labels stay together. The fragility and special attention needed for loose material currently mean that human involvement cannot be fully replaced by automation.

The conveyor belt component of these solutions could be used to automate digitisation workflows of other collections. It has indeed already been applied to pinned insects (Tegelberg et al. 2014, Tegelberg et al. 2017). Herbarium sheets are more standardised and closer to two -dimensional (flat), making it easier to automate digitisation. Issues such as depth of field and positioning of an object taking into consideration stability, standardised anatomical, and diagnostic views are greater when objects have greater depth. Still, if only label images are needed (e.g. for transcription purposes, either by humans or through machine vision), or lower resolution specimen images are sufficient, then the current herbarium digitisation conveyor systems may be adapted for objects such as open boxes of bones, molluscs and other dry material. There may also be scope to image labels on jars from liquid collections, although this may come with increased risks to manage, if using electronics in proximity to liquids, especially flammable liquids.

Another important aspect of these workflows is that image correction, such as cropping, straightening and colour calibration, can be done automatically because there are a number variables such as shape, camera position and settings which are controlled by software to an appropriate level in this workflow.

In general, conveyor belts can be an efficiency boosting component when certain conditions are met. First, it takes a certain amount of time, effort and space to set up a system with a conveyor belt. The quantity of material needs to be large enough to balance the set up time/cost. Secondly, the diversity of the material needs to be within a set range. For example, this includes dimensions, views to be captured, acceptable levels of vibration and similar parameters. Further, the handling needs to be able to be broken up into independent actions at certain stations of the workflow. By simplifying the handling into separate repetitive actions (and reducing the number of actions where possible) the speed of the whole process can be improved. Finally, all the necessary preparations to standardise and prepare the collection for automated digitisation need to match the speed of the conveyor belt. This is the key barrier to optimising the benefits of automation through these systems. The experience of many digitisation projects (in both cultural heritage and natural history) shows that the amount of preparation work and the impact on the collections department is often underestimated. This is especially relevant for digitisation projects procuring a conveyor belt system and/or service through outsourcing, where it is not economic to stop and start the process.

3.4 Automation of 3D imaging – CultLab3D

The Fraunhofer Institute for Computer Graphics Research IGD (Darmstadt, Germany) is a research institute focussing on various fields in applied science, which has a Competence Center Cultural Heritage Digitization, including the research lab CultLab3D. They have developed multiple mass-digitisation 3D scanning solutions, that have been used for industry (e.g scanning of shoes) and cultural heritage (e.g. globe digitisation at Friedrich Schiller University) (Santos et al. 2017a, Santos et al. 2017b, Ritz et al. 2018). The developers understand the limitations of working with heritage objects and the need of curators for high quality and calibrated, repeatable results. For this case study, conversations were held with Pedro Santos, head of the Competence Center, as well as attendance of an open day with demonstrations & discussion at the Getty Research Institute in Los Angeles (USA) on 2019-07-15. A report of the open day is given in Appendix 1 of Walsum (2020).

The first solution from CultLab3D is a conveyor belt system equipped with cameras on two half-arcs with ring lights and glass turntables to allow imaging of the object’s underside (Fig. 7). Multiple objects can be loaded onto the conveyor belt, so that the imaging system doesn’t have downtime while a human operator loads the next object.

Figure 7.  

The CultLab3D conveyor belt and automated imaging system (© CultLab3D).

The newer solutions, named CultArm3D, are mounted on robot arms and can be either photogrammetry, laser or structured light based (Fig. 8. This allows flexibility in choosing the right solution based on cost, resolution, object surface type and colour accuracy requirements. The system can work with standard DSLR cameras, as well as professional high-resolution Phase One cameras. With the current camera and lens combination, objects with dimensions between 40×40×60cm (max 20kg) can be digitised, with a maximum resolution of 20 microns. When different lenses and a robot arm with longer reach are used, smaller and larger objects can be captured. Highly controlled focus stacking for small objects is an option. These robot arm systems can also be attached to the end of the conveyor belt system when insufficient data is detected in specific areas after initial alignment, to target these areas only.

Figure 8.  

The CultArm 3D automated imaging system (© CultLab3D).

Figure 9.  

Manual Handling risk assessment template extract.

The CultArm3D system is fitted with a glass turntable so that of many objects the underside can be sufficiently captured. Two positions have been pre-programmed to minimise refraction from the glass as well as to ensure that the arm does not hit the turntable. This does mean that objects with complex undersides (or objects with no definable undersides, which includes many non-man-made objects) will still need to be repositioned to properly capture them from all sides. Another scenario is objects that will not sit sufficiently stable by themselves and require some support. As long as there is sufficient contrast between the supporting material and the object, this can be removed automatically from images.

The capturing system is not currently compatible with Mac. The computer specifications for capture are very low: it can run on a lightweight laptop. However, the requirements for the processing stage are much greater and can be a bottleneck. A typical capture results in hundreds of images.

The safety of the object and of humans in the vicinity needs to be assured to make this robotic system viable. To do this, multiple layers of security measures are present. The robotic arm (Universal Robotics UR10) is designed to be safe for human workers in its vicinity. Due to its inbuilt collision detection, contact such as with a human causes the arm to freeze. The same happens during a power failure, where some other robotic arms might unlock their joints so that gravity pulls the components down. The arm is programmed with predefined zones that it avoids, this includes the turntable and some other areas. Finally, based on an initial scan, a cylindrical space is defined that the arm cannot enter, giving a safe margin around the object.

After calibration and besides loading objects, the capturing process does not need a human operator. The object is initially photographed against a lit background so that only the silhouette is captured, which some software can use to improve photogrammetric algorithms. The robot arm is an off-the-shelf product which Fraunhofer can upgrade at a later point, including on existing systems. While Fraunhofer’s code should work with a new model robotic arm made by the same manufacturer from the same product line, only minimal code changes would be needed for a different arm, as long as these have the same degrees of freedom. Focus stacking processing, and detection and discarding of unsharp areas in the photos are done through scripts. To optimise the quality of the output, the pipeline generates depth maps to be able to ignore poor data (out of focus) in the images. The processing pipeline is not locked to a specific photogrammetric software, and can use either various commercial packages, open source software or software developed by Fraunhofer, as long as it is scriptable. Fraunhofer has developed software to optimise the 3D output for various presentation goals including web optimisation (this includes UV unwrapping, retopology, normal maps, publication in web specific format).

Fraunhofer’s 3D scanning robot has been designed to fully automate the capture process, as well as capturing extra data to be able to fully automate the 3D processing workflow. Some human interaction may still be required during post-processing, especially for the more complex objects. Compared to the herbarium imaging solution, the system deals with added complexity because of the 3D nature of the objects and many more processing steps that the raw captures and generated models need to go through.

Fraunhofer’s solutions are advanced in optimisation and automation, but adoption by collections is limited due to cost and floor space requirements of the original CultLab3D conveyor system. In some instances also, technologies like these can be a rather ‘gold plated’ solution for natural history collections - they are perhaps more suited to outstanding objects or particular object types than to typical ‘mass’ digitisation of thousands of natural history specimens. The components, both hardware and photography/photogrammetry software, are off-the-shelf solutions, so Fraunhofer’s goal is to make use of the development of more affordable options. These solutions are informative for ICEDIG’s automation recommendations because of the flexibility of components, while also being fully integrated. The original system used a conveyor belt to minimise downtime during loading, but constraints around object-loading and floor space mean this may become obsolete.

3.5 Remote microscopy

The use of remotely operable microscopy was proposed by Wheeler et al. (2012) to engage taxon experts in targeting and annotating information about type specimens, as part of a 4-step approach to improving nomenclatural benchmarking. This approach built on ‘telemedicine’ or ‘telepathology’ practice, which uses technology to facilitate remote sample examination and diagnosis using either digitised images of e.g. microscope preparations; or cameras, including mobile phone cameras, attached to microscopes to transmit images over the internet.

For this initiative, a system was developed known as ROBOT(E) (Remotely Operable Benchmarker Of Types, first used in Entomology) (Fig. 10). ROBOT(E) was designed to allow taxonomists to examine, manipulate, and digitally photograph type specimens through a Web connection. Three such instruments were set up in the major insect collections in Washington, London, and Paris. The goal was to minimise costs and maximise reliability and simplicity by using as much off-the-shelf technology as feasible, including a Canon 7D camera that allowed auto-focus and through-the-sensor high resolution viewing. The specimen holder was designed to allow pinned specimens, held in a tight bundle of fine acrylic cable, to be rotated 360 degrees and ‘rolled’ 180 degrees to reveal the ventral surface. Software was written to control the ROBOT(E) using mouse, arrow and button commands to move the system’s motors and e.g. control focusing. Images were stored in a temporary folder from which they could be downloaded to any target folder.

Figure 10.  

The ROBOT(E) remotely operable digital imaging system from Wheeler et al. 2012 (©Erik Holsinger, Arizona State University).

Figure 11.  

ROBOCASE with robot arm and automated retrieval designed for closer inspection of items in an exhibition (© Bruns).

This system appears not to have been well-used in practice when the pilot was deployed. The authors noted that ‘This is not a general solution to type accessibility or a substitute for creating e-types. We propose a broader strategy of which this direct connection of expert and type is merely one component.’ - in practice, digitisation in 2D or 3D is often the best method to satisfy the needs of remote access to specimens and their data, provided sufficient views are taken to satisfy diagnostic and research uses in most cases. Remote solutions require collaboration between human operators at both ‘ends’, e.g. to place material into the system, but the nature of tasks as each end may not align readily e.g. processes are needed to alert an operator that the remote examination is complete, and this timing may not fit well with other tasks they are performing. A strong internet connection is of course also required for remote examination - this may now be more readily available in more locations but is not universal. Overall, it seems likely that technological solutions for remote specimen examination are more likely to be useful in research and collaboration involving small numbers of specimens, or with a time-critical driver such as medical diagnosis, than in the mass handling or digitisation of collections where digitisation can usually provide high quality remote access in a repeatable, low-cost way.

3.6 Robotic handling of liquid vials

The Molecular Collections facility (MCf) at NHM London has a Hamilton Microlab Star Robotic Liquid Handling Workstation, which was purchased and installed in 2011 to function as part of the NHM's molecular pipeline (Fig. 12).

Figure 12.  

The Hamilton Microlab STAR automated liquid handling workstation (© Hamilton).

The robot performs pre-PCR (polymerase chain reaction) sample bulk analysis and processing (sample reformatting, quality control/assurance, DNA/RNA extractions and PCR reagent set up) prior to post-PCR processing (PCR, next gen sequencing) on its 'sister'/same model Microstar robot in the neighbouring sequencing facility. Workflow formats, consumables, scripts, service and maintenance contracts, including technical and apps support, are shared between the two robots for cost efficient high throughput molecular pipeline sample processing and workflow development.

The robot was first used in 2011⁠–⁠2012 to reformat and assess the quality of legacy molecular collections, e.g. transferring DNAs in solution into 2D barcoded cryo-vials for modern biobanking in the NHM's new cryofacility (-80°C ultracold and -196°C liquid nitrogen freezers). Since 2012 the robot has been used to perform bulk DNA extractions, mainly for automating researchers' preferred DNA extraction kit methods e.g. Qiagen's DNeasy kit. Additional robotic workstation deck accessories have been purchased since the robot's original installation, including integrated automatic 2D/1D cryo-vial and rack barcode readers, magnetic bead and vacuum extraction units. Further accessories can be added as required, plus 3rd party equipment integration, with interchangeable deck set up for flexibility and future proofing.

3.7 Robot arm for exhibition

As described, one of the limiting factors in automated specimen handling is the absence of proven solutions to pick up natural history objects and place them in a specific way in a specific place, without damage. One of the key solutions to this is increase standardisation of containers or supports. The Robocase, developed by exhibition designer Bruns in collaboration with software maker Kiss the Frog (both from the Netherlands), is of interest for this reason (Fig. 11). It is designed specifically for handling museum objects on a standardised base ‘plate’, for display. The robot arm sits on rails that can move on two axes, and the arm itself has multiple degrees of freedom. The robot arm takes an object on a base from a shelf or drawer in a specifically designed cabinet, then transports the object to the viewer, usually placing the object on a turntable so that it can be viewed from all angles. This displaying behaviour depends on the object and the computer code behind it. The idea is that a visitor can use a screen to choose an item to view up close, then the robot retrieves, displays and returns the object. It is possible to have multiple robot arms in one Robocase. The whole process has been designed with reliability and safety of the objects in mind.

To achieve this, Bruns undertook extensive research on robot arms for museum objects. Initially, their idea was to develop a system that could handle the objects directly. The robot arm had to be able to work in slow and steady movements with high repeatability. Also, for this purpose a range of grippers were investigated and tested. Due to the variety of objects and the desire to also interact with cabinet doors and drawers, it proved too hard to find a reliable flexible gripper for handling the objects directly. Instead, a design was made of a hook end effector, combined with low pedestals/plates on which the objects were placed. These pedestals/bases have a slot in the side, which the hook slides into so that the pedestal with object can be moved. This hook was also designed to be able to interact with the cabinet doors and drawers.

This means that this robot arm does not directly handle the object. Also, the system does not have sensors to locate and precisely pick up objects, this is done by storing the location programmatically. For natural history collections the lesson here is that it is easier to get a robot to interact with standardised containers, than it is to develop a system that can handle a wide range of specimens. This solution also highlights the potential for robotics in engaging displays that respond to viewer interests, potentially offering access to a greater range of objects (including e.g. those that cannot be left out exposed to too much light) in a relatively small space. The contents in the Robocase can also be changed and updated regularly, within the relevant size constraints.

4. Discussion

In this section, we consider how institutes can examine the investment case and return on investment for robotics and automated warehousing solutions; how these solutions intersect with and in many cases can support Health and Safety risk management; some factors that might help better collaboration with suppliers / subject matter experts; and the next steps for DiSSCo to advance the sector's understanding and use of physical automation.

4.1 Analysing potential automated solutions - the investment case

While this report makes a range of general points about automation technologies; their potential applications and limitations in our sector; and some case study examples; in practice any individual institution will need to consider their own specific context and challenges; and the costs and benefits of specific investments and solutions in relation to these. This section offers some key areas to consider.

McKinsey & Company identify five general factors relevant to the take up and success of automation in any sector (Chui et al. 2017). These are (1) technical feasibility; (2) costs of hardware & software; (3) labour supply & demand (cost & availability of human resources in relation to automation costs, and costs/availability of skilled resources to implement automation); (4) benefit beyond labour substitution (i.e. reasons to automate besides replacing human resources); and (5) the acceptability or otherwise of automation in the relevant sector (social, legal etc). These five factors are a useful framework for considering automation in the natural collections sector, both in general and in terms of the return on investment in specific cases. The five factors can be taken alongside three key stages for any particular institution and case - defining your problem or opportunity and your aims; understanding solution options; and understanding the business case including Return on Investment.

Technical feasibility

As is demonstrated in the variety of case studies in this document, many existing technologies have demonstrable or potential uses in the natural history collections sector. A more detailed mapping of existing technologies to workflows, including gap analysis, is recommended as one of the next steps for DiSSCo below. This further work is likely to help institutes hoping to understand both the opportunities that may be available to them, and the solutions and suppliers that may be relevant in their context. Discussion with suppliers, whether as part of a formal procurement process or as part of a research alliance for example, will be key to a more detailed and up to date understanding in relation to a particular challenge or set of requirements.

Costs - hardware, software and resource (labour)

It is of course critical to any investment case to understand the balance of costs and benefits/returns over time. Costs of some automation hardware such as robotic arms are falling, however costs will need to be looked at in detail in relation to any particular project and the most relevant technology solutions. Costs will include immediate purchase, but also maintenance over time, depreciation, and any ultimate costs of disposal. It remains the case that automation may come with relatively high ‘up front’ costs, for example, a robot arm may have a higher cost than employing a human operator over the first year; but this balance may shift when factors including working time, risk of injury, recruitment/replacement costs and so on are taken into account, and where costs are looked at over a longer period. It is also necessary to understand end to end workflows here, and any limitations such as human capacity to provide relevant material to the robotic solution - this may be a limiting factor for instance in relation to digitisation, which may prevent automation operating at maximum speed and efficiency. Even where the costs of automation outweigh the benefits in relation to a particular context and set of aims, it is worth bearing in mind that this may change - particularly where institutions are undertaking major changes to collections such as moves or mass digitisation it is worth revisiting the business case regularly, and trying to retain flexibility to future automation options.

Benefits - labour substitution and beyond

Automation is often perceived as a time and/or cost-saving technology, and this may indeed be key - for example institutes considering new storage facilities may aim to minimise their future resource costs by examining the scope for automated retrieval of storage units. There are, however, a variety of other aims to which automation may be relevant - for example managing key operational risks (health and safety; security); increasing speed and quality/consistency e.g. in performing digitisation; and ‘freeing up’ increasingly scarce human resources to focus on higher value, more rewarding activities that may increase wellbeing. Use of automation may be part of a wider business case, e.g. it may be that cost savings of new storage per square foot can only be realised if storage at height can be used with automated retrieval. It is important to fully establish the criteria against which you are assessing automated or other solutions, taking a rounded view of benefits over time.

Acceptability

Although, as shown in this paper, instances of physical automation are relatively widespread in collections, these instances may not be recognised widely (e.g. conveyor belts may not be talked about in terms of robotics or automation). Further pilots, as suggested below, and communication of their successes and lessons learned across the sector, are likely to be vital to building wider acceptance of the possibilities. Similarly, such pilots and communications are needed to build awareness in the industry of the needs and challenges in our sector, and to highlight this as an area of opportunity that is open to the possibility of automation. If pilots are isolated, and not communicated, it is likely to be a very long time before there is any systematic acceptance of automation within the sector, or any deeper engagement with potential suppliers.

Return on Investment

Overall return on investment (RoI) is the ratio of net returns to the cost of any investment (measured over a defined period of time). Usually, net returns are expressed in the form of profit, however in social investment for example monetary value is given to other forms of benefit or return. To understand the RoI of any investment in automation, the benefits set out above need to be calculable in financial terms. This is relatively straightforward for substitution benefits (i.e. relative savings compared to human time/cost), but can be more complex for wider benefits. Institutions may need to consider, for example, cost savings as a result of fewer worker days lost to injury; or the benefits to research of having access to greater volumes of digital specimen data more quickly. DiSSCo may be able to provide templates and methods to help understand the money value of benefits consistently.

4.2 Managing and mitigating health and safety risks

A key benefit of robots in industry is that they can manage many operations, particularly repetitive tasks and those involving objects or environments that pose risks to humans, with much lower chance of injury or damage than human operators. To understand a risk-based approach to assessing the benefits of automation for natural history collections, the Natural History Museum London (NHM) analysed their Health and Safety risk management system - the detailed outcomes and methodology are at Appendix 2.

Across discussion with the NHM Health and Safety team, and analysis of relevant risk categories and risk assessments, key risk areas fell under the broad categories of manual handling; broader environmental factors; and working alone. There is scope for automation to mitigate risks across these areas, although there are also some risk categories where the risk to robots is equivalent to or higher in severity than the risk to humans. In all cases, the usefulness of automation to mitigate risks depends on the desired outcomes (e.g. acceptable risk level) and on cost-benefit analysis relevant to the particular circumstances.

4.3 Working with industry

As discussed in the Case Studies section, it proved difficult to engage industry in this ICEDIG task in order to examine case studies; understand industry (including SME) needs, and develop specifications for research & development around robotics in relation to natural history collections.

The engineer that Picturae worked with during the development of their herbarium conveyor belt was one of the few who was available for extended contact. He indicated that during the development of the herbarium conveyor belt, a key factor was understanding the strong requirement for safe transport and automated recognition of the specimens, which for example meant that vibration was minimised. Clear requirements are central to effective solutions tailored to our sector, across smaller or larger industrial partners and subject matter experts. Industry can’t be expected to come up with solutions for problems that are specific for natural (and cultural) history collections - only through clear requests, and demarcated problems that need to be solved, can they attempt to contribute. One of the issues is that the term ‘biodiversity collections’ or similar is largely unknown to this sector, and is very unlikely to be recognised spontaneously as a field of opportunity.

A combination of clear requirements and the right financial framework and incentives is likely to be critical. A previous example during which natural history collections similarly failed to gain sufficient engagement from technologists was the Beyond the Box competition, by the American Institute of Biological Sciences and the National Science Foundation (2014-2015). This offered an award of $1 million (USD) to the person or team who created a technology that increased the speed and accuracy of digitisation of a drawer of insect specimens and their associated data (National Science Foundation 2014). After the deadline, no submissions were received which met the requirements, resulting in no award made. While this competition was backed with a large sum of prize money to attract competitors, this fund was specifically to further develop and to market the solution, putting off some not-for-profit entrants, while the wider conditions and rules of the competition were quite prescriptive. When seeking technical collaborators it is important to allow means for partners to become familiar with the requirements and limitations of specific natural history collections. This competition laid out strict levels of achievement and a set of abstracted test material (PDF, with dimension and location details of drawers, unit trays and specimens); giving a limited range of flexibility for the potential contestants. Because roboticised solutions for most heritage collections are high in complexity, more interaction is needed to establish a suitable solution. In any future competition, a better outcome may be achieved by defining the outcome and requirements in principle, rather than the approach. It may be helpful to consider examples of competitions in other sectors, such as the Amazon Picking Challenge.

Many businesses in the automation industry have a large enough market in various sectors (manufacture, logistics, food production). In comparison, the natural history and heritage sector is relatively small and complex. Developers need an incentive to focus on the heritage collection market, which often translates into making funds available for development (subject to relevant procurement regulations). Funds may come directly from institutions, or from e.g. government innovation funding in some countries. The right combination of funding and clarity of requirements and challenges is key to future work with the robotics and automation industry.

4.4 Next steps for DiSSCo

It has not yet been decided whether DiSSCo will own or operate centralised digitsation facilities, however it has been recommended (Hardisty et al. 2020 - also quoted below) that DiSSCo should consider Centres of Excellence in particular services, offering structured leadership and best practice, and 'should consider building an experimental facility (DiSSCo Centre of Excellence) for an ‘out-of-town’ fully automated, industrial scale specimen storage and digitization facility'.

Prior to such a facility being possible, further work and pilot projects are needed to better understand the technologies available and to explore and demonstrate their successful application to natural history collections. This is recognised in the recommendation that DiSSCo should 'work towards a future automated end-to-end digitization solution, development should focus on independent components (including storage and retrieval, transport, object picking, and imaging) which can be connected in the future' and develop a series of pilot projects.

In the first instance, the analysis in this paper should be expanded to a formal identification of relevant workflow stages; whether solutions are already marketed that could address these; and where there are specific challenges and gaps. Pilot projects can then be targetted to test solutions in practice, and to develop solutions to fill gaps. It's likely that this will require a 'mixed economy' approach, including some or all of:

  • identification and incentivisation of consortium members who may have opportunities to pilot physical automation approaches
  • innovation competitions targetting R&D challenges
  • partnering with industry to bid for innovation funding and work bilaterally on R&D
  • customer-supplier projects to test particular technologies

As set out above, the right combination of clear requirements/problem statements, coupled with incentives for subject matter experts to engage, will be critical. DiSSCo coordination of potentially disparate projects, perhaps through a Centre of Excellence, can support the process of making specifications and requirements clear, and can provide a sufficiently substantial platform to interest commercial suppliers. DiSSCo should also assure project alignment to workflow steps, sharing the successes and lessons learned from demonstrator projects to inform the wider community. Templates and/or consistent methodologies for valuing benefits in financial terms would also support understanding of return on investment in automation, and also in other kinds of investment such as wider investment in digitisation - DiSSCo may be able to access the support of economists to develop this area, which may not be available to individual institutions.

These steps could then lead towards the creation of a fully automated DiSSCO facility if desired.

5. Conclusion

For this report the current state of robotics and automated warehousing was investigated in relation to the storage, handling and digitisation of natural history collections. Several examples of automation from e-commerce and the library sector were explored. Two examples of a robotic arm in the heritage sector were also studied.

At this time, although there are many instances of automation being used, the natural history collections sector has not implemented any fully automated solution from storage to imaging (or other processing/use) and back to storage. By continuing to develop independent components which can be connected in the future, progress can already be made towards an end-to-end solution. DiSSCo has a role to play in catalysing such pilots; communicating the outcomes; stimulating engagment with industry; and ensuring pilots contribute to an overall end to end view.

Previous attempts at remote microscopy have been unsuccessful but may work in specific contexts where standardised specimen or sample preparation can occur. This could include electron microscopy using standardised stubs and mounts, or microscope slides with standard dimensions and preparations. Improvements in automatic loading using cassettes to remove the need of a permanent on-site operator and improved internet speeds make remote microscopy more viable, albeit at a limited scale.

While robots are clearly very successful at repetitive tasks on standardised items, they are developing rapidly into more complex kinds of handling, and wider physical automation from environmental control to conveyor belts has clear potential for our sector. Increased standardisation of storage units (drawers, jars, platforms etc) would greatly increase the possibilities for automated handling and movement of collections objects - as can be seen in molecular collections described section 3.6. It is likely that software and hardware automation combined will make a significant difference to the challenges of mass digitisation and data mobilisation in our sector, and perhaps also the wider issues of collections management and storage in the longer term.

The natural history sector will need to work with suppliers and subject matter experts, who have the required expertise to develop and integrate components. Natural history collections need to supply clear information about challenges, requirements and constraints. Competitions and tenders will need to allow for SMEs to see collections in action and to ask questions to gather information and experience that they require for development.

DiSSCo can play a further role in developing the expertise to better communicate with industry; and in particular to develop a consistent set of pilots that address gaps in workflows and build confidence and understanding where technology solutions already exist.

Appendices

Appendix 1: Using a risk-based approach to consider the benefits of robotics and automated warehousing in natural history collections

As part of assessing the potential for robotics, automated warehousing technologies and assistive technologies in relation to storing, handling and digitising natural history collections, we investigated key categories of risk in the Natural History Museum, London’s (NHM) risk management system. Sections i - iii below cover discussion, findings and suggestions for how these could be applied. The remainder of the Appendix is a methodological account of the system review.

Across discussion with the Museum's Health and Safety team, and analysis of relevant risk categories and risk assessments, key risk areas fell under the broad categories of manual handling; broader environmental factors; and working alone. There is scope for automation to mitigate risks across these areas, although there are also some risk categories where the risk to robots is equivalent to or higher in severity than the risk to humans. In all cases, the usefulness of automation to mitigate risks depends on the desired outcomes (e.g. acceptable risk level) and on cost-benefit analysis relevant to the particular circumstances.

i. Manual handling

It is clear that there are many unavoidable manual handling tasks and risks in working with collections, some of which are occasional e.g. only incurred during major moves, and some which recur regularly e.g. producing items on request for visitors/researchers. Figs 9, 13 show an extract of the manual handling risk assessment template. Review of particular risk assessments confirms the importance of the categories within this template, which cover the nature of the physical task required (e.g. bending, stretching, repetition); the environment in which the task is carried out (e.g. lighting, obstacles); the load (weight, bulk; hazards such as sharp edges); and the capability of the operator (e.g. restricted movement or conditions such as pregnancy).

Figure 13.  

Manual Handling risk assessment template extract (page 2).

Across all of these elements, mitigation currently includes a range of equipment, from personal protective equipment (PPE - e.g. gloves, masks) to other hardware such as trucks or ladders. There is scope for automation to reduce any of these categories of risk - as set out in the main report above, robots are excellent at repetitive tasks (which risk human injury); and robots or assistive technologies such as exosuits can increase the capability of humans or offer alternative capabilities e.g. in managing heavy loads. In relation to environment, specific to manual handling, robots are likely to have different requirements from their environment than humans, however their need for consistency/standardisation is likely to make risks such as access obstacles or uneven floors as much or more of a problem for robots as for humans.

ii. Broader environment

Risk assessments and discussions with the NHM H&S team identify a range of environmental considerations beyond manual handling, where automation may help to address risk. These include security considerations e.g. where there is a need to limit human access to high value materials; pest control, where again more human movement and access increases risks to collections; factors such as air quality, temperature and noise, whether deliberate as in a low-oxygen or cold store, or unplanned; and hazards such as biological and chemical hazards, sharps; very high or low working etc.

Clearly where human access is a key element of risk, there may be scope for automated solutions - although the need for human operators to work with/on robots in many cases may reduce these benefits. Similarly, robots may have a greater tolerance to certain environmental conditions than humans - they are unlikely to be impacted by noise, and can operate in planned environments that are not suitable for prolonged human access, such as those with low temperatures, reduced humidity or low oxygen. Again, consideration must be given to the need for humans to work with the robots and to maintain them.

In some instances, robots may have better tolerance to hazards, however there are many hazards which will pose an equivalent or greater risk to robots compared to humans, for example electrical faults, fire, leaks and corrosive materials.

iii. Working alone

Where robots can operate without (or with minimal or remote) supervision, they may reduce the instance of collections staff having to work alone or in isolation. In general terms, it seems likely also that technologies such as environmental sensing and remote communications could increase the safety of those working alone, particularly in laboratory environments where various types of hazards may exist.

iv. Risk review - NHM risk management system

The NHM uses a software called Rivo Safeguard (now on SpheraCloud) to record and manage risks. There are limits on the functionality of the version in use at the time of this assessment, particularly in terms of searching or exporting data from within risk assessments. In addition, details of incidents are protected personal data.

This exercise therefore included: discussion with relevant personnel e.g. the NHM Health and Safety team for their insights; examination of the categories of data/information within Safeguard to determine where relevant information or risk assessments might be found; a manual examination of a sample of current risk assessments across relevant categories, to identify and capture relevant content; and sorting of this content into thematic areas for further examination in relation to robotics and automated warehousing.

Key content categories within Safeguard are ‘hazards’, ‘assessments’, ‘incidents’, and ‘risk assessments’. Assessments and hazards relate solely to assessments of display screen equipment (DSE) for individuals, and resolution of related issues e.g. the provision of eye test vouchers for DSE users. These categories are therefore not relevant to automation or robotics - they apply to any environment where staff are using computers. Incidents and risk assessments are covered in more detail below.

v. Incidents

Incidents in Safeguard record Health and Safety incidents and near misses, subdivided by relevant areas of the Museum organisation. Areas most relevant to collections and warehousing are Conservation; Life Sciences; Earth Sciences; Imaging and Analysis Centre (IAC); and Libraries.

Between January 2014 and August 2019, Safeguard recorded just over 150 incidents (accidents and near misses, excluding pre-existing medical conditions) across those areas, of which all but 8 involved employees (5 were departmental visitors and 3 were scientific associates); and only 3 record any days lost as a result.

Summary data does not provide any details of incidents or causes, but examples checked by the NHM Health and Safety manager suggest the majority are related to factors such as slips, trips and falls, rather than directly to collections handling or similar. Incidents in Safeguard therefore do not provide useful data for this report. However, the NHM Health and Safety manager has suggested the following themes/areas which have been present in incidents over the years and may be relevant to future facilities and to automation: security (e.g. conditions around physical access to collections); manual handling; and environmental conditions including pest control.

Risk assessments

Risk assessments are completed by staff in relation to all activities that may result in a health and safety risk, to record risks and mitigation. They therefore provide the key information for this evaluation of areas that may be relevant to robotics, automation and assistive technologies.

Risk assessments fall into user-selected categories known as 'types'. Those types considered to be in scope for this analysis were:

  • NHM standard multi-hazard
  • Manual Handling
  • Working alone

Types excluded as less or not relevant to this exercise were Manager's health and safety control; Fieldwork; Young persons (under age 18); Quarterly inspections; and Business traveller's risk assessments.

Risk assessments are ‘current’ or ‘archived’ - this is at the discretion of the author. For this exercise, only current risk assessments were considered (with one exception for a relevant completed project whose risk records had been archived promptly).

As with incidents, risk assessments are further subdivided into areas of the organisation - there are more of these organisational categories for risk assessments than for incidents. These include some ‘generic’ areas used to hold template risk assessments, and also some projects in addition to regular organisational units or teams.

Out of a longer list including many other Museum teams, the following areas involving collections were identified as having relevant current risk assessments for review:

  • Conservation Centre
  • Digital Collections Programme
  • Earth Science (ES) Collections
  • ES Economic and Environmental Earth Sciences
  • ES Invertebrates and Plants Paleobiology
  • ES Mineral and Planetary Sciences
  • ES Vertebrates and Anthropology Paleobiology
  • Imaging and Analysis Centre
  • LA Operations
  • Life Science (LS) Algae, Fungi & Plants
  • LS Angela Marmont Centre (for UK biodiversity)
  • LS Department Operations
  • LS Diversity and Informatics
  • LS GENERIC on site
  • LS Insects
  • LS Invertebrates
  • LS Parasites & Vectors
  • LS Vertebrates
  • Molecular Biology Labs GENERIC
  • PROJECT - Anthropology Store Refurbishment
  • PROJECT - Airless (archived assessment)
  • Tring (NHM’s Museum and collections at Tring)

A range of other areas were checked but found to be obsolete/duplicate or having no current risk assessments.

For each area above, risk assessments were ordered by the ‘type’ to identify current risk assessments of three relevant types. These lists were downloaded to a spreadsheet, identifying 642 potentially relevant risk assessments. Of these, the vast majority were the NHM standard multi hazard type, which is clearly used as default and has overlap with other types. Two were for working alone, and 15 for manual handling.

All manual handling and working alone risk assessments were opened and manually reviewed.

Over 600 multi hazard assessments was too many for detailed examination in this project. In practice, however, many in fact were of other types - for instance it is possible in the title to identify fieldwork as the main focus. Other titles also gave good indication of relevance, for instance it is possible to see which assessments focus on building work, behaviour of contractors or users and other areas which are not relevant to the potential for robotics and assistive technologies. A manual exercise was therefore carried out to identify the most relevant titles and examine these risk assessments in detail for the multi hazard type.

Results of risk assessment review - manual handling

Screenshots of the Manual Handling risk assessment template are given in Fig. 9 and Fig. 13. As well as text fields, this template includes sections with radio button lists which highlight possible areas for intervention in relation to task, environment, load and capability. In addition (not shown in the figure) the template goes on to cover additional hazards; individuals with specific needs (e.g. disabilities, pregnancy); safe systems of work and control measures.

Fifteen risk assessments relating to manual handling were identified in our work, and these were all opened and reviewed. Of these, 9 were assessed as low and 3 as medium risk after control measures. 3 were not scored (e.g. because they are generic templates, not specific assessments of risks in practice).

Areas covered in these templates were:

  • Moving, packing and lifting specimens and artworks
  • Handling heavy and/or bulky specimens or artworks
  • Moving and lifting equipment, furniture, boxes/storage and supplies
  • Acids (in practice this would expand to include many hazardous supplies, preservatives and specimens, whether chemical, radioactive etc)
  • Obstacles in routes e.g. bumps, ledges, narrow areas, turns
  • Handling drawers, trays, folders etc of specimens
  • Protruding drawers or handles and similar
  • Low/high storage areas and similar
  • Use and movement of stepladders and similar
  • Working in compacterised storage areas
  • Use of roller racking
  • Temperature
  • Use of presses (e.g. in plant mounting)

Control/mitigation measures made reference to relevant Museum training, policies, relevant regulations and assistive kit. In particular, approaches used in manual handling may (subject to training and policies) include any of:

  • Wheeled trolleys
  • Hydraulic / scissor-lift trolleys
  • Cranes
  • Pallet and fork lift trucks
  • Skates
  • Hoists
  • Step ladders
  • Lifts (used instead of stairs where possible)
  • Extraction fans and hoods
  • Suitable height tables and similar
  • Wrapping and packing (e.g. to cover sharp or protruding areas while moving)
  • Personal Protective Equipment - clothes, footwear, gloves, high-vis, helmets etc
  • Use of trained staff including porters and specialised movers (contractors)
  • Working with colleagues e.g. to coordinate lifting by two or more people and to supervise such activity
  • Avoiding involving visitors or those with known health considerations in manual handling
  • Use of imaging/digitisation to reduce task recurrence

It is clear that there are many unavoidable manual handling tasks and risks in working with collections, some of which are occasional e.g. only incurred during major moves, and some which recur regularly e.g. producing items on request for visitors/researchers.

Both the individual risk assessments, and the radio button lists in the template at Fig. 9, suggest areas where robotic and assistive technologies could potentially add to solutions and control measures already mentioned above. All of the radio button areas are considered relevant to some of the templates reviewed. Machines are likely to have strengths relevant to human handling in relation to weight; environmental conditions; and repetitive handling. These benefits are likely to apply to repetitive tasks, but may not apply for example where space is restricted or where obstacles exist in routes.

Results of risk assessment review - working alone

This template has text fields only, not ‘built-in’ categories or lists like manual handling. Fields cover description; named employees at risk; hazards to be added; specific needs; and safe system of work.

Only two working alone templates were identified in relevant areas and these were both opened and reviewed. These were both rated low risk after control measures. One related to working from home - this identifies risks/hazards while working at home: slip, trip or fall; and fire or CO leak. The other related to lab work involving faecal matter, but included very few details of processes/ mitigations.

Overall, therefore, our exploration showed no clear evidence or examples in relation to working alone risks and whether these could be mitigated through robotics or automated handling. It seems likely in general terms, however, that technologies such as environmental sensing and remote communications (radio, video or phone link) could increase the safety of those working alone, particularly in laboratory environments where various types of hazards may exist.

Results of risk assessment review - multi-hazard

The Multi-hazard risk assessment ‘type’ offers primarily text fields to create and add hazards, covering description; hazards (with ratings); individuals with specific needs; safe system of work; related documents/tasks and review.

Of the 625 multi-hazard risk assessments identified as relevant to this project, a selection of 37 was made for individual review, based on title information, e.g. excluding the many in which fieldwork of exhibition set up was a key emphasis; attempting to avoid duplication e.g. there are very similar risk assessments from different areas regarding use of knives/scalpels; and covering all the sampled organisational areas. Fieldwork was a key category for both Earth and Life Science risk assessments so the majority of these assessments were excluded.

These templates cover a wide range of areas from sections of Museum buildings; to use of specialist equipment; specimen handling (by hand and otherwise e.g. with forceps etc); public events and so on. The key information in these templates are the hazards and control measures.

Relevant hazards identified include the following (it is notable that manual handling and working alone form hazards within multi-hazard, explaining why there are fewer of these templates alone):

  • Slips, trips and falls
  • Falling items
  • Damage to and breakage of specimens, kit etc
  • Spillage
  • Manual handling
  • Accidents or injury including back strain, RSI, crushing/pinching, becoming trapped
  • Electrical hazards including electric shock, power surges, incorrect supply/connection
  • Fire
  • Power failure
  • Explosion and explosive devices
  • Low/high temperatures (area or equipment, and consequences e.g. burns)
  • Working alone
  • Sharp tools or items (from knives to paper, and consequences e.g. cuts)
  • Suspicious packages
  • Abusive or violent behaviour
  • Inappropriate behaviour (to staff or collections)
  • High or low working including step ladders etc
  • Use of hazardous chemicals e.g. corrosive, flammable etc (includes glues and solvents, arsenic, ammonia, lead, mercury, ethanol, acids and others)
  • Biological hazards and pathogens e.g. work with cultures, spores, bird corpses etc
  • Use of X-ray emitting equipment
  • Dust inhalation
  • Noise
  • Light sensitivity (e.g. lasers, flicker)
  • Theft (of collections, donations etc)
  • Leakage
  • Damage to building fabric
  • Risks of working outside, e.g. weather, wildlife, cliffs, tides, bites etc
  • Traffic accidents

Control/mitigation measures largely make reference to training, and to procedures and policies including detailed instructions for relevant equipment and similar. In addition to measures identified under manual handling above, such as personal protective equipment, approaches used may include any of:

  • Carrying radio/phone
  • Security practices e.g. sign in/out and patrols
  • Keeping areas tidy, with appropriate treatment of waste including clinical waste, chemical waste etc
  • Correct use of designated areas, equipment and features (e.g. extraction cabinets, filters, wheel locks)
  • Regular servicing of equipment/labs
  • Environmental controls e.g. temperature, humidity
  • Appropriate use of alarms e.g. in walk in freezers or for fire etc, and relevant response procedures e.g. evacuation
  • Appropriate supervision
  • Taking appropriate time / not rushing tasks
  • Reporting incidents and near misses

Overall, the majority of risk types identified in this review show possibilities for improvement with automation, subject to cost-benefit analysis in particular instances. There are a smaller number of categories, however, where risks identified are likely to be the same or higher for robots as for humans, for example uneven floors, leakage, and power or electrical faults.

Funding program

H2020-EU.1.4.1.1. - Developing new world-class research infrastructures

Grant title

ICEDIG – “Innovation and consolidation for large scale digitisation of natural heritage”, Grant Agreement No. 777483

Author contributions

Authors:

Helen Hardy: Conceptualization, Investigation, Writing - Original Draft, Writing - Review & Editing. Myriam van Walsum: Conceptualization, Investigation, Writing - Original Draft. Laurence Livermore: Writing - Review & Editing. Stephanie Walton: Writing - Review & Editing

Contribution types are drawn from CRediT - Contributor Roles Taxonomy.

References

Endnotes
*1

Visited by Helen Hardy & Laurence Livermore, January 2020

*2

NHM London/Helen Hardy visit to Smithsonian Institution, December 2019