Non-destructive survey techniques are replacing or augmenting traditional sampling tools in sensitive marine ecosystems. Autonomous and remotely -operated camera platforms and acoustic recorders have allowed us to observe, record and store biodiversity information much faster than manually transcribing field observations (Zhang et al. 2015, McEver et al. 2023). Moreover, the technology has enabled scientists to explore previously inaccessible areas and depths (Durden et al. 2017). Despite advanced camera sensors producing high-resolution images, there remains a bottleneck to converting digital data into relevant biodiversity information. Innovations in the field of artificial intelligence (AI) present opportunities for numerous applications in marine sciences to address this and allow for rapid biodiversity assessment and monitoring (Mahmood et al. 2016, Marburg and Bigham 2016, Moniruzzaman et al. 2017, McEver et al. 2023).

Computer vision (CV), a discipline of AI utilising neural networks to analyse digital images, has been widely applied in ecology (Weinstein 2017). As seen in the terrestrial realm, it has achieved success in identifying and counting large megafauna through camera trap images (Crall et al. 2013) and drone imagery (Torney et al. 2016), as well as creating detailed penguin colony maps (McDowall and Lynch 2017). The marine realm also boasts successful applications of CV, mostly pertaining to marine species identification (Storbeck and Daan 2001, Jalal et al. 2020, Mohamed et al. 2020), measurement (White et al. 2006), behaviour (Papadakis et al. 2012) and estimates of abundance (Ditria et al. 2020). There is also growing interest in CV’s application in benthic invertebrate detection for ecosystem classification (Piechaud et al. 2019, Piechaud and Howell 2022). R-CNN (Girshick 2015, He et al. 2017) and YOLO (Redmon and Farhadi 2018) are popular deep-learning algorithms that utilise neural networks to identify patterns in images to recognise objects, classes and categories. Additionally, CV can be effective in classifying audio, time-series and signal data and have been applied to the analysis of underwater soundscapes. Active and passive acoustic sampling methods can supplement visual surveys to assess components of ecosystems potentially under-represented by visual methods alone. For example, using acoustic complexity analysis to monitor large marine mammals, such as whales and dolphins (Davies et al. 2020, Duan et al. 2022), to quantify the biodiversity of benthic assemblages (Davies et al. 2020). The accuracy and speed of such algorithms provide a blueprint for future CV studies to use and expand on.

Specifically, CV presents the opportunity to automate parts of or the entire process transforming digital images and sounds into relevant biodiversity data and address the manual analysis bottleneck. Computer vision can be applied to real-time data collection, post hoc after samples are collected or even implemented retrospectively to extract data from previously collected data resources. Object detection is a type of CV algorithm widely used to count “things”, i.e. organisms in an image (Papageorgiou et al. 1998, Zhao et al. 2019). Object detection can be used on a single class, focused on a specific species to understand the distribution and abundance of that species; or multiple classes to understand entire biological communities (Davies et al. 2020). Hierarchical class object detection presents a promising classification framework for marine biologists because it allows established classification trees, such as the World Register of Marine Species (WoRMS), to be embedded (Costello et al. 2013, WoRMS Editorial Board 2023). Pilot studies using this type of algorithm to detect and classify fish to varying taxonomic certainty has shown promising results (Kalhagen and Olsen 2020).

South Africa has an established and growing suite of underwater camera platforms funded through the National Research Foundation (NRF), with a long history of monitoring and exploratory surveys. Remotely Operated underwater Vehicles (ROVs), Baited Remote Underwater Videos (BRUVs) and drop and towed cameras are the most popular approaches used to explore and survey the benthic environment (Mallet and Pelletier 2014). Over the past 10 years, more than 50 underwater video surveys from various underwater camera platforms have been conducted, collecting roughly 10,000 hours of video footage and thousands of images. Of these, only a fraction has been processed, as manual annotation of video and images is laborious and time-consuming. The processing bottleneck is exacerbated by the lack of standardisation of formats, techniques and workflows. A standard of best practice is, therefore, essential to facilitate knowledge exchange, align with current field-specific best practices and advance the application of CV in South Africa.

The use of CV is not confined to benthic research, as electronic monitoring schemes to detect and quantify catch and bycatch are finding their way into commercial fisheries operations (Honarmand Ebrahimi et al. 2021, BirdLife South Africa 2023). Pilot programmes have been initiated in pelagic shark and demersal fisheries, whilst animal identification and use of bycatch management measures via CV have been successfully tested in a commercial trawl fishery (Honarmand Ebrahimi et al. 2021, BirdLife South Africa 2023).

South African researchers have applied these state-of-the-art CV techniques to fields from marine geology (Pillay et al. 2020, Pillay et al. 2021a, Pillay et al. 2021b) to seabird ecology (Schoombie et al. 2019), demonstrating the presence of local expertise. Computer vision and AI recognition have also been used in aerial counts of whales and seals (Schneider et al. 2019), as well as being used for taxonomic classification and quantification of various marine organisms, from plankton to higher vertebrates, such as fishes (Lopez‐Marcano et al. 2020, Laplante et al. 2021 Li et al. 2022Salman 2023). Despite this, the application of CV in marine science in South Africa is still in its infancy, with independent knowledge-bases being developed and held in isolation by a small number of researchers, students and research labs. This can lead to the unnecessary duplication of efforts, pitfalls and ultimately limit advancement of the field. Establishing a community of practice that openly shares knowledge of workflows, algorithm selection and annotated libraries between research groups is key to addressing this issue. Here, we present the findings of the first Computer Vision in Marine Sciences (COVIMSA) workshop held at the 17^th South African Marine Science Symposium (SAMSS). We aim to assess the state of knowledge of CV in South Africa and identify the way forward.

The COVIMSA workshop was held as a hybrid workshop at the 17th SAMSS on the 24 June 2022 in Durban, South Africa.

The hybrid workshop included marine scientists currently involved in projects or with interests in CV or related technology; computer scientists, AI researchers, robotics engineers, developers and service providers that are interested in applying CV to marine science challenges. In total, 97 participants from 42 different institutions and eight countries, with diverse professional backgrounds and affiliations participated in the workshop (Table 1). Most of the participants were from South Africa, but there were participants from China, Australia, Ireland, Italy, Namibia, the Netherlands and the United Kingdom. Most (69) participants came from a biological sciences background, but 29 of the participants had backgrounds in engineering, machine learning, robotics and AI.

Computer vision is a field of AI that enables computers to process information from digital images, videos, audio recordings (through spectrograms) and other visual inputs, thereby significantly decreasing time spent manually analysing digital input, especially for long-term monitoring. This field seeks to streamline and automate tasks that the human classification process can do. The field of CV is concerned with automatic extraction, analysis and understanding of data from a single image, sequence of images, videos or sound files, through development of a theoretical and algorithmic basis to achieve automatic visual understanding. Computer vision has the potential to significantly accelerate South Africa’s ecological and environmental observation, monitoring and analysis capabilities. It can revolutionise many cost- or otherwise labour-intensive tasks in marine science, conservation and fisheries applications, while providing easy replicability for long-term studies.

Computer vision has wide-ranging applications in marine science and the management of the marine space, for example:

• automatically classify, identify and quantify catch and bycatch species on fishing vessels during fishing, sorting or offloading operations;
• quantify marine pinnipeds and birds in breeding colonies via aerial counts;
• automatically classify and identify marine animals according to taxonomic features (e.g. sponge and sea-cucumber spicules, fish otoliths/scales, shark denticles);
• automatically classify and quantify habitats and/or species from underwater or aerial footage;
• automatically identify marine related events (boats, fishers, whale-blows, bird activity, algal blooms, low oxygen events and consequent marine animal strandings/walkouts) via aerial footage, fixed-point and/or motion-sensing cameras along the shore, at harbours or slipways;
• automatically detect and identify sounds of marine organisms through passive acoustic monitoring;
• automatically classify species abundance and richness and quantify biodiversity, according to acoustic signatures recorded in marine soundscapes;
• individual identification of marine organisms through pattern recognition.

The COVIMSA workshop's main goal was to connect marine scientists interested in this field with CV engineers and programmers. Furthermore, the workshop further aimed to:

1. Showcase existing CV efforts in Southern African Marine Science;
2. Create awareness of the latest developments in the application of these technologies worldwide and their potential applications.

Drs. Sven Kerwath, Toufiek Samaai and Charlene da Silva led the hybrid workshop on 24 June 2022. Presentations and discussions were acilitated by Justin Kiley, with Danielle Stephenson assisting online participants via the Zoom interactive platform. Bruce Dorrofield provided technical support to integrate the online participants with the physical workshop. The workshop was organised into four sessions, loosely grouped into different aspects of CV and its applications in marine science (Table 2). The last session doubled as a final general discussion and outlook for the future.

All talks and discussions related to CV and AI from the workshop are available on the website: http://sharksunderattackcampaign.co.za/aiworkshop/. Those keen on exploring specific topics can either download or listen to them online. For further details, you can also reach out to the individual presenters.

A short summary of the context of each session is provided below.

The first session provided an informative introductory discussion on the topic of CV, with the fundamental concepts of CV and its importance in today's technological world explained to participants. The workshop also addressed the primary tools available for usage in CV, including live-code walk-throughs to show how these tools are implemented (Table 3).

Notably, machine-learning photogrammetry in the form of Automated Fish ID (AFID; www.afid.io), which aims to reduce the cost and labour required to process BRUV imagery was discussed. This presentation’s focus was around the AFID Digital Assistant, which is currently being developed for the SeaGIS EventMeasure image processing software, a tool used by numerous South African marine scientists. The open-source web platform BIIGLE (https://biigle.de/) was also highlighted as a noteworthy resource for marine CV enthusiasts, with its built-in AI function that can be used for rapid annotation of images and still videos. The Machine Assisted Image Annotation (MAIA) capabilities of this platform and its usefulness in seamlessly analysing large images and video collections was demonstrated during a live presentation. Participants were encouraged to engage with the developers to gain a better understanding of BIIGLE’s software capabilities and potential uses. The other presentations during this session used live code walk-throughs to demonstrate space-time image sequencing as well as to demonstrate the importance of comprehensive data pre-processing, training, testing and validation.

It became evident during this session’s discussion that there is considerable expertise in CV applications in South Africa. Nonetheless, most of the expertise is within the lucrative private sector, limiting involvement in academic and research activity in marine sciences. It was also noted that there are numerous large datasets that could be unlocked using CV applications, but their utilisation is limited due to a lack of capacity. This highlights the need for additional infrastructure and skill development investment in South Africa to enable the effective application of CV technologies.

The application of CV technology in visual census was the subject of the second session. The session specifically focused on how remote technology, such as BRUVs, are being used to count and recognise fish in marine environments, with presenters providing examples of the different AI applications/software which are currently available for use in BRUV research (e.g. AFID, VIAME, FishID, BlueCounter). One of the presentations highlighted the success of employing CV to analyse data recorded by bird-borne video loggers and alluded to the potential use of these data to train deep-learning models in future research.

Unfortunately, some of the software now available for this type of research is not open-source and must be purchased, which may limit access for researchers with minimal resources. It also emphasises the need for further initiatives that use AI to identify invertebrate species, quantify fish counts during trawl surveys and count and distinguish species groups in rocky coast quadrats.

While studies in this field have been conducted independently, it was observed that researchers often operate in isolation, potentially hindering the collaborative potential and broader impact of future research. Increased funding opportunities would help to streamline AI research and to promote collaboration across diverse programmes to take South Africa's marine science to the next level. Researchers can utilise the potential of CV technologies and make substantial advances in our understanding of marine ecosystems by doing so.

The third session focused on monitoring fishing and bycatch in South Africa. A series of talks covered a variety of issues, including the use of cameras in trawl and longline fisheries, where the cameras are now being trialled.

Numerous presentations were delivered throughout the session that showed the possibilities of CV technologies in the fishing sector. One presentation, for example, used CV techniques to quantify trawl bycatch from camera footage taken over the conveyor belt that transports the catch underdeck towards the processing facility of the vessel. In another example, camera footage was used to develop an Adaptive Intelligence demo model for the continuous detection, tracking and reporting of the interaction between seabirds and trawling gear, a significant source of seabird mortality. The model, a first of its kind, designed to deliver real-time analysis on constantly changing data in motion, was trained on images from surveillance cameras positioned at the stern of trawl vessels. It was able to track the flight path of multiple birds, detect seabird collisions with trawl gear, as well as record the presence/absence of bird-scaring lines, the principal mitigation measure used by these fisheries to prevent seabird mortalities. Increasing demands for at-sea monitoring and data collection emphasises the importance of deploying models capable of rapidly learning and processing extensive amounts of biological data in real-time. Such implementation would serve as a powerful tool for enhancing fisheries management and conservation efforts (see: https://www.cognitivesystems.ai/).

Another presentation used Mask R-CNN to auto-identify roman seabream (Chrysoblephus laticeps) from BRUV footage. Aside from these applications, the discussion focused on the potential of technology to promote the development of mechanical and electrical engineering, as well as AI solutions in the fishing sector. Sea Technology Services, a South African-based company that specialises in the development of undersea technology, presented on their ability to assist with the development of these solutions. Sea Technology Services highlighted their in-house expertise with examples of underwater camera platforms that they have designed for both international and local institutions (see: https://www.seatechnology.co.za/).

The session demonstrated the potential of CV technology to provide creative solutions to support the South African fishing industry. Showcasing innovative solutions revealed how CV can contribute to monitoring efforts and promote sustainable fishing practices, while minimising the impact on marine ecosystems.

The final session included structured discussions on best practice, such as data hygiene (standards and formats) and annotation workflows, two critical but frequently forgotten aspects of data processing that are integral for the facilitation of machine-learning applications. This session also touched on aspects of future funding, data availability and confidentiality, all with the goal of increasing momentum to increase uptake and implementation of CV in marine science in South Africa.

The discussion sessions that followed each presentation included ideas for best practices and the usage of various tools and analytical software. Participants talked on open access data and standardised data analysis methods, which could help to simplify the usage of CV technologies across the marine sciences field. A key discussion point was the challenges associated with data management, particularly the backlog of videos and images that often require expert knowledge and considerable time investment to analyse. To overcome these barriers, participants discussed the need for standardised workflows and automated systems that could help speed up data processing. The importance of developing collaborations and infrastructure for CV was also discussed, as this could help promote the use of these technologies in marine science research. Participants highlighted the need to increase data transparency while maintaining confidentiality to protect ecologically-sensitive data. Finally, the need to secure funding for start-up projects was emphasised. Dr. Angus Patterson, the Managing Director of South African Institute for Aquatic Biodiversity and coordinator of the NRF African Coelacanth Ecosystem Programme funding and projects, offered his assistance in this regard.

In conclusion, the discussion sessions were aimed at overcoming obstacles and advancing the utilisation of CV in marine imagery analysis. Through the sharing of best practices and the development of collaborative infrastructure, it is possible to build momentum in marine science CV in South Africa and promote the adoption of more sustainable and efficient practices for utilising our marine resources.

Computer vision can be defined as an interdisciplinary field of AI that enables computers to interpret objects and sound across vast amounts of digital data (i.e. images, videos, acoustic recordings) (Hassaballah and Awad 2020). Computer vision has become a growing area of research in recent times owing to the technological advances in the field of marine science and the associated influx of vast amounts of underwater image and video data which contains an abundance of marine life (Lu et al. 2017). Furthermore, there are many challenges associated with visual processing underwater images and videos, including poor contrasts, out of focus images, colour deviations, low lighting and organic debris (marine snow). These challenges only exacerbate processing efforts and have contributed to making underwater CV a topic of interest amongst marine researchers.

Over the last few decades, there has been an emergence of CV studies in the field of marine science (Shihavuddin et al. 2013, Akkaynak and Treibitz 2019, Wang et al. 2021, Honarmand Ebrahimi et al. 2021, Saleh et al. 2022); however, similar research in South Africa is sparse (Pillay et al. 2021a, Pillay et al. 2021b, Conrady et al. 2022) and appears to lag behind that of the international community.

The workshop achieved its goals in creating awareness about the latest developments of CV by showcasing current applications and projects that have the potential for wider use across marine science in South Africa. It became evident that there are immense opportunities to increase the efficiency of marine research programmes in South Africa by utilising this technology. The field is advancing in South Africa in multiple areas of marine research, but there is a lack of cohesion within the marine science community regarding sharing these ideas, methods, applications and technology. The whole field of bioacoustics, for example, was not addressed in this workshop although the application of CV has been increasing in the topic (Ruff et al. 2021), which might also reflect the lack of researchers in the country using CV in acoustics. On the other hand, while there are multiple applications of CV across many fields of marine research, most projects focus on single applications, carried out by independent research groups on a particular topic. In some cases, such as analyses of BRUV videos, several groups work independently on solving the same problem, i.e. fish species identification, using different algorithms, with little cross pollination.

South Africa possesses a significant amount of technical expertise in engineering and modification of existing systems (e.g. building of camera rigs, BRUVs etc.). However, although there is a growing pool of skilled programmers, the presented applications usually rely on established packages and systems, mostly developed in other parts of the world, principally the USA, Australia, Europe and China.

There is keen interest in applying CV in South African marine sciences, yet scientific output employing these techniques remains scarce. The group collectively agreed that establishing a formal Computer Vision in Marine Sciences South Africa (COVIMSSA) working group would be beneficial in moving the field forward. COVIMSSA could play a vital role in arranging meetings, organising funding opportunities, hackathons and sharing the latest and most useful applications for CV. Furthermore, within this contribution, the workshop's details, outcomes and the contacts of the participants are accessible to the broader marine science community.

In summary, the CV workshop revealed that there is significant potential to enhance the efficiency of marine research programmes in South Africa by leveraging technology and harnessing current programming expertise through interdisciplinary knowledge-sharing. The establishment of a working group and a platform that would promote collaboration, funding opportunities and knowledge-sharing amongst experts in marine science, engineering and programming will undoubtedly lead to ground-breaking achievements in the future.

This workshop was possible thanks to the SAMSS steering committee and WildTrust sponsored by Oceans5 and the Rainforest Trust. We would like to express our sincere gratitude to our institutions for allowing us to host and attend the COVISMA workshop. This workshop was a multi-institutional collaborative effort and we are thankful to all participants for taking the time to attend. The workshop would not have been possible without the participants and we thank them for engaging with us on this subject.

National Research Foundation-South African Institute for Aquatic Biodiversity, Makhanda, South Africa (NRF African Coelacanth Ecosystem Programme funding and projects)

Department of Forestry, Fisheries and the Environment: Oceans and Coasts Research and Fisheries Research

WildTrust

Department of Forestry, Fisheries and the Environment

N/A

CdaS, SK and TS organised the workshop and wrote the report. Other authors contributed to the discussion and write-up of the report.