Research Ideas and Outcomes : Grant Proposal
Print
Grant Proposal
Jellyfish Identification Software for Underwater Laser Cameras (JTRACK)
expand article info Patrizio Mariani
‡ National Institute of Aquatic Resources (DTU Aqua), Copenhagen, Denmark
Open Access

Abstract

Jellyfish can form erratic blooms in response to seasonal and irregular changes in environmental conditions with often large, transient effects on local ecosystem structure as well as effects on several sectors of the marine and maritime economy. Early warning systems able to detect conditions for jelly fish proliferation can enable management responses to mitigate such effects providing benefit to local ecosystems and economies. We propose here the creation of a research team in response to the EU call for proposal under the European Maritime and Fisheries Fund called “Blue Labs: innovative solutions for maritime challenges”. The project will establish a BLUELAB team with a strong cross-sectorial component that will benefit of the expertise of researchers in IT and Marine Biology, Computer Vision and embedded systems, which will work in collaboration with Industry and Policy maker to develop an early warning system using a new underwater imaging system based on Time of Flight Laser cameras. The camera will be combined to machine learning algorithm allowing autonomous early detection of jellyfish species (e.g. polyp, ephyra and planula stages). The team will develop the system and the companion software and will demonstrate its applications in real case conditions.

Keywords

Machine Learning, Lidar, Species Identification, Computer Vision

Objectives

The overall aim of the project is to enable automatic jellyfish species identification and recognition necessary for early-detection of harmful proliferation of jellyfish and the general management of alien species, through exploiting state-of-the-art underwater imaging technologies based on innovative pulsed laser illumination (LiDAR).

This goal will be reached through six distinct and achievable objectives in terms of Scientific and Technical Excellence (ST), Training and Networking Excellence (TN) and Market and Outreach Excellence (MO). Those objectives are:

  1. To establish a cross-disciplinary working team (experts and young-scientists) on underwater image acquisition and object recognition, including engineers, mathematicians, marine ecologists and economists from industry and academia (TN1).

  2. To design, develop and commercially assess efficient and reliable software (called JTRACK) for automatic detection, identification and tracking of jellyfish (ST1).

  3. To tailor JTRACK to specific stakeholder needs and to images obtained by the next generation underwater LiDAR imaging technologies (ST2).

  4. To demonstrate to selected stakeholders the jelly identification system in the field along with conventional oceanographic methods to allow for direct comparison (MO1).

  5. To identify go-to-market strategies and any additional technical requirement to commercialize the software (MO2).

  6. To effectively disseminate project results to the large scale marine and maritime stakeholders (MO3).

Other goals

Blue Lab. The interaction within the established Blue team will produce research development and innovation with long-lasting benefit at regional and trans-national scales. The team will have regular (weekly and monthly) meetings (including teleconferences) and it will be composed by:

  1. a selected group of post-graduate scientists coming from DTU (DK), UniPARTH (IT) and GEOMAR (GER);
  2. senior researchers from the involved Universities;
  3. industrial tutors from BIORAS (DK) and advisors from SINTEF (NO).

An internship program with short-term mobility for students will be established at the beginning of the program and will involve all Partners; MSc projects linked to action will alos be supported during the life of the project. Postgraduate students will allow knowledge transfer and capacity building within the consortium.

Regional approach, cooperation and coordination amongst marine stakeholders. Preliminary contacts with stakeholders have been established and specific needs have been already identified. Some of the stakeholders will also sponsor the action (please refer to the sponsor letters). One-to-one tailored meetings with relevant stakeholders will be planned during the lifetime of the project and a larger forum for a final general assembly meeting will be established, supporting the EU Strategy for the Baltic Sea Region by enhancing the coherence of the cooperation within the region.

Exploiting previous results. JTRACK will build upon an on-going project and a previous project, namely H2020 UTOFIA (UTOFIA 2015) (Underwater Time Of Flight Image Acquisition system) and EU FP7 VECTORS. SINTEF is coordinating UTOFIA and is Partner in JTRACK, DTU Aqua participated to VECTORS with main focus in the Baltic Sea Region. UTOFIA will provide the camera system for imaging while results from VECTORS for invasive species in the Bornholm basin will guide our imaging program.

Contribute to the objectives set in the Call for Proposals

The proposal contributes to all the relevant objectives set in the call text Table 1.

Comparison of call scope vs. JTRACK objectives and project design.

Scope as defined in the call for proposals

Objectives and specific activities in JTRACK

Priorities

4. Invasive alien species and jelly fish proliferation:…

Obj. ST1 Design, develop and commercially assess an efficient and reliable software (called JTRACK) for automatic detection, identification and tracking of jellyfish

Work Packages 2, 3 and 4

this action aims at promoting innovative "laboratories" (hereinafter called Blue Labs) to pilot new and viable solutions addressing selected maritime and marine challenges and opportunities in the blue economy. The focus of this action is to support a novel way of working, where young scientists supported by researchers, industry and local stakeholders, team up to develop innovative solutions to support the development of a sustainable blue economy, while preserving marine resources and ecosystems.

Obj. TN1 Establish a cross-disciplinary working team (experts and young-scientists) on underwater image acquisition and object recognition, including engineers, mathematicians, marine ecologists and economists from industry and academia

Work Package 1 – task 1.2

The Blue Labs concept works at the interface between research and commercial exploitation so that results from research are taken forward to the market. By doing so, it aims at contributing to generate jobs and business opportunities in the blue economy

Obj. MO2 Identify go-to-market strategies and any additional technical requirement to commercialize the software

Work Package 4 – task 4.3

leveraging young peoples’ skills and creativity and increasing awareness of marine challenges and opportunities

Obj. TN1 as above, and Obj. MO1 : Demonstrate to selected stakeholders the jelly identification system (camera and JTRACK) in the field

Work Package 4 – task 4.1, task 4.2

supporting pioneering partnerships between maritime stakeholders and fostering multidisciplinary approaches by combining competences from businesses, the public sector and research bodies

Obj. TN1 and Obj. MO1 as above and Obj. MO3 Effectively disseminate project results to the large scale marine and maritime stakeholders

Work Package 1 – task 1.2

supporting enhanced cooperation and coordination amongst maritime stakeholders at local and sea basin levels (including at sub-region level of the sea basin), whilst capitalising on local stakeholders’ knowledge

Obj. MO1 and Obj. MO3 as above and Obj. ST2 Tailor JTRACK to specific stakeholder needs and to images obtained by the next generation underwater LiDAR imaging technologies

Work Package 1 – task 1.2

the "Blue Labs" action under this call for proposals, aims to build on the results of relevant projects and support innovative and multidisciplinary approaches that can help move those results forward closer to the commercial stage

Obj. TN1 and Obj. MO2 Identify go-to-market strategies and any additional technical requirement to commercialize the software

Work Package 4 – task 4.3

General description of the action

State-of-the-art. Within the variety of life forms populating the oceans jellyfish are certainly among the most intriguing because of their inflated body plan, diverse life cycles, and ecological role. During the last few decades, this species has raised public concerns about their impact on our societies (Condon et al. 2012), often supported by spectacular cases where large jellyfish have proliferated, causing extensive economic damages (Robinson et al. 2014) and dramatic changes to the local food web (Daskalov et al. 2007). Jellyfish can cause direct damages to tourism, infrastructure and fishing gears (Graham et al. 2014) and have impacts on food production, as jellyfish and forage fish (small pelagic fish) feed at similar trophic level (Azaña Schnedler-Meyer et al. 2016). Hence, jellyfish proliferation is a common concern in all the European seas and detection and management of these species are acknowledged in the present call text as priority theme (priority 4). Jellyfish management is identified as one of the main sub-objectives to maintain a “Rich and healthy wildlife” of the Baltic as indicated in the European Union Strategy for the Baltic Sea Region (COM 2015). Quantitative systematic data on the species are largely lacking since sampling methods for jellyfish require filtration of large water volumes as well as analyses of the samples right after the catch since most species are either too large or do not withstand conventional preservation methods (Engell-Sørensen et al. 2009). An image based quantification system would circumvent the current constrain on high quality data acquisition for small and large jellyfish and will significantly contribute to main policy areas (MSFD and CFP) providing an early-detection system for these species.

Knowledge gaps and opportunities. Present-day technologies are unable to provide reliable early detection systems of jellyfish invasion in most real-life situations. Indeed there is a need to develop fast-accurate and reliable identification system for marine species as indicated by a dedicated FAO working group (FAO 2012): “… although only few question the urgent need for improved species identification [ID] in the context of sustainable resource management and biodiversity conservation, there is still a shortage of funding for the adequate research and development of these new ID tools, and this is hindering their use. “. This lack of knowledge and technologies for jellyfish early detection and species identification has direct economical consequences, with several examples present in the Baltic Sea area as indicated by the stakeholder target group we have recently interviewed:

Jellies and mussels farming: high concentrations of jellies could lower the production of mussels through reducing zooplankton abundance allowing phytoplankton to proliferate to harmful levels including oxygen depletion in near bottom areas, as has been shown due to jellyfish blooms in Limfjorden, Denmark (Riisgård et al. 2015). In such cases, it is relevant to move the mussels to more favorable places. A jellyfish detection system in mussels farm may therefore alert the mussel farm producers, so they can make actions to mitigate the problem.

Jellies and shrimp production/fish farming: A jellyfish bloom completely damaged the Norwegian shrimp fishery in 2011 because of clogging of the fishing nets. Similarly, a large jellyfish bloom negatively impacted salmon aquaculture revenues in Ireland (Baxter et al. 2011). Detection system and species identification could prevent negative economic impacts.

Jellies and coastal management: In southeastern Sweden, one of the three reactors that make up the Oskarshamn nuclear power plant was temporary shut down for one day in 2013 as the intake pipes was felled by jellyfish. For Japanese waters it has been shown that during the 2000´s, incidences between jellyfish and power stations drastically increased compared to earlier years (Purcell 2012).

JTRACK: technical advantages. Limitations of present-day organisms identification systems are linked to the poor imaging system, difficulties in object detection and low accuracy algorithms. Imaging quality of common underwater cameras are exposed to the common rapid changes in underwater light conditions due to water turbidity, surface waves, cloud cover, etc. These limits make the object identification challenging especially when dealing with mimetic or transparent objects (e.g., fish and jellyfish). Moreover the difficulties in sharp object detection and the little information about the object when detected (i.e., morphology and color) makes the training of the machine-learning algorithm difficult. JTRACK will focus on detection, tracking and identification solutions tailored for the next generation range-gated camera technology based on state-of-the-art “time of flight” image sensors and innovative pulsed laser illumination (LiDAR). Those new cameras provide undoubted advantages respect to regular cameras, since the autonomous light source makes them independent from local underwater light conditions. Moreover the range-gating technology is able to minimize the effects of water turbidity allowing for larger observed volumes (by factor 2 to 3 over regular cameras) making them much more suitable for early detection systems of marine species. Finally the camera can provide three-dimensional information about the observed objects including their size, shape, and speed. Species identification can then be largely improved when those additional data are integrated in the machine-learning algorithm.

JTRACK: project activities. The project is developed along different activities:

Scientific and Technical (ST) – (1) Design and execute laboratory and field work for jellyfish imaging. (2) Develop, test and apply software for jellyfish species detection, tracking, identification and sizing. (3) Apply a demo JTRACK system (camera + software) in the field (see WP 2 and 3).

Training and Networking (TN) – (1) Postgraduate training in: ST aspects, stakeholder management. (2) Training of stakeholders using the JTRACK system. (3) Facilitate transferability of project results to the other regions. (4) Identify synergies with relevant on-going EU H2020 projects (e.g., COLUMBUS 2016, UTOFIA 2015) (see WP 1 and 4).

Market and Outreach (MO) – (1) define go-to-market strategy including a detailed business plan. (2) Plan actions to ensure project sustainability after the end of JTRACK. (3) Dissemination actions (website, etc.) and build-upon COLUMBUS team activities and the “Future Ocean” cluster in Germany (see WP 4).

While ST and MO activities will be mainly implemented in Denmark (DTU, BIORAS) TN tasks will directly involve different Countries and especially Italy (UniPARTH) in dissemination and transferability of the results, Norway (SINTEF) for liasoning with other projects and advise in go-to-market strategies. Additionally, broad dissemination to the public and policy makers via open science days and demonstration of the system is planned in Denmark (DTU) and Germany (GEOMAR). During the 24 months life of the project the Blue Lab team will have dedicated physical office-space and facilities available at DTU for daily work and extended group meetings.

JTRACK: added value. The innovative software would be the first of this kind in fully exploiting the advantages of LiDAR cameras for object detection and identification. We expect to produce new solutions in machine-learning algorithm to be efficiently designed to utilize the additional imaging features provided by these cameras. The project will clearly build upon H2020 UTOFIA project that is developing state-of-the art LiDAR cameras which will be make available to develop JTRACK’s activities. JTRACK will significantly expand the commercial exploitation for the UTOFIA cameras adding significant value to the project including new markets.

The working team includes experts on: computer vision, jellyfish ecology, tracking technology, imaging systems, economy and industry. Moreover the overlapping interests of many of the post-graduate students associated to the project will guarantee communications within the team to efficiently exploit the expertise available in JTRACK and reach all the goals. Students will have tutors from University (DTU, UniPARTH), research institute (GEOMAR) and Industry (BIORAS) with additional industrial partner (SINTEF) providing advice on imaging system, identification algorithm and go-to-market strategies. The Norwegian research institution SINTEF is vital to the implementation of the project since it will give access to the UTOFIA camera (prototype and final systems), and will be able to provide advice on both imaging systems and interpretation of such images. The University of Naples “Parhtenope” will help the technological advance in WP3, and the transfer of knowledge generated in the project to other regions: both policy makers and a larger group of stakeholders during the dissemination activities. Additionally, on-going monitoring activities and free (in-kind) access to different research vessels at GEOMAR allow for testing and quantitative ground-truth the detection system throughout the project.

Expected Results

The main results of the project are:

  • Compilation of a new annotated training database for jellyfish identification that includes: taxonomic names, sizes and shape categories of the jellies

  • Release of the software JTRACK version 1 for jellyfish detection, identification and tracking with underwater time-of-flight and LiDAR image acquisition systems

  • An international cross-disciplinary long-lasting team between academia and industry on underwater marine species identification will be established

  • Establishment of a long-lasting forum for Baltic Sea marine and maritime stakeholders

  • Development and test of new and alternative sampling technology for delicate, non-preservable plankton organisms

The project will build upon on-going EU projects such as H2020 UTOFIA, which is developing time-of-flight imaging systems based on LiDAR and the completed project FP7 VECTORS in identifying suitable Baltic regions for jellies imaging. Results of the project will have major impacts in the implementation of the EU Strategy for the Baltic Sea Region (EUSBSR) as reported in Table 2

Results, impacts and indicators of the JTRACK project.

Main result

Impact

Indicator

Compilation of a new annotated training database for jellyfish identification

Set baseline for MSFD indicators supporting the HELCOM Baltic Sea Action Plan (EU 2009)

Database available on PANGAEA (PANGEA 2017)

Release of the JTRACK software for jellyfish detection, identification and tracking with underwater time-of-flight image acquisition systems

Filling gap in present species identification methods; support Policy Areas on Ship, Hazards and Nutrients in e.g., Managing the invasive alien species risks at the Baltic Sea, Development of Baltic Marine Environment Protection Commission (HELCOM) core set indicators, Baltic Blue Growth (BBG) (EU 2009)

Software available for download on gitHub and project webpage

Establishing an international cross-disciplinary long-lasting team between academia and industry on underwater marine species identification

Support Policy Area Education of the EUSBSR and Integrated Maritime Policy

Signature of a Memorandum of Understanding for internship student mobility between Partners

Establishing a long-lasting forum for Baltic Sea marine and maritime stakeholders

Supporting several Policy Area of the EUSBSR

Website with a dedicated section for stakeholder news and comments

New sampling technology for delicate, non-preservable plankton organisms

Baseline for MSFD indicators supporting the HELCOM Baltic Sea Action Plan (EU 2009)

Demonstration to stakeholders month 22

Technology Readiness Level (TRL)

From already available basic technology on machine learning algorithm (TRL 1) during the course of the project we will research the feasibility for jellyfish identification software (TRL 2) producing a first proof of concept (JTRACK version 0; TRL 3) within 6 months of the project. The version 0 of JTRACK will then be integrated with object identification and tracking using regular cameras in experimental setup (TRL 4, Month 12). LiDAR cameras will then be used in the field to acquire images that will be processed by an extended version of the identification software (JTRACK version 1, TRL 5). Finally the system camera JTRACK will be fully integrated and the demo system demonstrated to stakeholders in relevant marine environment (TRL 6, Month 22).

Composition of the Consortium and Blue Lab Team

We have 5 applicant Organisations (DTU, GEOMAR, UniPARTH, BIORAS Aps, SINTEF ICT) from 4 different Countries (Denmark, Germany, Norway and Italy). The team has all the necessary skills and expertize to successfully implement the project providing results that will have impacts at Baltic Sea regional scale and beyond. We describe below the specific people involved in the Blue Lab Team from the different Organizations and provide in each WP (described in section “f”) specific role and responsibilities. In Table 3 we provide a synthetic overview on the relevant profiles for the different Organizations and their involvement in the specific Work Packages (WP) with a general task description.

Partners’ description, role and specific tasks in the JTRACK.

P1- Technical University of Denmark (DTU) www.dtu.dk - is the leading technical university in the Nordic Countries and is among the most innovative Universities globally. DTU is characterized by research-based education at the highest international level, with close interaction between researchers, Authorities and business, as well as strategic alliances with several international partners. Two Departments at DTU will participate to this action: DTU Aqua, the National institute for Aquatic Resources that is a leading European institute in marine and maritime science and a major player for Baltic Sea fisheries; DTU Compute, the department for applied mathematics and computer science is an internationally leading research department within e.g. machine learning, computer vision and statistical image analysis. Both departments have very-long records in management of large EU projects and world class expertize in generating innovative products for public sector consultancy and to the benefit of both the industry and public sector institutions.

WP

Time

Description of work

WP1

30%

DTU Aqua will coordinate the project and will provide logistic and IT support. The coordinator will establish the management body of the project, will organize the meetings and communicate with EC to report on activities; together with the Steering Committee will implement specific actions (e.g. mobility plan) and will manage risks. DTU Aqua will also be responsible of setting up the website and its administration.

WP2

10%

DTU Aqua will support the field and laboratory activities for imaging acquisition and compilation of the training database. DTU (both Aqua and Compute) will be responsible for the testing of the complete demo system.

WP3

30%

DTU Compute will lead the final software development and interact with UniPARTH to optimize the machine-learning algorithm for the LiDAR camera.

WP4

30%

DTU Aqua (supported by Compute) will guarantee coordination with other on-going EU projects (namely: COLUMBUS, UTOFIA) and marine science networks (e.g. EUROMARINE) and coordinate the outreach and dissemination activities supporting the stakeholder training with the complete demo system.

P2- GEOMAR www.geomar.de - is the Helmholtz Centre for Ocean Research and has an excellence expertise in ocean research within the Baltic Sea – as exemplified by its coordination of the Bonus EU Project BIO-C3. GEOMAR covers all aspects of ocean science from chemical, physical and biological oceanography to geology as well as ocean atmosphere interactions. With this breadth of research, GEOMAR is unique in Germany, combining coastal ecosystems with blue water research. It has an annual budget of ca. 72 Mio. EUR, ca. 950 employees and 4 research vessels. The personnel contributing to JTRACK include the post-graduate student Dr. Cornelia Jaspers, who is a world known expert on gelatinous zooplankton and jellyfish ecology and Dr. Catriona Clemmesen-Bockelmann, expert on fish larvae ecology and the Kiel Fjord system and will provide valuable background information for JTRACK.

WP

Time

Description of work

WP2

60%

GEOMAR will lead all the laboratory and field activities to perform jellyfish imaging. GEOMAR will provide ship-of opportunities and all the knowledge of jellyfish taxonomy and ecology to compile the training database.

WP3

10%

GEOMAR contributes to jellyfish identification

WP4

30%

GEOMAR will participate to all the dissemination and outreach activities providing link to their network of stakeholders in the Baltic Sea Region, providing content for e-news and website and disseminating project results in conferences and meetings. Moreover GEOMAR will lead the stakeholder training activities providing ship of opportunities and coordination with the “Future Ocean” cluster in Germany.

P3- BIORAS Aps www.bioras.com - is a Danish software development company, with many years of experience in integrating instrumentation and optical systems. The company supplies advanced specialized digital image analysis solutions for analysis of still images and video sequences in 2D or 3D.

Bioras develops laboratory equipment and monitoring systems: FishGuard system for monitoring fish behavior in commercial aquaculture plants; BallastWISE for quantifying the number of living micro-organisms in water samples from ballast water tanks for the shipping industry; an automatic plate counter, branded as PetrilyzerTM. Bioras is involved in a range of development projects, and has established an extensive network of scientific and industrial collaborators. Bioras has provided training and internship opportunities for a number of students and graduates.

WP

Time

Description of work

WP1

15%

BIORAS will be the main industrial tutor for post-graduate students and help in defining the mobility plan.

WP2

15%

BIORAS will support the final demo system test in the field.

WP3

30%

BIORAS will contribute in the development of the object detection and tracking.

WP4

40%

BIORAS will participate to the stakeholder training and will lead the task identifying a go-to-market strategy including a business model and business plan which can guarantee the future sustainability of JTRACK.

P4- University of Naples “Parthenope” (UniPARTH) www.uniparthenope.it -The Science and Technology Department of University of Naples “Parthenope” (UniPARTH) develops activity of basic and applied research for the knowledge advancement, for new methodologies and tools definition and the development of meaningful applications in the sectors included in the following research areas: Computer science, Physical Sciences, Earth Sciences, Geodetic Sciences, Maritime and Aerial Navigation Sciences and technologies. UniPARTH is active in the base and applied research for the disciplines of Computational Mathematics, Artificial Vision and Forms Recognition and Classification, Images Elaboration and in the associated education.

WP

Time

Description of work

WP3

60%

UniPARTH will lead WP3 and drive design and implementation of the JTRACK algorithm including testing different methodologies for segmentation and tracking of the objects.

WP4

40%

UniPARTH will contribute to the dissemination of the results interacting with the Mediterranean Region networks, organizing a stakeholder meeting and contributing to website and news digest.

P5- SINTEF www.sintef.no - SINTEF is Scandinavia's largest independent research organisation, and is a non-profit research foundation. We employ 2100 people and more than 90% of our annual turnover derives from contract research for industry and the public sector. We receive minimal state funding (around 6%). SINTEF is multidisciplinary, with international top-level expertise in a wide range of technological and scientific disciplines, medicine and the social sciences. Our company vision is "technology for a better society", and it is an important aspect of our societal role to contribute to the creation of more jobs. SINTEF has 20+ years of experience developing optical measurement systems for industrial problems and interpreting the data from these sensor systems. They have worked with core UTOFIA technology – Time-of-Flight sensors – for the past ten years, with a particular focus on characterisation and usage towards robotics.

WP

Time

Description of work

WP2

10%

SINTEF will through UTOFIA provide the LiDAR camera for WP2 work.

WP3

30%

SINTEF will advice on best practice for integration of software and LiDAR camera system.

WP4

60%

SINTEF will advice in producing a business plan for JTRACK coherent with LiDAR camera market potential; will participate to the kickoff and final meeting and contribute to the dissemination of the results.

Coordinator: Dr. Patrizio Mariani is Senior Researcher at DTU Aqua and will be the project coordinator. He is an expert marine ecologist and is involved as PI in several technology projects at DTU including underwater visualization systems, tracking algorithm for fish behaviour and underwater robotics. He is chair of the Steering Committee of EUROMARINE EUROMARINE 2014), a European marine science network with 72 Member Organizations with focus on foresight science, science-governance interface and marine technology development. He is main PI in COLUMBUS and UTOFIA, member of the European Marine Board and involved in several working groups including the CIESM (Mediterranean Science Commission) working group on marine connectivity. He is main PI of the Centre of Ocean Life (CentreForOceanLife 2012 a centre of Excellence with currently about 25 PhDs and 10 Postdocs; he is leading a lab in Marine Ecosystems and Technology (BlueLab 2017) with 2 MSc students, 1 PhD and 1 postdoc.

Added value

The Blue Lab established in this project is an international and interdisciplinary team providing innovation via research and technology-development by designing, testing and demonstrating a jellyfish detection systems, which will have direct impacts across several Blue Growth Sectors*1: monitoring and observation, marine environment, coastal tourism, aquaculture, fisheries, maritime transport. As described above we can group the activities in JTRACK around three main themes: Scientific and Technical (ST); Training and Networking (TN); Market and Outreach (MO).

- Ecological ST activities include: jellyfish ecology, execute laboratory- and field-work.

All Partners are main world lead players in marine and maritime sectors. There is a strong regional component in our consortium with DTU and GEOMAR being main actors in the Baltic Sea Regional strategies. Moreover the transnational component in the Consortium is essential for the efficient execution of ST and TN activities. The composition of the Consortium guarantees proper capacity building and training opportunities for all the post-graduate students participating to the project. As formation of students is at the centre of JTRACK, students will be main players in stakeholder management and interactions, as well as the main engine for the ST aspects. BIORAS participation in tutoring and marketing strategies will give both the commercial dimension to the students (which is nonetheless already traditionally in focus at DTU) and provide the best possible framework for efficient commercialization of the software after the end of the project since it matches the core activities of this SME.

Expected constraints and risks related to the project implementation and mitigation measures

RISK MANAGEMENT PLAN: Major technical risks connected to the individual work packages and phases of work have been identified in the course of project preparation. Risks listed have been selected based on the probability and consequence for the project; only high-probability and high-consequence risks are listed. This has been summarized in Table 4.

Risks and mitigation measures planned.

Risk no.

Description

WP involved

Possible mitigation measures

R01

Postgraduates moving to other jobs before the end of the project

WP 1-4

Discuss with EC possibility of substituting them with other postgraduates

Distribute the work among the expert PIs of the working team

R02

Delay in recruiting the postgraduate needed at DTU

WP 1-4

Identify potential candidates before the kickoff meeting and possibly among those already working in the different Labs

WP leaders and task leaders to advance the work until positions are fulfilled

R03

Commercial LiDAR cameras not available during the life of the project

WP 1-4

Use prototypes developed in the UTOFIA project (system 0, 1, 2)

R04

LiDAR camera offers poor imaging of jellies

WP2-4

Early test of the system

Algorithm developed for regular cameras

R05

Driver and software interface development more complex than originally estimated

WP 2-3

Review of interface requirements early within each project phase

R06

Algorithm development does not identify substantial improvements

WP 3

Early and regular review of algorithm scope and implementation

R07

Underperforming partners

WP 1

Close contact between WP Leaders and Technical Leader, short feedback loops and personal contact

R08

Stakeholders not interested in participating in the project

WP 4

Define strategy for one-to-one meetings with stakeholders during Trades, Conferences and other meetings

R09

Mis-match between jellyfish presence in the field and system development and testing

WP 2-3

Apply for more ship-of-opportunities and regular review the workplan based on observational evidences on jellyfish presence

R10

Demo-test application delayed

WP 2, 4

Revise dissemination and exploitation plans. Define alternate application cases

Risks will be reviewed in Steering Committee meetings, to both monitor their status and to identify new possible risks found during development.

Work Plan

Work Package 1

P1 Project Management

Work Package Leader - DTU

Participant Partners - BIORAS

Type of Activity - Training and Networking, Market and Outreach

Objectives:

  • Establish the team work and the management structures and processes
  • Implement the internship and short mobility program
  • Establish a stakeholder forum
  • Monitor performances, development and implementation of the work plan
  • Report to the European Commission, preparation of review meetings and documents

Description of Work:

Task 1.1: Administration, self-assessment and risk management (M1 – M24). Ensure the distribution of responsibilities and the flow of information and to provide the technical and organisational guidance to the project and coordinate the research efforts. Delivery of reports and deliverables in time and in the expected quality. Establish the management structure and schedule central meetings (kickoff, midterm and final) to get feedback on the project planning and progress to make sure the project will give maximum impact on marine science and technology. Establish a clear conflict management to ensure fast and acceptable conflict resolution while reducing the risks of escalations of disputes. Do regular assessment and the analysis of potential risks to prepare corrective actions.

Organisational structure Several management bodies will be implemented in the project Table 5. In particular, a Steering Committee (SC) will be established to help monitor the overall technical progress and to ensure that the decisions are taken on a sound technical and scientific basis. The risk management plan defined in point e) of this proposal will guide discussions about possible mitigation actions, which are taken by the SC. WP leaders and a representative of the post-graduate students compose the SC and its decision taken with verbal consensus or explicit majority vote when needed. A procedure for internal review of each deliverable is established with responsible for review indicated along the deliverables. A support team will help the Coordinator (Dr. Patrizio Mariani) in the management of the project. The support team will be composed by personnel at DTU Aqua: Ivo Grigorov from the Research Office, Lilian Andersen from the secretariat and Dr. Thomas Lindeberg Thørgersen, who has a formation in economy, he is post-graduate students at DTU Aqua and will be associated to JTRACK. He has experience in project management, stakeholder management (node leader in COLUMBUS) and will assist in the management of JTRACK.

Role and Responsibilities.

Responsibility level

Management Body

Composition

Main Responsibilities

Task

Task leader

One person from Partners

· Execute the actions described in the task

· Report to the responsible WP leader risks and impediments in the execution of the task

WP

WP leader

One person from Partners

· Co-ordinate and report on progress of detailed work in the WP

· Responsible for the timely delivery of all the results from their work packages, as defined in the project work plan.

Project as a whole

Steering Committee

- Work package leaders

- Project Coordinator

- Postgraduate representative

· Make strategic decisions concerning project co-ordination, direction, and overall management and planning.

· Project Risk Management

· Plan and implement the internship and short mobility program

Exploitation manager

Leader within WP4 (e.g., BIORAS, DTU)

· Provide an exploitation and business plan

· Oversees IPR management

Project management Team

Project coordinator + support team (see below)

· Organize the SC meetings and prepare the agenda and report the minutes

· Implement decision of the Steering Committee

· Assist all other management bodies

Project coordinator

Dr. Patrizio Mariani – DTU Aqua

· Co-ordinate and report on progress of JTRACK

· Engage with stakeholders

Internal communication. Kick-off (Month 3) mid-term (Month 12) and final meeting (Month 22) will be organized by the Project Management team. Regular monthly meetings with relevant partners will be scheduled as needed. The Project Management team will organize weekly Blue Lab meetings around core activities and will compile minutes for each meeting and make them available on the webpage.

Deliverables: D1.1, D1.5, D1.6, D1.7, D1.8

Responsible: DTU

Task 1.2: Mobility, networking and tutoring (M1 – M24). Implement weekly meetings of the working team and organize the SC monthly meetings. Establish the stakeholder forum and provide methods for stakeholder engagement on a regular basis to guide stakeholder interaction in WP4 and technical development in WP3. Support tutoring of the post-graduate and their interactions with the industrial tutor, internal advisors and stakeholders, including entrepreneurship training provided by FOUNDERS A/S (please refer to Letter of Sponsors). Deliver a plan – agreed among all involved Partners – and monitor the internship and mobility program for post-graduate students, which will constitute the basis for a preliminary Memorandum of Understanding between partners for long term mobility.

Deliverables: D1.2, D1.3

Responsible: DTU, Partner BIORAS

Task 1.3: Creation and maintenance of the website (M1 – M24). Design and develop a webpage for the BlueLab team with dedicated sections for partners contributes (e.g., a blog) and ensure each participants, advisors and stakeholder to contribute to the content including publication of: ideas, results, data, test cases, demo. The task responsible will also publish bi-monthly electronic newsletters to update on project status. All generated image data by this project will be openly accessible via our homepage and stored in PANGAEA to ensure long-term accessibility.

Deliverables: D1.4

Responsible: DTU with contributes from all Partners

Role of Partners:

DTU will be the Coordinator of JTRACK and will lead this WP and be responsible for administration of resources, funding and effort put in the project, coordination of reporting to the EC (deliverables, milestones, periodic reports). This includes the setup and maintenance of IT services like Web Content Management systems (CMS), internal wiki pages, and Mailing list Services. DTU will also be in charge of the scientific leadership and is therefore responsible for the guidance and controlling of technical effort, reaching the targeted objectives and performance of partners. Most of the technical coordination will be carried out as part of work in WP3, DTU work in WP1 will only relate to overall technical management.

BIORAS will be the main industrial tutor for the post-graduate students in the working team and act as facilitators to stakeholder meetings. BIORAS will provide support in the technical coordination (in WP3) and technical management of the project.

Deliverables (Lead Partner, delivery month, reviewer)

D1.1 Project internal management structure and communication infrastructure established and central meetings planned (kickoff, midterm, final) (DTU, M3, SINTEF)

D1.2 Report on plan for post-graduate internship and mobility and draft MoU among partners for long term mobility (DTU, M6, GEOMAR)

D1.3 Guidelines for stakeholder interactions and questionnaire on technical feedbacks (DTU, M10, BIORAS)

D1.4 Website and data repository available (DTU, M3, UniPARTH)

D1.5 Progress report month 6 (DTU, M6, UniPARTH)

D1.6 Interim report month 12 (DTU, M12, GEOMAR)

D1.7 Progress report month 18 (DTU, M18, BIORAS)

D1.8 Final report (DTU, M24, BIORAS)

Work Package 2

WP2 - Jellyfish imaging, training database, field testing and quantitative ground truthing

Work Package Leader - GEOMAR

Participant Partners - DTU, BIORAS, SINTEF

Type of Activity

Scientific and Technical, Training and Networking

Objectives:

  • Perform testing of the imaging system under controlled laboratory conditions
  • Establish a database on jellyfish imaging and features from the Baltic Sea Region
  • Quantitative comparison between camera and conventional sampling methods

Description of Work:

Task 2.1: System testing and parameters definition (M1 – M6). Design and perform laboratory experiments where different jellyfish groups (with regard to shape and size) will be observed and analysed under controlled conditions. Classical cameras e.g., GoPro and machine vision cameras in underwater housings, and a LiDAR camera will be employed simultaneously to allow for direct comparison of different technologies along with actual size and shape measurements of the organisms. Further, different turbidity levels will be simulated to compare cameras’ performances. Images and data will contribute to the training database developed in Task 2.2 and will define parameters range for jellyfish observations by the LiDAR camera. Experiments will be supported by the National Aquarium in Denmark “Den BlåPlanet” that is a sponsor of JTRACK (please ref. Letter of Sponsors) and provide in-kind free access to their facilities.

Deliverables: D2.1

Responsible: GEOMAR, Partners DTU, BIORAS, SINTEF

Task 2.2: Imaging acquisition and database compilation (M3 – M24). Weekly or bi-weekly monitoring cruises in the Kiel Bight will be used to gather data from spring to late autumn. Data acquisition will consist of regular cameras and LiDAR type camera systems along with classical net borne methods for inter-calibration and method comparison. We envision to follow the population development of the common moon jellyfish (Aurelia aurita) (from early stage to adult stages) as well as detecting first occurrence and population dynamics of the invasive comb jelly Mnemiopsis leidyi, which covers a size range from 1 to 60 mm. Additionally other jellyfish that might occur will be imaged and annotated with taxonomic names, size and shape categories. Depending on inflow events of higher saline waters from the Kattegat potential species present in the area are i.e. Cyanea capillata, Obelia octopunctata, Sarsia tubulosa. Further, to have a large cover of the Baltic Region, cruises to offshore waters in the central Baltic Basins will be performed in March, April, May and September of the second year (2018) collecting images both at night and during daylight. An annotated database to be used in WP3 will be compiled. Previous sampling showed abundance of jelly species in the order of 50 per cubic meter in the area. Hence we expect to collect on the order of tens of thousands of images which will be included in the annotated training database.

Deliverables: D2.2

Responsible: GEOMAR, Partner DTU

Task 2.3: Field test of a complete demo system (M18 – M24). The final JTRACK demo will be tested in a workshop organized back-to-back with the stakeholder training session in WP4 (Task 4.2). Field test will be conducted using the R/V Littorina and using the demo-system mounted and deployed on available ROV. We will test the suitability of the system in contrasting ecosystems i.e., turbid coastal waters (Kiel Fjord) in comparison to more offshore waters (Arkona Basin). The system will be used in combination with regular sampling and conventional underwater cameras to establish level of accuracy of jellyfish detection.

Deliverables: D2.3, D2.4

Responsible: DTU Partners GEOMAR, BIORAS

Role of Partners:

GEOMAR will lead this WP and ensure that laboratory and field data will be gathered and is responsible for the quality control of jellyfish identification. This includes the test of the camera and laser system in the field as well as in the laboratory under controlled conditions with cultured jellyfish species of different size and shape. GEOMAR will arrange the workshop and ensure that all Partners will have access to the gathered data and training sets that will be compiled. GEOMAR will give advice on the relevant biology, taxonomy and ecology of jellyfish and will offer ship of-opportunities.

BIORAS will aid in expertise regarding acquisition of real time video using proprietary underwater housings together with camera- and optical configurations.

DTU will assist the experimental studies at “Den BlåPlanet” managing the logistic and will ensure that the data collection is suitable for the algorithm development implemented in WP3. DTU will be main responsible for testing the final demo-system

SINTEF will provide access to the LiDAR cameras for laboratory and field-work.

Deliverables (Lead Partner, delivery month, reviewer)

D2.1 Report on system comparison under different turbidity levels (GEOMAR, M6, DTU)

D2.2 Training database for jellyfish identification available on open access data repository (GEOMAR, M12, UniPARTH)

D2.3 Report on demo performance (DTU, M22, UniPARTH)

D2.4 Manuscript on recent developments in jellyfish quantification (GEOMAR, M24, DTU, + all partners invited to co-authors)

Work Package 3

WP3 - Detection, identification and tracking algorithm

Work Package Leader - UniPARTH

Participant Partners - DTU, BIORAS, SINTEF, GEOMAR

Type of Activity

Scientific and Technical

Objectives:

  • Develop a model for automatic detection, tracking and species identification of jellyfish
  • Develop a model for relating jellyfish measured in image data to populations
  • Quantitative evaluation of the performance of the model under laboratory and operating conditions

Description of Work:

Task 3.1: Initial model development (M1-M6). Monitoring the development of jellyfish populations from image data requires a method for quantifying the jellyfish in the recorded images and a model that relates this quantity to the population. Detection of jellyfish and identification of their species from image data typically involves detection of individual jellyfish, segmenting them to measure their size and shape, tracking individuals through consecutive frames, and classifying them to species based on image features. The initial method development will be based on images of fish from 1) LiDAR and 2) regular underwater cameras, which have been recorded in 2016 while images containing jellyfish will first be available in the project in early 2017. This initial model development will allow gaining experience on segmentation, size measurements, tracking and feature extraction. This analysis will provide information about the sampling possibilities with the system, i.e. what sizes that can be measured at given depths. The depth measurements obtained by LiDAR provides a very good basis for segmentation and accurate size measurements. Features of both desirable objects and non-desirable objects will be assimilated for classification using feature learning mechanisms based on deep neural networks, capable of reaching high identification percentages in most computer vision tasks, and in particular Deep Belief Networks.

Deliverables: D3.1

Responsible: UniPARTH, Partners DTU, BIORAS

Task 3.2: Development and test of minimum viable product (JTRACK version 0) (M6-M15). Based on experience gained in Task 3.1 a plan for data collection will be coordinated with Task 2.2 in order to obtain data for model development including both training and test data. The design of the algorithm will be based on the acquired data, where image features including shape, texture and size features will be extracted. Tracking can be used to extract movement patterns and for offering optimal frames and subsequences for feature extraction. All these features will be used in a classification method trained from the image data collected from regular cameras (early stage) and when possible from LiDAR imaging under controlled conditions. Reference data to train and verify the jellyfish classification will be based on expert annotations. Relating the measurements obtained from image data to population data requires an additional model. Reference data for the population model will be obtained from field-sampling. Robustness of the chosen algorithm will be verified through images acquired both from laboratory and data acquired from the sea. Tracking objects is challenging because the system must deal with changes of appearance, illuminations, occlusions, out-of-plane rotations and real-time processing requirements. The main challenge is the difficulty to handle the appearance changes of the target object. The appearance changes can be caused by intrinsic changes such as pose, scale and shape variation and by extrinsic changes such as illumination, camera motion, camera viewpoint, and occlusions. A preliminary but fully functional version of the software JTRACK version 0 will be presented at the International ASLO conference in February 2018.

Deliverables: D3.2

Responsible: DTU, Partners BIORAS, UniPARTH, GEOMAR

Task 3.3: Operational version of the model and assessment in operation (JTRACK version 1) (M14-M24). Bringing an algorithm from an experimental stage to operation typically requires modifications and adjustments. This can both be caused by change in the characteristics of the data and in the requirements of the requests by the user of the equipment. The purpose of this task is to adjust and optimize the chosen algorithm to meet these changes and requests. This involves an implementation for real-time analysis of the acquired image data, and a reporting module that allows easy interpretation of the measurements including quantity and a possibility for an uncertainty estimate. This will be specifically tailored for the needs of the stakeholders and therefore developed in close collaboration with the end-users. In particular we will follow advice coming from task 4.2 and after the first stakeholder meeting (Month 12).

Deliverables: D3.3, D3.4

Responsible: DTU, Partners UniPARTH, BIORAS

Role of Partners:

UniPARTH will lead this WP and be the main responsible for the initial algorithm development providing an early proof of concept that will be further developed at DTU. UniPARTH will then advice on system developing for JTRACK version 0 and JTRACK version 1 assisting post-graduate students work and providing specific capacity building and training to them.

DTU will be responsible for the release of JTRACK algorithm version 0 and version 1. Will guide post-graduate students’ work and design, test and optimize machine-learning algorithm for detection, tracking and identification.

BIORAS will provide advice on all steps of the technical development. They will be primary responsible for algorithm development for regular camera images and will provide totoring to all post-graduate students.

GEOMAR will provide expert annotation on images and provide guidance and advise on aspects of jellyfish ecology relevant to algorithm development (e.g., shape modification, drifting with currents, etc.)

SINTEF will advice on the integration of the identification software and LiDAR camera settings.

Deliverables (Lead Partner, delivery month, reviewer)

D3.1 Technical report on performance and importance of different image features for quantifying fish: comparison between LiDAR and regular cameras (UniPARTH, M6, DTU)

D3.2 Release of the minimum viable software product (JTRACK version 0) allowing quantification of jellyfish (DTU, M15, UniPARTH)

D3.3 Release operational version of the algorithm (JTRACK version 1) – (DTU, M22, BIORAS)

D3.4 Manuscript on recent developments in jellyfish detection, tracking and identification (UniPARTH, M24, + all partners invited to co-authors)

Work Package 4

WP4 - Dissemination activities and go-to-market plan

Work Package Leader - BIORAS

Participant Partners - UniPARTH, DTU, GEOMAR, SINTEF

Type of Activity

Market and Outreach, Training and Networking

Objectives:

  • Promote and support dissemination activities of the project results and generate the content for internal communication flow and webpage
  • Identify best go-to-market strategies for JTRACK

Description of Work:

Task 4.1: Outreach and international awareness raising (M1 – M24). Information and awareness raising activities regarding the project to the general public and stakeholders include (see also Table 5): participation to conferences (e.g. ASLO meeting 2018 see task 3.2), presentation of the technology to science forum (e.g. EUROMARINE, Marine Board), liasoning with H2020 COLUMBUS project (Knowledge transfer CSA) and engaging the “Future Ocean” cluster in Germany. A dissemination plan will be prepared at Month 3 by the task leader and updated at Month 15 of the project. These activities with stakeholders will be mainly developed by the post-graduate students of the group in order to promote acquisition of transferable skills in communication and to expose them to the challenges and opportunities present in marine and maritime sectors. A post-graduate will be also responsible of updating the e-news and blog sections of the Blue Lab website developed in WP1.

Table 5. Communication and dissemination activities

Deliverables: D4.1, D4.3

Responsible: DTU, Partners UniPARTH, GEOMAR, BIORAS, SINTEF

Task 4.2: Demonstration and training of stakeholders (M10 – M24). Early (JTRACK version 0) and final (JTRACK version 1) versions of JTRACK software will be presented to stakeholders in dedicated meeting at month 12 and Month 22. The early meeting will collect feedbacks on technical aspects (e.g. graphical user interface, identification system, etc) following guidelines provided in task 1.2 and to support WP3 algorithm development in task 3.3. The final JTRACK version 1 demo will be presented to stakeholders in a workshop and training session organized in the last quarter of the project (e.g., October-November 2018). We envision conducting a 3-day event with the first day dedicated to equipment preparation and software testing while two days organized around a training session with invited stakeholders and other scientists. The training session will introduce to system deployment and operation (e.g., using a ROV) and data acquisition and processing. A day cruises with the R/V Littorina, will allow the participation of ca. 25 stakeholders from different groups that will be invited to join. We will demonstrate the suitability of the system in contrasting ecosystems i.e., turbid coastal waters (Kiel Fjord) in comparison to more offshore waters (Arkona Basin). Post-graduate students will organize the event introduce the system to stakeholders and demonstrate the functionality of the system.

Deliverables: D4.4

Responsible: GEOMAR, Partners DTU, BIORAS, UniPARTH

Task 4.3: Evaluation of go-to-market strategies for JTRACK (M3 – M24). The leader of this task will act as Exploitation Manager and during the life of the project will implement activities to generate a comprehensive business model and business plan agreed among all Partners and including the following elements:

1. Intellectual Property (IP) protection and management plan agreement between partners will be discussed and agreed upon at project start.

2. Identify the market and potential customers (exploitation plan). In general, companies that could have an interest in this invention could be (i) government bodies and agencies (environmental agencies, public marine labs, Navies, civil security, Coast guards, etc.); (ii) engineering consulting companies responsible for monitoring of water quality for example near aquaculture facilities, or near water intake for cooling of power plants or atomic reactors; (iii) research institutes and universities; (iv) sensor manufacturing companies interested in purchasing the rights to manufacture and commercialize the sensor. We will contact at least 3 potential customers from each segment to analyze current interest.

3. Testing of the patentability of JTRACK software as a whole or parts of the system. If the JTRACK as a whole or parts of the system are patentable, a patent application will be written during the project period. DTU has a patent office, which can assist during this process. An alternative to patenting is to license the software. The project partners will discuss possible licensing strategies.

4. Define a go to market strategy. We are expecting an interest in the JTRACK system globally from:

  • Government bodies
  • Universities and research institutions
  • Private engineering companies.

Universities and research institutions and government bodies and agencies (environmental agencies, public marine labs, Navies, civil security, Coast guards, etc.) are expected to purchase through public call for tenders. In order to achieve an efficient market introduction and a global distributor network, we are planning to contact distributors specialized in underwater technology and marine survey equipment with a global network, towards the end of the project when we have fully functional prototype and the patentability has been clarified.

Examples of potential distributors might be: Eiva A/S (www.eiva.com), a Danish company specialized in marine subsea engineering and marine survey equipment; McArtney (http://www.macartney.com) a global supplier of underwater technology specializing in design, manufacture, sales and service of a wide range of systems to offshore operators, ocean science institute).

5. JTRACK business plan. A business plan will be written based the IP management plan, market potential analysis and the go to market strategy. If a distributor has been identified, the distributor will be invited to contribute to the marketing section of the business plan.

Deliverables: D4.2, D4.5

Responsible: BIORAS, Partners DTU, SINTEF

Role of Partners:

BIORAS will coordinate this work package and ensure delivery and reporting being on time. Moreover, BIORAS will lead task 4.3 identifying during the duration of the project all the elements and ideas contributing to the definition of the business model and business plan. BIORAS will also contribute to the communication and dissemination activities and demonstration and training of stakeholders.

DTU will lead the dissemination and communication activates providing overview of the opportunities and facilitating interactions with on-going EU projects (e.g., COLUMBUS) and marine science networks (e.g., EUROMARINE) as well as motivating post-graduate students and the entire Blue Lab team to provide content for the webpage. DTU will also participate to the identification of business model in task 4.3.

UniPARTH will contribute to dissemination and communication activities providing access to stakeholders from the Mediterranean Sea Region. They will organize the first stakeholder meeting in Month 12 in task 4.2 and will contribute to all aspect of task 4.3.

GEOMAR will be the main leader for stakeholder interactions in task 4.2, providing access and facilities for the demonstration and training. They will be responsible of the organization of the final stakeholder meeting and contribute to all communication and dissemination activities in task 4.1.

SINTEF will advice on an effective business plan for JTRACK to be coherent with present market potential of LiDAR cameras and contribute to dissemination and communiation activities.

Deliverables (Lead Partner, delivery month, reviewer)

D4.1 Dissemination and exploitation plan (DTU, M3, UniPARTH)

D4.2 Exploitation plan (BIORAS, M8, DTU)

D4.3 Final revised dissemination and exploitation plan (DTU, M15, BIORAS)

D4.4 Report on JTRACK system for application (GEOMAR, M23, SINTEF)

D4.5 Business model and business plan beyond the end of the project (BIORAS, M24, SINTEF)

Timeline

GANTT chart with task number and name in each WP, months duration since the start of the project, duration of the task (in blue) and deliverables number (in green with the number).

Figure 1.  

Gannt Chart.

Funding program

EMFF Work Programme 2016 Call for Proposals EASME/EMFF/2016/1.2.1.4

STATUS: Reserve List

Hosting institution

Technical University of Denmark, DTU Aqua

References

Endnotes
*1

Blue Growth Sectors as identified by the EU H2020 (COLUMBUS 2016)