Authors: Edmundo Sanz-Gadea López, Aaron van Diepen
Codebase: Diving Deeper - GitHub
In the field of scientific biological research, the identification of deep-sea creatures in images plays a crucial role in understanding marine biodiversity, monitoring population changes, and discovering new species. Advancements in ocean-going camera systems have granted scientists remarkable access to underwater imagery. Additionally, machine learning models have proven effective in sorting visual data and show great potential in the detection and classification of deep-sea creatures. However, these models often struggle to generalize to new scenarios, such as varying cameras, illumination, the emergence of new organisms, and changes in seafloor appearance. This challenge is particularly complex in the dynamic ocean environment. Furthermore, imbalanced datasets and inadequate deep-sea data collection pose additional challenges. The consequence is that existing machine learning models, trained primarily on images from the upper ocean region, struggle to generalize to greater ocean depths.
In this project we aimed to develop a robust computer vision model capable of identifying deep-sea creatures and performing out-of-sample detection. Out-of-sample detection refers to the assessment of whether new images come from a different distribution relative to the training data. Out-of-sample detection can help scientists identify species that were not adequately represented in the training data and identify changes in species distributions. Our project aimed to enhance out-of-sample detection for multi-label classification of deep-sea creature images while improving the model's ability to handle varying underwater image illumination and visibility conditions, which results in better generalization for images in conditions and depths not present in the training set.
To accurately reflect the real-world challenges presented, the model is trained and tested using different data distributions. The training images are collected within the upper ocean region, up to a depth of 800 meters, while the testing images cover a greater range of depths, up to 1300 meters. This scenario aligns with the common situation in ocean research where accessing deeper waters proves difficult, and annotated data is more readily available closer to the surface. The species distributions often overlap but diverge as the vertical distance between samples increases. The evaluation of the model assesses the ability of the model to identify creatures in the image and accurately perform out-of-sample detection, predicting whether an image originates from a different distribution compared to the training data.
The project utilizes the FathomNet dataset, which is an open-source collection of annotated underwater images depicting various species. These images were captured using a camera system deployed by the Monterey Bay Aquarium Research Institute (MBARI) on Remotely Operated Vehicles along the Central California coast. The depth of the image capture ranges from the surface to 1300 meters.
To ensure effective training and evaluation, the dataset is divided into distinct sets: training, validation, and a final test set. The training and validation sets specifically include images from the upper ocean, with depths less than 800 meters. In contrast, the test set is derived from deeper waters, covering the entire range from the surface to a depth of 1300 meters. Although the species distributions in these regions overlap, they are not identical, presenting a challenge in the classification task.
The training and validation sets consist of 5950 images of different resolutions. Each image is identified by a unique ID and the dataset provides annotations consisting of a list of categories associated with the species present in the image. In the provided CSV file there are a total of 290 categories representing bottom dwelling animal species, denoted by numerical values. However, in the training and validation datasets, only 133 species are present. Additionally, the categories are further classified into 20 semantic parent categories, such as fish, urchin, and anemone, among others. The figure below displays nine randomly selected images along with their respective parent categories. Observing the images, it becomes evident that they present issues such as inadequate illumination and focus, making it challenging to discern the animals. These factors pose significant challenges for the model's performance.
The dataset presents an additional challenge arising from the data imbalance. As illustrated in the figure below, which showcases the distribution of parent categories, only a few animal species are heavily represented in the dataset, resulting in a very uneven distribution. The model's ability to generalize across all categories is hindered by this imbalance because the model becomes biased towards recognizing and classifying the overrepresented classes, due to the limited exposure it has to the minority classes during training.
The challenge becomes even more pronounced when we consider the fine-grained categories instead of the parent categories. The most represented category contains over 2500 samples, while there is a long tail of underrepresented categories. There are 102 species that have fewer than 50 images, 73 of which have fewer than 10 images, and 25 species that are represented by only a single image. This presents an immense challenge for the model to effectively learn and classify species with such a limited number of instances.