Cameras help predict survival probability of fish

It is a follow-up of a larger study that focused on improving fisheries in such a way that the fishing techniques used minimise stress and damage to the fish, so that less fish die in the capture process. Based on visual observation of several behavioural features and external damages, experts can give a qualitative estimate of the survival probability of individual fish that are returned to sea as discards. This assessment of fish condition is a useful tool for the evaluation of the effects various fishing techniques or fishing gears might have on the survival probabilities of discards.

Angelo Mencarelli, researcher at WUR’s Agro Food Robotics program: “Classification of survival probability of fish has so far been done manually by an expert observer on board of the fishing vessel. But this is costly and humans are subjective, so one observer may interpret the markings on the fish differently than their colleague on another vessel. Part of this classification could be done automatically using RGB and spectral cameras and AI.”

An objective, automated condition scoring system could make much quicker observations at a lower cost, making discards survival research much easier.

Spectral cameras detect blood and scale damage on fish

Using his background of working with spectral cameras to detect blood in human medicine, Agro Food Robotics spectroscopy expert Joseph Peller suggested to use those same wavelengths of light to highlight the fish damages.

The team scanned a load of North Sea plaice and let a Wageningen Marine Research expert classify the condition of the fish based on external damages in the images. In the images taken with a regular RGB camera, the skin pigmentations on the plaice could be confused with blood, both being about the same apparent colour. But with the team’s new spectral camera setup the results were completely different: not only was it very easy to separate blood from the fish’s normal skin colour, other damages, such as damage to the scales, could also be identified successfully. All of this was possible using a spectral camera with just five wavelengths of light. The collected images will serve as a dataset to train artificial intelligence to determine various characteristics and defects of the fish (a proxy for survival rate).

“What you gain here is not only objectivity of classification and knowledge, but also speed of the analysis. What is now done manually, could be done automatically in future. The fact that we are going to do this on board a fishing vessel is completely new,” Angelo says.

Bringing electronics to the most hostile environment after space

Spectral cameras have been widely used in laboratory and industrial work, but manufacturers have only started hardening spectral cameras in the past 10-15 years when their use in drones increased. Bringing electronics on board a fishing vessel poses its own further challenges. The newest spectral cameras might be more robust, but they’re still not suited for use on board a fishing vessel. “This is one of the dirtiest, saltiest, and shakiest places in the world, and all three of these are miserable for cameras. So a lot of our research now is on finding a way to bring our system on board a vessel and not have it immediately fall apart,” Joseph explains.

The team are looking into waterproof and airtight lenses with a fixed focus to deal with the on-board vibrations, and a special construction around the camera that has the camera isolated and pointing down, so as to reduce salt deposition. Angelo: “It’s really no joke, this is the most hostile environment for electronics after space. And we also have to remember simple little things. For example, how will the fishers take a picture? They cannot type on a keypad or push a tiny button with a fishy, gloved hand. So we have to think of a separate button that can be smashed with a fist or even an elbow to activate the camera system. And it has to be simple yet robust: it cannot completely move the camera calibration.” Testing is currently being done in a laboratory setting. The team will then move to the harbour as a second step and finally Wageningen Marine Research will run their first trials with the camera system on a commercial fishing vessel to see how well it works and collect images for the dataset.

The two researchers agree that there is strong potential in using spectral imaging in fisheries. “The current project focuses on fish damage, but we are already starting a separate project where we look into non-destructive fish quality measurements, and use a spectral camera to look in the infrared to see fat bands to quantify the amount of fat in a fish,” says Joseph.