Abstract This article presents a non-invasive fully automatic procedure for Bluefin Tuna sizing, based on a stereoscopic vision system and a deformable model of the fish ventral silhouette. An image processing procedure is performed on each video frame to extract individual fish, followed by a fitting procedure to adjust the fish model to the extracted targets, adapting it to the bending movements of the fish. The proposed system is able to give accurate measurements of tuna snout fork length (SFL) and widths at five predefined silhouette points without manual intervention. In this work, the system is used to study size evolution in adult Atlantic Bluefin Tuna (Thunnus Thynnus) over time in a growing farm. The dataset is composed of 12 pairs of videos, which were acquired once a month in 2015, between July and October, in three grow-out cages of tuna aquaculture facilities on the west Mediterranean coast. Each grow out cage contains between 300 and 650 fish on an approximate volume of 20 000 m3. Measurements were automatically obtained for the 4 consecutive months after caging and suggest a fattening process: SFL shows an increase of just a few centimetres (2%) while the maximum width (A1) shows a relative increase of more than 20%, mostly in the first 2 months in farm. Moreover, a linear relation (with coefficient of determination R2 > 0.98) between SFL and widths for each month is deduced, and a fattening factor (F) is introduced. The validity of the measurements is proved by comparing 15 780 SFL measurements, obtained with our automatic system in the last month, versus ground truth data of a high percentage of the stock under study (1143 out of 1579), obtaining no statistically significant difference. This procedure could be extended to other species to assess the size distribution of stocks, as discussed in the article. Introduction The early detection of impacts from natural and anthropogenic activities is very important to the sustainability of the marine environment. Fishing, climate change and pollution have high implications for fish stocks. Reliable fish measurements like length, height and width can be a very important indicator of the health of wild fish stocks (Dunbrack, 2006; Rosen et al., 2013; Shortis et al., 2013; Shafait et al., 2017). Sampling methods to take fish measurements that involve capturing and handling live fish must be discarded, because they not only cause fish stress and possible death but also hinder the achievement of a large number of measurements. Monitoring wild fish stock and inspection in aquaculture require extremely gentle handling of the target to avoid damage. Thus underwater computer vision systems have been frequently used, as reported in recent reviews (Zion, 2012; Shortis et al., 2013; Mallet and Pelletier, 2014; Boutros et al., 2015; Hao et al., 2016; Saberioon et al., 2016), because it is a very appropriate non-intrusive method that permits work even when the fish are alive. In the particular case of using stereoscopic vision systems (two cameras in a side-by-side arrangement), the following applications have been achieved: fish sizing (Ruff et al., 1995; Tillett et al., 2000; Lines et al., 2001; Harvey et al., 2003; Costa et al., 2006; Dunbrack, 2006; Torisawa et al., 2011; Letessier et al., 2015; Williams and Lauffenburger, 2016); fish counting and sizing (Costa et al., 2009; Rosen et al., 2013); fish sizing in combination with acoustic techniques (Sawada et al., 2009; Espinosa et al., 2011; Kloser et al., 2011); fish farm automation (Martinez-de Dios et al., 2003); wild fish stock assessment (Willis and Babcock, 2000; Watson et al., 2009; Harvey et al., 2012; Langlois et al., 2012; Seiler et al., 2012; Smale et al., 2012; Zintzen et al., 2012; Wakefield et al., 2013; Santana-Garcon et al., 2014; McLaren et al., 2015). Nevertheless, vision sensors and image processing methods have to overcome limited visibility, temporal and spatial variations in lighting, varying distances and relative orientations between cameras and objects, motion and density of the monitored targets, and even lack of physical stability. All these conditions represent a very demanding challenge, which have limited the development of fully automatic commercial solutions. In fact, most of the aforementioned applications are manual or semi-automatic and require human intervention in some of their stages. In regard to biomass estimation, the most widely used commercial systems are AQ1 AM100 (Phillips et al., 2009) and AKVAsmart, formerly VICASS (Shieh and Petrell, 1998), which belong to the semi-automatic category. In both systems, human operators must inspect the videos and select frames in which the fish is isolated and straight, to then manually mark fish snout and fork in both stereo images to estimate its length. The International Commission for the Conservation of Atlantic Tunas (ICCAT) establishes a catch reporting system which covers the full chain of Atlantic Bluefin Tuna (ABT) fishery from capture to marketing (Costa et al., 2009). The use of a stereoscopic system to estimate catch quotas is established in (ICCAT, 2015). The number of individuals, counted during the transfer from tow to grow-out cages, is multiplied by the average weight of a subsample of the stock to derive the total biomass per tow cage. As mentioned before, current stereoscopic vision systems need human operation, making the process slow and laborious, and introduce the variability of manual measuring in the biomass estimation. Therefore, a vision-based automatic procedure for ABT biomass estimation is required. One of the most significant aspects for farmers, biologists and researchers would be the definition of growing models for different species (Aguado-Gimenez and Garcia-Garcia, 2005), but periodic systematic monitoring is required. Aquaculture farms are a good environment for this purpose. Species such as tuna and salmon are most commonly farmed due to market acceptance and rapid growth (Shortis, 2015; Sture et al., 2016). Monitoring would provide information on abnormal growth so that the causes such as parasites, stress caused by environmental conditions and diseases could be tackled. Additionally, fish behaviour depending on size or seasonal habits could also be studied and feed regimens and harvest strategies in aquaculture could be optimized. As indicated in (Harvey et al., 2003), collecting numerous, precise accurate data on length or age without the need to physically handle live fish has been identified as an urgent requirement for fisheries and aquaculture managers. Some authors, such as (Lines et al., 2001; Zion, 2012; Shortis et al., 2013; Atienza-Vanacloig et al., 2016; Shafait et al., 2017), highlight the need for fully automatic methods for underwater video processing. The automatic identification of a single fish is an essential step in achieving a fully automatic process. However, body bending while free-swimming means the same individual is observed with very different shapes, sizes and orientations depending on the visualized frame. So, robust fish detection methods dealing with these variations are required (Lines et al., 2001). In Atienza-Vanacloig et al. (2016) a deformable adaptive model based on computer vision methods that automatically fit the body ventral silhouette of Bluefin Tuna while swimming was proposed. This model achieved very high success rates (up to 90%) identifying individuals in complex images acquired in real conditions. The main advantages of using silhouette model fitting are the following: (i) foreground fish in crowded images can be detected (ii) fish can be measured even in images with bad segmentation due to noise or variable lighting, and (iii) fish direction and body bending can be deduced. When the target has been identified and characterized, 3D biometric measurements can be obtained from a calibrated stereo vision system. Moreover, the validity of any semi-automatic or automatic procedure must be demonstrated by comparing the results with ground truth measurements. In this work, we present a fully automatic procedure based on a stereoscopic vision system and a deformable model of the Bluefin Tuna ventral silhouette to estimate length and widths. Our proposal can provide accurate measurements under real conditions and without human intervention, as shown by comparing the results with ground truth data. Furthermore, fish growing and fattening models are deduced analysing the data collected by us through systematic periodic acquisition for 4 consecutive months in grow-out cages. Although this article is mainly focused on achieving good precision in biometric measurements of fish in cages, the work is currently being adapted so that biomass can be estimated during the fish transfer process. Material and methods The computer vision algorithms involved in the process of fish sizing are described and summarized in Figure 1, as well as the offline manual and supervised operations we performed to check the goodness of our algorithms. Figure 1. View largeDownload slide Sequence of processes performed automatically in our proposal, in the first row, and the results of each step, in the second row. FEI is a coefficient that represents the goodness of the model fitting. Figure 1. View largeDownload slide Sequence of processes performed automatically in our proposal, in the first row, and the results of each step, in the second row. FEI is a coefficient that represents the goodness of the model fitting. Video acquisition In order to study the evolution of the ABT dimensions, videos were acquired in three grow-out cages in the Grup Balfegó aquaculture facilities. The three grow-out cages are located next to each other and 2.5 miles off the port of l’Ametlla de Mar (Spain). The cages are cone shaped with a base of 50 m of diameter in the water surface and 30 m high, that is an approximate volume of 20 000 m3. The recordings were taken using the AM100 stereovision system (www.aq1systems.com). It uses two Gigabit Ethernet cameras, with image resolution of 1360 × 1024 pixels and framerate of 12 fps. The cameras are mounted in an underwater housing, with a baseline of 80 cm and an inward convergence of 6°. The system is rated to 40 m deep and has an umbilical cable that supplies power to the cameras and transfers images to a logging computer, which generates synchronized left and right videos. The recordings were taken once a month in 2015 between July and October for each grow-out cage, using the same AM100 stereovision system. The resulting dataset consists of 12 pairs of videos, one per month per cage, of 120 minutes duration each one, enough to extract a statistically representative amount of measurements. The cameras were positioned 15 m deep in the grow-out cages and looking towards the surface to obtain a ventral silhouette of the fish (Figure 2). This camera arrangement has three advantages: first, with this orientation, the sunlight acts like a backlight system so objects are always darker than water; second, in this set up, body bending can be clearly appreciated and dealt with; third, the most reliable measurements are obtained when the fish are swimming in a plane orthogonal to the visual axis (Dunbrack, 2006). The acquired videos are processed automatically using the computer vision algorithms described. Figure 2. View largeDownload slide (a) Sensors platform in grow-out cages, including the AM100 stereoscopic vision system used for this study. (b) Snapshot of recordings with the AM100. Figure 2. View largeDownload slide (a) Sensors platform in grow-out cages, including the AM100 stereoscopic vision system used for this study. (b) Snapshot of recordings with the AM100. Stereo vision system calibration Images for calibration were acquired in a tank containing seawater at IEO (Spanish Oceanographic Institute) facilities in Mazarrón (Spain). A 1.40 × 1.10 m checkerboard pattern was guided from −45° to 45° with respect to the optical axis and moved between 1 and 10 m away from the cameras. The MATLAB Stereo Calibration Application based on (Heikkila and Silven, 1997) and (Zhang, 2000) was used to estimate the calibration parameters. The diagonal length of the checkerboard pattern was computed in 5018 stereo images to analyse our calibration accuracy in terms of proportional error between true and measured lengths. 95.91% of the measurements fall within a margin of error of 1% and 100% of the measurements fall within a margin of error of 3% for all ranges. Measurements of a scale bar with known length (1.5 m) are done over a range of distances (2–10 m) before each recording to guarantee that the camera calibration is still valid. Processing frames: segmentation, filtering, and tuna model fitting In this article, a variant of the tuna model presented in (Atienza-Vanacloig et al., 2016) was implemented to achieve our objective of estimating biometric measurements. Segmentation and filtering process Image segmentation was implemented using local thresholding technique (Petrou and Petrou, 2011) to extract individual objects (fish) from video frames. Local thresholding examines statistically the intensity values of the local neighbourhood of each pixel assuming that illumination is approximately uniform in the neighbourhood. In our case, the pixel in i-th row and j-th column of the image is selected as foreground if its intensity value pij is below a local threshold Mij. The local thresholding technique can be expressed as: Mij=1w*w ∑l=i-w2i+w2 ∑k=j-w2j+w2plk; pij≥Mij ; pij is backgroundpij<Mij; pij is foreground; ∀ pij∈Ft (1) where w=25 is the size of the neighbourhood, plk the intensity values of the neighbour pixels and Ft the video frame. Open, close and fill morphological operations complete the segmentation process. The segmented blobs are geometrically characterized and sifted using shape (aspect ratio), pixel density and dimensional filters. An edge detection algorithm is then applied and a fitting of the deformable model of the fish, defined as a nonlinear multivariable function, is obtained using a minimization algorithm. Deformable tuna model and fitting process Figure 3e shows the deformable model M of tuna fish defined in (Atienza-Vanacloig et al., 2016) as a vector of five parameters M=[sx, sy, l, α, θ] where: sx and sy give the image location of the snout tip; l is the length of the vertebral column; α denotes the angle of the fish head in relation to the horizontal axis, and θ is the bending angle of the vertebral column. The capabilities of this fish model have been increased, from discrimination of individuals to accurate measuring: (i) the number of vertebral points has been increased from 16 to 18 and are now not equidistantly distributed along the fish length l, (ii) more points are concentrated in the tail, a crucial zone for length measurements, (iii) the area around the pectoral fin is not considered, as its many shapes can hinder model fitting, and (iv) a new width vector parameter w, containing a width coefficient for each vertebral point, has been added to already existing model parameters. While in the deformable model presented in (Atienza-Vanacloig et al., 2016), the width is considered a function of length with constant coefficients, our model assigns a variable-bounded coefficient for each vertebral point. Figure 3. View largeDownload slide Image processing steps: (a) original image, (b) image segmentation, (c) edge detection, (d) deformable model fitting with the 18 non equidistant vertebral points and their respective profile points (landmarks), (e) deformable tuna model presented in Atienza-Vanacloig et al. (2016), (f) graphical representation of the ML, SFL, and the five widths defined to study the fattening evolution, (g) polynomial fitting for the ML–SFL relation, the continuous line is the linear fitting and the dashed lines are the 95% CI. Figure 3. View largeDownload slide Image processing steps: (a) original image, (b) image segmentation, (c) edge detection, (d) deformable model fitting with the 18 non equidistant vertebral points and their respective profile points (landmarks), (e) deformable tuna model presented in Atienza-Vanacloig et al. (2016), (f) graphical representation of the ML, SFL, and the five widths defined to study the fattening evolution, (g) polynomial fitting for the ML–SFL relation, the continuous line is the linear fitting and the dashed lines are the 95% CI. This new model is defined as a vector of six parameters Mw=[sx, sy, l, α, θ, w] where w is a vector of coefficients for width fitting. The model is characterized by 18 vertebras vi=(xiv, yiv) distributed along the fish length, whose position is computed according to the parameters using the following equation: xiv yiv=SxSy +cos α-sin αsin αcos αli cos (θi)li sin (θi) (2) where li is the length from the snout to the i-th vertebra and θi the bending angle of the i-th vertebra. The model consists of 35 landmarks, 1 landmark for the snout tip and 17 landmarks for each side of the tuna body profile. The landmarks ki=xik, yik that configure the Mw model silhouette are obtained from the vertebral points vi with the following expressions: xik=xiv ± wi li ci sin θiyik= yiv ± wi li ci cos θi i=1…n (3) where the positive or negative sign depends on the side of the tuna body profile, while ci is the i-th coefficient from a constant vector defining the distance from vertebras to landmarks and wi is the width coefficient of i-th vertebra. A fitting error index (FEI), based on the quadratic distance between the model points and the target edges points, is computed to analyse the goodness of the fitting. FEI takes values in the [0.10] range, where FEI = 0 denotes a perfect fit between the segmented blob and the theoretical model Mw. Fittings with high values (FEI > 6) are discarded. See (Atienza-Vanacloig et al., 2016) for further details on the model definition and fitting procedure. Figure 3a–d shows the image processing steps. The model comprises from the fish snout to the end of the caudal peduncle keel, as the caudal fin cannot be modelled due to its great variability. A set of blobs with good FEI, that is, with good model fitting, is provided after the processing frames stage. Up to this point, the videos acquired by both cameras are processed independently, as can be seen in Figure 1. Stereo correspondence The results for left and right videos, obtained separately in “Processing frames: segmentation, filtering, and tuna model fitting” Section, can be merged to calculate fish measurements if the same individuals can be identified in both videos. With the calibration described in “Stereo vision system calibration” Section the relative position and orientation of the two cameras is known, so the following epipolar geometry restriction can be used: given two characteristic points of the fish model, like snout and tail, in one image, the matching points in the other image must lie on the epipolar line defined by the calibration parameters. The solution is not unique in the image plane so geometrical filters must be added to guarantee the correspondence. Only the samples with similar model parameters (length, orientations, bending and widths) are considered. Length and widths: 3D measurements When stereo correspondence has been guaranteed, the image plane information can be transformed to 3D measurements using the calibration matrices and 3D triangulation. Fish are deformable due to the swimming motion and, consequently, measurements taken from a single frame may not be reliable (Shortis et al., 2013). Two main options are used in the literature to reduce the effect of swimming motion on length measurement: (i) take measurements in all frames and deduce straight body length from a sinusoid-like pattern (Shortis et al., 2013); (ii) account for body bending by adding contiguous linear segments (Williams and Lauffenburger, 2016). In our case, the swimming length problem is resolved using the tuna model bending angle θ, by identifying as valid samples the ones whose vertebral points form a straight line and discarding the others. model length (ML) is computed as the Euclidean distance between the 3D coordinates of the snout and tail fork model points. As explained in “Processing frames: segmentation, filtering, and tuna model fitting” Section and as can be appreciated in Figure 3a and b, the caudal fin cannot be modelled due to its great variability, so a relation between snout fork length ( SFL), usually used in the literature, and ML is needed. For this purpose, 279 samples from within the automatic measurements were selected with the following requirements: the tail fork must be clearly identifiable and aligned with the snout and tail model points, as shown in Figure 3f. For these samples, the tail fork point was manually selected to have SFL. A polynomial fitting was done, resulting in an SFL-ML linear relation as shown in Equation (2) and Figure 3g. SFL=1.0312 ML+0.065641 (4) For the case of fish width measurements, the 3D coordinates of the model points that are symmetrically paired to the vertebral column could be used, see Figure 3d. However, these points are influenced more by the camera perspective because they are not extreme points in the silhouette. Triangulation may lead to major errors if the matching points in both images do not correspond to the same actual point. Therefore, the proposal for width measurements differs from the one for SFL measurements. In this case, only the model with better FEI from the left or right image is considered. The pairs of characteristic model points defining the widths are transformed to the 3D space assuming each pair is at the same distance from the cameras. For this study, five fish widths Ai are considered, whose location in the model can be seen in Figure 3f. The distance to the cameras Zi for each width Ai is computed with the following expression: Zi=Zs+lil(Zt-Zs) (5) where Zs and Zt, are Z coordinates of snout and tail fork, l is ML, and li is length from the snout to the vertebral axis corresponding to Ai. Note that Equation (3) represents the equation of a line between Zs and Zt and the calculated distance depends on the position in the model of the pair of points associated to each width Ai. Once the points are in the 3D space, the Euclidean distances for each pair of points are selected as the fish 3D widths. Results The dataset was recorded with a stereoscopic system for 4 consecutive months immediately after caging, from July to October 2015, on three grow-out cages. Targets were extracted from a total of 12 pairs of videos of 120 min duration and around 100 000 frames each. Our tuna model had a successful fitting, i.e. good FEI, in individual images in more than 1.4 million blobs, and more than 100 000 3D measurements where obtained after stereo correspondence. Redundant information in the statistical distribution is intrinsic to the case of grow-out cage monitoring due to sample repetition, but its impact decreases with the number of measurements. Moreover, the stock under study is considered almost constant, as the population only changes considerably in one cage and month (cage 1 in October). Table 1 summarizes the number of video frames, number of good model fitting samples, automatic 3D measurements, and number of fish in cages when the recordings were taken. The videos of cage 2 in August were corrupted, so no information can be extracted from them. Table 1. Number of video frames, number of good model fitting samples for left and right videos, automatic 3D measurements (when stereo correspondence is ensured) and number of fish in cages when the recordings were taken. Number of frames Good model fitting 3D measurements/Number of fish in cages Left Right Cage 1 Cage 2 Cage 3 Total July 648 180 212 562 231 973 3923/646 20 143/647 7651/647 31 717 / 1940 August 324 750 184 316 192 719 8180/636 – 17 209/626 25 389 / 1893 September 646 220 199 090 181 340 3996/538 15 208/625 12 011/625 31 215 / 1792 October 650 210 124 712 129 521 6706/326 3365/624 5709/624 15 780 / 1579 Total 2 269 360 720 680 735 553 22 805 38716 42580 104 101 Number of frames Good model fitting 3D measurements/Number of fish in cages Left Right Cage 1 Cage 2 Cage 3 Total July 648 180 212 562 231 973 3923/646 20 143/647 7651/647 31 717 / 1940 August 324 750 184 316 192 719 8180/636 – 17 209/626 25 389 / 1893 September 646 220 199 090 181 340 3996/538 15 208/625 12 011/625 31 215 / 1792 October 650 210 124 712 129 521 6706/326 3365/624 5709/624 15 780 / 1579 Total 2 269 360 720 680 735 553 22 805 38716 42580 104 101 It should be noticed that this work focuses on obtaining as many samples as possible, so computing time is not an issue. The overall process shown in Figure 1 had a computing time of around 5 h for each cage and month. As further work, both the code and the algorithms will be optimized to adapt the proposal to applications where computing time is important, like biomass estimation in transfers, measurements, and number of fish in cages when the recordings were taken. The study comprises three grow-out cages from July to October. The data collected over 4 months have been processed using the computer vision algorithms described in “Material and methods” Section, and the resulting 3D measurements of 104 101 fish are analysed in this section with the following structure: evolution. Ai evolution. SFL— A1 ratio evolution. Fattening factor. Measurements validation. At least 3000 samples have been extracted for each cage and month, so the results are considered statistically significant. Measurements are validated by comparing ground truth data from harvests in October with automatic measurements in that month. SFL evolution Figure 4 shows normalized SFL frequency histograms and SFL means ( SFL¯) for each month and cage. Distributions are very similar for all months and SFL¯ variation for the 4 months is of only a few centimetres (2%). The great majority of the samples are located in the interval SFL ϵ [1.70,2.60]. The same results seem to apply to cage 2, despite the missing data in August. Figure 4. View largeDownload slide Normalized SFL frequency histograms and evolution of SFL means ( SFL¯) for each month and cage under study. Figure 4. View largeDownload slide Normalized SFL frequency histograms and evolution of SFL means ( SFL¯) for each month and cage under study. Widths (Ai) evolution The five fish widths defined in Figure 3f are considered. The normalized frequency histogram for each width Ai and month, and the evolution of the mean widths A-i are shown in Figure 5 for cages 1 and 3. It can be seen that the form of the distribution is similar over months and the evolution of the widths is different for each point: whereas A2 and A5 remain almost constant, A1, A3, and A4 show a clear fattening evolution. Moreover, those widths increase most in the first two months in the cages. Similar results apply to cage 2, although the representation of those results has been omitted due to the lack of information in August for cages 1 and 3. Figure 5. View largeDownload slide Normalized frequency histograms for each width Ai and evolution of mean widths A-i over months for cages 1 and 3. Figure 5. View largeDownload slide Normalized frequency histograms for each width Ai and evolution of mean widths A-i over months for cages 1 and 3. SFL—maximum width (A1) ratio evolution The study of the SFL-widths ratio evolution is focused on the maximum width A1 and for SFL ϵ [1.70,2.60], because the evolution is similar for all widths that vary over time and there are few samples outside that length interval. The relation between SFL and A1 over months can be seen in Figure 6 on the left column, a scatter plot with the data of the first and last study months (July and October) is shown; on the right, SFL is split in intervals of 5 cm and the mean width for each interval is calculated. A strong linear relation can be observed for all months. The fitting to a linear model and the coefficient of determination R2 is summarized in Table 2. Table 2. Linear fitting of the relation between SFL and maximum width ( A1) over months. July August September October Cage 1 A1 = 0.164· SFL + 0.059 A1 = 0.145· SFL + 0.125 A1 = 0.194· SFL + 0.062 A1 = 0.195· SFL + 0.060 R2 = 0.985 R2 = 0.996 R2 = 0.995 R2 = 0.981 Cage 2 A1 = 0.141· SFL + 0.106 – A1 = 0.218· SFL + 0.156 A1 = 0.222· SFL + 0.015 R2 = 0.988 – R2 = 0.991 R2 = 0.985 Cage 3 A1 = 0.159· SFL + 0.069 A1 = 0.175· SFL + 0.067 A1 = 0.189· SFL + 0.071 A1 = 0.188· SFL + 0.083 R2 = 0.990 R2 = 0.993 R2 = 0.998 R2 = 0.993 July August September October Cage 1 A1 = 0.164· SFL + 0.059 A1 = 0.145· SFL + 0.125 A1 = 0.194· SFL + 0.062 A1 = 0.195· SFL + 0.060 R2 = 0.985 R2 = 0.996 R2 = 0.995 R2 = 0.981 Cage 2 A1 = 0.141· SFL + 0.106 – A1 = 0.218· SFL + 0.156 A1 = 0.222· SFL + 0.015 R2 = 0.988 – R2 = 0.991 R2 = 0.985 Cage 3 A1 = 0.159· SFL + 0.069 A1 = 0.175· SFL + 0.067 A1 = 0.189· SFL + 0.071 A1 = 0.188· SFL + 0.083 R2 = 0.990 R2 = 0.993 R2 = 0.998 R2 = 0.993 Figure 6. View largeDownload slide On the left, scatter plots of SFL and maximum width ( A1) for first and last study months. On the right, relation between SFL and A1- over months. SFL ϵ [1.70,2.60] is split in intervals of 5 cm and the mean width for each interval is calculated. Figure 6. View largeDownload slide On the left, scatter plots of SFL and maximum width ( A1) for first and last study months. On the right, relation between SFL and A1- over months. SFL ϵ [1.70,2.60] is split in intervals of 5 cm and the mean width for each interval is calculated. Fattening factor F As can be seen in Figure 7, the evolution over time of maximum width A1 for different SFL is very similar, so a global fattening factor can be defined independently of SFL: F=A1M¯A1J¯ (6) where A1M¯ is the mean A1 for each month, and A1J¯ mean A1 in July. Its evolution over time can be seen on the last subplot in Figure 7, the fattening factor increases mostly, and almost linearly, in the first 2 months and less in the third month. Figure 7. View largeDownload slide (a–c) Mean maximum width ( A1-) for SFL intervals over months. SFL ϵ [1.70, 2.60] is split in intervals of 5 cm and the mean width for each interval is calculated. (d) Fattening factors over months. Line dashed to represent the missing data in August. Figure 7. View largeDownload slide (a–c) Mean maximum width ( A1-) for SFL intervals over months. SFL ϵ [1.70, 2.60] is split in intervals of 5 cm and the mean width for each interval is calculated. (d) Fattening factors over months. Line dashed to represent the missing data in August. Measurements validation To validate the procedure, the system measurements and ground truth data are compared. The ground truth data is provided by Grup Balfegó, which measures SFL of the fish in the cages at harvesting. They are dated mostly between October and December, so it was decided to compare them with automatic measurements of the recordings in October. The analysis is also run on pooled data, although the fish in each cage constitute an independent stock and merging data from different stocks can lead to differences in distributions. Figure 8 shows the normalized SFL frequency histograms of the automatic measurements and ground truth data, for each cage and with pooled data. Differences in SFL¯ among harvests and automatic measurements were examined with analysis of variance tests. Since the two groups have unequal sample sizes and homoscedasticity (homogeneity of variance) cannot be ensured, Welch’s ANOVA test (Welch, 1951) is used, as recommended in Rasch et al. (2011) and McDonald (2014). The differences in SFL frequency distributions are analysed with the Kolmogorov-Smirnov test (Massey, 1951). Figure 8. View largeDownload slide Normalized SFL frequency histograms. Ground truth in dark-blue and automatic measurements in light-yellow. SFL¯ is the mean SFL, f the number of fish and n the number of samples. Figure 8. View largeDownload slide Normalized SFL frequency histograms. Ground truth in dark-blue and automatic measurements in light-yellow. SFL¯ is the mean SFL, f the number of fish and n the number of samples. As Table 3 shows, the tests for SFL¯ give p-values higher than the 5% significance level for each cage and with data pooled, and the tests for SFL distribution frequency give p-values higher than the 5% significance level, except when the cages are pooled. In conclusion, there is no statistically significant difference between ground truth and automatic measurements, thereby validating the measurements obtained with the proposed automatic system. Table 3. Automatic system measurements vs ground truth statistical comparison in the three grow-out cages and with data pooled. Cage 1 Cage 2 Cage 3 Pooled No. fish 326 629 624 1579 No. harvests (ground truth) 316 511 316 1143 No. automatic measurements 6706 3365 5709 15 780 Welch’s ANOVA test p-value 0.9928 0.7793 0.4118 0.3884 Kolmogorov-Smirnov test p-value 0.3553 0.2944 0.3075 0.0183 Cage 1 Cage 2 Cage 3 Pooled No. fish 326 629 624 1579 No. harvests (ground truth) 316 511 316 1143 No. automatic measurements 6706 3365 5709 15 780 Welch’s ANOVA test p-value 0.9928 0.7793 0.4118 0.3884 Kolmogorov-Smirnov test p-value 0.3553 0.2944 0.3075 0.0183 Discussion The need for a fully automatic system to accurately estimate the length of free swimming fish with a non-intrusive procedure has often been pointed out in recent years (Costa et al., 2009; Zion, 2012; Shortis et al., 2013; Rosen et al. 2013; Williams and Lauffenburger, 2016; Shafait et al., 2017). Fish length information is an important indicator of the health of wild fish stocks and for predicting biomass using length-weight relations (Lines et al., 2001; Martinez-de Dios et al., 2003). The most common mathematical model between fish length ( L) and mass ( W) is W=aLb, where a and b are parameters dependent on fish species (Zion, 2012) and on growth, in captivity or wild, (Aguado-Gimenez and Garcia-Garcia, 2005; Katavić et al., 2016). The total biomass of a fish stock is commonly determined by obtaining the mean length of a statistically representative number of fish and counting the number of fish (Costa et al., 2009; Shafait et al., 2017). The proposed automated system allowed us to process more than 2 million video frames, producing more than 100 000 3D length and width measurements. Stereo-cameras were positioned 15 m deep in the grow-out cages with fish measurements ranging from 3 to 10 m. The limitations of using computer vision, namely high turbidity in water and crowded fish situations, were revealed and the videos in November were dismissed because of poor water visibility. The results demonstrate highly accurate SFL estimation and validate the automatic procedure. As Figure 8 and Table 3 show, there is no statistically significant difference between ground truth and automatic measurements. The periodicity of our recordings on the same individuals and the large number of samples collected, more than 3000 per cage and month, enables us to analyse evolution over time (four months) of the length and width measurements. This analysis may be of use for solving some paradigms of interest regarding ABT for farmers, biologists and researchers such as: How does SFL evolve over time? The obtained SFL¯ variation presents an increase of only 2% from first to last month (Figure 4). How do widths Ai evolve over time? The obtained evolution differs depending on the fish body section considered: whereas no increasing is shown in head and caudal peduncle keel sections ( A2,A5), sections between the pectoral fin and caudal peduncle keel ( A1,A3, A4) show clear increasing, mostly in the first two months (Figure 5). Is there any relation between SFL and A1? A strong linear relation has been observed: high coefficients of determination R2 for linear model fitting have been obtained for all months (Figure 6, and Table 2). Can a fattening factor for tuna in grow-out cages be established? Fattening factor F, defined as the relative increase over time of maximum width A1, shows a fattening evolution that increases almost linearly in the first two months and less in the third month (Figure 7) for SFL ϵ [1.70,2.60]. A simulation of fattening evolution according to fattening factor F is shown in Figure 9. Figure 9. View largeDownload slide Bluefin Tuna fattening evolution according to the calculated fattening factor. The image corresponds to a fish in July, the continuous white line is the model fitting in that month, and the dashed line is the simulation of the fattening evolution for the last month. Figure 9. View largeDownload slide Bluefin Tuna fattening evolution according to the calculated fattening factor. The image corresponds to a fish in July, the continuous white line is the model fitting in that month, and the dashed line is the simulation of the fattening evolution for the last month. Automatic extraction of a large number of silhouettes, precise tuna model fitting and accurate 3D measurements were a priority in our developments, without paying much attention to processing time. Working with tuna in cages has the advantage of being able to record the time necessary to obtain a good statistical representation of the stock. Thus, on a two hours recording, the automatic system estimates on average about 10 000 fish measurements ( SFL and five widths Ai) with a computational cost of 5 h (1.38 s per sample). We are sure it will be possible to improve the processing time, but currently it is obviously much lower than the time necessary to obtain the same measurements with a manual or semiautomatic application. The whole fully automatic process is the main difference of this work with respect to other studies with similar goals. Also the following aspects should be highlighted: Our measurements have been validated with a large number of measurements (15 780), a large amount of ground truth data (1143 harvests out of 1579 fish), and wide measuring range (from 3 to 10 m). Other authors obtained good results measuring fish lengths with stereovision systems, but their proposals have one or several of the following common limitations: measurements are not extracted fully automatically, measurements are taken in a narrow range, the number of measurements is relatively small, or the ground truth comprises only a few samples. In fact, (Lines et al., 2001) reported that the linear dimension of salmon in sea cages could be extracted automatically with a mean error below 10%, but they work with only 60 images of 17 fish and measure in a range from 1 to 2 m. (Harvey et al., 2003) predicted the SFL of Southern Bluefin Tuna (SBT) inside a cage with a relative error of 0.16% (with SFL from 830 to 1412 mm), but harvesting only 54 SBT from thousands in the cage and measuring in a range of about 1 m. (Shafait et al., 2017) present a semiautomatic method for estimating the fish lengths of 22 138 SBT in transfers in a range from 1 to 4 m, but it is not fully automatic and the results are compared with manual measurements and not ground truth data. Fully automatic estimation of five widths in addition to SFL. Recent studies attempt to show that biomass can be better estimated if fish measurements in dimensions other than length, like width and depth, are available (Harvey et al., 2003; Aguado-Gimenez and Garcia-Garcia, 2005). Nevertheless, as stated in (Harvey et al., 2003), measuring the width of a fish is relatively subjective due to the lack of defined points in the fish silhouette. Those authors use simple cursor positioning and mouse clicks to measure maximum body depth (MBD). Instead, we use our tuna model features to obtain the maximum width (equivalent to MBD but in width) in the body section close to the pectoral fin (Figure 3f). Our automatic system can produce a lot of SFL and maximum width measurements in a relatively short time, so a statistical distribution with a high number of samples can be obtained, which would allow better biomass estimation. Videos are acquired in real world conditions without using any background screen, contrast element, or reference object, as it is done for example in (Shafait et al., 2017). The acquisition configuration meets the requirements for automatic sizing and counting of tuna in transfers according to (ICCAT, 2015). The stereo-videos were recorded from 15 m deep in grow-out cages with a measuring range from 3 to 10 m. This position and range were selected, as a first approach, to be able to apply this method to fish transfers, where tuna have to pass from transport to grow-out cages through a 10 × 10 m door between cages. Although measuring fish at higher range to the cameras should lead to greater measurement error, the results prove that our automatic system is able to give accurate measurements in that range interval. The present regulations for ABT establish the use of stereoscopic vision systems to estimate catch quotas when the fish are transferred from tow cages to grow-out cages. But the current systems need human operation, making the process slow and laborious, and introduce the variability of manual measuring into the biomass estimation. Therefore, the proposed vision-based fully automatic procedure for Bluefin Tuna individual biomass estimation makes a necessary and valuable contribution. To complete the system and be able to estimate total biomass in transfer operations, an automatic counting procedure is currently under development. Conclusions and further work The proposed procedure might be a significant contribution towards a commercial system for fully automatic Bluefin Tuna biomass estimation. The authors consider this system a potential tool to ensure the reliable accomplishment of catch quotas following ICCAT recommendations and to support farmers, biologists and researchers in important aspects of fish growth and marine environment. It is also reasonable to think that better biomass estimation could be achieved using more dimensions of the tuna than just SFL. Our system estimates SFL and five widths in different sections of the fish silhouette which can be used to compute biomass. As further work, we plan to improve the robustness of the method by adding a time-dependent analysis, as well as other developments, such as: improved segmentation procedures, accurate measurements in bended fish and accurate measurements from other perspectives (not only ventral silhouette). Moreover, we want to combine this computer vision procedure with echosounder information to estimate biomass in more complex situations, such as wild environments and transfers from tow to grow-out cages. Acknowledgements We acknowledge the assistance provided by the Spanish company Grup Balfegó S.L. in supplying boats and divers to acquire underwater video in the Mediterranean Sea. Funding This work was supported by funding from ACUSTUNA project ref. CTM2015-70446-R (MINECO/ERDF, EU). This project has been possible thanks to the collaboration of IEO (Spanish Oceanographic Institute). References Aguado-Gimenez F., Garcia-Garcia B. 2005. Growth, food intake and feed conversion rates in captive Atlantic bluefin tuna (Thunnus thynnus Linnaeus, 1758) under fattening conditions. Aquaculture Research , 36: 610– 614. Google Scholar CrossRef Search ADS Atienza-Vanacloig V., Andreu-García G., López-García F., Valiente-Gonzólez J. M., Puig-Pons V. 2016. Vision-based discrimination of tuna individuals in grow-out cages through a fish bending model. Computers and Electronics in Agriculture , 130: 142– 150. Google Scholar CrossRef Search ADS Boutros N., Shortis M. R., Harvey E. S. 2015. A comparison of calibration methods and system configurations of underwater stereo-video systems for applications in marine ecology. Limnology and Oceanography: Methods , 13: 224– 236. Google Scholar CrossRef Search ADS Costa C., Loy A., Cataudella S., Davis D., Scardi M. 2006. Extracting fish size using dual underwater cameras. Aquacultural Engineering , 35: 218– 227. Google Scholar CrossRef Search ADS Costa C., Scardi M., Vitalini V., Cataudella S. 2009. A dual camera system for counting and sizing Northern Bluefin Tuna (Thunnus thynnus; Linnaeus, 1758) stock, during transfer to aquaculture cages, with a semi automatic Artificial Neural Network tool. Aquaculture , 291: 161– 167. Google Scholar CrossRef Search ADS Dunbrack R. L. 2006. In situ measurement of fish body length using perspective-based remote stereo-video. Fisheries Research , 82: 327– 31. Google Scholar CrossRef Search ADS Espinosa V., Soliveres E., Cebrecos A., Puig V., Sainz-Pardo S., de la Gándara F. 2011. Growing Monitoring in Sea Cages: Ts Measurements Issues. In Proceedings of the 34th Scandinavian Symposium on Physical Acoustics. Geilo, pp. 1– 6. Hao M., Yu H., Li D. 2016. The measurement of fish size by machine vision - a review. In Computer and Computing Technologies in Agriculture IX: 9th IFIP WG 5.14 International Conference, CCTA 2015, Beijing, China, September 27–30, 2015, Revised Selected Papers, Part II 9, pp. 15–32. Harvey E., Butler J. J., McLean D. L., Shand J. 2012. Contrasting habitat use of diurnal and nocturnal fish assemblages in temperate Western Australia. Journal of Experimental Marine Biology and Ecology , 426: 78– 86. Google Scholar CrossRef Search ADS Harvey E., Cappo M., Shortis M., Robson S., Buchanan J., Speare P. 2003. The accuracy and precision of underwater measurements of length and maximum body depth of southern bluefin tuna (Thunnus maccoyii) with a stereo-video camera system. Fisheries Research , 63: 315– 326. Google Scholar CrossRef Search ADS Heikkila J., Silven O. 1997. A Four-step Camera Calibration Procedure with Implicit Image Correction. Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR ’97), CVPR ’97, pp. 1106. Washington, DC, USA: IEEE Computer Society. ICCAT. 2015. Recommendation by ICCAT amending the recommendation 13-07 by ICCAT to establish a multi-annual recovery plan for Bluefin Tuna in the eastern Atlantic and Mediterranean. Rec [14-04]. 2015 Compendium management recommendations and resolutions adopted by ICCAT for conservation of Atlantic tunas and tuna-like species, pp. 47– 82. Katavić I., Šegvić-Bubić T., Grubišić L., Talijančić I. 2016. Reliability of Bluefin Tuna Size Estimates Using A Stereoscopic Camera System. Collective Volume of Scientific Papers ICCAT 72: 1848– 1861. Kloser R. J., Ryan T. E., Macaulay G. J., Lewis M. E. 2011. In situ measurements of target strength with optical and model verification: a case study for blue grenadier, Macruronus novaezelandiae. ICES Journal of Marine Science , 68: 1986– 1995. Google Scholar CrossRef Search ADS Langlois T. J., Harvey E. S., Meeuwig J. J. 2012. Strong direct and inconsistent indirect effects of fishing found using stereo-video: testing indicators from fisheries closures. Ecological Indicators , 23: 524– 534. Google Scholar CrossRef Search ADS Letessier T. B., Juhel J.-B., Vigliola L., Meeuwig J. J. 2015. Low-cost small action cameras in stereo generates accurate underwater measurements of fish. Journal of Experimental Marine Biology and Ecology , 466: 120– 126. Google Scholar CrossRef Search ADS Lines J. A., Tillett R. D., Ross L. G., Chan D., Hockaday S., McFarlane N. J. B. 2001. An automatic image-based system for estimating the mass of free-swimming fish. Computers and Electronics in Agriculture , 31: 151– 168. Google Scholar CrossRef Search ADS Mallet D., Pelletier D. 2014. Underwater video techniques for observing coastal marine biodiversity: A review of sixty years of publications (1952–2012). Fisheries Research , 154: 44– 62. Google Scholar CrossRef Search ADS Martinez-de Dios J. R., Serna C., Ollero a. 2003. Computer vision and robotics techniques in fish farms. Robotica , 21: 233– 243. Google Scholar CrossRef Search ADS Massey F. J. 1951. The Kolmogorov-Smirnov test for goodness of fit. Journal of the American Statistical Association , 46: 68– 78. Google Scholar CrossRef Search ADS McDonald J. H. 2014. Handbook of Biological Statistics ., 3rd ed. Baltimore, Maryland: Sparky House Publishing. McLaren B. W., Langlois T. J., Harvey E. S., Shortland-Jones H., Stevens R. 2015. A small no-take marine sanctuary provides consistent protection for small-bodied by-catch species, but not for large-bodied, high-risk species. Journal of Experimental Marine Biology and Ecology , 471: 153– 163. Google Scholar CrossRef Search ADS Petrou M., Petrou C. 2011. Image segmentation and edge detection. Image Processing: The Fundamentals , pp. 527– 668. John Wiley & Sons, Ltd: Chichester, UK. Google Scholar CrossRef Search ADS Phillips K., Rodriguez V. B., Harvey E., Ellis D., Seager J., Begg G., Hender J. 2009. Assessing the operational feasibility of stereo-video and evaluating monitoring options for the Southern Bluefin Tuna Fishery ranch sector, Fisheries Research and Development Corporation and Bureau of Rural Sciences (Australia). Report 2008/44, ISBN 978-1-921192-32-6. Rasch D., Kubinger K. D., Moder K. 2011. The two-sample t test: pre-testing its assumptions does not pay off. Statistical Papers , 52: 219– 231. Google Scholar CrossRef Search ADS Rosen S., Jörgensen T., Hammersland-White D., Holst J. C., Grant J. 2013. DeepVision: a stereo camera system provides highly accurate counts and lengths of fish passing inside a trawl. Canadian Journal of Fisheries and Aquatic Sciences , 70: 1456– 1467. Google Scholar CrossRef Search ADS Ruff B. P., Marchant J. A., Frost A. R. 1995. Fish sizing and monitoring using a stereo image analysis system applied to fish farming. Aquacultural Engineering , 14: 155– 173. Google Scholar CrossRef Search ADS Saberioon M., Gholizadeh A., Cisar P., Pautsina A., Urban J. 2016. Application of machine vision systems in aquaculture with emphasis on fish: State-of-the-art and key issues. Reviews in Aquaculture , doi: 10.1111/raq.12143 Santana-Garcon J., Newman S. J., Harvey E. S. 2014. Development and validation of a mid-water baited stereo-video technique for investigating pelagic fish assemblages. Journal of Experimental Marine Biology and Ecology , 452: 82– 90. Google Scholar CrossRef Search ADS Sawada K., Takahashi H., Abe K., Ichii T., Watanabe K., Takao Y. 2009. Target-strength, length, and tilt-angle measurements of Pacific saury (Cololabis saira) and Japanese anchovy (Engraulis japonicus) using an acoustic-optical system. ICES Journal of Marine Science , 66: 1212– 1218. Google Scholar CrossRef Search ADS Seiler J., Williams A., Barrett N. 2012. Assessing size, abundance and habitat preferences of the Ocean Perch Helicolenus percoides using a AUV-borne stereo camera system. Fisheries Research , 129: 64– 72. Google Scholar CrossRef Search ADS Shafait F., Harvey E. S., Shortis M. R., Mian A., Ravanbakhsh M., Seager J. W., Culverhouse P. F. et al. 2017. Towards automating underwater measurement of fish length: a comparison of semi-automatic and manual stereo–video measurements. ICES Journal of Marine Science , doi: 10.1093/icesjms/fsx007. Shieh A. C. R., Petrell R. J. 1998. Measurement of fish size in atlantic salmon (salmo salar l.) cages using stereographic video techniques. Aquacultural Engineering , 17: 29– 43. Google Scholar CrossRef Search ADS Shortis M. 2015. Calibration techniques for accurate measurements by underwater camera systems. Sensors , 15: 30810– 30827. Google Scholar CrossRef Search ADS PubMed Shortis M., Ravanbakskh M., Shaifat F., Harvey E. S., Mian A., Seager J. W., Culverhouse P. F. et al. 2013. A review of techniques for the identification and measurement of fish in underwater stereo-video image sequences. Proc. SPIE 8791, Videometrics, Range Imaging, and Applications XII; and Automated Visual Inspection, 87910G, doi: 10.1117/12.2020941 Smale D. A., Kendrick G. A., Harvey E. S., Langlois T. J., Hovey R. K., Van Niel K. P., Waddington K. I. et al. 2012. Regional-scale benthic monitoring for ecosystem-based fisheries management (EBFM) using an autonomous underwater vehicle (AUV). ICES Journal of Marine Science 69: 1108– 1118. Google Scholar CrossRef Search ADS Sture Ø., Øye E. R., Skavhaug A., Mathiassen J. R. 2016. A 3D machine vision system for quality grading of Atlantic salmon. Computers and Electronics in Agriculture 123: 142– 148. Google Scholar CrossRef Search ADS Tillett R., Mcfarlane N., Lines J. 2000. Estimating dimensions of free-swimming fish using 3d point distribution models. Computer Vision and Image Understanding , 79: 123– 141. Google Scholar CrossRef Search ADS Torisawa S., Kadota M., Komeyama K., Suzuki K., Takagi T. 2011. A digital stereo-video camera system for three-dimensional monitoring of free-swimming Pacific bluefin tuna, Thunnus orientalis, cultured in a net cage. Aquatic Living Resources , 24: 107– 112. Google Scholar CrossRef Search ADS Wakefield C. B., Lewis P. D., Coutts T. B., Fairclough D. V., Langlois T. J. 2013. Fish assemblages associated with natural and anthropogenically-modified habitats in a marine embayment: comparison of baited videos and opera-house traps (S. J. Goldstien, Ed.). PLoS One , 8: e59959. Google Scholar CrossRef Search ADS PubMed Watson D., Anderson M., Kendrick G., Nardi K., Harvey E. 2009. Effects of protection from fishing on the lengths of targeted and non-targeted fish species at the Houtman Abrolhos Islands, Western Australia. Marine Ecology Progress Series , 384: 241– 249. Google Scholar CrossRef Search ADS Welch B. L. 1951. On the comparison of several mean values: an alternative approach. Biometrika , 38: 330– 336. Google Scholar CrossRef Search ADS Williams K., Lauffenburger N. 2016. Automated measurements of fish within a trawl using stereo images from a Camera-Trawl device (CamTrawl). Methods in Oceanography , 17: 138– 152. Google Scholar CrossRef Search ADS Willis T. J., Babcock R. C. 2000. A baited underwater video system for the determination of relative density of carnivorous reef fish. Marine and Freshwater Research , 51: 755. CSIRO PUBLISHING, doi: 10.1071/MF00010 Google Scholar CrossRef Search ADS Zhang Z. 2000. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence , 22: 1330– 1334. Google Scholar CrossRef Search ADS Zintzen V., Anderson M. J., Roberts C. D., Harvey E. S., Stewart A. L., Struthers C. D. 2012. Diversity and composition of demersal fishes along a depth gradient assessed by baited remote underwater stereo-video (B. R. MacKenzie, Ed.). PLoS One , 7: e48522. Google Scholar CrossRef Search ADS PubMed Zion B. 2012. The use of computer vision technologies in aquaculture - a review. Computers and Electronics in Agriculture , 88: 125– 132. Google Scholar CrossRef Search ADS © International Council for the Exploration of the Sea 2017. All rights reserved. For Permissions, please email: email@example.com
ICES Journal of Marine Science – Oxford University Press
Published: Jan 1, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.
All for just $49/month
Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly
Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.
Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera