3DPhenoFish: Application for two- and three-dimensional fish morphological phenotype extraction from point cloud analysis
摘要: 鱼类形态表型是水产养殖和生态学研究中人工育种、功能基因定位和群体遗传分析的重要资源。传统的形态学表型测量需要耗费大量的时间与劳动力，更重要的是人工测量高度依赖于操作经验，导致表型测量的结果具有一定的主观性。因此，我们开发了一个可以从三维点云数据中提取鱼类形态表型的软件3DPhenoFish。该软件提供了一个直观的用户界面，将背景剔除、坐标归一化、三维分割、关键点识别和表型提取的功能进行了整合。用户可以自动获取鱼体上18个关键形态点，基于关键点的二维表型，以及鱼体表面积和体积的三维表型。同时，3DPhenoFish还允许用户为自动识别的关键点进行微调，并自定义个性化的表型。基于3DPhenoFish，我们对四种高原特有的裂腹鱼亚科鱼类进行了高通量表型分析，包括拉萨裸裂尻、拉萨河尖裸鲤、双须叶须鱼和异齿裂腹鱼。结果表明，使用3DPhenoFish高通量提取的形态表型与人工测量结果表现出高度的线性相关性（>0.94）。基于高通量提取的形态学表型，我们可以很好地将不同物种进行区分，甚至可以区分同一物种的不同种群。综上所述，我们开发了高效、准确和可定制的鱼类表型分析工具3DPhenoFish，用于从三维点云数据中批量提取形态学表型，有助于克服人工测量中的低通量和高成本的一些传统挑战，因此3DPhenoFish可用于功能基因定位、新品种培育和资源保护研究中的表型分析。3DPhenoFish是一个开源软件，并可以在https://github.com/lyh24k/3DPhenoFish/tree/master免费下载。Abstract: Fish morphological phenotypes are important resources in artificial breeding, functional gene mapping, and population-based studies in aquaculture and ecology. Traditional morphological measurement of phenotypes is rather expensive in terms of time and labor. More importantly, manual measurement is highly dependent on operational experience, which can lead to subjective phenotyping results. Here, we developed 3DPhenoFish software to extract fish morphological phenotypes from three-dimensional (3D) point cloud data. Algorithms for background elimination, coordinate normalization, image segmentation, key point recognition, and phenotype extraction were developed and integrated into an intuitive user interface. Furthermore, 18 key points and traditional 2D morphological traits, along with 3D phenotypes, including area and volume, can be automatically obtained in a visualized manner. Intuitive fine-tuning of key points and customized definitions of phenotypes are also allowed in the software. Using 3DPhenoFish, we performed high-throughput phenotyping for four endemic Schizothoracinae species, including Schizopygopsis younghusbandi, Oxygymnocypris stewartii, Ptychobarbus dipogon, and Schizothorax oconnori. Results indicated that the morphological phenotypes from 3DPhenoFish exhibited high linear correlation (>0.94) with manual measurements and offered informative traits to discriminate samples of different species and even for different populations of the same species. In summary, we developed an efficient, accurate, and customizable tool, 3DPhenoFish, to extract morphological phenotypes from point cloud data, which should help overcome traditional challenges in manual measurements. 3DPhenoFish can be used for research on morphological phenotypes in fish, including functional gene mapping, artificial selection, and conservation studies. 3DPhenoFish is an open-source software and can be downloaded for free at https://github.com/lyh24k/3DPhenoFish/tree/master.
Figure 4. Semantic segmentation for fish point cloud
A: Pre-segment point cloud using super voxel method. B: Fin segmentation is performed using adaptive weighted region growth segmentation. C: Head, eye, body, and fins are segmented from point cloud, then used for following key point recognition and morphological phenotype extraction.
Figure 5. Key point recognition and phenotype extraction
A: Main 2D phenotypes determined from distances among key points estimated directly on background plane of fish point cloud. B: Main 3D phenotypes estimated from point cloud conformation of fish, including arc length, surface, and volume. C: 3D phenotypes for head. Key points recognized in point cloud include snout point (A), front point of eye (B), back point of eye (C), external point of opercular (D), starting point of pectoral fin (E), end point of pectoral fin base (F), lowest point of ventral margin (G), starting point of ventral fin (H), end point of ventral fin base (I), starting point of anal fin (J), end point of anal fin (K), lower point of caudal peduncle (L), end point of coccyx (M), end point of tail fin (N), upper point of caudal peduncle (O), end point of dorsal fin (P), starting point of dorsal fin (Q), and highest point of dorsal margin (R).
Figure 6. Main interface of 3DPhenoFish
A: Main point cloud image viewer. B: List of point clouds that need to be processed. C: Properties of current point cloud. D: List of key points for fish point cloud. E: List of morphological phenotypes for fish point cloud. F: List of operation records. G: Toolbar used to open and save files, adjust visual interface of point cloud, and automatically segment fish point cloud. Morphological phenotype extraction in the software must be executed strictly by down-sampling (Ⅰ), background removal (Ⅱ), and key point recognition (Ⅲ).
Figure 7. Linear correlation analysis of morphological phenotypes from 3DPhenoFish and manual measurement
2D phenotypes were extracted from 3DPhenoFish and manual measurements were collected and compared for 30 randomly selected fish samples. Correlation coefficients of 17 morphological phenotypes were calculated (Supplementary Table S1), including full length (A), body length (B), dorsal snout distance (C), body height (D), caudal peduncle height (E), and head length (F).
Figure 8. Phenotype-based clustering of sample classifications of species and populations using linear discriminant analysis
Samples from Schizopygopsis younghusbandi, Oxygymnocypris stewartii, Ptychobarbus dipogon, and Schizothorax oconnori were used for analysis. Clustering of samples using traditional 2D morphological phenotypes (A) and 2D and 3D morphological phenotypes (B). Clustering of S. younghusbandi samples using traditional 2D morphological phenotypes (C) and 2D and 3D morphological phenotypes (D).
Figure 9. Morphological phenotypes exhibited significant differences among species and Schizopygopsis younghusbandi populations
Distribution of head height/head length (A) and dorsal snout distance/body length (B) for Schizothoracinae species and dorsal arc width/caudal arc width (C) and head volume/head length (D) for S. younghusbandi populations. Significant differences are shown by labels above bars, samples sharing no label letter indicate significant difference between two groups (P≤0.05).
 Aliyu I, Gana KJ, Musa AA, Agajo J, Orire AM, Abiodun FT, et al. 2017. A proposed fish counting algorithm using digital image processing technique. ATBU, Journal of Science, Technology & Education, 5(1): 1−11.  Balakrishnama S, Ganapathiraju A. 1998. Linear discriminant analysis-a brief tutorial. Institute for Signal and information Processing, 18(1998): 1−8.  Balta H, Velagic J, Bosschaerts W, De Cubber G, Siciliano B. 2018. Fast statistical outlier removal based method for large 3D point clouds of outdoor environments. IFAC-PapersOnLine, 51(22): 348−353. doi: 10.1016/j.ifacol.2018.11.566  Bär T, Reuter JF, Zöllner JM. 2012. Driver head pose and gaze estimation based on multi-template ICP 3-D point cloud alignment. In: Proceedings of 2012 15th International IEEE Conference on Intelligent Transportation Systems. Anchorage: IEEE, 1797–1802.  Batanov S D, Starostina O S, Baranova I A. 2019. Non-contact methods of cattle conformation assessment using mobile measuring systems. IOP Conference Series: Earth and Environmental Science, 315(3): 032006.  Besl PJ, McKay ND. 1992. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2): 239−256. doi: 10.1109/34.121791  Bradski G, Kaehler A. 2008. Learning OpenCV: Computer Vision with the OpenCV library. O'Reilly Media, Inc.  Chen XW, Zhu YL, Wu T, Wang ZQ. 2017. The point cloud registration technology based on SAC-IA and improved ICP. J Xi’an Polytech Univ, 31(3): 395−401. (in Chinese)  Comba L, Biglia A, Aimonino DR, Gay P. 2018. Unsupervised detection of vineyards by 3D point-cloud UAV photogrammetry for precision agriculture. Computers and Electronics in Agriculture, 155: 84−95. doi: 10.1016/j.compag.2018.10.005  Fernandes AFA, Turra EM, de Alvarenga ÉR, Passafaro TL, Lopes FB, Alves GFO, et al. 2020. Deep Learning image segmentation for extraction of fish body measurements and prediction of body weight and carcass traits in Nile tilapia. Computers and Electronics in Agriculture, 170: 105274. doi: 10.1016/j.compag.2020.105274  Gongal A, Karkee M, Amatya S. 2018. Apple fruit size estimation using a 3D machine vision system. Information Processing in Agriculture, 5(4): 498−503. doi: 10.1016/j.inpa.2018.06.002  Hao M M, Yu H L, Li D L. 2015. The measurement of fish size by machine vision-a review. In: Proceedings of 9th IFIP WG 5.14 International Conference on Computer and Computing Technologies in Agriculture. Cham: Springer, 15–32.  Hartmann A, Czauderna T, Hoffmann R, Stein N, Schreiber F. 2011. HTPheno: an image analysis pipeline for high-throughput plant phenotyping. BMC Bioinformatics, 12(1): 148. doi: 10.1186/1471-2105-12-148  Huang LW, Li SQ, Zhu AQ, Fan XY, Zhang CY, Wang HY. 2018. Non-contact body measurement for qinchuan cattle with LiDAR sensor. Sensors, 18(9): 3014. doi: 10.3390/s18093014  Le Cozler Y, Allain C, Caillot A, Delouard JM, Delattre L, Luginbuhl T, et al. 2019. High-precision scanning system for complete 3D cow body shape imaging and analysis of morphological traits. Computers and Electronics in Agriculture, 157: 447−453. doi: 10.1016/j.compag.2019.01.019  Li ML, Sun CM. 2018. Refinement of LiDAR point clouds using a super voxel based approach. ISPRS Journal of Photogrammetry and Remote Sensing, 143: 213−221. doi: 10.1016/j.isprsjprs.2018.03.010  Lin GC, Tang YC, Zou XJ, Xiong JT, Li JH. 2019. Guava detection and pose estimation using a low-cost RGB-D sensor in the field. Sensors, 19(2): 428. doi: 10.3390/s19020428  López-Fanjul C, Toro M Á. 2007. Fundamentos de la Mejora Genética en Acuicultura. Madrid: Genética y Genómica en Acuicultura, 155–182.  Navarro A, Lee-Montero I, Santana D, Henríquez P, Ferrer MA, Morales A, et al. 2016. IMAFISH_ML: a fully-automated image analysis software for assessing fish morphometric traits on gilthead seabream (Sparus aurata L.), meagre (Argyrosomus regius) and red porgy (Pagrus pagrus). Computers and Electronics in Agriculture, 121: 66−73. doi: 10.1016/j.compag.2015.11.015  Oliphant TE. 2007. Python for scientific computing. Computing in Science & Engineering, 9(3): 10−20.  Orts-Escolano S, Morell V, García-Rodríguez J, Cazorla M. 2013. Point cloud data filtering and downsampling using growing neural gas. In: Proceedings of the 2013 International Joint Conference on Neural Networks. Dallas: IEEE, 1–8.  Pérez-Ruiz M, Tarrat-Martín D, Sánchez-Guerrero MJ, Valera M. 2020. Advances in horse morphometric measurements using LiDAR. Computers and Electronics in Agriculture, 174: 105510. doi: 10.1016/j.compag.2020.105510  Pezzuolo A, Guarino M, Sartori L, Marinello F. 2018. A feasibility study on the use of a structured light depth-camera for three-dimensional body measurements of dairy cows in free-stall barns. Sensors, 18(2): 673.  Rusu RB, Cousins S. 2011. 3D is here: point cloud library (PCL). In: Proceedings of 2011 IEEE International Conference on Robotics and Automation. Shanghai: IEEE, 1–4.  Schnabel R, Wahl R, Klein R. 2007. Efficient RANSAC for point‐cloud shape detection. Computer Graphics Forum, 26(2): 214−226. doi: 10.1111/j.1467-8659.2007.01016.x  Schroeder WJ, Avila LS, Hoffman W. 2000. Visualizing with VTK: a tutorial. IEEE Computer Graphics and Applications, 20(5): 20−27. doi: 10.1109/38.865875  Shah SZH, Rauf HT, IkramUllah M, Khalid MS, Farooq M, Fatima M, et al. 2019. Fish-pak: fish species dataset from pakistan for visual features based classification. Data in Brief, 27: 104565. doi: 10.1016/j.dib.2019.104565  Spampinato C, Chen-Burger YH, Nadarajan G, Fisher RB. 2008. Detecting, tracking and counting fish in low quality unconstrained underwater videos. In: Proceedings of the 3rd International Conference on Computer Vision Theory and Applications. Madeira: DBLP, 514–519.  Spampinato C, Giordano D, Di Salvo R, Chen-Burger YHJ, Fisher RB, Nadarajan G. 2010. Automatic fish classification for underwater species behavior understanding. In: Proceedings of the 1st ACM International Workshop on Analysis and Retrieval of Tracked Events and Motion in Imagery Streams. Firenze: ACM, 45–50.  Vo AV, Truong-Hong L, Laefer DF, Bertolotto M. 2015. Octree-based region growing for point cloud segmentation. ISPRS Journal of Photogrammetry and Remote Sensing, 104: 88−100. doi: 10.1016/j.isprsjprs.2015.01.011  Wang G A, Hwang JN, Wallace F, Rose C. 2019. Multi-scale fish segmentation refinement and missing shape recovery. IEEE Access, 7: 52836−52845. doi: 10.1109/ACCESS.2019.2912612  Wang ZL, Walsh KB, Verma B. 2017. On-tree mango fruit size estimation using RGB-D images. Sensors, 17(12): 2738.  Wu YX, Li F, Liu FF, Cheng LN, Guo LL. 2016. Point cloud segmentation using Euclidean cluster extraction algorithm with the Smoothness. Meas Control Technol, 35(3): 36−38. (in Chinese)  Zermas D, Morellas V, Mulla D, Papanikolopoulos N. 2020. 3D model processing for high throughput phenotype extraction–the case of corn. Computers and Electronics in Agriculture, 172: 105047. doi: 10.1016/j.compag.2019.105047
ZR-2021-141 Supplementary Tables and Figures.pdf