BARN: Behavior-Aware Relation Network for multi-label behavior detection in socially housed macaques

Sen Yang; Zhi-Yuan Chen; Ke-Wei Liang; Cai-Jie Qin; Yang Yang; Wen-Xuan Fan; Chen-Lu Jie; Xi-Bo Ma

doi:10.24272/j.issn.2095-8137.2022.485

Volume 44 Issue 6

Nov. 2023

Turn off MathJax

Article Contents

Article Navigation > Zoological Research > 2023 > 44(6): 1026-1038

Sen Yang, Zhi-Yuan Chen, Ke-Wei Liang, Cai-Jie Qin, Yang Yang, Wen-Xuan Fan, Chen-Lu Jie, Xi-Bo Ma. BARN: Behavior-Aware Relation Network for multi-label behavior detection in socially housed macaques. Zoological Research, 2023, 44(6): 1026-1038. doi: 10.24272/j.issn.2095-8137.2022.485

Citation:

Sen Yang, Zhi-Yuan Chen, Ke-Wei Liang, Cai-Jie Qin, Yang Yang, Wen-Xuan Fan, Chen-Lu Jie, Xi-Bo Ma. BARN: Behavior-Aware Relation Network for multi-label behavior detection in socially housed macaques. Zoological Research, 2023, 44(6): 1026-1038. doi: 10.24272/j.issn.2095-8137.2022.485

Citation:

Sen Yang, Zhi-Yuan Chen, Ke-Wei Liang, Cai-Jie Qin, Yang Yang, Wen-Xuan Fan, Chen-Lu Jie, Xi-Bo Ma. BARN: Behavior-Aware Relation Network for multi-label behavior detection in socially housed macaques. Zoological Research, 2023, 44(6): 1026-1038. doi: 10.24272/j.issn.2095-8137.2022.485

PDF( 16303 KB)

BARN: Behavior-Aware Relation Network for multi-label behavior detection in socially housed macaques

doi: 10.24272/j.issn.2095-8137.2022.485

Sen Yang^{1, 2, 3},
Zhi-Yuan Chen^{2, 3},
Ke-Wei Liang^{2, 3},
Cai-Jie Qin^{2, 3},
Yang Yang^{2, 3},
Wen-Xuan Fan^{1, 2, 3},
Chen-Lu Jie^{2, 3, 4},
Xi-Bo Ma^{1, 2, 3
,
,}

1.
College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, Liaoning 110169, China
2.
MAIS, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
3.
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China
4.
School of Automation, Harbin University of Science and Technology, Harbin, Heilongjiang 150080, China

Our code and macaque datasets are freely available online (https://github.com/BertonYang18/BARN-monkey) along with the publication of this study. We hope to receive feedback on any potential bugs or issues. Supplementary Videos S1 and S2 can also be found online (https://drive.google.com/drive/folders/1v2ZcXlrAR7rB0Pws4SWUqZPTKupVQIw7?usp=share_link).
Supplementary data to this article can be found online.
The authors declare that they have no competing interests.
S.Y. completed the main experiment, wrote the manuscript draft, and conducted the literature review. S.Y., Z.Y.C., K.W.L., W.X.F., and C.L.J. participated in the design of data acquisition and annotation scheme. X.B.M., Y.Y., and C.J.Q discussed the study, provided suggestions to improve the experimental scheme, and edited the manuscript. All authors read and approved the final version of the manuscript.

Funds: This work was supported by the Major Project of the National Natural Science Foundation of China (82090051, 81871442) and Outstanding Member Project of Youth Innovation Promotion Association of the Chinese Academy of Sciences (Y201930)

More Information

Corresponding author: E-mail: xibo.ma@nlpr.ia.ac.cn
Received Date: 2023-03-23
Accepted Date: 2023-04-20

Published Online: 2023-09-12

Publish Date: 2023-11-18

Abstract

Abstract

Quantification of behaviors in macaques provides crucial support for various scientific disciplines, including pharmacology, neuroscience, and ethology. Despite recent advancements in the analysis of macaque behavior, research on multi-label behavior detection in socially housed macaques, including consideration of interactions among them, remains scarce. Given the lack of relevant approaches and datasets, we developed the Behavior-Aware Relation Network (BARN) for multi-label behavior detection of socially housed macaques. Our approach models the relationship of behavioral similarity between macaques, guided by a behavior-aware module and novel behavior classifier, which is suitable for multi-label classification. We also constructed a behavior dataset of rhesus macaques using ordinary RGB cameras mounted outside their cages. The dataset included 65 913 labels for 19 behaviors and 60 367 proposals, including identities and locations of the macaques. Experimental results showed that BARN significantly improved the baseline SlowFast network and outperformed existing relation networks. In conclusion, we successfully achieved multi-label behavior detection of socially housed macaques with both economic efficiency and high accuracy.
- Macaque behavior,
- Drug safety assessment,
- Multi-label behavior detection,
- Behavioral similarity,
- Relation network

FullText(HTML)

References(37)

References

[1]	Bala PC, Eisenreich BR, Yoo SBM, et al. 2020. Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio. Nature Communications, 11(1): 4560. doi: 10.1038/s41467-020-18441-5
[2]	Ballesta S, Reymond G, Pozzobon M, et al. 2014. A real-time 3D video tracking system for monitoring primate groups. Journal of Neuroscience Methods, 234: 147−152. doi: 10.1016/j.jneumeth.2014.05.022
[3]	Collobert R, Bengio S, Mariéthoz J. 2002. Torch: a modular machine learning software library. REP_WORK (30 October, 2002). Idiap, https://os.unil.cloud.switch.ch/tind-customer-epfl/5ea06583-58ae-4f33-bcc5-ae8feb746af1?response-content-disposition=attachment%3B%20filename%2A%3DUTF-8%27%27rr02-46.pdf&response-content-type=application%2Fpdf&AWSAccessKeyId=ded3589a13b4450889b2f728d54861a6&Expires=1682421084&Signature=gej7yCqVtYuJiwgttsO0YPoiqXo%3D.
[4]	Defler TR. 2000. Locomotion and posture in Lagothrix lagotricha. Folia Primatologica, 70(6): 313–327.
[5]	Everingham M, Van Gool L, Williams CKI, et al. 2010. The pascal visual object classes (VOC) challenge. International journal of computer vision, 88(2): 303−338. doi: 10.1007/s11263-009-0275-4
[6]	Feichtenhofer C, Fan HQ, Malik J, et al. 2019. Slowfast networks for video recognition. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 6201–6210.
[7]	Glander KE. 1975. Habitat description and resource utilization: a preliminary report on mantled howling monkey ecology. In: Tuttle RH. Socioecology and Psychology of Primates. Berlin: De Gruyter Mouton, 37–58.
[8]	Goyal P, Dollár P, Girshick R, et al. 2017. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv: 1706.02677.
[9]	Gu CH, Sun C, Ross DA, et al. 2018. AVA: a video dataset of spatio-temporally localized atomic visual actions. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 6047–6056.
[10]	Han YN, Huang K, Chen K, et al. 2022. MouseVenue3D: a markerless three-dimension behavioral tracking system for matching two-photon brain imaging in free-moving mice. Neuroscience Bulletin, 38(3): 303−317. doi: 10.1007/s12264-021-00778-6
[11]	He KM, Gkioxari G, Dollár P, et al. 2017. Mask R-CNN. In: Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice: IEEE, 2980–2988.
[12]	Holman CC. 1948. Hæmangioma of the sigmoid colon. Report of a case. British Journal of Surgery, 36(142): 210.
[13]	Hsu AI, Yttri EA. 2019. B-SOiD: an open source unsupervised algorithm for discovery of spontaneous behaviors. BioRxiv, 770271.
[14]	Jafrasteh B, Suárez A. 2021. Objective functions from Bayesian optimization to locate additional drillholes. Computers & Geosciences, 147: 104674.
[15]	Kim NY, Kim SJ, Jang SY, et al. 2017. Behavioral characteristics of Hanwoo (Bos taurus coreanae) steers at different growth stages and seasons. Asian-Australasian Journal of Animal Sciences, 30(10): 1486−1494. doi: 10.5713/ajas.16.0992
[16]	Kops MS, Pesic M, Petersen KU, et al. 2021. Impact of concurrent remifentanil on the sedative effects of remimazolam, midazolam and propofol in cynomolgus monkeys. European Journal of Pharmacology, 890: 173639. doi: 10.1016/j.ejphar.2020.173639
[17]	Li C, Xiao Z, Li Y, et al. 2023. Deep learning-based activity recognition and fine motor identification using 2D skeletons of cynomolgus monkeys. Zoological Research, 44(5): 967−980. doi: 10.24272/j.issn.2095-8137.2022.449
[18]	Liu MS, Gao JQ, Hu GY, et al. 2022. MonkeyTrail: a scalable video-based method for tracking macaque movement trajectory in daily living cages. Zoological Research, 43(3): 343−351. doi: 10.24272/j.issn.2095-8137.2021.353
[19]	Marks M, Jin QH, Sturman O, et al. 2022. Deep-learning-based identification, tracking, pose estimation and behaviour classification of interacting primates and mice in complex environments. Nature Machine Intelligence, 4(4): 331−340. doi: 10.1038/s42256-022-00477-5
[20]	Meunier B, Pradel P, Sloth KH, et al. 2018. Image analysis to refine measurements of dairy cow behaviour from a real-time location system. Biosystems engineering, 173: 32−44. doi: 10.1016/j.biosystemseng.2017.08.019
[21]	Morimoto Y, Fujita K. 2011. Capuchin monkeys (Cebus apella) modify their own behaviors according to a conspecific’s emotional expressions. Primates, 52(3): 279−286. doi: 10.1007/s10329-011-0249-3
[22]	Negrete SB, Labuguen R, Matsumoto J, et al. 2021. Multiple monkey pose estimation using OpenPose. bioRxiv: 428726.
[23]	Pan JT, Chen SY, Shou MZ, et al. 2021. Actor-context-actor relation network for spatio-temporal action localization. In: Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville: IEEE, 464–474.
[24]	Röder EL, Timmermans PJA. 2002. Housing and care of monkeys and apes in laboratories: adaptations allowing essential species-specific behaviour. Laboratory Animals, 36(3): 221−242. doi: 10.1258/002367702320162360
[25]	Singh GB, Bani S, Singh S. 1996. Toxicity and safety evaluation of Boswellic acids. Phytomedicine, 3(1): 87−90. doi: 10.1016/S0944-7113(96)80018-3
[26]	Sun C, Shrivastava A, Vondrick C, et al. 2018. Actor-centric relation network. In: Proceedings of the 15th European Conference on Computer Vision (ECCV). Munich: Springer, 335–351.
[27]	Volkow ND. 2012. Long-term safety of stimulant use for ADHD: findings from nonhuman primates. Neuropsychopharmacology, 37(12): 2551−2552. doi: 10.1038/npp.2012.127
[28]	Wang CY, Bochkovskiy A, Liao HYM. 2022. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv: 2207.02696.
[29]	Wbreza. 2021. VOTT.https://github.com/microsoft/VoTT.
[30]	Westlund K, Fernström AL, Wergård EM, et al. 2012. Physiological and behavioural stress responses in cynomolgus macaques (Macaca fascicularis) to noise associated with construction work. Laboratory Animals, 46(1): 51−58. doi: 10.1258/la.2011.011040
[31]	Wiltschko AB, Johnson MJ, Iurilli G, et al. 2015. Mapping sub-second structure in mouse behavior. Neuron, 88(6): 1121−1135. doi: 10.1016/j.neuron.2015.11.031
[32]	Wit HD. 2011. Sex hormones: a new treatment for cocaine abuse?. Neuropsychopharmacology, 36(11): 2155−2156. doi: 10.1038/npp.2011.146
[33]	Wu CY, Feichtenhofer C, Fan HQ, et al. 2019. Long-term feature banks for detailed video understanding. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 284–293.
[34]	Xu Q, Zhang M, Gu ZH, et al. 2019. Overfitting remedy by sparsifying regularization on fully-connected layers of CNNs. Neurocomputing, 328: 69−74. doi: 10.1016/j.neucom.2018.03.080
[35]	Yu S. 2016. New challenge for bionics—brain-inspired computing. Zoological Research, 37(5): 261−262.
[36]	Zhang YB, Tokmakov P, Hebert M, et al. 2019. A structured model for action detection. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 9967–9976.
[37]	Zhang YY, Li XY, Marsic I. 2021. Multi-label activity recognition using activity-specific features and activity correlations. In: Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville: IEEE, 14620–14630.