AI based Animal Behavior Coding Project

Artificial intelligence (AI) has potentially profound implications for scientific practice in the field of animal behavior. Animal behavior research typically starts with a laborious human labelling of video, audio or other data. Manually labeling behavioral data is both extremely time-consuming, and inconsistent across studies and species. These restrictions currently severely restrict our capacity to analyze behavioral data across species.

Modern deep-AI methods could potentially revolutionize this field by allowing rapid objective machine-assisted coding of data. However, current methods require extensive human-verified data to train and test AI systems, and such data must be in consistent formats to allow machine learning models to operate.

We will develop, test, and release a powerful AI-assisted, open-source framework - AIHAB - capable of processing behavioral data and generalizing across species. We aim to harness the power of pre trained deep-AI models, leveraging our substantial repository of human-coded animal behavior data (video, audio and bio-logging data) across multiple species. We will customize existing "foundation models" by fine-tuning them with our animal behavior data. Although initial training for one species will require very large datasets, pre-trained models can then be further refined for a different species with much less data, providing a generalizable AI-based system.

Our system will allow researchers to easily fine-tune existing open-source foundation models (Large Language Models and machine vision models) in a flexible and extendable manner. We will train and test models using our large existing labeled datasets, and extend and test the system on new datasets, with a focus on cross-species generalization in multiple bird species. We will deploy our system as open-source Python code, free to the international community. Our project will thus demonstrate the potential of AI to revolutionize the study of animal behavior and cognition.