Birdsong From AI: Synthetic Audio Boosts Species Classification

In practice, bird sound datasets often suffer from data insufficiency and class imbalance: some common species have thousands of recordings, while rare or less-studied species have very few. This imbalance skews model performance — models tend to excel on well-represented classes but struggle with under-represented ones. For example, field survey datasets and challenges like BirdCLEF (which includes xeno-canto) have shown that a handful of species dominate the recordings, whereas some species are recorded only once or twice. In BirdCLEF 2022, certain bird call classes had only a single training sample available — a stark illustration of data scarcity in real-world scenarios. This lack of data makes it hard to train a robust classifier for every species.

subtitle

The Data Challenge in Bird Sound Classification

Insufficient data and class imbalance pose serious challenges for bird audio classification. A model trained on imbalanced data may overfit to common species and fail to recognize the rarer birds. Indeed, researchers have observed that even strong bird sound detection pipelines show weak performance on underrepresented classes. Traditionally, one might apply data augmentation (adding noise, time-shifting, etc.) to improve generalization. But while standard augmentation can expand the dataset, it doesn’t create new bird calls for the species that have few or no recordings — it merely modifies existing ones. What if we could generate entirely new, realistic bird audio samples for the species lacking data?

Recent advances in text-to-audio generative models suggest a compelling solution: use AI to synthesize audio of bird calls from textual descriptions. Such models have matured to the point of producing high-quality sounds from prompts, opening the door to dataset augmentation with synthetic audio. Researchers note that these Text-To-Audio (TTA) systems “offer significant potential for dataset synthesis or augmentation” and could be “directly integrated into the training pipeline of deep learning systems”. By leveraging a generative model, we can create artificial bird chirp recordings to balance our training data. The key is to ensure the generated sounds are realistic enough to be useful. This is where AudioLDM2 comes into play.

Synthetic Bird Calls with AudioLDM2

AudioLDM2 can create high-quality, realistic audio from simple text prompts — yes, just a descriptive sentence! How does it work? It’s based on something called a “latent diffusion model”, similar to popular image generators you might’ve seen online. It translates text descriptions into audio, without needing exact pairs of audio and text during its training.

AudioLDM2 cleverly uses what’s called CLAP embeddings — essentially, it understands the relationship between text and audio intuitively. This makes it super efficient and remarkably accurate at generating realistic audio clips, like bird chirps.

Using AudioLDM2 in the Training Pipeline

Integrating AudioLDM2 into a bird classification training pipeline is straightforward. Here’s a quick look at how we generate synthetic bird audio:

Step 1: Initializing AudioLDM2:

import torch
from diffusers import AudioLDM2Pipeline

output:
pipe = AudioLDM2Pipeline.from_pretrained("cvssp/audioldm2", torch_dtype=torch.float16)
pipe = pipe.to("cuda")  # GPU speeds things up!

Step 2: Crafting a Prompt Describe the bird sound you need:

prompt = "Clear chirping of a small sparrow in the early morning forest"

Step 3: Generating the Audio Generate a 5-second clip:

audio_output = pipe(prompt, num_inference_steps=150, audio_length_in_s=5.0)
audio_array = audio_output.audios[0]

Step 4: Saving Your Clip Finally, save this newly generated audio:

import scipy.io.wavfile as wavfile
wavfile.write("synthetic_sparrow.wav", rate=16000, data=audio_array)

Now, repeat this process for species needing more data, and suddenly your dataset becomes richer and more balanced.

Conclusion

By incorporating AudioLDM2-generated bird calls into the training set, we address the twin problems of data insufficiency and class imbalance. The model now has more examples to learn from for the rare species, which can lead to improved recognition accuracy on those classes. Using synthetic audio as training data is a form of advanced data augmentation — one that actually creates new signal patterns rather than tweaking existing ones.

Models trained with a healthy mix of real and synthetic data might become more robust against variances in recording conditions as well — since we can simulate different backgrounds or environments in the prompts (e.g. “song of a sparrow with city traffic noise in the background”). Moreover, this approach is not limited to birds: the same pipeline can generate insect sounds, amphibian calls, or mammal vocalizations given the right prompts, potentially aiding various ecological audio surveys.

Looking ahead, using tools like AudioLDM2 could revolutionize biodiversity conservation. Imagine easily training models to detect critically endangered birds, amphibians, insects, or mammals, even if real recordings are scarce.

With AI-driven audio synthesis, we’re not just overcoming data limitations — we’re enhancing our ability to protect and understand the natural world.

Reference

• Martynov, Eduard, and Yuuichiroh Uematsu. “Dealing with Class Imbalance in Bird Sound Classification.” In CLEF (Working Notes), pp. 2151–2158. 2022.
• Ronchini, Francesca, Luca Comanducci, and Fabio Antonacci. “Synthesizing soundscapes: Leveraging text-to-audio models for environmental sound classification.” arXiv e-prints (2024): arXiv-2403.
• Ronchini, Francesca, Luca Comanducci, and Fabio Antonacci. “Synthetic training set generation using text-to-audio models for environmental sound classification.” arXiv preprint arXiv:2403.17864 (2024).
• Liu, Haohe, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. “Audioldm 2: Learning holistic audio generation with self-supervised pretraining.” IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024).