AudioCraft Audio Generation: MusicGen and AudioGen Guide

AudioCraft is Meta's suite of models for generating audio from text. This guide covers the two main generation models, MusicGen for music and AudioGen for sound effects, plus the EnCodec codec that sits underneath them. You would reach for this when you need to turn a text description into a playable audio file, whether that is a 30-second orchestral piece, a dog barking in a park, or a batch of UI sound effects for a game. It is a practical reference for anyone building music generation tools, sound design pipelines, or interactive demos.

What it does

AudioCraft gives you three capabilities in one package. MusicGen turns text prompts into music, with optional conditioning on a melody or a reference style. AudioGen does the same for sound effects and environmental audio. EnCodec compresses and reconstructs audio, which is useful for preprocessing or for understanding how the other models represent sound internally.

The models come in several sizes, from the 300M parameter musicgen-small up to the 3.3B musicgen-large. Bigger models generally produce better quality but need more GPU memory. You can also choose specialized variants: musicgen-melody for melody conditioning, musicgen-stereo-* for stereo output, and musicgen-style for style transfer from a reference clip.

Generation is controlled by a few key parameters. duration sets the length in seconds, top_k and top_p control sampling diversity, temperature adjusts creativity, and cfg_coef sets how strictly the output follows the text prompt. You set these once per generation call with set_generation_params.

Before you start

This skill is optional and installed on demand. It runs on Linux and macOS. You will need Python and a working PyTorch installation with CUDA if you want reasonable performance on the larger models. The source lists three installation routes: PyPI, GitHub, or HuggingFace Transformers. The PyPI route is simplest for most users; the GitHub route gets you the latest code; the Transformers route is useful if you already work in that ecosystem.

# From PyPI
pip install audiocraft

# From GitHub (latest)
pip install git+https://github.com/facebookresearch/audiocraft.git

# Or use HuggingFace Transformers
pip install transformers torch torchaudio

You will also need torchaudio for loading and saving audio, and scipy if you use the Transformers path. The models download from HuggingFace on first use, so expect a one-time download of several gigabytes depending on the model size.

Quick start

Basic text-to-music (AudioCraft)

The fastest way to generate music is to load a pretrained MusicGen model, set your parameters, and call generate with a list of text descriptions. The model returns a tensor of audio samples, which you save with torchaudio.save. The sample rate for MusicGen is 32000 Hz.

import torchaudio
from audiocraft.models import MusicGen

# Load model
model = MusicGen.get_pretrained('facebook/musicgen-small')

# Set generation parameters
model.set_generation_params(
    duration=8,  # seconds
    top_k=250,
    temperature=1.0
)

# Generate from text
descriptions = ["happy upbeat electronic dance music with synths"]
wav = model.generate(descriptions)

# Save audio
torchaudio.save("output.wav", wav[0].cpu(), sample_rate=32000)

Using HuggingFace Transformers

If you prefer the Transformers API, the same generation is available through MusicgenForConditionalGeneration. You load a processor and model, move them to your device, and pass the text through the processor. The guidance_scale parameter plays the role of cfg_coef, and max_new_tokens controls the output length.

from transformers import AutoProcessor, MusicgenForConditionalGeneration
import scipy

# Load model and processor
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
model.to("cuda")

# Generate music
inputs = processor(
    text=["80s pop track with bassy drums and synth"],
    padding=True,
    return_tensors="pt"
).to("cuda")

audio_values = model.generate(
    **inputs,
    do_sample=True,
    guidance_scale=3,
    max_new_tokens=256
)

# Save
sampling_rate = model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write("output.wav", rate=sampling_rate, data=audio_values[0, 0].cpu().numpy())

Text-to-sound with AudioGen

AudioGen works the same way as MusicGen but is trained for sound effects. The default sample rate is 16000 Hz, lower than MusicGen's 32000 Hz. The example below generates a short clip of a dog barking in a park.

from audiocraft.models import AudioGen

# Load AudioGen
model = AudioGen.get_pretrained('facebook/audiogen-medium')

model.set_generation_params(duration=5)

# Generate sound effects
descriptions = ["dog barking in a park with birds chirping"]
wav = model.generate(descriptions)

torchaudio.save("sound.wav", wav[0].cpu(), sample_rate=16000)

Core concepts

Architecture overview

AudioCraft uses a three-stage pipeline. A text encoder (T5) converts your prompt into embeddings. A transformer decoder generates audio tokens autoregressively, using efficient interleaving patterns. Finally, the EnCodec decoder converts those tokens back into a waveform. Understanding this helps when you tweak generation parameters: the text encoder determines how well the model understands your prompt, the transformer controls the musical structure, and EnCodec affects the final audio quality.

AudioCraft Architecture:
┌──────────────────────────────────────────────────────────────┐
│                    Text Encoder (T5)                          │
│                         │                                     │
│                    Text Embeddings                            │
└────────────────────────┬─────────────────────────────────────┘
                         │
┌────────────────────────▼─────────────────────────────────────┐
│              Transformer Decoder (LM)                         │
│     Auto-regressively generates audio tokens                  │
│     Using efficient token interleaving patterns               │
└────────────────────────┬─────────────────────────────────────┘
                         │
┌────────────────────────▼─────────────────────────────────────┐
│                EnCodec Audio Decoder                          │
│        Converts tokens back to audio waveform                 │
└──────────────────────────────────────────────────────────────┘

Model variants

The table below summarizes the available models. The size column shows parameter counts. The "Use Case" column gives a rough idea of when to pick each one. For quick experiments, musicgen-small is enough. For production quality, musicgen-large is better but requires more memory. The melody and style variants are specialized for their respective conditioning modes.

Model	Size	Description	Use Case
musicgen-small	300M	Text-to-music	Quick generation
musicgen-medium	1.5B	Text-to-music	Balanced
musicgen-large	3.3B	Text-to-music	Best quality
musicgen-melody	1.5B	Text + melody	Melody conditioning
musicgen-melody-large	3.3B	Text + melody	Best melody
musicgen-stereo-*	Varies	Stereo output	Stereo generation
musicgen-style	1.5B	Style transfer	Reference-based
audiogen-medium	1.5B	Text-to-sound	Sound effects

Generation parameters

These parameters control the sampling process. duration is straightforward. top_k limits sampling to the top K tokens; top_p uses nucleus sampling, with 0 meaning disabled. temperature scales the logits before sampling, so higher values produce more varied output. cfg_coef is the classifier-free guidance weight, where higher values make the output adhere more strictly to the text prompt.

Parameter	Default	Description
duration	8.0	Length in seconds (1-120)
top_k	250	Top-k sampling
top_p	0.0	Nucleus sampling (0 = disabled)
temperature	1.0	Sampling temperature
cfg_coef	3.0	Classifier-free guidance

MusicGen usage

Text-to-music generation

This example shows a more complete configuration with all parameters set explicitly. It also demonstrates generating multiple samples in one call, which is more efficient than looping. The output tensor has shape [batch, channels, samples], so you iterate over the batch dimension to save each clip.

from audiocraft.models import MusicGen
import torchaudio

model = MusicGen.get_pretrained('facebook/musicgen-medium')

# Configure generation
model.set_generation_params(
    duration=30,          # Up to 30 seconds
    top_k=250,            # Sampling diversity
    top_p=0.0,            # 0 = use top_k only
    temperature=1.0,      # Creativity (higher = more varied)
    cfg_coef=3.0          # Text adherence (higher = stricter)
)

# Generate multiple samples
descriptions = [
    "epic orchestral soundtrack with strings and brass",
    "chill lo-fi hip hop beat with jazzy piano",
    "energetic rock song with electric guitar"
]

# Generate (returns [batch, channels, samples])
wav = model.generate(descriptions)

# Save each
for i, audio in enumerate(wav):
    torchaudio.save(f"music_{i}.wav", audio.cpu(), sample_rate=32000)

Melody-conditioned generation

When you have an existing melody and want the generated music to follow it, use the musicgen-melody model and the generate_with_chroma method. You pass the melody audio and its sample rate alongside the text description. The model extracts chroma features from the melody and conditions the generation on them.

from audiocraft.models import MusicGen
import torchaudio

# Load melody model
model = MusicGen.get_pretrained('facebook/musicgen-melody')
model.set_generation_params(duration=30)

# Load melody audio
melody, sr = torchaudio.load("melody.wav")

# Generate with melody conditioning
descriptions = ["acoustic guitar folk song"]
wav = model.generate_with_chroma(descriptions, melody, sr)

torchaudio.save("melody_conditioned.wav", wav[0].cpu(), sample_rate=32000)

Stereo generation

For stereo output, load a musicgen-stereo-* variant. The generated tensor will have two channels. The example prints the shape, which for a 15-second clip at 32000 Hz would be [1, 2, 480000]. Saving works the same as mono, but the resulting file will have two channels.

from audiocraft.models import MusicGen

# Load stereo model
model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium')
model.set_generation_params(duration=15)

descriptions = ["ambient electronic music with wide stereo panning"]
wav = model.generate(descriptions)

# wav shape: [batch, 2, samples] for stereo
print(f"Stereo shape: {wav.shape}")  # [1, 2, 480000]
torchaudio.save("stereo.wav", wav[0].cpu(), sample_rate=32000)

Audio continuation

You can extend an existing audio clip by providing it as input along with a text prompt. This uses the Transformers API. The processor takes the audio array and its sample rate, plus the text. The model then generates a continuation that follows the prompt.

from transformers import AutoProcessor, MusicgenForConditionalGeneration

processor = AutoProcessor.from_pretrained("facebook/musicgen-medium")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-medium")

# Load audio to continue
import torchaudio
audio, sr = torchaudio.load("intro.wav")

# Process with text and audio
inputs = processor(
    audio=audio.squeeze().numpy(),
    sampling_rate=sr,
    text=["continue with a epic chorus"],
    padding=True,
    return_tensors="pt"
)

# Generate continuation
audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=512)

MusicGen-Style usage

Style-conditioned generation

The musicgen-style model lets you generate music that matches the style of a reference audio clip. You set two extra parameters: cfg_coef_beta controls the influence of the style, and eval_q sets the number of RVQ quantizers used for the style conditioning (1 to 6). The excerpt_length is how many seconds of the reference are used. Then you call generate_with_style with the text and the reference audio.

from audiocraft.models import MusicGen

# Load style model
model = MusicGen.get_pretrained('facebook/musicgen-style')

# Configure generation with style
model.set_generation_params(
    duration=30,
    cfg_coef=3.0,
    cfg_coef_beta=5.0  # Style influence
)

# Configure style conditioner
model.set_style_conditioner_params(
    eval_q=3,          # RVQ quantizers (1-6)
    excerpt_length=3.0  # Style excerpt length
)

# Load style reference
style_audio, sr = torchaudio.load("reference_style.wav")

# Generate with text + style
descriptions = ["upbeat dance track"]
wav = model.generate_with_style(descriptions, style_audio, sr)

Style-only generation (no text)

You can also generate purely from a style reference, without any text prompt. Pass None as the description and set cfg_coef_beta to None to disable the double classifier-free guidance. This is useful when you want a continuation of the style without any textual direction.

# Generate matching style without text prompt
model.set_generation_params(
    duration=30,
    cfg_coef=3.0,
    cfg_coef_beta=None  # Disable double CFG for style-only
)

wav = model.generate_with_style([None], style_audio, sr)

AudioGen usage

Sound effect generation

AudioGen is the text-to-sound model. The workflow is identical to MusicGen, but the sample rate is 16000 Hz. You can generate multiple sound effects in one batch, which is efficient for building a sound library. The example generates four distinct environmental sounds.

from audiocraft.models import AudioGen
import torchaudio

model = AudioGen.get_pretrained('facebook/audiogen-medium')
model.set_generation_params(duration=10)

# Generate various sounds
descriptions = [
    "thunderstorm with heavy rain and lightning",
    "busy city traffic with car horns",
    "ocean waves crashing on rocks",
    "crackling campfire in forest"
]

wav = model.generate(descriptions)

for i, audio in enumerate(wav):
    torchaudio.save(f"sound_{i}.wav", audio.cpu(), sample_rate=16000)

EnCodec usage

Audio compression

EnCodec is a neural audio codec that compresses audio into discrete tokens and reconstructs it. This is useful for preprocessing audio before feeding it to other models, or for understanding the token representation. The example loads a 32 kHz EnCodec model, resamples the input if needed, encodes to codes, and decodes back to a waveform.

from audiocraft.models import CompressionModel
import torch
import torchaudio

# Load EnCodec
model = CompressionModel.get_pretrained('facebook/encodec_32khz')

# Load audio
wav, sr = torchaudio.load("audio.wav")

# Ensure correct sample rate
if sr != 32000:
    resampler = torchaudio.transforms.Resample(sr, 32000)
    wav = resampler(wav)

# Encode to tokens
with torch.no_grad():
    encoded = model.encode(wav.unsqueeze(0))
    codes = encoded[0]  # Audio codes

# Decode back to audio
with torch.no_grad():
    decoded = model.decode(codes)

torchaudio.save("reconstructed.wav", decoded[0].cpu(), sample_rate=32000)

Common workflows

Workflow 1: Music generation pipeline

This class wraps MusicGen into a reusable generator. It handles loading the model, setting parameters, generating, and saving. The generate method returns a CPU tensor, and generate_batch processes multiple prompts at once. This is a good starting point for integrating MusicGen into a larger application.

import torch
import torchaudio
from audiocraft.models import MusicGen

class MusicGenerator:
    def __init__(self, model_name="facebook/musicgen-medium"):
        self.model = MusicGen.get_pretrained(model_name)
        self.sample_rate = 32000

    def generate(self, prompt, duration=30, temperature=1.0, cfg=3.0):
        self.model.set_generation_params(
            duration=duration,
            top_k=250,
            temperature=temperature,
            cfg_coef=cfg
        )

        with torch.no_grad():
            wav = self.model.generate([prompt])

        return wav[0].cpu()

    def generate_batch(self, prompts, duration=30):
        self.model.set_generation_params(duration=duration)

        with torch.no_grad():
            wav = self.model.generate(prompts)

        return wav.cpu()

    def save(self, audio, path):
        torchaudio.save(path, audio, sample_rate=self.sample_rate)

# Usage
generator = MusicGenerator()
audio = generator.generate(
    "epic cinematic orchestral music",
    duration=30,
    temperature=1.0
)
generator.save(audio, "epic_music.wav")

Workflow 2: Sound design batch processing

This function processes a list of sound specifications and saves each generated clip to an output directory. It returns a list of dictionaries with the name, path, and description of each sound. This is useful for generating a consistent set of UI sounds or game effects.

import json
from pathlib import Path
from audiocraft.models import AudioGen
import torchaudio

def batch_generate_sounds(sound_specs, output_dir):
    """
    Generate multiple sounds from specifications.

    Args:
        sound_specs: list of {"name": str, "description": str, "duration": float}
        output_dir: output directory path
    """
    model = AudioGen.get_pretrained('facebook/audiogen-medium')
    output_dir = Path(output_dir)
    output_dir.mkdir(exist_ok=True)

    results = []

    for spec in sound_specs:
        model.set_generation_params(duration=spec.get("duration", 5))

        wav = model.generate([spec["description"]])

        output_path = output_dir / f"{spec['name']}.wav"
        torchaudio.save(str(output_path), wav[0].cpu(), sample_rate=16000)

        results.append({
            "name": spec["name"],
            "path": str(output_path),
            "description": spec["description"]
        })

    return results

# Usage
sounds = [
    {"name": "explosion", "description": "massive explosion with debris", "duration": 3},
    {"name": "footsteps", "description": "footsteps on wooden floor", "duration": 5},
    {"name": "door", "description": "wooden door creaking and closing", "duration": 2}
]

results = batch_generate_sounds(sounds, "sound_effects/")

Workflow 3: Gradio demo

This example builds a simple web interface for MusicGen using Gradio. The function takes a text prompt, duration, temperature, and CFG coefficient, generates audio, saves it to a temporary file, and returns the path. The interface includes sliders for the numeric parameters.

import gradio as gr
import torch
import torchaudio
from audiocraft.models import MusicGen

model = MusicGen.get_pretrained('facebook/musicgen-small')

def generate_music(prompt, duration, temperature, cfg_coef):
    model.set_generation_params(
        duration=duration,
        temperature=temperature,
        cfg_coef=cfg_coef
    )

    with torch.no_grad():
        wav = model.generate([prompt])

    # Save to temp file
    path = "temp_output.wav"
    torchaudio.save(path, wav[0].cpu(), sample_rate=32000)
    return path

demo = gr.Interface(
    fn=generate_music,
    inputs=[
        gr.Textbox(label="Music Description", placeholder="upbeat electronic dance music"),
        gr.Slider(1, 30, value=8, label="Duration (seconds)"),
        gr.Slider(0.5, 2.0, value=1.0, label="Temperature"),
        gr.Slider(1.0, 10.0, value=3.0, label="CFG Coefficient")
    ],
    outputs=gr.Audio(label="Generated Music"),
    title="MusicGen Demo"
)

demo.launch()

Performance optimization

Memory optimization

Running large models can exhaust GPU memory. The source suggests several strategies: use a smaller model, clear the CUDA cache between generations, generate shorter durations, and use half precision. The example shows all four.

# Use smaller model
model = MusicGen.get_pretrained('facebook/musicgen-small')

# Clear cache between generations
torch.cuda.empty_cache()

# Generate shorter durations
model.set_generation_params(duration=10)  # Instead of 30

# Use half precision
model = model.half()

Batch processing efficiency

Generating multiple prompts in a single call is much faster than looping. The model is designed to handle a batch of descriptions at once. The example contrasts the efficient batch call with the slower loop.

# Process multiple prompts at once (more efficient)
descriptions = ["prompt1", "prompt2", "prompt3", "prompt4"]
wav = model.generate(descriptions)  # Single batch

# Instead of
for desc in descriptions:
    wav = model.generate([desc])  # Multiple batches (slower)

GPU memory requirements

The table below shows approximate VRAM usage for the main MusicGen models in FP32 and FP16. Use this to decide which model fits your GPU. FP16 roughly halves the memory requirement.

Model	FP32 VRAM	FP16 VRAM
musicgen-small	~4GB	~2GB
musicgen-medium	~8GB	~4GB
musicgen-large	~16GB	~8GB

Common issues

Issue	Solution
CUDA OOM	Use smaller model, reduce duration
Poor quality	Increase cfg_coef, better prompts
Generation too short	Check max duration setting
Audio artifacts	Try different temperature
Stereo not working	Use stereo model variant

When not to use it

The source lists several alternatives for specific use cases. If you need longer commercial music generation, Stable Audio is a better fit. For text-to-speech with music or sound effects, Bark is designed for that. Riffusion offers spectrogram-based music generation, which is a different approach. OpenAI Jukebox can generate raw audio with lyrics, which AudioCraft does not support. Choose AudioCraft when your primary need is text-to-music or text-to-sound without lyrics.

Limits and gotchas

• The duration parameter has a range of 1 to 120 seconds, but the actual maximum may be lower depending on the model and your GPU memory. The source notes that generation can be too short if you hit the max duration setting.
• The top_p default is 0.0, which means it is disabled and only top_k is used. If you want nucleus sampling, you must set it to a value between 0 and 1.
• The cfg_coef default is 3.0. Increasing it makes the output adhere more strictly to the text, but too high can cause artifacts. The source suggests adjusting it if quality is poor.
• For stereo generation, you must use a musicgen-stereo-* variant. Using a mono model will not produce stereo output.
• The style model's eval_q parameter accepts values from 1 to 6. Setting it too low may lose style detail, too high may overfit.
• AudioGen uses a sample rate of 16000 Hz, while MusicGen uses 32000 Hz. Make sure you save with the correct rate for each model.
• The source lists common issues: CUDA OOM, poor quality, short generation, audio artifacts, and stereo not working. The solutions are in the table above.

Related skills

This skill pairs well with other creative skills in the Hermes Agent ecosystem. The source lists heartmula and songwriting-and-ai-music as related. If you are building a music production pipeline, you might combine AudioCraft for generation with songwriting-and-ai-music for lyrics or structure, and heartmula for something else in the creative workflow. Check those skill pages for details.

References

• Advanced Usage - Training, fine-tuning, deployment
• Troubleshooting - Common issues and solutions

Resources

• GitHub: https://github.com/facebookresearch/audiocraft
• Paper (MusicGen): https://arxiv.org/abs/2306.05284
• Paper (AudioGen): https://arxiv.org/abs/2209.15352
• HuggingFace: https://huggingface.co/facebook/musicgen-small
• Demo: https://huggingface.co/spaces/facebook/MusicGen