HeartMuLa Reference Guide

HeartMuLa is an open-source family of music foundation models (Apache-2.0) that generates full songs conditioned on lyrics and tags, with multilingual support. It is comparable to Suno for open-source use and can run locally or offline. This guide covers installation, configuration, generation, and known limitations as of February 2026.

When to Use HeartMuLa

Use HeartMuLa when you want to generate music or songs from text descriptions, need an open-source alternative to Suno, require local or offline music generation, or when asked about HeartMuLa, heartlib, or AI music generation.

Capabilities

HeartMuLa generates full songs from lyrics and tags. It includes several components:

• HeartMuLa – music language model (3B or 7B parameters)
• HeartCodec – 12.5Hz music codec for high-fidelity audio reconstruction
• HeartTranscriptor – Whisper-based lyrics transcription
• HeartCLAP – audio-text alignment model

Output is MP3 at 48kHz stereo, 128kbps. Real-time factor (RTF) is approximately 1.0, meaning a 4-minute song takes about 4 minutes to generate.

Prerequisites

• NVIDIA GPU with PyTorch CUDA support (or CPU with significant RAM for extremely slow generation)
• Python 3.10
• uv package manager
• Internet connection for downloading models and dependencies
• Git for cloning the repository

Installation

Step 1: Clone the Repository

cd ~/  # or desired directory
git clone https://github.com/HeartMuLa/heartlib.git
cd heartlib

Step 2: Create Virtual Environment and Install Dependencies

uv venv --python 3.10 .venv
. .venv/bin/activate
uv pip install -e .

Step 3: Fix Dependency Compatibility

As of February 2026, pinned dependencies have conflicts with newer packages. Upgrade datasets and transformers:

# Upgrade datasets (old version incompatible with current pyarrow)
uv pip install --upgrade datasets

# Upgrade transformers (needed for huggingface-hub 1.x compatibility)
uv pip install --upgrade transformers

Step 4: Patch Source Code for Transformers 5.x

In src/heartlib/heartmula/modeling_heartmula.py, inside the setup_caches method of the HeartMuLa class, add RoPE reinitialization after the reset_caches try/except block and before the with device: block. Insert this code:

# Re-initialize RoPE caches that were skipped during meta-device loading
from torchtune.models.llama3_1._position_embeddings import Llama3ScaledRoPE
for module in self.modules():
    if isinstance(module, Llama3ScaledRoPE) and not module.is_cache_built:
        module.rope_init()
        module.to(device)

In src/heartlib/pipelines/music_generation.py, add ignore_mismatched_sizes=True to ALL HeartCodec.from_pretrained() calls. There are two such calls: one in the eager load in __init__ and one in the lazy load in the codec property.

Step 5: Download Model Checkpoints

All three downloads can run in parallel. Total size is several GB.

cd heartlib  # project root
hf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen'
hf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year'
hf download --local-dir './ckpt/HeartCodec-oss' 'HeartMuLa/HeartCodec-oss-20260123'

Basic Generation

Step 1: Activate Environment and Run Generation

cd heartlib
. .venv/bin/activate
python ./examples/run_music_generation.py \
  --model_path=./ckpt \
  --version="3B" \
  --lyrics="./assets/lyrics.txt" \
  --tags="./assets/tags.txt" \
  --save_path="./assets/output.mp3" \
  --lazy_load true

Input Formats

Tags: Comma-separated, no spaces. Examples:

piano,happy,wedding,synthesizer,romantic

rock,energetic,guitar,drums,male-vocal

Lyrics: Use bracketed structural tags like [Intro], [Verse], [Chorus], [Bridge], [Outro]. Example:

[Intro]

[Verse]
Your lyrics here...

[Chorus]
Chorus lyrics...

[Bridge]
Bridge lyrics...

[Outro]

Parameters

All parameters are optional unless noted.

Parameter	Meaning	Default
--max_audio_length_ms	Maximum length in milliseconds	240000 (4 minutes)
--topk	Top-k sampling	50
--temperature	Sampling temperature	1.0
--cfg_scale	Classifier-free guidance scale	1.5
--lazy_load	Load/unload models on demand to save VRAM	false
--mula_dtype	Dtype for HeartMuLa (bfloat16 recommended)	bfloat16
--codec_dtype	Dtype for HeartCodec (float32 recommended for quality)	float32

Use --lazy_load true to reduce VRAM usage. For multi-GPU setups, use --mula_device cuda:0 --codec_device cuda:1. For CPU-only mode (extremely slow), use --mula_device cpu --codec_device cpu.

Constraints and Caveats

• Minimum hardware: 8GB VRAM with --lazy_load true; recommended 16GB+ VRAM for single GPU. The 3B model with lazy load peaks at approximately 6.2GB VRAM.
• Python 3.10 is required.
• Do NOT use bf16 for HeartCodec – it degrades audio quality. Use fp32 (the default).
• Tags may be ignored (known issue #90); lyrics tend to dominate. Experiment with tag ordering.
• Triton is not available on macOS; Linux/CUDA only for GPU acceleration.
• RTX 5080 incompatibility has been reported in upstream issues.
• CPU mode is extremely slow (30–60+ minutes per song versus ~4 minutes on GPU) and requires approximately 12GB+ free RAM. Consider a cloud GPU service or the online demo if you do not have an NVIDIA GPU.
• Pinned dependencies have conflicts with newer packages; manual upgrades and patches are required as shown in the installation steps.
• The torchtune package version 0.4.0+cpu may be needed for RoPE reinitialization. The torch==2.4.1 version is pinned in the project.

Failure Modes

• Tags may be ignored (known issue #90).
• RTX 5080 incompatibility reported.
• Dependency pin conflicts require manual upgrades and patches.
• Using bf16 for HeartCodec degrades audio quality.
• Triton not available on macOS.
• If the codec property fails to load, check that ignore_mismatched_sizes=True is added to both HeartCodec.from_pretrained() calls in src/heartlib/pipelines/music_generation.py. The initted flag and [1]/[] indexing in the codec property may also need inspection if errors occur.

Multi-GPU Configuration

For multi-GPU setups, assign devices explicitly:

--mula_device cuda:0 --codec_device cuda:1

This allows the HeartMuLa model and HeartCodec to run on separate GPUs, reducing per-GPU memory pressure.

Notes on Model Loading

The Llama3ScaledRoPE.rope_init() call is critical for models loaded on meta devices. Without it, RoPE caches remain uninitialized, causing generation errors. Always apply the patch in src/heartlib/heartmula/modeling_heartmula.py after upgrading transformers.

The from_pretrained method for HeartCodec must include ignore_mismatched_sizes=True in both the __init__ eager load and the codec property lazy load. This prevents size mismatch errors when loading checkpoints from February 2026.