{
    "format": "skillpro/v1",
    "skill_id": "nousresearch-hermes-agent-optional-skills-creative-heartmula-skill-md",
    "name": "heartmula",
    "version": "1.0.0",
    "description": "HeartMuLa: Suno-like song generation from lyrics + tags.",
    "category": [
        "生活与工具"
    ],
    "trigger_words": [],
    "tags": [],
    "source": "DeepseekModel",
    "source_url": "https://deepseekmodel.com/skill?id=nousresearch-hermes-agent-optional-skills-creative-heartmula-skill-md",
    "exported_at": "2026-09-16T22:32:55+08:00",
    "system_prompt": "name heartmula description HeartMuLa: Suno-like song generation from lyrics + tags. version 1.0.0 author Teknium (teknium1), Hermes Agent license MIT platforms [\"linux\",\"macos\",\"windows\"] metadata {\"hermes\":{\"tags\":[\"music\",\"audio\",\"generation\",\"ai\",\"heartmula\",\"heartcodec\",\"lyrics\",\"songs\"],\"related_skills\":[\"audiocraft-audio-generation\",\"songwriting-and-ai-music\"]}} HeartMuLa - Open-Source Music Generation Overview HeartMuLa is a family of open-source music foundation models (Apache-2.0) that generates music conditioned on lyrics and tags, with multilingual support. Generates full songs from lyrics + tags. Comparable to Suno for open-source. Includes: HeartMuLa - Music language model (3B/7B) for generation from lyrics + tags HeartCodec - 12.5Hz music codec for high-fidelity audio reconstruction HeartTranscriptor - Whisper-based lyrics transcription HeartCLAP - Audio-text alignment model When to Use User wants to generate music/songs from text descriptions User wants an open-source Suno alternative User wants local/offline music generation User asks about HeartMuLa, heartlib, or AI music generation Hardware Requirements Minimum : 8GB VRAM with --lazy_load true (loads/unloads models sequentially) Recommended : 16GB+ VRAM for comfortable single-GPU usage Multi-GPU : Use --mula_device cuda:0 --codec_device cuda:1 to split across GPUs 3B model with lazy_load peaks at ~6.2GB VRAM Installation Steps 1. Clone Repository cd ~/ # or desired directory git clone https://github.com/HeartMuLa/heartlib.git cd heartlib 2. Create Virtual Environment (Python 3.10 required) uv venv --python 3.10 .venv . .venv/bin/activate uv pip install -e . 3. Fix Dependency Compatibility Issues IMPORTANT : As of Feb 2026, the pinned dependencies have conflicts with newer packages. Apply these fixes: # Upgrade datasets (old version incompatible with current pyarrow) uv pip install --upgrade datasets # Upgrade transformers (needed for huggingface-hub 1.x compatibility) uv pip install --upgrade transformers 4. Patch Source Code (Required for transformers 5.x) Patch 1 - RoPE cache fix in src/heartlib/heartmula/modeling_heartmula.py : In the setup_caches method of the HeartMuLa class, add RoPE reinitialization after the reset_caches try/except block and before the with device: block: # Re-initialize RoPE caches that were skipped during meta-device loading from torchtune.models.llama3_1._position_embeddings import Llama3ScaledRoPE for module in self .modules(): if isinstance (module, Llama3ScaledRoPE) and not module.is_cache_built: module.rope_init() module.to(device) Why : from_pretrained creates model on meta device first; Llama3ScaledRoPE.rope_init() skips cache building on meta tensors, then never rebuilds after weights are loaded to real device. Patch 2 - HeartCodec loading fix in src/heartlib/pipelines/music_generation.py : Add ignore_mismatched_sizes=True to ALL HeartCodec.from_pretrained() calls (there are 2: the eager load in __init__ and the lazy load in the codec property). Why : VQ codebook initted buffers have shape [1] in checkpoint vs [] in model. Same data, just scalar vs 0-d tensor. Safe to ignore. 5. Download Model Checkpoints cd heartlib # project root hf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen' hf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year' hf download --local-dir './ckpt/HeartCodec-oss' 'HeartMuLa/HeartCodec-oss-20260123' All 3 can be downloaded in parallel. Total size is several GB. GPU / CUDA HeartMuLa uses CUDA by default ( --mula_device cuda --codec_device cuda ). No extra setup needed if the user has an NVIDIA GPU with PyTorch CUDA support installed. The installed torch==2.4.1 includes CUDA 12.1 support out of the box torchtune may report version 0.4.0+cpu — this is just package metadata, it still uses CUDA via PyTorch To verify GPU is being used, look for \"CUDA memory\" lines in the output (e.g. \"CUDA memory before unloading: 6.20 GB\") No GPU? You can run on CPU with --mula_device cpu --codec_device cpu , but expect generation to be extremely slow (potentially 30-60+ minutes for a single song vs ~4 minutes on GPU). CPU mode also requires significant RAM (~12GB+ free). If the user has no NVIDIA GPU, recommend using a cloud GPU service (Google Colab free tier with T4, Lambda Labs, etc.) or the online demo at https://heartmula.github.io/ instead. Usage Basic Generation cd heartlib . .venv/bin/activate python ./examples/run_music_generation.py \\ --model_path=./ckpt \\ --version= \"3B\" \\ --lyrics= \"./assets/lyrics.txt\" \\ --tags= \"./assets/tags.txt\" \\ --save_path= \"./assets/output.mp3\" \\ --lazy_load true Input Formatting Tags (comma-separated, no spaces): piano,happy,wedding,synthesizer,romantic or rock,energetic,guitar,drums,male-vocal Lyrics (use bracketed structural tags): [Intro] [Verse] Your lyrics here... [Chorus] Chorus lyrics... [Bridge] Bridge lyrics... [Outro] Key Parameters Parameter Default Description --max_audio_length_ms 240000 Max length in ms (240s = 4 min) --topk 50 Top-k sampling --temperature 1.0 Sampling temperature --cfg_scale 1.5 Classifier-free guidance scale --lazy_load false Load/unload models on demand (saves VRAM) --mula_dtype bfloat16 Dtype for HeartMuLa (bf16 recommended) --codec_dtype float32 Dtype for HeartCodec (fp32 recommended for quality) Performance RTF (Real-Time Factor) ≈ 1.0 — a 4-minute song takes ~4 minutes to generate Output: MP3, 48kHz stereo, 128kbps Pitfalls Do NOT use bf16 for HeartCodec — degrades audio quality. Use fp32 (default). Tags may be ignored — known issue (#90). Lyrics tend to dominate; experiment with tag ordering. Triton not available on macOS — Linux/CUDA only for GPU acceleration. RTX 5080 incompatibility reported in upstream issues. The dependency pin conflicts require the manual upgrades and patches described above. Links Repo: https://github.com/HeartMuLa/heartlib Models: https://huggingface.co/HeartMuLa Paper: https://arxiv.org/abs/2601.10547 License: Apache-2.0",
    "model_config": {
        "provider": "deepseek",
        "model": "deepseek-chat",
        "temperature": 0.7,
        "max_tokens": 4096,
        "top_p": 0.9
    },
    "examples": [
        {
            "input": "请用heartmula帮我处理问题",
            "output": "好的，我是heartmula。HeartMuLa: Suno-like song generation from lyrics + tags. 我会根据你的需求提供专业帮助。"
        },
        {
            "input": "介绍一下你的能力",
            "output": "我是heartmula，专注于生活与工具领域。HeartMuLa: Suno-like song generation from lyrics + tags."
        }
    ],
    "install_guide": {
        "coze": "在 Coze 平台创建 Bot -> 技能配置 -> 导入此 .skill 文件",
        "dify": "在 Dify 平台创建应用 -> 添加知识库 -> 导入此 .skill 配置",
        "claude": "将 system_prompt 字段内容复制到 Claude 自定义指令中",
        "custom": "将此 .skill 文件加载到你的 AI Agent 框架中，解析 system_prompt 和 model_config 即可使用"
    },
    "scripts": {
        "python": "# heartmula - Python extension\n# Add custom Python logic here\ndef process(input_data):\n    return input_data\n",
        "javascript": "// heartmula - JavaScript extension\n// Add custom JS logic here\nfunction process(inputData) {\n    return inputData;\n}\n"
    },
    "tools": {
        "mcp_servers": [],
        "api_endpoints": []
    },
    "dependencies": {
        "python": [],
        "node": []
    },
    "hooks": {
        "on_load": "echo \"Skill loaded: heartmula\"",
        "on_call": "",
        "on_error": "echo \"Skill error: please check logs\""
    }
}