{
    "format": "skillpro/v1",
    "skill_id": "nousresearch-hermes-agent-optional-skills-mlops-accelerate-skill-md",
    "name": "accelerate",
    "version": "1.0.0",
    "description": "Run PyTorch training across GPUs with minimal changes.",
    "category": [
        "学习教育"
    ],
    "trigger_words": [],
    "tags": [
        "ai"
    ],
    "source": "DeepseekModel",
    "source_url": "https://deepseekmodel.com/skill?id=nousresearch-hermes-agent-optional-skills-mlops-accelerate-skill-md",
    "exported_at": "2026-09-17T06:40:08+08:00",
    "system_prompt": "name accelerate description Run PyTorch training across GPUs with minimal changes. version 1.0.1 author Orchestra Research license MIT dependencies [\"accelerate\",\"torch\",\"transformers\"] platforms [\"linux\",\"macos\",\"windows\"] metadata {\"hermes\":{\"tags\":[\"Distributed Training\",\"HuggingFace\",\"Accelerate\",\"DeepSpeed\",\"FSDP\",\"Mixed Precision\",\"PyTorch\",\"DDP\",\"Unified API\",\"Simple\"]}} HuggingFace Accelerate - Unified Distributed Training Quick start Accelerate simplifies distributed training to 4 lines of code. Installation : pip install accelerate Convert PyTorch script (4 lines): import torch + from accelerate import Accelerator + accelerator = Accelerator() model = torch.nn.Transformer() optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset) + model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) for batch in dataloader: optimizer.zero_grad() loss = model(batch) - loss.backward() + accelerator.backward(loss) optimizer.step() Run (single command): accelerate launch train.py Common workflows Workflow 1: From single GPU to multi-GPU Original script : # train.py import torch model = torch.nn.Linear( 10 , 2 ).to( 'cuda' ) optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset, batch_size= 32 ) for epoch in range ( 10 ): for batch in dataloader: batch = batch.to( 'cuda' ) optimizer.zero_grad() loss = model(batch).mean() loss.backward() optimizer.step() With Accelerate (4 lines added): # train.py import torch from accelerate import Accelerator # +1 accelerator = Accelerator() # +2 model = torch.nn.Linear( 10 , 2 ) optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset, batch_size= 32 ) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # +3 for epoch in range ( 10 ): for batch in dataloader: # No .to('cuda') needed - automatic! optimizer.zero_grad() loss = model(batch).mean() accelerator.backward(loss) # +4 optimizer.step() Configure (interactive): accelerate config Questions : Which machine? (single/multi GPU/TPU/CPU) How many machines? (1) Mixed precision? (no/fp16/bf16/fp8) DeepSpeed? (no/yes) Launch (works on any setup): # Single GPU accelerate launch train.py # Multi-GPU (8 GPUs) accelerate launch --multi_gpu --num_processes 8 train.py # Multi-node accelerate launch --multi_gpu --num_processes 16 \\ --num_machines 2 --machine_rank 0 \\ --main_process_ip $MASTER_ADDR \\ train.py Workflow 2: Mixed precision training Enable FP16/BF16 : from accelerate import Accelerator # FP16 (with gradient scaling) accelerator = Accelerator(mixed_precision= 'fp16' ) # BF16 (no scaling, more stable) accelerator = Accelerator(mixed_precision= 'bf16' ) # FP8 (H100+) accelerator = Accelerator(mixed_precision= 'fp8' ) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # Everything else is automatic! for batch in dataloader: with accelerator.autocast(): # Optional, done automatically loss = model(batch) accelerator.backward(loss) Workflow 3: DeepSpeed ZeRO integration Enable DeepSpeed ZeRO-2 (pass a DeepSpeedPlugin , not a raw dict): from accelerate import Accelerator, DeepSpeedPlugin deepspeed_plugin = DeepSpeedPlugin( zero_stage= 2 , # ZeRO-2 offload_optimizer_device= \"none\" , # or \"cpu\" to offload gradient_accumulation_steps= 4 , ) accelerator = Accelerator( mixed_precision= 'bf16' , deepspeed_plugin=deepspeed_plugin, # DeepSpeedPlugin instance (or dict[str, DeepSpeedPlugin]) ) # Same code as before! model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) Or point at a full DeepSpeed JSON config via the plugin : from accelerate import Accelerator, DeepSpeedPlugin # hf_ds_config accepts a path to a DeepSpeed config JSON (or a dict) deepspeed_plugin = DeepSpeedPlugin(hf_ds_config= \"ds_config.json\" ) accelerator = Accelerator(mixed_precision= 'bf16' , deepspeed_plugin=deepspeed_plugin) ds_config.json (a raw DeepSpeed config — passed via the plugin, NOT via --config_file ): { \"fp16\" : { \"enabled\" : false } , \"bf16\" : { \"enabled\" : true } , \"zero_optimization\" : { \"stage\" : 2 , \"offload_optimizer\" : { \"device\" : \"cpu\" } , \"allgather_bucket_size\" : 5e8 , \"reduce_bucket_size\" : 5e8 } } Or via interactive config : accelerate config # Select: DeepSpeed → ZeRO-2 # This writes an accelerate YAML config (default: ~/.cache/huggingface/accelerate/default_config.yaml) Launch ( --config_file expects an accelerate YAML, not a raw DeepSpeed JSON): # Uses the default accelerate config written by `accelerate config` accelerate launch train.py # Or point at a specific accelerate YAML accelerate launch --config_file accelerate_deepspeed.yaml train.py Workflow 4: FSDP (Fully Sharded Data Parallel) Enable FSDP : from accelerate import Accelerator, FullyShardedDataParallelPlugin fsdp_plugin = FullyShardedDataParallelPlugin( sharding_strategy= \"FULL_SHARD\" , # ZeRO-3 equivalent auto_wrap_policy= \"transformer_based_wrap\" , # valid: transformer_based_wrap | size_based_wrap | no_wrap cpu_offload= False ) accelerator = Accelerator( mixed_precision= 'bf16' , fsdp_plugin=fsdp_plugin ) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) Or via config : accelerate config # Select: FSDP → Full Shard → No CPU Offload Workflow 5: Gradient accumulation Accumulate gradients : from accelerate import Accelerator accelerator = Accelerator(gradient_accumulation_steps= 4 ) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) for batch in dataloader: with accelerator.accumulate(model): # Handles accumulation optimizer.zero_grad() loss = model(batch) accelerator.backward(loss) optimizer.step() Effective batch size : batch_size * num_gpus * gradient_accumulation_steps When to use vs alternatives Use Accelerate when : Want simplest distributed training Need single script for any hardware Use HuggingFace ecosystem Want flexibility (DDP/DeepSpeed/FSDP/Megatron) Need quick prototyping Key advantages : 4 lines : Minimal code changes Unified API : Same code for DDP, DeepSpeed, FSDP, Megatron Automatic : Device placement, mixed precision, sharding Interactive config : No manual launcher setup Single launch : Works everywhere Use alternatives instead : PyTorch Lightning : Need callbacks, high-level abstractions Ray Train : Multi-node orchestration, hyperparameter tuning DeepSpeed : Direct API control, advanced features Raw DDP : Maximum control, minimal abstraction Common issues Issue: Wrong device placement Don't manually move to device: # WRONG batch = batch.to( 'cuda' ) # CORRECT # Accelerate handles it automatically after prepare() Issue: Gradient accumulation not working Use context manager: # CORRECT with accelerator.accumulate(model): optimizer.zero_grad() accelerator.backward(loss) optimizer.step() Issue: Checkpointing in distributed Use accelerator methods: # Save only on main process if accelerator.is_main_process: accelerator.save_state( 'checkpoint/' ) # Load on all processes accelerator.load_state( 'checkpoint/' ) Issue: Different results with FSDP Ensure same random seed: from accelerate.utils import set_seed set_seed( 42 ) Advanced topics Megatron integration : See references/megatron-integration.md for tensor parallelism, pipeline parallelism, and sequence parallelism setup. Custom plugins : See references/custom-plugins.md for creating custom distributed plugins and advanced configuration. Performance tuning : See references/performance.md for profiling, memory optimization, and best practices. Hardware requirements CPU : Works (slow) Single GPU : Works Multi-GPU : DDP (default), DeepSpeed, or FSDP Multi-node : DDP, DeepSpeed, FSDP, Megatron TPU : Supported Apple MPS : Supported Launcher requirements : DDP : torch.distributed.run (built-in) DeepSpeed : deepspeed (pip install deepspeed) FSDP : PyTorch 1.12+ (built-in) Megatron : Custom setup Resources Docs: https://huggingface.co/docs/accelerate GitHub: https://github.com/huggingface/accelerate Version: 1.11.0+ Tutorial: \"Accelerate your scripts\" Examples: https://github.com/huggingface/accelerate/tree/main/examples Used by: HuggingFace Transformers, TRL, PEFT, all HF libraries",
    "model_config": {
        "provider": "deepseek",
        "model": "deepseek-chat",
        "temperature": 0.7,
        "max_tokens": 4096,
        "top_p": 0.9
    },
    "examples": [
        {
            "input": "请用accelerate帮我处理问题",
            "output": "好的，我是accelerate。Run PyTorch training across GPUs with minimal changes. 我会根据你的需求提供专业帮助。"
        },
        {
            "input": "介绍一下你的能力",
            "output": "我是accelerate，专注于学习教育领域。Run PyTorch training across GPUs with minimal changes."
        }
    ],
    "install_guide": {
        "coze": "在 Coze 平台创建 Bot -> 技能配置 -> 导入此 .skill 文件",
        "dify": "在 Dify 平台创建应用 -> 添加知识库 -> 导入此 .skill 配置",
        "claude": "将 system_prompt 字段内容复制到 Claude 自定义指令中",
        "custom": "将此 .skill 文件加载到你的 AI Agent 框架中，解析 system_prompt 和 model_config 即可使用"
    },
    "scripts": {
        "python": "# accelerate - Python extension\n# Add custom Python logic here\ndef process(input_data):\n    return input_data\n",
        "javascript": "// accelerate - JavaScript extension\n// Add custom JS logic here\nfunction process(inputData) {\n    return inputData;\n}\n"
    },
    "tools": {
        "mcp_servers": [],
        "api_endpoints": []
    },
    "dependencies": {
        "python": [],
        "node": []
    },
    "hooks": {
        "on_load": "echo \"Skill loaded: accelerate\"",
        "on_call": "",
        "on_error": "echo \"Skill error: please check logs\""
    }
}