modal

name modal description Serverless GPU cloud for ML jobs and model APIs. version 1.0.1 author Orchestra Research license MIT dependencies ["modal>=1.0"] platforms ["linux","macos","windows"] metadata {"hermes":{"tags":["Infrastructure","Serverless","GPU","Cloud","Deployment","Modal"]}} Modal Serverless GPU Guide to running ML workloads on Modal's serverless GPU cloud platform. When to use Modal Use Modal when: Running GPU-intensive ML workloads without managing infrastructure Deploying ML models as auto-scaling APIs Running batch processing jobs (training, inference, data processing) Need pay-per-second GPU pricing without idle costs Prototyping ML applications quickly Running scheduled jobs (cron-like workloads) Key features: Serverless GPUs : T4, L4, A10G, L40S, A100, H100, H200, B200 on-demand Python-native : Define infrastructure in Python code, no YAML Auto-scaling : Scale to zero, scale to 100+ GPUs instantly Sub-second cold starts : Rust-based infrastructure for fast container launches Container caching : Image layers cached for rapid iteration Web endpoints : Deploy functions as REST APIs with zero-downtime updates Use alternatives instead: RunPod : For longer-running pods with persistent state Lambda Labs : For reserved GPU instances SkyPilot : For multi-cloud orchestration and cost optimization Kubernetes : For complex multi-service architectures Quick start Installation pip install modal modal setup # Opens browser for authentication Hello World with GPU import modal app = modal.App( "hello-gpu" ) @app.function( gpu= "T4" ) def gpu_info (): import subprocess return subprocess.run([ "nvidia-smi" ], capture_output= True , text= True ).stdout @app.local_entrypoint() def main (): print (gpu_info.remote()) Run: modal run hello_gpu.py Basic inference endpoint import modal app = modal.App( "text-generation" ) image = modal.Image.debian_slim().pip_install( "transformers" , "torch" , "accelerate" ) @app.cls( gpu= "A10G" , image=image ) class TextGenerator : @modal.enter() def load_model ( self ): from transformers import pipeline self .pipe = pipeline( "text-generation" , model= "gpt2" , device= 0 ) @modal.method() def generate ( self, prompt: str ) -> str : return self .pipe(prompt, max_length= 100 )[ 0 ][ "generated_text" ] @app.local_entrypoint() def main (): print (TextGenerator().generate.remote( "Hello, world" )) Core concepts Key components Component Purpose App Container for functions and resources Function Serverless function with compute specs Cls Class-based functions with lifecycle hooks Image Container image definition Volume Persistent storage for models/data Secret Secure credential storage Execution modes Command Description modal run script.py Execute and exit modal serve script.py Development with live reload modal deploy script.py Persistent cloud deployment GPU configuration Available GPUs GPU VRAM Best For T4 16GB Budget inference, small models L4 24GB Inference, Ada Lovelace arch A10G 24GB Training/inference, 3.3x faster than T4 L40S 48GB Recommended for inference (best cost/perf) A100-40GB 40GB Large model training A100-80GB 80GB Very large models H100 80GB Fastest, FP8 + Transformer Engine H200 141GB Auto-upgrade from H100, 4.8TB/s bandwidth B200 Latest Blackwell architecture GPU specification patterns # Single GPU @app.function( gpu= "A100" ) # Specific memory variant @app.function( gpu= "A100-80GB" ) # Multiple GPUs (up to 8) @app.function( gpu= "H100:4" ) # GPU with fallbacks @app.function( gpu=[ "H100" , "A100" , "L40S" ] ) # Any available GPU @app.function( gpu= "any" ) Container images # Basic image with pip image = modal.Image.debian_slim(python_version= "3.11" ).pip_install( "torch==2.1.0" , "transformers==4.36.0" , "accelerate" ) # From CUDA base image = modal.Image.from_registry( "nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04" , add_python= "3.11" ).pip_install( "torch" , "transformers" ) # With system packages image = modal.Image.debian_slim().apt_install( "git" , "ffmpeg" ).pip_install( "whisper" ) Persistent storage volume = modal.Volume.from_name( "model-cache" , create_if_missing= True ) @app.function( gpu= "A10G" , volumes={ "/models" : volume} ) def load_model (): import os model_path = "/models/llama-7b" if not os.path.exists(model_path): model = download_model() model.save_pretrained(model_path) volume.commit() # Persist changes return load_from_path(model_path) Web endpoints FastAPI endpoint decorator @app.function() @modal.fastapi_endpoint( method= "POST" ) def predict ( text: str ) -> dict : return { "result" : model.predict(text)} Full ASGI app from fastapi import FastAPI web_app = FastAPI() @web_app.post( "/predict" ) async def predict ( text: str ): return { "result" : await model.predict.remote.aio(text)} @app.function() @modal.asgi_app() def fastapi_app (): return web_app Web endpoint types Decorator Use Case @modal.fastapi_endpoint() Simple function → API @modal.asgi_app() Full FastAPI/Starlette apps @modal.wsgi_app() Django/Flask apps @modal.web_server(port) Arbitrary HTTP servers Dynamic batching @app.function() @modal.batched( max_batch_size= 32 , wait_ms= 100 ) async def batch_predict ( inputs: list [ str ] ) -> list [ dict ]: # Inputs automatically batched return model.batch_predict(inputs) Secrets management # Create secret modal secret create huggingface HF_TOKEN=hf_xxx @app.function( secrets=[modal.Secret.from_name( "huggingface" )] ) def download_model (): import os token = os.environ[ "HF_TOKEN" ] Scheduling @app.function( schedule=modal.Cron( "0 0 * * *" ) ) # Daily midnight def daily_job (): pass @app.function( schedule=modal.Period( hours= 1 ) ) def hourly_job (): pass Performance optimization Cold start mitigation # Modal 1.0 autoscaler params: scaledown_window (was container_idle_timeout). # Input concurrency moved to the @modal.concurrent decorator. @app.function( scaledown_window= 300 ) # Keep warm 5 min @modal.concurrent( max_inputs= 10 ) # Handle concurrent requests per container def inference (): pass Model loading best practices @app.cls( gpu= "A100" ) class Model : @modal.enter() # Run once at container start def load ( self ): self .model = load_model() # Load during warm-up @modal.method() def predict ( self, x ): return self .model(x) Parallel processing @app.function() def process_item ( item ): return expensive_computation(item) @app.function() def run_parallel (): items = list ( range ( 1000 )) # Fan out to parallel containers results = list (process_item. map (items)) return results Common configuration @app.function( gpu= "A100" , memory= 32768 , # 32GB RAM cpu= 4 , # 4 CPU cores timeout= 3600 , # 1 hour max scaledown_window= 120 , # Keep warm 2 min (was container_idle_timeout) retries= 3 , # Retry on failure max_containers= 10 , # Max concurrent containers (was concurrency_limit) min_containers= 1 , # Keep N containers warm (was keep_warm) ) def my_function (): pass Modal 1.0 autoscaler renames (see the migration guide ): container_idle_timeout → scaledown_window concurrency_limit → max_containers keep_warm → min_containers allow_concurrent_inputs=N → the @modal.concurrent(max_inputs=N) decorator Debugging # Test locally if __name__ == "__main__" : result = my_function.local() # View logs # modal app logs my-app Common issues Issue Solution Cold start latency Increase scaledown_window , use @modal.enter() GPU OOM Use larger GPU ( A100-80GB ), enable gradient checkpointing Image build fails Pin dependency versions, check CUDA compatibility Timeout errors Increase timeout , add checkpointing References Advanced Usage - Multi-GPU, distributed training, cost optimization Troubleshooting - Common issues and solutions Resources Documentation : https://modal.com/docs Examples : https://github.com/modal-labs/modal-examples Pricing : https://modal.com/pricing Discord : https://discord.gg/modal