DeepSeek Multimodal Tutorial

Comparison Dimension	DeepSeek VL2	DeepSeek Janus
Core Capability	Visual language understanding (unidirectional)	Multimodal understanding + image generation (bidirectional)
Architecture Design	Visual encoder + language model projection	Unified autoregressive Transformer
Image Understanding	Professional-grade, dynamic resolution	Supported, fixed 384x384 resolution
Image Generation	Not supported	Supported, text-to-image generation
OCR Capability	Extremely strong, enhanced by dynamic resolution	Basic support
Model Scale	Tiny / Small / Full three tiers	Janus-Pro-7B (7B parameters)
Typical Scenarios	OCR document recognition, chart analysis, visual question answering	Image captioning, multimodal dialogue, text-to-image creation

• Need high-precision OCR and document understanding: Choose VL2; its dynamic resolution mechanism makes it excel in text recognition
• Need image generation capability: Choose Janus; it supports text-to-image generation at 384x384 resolution
• Need visual question answering and chart analysis: VL2 is the first choice, with higher visual understanding accuracy
• Need a unified multimodal dialogue experience: Janus switches more naturally between understanding and generation
• Limited resources: VL2 Tiny (3B) or VL2 Small (16B) can run on consumer-grade GPUs

Model Version	Parameter Size	Vision Encoder	GPU Memory Requirement	Use Cases
VL2-Tiny	3B	SigLIP-SO400M	~8GB	Rapid prototyping, mobile deployment
VL2-Small	16B	SigLIP-SO400M	~40GB	Research, high-precision OCR
VL2-Full	27B (MoE)	SigLIP-SO400M	~80GB	Professional-grade visual understanding, complex document analysis

# Create virtual environment python -m venv vl2-env vl2-env\Scripts\activate # Install PyTorch (choose based on CUDA version) pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 # Install core dependencies pip install transformers accelerate sentencepiece pip install pillow numpy opencv-python # Install huggingface_hub for model download pip install huggingface_hub # Verify installation python -c "import torch; import transformers; print('VL2 environment ready')"

from huggingface_hub import snapshot_download # Download VL2-Tiny (recommended for beginners) snapshot_download( repo_id="deepseek-ai/deepseek-vl2-tiny", local_dir="./models/deepseek-vl2-tiny", local_dir_use_symlinks=False, resume_download=True, ) # Download VL2-Small (requires more GPU memory) # snapshot_download( # repo_id="deepseek-ai/deepseek-vl2-small", # local_dir="./models/deepseek-vl2-small", # local_dir_use_symlinks=False, # resume_download=True, # ) print("Model download complete")

Tip

If the download speed is slow, you can set the HF mirror: export HF_ENDPOINT=https://hf-mirror.com. For more model download tips, see the DeepSeek Model Download Guide.

from transformers import AutoModelForCausalLM from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM import torch from PIL import Image # Load model and processor model_path = "./models/deepseek-vl2-tiny" vl2_processor = DeepseekVLV2Processor.from_pretrained(model_path) vl2_model = DeepseekVLV2ForCausalLM.from_pretrained( model_path, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, ) # Load image image = Image.open("images/document.jpg").convert("RGB") print(f"Image size: {image.size}")

# Single-image visual question answering conversation = [ { "role": "user", "content": [ {"type": "image", "image": "images/photo.jpg"}, {"type": "text", "text": "What is in this photo? Please describe in detail."}, ], } ] # Prepare inputs prepare_inputs = vl2_processor( conversations=conversation, images=[image], force_batchify=True, system_prompt="", ).to(vl2_model.device) # Inference with torch.no_grad(): inputs_embeds = vl2_model.prepare_inputs_embeds(**prepare_inputs) outputs = vl2_model.language_model.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, max_new_tokens=512, do_sample=False, temperature=0.0, ) answer = vl2_processor.tokenizer.decode( outputs[0].cpu().tolist(), skip_special_tokens=True, ) print("Answer:", answer)

# OCR text extraction - core strength of VL2 conversation = [ { "role": "user", "content": [ {"type": "image", "image": "images/document.jpg"}, {"type": "text", "text": "Please extract all text from this image, preserving the original format and layout."}, ], } ] prepare_inputs = vl2_processor( conversations=conversation, images=[image], force_batchify=True, system_prompt="", ).to(vl2_model.device) with torch.no_grad(): inputs_embeds = vl2_model.prepare_inputs_embeds(**prepare_inputs) outputs = vl2_model.language_model.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, max_new_tokens=1024, do_sample=False, ) ocr_text = vl2_processor.tokenizer.decode( outputs[0].cpu().tolist(), skip_special_tokens=True, ) print("OCR extraction result:\n", ocr_text)

# Chart analysis and data extraction conversation = [ { "role": "user", "content": [ {"type": "image", "image": "images/chart.png"}, {"type": "text", "text": "Please analyze this chart: 1) What type of chart is it? 2) What do the x-axis and y-axis represent? 3) What is the data trend? 4) What are the key findings?"}, ], } ] prepare_inputs = vl2_processor(

Dynamic Resolution Processing Pipeline:

• Calculate the optimal tiling scheme based on the image aspect ratio (e.g., 2x2, 3x2, 1x3, etc.)
• Resize the image to the target resolution and split it into multiple tiles
• Generate a global thumbnail to preserve overall layout information
• Feed all tiles and the thumbnail into the vision encoder separately
• Concatenate the encoded visual features and feed them into the language model

# VL2 processor has built-in dynamic resolution support # Control the tiling mode via the resolution parameter # Custom resolution configuration from deepseek_vl2.models.processing_vlm import VLChatProcessor # Method 1: Use default dynamic resolution vl2_processor = DeepseekVLV2Processor.from_pretrained( model_path, resolution="dynamic", # dynamic resolution mode ) # Method 2: Manually specify tiling options # Set candidate resolution list custom_resolutions = [ (1, 1), # 1x1 = 384x384 (1, 2), # 1x2 = 384x768 (2, 1), # 2x1 = 768x384 (2, 2), # 2x2 = 768x768 (2, 3), # 2x3 = 768x1152 (3, 2), # 3x2 = 1152x768 (3, 3), # 3x3 = 1152x1152 ] # The processor automatically selects the best matching resolution based on the image aspect ratio # No extra configuration needed at inference; the processor handles it automatically conversation = [ { "role": "user", "content": [ {"type": "image", "image": "images/high_res_doc.jpg"}, {"type": "text", "text": "Please extract all text from this high-resolution document."}, ], } ] prepare_inputs = vl2_processor( conversations=conversation, images=[high_res_image], force_batchify=True, system_prompt="", ).to(vl2_model.device) # Check the actual number of tiles used num_tiles = prepare_inputs.get("images_seq_mask", torch.tensor([0])).sum().item() print(f"Dynamic resolution used {num_tiles} tiles + 1 global thumbnail")

Comparison	Fixed Resolution	Dynamic Resolution
OCR Accuracy	Low, small text blurry	High, small text clearly readable
Inference Speed	Fast, single tile	Slower, multiple tiles in parallel
Memory Usage	Low	Higher, increases with number of tiles
Applicable Scenarios	Quick preview, low-resolution images	High-precision OCR, document analysis, chart reading

Best Practices

For document OCR and chart analysis, always use dynamic resolution mode. For simple scene classification or object recognition, you can use fixed resolution to save computational resources. It is recommended to keep the number of tiles within 9 (approximately 1152x1152); beyond that, marginal returns diminish.

def vl2_multi_turn_chat(model, processor, image_paths, questions): """VL2 multi-turn visual dialogue Args: model: Loaded VL2 model processor: VL2 processor image_paths: List of image paths corresponding to each turn questions: List of questions for each turn """ conversation_history = [] images_so_far = [] for i, (img_path, question) in enumerate(zip(image_paths, questions)): # Load current turn image current_image = Image.open(img_path).convert("RGB") images_so_far.append(current_image) # Build user message user_content = [] for j, img in enumerate(images_so_far): user_content.append({"type": "image", "image": img}) user_content.append({"type": "text", "text": question}) conversation_history.append({ "role": "user", "content": user_content, }) # Prepare inputs prepare_inputs = processor( conversations=conversation_history, images=images_so_far, force_batchify=True, system_prompt="", ).to(model.device) # Inference with torch.no_grad(): inputs_embeds = model.prepare_inputs_embeds(**prepare_inputs) outputs = model.language_model.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, max_new_tokens=512, do_sample=False, ) answer = processor.tokenizer.decode( outputs[0].cpu().tolist(), skip_special_tokens=True, ) print(f"[Turn {i+1}] Question: {question}") print(f"[Turn {i+1}] Answer: {answer}\n") # Add assistant reply to conversation history conversation_history.append({ "role": "assistant", "content": answer, }) return conversation_history # Usage example images = ["images/doc_page1.jpg", "images/doc_page2.jpg"] questions = [ "What is the main content of the first page?", "Compared to the second page, what has changed?", ] history = vl2_multi_turn_chat(vl2_model, vl2_processor, images, questions)

• Context length limit: VL2's context window is limited; for long conversations, you need to trim history or use summarization strategies.
• Image accumulation strategy: Each turn you can choose whether to carry historical images; carrying all images consumes more GPU memory.
• Dialogue history structure: Maintain the alternating structure of role (user/assistant) and content to ensure the model understands the conversation flow.
• Image reference: Explicitly reference images in questions (e.g., "in the first image...") to avoid confusion.
• Timely cleanup: When switching conversation topics, it is recommended to reset conversation_history to avoid context pollution.

Optimization Suggestions

For long conversations, it is recommended to keep only the last 3-5 turns as context each turn to avoid exceeding the token limit. If you need to compare multiple images, you can put all images in a single turn instead of sending them across multiple turns.

# Create virtual environment python -m venv janus-env janus-env\Scripts\activate # Install PyTorch pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 # Install Janus core dependencies pip install transformers accelerate pip install pillow numpy pip install sentencepiece # Install huggingface_hub pip install huggingface_hub # Verify installation python -c "import torch; import transformers; print('Janus environment ready')"

from huggingface_hub import snapshot_download # Download Janus-Pro-7B snapshot_download( repo_id="deepseek-ai/Janus-Pro-7B", local_dir="./models/Janus-Pro-7B", local_dir_use_symlinks=False, resume_download=True, ) # You can also download Janus-1.3B (lightweight version) # snapshot_download( # repo_id="deepseek-ai/Janus-1.3B", # local_dir="./models/Janus-1.3B", # local_dir_use_symlinks=False, # resume_download=True, # ) print("Janus model download complete")

Model Version	Parameter Size	Image Understanding	Image Generation	GPU Memory Requirement
Janus-1.3B	1.3B	Basic support	384x384	~6GB
Janus-Pro-7B	7B	Enhanced	384x384 (improved quality)	~20GB

import torch from transformers import AutoModelForCausalLM from janus.models import MultiModalityCausalLM, VLChatProcessor from PIL import Image # Load the Janus model model_path = "./models/Janus-Pro-7B" vl_chat_processor = VLChatProcessor.from_pretrained(model_path) tokenizer = vl_chat_processor.tokenizer vl_gpt = MultiModalityCausalLM.from_pretrained( model_path, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, ) vl_gpt.eval() print("Janus model loaded successfully")

def janus_image_caption(model, processor, image_path): """Janus image captioning""" image = Image.open(image_path).convert("RGB") conversation = [ { "role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": "Please describe the content of this image in detail."}, ], }, ] prepare_inputs = processor( conversations=conversation, images=[image], force_batchify=True, ).to(model.device) with torch.no_grad(): inputs_embeds = model.prepare_inputs_embeds(**prepare_inputs) outputs = model.language_model.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, max_new_tokens=512, do_sample=False, pad_token_id=tokenizer.eos_token_id, ) caption = tokenizer.decode( outputs[0].cpu().tolist(), skip_special_tokens=True, ) return caption # Usage caption = janus_image_caption(vl_gpt, vl_chat_processor, "images/photo.jpg") print("Image caption:", caption)

def janus_vqa(model, processor, image_path, question): """Janus visual question answering""" image = Image.open(image_path).convert("RGB") conversation = [ { "role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": question}, ], }, ] prepare_inputs = processor( conversations=conversation, images=[image], force_batchify=True, ).to(model.device) with torch.no_grad(): inputs_embeds = model.prepare_inputs_embeds(**prepare_inputs) outputs = model.language_model.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, max_new_tokens=512, do_sample=False, pad_token_id=tokenizer.eos_token_id, ) answer = tokenizer.decode( outputs[0].cpu().tolist(), skip_special_tokens=True, ) return answer # Usage answer = janus_vqa(vl_gpt, vl_chat_processor, "images/diagram.png", "How many steps are in this diagram? What is each step?") print("Answer:", answer)

# Janus supports logical reasoning based on image content question = """Please analyze the code logic in this screenshot: 1. What functionality does this code implement? 2. Are there any potential bugs? 3. How can this code be optimized?""" answer = janus_vqa(vl_gpt, vl_chat_processor, "images/code_screenshot.png", question) print("Multimodal reasoning result:\n", answer)

import numpy as np from janus.utils.io import save_image def janus_text_to_image(model, processor, prompt, output_path, **gen_kwargs): """Janus text-to-image generation Args: model: Janus model processor: VLChatProcessor prompt: Image generation prompt output_path: Output image path gen_kwargs: Generation parameters """ # Build generation conversation conversation = [ { "role": "user", "content": [ {"type": "text", "text": prompt}, ], }, ] prepare_inputs = processor( conversations=conversation, images=[], force_batchify=True, ).to(model.device) with torch.no_grad(): inputs_embeds = model.prepare_inputs_embeds(**prepare_inputs) outputs = model.language_model.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, max_new_tokens=gen_kwargs.get("max_new_tokens", 1024), do_sample=gen_kwargs.get("do_sample", True), temperature=gen_kwargs.get("temperature", 1.0), top_p=gen_kwargs.get("top_p", 0.95), pad_token_id=tokenizer.eos_token_id, ) # Decode generated image tokens generated_ids = outputs[0].cpu().tolist() image_tokens = model.decode_image_tokens(generated_ids) # Convert tokens to image image = model.gen_vision_model.decode_code( image_tokens.to(model.device), shape=[1, 8, 24, 24], # Janus image shape ) # Save image decoded_image = image[0].cpu().float().numpy().transpose(1, 2, 0) decoded_image = np.clip((decoded_image * 0.5 + 0.5) * 255, 0, 255).astype(np.uint8) Image.fromarray(decoded_image).save(output_path) print(f"Image saved to: {output_path}") # Generation example prompt = "A serene lake at sunset with mountains in the background, oil painting style" janus_text_to_image( vl_gpt, vl_chat_processor, prompt=prompt, output_path="output/sunset_lake.png", temperature=1.0, do_sample=True, )

Parameter	Description	Recommended Value
temperature	Controls generation diversity. Higher is more random, lower is more deterministic	0.8 - 1.2
top_p	Nucleus sampling threshold, controls candidate token range	0.9 - 0.95
max_new_tokens	Maximum number of tokens to generate, affects image detail	1024 - 2048
do_sample	Whether to use sampling. If False, greedy decoding is used	True (for image generation)

def janus_batch_generate(model, processor, prompts, output_dir="output"): """Generate images in batch, each prompt generates multiple variants""" import os os.makedirs(output_dir, exist_ok=True) for idx, prompt in enumerate(prompts): for variant in range(4): # Generate 4 images per prompt output_path = f"{output_dir}/gen_{idx+1}_v{variant+1}.png" janus_text_to_image( model, processor, prompt=prompt, output_path=output_path, temperature=0.9 + variant * 0.1, # Slightly different each time do_sample=True, ) print(f"Batch generation complete, total {len(prompts) * 4} images") # Usage prompts = [ "A futuristic city with flying cars at night, neon lights", "A traditional Chinese garden with a koi pond and pavilion", "A cute robot reading a book under a tree", ] janus_batch_generate(vl_gpt, vl_chat_processor, prompts)

Tip

Janus's image generation is based on an autoregressive Transformer, and the generation speed depends on max_new_tokens. The 384x384 resolution is suitable for most scenarios; if higher resolution is needed, you can later use a super-resolution model to upscale. It is recommended to write prompts in English; the more specific the description, the better the generation results.

Capability Dimension	DeepSeek VL2	DeepSeek Janus (Pro-7B)
OCR Text Recognition	Extremely strong (with dynamic resolution)	Average
Chart/Document Understanding	Professional grade	Basic support
Visual Question Answering	High accuracy	Medium
Image Captioning	Detailed and precise	Natural and fluent
Image Generation	Not supported	Supported (384x384)
Text-only Conversation	Supported	Supported (more natural)
Inference Speed	Medium (slower with many tiles)	Faster (fixed resolution)
Model Scale	3B / 16B / 27B(MoE)	1.3B / 7B

Scenario	Recommended Model	Reason
Document OCR Recognition	VL2	Dynamic resolution, clear small text
Financial Statement Analysis	VL2	Chart understanding + data extraction
Multimodal Chatbot	Janus	Unified experience of understanding + generation
AI Painting Application	Janus	Native support for text-to-image
Medical Image Analysis	VL2	High-precision fine-grained recognition
Creative Content Generation	Janus	Closed loop of understanding + generation

• VL2 for understanding + Janus for generation: Use VL2 to understand user-uploaded images with high precision, extract key information, then use Janus to generate responses or new images
• VL2 for preprocessing + Janus for conversation: VL2 handles OCR and document parsing, Janus handles natural language conversation and creative generation
• Route by scenario: Route document-related requests to VL2, creative requests to Janus, achieving optimal resource utilization

"""DeepSeek Multimodal AI Application — VL2 Understanding + Janus Generation""" import streamlit as st import torch from PIL import Image import os # Page configuration st.set_page_config( page_title="DeepSeek Multimodal AI", page_icon=None, layout="wide", ) st.title("DeepSeek Multimodal AI Application") st.markdown("Supports VL2 visual understanding and Janus image generation") # Sidebar: Model selection with st.sidebar: st.header("Model Configuration") model_choice = st.radio( "Select Model", ["DeepSeek VL2 (Visual Understanding)", "DeepSeek Janus (Understanding + Generation)"], ) if "VL2" in model_choice: vl2_size = st.selectbox( "VL2 Model Size", ["Tiny (3B)", "Small (16B)", "Full (27B MoE)"], ) st.caption("Tiny recommended for quick testing, Full for production") if "Janus" in model_choice: st.caption("Using Janus-Pro-7B model") gen_mode = st.radio("Mode", ["Image Understanding", "Image Generation"]) st.divider() st.markdown("### About") st.markdown("DeepSeek Multimodal Model Hands-on Tutorial") st.markdown("VL2: OCR / Charts / Visual QA") st.markdown("Janus: Understanding + Text-to-Image") # Load model (cached) @st.cache_resource def load_vl2_model(size="tiny"): """Load VL2 model""" from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM model_map = { "tiny": "deepseek-ai/deepseek-vl2-tiny", "small": "deepseek-ai/deepseek-vl2-small", "full": "deepseek-ai/deepseek-vl2", } model_path = model_map.get(size, model_map["tiny"]) processor = DeepseekVLV2Processor.from_pretrained(model_path) model = DeepseekVLV2ForCausalLM.from_pretrained( model_path, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, ) return model, processor @st.cache_resource def load_janus_model(): """Load Janus model""" from janus.models import MultiModalityCausalLM, VLChatProcessor model_path = "deepseek-ai/Janus-Pro-7B" processor = VLChatProcessor.from_pretrained(model_path) model = MultiModalityCausalLM.from_pretrained( model_path, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, ) model.eval() return model, processor # Main interface if "VL2" in model_choice: st.header("VL2 Visual Understanding") uploaded_file = st.file_uploader( "Upload Image", type=["png", "jpg", "jpeg", "webp"], help="Supports common image formats", ) question = st.text_input( "Enter your question", placeholder="e.g., Please extract all text from this image...", ) task_type = st.selectbox( "Task Type", ["General QA", "OCR Text Extraction", "Chart Analysis", "Visual Grounding", "Image Captioning"], ) if uploaded_file and question: image = Image.open(uploaded_file).convert("RGB") col1, col2 = st.columns(2) with col1: st.image(image, caption="Uploaded Image", use_container_width=True) if st.button("Start Analysis", type="primary"): with col2: with st.spinner("VL2 is analyzing..."): # Build prompt based on task type task_prompts = { "General QA": question, "OCR Text Extraction": f"Please extract all text from this image, preserving original format: {question}", "Chart Analysis": f"Please analyze this chart: {question}", "Visual Grounding": f"Please locate the target object in the image: {question}", "Image Captioning": "Please describe the content of this image in detail.", } # Call VL2 inference model, processor = load_vl2_model(vl2_size.lower().split()[0]) conversation = [{ "role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": task_prompts[task_type]}, ], }] prepare_inputs = processor( conversations=conversation, images=[image], force_batchify=True, system_prompt="", ).to(model.device) with torch.no_grad(): inputs_embeds = model.prepare_inputs_embeds(**prepare_inputs) outputs = model.language_model.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, max_new_tokens=1024, do_sample=False, ) answer = processor.tokenizer.decode( outputs[0].cpu().tolist(), skip_special_tokens=True, ) st.markdown("### Analysis Result") st.markdown(answer) elif "Janus" in model_choice: if gen_mode == "Image Understanding": st.header("Janus Image Understanding") uploaded_file = st.file_uploader( "Upload Image", type=["png", "jpg", "jpeg", "webp"], ) question = st.text_input( "Enter your question", placeholder="Please describe this image...", ) if uploaded_file and question: image = Image.open(uploaded_file).convert("RGB") col1, col2 = st.columns(2) with col1: st.image(image, caption="Uploaded Image", use_container_width=True) if st.button("Start Analysis", type="primary"): with col2: with st.spinner("Janus is analyzing..."): model, processor = load_janus_model() tokenizer = processor.tokenizer conversation = [{ "role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": question}, ], }] prepare_inputs = processor( conversations=conversation, images=[image], force_batchify=True, ).to(model.device) with torch.no_grad(): inputs_embeds = model.prepare_inputs_embeds(**prepare_inputs) outputs = model.language_model.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, max_new_tokens=512, do_sample=False, pad_token_id=tokenizer.eos_token_id, ) answer = tokenizer.decode( outputs[0].cpu().tolist(), skip_special_tokens=True, ) st.markdown("### Analysis Result") st.markdown(answer) else: st.header("Janus Image Generation") prompt = st.text_area( "Enter generation prompt (English recommended)", placeholder="A serene lake at sunset with mountains in the background, oil painting style", height=100, ) col1, col2 = st.columns(2) with col1: temperature = st.slider("Temperature", 0.5, 1.5, 1.0, 0.1) with col2: num_images = st.selectbox("Number of images", [1, 2, 4], index=0) if prompt and st.button("Generate Images", type="primary"): model, processor = load_janus_model() tokenizer = processor.tokenizer for i in range(num_images): with st.spinner(f"Generating image {i+1}/{num_images}..."): conversation = [{ "role": "user", "content": [{"type": "text", "text": prompt}], }] prepare_inputs = processor( conversations=conversation, images=[], force_batchify=True, ).to(model.device) with torch.no_grad(): inputs_embeds = model.prepare_inputs_embeds(**prepare_inputs) outputs = model.language_model.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, max_new_tokens=1024, do_sample=True, temperature=temperature, top_p=0.95, pad_token_id=tokenizer.eos_token_id, ) generated_ids = outputs[0].cpu().tolist() image_tokens = model.decode_image_tokens(generated_ids) gen_image = model.gen_vision_model.decode_code( image_tokens.to(model.device), shape=[1, 8, 24, 24], ) import numpy as np decoded = gen_image[0].cpu().float().numpy().transpose(1, 2, 0) decoded = np.clip((decoded * 0.5 + 0.5) * 255, 0, 255).astype(np.uint8) st.image(decoded, caption=f"Generated image {i+1}", use_container_width=True) # Footer st.divider() st.caption("DeepSeek Multimodal AI Application | VL2 + Janus | Local Deployment, Data Security")

# Install Streamlit pip install streamlit # Run the application streamlit run multimodal_app.py # Open in browser # http://localhost:8501

• VL2 Visual Understanding: Supports image upload, multiple task types (OCR/chart/detection/description), custom questions
• Janus Image Understanding: Upload images and conduct natural language Q&A
• Janus Image Generation: Text-to-image generation, supports adjusting temperature and batch generation
• Model Caching: Uses Streamlit caching mechanism, model loaded only once
• Sidebar Configuration: Flexible switching between models and modes, adjust generation parameters

Deployment Recommendations

For production environments, it is recommended to use VL2-Tiny or Janus-1.3B to reduce hardware costs. For high-performance inference, you can use vLLM or TGI to deploy the model service, and the frontend calls via API. See DeepSeek Deployment Tutorial.