DeepSeek Multimodal Development in Practice: Janus-Pro Image Understanding and Generation

Janus-Pro Overview

Janus-Pro-7B is an open-source unified multimodal model released by DeepSeek in January 2025. Its core innovation lies in Decoupled Visual Encoding, which designs separate visual encoders for the "understanding" and "generation" paths, breaking through the trade-off between understanding and generation quality in traditional unified multimodal models.

Core Architecture

Image Understanding: Visual Question Answering

from transformers import AutoModelForCausalLM
from janus.models import MultiModalityCausalLM, VLChatProcessor
from PIL import Image
import torch

model_path = "deepseek-ai/Janus-Pro-7B"
vl_chat_processor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto"
).eval()

# Image Question Answering
conversation = [
    {
        "role": "<|User|>",
        "content": "\nWhat is in this image? Please describe in detail.",
        "images": ["images/demo.jpg"],
    },
    {"role": "<|Assistant|>", "content": ""},
]

prepare_inputs = vl_chat_processor(
    conversations=conversation,
    images=[Image.open("images/demo.jpg")],
    force_batchify=True
).to(vl_gpt.device)

inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)
outputs = vl_gpt.language_model.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
)

answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Understanding result: {answer}")

Image Generation: Text-to-Image

# Text-to-Image Generation
gen_conversation = [
    {
        "role": "<|User|>",
        "content": "Generate an image: a cute orange cat sitting on a windowsill, with sunlight shining on it.",
    },
    {"role": "<|Assistant|>", "content": ""},
]

gen_inputs = vl_chat_processor(
    conversations=gen_conversation,
    force_batchify=True
).to(vl_gpt.device)

gen_inputs_embeds = vl_gpt.prepare_inputs_embeds(**gen_inputs)
gen_outputs = vl_gpt.language_model.generate(
    inputs_embeds=gen_inputs_embeds,
    attention_mask=gen_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    max_new_tokens=2048,
    do_sample=True,
    temperature=0.8,
)

# Decode generated visual tokens into an image
generated_tokens = gen_outputs[0]
decoded_image = vl_gpt.gen_vision_model.decode_code(
    generated_tokens,
    shape=[384, 384]  # Janus-Pro fixed resolution
)
decoded_image.save("generated_cat.png")
print("Image saved as generated_cat.png")

Understanding Capability Scenarios

Generation Tips

• Be specific in prompts: describe scene, style, color, composition
• Supports stylized descriptions: "Van Gogh style", "cyberpunk", "watercolor"
• Fixed resolution 384×384, not suitable for very large images
• Temperature controls creativity: 0.5 for realism, 0.9 for creativity
• Each generation takes about 5-15 seconds (depending on hardware)

Deployment Requirements

Janus-Pro can run smoothly on consumer-grade GPUs (such as RTX 4090), making it an ideal choice for getting started with multimodal development.