This is the third deployment playbook in the Tanzu AI Services series. It covers google/gemma-4-31B-it — Google's 31B-parameter instruction-tuned Gemma 4, a multimodal model that handles text, tools, and vision — running on a single NVIDIA RTX PRO 6000 Blackwell. At the concurrency sweet spot, that one GPU serves ~2,700 tokens/sec aggregate with a 256k-token context window.

The main difference from the Qwen3.5 and GPT-OSS recipes is the lack of multi-GPU plumbing: one card, no tensor-parallel split, and no NCCL tuning. The variables are the model fields, the FP8 settings, and how a single Blackwell behaves under load.

The setup

  • Tanzu Platform 10.4 with the AI Services 10.4 tile
  • Model: google/gemma-4-31B-it (multimodal: chat, tools, vision; FP8 weights, FP8 KV cache)
  • Worker VM: 1× NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM), Ubuntu Jammy stemcell
  • vLLM 0.19.1 (V1 engine)

The worker hardware

Lab note: this worker is part of my homelab. The card is the Max-Q workstation edition of the RTX PRO 6000 Blackwell — a single-slot, power-efficient variant. Production AI Services deployments will usually run on data-center accelerators (the full RTX PRO 6000, L40S, H100, or newer), where the per-card power and clock ceilings are higher.

Use the values below as a starting point. The recipe is the same across GPUs — the engine sizes itself to whatever VRAM is present — but throughput tracks the specific card. Validate in your own environment.

Tanzu AI Services hands the IaaS-level GPU plumbing to the platform. BOSH brings up the worker VM, claims the GPU from the hypervisor, lays down the vLLM tile package, and supervises the process. Your job is one-time: define a matching BOSH vm_type in OpsMan's Resource Config (here named 1x-blackwell-6000-pro) and assign it to your model in Step 1.

The GPU

  • 1× NVIDIA RTX PRO 6000 Blackwell (Max-Q Workstation Edition), Blackwell generation
  • 96 GB GDDR7 — enough to hold the 31B model in FP8 plus a large KV cache and the vision encoder, all on one card
  • Passed through from vSphere into the BOSH-managed worker VM via vCenter's PCI device management
  • Native FP8 tensor cores — Blackwell runs the FP8-quantized weights and FP8 KV cache directly, no upcast

Single-GPU simplicity

With one GPU, vLLM runs --tensor-parallel-size 1 (set automatically from the worker's GPU count). There's no tensor-parallel all-reduce, no NVLink topology to map, no NCCL tuning. The model, KV cache, and vision tower all live in the one card's 96 GB. This is the simplest possible vLLM serving shape, and on a card this large it's plenty for a 31B model.

Power

The card runs at its stock 300 W limit. Under sustained inference load it sits at 100% utilization and pulls the full 300 W — this is a card that runs flat out when you feed it concurrency. Confirm the limit any time with:

nvidia-smi --query-gpu=index,power.limit,power.max_limit,utilization.gpu --format=csv

BOSH vm_type

The 1x-blackwell-6000-pro vm_type referenced in Step 1 is defined in OpsMan's Resource Config tab on the BOSH tile. On vSphere foundations the definition includes the resource pool and the PCI passthrough device ID for the card. Once the vm_type exists you can attach it to any vLLM model in the AI Services tile by name.

Step 1 — Add the vLLM model

In OpsMan, open AI Services tile → Model Config → Add a vLLM Model. Fill in these fields:

FieldValue
Namegemma-4-31B-it
Model Namegoogle/gemma-4-31B-it
Model Capabilitieschat, tools, vision
Wire Formatopenai
Tool Call Parsergemma4
Enable Auto Tool Choice✓
Quantizationfp8
Max Model Length(leave blank — uses the model's native 262144)
GPU Memory Utilization0.9
Model Implementationauto
Load 8-bit(unchecked)
VM Type1x-blackwell-6000-pro (or any single-GPU VM type with enough VRAM)
AZsaz1
Instances1

Step 2 — Chat template

Leave the Chat Template field blank. Gemma 4 ships its chat template in the tokenizer config, including the multimodal turn structure, and vLLM picks it up automatically on model load. No override is needed for chat, tool-calling, or vision inputs.

Step 3 — Environment Variables

None required. Gemma 4 on a single Blackwell needs no special environment tuning — leave the Environment Variables field empty.

Step 4 — Advanced Configuration Options

JSON object — each entry is a vLLM CLI flag wrapped as { "value": ... }. Boolean flags use an empty-string value:

{
  "kv-cache-dtype":          { "value": "fp8" },
  "enable-chunked-prefill":  { "value": "" },
  "enable-prefix-caching":   { "value": "" },
  "async-scheduling":        { "value": "" },
  "reasoning-parser":        { "value": "gemma4" }
}

That's the whole config. On a single large card there's no batching or memory dial worth overriding — the defaults already pack the engine to the card's compute ceiling (more on that in the performance section).

Apply changes

Save the model entry, review pending changes, and run Apply Changes. The tile renders a fresh /var/vcap/jobs/vllm-worker/bin/run_worker.sh, brings up the worker, and registers the model with the AI Services broker. Model load + warmup is roughly 1–2 minutes once the weights are cached locally.

If you prefer infrastructure-as-code, express the same configuration as an om ops file targeting /product-properties/.errands.vllm_models/value/N (replace N with your model's index in the tile's list) and apply via:

om staged-config -p genai > genai-current.yml
om configure-product -c genai-current.yml -o gemma4-add.ops.yml
om apply-changes --product-name genai

Performance numbers

All benches: google/gemma-4-31B-it, 256-token completions of a deterministic prose prompt, on the single Blackwell worker.

Throughput + time-to-first-token across concurrency

Concurrency  Aggregate     Avg TTFT
1            41 tok/s      63 ms
8            312 tok/s     ~90 ms
32           1,133 tok/s   164 ms
64           1,917 tok/s   120 ms
128          2,700 tok/s   139 ms

Single-stream decode sits at ~40 tok/s — that's the memory-bandwidth-bound case, reading the full weight set per token. Throughput then scales cleanly with concurrency as batching amortizes those weight reads: ~2,700 tokens/sec aggregate at 128 concurrent streams, a roughly 65× speedup over single-stream on the same card.

Watch the GPU during a concurrent run and you'll see it pinned at 100% utilization — the card runs flat out, which is exactly what you want from a serving worker. 128 concurrent is the sweet spot on this card; the 256k context window means the KV cache stays comfortable well past typical agent and chat traffic. TTFT holds in the 60–160 ms range across the band, so interactive latency stays snappy even under heavy batch.

What each flag does

A quick walk through the recipe so you can adapt it to your own model and card.

Model identity and capabilities

  • Name: gemma-4-31B-it — the handle the AI Services broker advertises, and what bound apps see in their multi-model binding.
  • Model Name: google/gemma-4-31B-it — the HuggingFace repo ID the tile pulls weights from on first start.
  • Wire Format: openai — the broker exposes an OpenAI-compatible /v1/chat/completions surface.
  • Model Capabilities: chat, tools, vision — surfaces this model to bound apps for chat completion, tool calling, and image input. The vision tower loads alongside the language model on the same card.
  • Tool Call Parser: gemma4 + Enable Auto Tool Choice — enables Gemma 4's native tool-calling format with automatic tool selection.

Quantization and KV cache

  • quantization: fp8 — serves the model in FP8. On Blackwell this maps directly onto native FP8 tensor cores, so there's no dequant overhead and the 31B model fits comfortably with room to spare for KV cache.
  • kv-cache-dtype: fp8 — stores the KV cache in FP8 instead of FP16, halving per-token KV memory and roughly doubling concurrent capacity at the same context length. Pairs naturally with FP8 weights on Blackwell.
  • GPU Memory Utilization: 0.9 — budgets 90% of the card's 96 GB for model + KV cache + the vision encoder, leaving headroom for activation spikes during multimodal prefill.

Throughput optimizations

  • async-scheduling — overlaps token sampling on the GPU with scheduling work on the CPU.
  • enable-chunked-prefill — lets the scheduler interleave new request prefills with ongoing decodes in the same step, which is what keeps TTFT low even at 128 concurrent.
  • enable-prefix-caching — reuses the KV cache for previously seen prompt prefixes. Agent and chat workloads with repeated system prompts benefit directly.

Model behavior

  • reasoning-parser: gemma4 — parses Gemma 4's reasoning markers into a separate reasoning_content field on the response, so apps can show or hide supported reasoning output independently of the final answer.
  • No custom chat template — Gemma 4's tokenizer config carries the template vLLM uses by default, including the vision turn structure.
  • Native 256k context — leaving Max Model Length blank keeps the model's full 262,144-token window, which the 96 GB card backs with a healthy KV cache.

Wrap up

Gemma 4 31B is the easiest of the three recipes in this series: one card, five advanced-config lines, no environment variables, no multi-GPU plumbing. Drop in a single Blackwell, point the tile at google/gemma-4-31B-it, and you get a multimodal model with a 256k context window serving ~2,700 tokens/sec at the concurrency sweet spot.

After Apply Changes, the model is bound automatically to any tanzu-all-models service instance on the foundation. Keep the configuration in your platform-config repo so teammates can reuse it and you can roll the same recipe to other foundations with om apply-changes.