Tanzu AI Services lets you add open-weight LLMs from the tile without custom deployment code. This walkthrough covers OpenAI's openai/gpt-oss-120b Mixture-of-Experts model on a 4× RTX 3090 worker, including every tile field and the MoE-aware tunings that reached 655 tokens/sec aggregate at 8 concurrent streams and 94 ms time to first token on short prompts.

If you've already read the Qwen3.5 recipe on this blog, much of the field layout will look familiar. The main differences are in the engine tunings: expert parallelism, NCCL all-reduce tuning, and a more conservative memory budget for a 120B-parameter MoE model.

The setup

  • Tanzu Platform 10.4 with the AI Services 10.4 tile
  • Model: openai/gpt-oss-120b (Mixture-of-Experts, native MXFP4 weights, FP8 KV cache)
  • Worker VM: 4× NVIDIA RTX 3090 (96 GB total VRAM), Ubuntu Jammy stemcell
  • vLLM 0.19.1 (V1 engine)

The worker hardware

Lab note: this worker is part of my homelab, so the 4× RTX 3090s are consumer-grade Ampere cards. Production AI Services deployments will usually run on enterprise NVIDIA accelerators — A100, H100, L40S, or newer — where a 120B-class MoE model benefits from faster inter-GPU links.

Use the values below as a starting point. MoE workloads are sensitive to GPU topology because expert tensors are sharded across cards and all-reduce bandwidth sets the throughput ceiling. Validate the settings against your own GPU mix.

Tanzu AI Services hands the IaaS-level GPU plumbing to the platform. BOSH brings up the worker VM, claims the GPUs from the hypervisor, lays down the vLLM tile package, and supervises the process. Your job is one-time: define a matching BOSH vm_type in OpsMan's Resource Config (here named 4x-3090) and assign it to your model in Step 1.

GPUs

  • 4× NVIDIA RTX 3090, Ampere generation (sm_86)
  • 24 GB GDDR6X per card, 96 GB total VRAM across the worker
  • Passed through from vSphere into the BOSH-managed worker VM via vCenter's PCI device management
  • NVIDIA driver 570.x with CUDA 12.8 runtime, shipped inside the vLLM tile package

PCIe layout

Confirm the topology from inside the worker with nvidia-smi topo -m:

        GPU0    GPU1    GPU2    GPU3
GPU0     X      NV4     PHB     PHB
GPU1    NV4     X      PHB     PHB
GPU2    PHB    PHB      X      NV4
GPU3    PHB    PHB     NV4      X

GPU0↔GPU1 and GPU2↔GPU3 each have a 4-lane NVLink bridge within the pair; the two pairs talk over the PCIe host bridge. With --tensor-parallel-size 4 (set automatically from the worker's GPU count) plus enable-expert-parallel, MoE expert tensors are sharded so the all-reduces flow naturally over the NVLink pairs first, with cross-pair PCIe traffic minimized. The NCCL tuning in Step 3 is sized for this layout.

Power

The cards are set to a 200 W per-card power limit (the stock cap is 350 W) via nvidia-smi -pl 200 at host boot. This keeps the worker's aggregate GPU power envelope at 800 W — comfortable for a dual-PSU workstation chassis. Confirm any time with:

nvidia-smi --query-gpu=index,power.limit,power.max_limit --format=csv

BOSH vm_type

The 4x-3090 vm_type referenced in Step 1 is defined in OpsMan's Resource Config tab on the BOSH tile. On vSphere foundations the definition includes the resource pool and the PCI passthrough device IDs for the four cards. Once the vm_type exists you can attach it to any vLLM model in the AI Services tile by name — no further IaaS work needed.

Step 1 — Add the vLLM model

In OpsMan, open AI Services tile → Model Config → Add a vLLM Model. Fill in these fields:

FieldValue
Namegpt-oss-120b
Model Nameopenai/gpt-oss-120b
Model Capabilitieschat, tools
Wire Formatopenai
Tool Call Parseropenai
Enable Auto Tool Choice✓
Quantization(leave blank — gpt-oss ships native MXFP4)
Max Model Length131072
GPU Memory Utilization0.9
Model Implementationauto
Load 8-bit(unchecked)
VM Type4x-3090 (or any 4×GPU VM type)
AZsaz1
Instances1

Step 2 — Chat template

Leave the Chat Template field blank. GPT-OSS ships with the OpenAI harmony chat template baked into its tokenizer config, and vLLM picks it up automatically on model load. No override is needed for chat or tool-calling workflows.

Step 3 — Environment Variables

The field accepts a comma-separated list of VAR=VAL pairs (no spaces). The first four are NCCL all-reduce tuning — essential for MoE on multi-GPU since expert outputs cross the network on every token. The last three are the same memory-profiler + sampler tunings you'd use on any vLLM model.

NCCL_BUFFSIZE=4194304,NCCL_NTHREADS=64,NCCL_P2P_NET_CHUNKSIZE=131072,NCCL_DEBUG=WARN,PYTORCH_ALLOC_CONF=expandable_segments:True,VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1,VLLM_USE_FLASHINFER_SAMPLER=1

Step 4 — Advanced Configuration Options

JSON object — each entry is a vLLM CLI flag wrapped as { "value": ... }. Boolean flags use an empty-string value:

{
  "enable-chunked-prefill":       { "value": "" },
  "enable-prefix-caching":        { "value": "" },
  "max-num-seqs":                 { "value": 256 },
  "enable-expert-parallel":       { "value": "" },
  "async-scheduling":             { "value": "" },
  "reasoning-parser":             { "value": "openai_gptoss" },
  "block-size":                   { "value": 32 },
  "max-cudagraph-capture-size":   { "value": 128 },
  "enable-flashinfer-autotune":   { "value": "" },
  "performance-mode":             { "value": "throughput" }
}

Apply changes

Save the model entry, review pending changes, and run Apply Changes. The tile renders a fresh /var/vcap/jobs/vllm-worker/bin/run_worker.sh, brings up the worker, and registers the model with the AI Services broker. First-boot model download + warmup is roughly 6–8 minutes — gpt-oss-120b is a large model and the inductor/Triton kernel cache rebuilds from scratch on a fresh VM.

If you prefer infrastructure-as-code, express the same configuration as an om ops file targeting /product-properties/.errands.vllm_models/value/N (replace N with your model's index in the tile's list) and apply via:

om staged-config -p genai > genai-current.yml
om configure-product -c genai-current.yml -o gpt-oss-add.ops.yml
om apply-changes --product-name genai

Performance numbers

Numbers below are from a sanity bench on the 4× RTX 3090 worker, 256-token completions of a deterministic prose prompt, streamed via /v1/chat/completions.

Throughput + time-to-first-token

Concurrency  Aggregate    Avg TTFT   Per-stream decode
1            -            94 ms      43 tok/s
8            655 tok/s    ~120 ms    ~82 tok/s

Single-stream decode at ~43 tok/s shows the MoE bottleneck clearly: the active-expert subset of weights is small, but every token traverses the all-reduce, so per-stream latency is gated by NVLink + PCIe rather than raw compute. With concurrency, the same all-reduce amortizes over the batch and aggregate throughput climbs to 655 tok/s at 8 concurrent streams, roughly a 15× speedup over single-stream on the same hardware.

TTFT stays under 200 ms across the concurrency range tested. Bench further on your hardware — on enterprise GPUs with proper NVLink mesh (A100, H100), aggregate throughput at 16 and 32 concurrent streams should scale beyond what's shown here.

What each flag does

A quick walk through the recipe so you can adapt it to your own MoE model and hardware.

Model identity and capabilities

  • Name: gpt-oss-120b — the handle the AI Services broker advertises, and what bound apps see in their multi-model binding.
  • Model Name: openai/gpt-oss-120b — the HuggingFace repo ID the tile pulls weights from on first start.
  • Wire Format: openai — the broker exposes an OpenAI-compatible /v1/chat/completions surface.
  • Model Capabilities: chat, tools — surfaces this model to bound apps for chat completion and tool calling.
  • Tool Call Parser: openai + Enable Auto Tool Choice — uses the OpenAI-native tool-calling format that gpt-oss emits, with automatic tool selection.
  • Quantization: (blank) — gpt-oss ships pre-quantized in MXFP4 and vLLM detects this automatically from the model's config; no quantization-type override is needed.

MoE-specific knobs

  • enable-expert-parallel — the key MoE flag. Shards expert tensors across the GPUs alongside the standard tensor-parallel split, so each card holds a slice of every expert rather than a copy. Without this, 4× 24 GB is not enough to hold 120B parameters.
  • NCCL_BUFFSIZE=4194304, NCCL_NTHREADS=64, NCCL_P2P_NET_CHUNKSIZE=131072 — tunes NCCL's all-reduce for the burst of small messages that MoE expert routing produces. Larger buffers + more worker threads cut all-reduce overhead noticeably on this 4-GPU topology.
  • NCCL_DEBUG=WARN — surfaces NCCL warnings in the vLLM log without flooding it with INFO chatter. Useful when you're profiling expert-parallel traffic.

Sizing the engine to the hardware

  • max-num-seqs: 256 — cap on concurrent sequences vLLM will batch. MoE benefits substantially from larger batches because the routing all-reduce amortizes over more tokens per step.
  • Max Model Length: 131072 — 128k-token context window, matching gpt-oss-120b's training-time max.
  • block-size: 32 — KV cache block size (default 16). Larger blocks halve per-block metadata overhead.

Memory headroom

  • GPU Memory Utilization: 0.9 — tells vLLM to budget 90% of GPU memory for model + KV cache. We deliberately stop short of the 0.95 you'd use on a 27B model because gpt-oss-120b's expert weights + activations leave less headroom for the profiler's CUDA-graph reservation.
  • PYTORCH_ALLOC_CONF=expandable_segments:True — PyTorch's CUDA allocator uses expandable memory segments, reducing fragmentation during burst allocations.
  • VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 — makes vLLM's memory profiler include CUDA-graph reservations in its budget calculation, keeping the 0.9 utilization accurate at runtime.
  • max-cudagraph-capture-size: 128 — caps the largest captured batch size. On a 120B MoE this is half the 256 you'd use on a smaller dense model — capturing graphs at every size up to 256 costs more memory than it's worth here.

Throughput optimizations

  • async-scheduling — overlaps token sampling on the GPU with scheduling work on the CPU.
  • enable-chunked-prefill — lets the scheduler interleave new request prefills with ongoing decodes in the same step.
  • enable-prefix-caching — reuses the KV cache for previously seen prompt prefixes. Agent workloads with repeated system prompts and growing tool histories benefit substantially.
  • enable-flashinfer-autotune + VLLM_USE_FLASHINFER_SAMPLER=1 — engages FlashInfer kernel autotune and the FlashInfer top-p/top-k sampler. Look for Using FlashInfer for top-p & top-k sampling in the init log to confirm.
  • performance-mode: throughput — vLLM 0.19 preset that bundles throughput-oriented internal defaults.

Quantization and KV cache

  • Native MXFP4 weights — gpt-oss-120b's weights ship in OpenAI's MXFP4 mixed-precision format. vLLM detects this from the model config and loads the appropriate kernels without any tile-level quantization override.
  • FP8 KV cache — not explicitly set here because vLLM picks an appropriate KV dtype for gpt-oss automatically. If you want to force FP8, add "kv-cache-dtype": { "value": "fp8" } to the Advanced Configuration Options.

Model behavior

  • reasoning-parser: openai_gptoss — parses gpt-oss's reasoning markers into a separate reasoning_content field on the response. Apps that want only the final answer ignore the field; agent UIs that support reasoning metadata can display it separately.
  • No custom chat template — gpt-oss's tokenizer config carries the harmony template that vLLM uses by default. You only need to set the Chat Template field if you want to override that behavior.

Wrap up

After Apply Changes, the tile-rendered run_worker.sh and process environment reflect this config exactly, and the model is bound automatically to any tanzu-all-models service instance on the foundation. The MoE-specific moves — enable-expert-parallel plus the NCCL all-reduce tuning — are what unlock the throughput on a 4-GPU box. Everything else is the same vLLM kit you'd use for any 0.19-era deployment.

Keep the configuration in your platform-config repo so teammates can reuse it and you can roll the same recipe to other foundations with om apply-changes.