Deploying Qwen3.5-27B on Tanzu AI Services

Tanzu AI Services keeps the deployment path straightforward: open the tile, click Add Model, paste the model settings, and run Apply Changes. This post is a field guide for Qwen/Qwen3.5-27B-GPTQ-Int4 on a 4× RTX 3090 worker, including every field and the tuning settings that reached 615 tokens/sec aggregate at 16 concurrent prose streams and 91 ms time to first token on short prompts.

The setup

• Tanzu Platform 10.4 with the AI Services 10.4 tile
• Model: Qwen/Qwen3.5-27B-GPTQ-Int4 (GPTQ Marlin, FP8 KV cache)
• Worker VM: 4× NVIDIA RTX 3090 (96 GB total VRAM), Ubuntu Jammy stemcell
• vLLM 0.19.1 (V1 engine)

The worker hardware

Lab note: this worker is part of my homelab, so the 4× RTX 3090s are consumer-grade Ampere cards. Production AI Services deployments will usually run on enterprise NVIDIA accelerators — A100, H100, L40S, or newer — with higher per-token throughput.

Use the values below as a starting point. The recipe scales cleanly — the engine-sizing dials (max-num-seqs, max-num-batched-tokens, GPU Memory Utilization) adjust to the worker's VRAM — but the right settings still depend on the GPU mix, model, and traffic shape. Validate and tune in your own environment.

Tanzu AI Services hands the IaaS-level GPU plumbing to the platform — BOSH brings up the worker VM, claims the GPUs from the hypervisor, lays down the vLLM tile package, and supervises the process. Your job is one-time: define a matching BOSH vm_type in OpsMan's Resource Config (here named 4x-3090) and assign it to your model in Step 1. Everything below is what that VM looks like under the hood.

GPUs

• 4× NVIDIA RTX 3090, Ampere generation (sm_86)
• 24 GB GDDR6X per card, 96 GB total VRAM across the worker
• Passed through from vSphere into the BOSH-managed worker VM via vCenter's PCI device management
• NVIDIA driver 570.x with CUDA 12.8 runtime, shipped inside the vLLM tile package

PCIe layout

Confirm the topology from inside the worker with nvidia-smi topo -m:

        GPU0    GPU1    GPU2    GPU3
GPU0     X      NV4     PHB     PHB
GPU1    NV4     X      PHB     PHB
GPU2    PHB    PHB      X      NV4
GPU3    PHB    PHB     NV4      X

GPU0↔GPU1 and GPU2↔GPU3 each have a 4-lane NVLink bridge within the pair; the two pairs talk over the PCIe host bridge. vLLM's default tensor-parallel layout (--tensor-parallel-size 4, automatic from the worker's GPU count) handles this gracefully via its custom all-reduce kernel, so the recipe doesn't need to pin the layout manually.

Power

The cards are set to a 200 W per-card power limit (the stock cap is 350 W) via nvidia-smi -pl 200 at host boot. This keeps the worker's aggregate GPU power envelope at 800 W — comfortable for a dual-PSU workstation chassis — with plenty of headroom for the throughput numbers below. Confirm any time with:

nvidia-smi --query-gpu=index,power.limit,power.max_limit --format=csv

BOSH vm_type

The 4x-3090 vm_type referenced in Step 1 is defined in OpsMan's Resource Config tab on the BOSH tile. On vSphere foundations the definition includes the resource pool and the PCI passthrough device IDs for the four cards. Once the vm_type exists you can attach it to any vLLM model in the AI Services tile by name — no further IaaS work needed.

Step 1 — Add the vLLM model

In OpsMan, open AI Services tile → Model Config → Add a vLLM Model. Fill in these fields:

Step 2 — Chat template

Paste this into the Chat Template field. It builds on Qwen3.5's shipped chat template and adds graceful handling for multi-turn agent workflows where the system prompt may be re-injected across tool turns.

If you are serving simple chat only, the stock template may be enough. Use the full template below when you expect agent loops, tool calls, or long multi-turn conversations where the same system context shows up more than once.

{%- set image_count = namespace(value=0) %}
{%- set video_count = namespace(value=0) %}
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
    {%- if content is string %}
        {{- content }}
    {%- elif content is iterable and content is not mapping %}
        {%- for item in content %}
            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
                {%- if is_system_content %}
                    {{- raise_exception('System message cannot contain images.') }}
                {%- endif %}
                {%- if do_vision_count %}
                    {%- set image_count.value = image_count.value + 1 %}
                {%- endif %}
                {%- if add_vision_id %}
                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
                {%- endif %}
                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
            {%- elif 'video' in item or item.type == 'video' %}
                {%- if is_system_content %}
                    {{- raise_exception('System message cannot contain videos.') }}
                {%- endif %}
                {%- if do_vision_count %}
                    {%- set video_count.value = video_count.value + 1 %}
                {%- endif %}
                {%- if add_vision_id %}
                    {{- 'Video ' ~ video_count.value ~ ': ' }}
                {%- endif %}
                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
            {%- elif 'text' in item %}
                {{- item.text }}
            {%- else %}
                {{- raise_exception('Unexpected item type in content.') }}
            {%- endif %}
        {%- endfor %}
    {%- elif content is none or content is undefined %}
        {{- '' }}
    {%- else %}
        {{- raise_exception('Unexpected content type.') }}
    {%- endif %}
{%- endmacro %}
{%- if not messages %}
    {{- raise_exception('No messages provided.') }}
{%- endif %}
{%- if tools and tools is iterable and tools is not mapping %}
    {{- '<|im_start|>system
' }}
    {{- "# Tools

You have access to the following functions:

<tools>" }}
    {%- for tool in tools %}
        {{- "
" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "
</tools>" }}
    {{- '

If you choose to call a function ONLY reply in the following format with NO suffix:

<tool_call>
<function=example_function_name>
<parameter=example_parameter_1>
value_1
</parameter>
<parameter=example_parameter_2>
This is the value for the second parameter
that can span
multiple lines
</parameter>
</function>
</tool_call>

<IMPORTANT>
Reminder:
- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags
- Required parameters MUST be specified
- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after
- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls
</IMPORTANT>' }}
    {%- if messages[0].role == 'system' %}
        {%- set content = render_content(messages[0].content, false, true)|trim %}
        {%- if content %}
            {{- '

' + content }}
        {%- endif %}
    {%- endif %}
    {{- '<|im_end|>
' }}
{%- else %}
    {%- if messages[0].role == 'system' %}
        {%- set content = render_content(messages[0].content, false, true)|trim %}
        {{- '<|im_start|>system
' + content + '<|im_end|>
' }}
    {%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
    {%- set index = (messages|length - 1) - loop.index0 %}
    {%- if ns.multi_step_tool and message.role == "user" %}
        {%- set content = render_content(message.content, false)|trim %}
        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
            {%- set ns.multi_step_tool = false %}
            {%- set ns.last_query_index = index %}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if ns.multi_step_tool %}
    {{- raise_exception('No user query found in messages.') }}
{%- endif %}
{%- for message in messages %}
    {%- set content = render_content(message.content, true)|trim %}
    {%- if message.role == "system" %}
        {%- if not loop.first %}
            {#- RELAXED for agent_buildpack compat: render mid-stream / stacked system messages as their own system block instead of raising. The first system is rendered in the preamble (lines 54-65); only non-first systems land here. #}
            {{- '<|im_start|>system
' + content + '<|im_end|>' + '
' }}
        {%- endif %}
    {%- elif message.role == "user" %}
        {{- '<|im_start|>' + message.role + '
' + content + '<|im_end|>' + '
' }}
    {%- elif message.role == "assistant" %}
        {%- set reasoning_content = '' %}
        {%- if message.reasoning_content is string %}
            {%- set reasoning_content = message.reasoning_content %}
        {%- else %}
            {%- if '</think>' in content %}
                {%- set reasoning_content = content.split('</think>')[0].rstrip('
').split('<think>')[-1].lstrip('
') %}
                {%- set content = content.split('</think>')[-1].lstrip('
') %}
            {%- endif %}
        {%- endif %}
        {%- set reasoning_content = reasoning_content|trim %}
        {%- if loop.index0 > ns.last_query_index %}
            {{- '<|im_start|>' + message.role + '
<think>
' + reasoning_content + '
</think>

' + content }}
        {%- else %}
            {{- '<|im_start|>' + message.role + '
' + content }}
        {%- endif %}
        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
            {%- for tool_call in message.tool_calls %}
                {%- if tool_call.function is defined %}
                    {%- set tool_call = tool_call.function %}
                {%- endif %}
                {%- if loop.first %}
                    {%- if content|trim %}
                        {{- '

<tool_call>
<function=' + tool_call.name + '>
' }}
                    {%- else %}
                        {{- '<tool_call>
<function=' + tool_call.name + '>
' }}
                    {%- endif %}
                {%- else %}
                    {{- '
<tool_call>
<function=' + tool_call.name + '>
' }}
                {%- endif %}
                {%- if tool_call.arguments is defined %}
                    {%- for args_name, args_value in tool_call.arguments|items %}
                        {{- '<parameter=' + args_name + '>
' }}
                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
                        {{- args_value }}
                        {{- '
</parameter>
' }}
                    {%- endfor %}
                {%- endif %}
                {{- '</function>
</tool_call>' }}
            {%- endfor %}
        {%- endif %}
        {{- '<|im_end|>
' }}
    {%- elif message.role == "tool" %}
        {%- if loop.previtem and loop.previtem.role != "tool" %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '
<tool_response>
' }}
        {{- content }}
        {{- '
</tool_response>' }}
        {%- if not loop.last and loop.nextitem.role != "tool" %}
            {{- '<|im_end|>
' }}
        {%- elif loop.last %}
            {{- '<|im_end|>
' }}
        {%- endif %}
    {%- else %}
        {{- raise_exception('Unexpected message role.') }}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant
' }}
    {%- if enable_thinking is defined and enable_thinking is false %}
        {{- '<think>

</think>

' }}
    {%- else %}
        {{- '<think>
' }}
    {%- endif %}
{%- endif %}

Step 3 — Environment Variables

The field accepts a comma-separated list of VAR=VAL pairs (no spaces):

PYTORCH_ALLOC_CONF=expandable_segments:True,VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1,VLLM_USE_FLASHINFER_SAMPLER=1

Step 4 — Advanced Configuration Options

JSON object — each entry is a vLLM CLI flag wrapped as { "value": ... }. Boolean flags use an empty-string value:

{
  "kv-cache-dtype":               { "value": "fp8" },
  "enable-chunked-prefill":       { "value": "" },
  "enable-prefix-caching":        { "value": "" },
  "max-num-seqs":                 { "value": 256 },
  "max-num-batched-tokens":       { "value": 32768 },
  "async-scheduling":             { "value": "" },
  "language-model-only":          { "value": "" },
  "reasoning-parser":             { "value": "qwen3" },
  "block-size":                   { "value": 32 },
  "max-cudagraph-capture-size":   { "value": 256 },
  "enable-flashinfer-autotune":   { "value": "" },
  "performance-mode":             { "value": "throughput" },
  "default-chat-template-kwargs": { "value": "'{\"enable_thinking\": false}'" }
}

Apply changes

Save the model entry, review pending changes, and run Apply Changes. The tile renders a fresh /var/vcap/jobs/vllm-worker/bin/run_worker.sh, brings up the worker, and registers the model with the AI Services broker. Total model load + warmup is roughly 3–5 minutes.

If you prefer infrastructure-as-code, express the same configuration as an om ops file targeting /product-properties/.errands.vllm_models/value/N (replace N with your model's index in the tile's list) and apply via:

om staged-config -p genai > genai-current.yml
om configure-product -c genai-current.yml -o qwen35-add.ops.yml
om apply-changes --product-name genai

Performance numbers

All benches: Qwen/Qwen3.5-27B-GPTQ-Int4, 400-token deterministic prose completion, streaming requests on a 4× RTX 3090 worker.

Throughput + time-to-first-token across concurrency

Concurrency  Aggregate    Avg TTFT   Per-stream decode
1            -            91 ms      57 tok/s
4            200 tok/s    186 ms     51 tok/s
8            373 tok/s    240 ms     48 tok/s
16           615 tok/s    493 ms     40 tok/s

Long-prompt prefill (cold cache)

Input tokens   TTFT       Prefill rate
443            0.21 s     2,087 tok/s
3,105          1.27 s     2,452 tok/s
12,227         4.87 s     2,514 tok/s
24,387         10.34 s    2,358 tok/s

Single-stream prefill sits at ~2.4k tokens/sec and scales linearly with prompt length. Production agent traffic with prefix caching warm consistently shows ~96% prefix-cache hit rate, so real-world TTFT on long histories is much lower than these cold-cache numbers.

What each flag does

A quick walk through the recipe so you can adapt it to your own model and hardware.

Model identity and capabilities

• Name: Qwen3.5-27B — the handle the AI Services broker advertises, and what bound apps see in their multi-model binding.
• Model Name: Qwen/Qwen3.5-27B-GPTQ-Int4 — the HuggingFace repo ID the tile pulls weights from on first start.
• Wire Format: openai — the broker exposes an OpenAI-compatible /v1/chat/completions surface.
• Model Capabilities: chat, tools — surfaces this model to bound apps for chat completion and tool calling.
• Tool Call Parser: qwen3_coder + Enable Auto Tool Choice — enables Qwen3.5's native tool-calling format with automatic tool selection.

Sizing the engine to the hardware

• max-num-seqs: 256 — cap on concurrent sequences vLLM will batch. With 96 GB of VRAM and a 27B-Int4 model there's room to pack many decode steps in parallel.
• max-num-batched-tokens: 32768 — max tokens per scheduling step. A larger budget lets long-context agent prompts fit in a single prefill step, directly improving TTFT on long inputs.
• block-size: 32 — KV cache block size (default 16). Larger blocks halve per-block metadata overhead and suit long-context workloads.
• Max Model Length: 262144 — 256k-token context window, matching Qwen3.5's native max.

Memory headroom

• GPU Memory Utilization: 0.95 — tells vLLM to budget 95% of GPU memory for model + KV cache. Pair with the two profiler env vars below to keep this safe at runtime.
• PYTORCH_ALLOC_CONF=expandable_segments:True — PyTorch's CUDA allocator uses expandable memory segments, reducing fragmentation during burst allocations.
• VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 — makes vLLM's memory profiler include CUDA-graph reservations in its budget calculation, keeping the 0.95 utilization accurate at runtime.
• max-cudagraph-capture-size: 256 — caps the largest captured batch size to match max-num-seqs, freeing GPU memory for KV cache.

Throughput optimizations

• async-scheduling — overlaps token sampling on the GPU with scheduling work on the CPU.
• enable-chunked-prefill — lets the scheduler interleave new request prefills with ongoing decodes in the same step.
• enable-prefix-caching — reuses the KV cache for previously seen prompt prefixes. Agent workloads with repeated system prompts and growing tool histories benefit directly — ~96% hit rate observed in production.
• enable-flashinfer-autotune + VLLM_USE_FLASHINFER_SAMPLER=1 — engages FlashInfer kernel autotune and the FlashInfer top-p/top-k sampler. Look for Using FlashInfer for top-p & top-k sampling in the init log to confirm.
• performance-mode: throughput — vLLM 0.19 preset that bundles throughput-oriented internal defaults.

Quantization and KV cache

• quantization: gptq_marlin — Marlin kernel for GPTQ-Int4 weights, the right choice for Ampere-class GPUs running Int4 models.
• kv-cache-dtype: fp8 — stores KV cache in FP8 instead of FP16, halving per-token KV memory and roughly doubling concurrent capacity at the same context length.

Model behavior

• reasoning-parser: qwen3 — parses Qwen3.5's thinking-mode markers into a separate reasoning_content field on the response.
• default-chat-template-kwargs: '{"enable_thinking": false}' — sets enable_thinking=false as the chat-template default. Skips Qwen3.5's CoT output for use cases that want a direct answer.
• language-model-only — drops multimodal head loading since we're only serving text.
• Custom chat template — the version in Step 2 handles multi-turn agent workflows where the system prompt is re-injected across tool turns.

Wrap up

After Apply Changes, the tile-rendered run_worker.sh and process environment reflect this config exactly, and the model is bound automatically to any tanzu-all-models service instance on the foundation. The same field structure works for any vLLM model — treat max-num-seqs, max-num-batched-tokens, and GPU Memory Utilization as your primary tuning dials, and turn on enable-flashinfer-autotune + performance-mode: throughput as a low-risk throughput baseline.

Keep the configuration in your platform-config repo so teammates can reuse it and you can roll the same recipe to other foundations with om apply-changes.