Qwen3.8-27B-NVFP4 on a 24GB RTX 4090: What a Dense Model Can and Cannot Do

A 27B dense model is a different serving problem than a 35B MoE model. The MoE checkpoint offloads its expert pool to host RAM and leaves room for a 262K-token KV cache on a 24GB card. The dense checkpoint has nothing to offload: its weights sit in VRAM, and the KV cache gets whatever is left.

This article measures exactly what is left, on one specific deployment: Qwen3.8-27B-NVFP4 (NVIDIA Model Optimizer mixed-precision release) served by FreeToken on an RTX 4090 (24.5 GiB). The headline result is two-sided:

• The model fits and serves, and scores well on short-context evals (38/40 qualbench, 51/53 agentbench).
• The KV ceiling is ~5.6K tokens, not the advertised 262,144. On this card, the dense 27B is a short-context interactive model, not a long-context coding server.

Why dense changes the math

Qwen3.6-35B-A3B (the model this site's earlier FreeToken articles serve) is a sparse MoE: 256 experts, 8 active per token. FreeToken's --moe-backend offload keeps the 31.4 GiB expert pool in host RAM and only the hot working set in VRAM. That is why a 35 GB checkpoint runs on a 24 GB card with a 300K-token KV reservation.

Qwen3.8-27B is dense: num_experts == 0, every parameter executes for every token. There is no expert pool to offload, so the --moe-* knobs are inert. The full weight set must live in VRAM, and the KV cache is a second-class citizen.

The model and the setup

The chat template was patched to froggeric v22.4 (the same template the 3.6 deployment uses), with the stock template preserved as .orig.

What actually fits: measured, not estimated

The launch sequence tells the real story. Free VRAM before model load: 23.07 GiB. After the mixed-precision weights load and CUDA graphs are captured: 2.04 GiB free (nvidia-smi shows 22.0–22.3 GiB used of 24.5 GiB).

The KV allocator clamps to what fits, not what was requested. --kv-reserve-tokens 8192, 32768, and 65536 all produced the same result:

Allocating 5577 tokens for KV cache, K + V = 0.34 GiB

With --max-running-requests 4 and --kv-reserve-tokens 65536, the server refused to start at all:

AssertionError: Not enough memory for KV cache, try reducing --num-pages

The working profile is deliberately small:

ft serve \
  --model /mnt/data/berda-models/models/Qwen3.8-27B-NVFP4 \
  --served-model-name qwen3.8-27b-nvfp4 \
  --host 0.0.0.0 --port 8000 \
  --cuda-graph-max-bs 1 --max-running-requests 1 \
  --kv-reserve-tokens 32768 \
  --num-tokenizer 0 \
  --tool-call-parser qwen3_coder --reasoning-parser qwen3

--max-running-requests 1 also shrinks the GatedDeltaNet linear-state pool (25→21 slots), which is part of why this profile fits where the 4-request profile asserts.

In practice, "maximum context" for this model on this card is ~5.6K tokens, roughly the default floor, not the model's native 262K. The advertised max_model_len: 262144 in /v1/models is the model's ceiling, not the served budget; trust the startup log line, not the metadata.

Correctness: qualbench 38/40 (short-context categories)

The qualbench suite (50 tasks, 6 categories) was run against the live server. The long-context category (8K–150K-token needle-in-haystack) was excluded: it cannot fit in a 5.6K KV budget. The remaining 40 tasks:

The two SQL failures are migration-application errors (a unique-constraint dupe handling issue and a DROP COLUMN blocked by a dependent view), not truncation or server faults.

Correctness: agentbench 51/53

The wire-protocol suite (OpenAI chat completions, Responses, Anthropic Messages) ran 51 passed / 2 failed / 11 skipped / 2 xfailed in 116 seconds. The two failures share one pattern:

• test_reasoning_plus_tool_call_replays_cleanly
• test_full_bugfix_loop_completes_with_final_summary

Both send reasoning_effort="high" with a 300-token output budget. The model spent the entire budget on its reasoning trace (finish_reason: length, content: '') and never emitted the tool call. This is the same reasoning-budget trap documented for the 3.6 deployment, but the 3.8-NVFP4 checkpoint is more verbose in reasoning, so tests that pass on 3.6 fail here. It is model-specific behavior under a fixed budget, not a server bug and not a flake.

The 11 skipped tests are 3 chat-template checks (no local template path) and 8 opt-in tests (concurrency + large-context) that require more KV than this card can give the dense model.

Speed

Decode throughput on the 4090 was ~46 tok/s at --max-running-requests 1 (GPU 100%), with prefill at ~32 tok/s for a 587-token prompt. The full 40-task qualbench run took 821 seconds (~14 min), faster than the 3.6 MoE run on the same card, as expected for a smaller dense model.

What this means in practice

• For short-context interactive use (chat, quick tool calls, small prompts), the dense 27B-NVFP4 is a strong fit on a 24GB card: it serves, it is fast, and it scores well on the short-context evals.
• For long-context coding agents, it is not a replacement for the MoE-based 3.6-35B-A3B deployment. The 5.6K KV ceiling means any prompt over ~5K tokens fails, and the agentbench large-context tests cannot run.
• The advertised context is not the served context. This is the same lesson as the 3.6 article, in a sharper form: the model card says 262K, the server says 5.6K, and the difference is the weight footprint, not a misconfiguration.

Lessons

• Dense models do not get the MoE offload trick. The RAM/PCIe path that makes a 35B MoE fit a 24GB card is unavailable to a dense 27B. Weights live in VRAM; KV gets the remainder.
• The KV allocator clamps, it does not fail. An oversized --kv-reserve-tokens silently produces the largest allocation that fits (5,577 tokens here), or, with a larger request profile, a hard startup assertion. Read the startup log line, not the flag value.
• Reasoning-budget truncation is model-specific. The same reasoning_effort="high" + 300-token budget that passes on 3.6 fails on 3.8-NVFP4 because the checkpoint reasons more verbosely. When you change models, re-check the reasoning-budget tests.
• max_model_len in /v1/models is a model property, not a deployment property. The served budget is what the startup log says it is.

Resources

• FreeToken GitHub repository: https://github.com/FlashML-org/FreeToken
• nvidia/Qwen3.8-27B-NVFP4: https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4
• Qwen/Qwen3.8-27B-FP8: https://huggingface.co/Qwen/Qwen3.8-27B-FP8
• llm-tests (qualbench suite): https://github.com/berdachuk/llm-tests

Published on 9/9/2026