Running Qwen3.6-35B-A3B at 262K Context with FreeToken MoE Offload

A 35 GB FP8 model does not fit in a 24 GB RTX 4090. With a conventional GPU-only runtime, that usually ends the experiment.

The FreeToken serving engine removes that limit.

The model runs on one RTX 4090 by using GPU VRAM for latency-critical weights and caches, host RAM for the expert-weight pool, and PCIe as the transport path between them.

That is the practical value of FreeToken's architecture. It turns a consumer GPU plus ordinary system memory into an elastic inference platform for sparse Mixture-of-Experts (MoE) models. The trade-off is explicit: GPU VRAM must be shared between the MoE expert cache and the KV cache. If that balance is left to the default auto-allocator, the model may advertise a 262K-token context window while the server can actually accept only about 8K tokens. This guide shows how to expose the usable context window, size RAM realistically, and tune the server for coding-oriented workloads.

The tests below use Qwen3.6-35B-A3B-FP8 on one RTX 4090, served remotely through FreeToken's OpenAI-compatible API. They include a 60K-token LAN test, near-262K-context verification, an updated 17-category statistical rerun, concurrency tuning, a 50-task correctness eval, and a systemd autostart setup. A final section compares this deployment against a hosted alternative (deepseek-v4-flash via Ollama) on the same test suites.

Why this architecture matters

Qwen3.6-35B-A3B is a sparse MoE model. The full FP8 checkpoint is about 35 GB, but it does not execute every parameter for every generated token:

• It has 256 experts, with 8 routed experts active per token.
• Attention, routing, embeddings, and other always-needed components remain resident on the GPU.
• The large expert pool remains in host RAM until a requested expert is needed.
• FreeToken keeps recently used experts in a GPU-side cache and fetches cache misses from RAM over PCIe.

This makes a 24 GB GPU useful for a model whose complete checkpoint is larger than its VRAM. It does not make RAM equivalent to VRAM. GPU-resident experts are still much faster. Instead, FreeToken tries to keep the hot working set in GPU memory and only pays the RAM/PCIe cost when routing reaches an expert that is not already cached.

@startuml
skinparam componentStyle rectangle

package "FreeToken server (ft serve)" {
  component "OpenAI-compatible\nAPI" as API
  component "Scheduler and batching" as Scheduler
  component "Attention, router,\nand core weights\n(always on GPU)" as Core
  component "GPU MoE expert cache\n(LRU)" as ExpertCache
  component "GPU KV cache" as KV
}

component "Expert weights in host RAM\n(31.4 GiB observed)" as RAM
component "RTX 4090\n24.5 GiB VRAM" as GPU

API --> Scheduler : chat completions
Scheduler --> Core : token batches
Core --> ExpertCache : route top-8 experts
ExpertCache --> RAM : cache miss\nPCIe transfer
RAM --> ExpertCache : expert tensors
Core --> KV : read/write attention state

Core -[hidden]-> GPU
ExpertCache -[hidden]-> GPU
KV -[hidden]-> GPU
@enduml

The immediate payoff is simple: a local model that normally requires more VRAM than the GPU has becomes usable without splitting it across machines or moving the workload to a cloud endpoint. The system's RAM is not a fallback after VRAM runs out—it is an intentional part of the serving design.

Which checkpoint: BF16, FP8, or NVFP4?

The tests in this article use Qwen3.6-35B-A3B-FP8, but that is one of three published representations of the same post-trained multimodal MoE checkpoint: Qwen/Qwen3.6-35B-A3B (BF16), Qwen/Qwen3.6-35B-A3B-FP8, and nvidia/Qwen3.6-35B-A3B-NVFP4 all share the same 35B total / ~3B active parameters, 256 experts (8 routed + 1 shared active per token), 40 hybrid Gated DeltaNet/attention layers, vision encoder, and 262,144-token native context. They differ only in weight representation—and therefore in footprint, quality risk, and runtime compatibility.

On quality: Qwen states the FP8 checkpoint is nearly identical to the BF16 original, and FP8's larger numeric range makes its quality risk more predictable than 4-bit. NVIDIA's published BF16-versus-NVFP4 comparison (measured on a GB300 configuration) shows very small deltas—between −0.8 and +0.5 points across MMLU Pro, GPQA Diamond, τ²-Bench Telecom, SciCode, AIME 2025, AA-LCR, IFBench, and MMMU Pro. That is a strong result for FP4, but the scores are NVIDIA's, on one configuration; for a coding agent, still validate the workload you actually run (RU/EN coding, structured tool calls, your MCP tools, long multi-turn trajectories).

For this RTX 4090 deployment, the choice was straightforward. The 4090 is an Ada Lovelace card (compute capability 8.9), so NVFP4's Hopper/Blackwell kernels are not an option, and BF16 at ~70 GB of weights would be impractical even with offload. FP8 is the official Qwen format that best fits this GPU. A companion article extends this same FreeToken technique to a 16 GB Blackwell RTX 5060 Ti, where the NVFP4 checkpoint becomes the better choice and roughly doubles throughput relative to FP8.

Hardware and RAM sizing

All primary results in this article were measured on one headless Ubuntu server.

The RTX 4090 used PCIe 4.0 x8 under load. That matters for offload serving because PCIe becomes part of the inference path whenever an expert is absent from the GPU cache. CPU memory bandwidth, PCIe link width, NUMA placement, and the current expert-cache hit rate all affect the price of a cache miss.

At steady state, this deployment consumed about 38 GiB of host RAM, including roughly 31.4 GiB for expert weights ready for offload. The model's approximately 35 GB on-disk FP8 checkpoint should therefore not be confused with the exact live RAM number: FreeToken primarily keeps the expert pool in host memory, while GPU-resident core weights, caches, metadata, Python/runtime memory, and the operating system all consume additional resources.

How much RAM is the minimum?

There is no single safe RAM number for every FreeToken model and setting. It depends on the checkpoint format, the model's non-expert-weight size, the selected context length, the desired concurrent-request count, operating-system usage, and whether other services run on the host.

For this specific configuration—Qwen3.6-35B-A3B-FP8 with one GPU and CPU offload—the practical planning guide is:

The 64 GiB figure is a practical lower bound, not a promise. It leaves roughly 29 GiB after holding a 35 GiB-class model-sized footprint, before accounting for the OS, file cache, FreeToken metadata, process memory, temporary allocations, monitoring, containers, and concurrent users. A setup may boot at lower memory, but a server that barely starts is difficult to operate safely: any peak allocation, another container, or memory pressure can turn an otherwise valid inference request into a crash or swap storm.

For a machine intended to serve coding agents, 96–128 GiB is the more realistic target. The primary test machine has 125 GiB available and still uses only about 38 GiB for the shown Qwen deployment, leaving substantial room for the OS page cache, client tooling, containers, compilation jobs, embeddings, a vector database, or another local service. That margin is operationally valuable even when the model itself appears to fit in 64 GiB.

RAM and VRAM solve different problems

It is useful to separate the roles of system RAM and VRAM:

Adding RAM can make an oversized MoE checkpoint runnable. Adding VRAM generally increases expert-cache capacity and/or KV-cache capacity, reducing misses and improving usable concurrency. A wider PCIe link makes cache misses cheaper. These upgrades help different parts of the system and should not be treated as interchangeable.

Why coding needs long context

A large context window is not merely a benchmark feature for coding workloads. It changes the kind of task a local coding model can complete before the client must summarize, truncate, retrieve, or start a new session.

A realistic software-engineering task may need some combination of:

• A task description, acceptance criteria, and architectural constraints.
• Repository instructions such as AGENTS.md, CLAUDE.md, contribution rules, and style guides.
• Build files, dependency manifests, CI pipelines, Docker configuration, and deployment definitions.
• Several related source files, interfaces, tests, schemas, migrations, and API contracts.
• Stack traces, logs, compiler output, failing tests, and previous agent tool results.
• Conversation history: decisions already made, patches attempted, review feedback, and rejected approaches.

With only an 8K-token effective window, an agent often has to select a tiny slice of this information. That forces aggressive retrieval and summarization, increases the chance that it misses a constraint, and makes multi-file changes less coherent. A larger context does not eliminate the need for retrieval, but it reduces the frequency with which relevant code and instructions fall out of the working set.

Context budgets in practical coding tasks

The exact token count varies with language, whitespace, generated code, comments, and tokenizer behavior, but these are useful planning ranges:

These are working-context ranges, not claims that a model should blindly ingest an entire repository. Sending every file is wasteful and can dilute attention. The practical strategy is retrieval plus large context:

1. Use search, symbol navigation, dependency analysis, or a code index to find likely relevant files.
2. Put high-value repository instructions, interfaces, implementations, tests, and diagnostic output into the model context together.
3. Preserve the agent's prior decisions and tool results long enough to avoid repeated rediscovery.
4. Use the large context reserve for cross-cutting cases, not as a substitute for selecting relevant evidence.

For an agentic coding workflow, 64K is often where multi-file tasks become less brittle. At 128K–262K, the model can retain a much larger slice of a repository investigation, patch review, or migration session without continually compressing its own history. The value is particularly high for Java/Spring applications, monorepos, infrastructure-as-code, generated API clients, and tasks where code, tests, CI configuration, and logs must be reasoned about together.

The 262K context window of Qwen3.6-35B-A3B therefore matters in practice—but only if the inference server allocates enough GPU KV cache to make it available.

The context-window trap

The model reports a maximum context length of 262,144 tokens:

{
  "max_model_len": 262144,
  "context_length": 262144
}

At first, that number was misleading in practice. The FreeToken server started with automatic expert-cache sizing enabled, and its log reported:

Allocating 8255 tokens for KV cache, K + V = 0.16 GiB

The default policy favored the MoE expert cache. It used most available VRAM for cached experts and left KV cache at approximately the --kv-reserve-tokens default floor of 8192 tokens. The model technically supported 262K tokens, but this deployment could accommodate only about 8K prompt-plus-generation tokens.

This is a general deployment lesson: a model card's context limit is a model capability, not evidence that a particular inference server has allocated enough KV cache to expose that capability.

KV memory is unusually inexpensive here

The initial allocation exposed a useful measurement point:

That per-token cost is low for a 35B-class model because Qwen3.6-35B-A3B uses a hybrid attention design: most layers use linear attention, only a subset uses full attention, and grouped-query attention limits the number of KV heads.

The resulting rough KV-memory projection was:

A full 262K-token window is therefore feasible on a 24.5 GiB RTX 4090. The limiting factor was not absolute VRAM capacity; it was the allocation policy between two competing caches.

Reserving KV cache first

The fix was to explicitly reserve enough KV capacity before FreeToken auto-sized the MoE expert cache:

ft serve \
  --model-path ~/models/Qwen3.6-35B-A3B-FP8 \
  --moe-backend offload \
  --served-model-name qwen3.6-35b-a3b \
  --host 0.0.0.0 \
  --port 8000 \
  --cuda-graph-max-bs 8 \
  --max-running-requests 8 \
  --kv-reserve-tokens 300000

The reservation intentionally exceeds 262,144 tokens to provide headroom for page rounding and runtime accounting. On restart, FreeToken reported:

Allocating 300131 tokens for KV cache, K + V = 5.72 GiB
--moe-cache-auto resolved moe_cache_size=2966 num_pages=300131 (prefill_overlap=True)

The allocation moved from the default MoE-heavy layout to a practical full-context layout:

@startuml
title Explicit KV reservation makes the full context usable

rectangle "RTX 4090 VRAM: 24.5 GiB" as VRAM {
  rectangle "Core model weights\nattention and router\n~2 GiB" as Core #lightgray
  rectangle "MoE expert cache\n2,966 slots\n~16 GiB" as MoE #orange
  rectangle "KV cache\n300,131 tokens\n5.72 GiB" as KV #lightgreen
}

note bottom of KV
The model's 262,144-token context
now fits with operational headroom.
end note
@enduml

The cost is real: fewer cached experts means a higher probability of a cache miss, and a miss can require RAM-to-GPU movement over PCIe. But that is exactly why the split should be intentional. This deployment is optimized for a real use case—long-context coding and agent tasks—rather than the largest possible expert cache for short prompts.

Verifying 60K tokens remotely

A long-context setting is only useful if it works through the API from a real client, not merely in local metadata. The server listened on all interfaces:

--host 0.0.0.0 --port 8000

A separate LAN machine sent a synthetic request that tokenized to 60,028 prompt tokens. The request succeeded with HTTP 200 and completed in 25.5 seconds:

{
  "prompt_tokens": 60028,
  "total_tokens": 60077
}

The test confirmed three things:

• The remote client reached the OpenAI-compatible endpoint successfully.
• The server allocated and used far more than the default 8K KV floor.
• The usage.prompt_tokens value verified the context actually processed by the deployment.

For practical validation, usage.prompt_tokens is more trustworthy than a model-card claim or a /v1/models metadata field. Those fields describe what the model supports; usage shows what this specific server configuration accepted.

A minimal remote request looks like this:

curl -sS http://SERVER_IP:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d @long_ctx_request.json

And a Python client can use the same endpoint directly:

from openai import OpenAI

client = OpenAI(
    base_url="http://SERVER_IP:8000/v1",
    api_key="not-needed",
)

response = client.chat.completions.create(
    model="qwen3.6-35b-a3b",
    messages=[{"role": "user", "content": "..."}],
    max_tokens=500,
)

print(response.usage.prompt_tokens)

Verifying near-full 262K context

The 60,028-token test above confirmed that the KV reservation worked, but it left a real gap: it verified less than a quarter of the model's advertised 262,144-token window. A separate follow-up session closed that gap with a request built specifically to approach the model's actual ceiling rather than a convenient round number.

Two things had to be corrected first.

The tokenizer used to size the prompt was not the model's tokenizer. The test harness measures prompt length with tiktoken's cl100k_base encoding, which is convenient for consistent local prompt generation but is not what Qwen3.6-35B-A3B's own tokenizer produces. A prompt sized to 255,440 cl100k_base tokens was rejected by the server:

{"error": {"message": "prompt is too long: 281188 tokens > 262144 maximum (prompt + generation); shorten the prompt or increase the KV cache budget", "type": "invalid_request_error", "code": "context_length_exceeded"}}

The model's tokenizer produced about 10% more tokens than cl100k_base for the same English/code text. Any pipeline that sizes prompts by an approximate tokenizer and assumes it matches the target model's real count should validate against the server's own accounting, not just a client-side estimate. The prompt was rescaled and reverified directly against the running server before it was used in a real benchmark run, converging on 233,426 cl100k_base tokens, which the server measured as 256,903 actual model tokens—about 5,000 tokens under the 262,144 ceiling, leaving headroom for the response.

The benchmark harness itself failed silently on very large prompts. The script that sends each request built the JSON payload and passed it to curl as a literal command-line argument. Linux limits any single execve argument to MAX_ARG_STRLEN (128 KiB), so once the serialized payload crossed roughly a megabyte, subprocess.run raised OSError: [Errno 7] Argument list too long. The harness's broad except Exception: return None, 0, False swallowed that error and reported the iteration as a normal 0.00-second failure, with no indication that the request never reached the network. The fix was to write the JSON payload to a temporary file and invoke curl -d @file instead of passing the JSON inline—curl then reads the body from disk with no argument-length limit.

With both issues corrected, a real coding-review prompt of 256,903 model tokens—a synthetic multi-service codebase containing a deliberately embedded verification marker roughly halfway through the text—was sent end to end:

{
  "usage": {
    "prompt_tokens": 256903,
    "completion_tokens": 511,
    "total_tokens": 257414
  }
}

The request succeeded, and the model's reasoning trace correctly located and transcribed the embedded marker from deep inside the 257K-token input before producing a substantive, accurate review of the deliberately planted bugs (an unbounded buffer that was never cleared, a dict mutated without a lock, an O(n) scan held under a lock, and others). That is a stronger claim than "the request did not error": it is direct evidence that attention across a quarter-million tokens of context was actually being used, not merely accepted and ignored.

Run three times, the full statistical suite—17 prompt categories including this near-262K-token prompt—completed with 51 of 51 iterations succeeding (100%) and this prompt category was again measured as statistically reliable in the latest rerun:

Response time for this prompt averaged 14.76 seconds across three iterations (14.43–15.04 s), most of it prefill: the first iteration processes the entire ~257K-token input from scratch, while subsequent iterations in the same session benefit from FreeToken's prefix cache and reuse most of the previously computed KV state.

This closes the gap left by the earlier 60K test:

• Verified prompt length: 256,903 model tokens (233,426 cl100k_base tokens as generated).
• Verified total context: 257,414 tokens for the full round trip.
• Distance from the model's ceiling: approximately 5,241 tokens, intentionally reserved for the response.
• Content-level verification: the model correctly retrieved a specific fact planted near the middle of the input, not just a token-count success.

The original --kv-reserve-tokens 300000 configuration required no changes to reach this result—it was already sized correctly. What was missing was a prompt built and measured against the model's real tokenizer, and a test harness that did not silently drop large requests.

Throughput and concurrency

Long context and high concurrency are different tuning problems. The KV reservation determines how much context can fit. --max-running-requests determines how much work the scheduler may batch together.

The following test used eight simultaneous requests asking for approximately 200-token completions:

At batch size 8, all eight requests completed in about 14.1 seconds. Raising the ceiling to 16 did not exhaust RAM or VRAM, but it reduced output throughput. The useful concurrency limit arrived before a memory allocation failure.

The likely mechanism is cache pressure. Each request routes to a subset of experts. As more unrelated requests are batched, their combined expert working sets become less cache-friendly. More cache misses mean more PCIe transfers from host RAM, and eventually the additional batching no longer compensates for the fetch overhead.

For this system and model profile, 8 concurrent requests was the useful operating point. More concurrency was possible, but not beneficial.

The CUDA graph batch setting was kept aligned with the production concurrency target:

--cuda-graph-max-bs 8 \
--max-running-requests 8

Statistical single-request benchmark

A single headline number is easy to overinterpret, so the server was tested with a curl-based benchmark harness across 17 realistic prompt categories, three iterations each.

Configuration:

{
  "target": {
    "llm_url": "http://SERVER_IP:8000/v1/chat/completions",
    "model": "qwen3.6-35b-a3b",
    "quant": "fp8",
    "backend": "cuda",
    "runtime": "freetoken"
  },
  "max_tokens": 512,
  "temperature": 0.1,
  "iterations_per_prompt": 3
}

For this statistical rerun, the server used a dedicated single-request profile (--cuda-graph-max-bs 1, --max-running-requests 1, --kv-reserve-tokens 300000) to remove queueing effects. The benchmark sent one request at a time and limited each response to 200 output tokens. Prompt-token counts in this table use the harness tokenizer (cl100k_base) for internal consistency; server-side model-token validation is reported separately above for the near-262K test. The table reports client-observed generation throughput; it should not be interpreted as a pure GPU kernel-only measurement.

All 51 of 51 requests succeeded (100%). By the benchmark's statistical reliability criterion (CV < 20%, no outliers beyond two standard deviations), 16 of 17 prompt categories were reliable. The only exception was the extra-long multi-module prompt at 21.1% CV.

The observed range was about 31–55 tok/s. That range is not primarily explained by prompt length alone. The near-262K prompt ran at 34.6 tok/s, while some much shorter prompts ran slower or faster depending on routing and cache locality. In a batch-1 decode regime, speed is influenced more by per-token compute, expert routing, current cache locality, and API/runtime overhead than by the amount of context already prefetched.

The shortest output samples naturally show a larger relative variance: when only a small number of tokens are generated, fixed client and server timing overhead becomes a larger fraction of the calculated tok/s value.

How to read the numbers

Several distinct metrics should not be collapsed into one “speed” figure:

For example, a single non-concurrent request showed a roughly 95–101 tok/s decode baseline in a focused test. The broader statistical table reports lower client-observed rates because it deliberately includes varied prompts, normal API/runtime behavior, and the effects of reasoning/output composition. Neither is wrong; they answer different questions.

Likewise, the 60K-token request's 25.5-second completion time is an end-to-end long-context result, not a direct measure of token generation alone.

Correctness: a 50-task quality eval

Throughput numbers answer a speed question, not a correctness one. To check whether this deployment actually gets coding tasks right, the server was run through qualbench, a 50-task, model-in-the-loop regression suite (part of the llm-tests repository). Every task sends a fixed prompt to the live server and grades the response with a deterministic check—a real compiler/test runner, a schema validator, or a regex against required phrasing—rather than an LLM-judge. The suite spans six categories:

The run used the production configuration from this article (--moe-backend offload, --kv-reserve-tokens 300000, --cuda-graph-max-bs 4, --max-running-requests 4), with the froggeric v22.4 chat template installed. Every single-task failure was re-run two to three additional times before being logged as a finding, to separate genuine model behavior from the temperature-0 non-determinism this offloaded-MoE setup exhibits (the exact set of experts resident in VRAM depends on prior traffic, so identical prompts can route through slightly different effective weights).

Results: 46/50, no new model-quality defects

A fresh full-suite run on the 4090 (after a server reboot, FreeToken main @ af71ba4) produced the same aggregate score as the original run, but with a different failure mix — and a new, previously invisible failure mode surfaced by the harness's truncation detector:

The SQL-migrations category initially reported 0/6 because the qualbench-pg Postgres container was not running after the reboot; once restarted, the category re-ran at 4/6. The three truncation failures are the interesting new signal: the model reasoned correctly but exhausted max_tokens before producing its answer. This is the same reasoning-budget trap documented in the agentbench suite — a reasoning model can spend 100-200+ tokens deliberating, and with a tight budget the visible answer never appears. The harness now flags finish_reason=length explicitly instead of reporting a misleading "test failed" verdict, which is exactly how these three failures were identified.

Every non-pass is either a known model-family characteristic already documented on the smaller 16 GB RTX 5060 Ti deployment (see the companion article) or the reasoning-budget truncation mode described above — no new model-quality defects appeared on the 4090. Notably, the SQL idempotency task that is a reproducible regression on NVFP4 (and a rare FP8 reasoning-loop failure on the 5060 Ti) passed here, consistent with FP8's documented occasional-failure behavior.

The 4090 also ran the whole suite in one continuous session with no OOM and no restart — including long-context, which crashed the 16 GB card mid-suite under the same workload. The 24.5 GiB VRAM headroom absorbs the sustained-serving memory pressure that the smaller card cannot. This is the practical payoff of the RAM/VRAM sizing discussion above: the same workload that needs careful session management on 16 GB runs unattended on 24 GB.

One harness note if you reproduce this suite: the long-context category imports tiktoken (via its prompt generator), so the harness must be run with the repo's venv python rather than the system python, or that category silently runs 0/0 tasks.

agentbench on the 4090

The companion wire-protocol suite (OpenAI chat completions, Responses, and Anthropic Messages against the live server) ran 53 passed, 11 skipped, 2 xfailed on the default run. The two xfails are confirmed, documented FreeToken bugs (inline <|think_off|> tag content swallowed into reasoning; tool_choice="required" silently behaving like "auto"). The opt-in run (concurrency + large-context) added 59 passed / 2 failed / 3 skipped / 2 xfailed — concurrency 3/3 and large-context 5/5 all passed, and the two failures were transient ConnectionResetErrors under sustained load that both passed on immediate rerun.

Comparison: deepseek-v4-flash via Ollama

The same two test suites were also run against a hosted alternative: deepseek-v4-flash:0731-cloud served by Ollama (192.168.0.73:11434, OpenAI-compatible /v1, 1,048,576-token context). This is a different model family entirely, so the comparison is about the suites' verdicts on a different serving stack, not about Qwen vs DeepSeek on identical weights.

qualbench: 47/50 — the best raw pass rate of any model/hardware combination tested so far:

Two results stand out. First, mcp-tools task 06 passed — the by-design hallucination probe (incomplete information, the model must refuse to call a money-transfer tool) that is expected to fail on every Qwen run. deepseek-v4-flash correctly refused the premature call; it is the first model in the suite to do so. Second, the three SQL-migration failures on deepseek-v4-flash are all non-deterministic (each passed on some reruns: 1/3, 2/3, 3/3) and hit different tasks than the Qwen findings — the NVFP4 idempotency-guard regression and the ADD CONSTRAINT IF NOT EXISTS hallucination did not reproduce here. The suite also ran roughly 5-10x faster (~8 minutes total vs ~40 minutes for the 4090 FP8 run).

agentbench: 50 passed, 4 failed, 11 skipped, 1 xfailed on the default run, with the opt-in concurrency (3/3) and large-context (5/5) tests passing. The four failures are all explainable and none is a model-quality defect:

1. Ollama's /v1/models response has no max_model_len field (API surface gap; the real context is 1M tokens per /api/show).
2. /v1/messages/count_tokens returns 404 (endpoint not implemented by Ollama).
3. One arithmetic canary answered "15" instead of "19" for "12 + 7"; 4/5 direct reruns answered correctly — a flake, with read timeouts observed on the Ollama server under repeated load.
4. A strict-xfail test that documents a FreeToken bug (inline <|think_off|> tag content swallowed into reasoning) passed on Ollama — the behavior is correct there, so the strict-xfail reports a failure. This is a positive result.

The 11 skipped tests are 3 chat-template checks (Ollama does not use the froggeric template) and 8 opt-in tests that were run separately and passed.

The practical takeaway for this article's audience: the FreeToken + Qwen deployment remains the right choice when you need the model's full 262K context on your own hardware with the froggeric template's tool-calling guarantees. But if a hosted endpoint is acceptable, deepseek-v4-flash via Ollama is a strong alternative on these suites — better qualbench score, correct tool-call refusal, 1M-token context, and much faster — with the caveats that its SQL migration output is occasionally non-deterministic and its Ollama endpoint lacks max_model_len and /v1/messages/count_tokens.

Autostart with systemd

A long-context coding server is only useful if it comes back after a reboot. The 4090 host runs FreeToken as two systemd user units (~/.config/systemd/user/), with loginctl enable-linger so they start at boot without a login session:

• freetoken-daemon.service — the supervisor (ft daemon --host 127.0.0.1 --port 1900), Restart=on-failure.
• freetoken-engine.service — a oneshot that waits for the daemon's health endpoint, then loads the model via ft daemon start <model> -- <serve flags>.

The one gotcha that surfaced after a reboot: the units' PATH must include the CUDA 13 toolkit before the system nvcc. The flashinfer JIT compiler invokes nvcc with --compress-mode=size, which only CUDA 13 understands; with the default /usr/bin/nvcc (CUDA 12) the engine died at startup with nvcc fatal: Unknown option '--compress-mode=size' and the API never came up. The fix is to prepend /usr/local/cuda-13.0/bin to PATH and set CUDA_HOME=/usr/local/cuda-13.0 in both units:

Environment=PATH=/usr/local/cuda-13.0/bin:%h/freetoken/.venv/bin:%h/freetoken/.venv/lib/python3.12/site-packages/nvidia/cu13/bin:%h/.local/bin:/usr/local/bin:/usr/bin:/bin
Environment=CUDA_HOME=/usr/local/cuda-13.0

After editing, reload and restart:

systemctl --user daemon-reload
systemctl --user restart freetoken-daemon.service
systemctl --user status freetoken-engine.service

Verify the engine actually came up (not just the daemon):

curl -s http://127.0.0.1:8000/v1/models

This is the same CUDA-13-vs-12 shadowing issue that affects interactive launches — the systemd units just make it invisible until the first reboot, so check the engine's journal (journalctl --user -u freetoken-engine.service) rather than assuming the daemon's health implies the model is loaded.

Reasoning mode

Qwen3.6-35B-A3B can emit a reasoning trace. On very small output budgets, the model may use the available tokens for reasoning before it produces visible final content.

For direct, low-latency answers, disable thinking per request:

curl -sS http://SERVER_IP:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-35b-a3b",
    "messages": [{"role": "user", "content": "Say pong"}],
    "max_tokens": 20,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Whether reasoning should remain enabled is workload-dependent. For coding agents and difficult planning tasks, it may be valuable. For extraction, routing, classification, or simple chat, disabling it prevents small output budgets from being spent before a final answer appears.

Reproducibility checklist

FreeToken performance is sensitive to runtime version, CUDA stack, GPU placement, PCIe topology, RAM bandwidth, and allocator settings. Record the environment alongside benchmark results:

ft --version
python --version
uv pip freeze | grep -Ei 'freetoken|torch|triton|tvm'
nvidia-smi
nvcc --version
git -C ~/freetoken rev-parse HEAD 2>/dev/null || true

Capture PCIe state while the model is actively generating, not only while idle:

watch -n 1 \
'nvidia-smi --query-gpu=timestamp,name,utilization.gpu,memory.used,pcie.link.gen.current,pcie.link.width.current --format=csv,noheader'

Also retain the detailed negotiated link state from lspci:

lspci -vv | grep -A22 -Ei 'NVIDIA|VGA|3D'

A low pcie.link.gen.current observed while the GPU is idle may reflect power management, such as ASPM, rather than the link's true ceiling. Check it under sustained workload. The stable facts to report are the maximum supported link, the negotiated width under load, and the measured workload result.

Final launch command

This was the production-oriented command used for the RTX 4090 long-context configuration (the same flags are baked into the systemd engine unit above):

source ~/freetoken/activate.sh

setsid nohup ft serve \
  --model /mnt/data/berda-models/models/Qwen3.6-35B-A3B-FP8 \
  --port 1919 \
  --moe-backend offload \
  --served-model-name qwen3.6-35b-a3b \
  --host 0.0.0.0 \
  --port 8000 \
  --cuda-graph-max-bs 4 \
  --max-running-requests 4 \
  --kv-reserve-tokens 300000 \
  --num-tokenizer 0 \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  > ~/qwen36_fp8_4090_serve_v2.log 2>&1 < /dev/null &

Note the flags that differ from the earlier tuning experiments: --cuda-graph-max-bs 4 --max-running-requests 4 (the concurrency experiments above used 8; 4 is the production choice for this deployment), --num-tokenizer 0 (the model directory carries its own tokenizer), and the two --port flags — 1919 is the daemon's internal engine port, 8000 is the public API port.

At steady state, the process used approximately:

• 22.1–23.2 GiB of 24.5 GiB VRAM;
• about 38 GiB of 125 GiB RAM;
• about 31.4 GiB of RAM for the expert-weight pool;
• a dynamically scaled mamba-slot pool between 24 and 96 slots under load.

Lessons

• FreeToken makes system RAM operationally useful for local inference. A 35 GB FP8 MoE checkpoint ran on one 24 GB RTX 4090 because FreeToken retained the active GPU working set in VRAM and stored the wider expert pool in host RAM.
• 64 GiB is a practical entry point; 96–128 GiB is a server-grade target. For this FP8 model, 64 GiB may be sufficient for a basic single-user setup, but 96–128 GiB provides the operational margin needed for long-context coding, tools, containers, benchmarks, and predictable behavior under memory pressure.
• Long context is a coding feature, not a vanity metric. It lets agents retain instructions, source files, tests, logs, architecture decisions, and prior tool output together. Retrieval remains necessary, but a larger working window reduces destructive truncation and repeated rediscovery.
• Advertised context is not deployed context. The model supported 262,144 tokens, but the default allocation left room for only 8,255. Explicit KV reservation was required to make long context real.
• The long-context trade-off is manageable for this model. Reserving 300K KV tokens used 5.72 GiB of VRAM and still left enough memory for a 2,966-slot MoE cache. The server processed a verified 60,028-token remote prompt, and a later test with the same configuration pushed a real coding-review prompt to 256,903 model tokens—within about 5,000 tokens of the 262,144 ceiling—with the model correctly retrieving a fact planted deep in the input.
• Client-side tokenizers can disagree with the model's own tokenizer. A prompt sized to a target length using tiktoken's cl100k_base encoding was about 10% short of the model's real token count and was rejected as too long. Validate prompt size against the serving model directly before relying on an approximate tokenizer for anything near a context limit.
• A benchmark harness can fail silently at scale. Passing a large JSON payload to curl as a command-line argument hits the Linux kernel's 128 KiB per-argument limit; a broad exception handler turned that failure into a misleading "0.00 second failure" with no error surfaced. Writing the payload to a file and using curl -d @file removed the limit. Any load-testing tool built around shelling out to curl or similar should be checked against this ceiling before it is trusted for large-payload tests.
• System topology matters. In an offload runtime, PCIe and RAM bandwidth are part of the serving architecture. They influence cache-miss cost and determine where concurrency stops scaling.
• More concurrency is not automatically more throughput. On this RTX 4090 system, 8 running requests achieved the best aggregate result at about 114 tok/s. Raising the limit to 16 lowered aggregate throughput to about 79 tok/s because the workload became less cache-friendly and more fetch-bound.
• Measure the workload you intend to serve. A single fast decode figure does not describe long-context prefill, client-observed API latency, multi-request aggregate capacity, or expert-cache behavior. Keep these metrics separate.
• Verify context with actual usage and actual content. usage.prompt_tokens from a successful API response is stronger evidence than a model-card claim, but a request planted specifically to test retrieval from deep in the context—like the marker recovered from the 257K-token prompt—is stronger still. A token-count success does not by itself prove the model attended to the far end of the window.
• A 50-task correctness eval found no new model-quality defects on the 4090. The full qualbench suite ran 46/50; every non-pass was either a known model-family characteristic already documented on the 16 GB RTX 5060 Ti or the reasoning-budget truncation mode. The 4090's larger VRAM also removed the sustained-serving OOM risk entirely—no restart was needed across the whole suite, including long-context.
• A rerun surfaced a new failure mode: reasoning-budget truncation. Three of the four non-passes in the fresh run were finish_reason=length — the model reasoned correctly but exhausted max_tokens before emitting its answer. This was invisible in earlier runs because the harness reported a generic "test failed"; the fix was to flag truncation explicitly. If your eval harness does not check finish_reason, add it: a truncated-but-parseable answer can grade PASS or FAIL for the wrong reason.
• Autostart is only as good as the environment it inherits. The systemd units came up after a reboot, but the engine died because the units' PATH resolved nvcc to CUDA 12, which flashinfer's JIT cannot use. Prepend the CUDA 13 toolkit to PATH and set CUDA_HOME in the units, and verify the engine (not just the daemon) with a real /v1/models request after every reboot.
• Transient connection resets under sustained load are not regressions. Two agentbench tests failed with ConnectionResetError during the full opt-in run and both passed on immediate rerun. When a long suite runs against a busy server, rerun the failing tests before treating them as findings.
• A hosted alternative can beat the local deployment on these suites. deepseek-v4-flash via Ollama scored 47/50 on qualbench (best so far, including the first pass on the by-design tool-call-refusal probe) and passed all agentbench opt-in tests; its failures were API-surface gaps and one arithmetic flake. The trade-off is control: the local FreeToken deployment keeps the full 262K context and the froggeric template's tool-calling guarantees on your own hardware.

For a home AI server, this is the architectural payoff: FreeToken lets a single consumer GPU serve a model larger than its VRAM, while explicit KV-cache planning makes the model's advertised long context useful for real coding and agent workflows. The GPU still determines the hot-path speed, but host RAM and PCIe become first-class design resources rather than a fallback after VRAM runs out. And the number on the model card is only a starting point—treat it as verified only once a real request, sized against the model's own tokenizer, has actually reached that range and the model has demonstrably used what it read.

Resources

• FreeToken GitHub repository: https://github.com/FlashML-org/FreeToken
• Qwen/Qwen3.6-35B-A3B (BF16): https://huggingface.co/Qwen/Qwen3.6-35B-A3B
• Qwen/Qwen3.6-35B-A3B-FP8: https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8
• nvidia/Qwen3.6-35B-A3B-NVFP4: https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4
• llm-tests (qualbench suite): https://github.com/berdachuk/llm-tests

Published on 8/29/2026