Running Qwen3.6-35B-A3B at 262K Context with FreeToken MoE Offload

A 35 GB FP8 model does not fit in a 24 GB RTX 4090. With a conventional GPU-only runtime, that usually ends the experiment.

The FreeToken serving engine removes that limit.

The model runs on one RTX 4090 by using GPU VRAM for latency-critical weights and caches, host RAM for the expert-weight pool, and PCIe as the transport path between them.

That is the practical value of FreeToken's architecture. It turns a consumer GPU plus ordinary system memory into an elastic inference platform for sparse Mixture-of-Experts (MoE) models. The trade-off is explicit: GPU VRAM must be shared between the MoE expert cache and the KV cache. If that balance is left to the default auto-allocator, the model may advertise a 262K-token context window while the server can actually accept only about 8K tokens. This guide shows how to expose the usable context window, size RAM realistically, and tune the server for coding-oriented workloads.

The tests below use Qwen3.6-35B-A3B-FP8 on one RTX 4090, served remotely through FreeToken's OpenAI-compatible API. They include a 60K-token LAN test, near-262K-context verification, an updated 17-category statistical rerun, and concurrency tuning.

Why this architecture matters

Qwen3.6-35B-A3B is a sparse MoE model. The full FP8 checkpoint is about 35 GB, but it does not execute every parameter for every generated token:

• It has 256 experts, with 8 routed experts active per token.
• Attention, routing, embeddings, and other always-needed components remain resident on the GPU.
• The large expert pool remains in host RAM until a requested expert is needed.
• FreeToken keeps recently used experts in a GPU-side cache and fetches cache misses from RAM over PCIe.

This makes a 24 GB GPU useful for a model whose complete checkpoint is larger than its VRAM. It does not make RAM equivalent to VRAM. GPU-resident experts are still much faster. Instead, FreeToken tries to keep the hot working set in GPU memory and only pays the RAM/PCIe cost when routing reaches an expert that is not already cached.

@startuml
skinparam componentStyle rectangle

package "FreeToken server (ft serve)" {
  component "OpenAI-compatible\nAPI" as API
  component "Scheduler and batching" as Scheduler
  component "Attention, router,\nand core weights\n(always on GPU)" as Core
  component "GPU MoE expert cache\n(LRU)" as ExpertCache
  component "GPU KV cache" as KV
}

component "Expert weights in host RAM\n(31.4 GiB observed)" as RAM
component "RTX 4090\n24.5 GiB VRAM" as GPU

API --> Scheduler : chat completions
Scheduler --> Core : token batches
Core --> ExpertCache : route top-8 experts
ExpertCache --> RAM : cache miss\nPCIe transfer
RAM --> ExpertCache : expert tensors
Core --> KV : read/write attention state

Core -[hidden]-> GPU
ExpertCache -[hidden]-> GPU
KV -[hidden]-> GPU
@enduml

The immediate payoff is simple: a local model that normally requires more VRAM than the GPU has becomes usable without splitting it across machines or moving the workload to a cloud endpoint. The system's RAM is not a fallback after VRAM runs out—it is an intentional part of the serving design.

Which checkpoint: BF16, FP8, or NVFP4?

The tests in this article use Qwen3.6-35B-A3B-FP8, but that is one of three published representations of the same post-trained multimodal MoE checkpoint: Qwen/Qwen3.6-35B-A3B (BF16), Qwen/Qwen3.6-35B-A3B-FP8, and nvidia/Qwen3.6-35B-A3B-NVFP4 all share the same 35B total / ~3B active parameters, 256 experts (8 routed + 1 shared active per token), 40 hybrid Gated DeltaNet/attention layers, vision encoder, and 262,144-token native context. They differ only in weight representation—and therefore in footprint, quality risk, and runtime compatibility.

On quality: Qwen states the FP8 checkpoint is nearly identical to the BF16 original, and FP8's larger numeric range makes its quality risk more predictable than 4-bit. NVIDIA's published BF16-versus-NVFP4 comparison (measured on a GB300 configuration) shows very small deltas—between −0.8 and +0.5 points across MMLU Pro, GPQA Diamond, τ²-Bench Telecom, SciCode, AIME 2025, AA-LCR, IFBench, and MMMU Pro. That is a strong result for FP4, but the scores are NVIDIA's, on one configuration; for a coding agent, still validate the workload you actually run (RU/EN coding, structured tool calls, your MCP tools, long multi-turn trajectories).

For this RTX 4090 deployment, the choice was straightforward. The 4090 is an Ada Lovelace card (compute capability 8.9), so NVFP4's Hopper/Blackwell kernels are not an option, and BF16 at ~70 GB of weights would be impractical even with offload. FP8 is the official Qwen format that best fits this GPU. A companion article extends this same FreeToken technique to a 16 GB Blackwell RTX 5060 Ti, where the NVFP4 checkpoint becomes the better choice and roughly doubles throughput relative to FP8.

Hardware and RAM sizing

All primary results in this article were measured on one headless Ubuntu server.

The RTX 4090 used PCIe 4.0 x8 under load. That matters for offload serving because PCIe becomes part of the inference path whenever an expert is absent from the GPU cache. CPU memory bandwidth, PCIe link width, NUMA placement, and the current expert-cache hit rate all affect the price of a cache miss.

At steady state, this deployment consumed about 38 GiB of host RAM, including roughly 31.4 GiB for expert weights ready for offload. The model's approximately 35 GB on-disk FP8 checkpoint should therefore not be confused with the exact live RAM number: FreeToken primarily keeps the expert pool in host memory, while GPU-resident core weights, caches, metadata, Python/runtime memory, and the operating system all consume additional resources.

How much RAM is the minimum?

There is no single safe RAM number for every FreeToken model and setting. It depends on the checkpoint format, the model's non-expert-weight size, the selected context length, the desired concurrent-request count, operating-system usage, and whether other services run on the host.

For this specific configuration—Qwen3.6-35B-A3B-FP8 with one GPU and CPU offload—the practical planning guide is:

The 64 GiB figure is a practical lower bound, not a promise. It leaves roughly 29 GiB after holding a 35 GiB-class model-sized footprint, before accounting for the OS, file cache, FreeToken metadata, process memory, temporary allocations, monitoring, containers, and concurrent users. A setup may boot at lower memory, but a server that barely starts is difficult to operate safely: any peak allocation, another container, or memory pressure can turn an otherwise valid inference request into a crash or swap storm.

For a machine intended to serve coding agents, 96–128 GiB is the more realistic target. The primary test machine has 125 GiB available and still uses only about 38 GiB for the shown Qwen deployment, leaving substantial room for the OS page cache, client tooling, containers, compilation jobs, embeddings, a vector database, or another local service. That margin is operationally valuable even when the model itself appears to fit in 64 GiB.

RAM and VRAM solve different problems

It is useful to separate the roles of system RAM and VRAM:

Adding RAM can make an oversized MoE checkpoint runnable. Adding VRAM generally increases expert-cache capacity and/or KV-cache capacity, reducing misses and improving usable concurrency. A wider PCIe link makes cache misses cheaper. These upgrades help different parts of the system and should not be treated as interchangeable.

Why coding needs long context

A large context window is not merely a benchmark feature for coding workloads. It changes the kind of task a local coding model can complete before the client must summarize, truncate, retrieve, or start a new session.

A realistic software-engineering task may need some combination of:

• A task description, acceptance criteria, and architectural constraints.
• Repository instructions such as AGENTS.md, CLAUDE.md, contribution rules, and style guides.
• Build files, dependency manifests, CI pipelines, Docker configuration, and deployment definitions.
• Several related source files, interfaces, tests, schemas, migrations, and API contracts.
• Stack traces, logs, compiler output, failing tests, and previous agent tool results.
• Conversation history: decisions already made, patches attempted, review feedback, and rejected approaches.

With only an 8K-token effective window, an agent often has to select a tiny slice of this information. That forces aggressive retrieval and summarization, increases the chance that it misses a constraint, and makes multi-file changes less coherent. A larger context does not eliminate the need for retrieval, but it reduces the frequency with which relevant code and instructions fall out of the working set.

Context budgets in practical coding tasks

The exact token count varies with language, whitespace, generated code, comments, and tokenizer behavior, but these are useful planning ranges:

These are working-context ranges, not claims that a model should blindly ingest an entire repository. Sending every file is wasteful and can dilute attention. The practical strategy is retrieval plus large context:

1. Use search, symbol navigation, dependency analysis, or a code index to find likely relevant files.
2. Put high-value repository instructions, interfaces, implementations, tests, and diagnostic output into the model context together.
3. Preserve the agent's prior decisions and tool results long enough to avoid repeated rediscovery.
4. Use the large context reserve for cross-cutting cases, not as a substitute for selecting relevant evidence.

For an agentic coding workflow, 64K is often where multi-file tasks become less brittle. At 128K–262K, the model can retain a much larger slice of a repository investigation, patch review, or migration session without continually compressing its own history. The value is particularly high for Java/Spring applications, monorepos, infrastructure-as-code, generated API clients, and tasks where code, tests, CI configuration, and logs must be reasoned about together.

The 262K context window of Qwen3.6-35B-A3B therefore matters in practice—but only if the inference server allocates enough GPU KV cache to make it available.

The context-window trap

The model reports a maximum context length of 262,144 tokens:

{
  "max_model_len": 262144,
  "context_length": 262144
}

At first, that number was misleading in practice. The FreeToken server started with automatic expert-cache sizing enabled, and its log reported:

Allocating 8255 tokens for KV cache, K + V = 0.16 GiB

The default policy favored the MoE expert cache. It used most available VRAM for cached experts and left KV cache at approximately the --kv-reserve-tokens default floor of 8192 tokens. The model technically supported 262K tokens, but this deployment could accommodate only about 8K prompt-plus-generation tokens.

This is a general deployment lesson: a model card's context limit is a model capability, not evidence that a particular inference server has allocated enough KV cache to expose that capability.

KV memory is unusually inexpensive here

The initial allocation exposed a useful measurement point:

That per-token cost is low for a 35B-class model because Qwen3.6-35B-A3B uses a hybrid attention design: most layers use linear attention, only a subset uses full attention, and grouped-query attention limits the number of KV heads.

The resulting rough KV-memory projection was:

A full 262K-token window is therefore feasible on a 24.5 GiB RTX 4090. The limiting factor was not absolute VRAM capacity; it was the allocation policy between two competing caches.

Reserving KV cache first

The fix was to explicitly reserve enough KV capacity before FreeToken auto-sized the MoE expert cache:

ft serve \
  --model-path ~/models/Qwen3.6-35B-A3B-FP8 \
  --moe-backend offload \
  --served-model-name qwen3.6-35b-a3b \
  --host 0.0.0.0 \
  --port 8000 \
  --cuda-graph-max-bs 8 \
  --max-running-requests 8 \
  --kv-reserve-tokens 300000

The reservation intentionally exceeds 262,144 tokens to provide headroom for page rounding and runtime accounting. On restart, FreeToken reported:

Allocating 300131 tokens for KV cache, K + V = 5.72 GiB
--moe-cache-auto resolved moe_cache_size=2966 num_pages=300131 (prefill_overlap=True)

The allocation moved from the default MoE-heavy layout to a practical full-context layout:

@startuml
title Explicit KV reservation makes the full context usable

rectangle "RTX 4090 VRAM: 24.5 GiB" as VRAM {
  rectangle "Core model weights\nattention and router\n~2 GiB" as Core #lightgray
  rectangle "MoE expert cache\n2,966 slots\n~16 GiB" as MoE #orange
  rectangle "KV cache\n300,131 tokens\n5.72 GiB" as KV #lightgreen
}

note bottom of KV
The model's 262,144-token context
now fits with operational headroom.
end note
@enduml

The cost is real: fewer cached experts means a higher probability of a cache miss, and a miss can require RAM-to-GPU movement over PCIe. But that is exactly why the split should be intentional. This deployment is optimized for a real use case—long-context coding and agent tasks—rather than the largest possible expert cache for short prompts.

Verifying 60K tokens remotely

A long-context setting is only useful if it works through the API from a real client, not merely in local metadata. The server listened on all interfaces:

--host 0.0.0.0 --port 8000

A separate LAN machine sent a synthetic request that tokenized to 60,028 prompt tokens. The request succeeded with HTTP 200 and completed in 25.5 seconds:

{
  "prompt_tokens": 60028,
  "total_tokens": 60077
}

The test confirmed three things:

• The remote client reached the OpenAI-compatible endpoint successfully.
• The server allocated and used far more than the default 8K KV floor.
• The usage.prompt_tokens value verified the context actually processed by the deployment.

For practical validation, usage.prompt_tokens is more trustworthy than a model-card claim or a /v1/models metadata field. Those fields describe what the model supports; usage shows what this specific server configuration accepted.

A minimal remote request looks like this:

curl -sS http://SERVER_IP:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d @long_ctx_request.json

And a Python client can use the same endpoint directly:

from openai import OpenAI

client = OpenAI(
    base_url="http://SERVER_IP:8000/v1",
    api_key="not-needed",
)

response = client.chat.completions.create(
    model="qwen3.6-35b-a3b",
    messages=[{"role": "user", "content": "..."}],
    max_tokens=500,
)

print(response.usage.prompt_tokens)

Verifying near-full 262K context

The 60,028-token test above confirmed that the KV reservation worked, but it left a real gap: it verified less than a quarter of the model's advertised 262,144-token window. A separate follow-up session closed that gap with a request built specifically to approach the model's actual ceiling rather than a convenient round number.

Two things had to be corrected first.

The tokenizer used to size the prompt was not the model's tokenizer. The test harness measures prompt length with tiktoken's cl100k_base encoding, which is convenient for consistent local prompt generation but is not what Qwen3.6-35B-A3B's own tokenizer produces. A prompt sized to 255,440 cl100k_base tokens was rejected by the server:

{"error": {"message": "prompt is too long: 281188 tokens > 262144 maximum (prompt + generation); shorten the prompt or increase the KV cache budget", "type": "invalid_request_error", "code": "context_length_exceeded"}}

The model's tokenizer produced about 10% more tokens than cl100k_base for the same English/code text. Any pipeline that sizes prompts by an approximate tokenizer and assumes it matches the target model's real count should validate against the server's own accounting, not just a client-side estimate. The prompt was rescaled and reverified directly against the running server before it was used in a real benchmark run, converging on 233,426 cl100k_base tokens, which the server measured as 256,903 actual model tokens—about 5,000 tokens under the 262,144 ceiling, leaving headroom for the response.

The benchmark harness itself failed silently on very large prompts. The script that sends each request built the JSON payload and passed it to curl as a literal command-line argument. Linux limits any single execve argument to MAX_ARG_STRLEN (128 KiB), so once the serialized payload crossed roughly a megabyte, subprocess.run raised OSError: [Errno 7] Argument list too long. The harness's broad except Exception: return None, 0, False swallowed that error and reported the iteration as a normal 0.00-second failure, with no indication that the request never reached the network. The fix was to write the JSON payload to a temporary file and invoke curl -d @file instead of passing the JSON inline—curl then reads the body from disk with no argument-length limit.

With both issues corrected, a real coding-review prompt of 256,903 model tokens—a synthetic multi-service codebase containing a deliberately embedded verification marker roughly halfway through the text—was sent end to end:

{
  "usage": {
    "prompt_tokens": 256903,
    "completion_tokens": 511,
    "total_tokens": 257414
  }
}

The request succeeded, and the model's reasoning trace correctly located and transcribed the embedded marker from deep inside the 257K-token input before producing a substantive, accurate review of the deliberately planted bugs (an unbounded buffer that was never cleared, a dict mutated without a lock, an O(n) scan held under a lock, and others). That is a stronger claim than "the request did not error": it is direct evidence that attention across a quarter-million tokens of context was actually being used, not merely accepted and ignored.

Run three times, the full statistical suite—17 prompt categories including this near-262K-token prompt—completed with 51 of 51 iterations succeeding (100%) and this prompt category was again measured as statistically reliable in the latest rerun:

Response time for this prompt averaged 14.76 seconds across three iterations (14.43–15.04 s), most of it prefill: the first iteration processes the entire ~257K-token input from scratch, while subsequent iterations in the same session benefit from FreeToken's prefix cache and reuse most of the previously computed KV state.

This closes the gap left by the earlier 60K test:

• Verified prompt length: 256,903 model tokens (233,426 cl100k_base tokens as generated).
• Verified total context: 257,414 tokens for the full round trip.
• Distance from the model's ceiling: approximately 5,241 tokens, intentionally reserved for the response.
• Content-level verification: the model correctly retrieved a specific fact planted near the middle of the input, not just a token-count success.

The original --kv-reserve-tokens 300000 configuration required no changes to reach this result—it was already sized correctly. What was missing was a prompt built and measured against the model's real tokenizer, and a test harness that did not silently drop large requests.

Throughput and concurrency

Long context and high concurrency are different tuning problems. The KV reservation determines how much context can fit. --max-running-requests determines how much work the scheduler may batch together.

The following test used eight simultaneous requests asking for approximately 200-token completions:

At batch size 8, all eight requests completed in about 14.1 seconds. Raising the ceiling to 16 did not exhaust RAM or VRAM, but it reduced output throughput. The useful concurrency limit arrived before a memory allocation failure.

The likely mechanism is cache pressure. Each request routes to a subset of experts. As more unrelated requests are batched, their combined expert working sets become less cache-friendly. More cache misses mean more PCIe transfers from host RAM, and eventually the additional batching no longer compensates for the fetch overhead.

For this system and model profile, 8 concurrent requests was the useful operating point. More concurrency was possible, but not beneficial.

The CUDA graph batch setting was kept aligned with the production concurrency target:

--cuda-graph-max-bs 8 \
--max-running-requests 8

Statistical single-request benchmark

A single headline number is easy to overinterpret, so the server was tested with a curl-based benchmark harness across 17 realistic prompt categories, three iterations each.

Configuration:

{
  "target": {
    "llm_url": "http://SERVER_IP:8000/v1/chat/completions",
    "model": "qwen3.6-35b-a3b",
    "quant": "fp8",
    "backend": "cuda",
    "runtime": "freetoken"
  },
  "max_tokens": 512,
  "temperature": 0.1,
  "iterations_per_prompt": 3
}

For this statistical rerun, the server used a dedicated single-request profile (--cuda-graph-max-bs 1, --max-running-requests 1, --kv-reserve-tokens 300000) to remove queueing effects. The benchmark sent one request at a time and limited each response to 200 output tokens. Prompt-token counts in this table use the harness tokenizer (cl100k_base) for internal consistency; server-side model-token validation is reported separately above for the near-262K test. The table reports client-observed generation throughput; it should not be interpreted as a pure GPU kernel-only measurement.

All 51 of 51 requests succeeded (100%). By the benchmark's statistical reliability criterion (CV < 20%, no outliers beyond two standard deviations), 16 of 17 prompt categories were reliable. The only exception was the extra-long multi-module prompt at 21.1% CV.

The observed range was about 31–55 tok/s. That range is not primarily explained by prompt length alone. The near-262K prompt ran at 34.6 tok/s, while some much shorter prompts ran slower or faster depending on routing and cache locality. In a batch-1 decode regime, speed is influenced more by per-token compute, expert routing, current cache locality, and API/runtime overhead than by the amount of context already prefetched.

The shortest output samples naturally show a larger relative variance: when only a small number of tokens are generated, fixed client and server timing overhead becomes a larger fraction of the calculated tok/s value.

How to read the numbers

Several distinct metrics should not be collapsed into one “speed” figure:

For example, a single non-concurrent request showed a roughly 95–101 tok/s decode baseline in a focused test. The broader statistical table reports lower client-observed rates because it deliberately includes varied prompts, normal API/runtime behavior, and the effects of reasoning/output composition. Neither is wrong; they answer different questions.

Likewise, the 60K-token request's 25.5-second completion time is an end-to-end long-context result, not a direct measure of token generation alone.

Reasoning mode

Qwen3.6-35B-A3B can emit a reasoning trace. On very small output budgets, the model may use the available tokens for reasoning before it produces visible final content.

For direct, low-latency answers, disable thinking per request:

curl -sS http://SERVER_IP:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-35b-a3b",
    "messages": [{"role": "user", "content": "Say pong"}],
    "max_tokens": 20,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Whether reasoning should remain enabled is workload-dependent. For coding agents and difficult planning tasks, it may be valuable. For extraction, routing, classification, or simple chat, disabling it prevents small output budgets from being spent before a final answer appears.

Reproducibility checklist

FreeToken performance is sensitive to runtime version, CUDA stack, GPU placement, PCIe topology, RAM bandwidth, and allocator settings. Record the environment alongside benchmark results:

ft --version
python --version
uv pip freeze | grep -Ei 'freetoken|torch|triton|tvm'
nvidia-smi
nvcc --version
git -C ~/freetoken rev-parse HEAD 2>/dev/null || true

Capture PCIe state while the model is actively generating, not only while idle:

watch -n 1 \
'nvidia-smi --query-gpu=timestamp,name,utilization.gpu,memory.used,pcie.link.gen.current,pcie.link.width.current --format=csv,noheader'

Also retain the detailed negotiated link state from lspci:

lspci -vv | grep -A22 -Ei 'NVIDIA|VGA|3D'

A low pcie.link.gen.current observed while the GPU is idle may reflect power management, such as ASPM, rather than the link's true ceiling. Check it under sustained workload. The stable facts to report are the maximum supported link, the negotiated width under load, and the measured workload result.

Final launch command

This was the production-oriented command used for the RTX 4090 long-context configuration:

source ~/freetoken/.venv/bin/activate
export PATH=/usr/local/cuda/bin:$PATH

nohup ft serve \
  --model-path ~/models/Qwen3.6-35B-A3B-FP8 \
  --moe-backend offload \
  --served-model-name qwen3.6-35b-a3b \
  --host 0.0.0.0 \
  --port 8000 \
  --cuda-graph-max-bs 8 \
  --max-running-requests 8 \
  --kv-reserve-tokens 300000 \
  > ~/qwen36_35b_serve.log 2>&1 & disown

At steady state, the process used approximately:

• 22.1–23.2 GiB of 24.5 GiB VRAM;
• about 38 GiB of 125 GiB RAM;
• about 31.4 GiB of RAM for the expert-weight pool;
• a dynamically scaled mamba-slot pool between 24 and 96 slots under load.

Lessons

• FreeToken makes system RAM operationally useful for local inference. A 35 GB FP8 MoE checkpoint ran on one 24 GB RTX 4090 because FreeToken retained the active GPU working set in VRAM and stored the wider expert pool in host RAM.
• 64 GiB is a practical entry point; 96–128 GiB is a server-grade target. For this FP8 model, 64 GiB may be sufficient for a basic single-user setup, but 96–128 GiB provides the operational margin needed for long-context coding, tools, containers, benchmarks, and predictable behavior under memory pressure.
• Long context is a coding feature, not a vanity metric. It lets agents retain instructions, source files, tests, logs, architecture decisions, and prior tool output together. Retrieval remains necessary, but a larger working window reduces destructive truncation and repeated rediscovery.
• Advertised context is not deployed context. The model supported 262,144 tokens, but the default allocation left room for only 8,255. Explicit KV reservation was required to make long context real.
• The long-context trade-off is manageable for this model. Reserving 300K KV tokens used 5.72 GiB of VRAM and still left enough memory for a 2,966-slot MoE cache. The server processed a verified 60,028-token remote prompt, and a later test with the same configuration pushed a real coding-review prompt to 256,903 model tokens—within about 5,000 tokens of the 262,144 ceiling—with the model correctly retrieving a fact planted deep in the input.
• Client-side tokenizers can disagree with the model's own tokenizer. A prompt sized to a target length using tiktoken's cl100k_base encoding was about 10% short of the model's real token count and was rejected as too long. Validate prompt size against the serving model directly before relying on an approximate tokenizer for anything near a context limit.
• A benchmark harness can fail silently at scale. Passing a large JSON payload to curl as a command-line argument hits the Linux kernel's 128 KiB per-argument limit; a broad exception handler turned that failure into a misleading "0.00 second failure" with no error surfaced. Writing the payload to a file and using curl -d @file removed the limit. Any load-testing tool built around shelling out to curl or similar should be checked against this ceiling before it is trusted for large-payload tests.
• System topology matters. In an offload runtime, PCIe and RAM bandwidth are part of the serving architecture. They influence cache-miss cost and determine where concurrency stops scaling.
• More concurrency is not automatically more throughput. On this RTX 4090 system, 8 running requests achieved the best aggregate result at about 114 tok/s. Raising the limit to 16 lowered aggregate throughput to about 79 tok/s because the workload became less cache-friendly and more fetch-bound.
• Measure the workload you intend to serve. A single fast decode figure does not describe long-context prefill, client-observed API latency, multi-request aggregate capacity, or expert-cache behavior. Keep these metrics separate.
• Verify context with actual usage and actual content. usage.prompt_tokens from a successful API response is stronger evidence than a model-card claim, but a request planted specifically to test retrieval from deep in the context—like the marker recovered from the 257K-token prompt—is stronger still. A token-count success does not by itself prove the model attended to the far end of the window.

For a home AI server, this is the architectural payoff: FreeToken lets a single consumer GPU serve a model larger than its VRAM, while explicit KV-cache planning makes the model's advertised long context useful for real coding and agent workflows. The GPU still determines the hot-path speed, but host RAM and PCIe become first-class design resources rather than a fallback after VRAM runs out. And the number on the model card is only a starting point—treat it as verified only once a real request, sized against the model's own tokenizer, has actually reached that range and the model has demonstrably used what it read.

FreeToken GitHub repository: FlashML-org/FreeToken

Published on 8/29/2026