Merge remote-tracking branch 'macm5us/deepseek4-pro-correctness' into glm5.2

# Conflicts:
#	ds4_metal.m

.gitattributes

@@ -0,0 +1,3 @@
+# Preserve hosted-model continuation bytes exactly; these fixtures may contain
+# model-generated trailing whitespace that is meaningful to tokenizer scoring.
+gguf-tools/quality-testing/data/*/continuations/*.txt -diff

.gitignore

@@ -1,6 +1,8 @@
 /ds4
 /ds4-server
 /ds4-bench
+/ds4-eval
+/ds4-agent
 /ds4_native
 /ds4_server_test
 /ds4_test
@@ -9,6 +11,8 @@
 /gguf/
 *.o
 *.dSYM/
+__pycache__/
+*.pyc
 /misc/
 .*.swp
 .DS_Store

AGENT.md

@@ -7,32 +7,26 @@ Objective-C only where Metal requires it and Metal kernels under `metal/`.
 ## Goals
 
 - Keep the production path as whole-model Metal graph inference.
-- Keep model loading mmap-backed; do not eagerly copy the full GGUF.
+- Always make sure that the SSD streaming, CUDA, distributed inference, Metal default inference are not affected by fixes to other parts of the code.
+- Keep model loading mmap-backed for the Metal default case; do not eagerly copy the full GGUF. Keep the model loading for SSD streaming of routed experts explicit: allocated buffers, fast reads from disk, always try to hide loading of missing routed experts by loading them while performing the inference of the shared expert and routed experts already in RAM. Always try to hide loading of layers for prefill in SSD streaming mode using the inference time of the current layer as the next one is loaded.
 - Keep the CPU backend CPU-only and use it only as reference/debug code.
-- Preserve correctness before speed. Do not keep a faster path with unexplained
-  attention, KV cache, or logits drift.
-- Make long local agent sessions practical through live KV reuse and disk KV
-  checkpoints.
+- Preserve correctness before speed. Do not keep a faster path with unexplained attention, KV cache, or logits drift.
+- Make long local agent sessions practical through live KV reuse and disk KV checkpoints.
 
 ## Quality Rules
 
-- Comment important inference code where the model mechanics, cache lifetime,
-  memory policy, or API orchestration are not obvious from the local code.
+- Keep the implementation small, sharp, easy to understand. Try to write elegant code in a state of grace. Don't settle for the first thing that comes to mind, try to find the most minimal and better working design. Don't introduce slop: very fragile code that just patches specific cases, dead code, useless code and code ways more complicated of how it should be.
+- Comment important inference code where the model mechanics, cache lifetime, memory policy, or API orchestration are not obvious from the local code.
 - Prefer comments beside the implementation over separate design documents.
-- Keep comments instructive and compact: explain why a shape, ordering, cache
-  boundary, or memory choice exists.
+- Keep comments instructive and compact: explain why a shape, ordering, cache boundary, or memory choice exists.
 - Keep public APIs narrow. CLI/server code should not know tensor internals.
-- Do not add permanent semantic variants behind flags. Diagnostic switches are
-  fine when they validate the one release path.
+- Do not add permanent semantic variants behind flags. Diagnostic switches are fine when they validate the one release path.
 - Do not introduce C++.
 
 ## Safety
 
-- Avoid large CPU inference runs on macOS; the CPU path has previously exposed
-  kernel VM failures with very large mappings.
-- Do not run multiple huge model processes concurrently. The instance lock is
-  intentional.
-- Prefer short Metal smoke tests for build verification.
+- Avoid large CPU inference runs on macOS; the CPU path has previously exposed kernel VM failures with very large mappings.
+- Do not run multiple huge model processes concurrently. The instance lock is intentional.
 
 ## Layout
 
@@ -46,8 +40,17 @@ Objective-C only where Metal requires it and Metal kernels under `metal/`.
 - `tests/`: unit and live integration tests.
 - `misc/`: ignored notes, experiments, and old planning material.
 
+This list is not complete, check the files for more info.
+
 ## Testing
 
 Use `make` for build validation. Use `make test` for unit/regression tests when a
 model and Metal are available. Use live server tests only when intentionally
 testing the API surface.
+
+At every major change where one of the following could be affected, make sure to:
+
+1. Test the normal Metal path and that speed is still at the level it was.
+2. Test the SSD streaming path.
+3. Test the distributed inference if it could be affected, but ask the user before doing so.
+4. Check if CUDA could be broken after the change, and ask the user to give you access to the CUDA machine to actually test if everything is still fine.

MODEL_CARD.md

@@ -0,0 +1,243 @@
+# DeepSeek v4 model card synopsis
+
+This document extracts the most important information from the official
+DeepSeek-V4-Flash Hugging Face model card, with emphasis on facts that matter
+for local inference, DS4 development, and benchmark interpretation.
+
+Source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash
+
+## Model Family
+
+DeepSeek-V4 is a preview model family with two Mixture-of-Experts language
+models:
+
+| Model | Total parameters | Active parameters | Context length |
+|---|---:|---:|---:|
+| DeepSeek-V4-Flash | 284B | 13B | 1M tokens |
+| DeepSeek-V4-Pro | 1.6T | 49B | 1M tokens |
+
+Flash is the smaller and more efficient model. The model card says Flash-Max can
+approach Pro reasoning performance when given a larger thinking budget, while
+remaining behind Pro on pure knowledge and the most complex agentic tasks.
+
+## Architecture
+
+DeepSeek-V4 uses long-context compressed attention. The model card calls the
+hybrid design Compressed Sparse Attention (CSA) plus Heavily Compressed
+Attention (HCA). In DS4 terms, each layer keeps a raw sliding-window KV cache
+for the latest 128 tokens. This is the high-resolution local context.
+
+After that raw window, the model uses layer-dependent compressed KV rows:
+
+| 0-based layer indexes | DS4 ratio | Extra state | Meaning |
+|---|---:|---|---|
+| 0, 1 | none | none | Raw 128-token sliding window only |
+| even layers from 2 onward | 4 | compressed KV + indexer KV | One compressed row per 4 tokens, with an indexer selecting visible compressed rows |
+| odd layers from 3 onward | 128 | compressed KV | One compressed row per 128 tokens |
+
+So, after the first two layers, the model alternates ratio-4 and ratio-128
+compressed attention. A token in a compressed layer attends over both the raw
+latest-128-token window and the older compressed history. The compression here
+is time-axis compression: several token positions are pooled into one KV row.
+The attention rows still use the model attention/value dimensions, so raw and
+compressed rows can be consumed by the same mixed-attention computation.
+
+Ratio-4 layers are the selective compressed-attention layers. They maintain a
+second compressed stream for the indexer, and when the compressed history is
+larger than the configured top-k, DS4 scores the compressed rows and selects up
+to 512 of them for attention. Ratio-128 layers are the heavily compressed path:
+they do not have the indexer stream and use the available ratio-128 compressed
+rows directly.
+
+DS4 validates these details from the GGUF metadata. The relevant fixed
+implementation constants are:
+
+- Layers: 43
+- Raw sliding-window attention: 128 tokens
+- Indexer heads: 64
+- Indexer head dimension: 128
+- Indexer top-k: 512
+
+This is the practical reason the model can expose a 1M-token context without a
+standard full KV cache for every token in every layer. The model card reports
+that, at 1M tokens, DeepSeek-V4-Pro needs much less single-token inference
+compute and KV cache than DeepSeek-V3.2.
+
+The family also uses:
+
+- Manifold-Constrained Hyper-Connections (mHC), intended to improve signal
+  propagation stability across layers.
+- The Muon optimizer, used for faster convergence and training stability.
+- A post-training pipeline with domain expert cultivation followed by unified
+  consolidation via on-policy distillation.
+
+## Precision And Weights
+
+Official download entries include:
+
+| Model | Precision |
+|---|---|
+| DeepSeek-V4-Flash-Base | FP8 Mixed |
+| DeepSeek-V4-Flash | FP4 + FP8 Mixed |
+| DeepSeek-V4-Pro-Base | FP8 Mixed |
+| DeepSeek-V4-Pro | FP4 + FP8 Mixed |
+
+For the instruct models, the model card describes FP4 + FP8 Mixed as using FP4
+for MoE expert parameters and FP8 for most other parameters.
+
+## Reasoning Modes
+
+The instruct models support three reasoning-effort modes:
+
+| Mode | Intended behavior | Output shape |
+|---|---|---|
+| Non-think | Fast, intuitive replies | `</think>` summary |
+| High | Deliberate reasoning for harder tasks | `<think>... </think>` summary |
+| Max | Largest reasoning budget | Special system prompt plus thinking and summary |
+
+The model card recommends using at least a 384K-token context window for Think
+Max.
+
+## Important Flash Benchmarks
+
+### DeepSeek-V4-Flash Across Reasoning Modes
+
+| Benchmark | Non-Think | High | Max |
+|---|---:|---:|---:|
+| GPQA Diamond Pass@1 | 71.2 | 87.4 | 88.1 |
+| MMLU-Pro EM | 83.0 | 86.4 | 86.2 |
+| SimpleQA-Verified Pass@1 | 23.1 | 28.9 | 34.1 |
+| Chinese-SimpleQA Pass@1 | 71.5 | 73.2 | 78.9 |
+| HLE Pass@1 | 8.1 | 29.4 | 34.8 |
+| LiveCodeBench Pass@1 | 55.2 | 88.4 | 91.6 |
+| HMMT 2026 Feb Pass@1 | 40.8 | 91.9 | 94.8 |
+| IMOAnswerBench Pass@1 | 41.9 | 85.1 | 88.4 |
+| SWE Verified Resolved | 73.7 | 78.6 | 79.0 |
+| Terminal Bench 2.0 Acc | 49.1 | 56.6 | 56.9 |
+| MCPAtlas Pass@1 | 64.0 | 67.4 | 69.0 |
+| Toolathlon Pass@1 | 40.7 | 43.5 | 47.8 |
+
+### DeepSeek-V4-Flash-Base
+
+The base-model table reports these Flash-Base scores:
+
+| Benchmark | Shots | Score |
+|---|---:|---:|
+| SuperGPQA EM | 5-shot | 46.5 |
+| MMLU EM | 5-shot | 88.7 |
+| MMLU-Pro EM | 5-shot | 68.3 |
+| Simple-QA verified EM | 25-shot | 30.1 |
+| HumanEval Pass@1 | 0-shot | 69.5 |
+| GSM8K EM | 8-shot | 90.8 |
+| LongBench-V2 EM | 1-shot | 44.7 |
+
+The model card reports SuperGPQA for the base model table, not in the instruct
+reasoning-mode comparison table.
+
+## Chat Template And Encoding
+
+The release does not use a Jinja chat template as the source of truth. The
+official prompt renderer is the Python code in
+`encoding/encoding_dsv4.py`, with examples and tests in the same `encoding`
+directory:
+
+- https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/raw/main/encoding/encoding_dsv4.py
+- https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/raw/main/encoding/test_encoding_dsv4.py
+
+The important special tokens are:
+
+| Purpose | Token |
+|---|---|
+| Beginning of sequence | `<｜begin▁of▁sentence｜>` |
+| End of assistant turn | `<｜end▁of▁sentence｜>` |
+| User turn prefix | `<｜User｜>` |
+| Assistant turn prefix | `<｜Assistant｜>` |
+| Latest reminder prefix | `<｜latest_reminder｜>` |
+| Thinking start | `<think>` |
+| Thinking end / non-thinking marker | `</think>` |
+| DSML tool markup marker | `｜DSML｜` |
+
+The renderer accepts `system`, `user`, `assistant`, `tool`,
+`latest_reminder`, and `developer` roles. The `developer` role is described in
+the Python comments as an internal search-agent role, not as a normal public
+chat role.
+
+Normal chat mode starts with the BOS token, then system text if present, then
+alternating user and assistant markers. In non-thinking chat mode, a new
+assistant generation is opened with:
+
+```text
+<｜Assistant｜></think>
+```
+
+That immediate `</think>` tells the model to skip hidden reasoning and produce
+the visible answer. In thinking mode, a new assistant generation is opened with:
+
+```text
+<｜Assistant｜><think>
+```
+
+Completed assistant thinking turns are rendered as reasoning content inside
+`<think>...</think>`, followed by the visible answer and the EOS token.
+
+By default, the Python renderer drops earlier assistant reasoning content before
+the last user message. If tools are present on any message, it disables that
+reasoning drop and keeps the full reasoning/tool context. `reasoning_effort=max`
+also prepends a special high-effort instruction prefix before the first rendered
+message in thinking mode.
+
+Tool definitions are passed in OpenAI-compatible function schema form, but the
+model is instructed to emit DSML. A tool call is rendered as a DSML
+`tool_calls` block containing one or more `invoke` entries, each with named
+parameters. Parameters carry a `string="true"` flag for raw strings and
+`string="false"` for JSON values such as numbers, booleans, arrays, or objects.
+
+DeepSeek-V4 does not render standalone `tool` role messages. The Python
+preprocessor converts tool results into user content blocks and renders each
+result as:
+
+```text
+<tool_result>...</tool_result>
+```
+
+Tool-result bodies are rendered as raw text. Literal `<`, `>`, and `&` from
+file contents or shell output are preserved; only the exact closing sentinel
+`</tool_result>` is escaped so the wrapper cannot be terminated by data.
+
+When there are multiple tool results, the renderer sorts them to match the
+order of the preceding assistant tool calls.
+
+The same script also defines special task tokens for internal quick tasks such
+as title generation, search-query generation, action selection, authority
+classification, domain classification, and URL-read decisions. Those are
+separate from normal chat/tool rendering.
+
+## Local Running Notes
+
+The model card lists vLLM and SGLang examples for OpenAI-compatible serving.
+For local deployment, it recommends:
+
+- `temperature = 1.0`
+- `top_p = 1.0`
+- At least 384K context for Think Max
+
+These are deployment recommendations from the model card, not necessarily the
+same settings used for deterministic benchmarking. DS4 keeps `top_p=1.0` but
+adds a local `min_p=0.05` default to avoid sampling tokens whose probability is
+far below the best token.
+
+## Licensing
+
+The repository and model weights are licensed under the MIT License.
+
+## Citation
+
+The model card cites:
+
+```bibtex
+@misc{deepseekai2026deepseekv4,
+      title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
+      author={DeepSeek-AI},
+      year={2026},
+}
+```

Makefile

@@ -7,16 +7,19 @@ else
 NATIVE_CPU_FLAG ?= -march=native
 endif
 
-CFLAGS ?= -O3 -ffast-math $(NATIVE_CPU_FLAG) -Wall -Wextra -std=c99
-OBJCFLAGS ?= -O3 -ffast-math $(NATIVE_CPU_FLAG) -Wall -Wextra -fobjc-arc
+DEBUG_FLAGS ?= -g
+CFLAGS ?= -O3 -ffast-math $(DEBUG_FLAGS) $(NATIVE_CPU_FLAG) -Wall -Wextra -std=c99
+OBJCFLAGS ?= -O3 -ffast-math $(DEBUG_FLAGS) $(NATIVE_CPU_FLAG) -Wall -Wextra -fobjc-arc
+QUALITY_CFLAGS ?= -O3 $(DEBUG_FLAGS) $(NATIVE_CPU_FLAG) -Wall -Wextra -std=c11
 
 LDLIBS ?= -lm -pthread
 METAL_SRCS := $(wildcard metal/*.metal)
+ROCM_SRCS := $(wildcard rocm/*.cuh)
 
 ifeq ($(UNAME_S),Darwin)
 METAL_LDLIBS := $(LDLIBS) -framework Foundation -framework Metal
-CORE_OBJS = ds4.o ds4_metal.o
-CPU_CORE_OBJS = ds4_cpu.o
+CORE_OBJS = ds4.o ds4_distributed.o ds4_ssd.o ds4_metal.o
+CPU_CORE_OBJS = ds4_cpu.o ds4_distributed.o ds4_ssd.o
 else
 CFLAGS += -D_GNU_SOURCE -fno-finite-math-only
 CUDA_HOME ?= /usr/local/cuda
@@ -25,38 +28,55 @@ CUDA_ARCH ?=
 ifneq ($(strip $(CUDA_ARCH)),)
 NVCC_ARCH_FLAGS := -arch=$(CUDA_ARCH)
 endif
-NVCCFLAGS ?= -O3 --use_fast_math $(NVCC_ARCH_FLAGS) -Xcompiler $(NATIVE_CPU_FLAG) -Xcompiler -pthread
+NVCCFLAGS ?= -O3 -g -lineinfo --use_fast_math $(NVCC_ARCH_FLAGS) -Xcompiler $(NATIVE_CPU_FLAG) -Xcompiler -pthread
+CORE_OBJS = ds4.o ds4_distributed.o ds4_ssd.o ds4_cuda.o
+CPU_CORE_OBJS = ds4_cpu.o ds4_distributed.o ds4_ssd.o
 CUDA_LDLIBS ?= -lm -Xcompiler -pthread -L$(CUDA_HOME)/targets/sbsa-linux/lib -L$(CUDA_HOME)/lib64 -lcudart -lcublas
-CORE_OBJS = ds4.o ds4_cuda.o
-CPU_CORE_OBJS = ds4_cpu.o
+HIPCC ?= $(shell command -v hipcc 2>/dev/null || echo /opt/rocm/bin/hipcc)
+ROCM_ARCH ?= gfx1151
+ROCM_CFLAGS ?= -O3 -ffast-math -g -fno-finite-math-only -pthread -D__HIP_PLATFORM_AMD__ -Wno-unused-command-line-argument --offload-arch=$(ROCM_ARCH)
+ROCM_LDLIBS ?= -lm -pthread -lhipblas -lhipblaslt
+DS4_LINK ?= $(NVCC) $(NVCCFLAGS)
+DS4_LINK_LIBS ?= $(CUDA_LDLIBS)
 METAL_LDLIBS := $(LDLIBS)
 endif
 
-.PHONY: all help clean test cpu cuda cuda-spark cuda-generic cuda-regression
+.PHONY: all help clean test cpu cuda cuda-spark cuda-generic cuda-regression strix-halo rocm
 
 ifeq ($(UNAME_S),Darwin)
-all: ds4 ds4-server ds4-bench
+all: ds4 ds4-server ds4-bench ds4-eval ds4-agent
 
 help:
 	@echo "DS4 build targets:"
-	@echo "  make              Build Metal ./ds4, ./ds4-server, and ./ds4-bench"
-	@echo "  make cpu          Build CPU-only ./ds4, ./ds4-server, and ./ds4-bench"
+	@echo "  make              Build Metal ./ds4, ./ds4-server, ./ds4-bench, ./ds4-eval, and ./ds4-agent"
+	@echo "  make cpu          Build CPU-only ./ds4, ./ds4-server, ./ds4-bench, ./ds4-eval, and ./ds4-agent"
 	@echo "  make test         Build and run tests"
 	@echo "  make clean        Remove build outputs"
 
-ds4: ds4_cli.o linenoise.o $(CORE_OBJS)
-	$(CC) $(CFLAGS) -o $@ ds4_cli.o linenoise.o $(CORE_OBJS) $(METAL_LDLIBS)
+ds4: ds4_cli.o ds4_help.o linenoise.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ ds4_cli.o ds4_help.o linenoise.o $(CORE_OBJS) $(METAL_LDLIBS)
 
-ds4-server: ds4_server.o rax.o $(CORE_OBJS)
-	$(CC) $(CFLAGS) -o $@ ds4_server.o rax.o $(CORE_OBJS) $(METAL_LDLIBS)
+ds4-server: ds4_server.o ds4_help.o ds4_kvstore.o rax.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ ds4_server.o ds4_help.o ds4_kvstore.o rax.o $(CORE_OBJS) $(METAL_LDLIBS)
 
-ds4-bench: ds4_bench.o $(CORE_OBJS)
-	$(CC) $(CFLAGS) -o $@ ds4_bench.o $(CORE_OBJS) $(METAL_LDLIBS)
+ds4-bench: ds4_bench.o ds4_help.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ ds4_bench.o ds4_help.o $(CORE_OBJS) $(METAL_LDLIBS)
 
-cpu: ds4_cli_cpu.o ds4_server_cpu.o ds4_bench_cpu.o linenoise.o rax.o $(CPU_CORE_OBJS)
-	$(CC) $(CFLAGS) -o ds4 ds4_cli_cpu.o linenoise.o $(CPU_CORE_OBJS) $(LDLIBS)
-	$(CC) $(CFLAGS) -o ds4-server ds4_server_cpu.o rax.o $(CPU_CORE_OBJS) $(LDLIBS)
-	$(CC) $(CFLAGS) -o ds4-bench ds4_bench_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
+ds4-eval: ds4_eval.o ds4_help.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ ds4_eval.o ds4_help.o $(CORE_OBJS) $(METAL_LDLIBS)
+
+ds4-agent: ds4_agent.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS)
+	$(CC) $(CFLAGS) -o $@ ds4_agent.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS) $(METAL_LDLIBS)
+
+gguf-tools/quality-testing/score_official: gguf-tools/quality-testing/score_official.c ds4.h $(CORE_OBJS) rax.o
+	$(CC) $(QUALITY_CFLAGS) -I. -o $@ gguf-tools/quality-testing/score_official.c $(CORE_OBJS) rax.o $(METAL_LDLIBS)
+
+cpu: ds4_cli_cpu.o ds4_server_cpu.o ds4_bench_cpu.o ds4_eval_cpu.o ds4_agent_cpu.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o rax.o $(CPU_CORE_OBJS)
+	$(CC) $(CFLAGS) -o ds4 ds4_cli_cpu.o ds4_help.o linenoise.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-server ds4_server_cpu.o ds4_help.o ds4_kvstore.o rax.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-bench ds4_bench_cpu.o ds4_help.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-eval ds4_eval_cpu.o ds4_help.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-agent ds4_agent_cpu.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CPU_CORE_OBJS) $(LDLIBS)
 
 cuda-regression:
 	@echo "cuda-regression requires a CUDA build"
@@ -68,15 +88,17 @@ help:
 	@echo "  make cuda-spark          Build CUDA for DGX Spark / GB10"
 	@echo "  make cuda-generic        Build CUDA for a generic local CUDA GPU"
 	@echo "  make cuda CUDA_ARCH=sm_N Build CUDA with an explicit nvcc -arch value"
-	@echo "  make cpu                 Build CPU-only ./ds4, ./ds4-server, and ./ds4-bench"
+	@echo "  make strix-halo          Build ROCm for Strix Halo / gfx1151"
+	@echo "  make rocm                Alias for make strix-halo"
+	@echo "  make cpu                 Build CPU-only ./ds4, ./ds4-server, ./ds4-bench, ./ds4-eval, and ./ds4-agent"
 	@echo "  make test                Build and run tests"
 	@echo "  make clean               Remove build outputs"
 
 cuda-spark:
-	$(MAKE) ds4 ds4-server ds4-bench CUDA_ARCH=
+	$(MAKE) -B ds4 ds4-server ds4-bench ds4-eval ds4-agent CUDA_ARCH=
 
 cuda-generic:
-	$(MAKE) ds4 ds4-server ds4-bench CUDA_ARCH=native
+	$(MAKE) -B ds4 ds4-server ds4-bench ds4-eval ds4-agent CUDA_ARCH=native
 
 cuda:
 	@if [ -z "$(strip $(CUDA_ARCH))" ]; then \
@@ -84,41 +106,85 @@ cuda:
 		echo "       or use make cuda-spark / make cuda-generic"; \
 		exit 2; \
 	fi
-	$(MAKE) ds4 ds4-server ds4-bench CUDA_ARCH="$(CUDA_ARCH)"
+	$(MAKE) -B ds4 ds4-server ds4-bench ds4-eval ds4-agent CUDA_ARCH="$(CUDA_ARCH)"
 
-ds4: ds4_cli.o linenoise.o $(CORE_OBJS)
-	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+strix-halo:
+	$(MAKE) -B ds4 ds4-server ds4-bench ds4-eval ds4-agent \
+		CORE_OBJS="ds4.o ds4_distributed.o ds4_ssd.o ds4_rocm.o" \
+		CFLAGS="$(CFLAGS) -DDS4_ROCM_BUILD" \
+		DS4_LINK="$(HIPCC) $(ROCM_CFLAGS)" \
+		DS4_LINK_LIBS="$(ROCM_LDLIBS)"
 
-ds4-server: ds4_server.o rax.o $(CORE_OBJS)
-	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+rocm: strix-halo
 
-ds4-bench: ds4_bench.o $(CORE_OBJS)
-	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
+ds4: ds4_cli.o ds4_help.o linenoise.o $(CORE_OBJS)
+	$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
 
-cpu: ds4_cli_cpu.o ds4_server_cpu.o ds4_bench_cpu.o linenoise.o rax.o $(CPU_CORE_OBJS)
-	$(CC) $(CFLAGS) -o ds4 ds4_cli_cpu.o linenoise.o $(CPU_CORE_OBJS) $(LDLIBS)
-	$(CC) $(CFLAGS) -o ds4-server ds4_server_cpu.o rax.o $(CPU_CORE_OBJS) $(LDLIBS)
-	$(CC) $(CFLAGS) -o ds4-bench ds4_bench_cpu.o $(CPU_CORE_OBJS) $(LDLIBS)
+ds4-server: ds4_server.o ds4_help.o ds4_kvstore.o rax.o $(CORE_OBJS)
+	$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
+
+ds4-bench: ds4_bench.o ds4_help.o $(CORE_OBJS)
+	$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
+
+ds4-eval: ds4_eval.o ds4_help.o $(CORE_OBJS)
+	$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
+
+ds4-agent: ds4_agent.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS)
+	$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
+
+gguf-tools/quality-testing/score_official: gguf-tools/quality-testing/score_official.c ds4.h $(CORE_OBJS) rax.o
+	$(DS4_LINK) $(filter-out -ffast-math,$(QUALITY_CFLAGS)) -I. -o $@ gguf-tools/quality-testing/score_official.c $(CORE_OBJS) rax.o $(DS4_LINK_LIBS)
+
+cpu: ds4_cli_cpu.o ds4_server_cpu.o ds4_bench_cpu.o ds4_eval_cpu.o ds4_agent_cpu.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o rax.o $(CPU_CORE_OBJS)
+	$(CC) $(CFLAGS) -o ds4 ds4_cli_cpu.o ds4_help.o linenoise.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-server ds4_server_cpu.o ds4_help.o ds4_kvstore.o rax.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-bench ds4_bench_cpu.o ds4_help.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-eval ds4_eval_cpu.o ds4_help.o $(CPU_CORE_OBJS) $(LDLIBS)
+	$(CC) $(CFLAGS) -o ds4-agent ds4_agent_cpu.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CPU_CORE_OBJS) $(LDLIBS)
 
 cuda-regression: tests/cuda_long_context_smoke
 	./tests/cuda_long_context_smoke
 endif
 
-ds4.o: ds4.c ds4.h ds4_gpu.h
+ds4.o: ds4.c ds4.h ds4_ssd.h ds4_distributed.h ds4_gpu.h
 	$(CC) $(CFLAGS) -c -o $@ ds4.c
 
-ds4_cli.o: ds4_cli.c ds4.h linenoise.h
+ds4_ssd.o: ds4_ssd.c ds4_ssd.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_ssd.c
+
+ds4_cli.o: ds4_cli.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h linenoise.h
 	$(CC) $(CFLAGS) -c -o $@ ds4_cli.c
 
-ds4_server.o: ds4_server.c ds4.h rax.h
+ds4_distributed.o: ds4_distributed.c ds4_distributed.h ds4.h ds4_ssd.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_distributed.c
+
+ds4_help.o: ds4_help.c ds4_help.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_help.c
+
+ds4_server.o: ds4_server.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h rax.h
 	$(CC) $(CFLAGS) -c -o $@ ds4_server.c
 
-ds4_bench.o: ds4_bench.c ds4.h
+ds4_bench.o: ds4_bench.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h
 	$(CC) $(CFLAGS) -c -o $@ ds4_bench.c
 
-ds4_test.o: tests/ds4_test.c ds4_server.c ds4.h rax.h
+ds4_eval.o: ds4_eval.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_eval.c
+
+ds4_agent.o: ds4_agent.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h ds4_web.h linenoise.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_agent.c
+
+ds4_web.o: ds4_web.c ds4_web.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_web.c
+
+ds4_kvstore.o: ds4_kvstore.c ds4_kvstore.h ds4.h ds4_ssd.h
+	$(CC) $(CFLAGS) -c -o $@ ds4_kvstore.c
+
+ds4_test.o: tests/ds4_test.c ds4_server.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h rax.h
 	$(CC) $(CFLAGS) -Wno-unused-function -c -o $@ tests/ds4_test.c
 
+ds4_agent_test.o: tests/ds4_agent_test.c ds4_agent.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h ds4_web.h linenoise.h
+	$(CC) $(CFLAGS) -Wno-unused-function -c -o $@ tests/ds4_agent_test.c
+
 tests/cuda_long_context_smoke.o: tests/cuda_long_context_smoke.c ds4_gpu.h
 	$(CC) $(CFLAGS) -I. -c -o $@ tests/cuda_long_context_smoke.c
 
@@ -128,36 +194,58 @@ rax.o: rax.c rax.h rax_malloc.h
 linenoise.o: linenoise.c linenoise.h
 	$(CC) $(CFLAGS) -c -o $@ linenoise.c
 
-ds4_cpu.o: ds4.c ds4.h ds4_gpu.h
+ds4_cpu.o: ds4.c ds4.h ds4_ssd.h ds4_distributed.h ds4_gpu.h
 	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4.c
 
-ds4_cli_cpu.o: ds4_cli.c ds4.h linenoise.h
+ds4_cli_cpu.o: ds4_cli.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h linenoise.h
 	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_cli.c
 
-ds4_server_cpu.o: ds4_server.c ds4.h rax.h
+ds4_server_cpu.o: ds4_server.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h rax.h
 	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_server.c
 
-ds4_bench_cpu.o: ds4_bench.c ds4.h
+ds4_bench_cpu.o: ds4_bench.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h
 	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_bench.c
 
+ds4_eval_cpu.o: ds4_eval.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h
+	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_eval.c
+
+ds4_agent_cpu.o: ds4_agent.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h ds4_web.h linenoise.h
+	$(CC) $(CFLAGS) -DDS4_NO_GPU -c -o $@ ds4_agent.c
+
 ds4_metal.o: ds4_metal.m ds4_gpu.h $(METAL_SRCS)
 	$(CC) $(OBJCFLAGS) -c -o $@ ds4_metal.m
 
 ds4_cuda.o: ds4_cuda.cu ds4_gpu.h ds4_iq2_tables_cuda.inc
 	$(NVCC) $(NVCCFLAGS) -c -o $@ ds4_cuda.cu
 
+ds4_rocm.o: ds4_rocm.cu ds4_gpu.h ds4_iq2_tables_cuda.inc $(ROCM_SRCS)
+	$(HIPCC) $(ROCM_CFLAGS) -c -o $@ ds4_rocm.cu
+
 tests/cuda_long_context_smoke: tests/cuda_long_context_smoke.o ds4_cuda.o
 	$(NVCC) $(NVCCFLAGS) -o $@ $^ $(CUDA_LDLIBS)
 
-ds4_test: ds4_test.o rax.o $(CORE_OBJS)
+ds4_test: ds4_test.o ds4_help.o ds4_kvstore.o rax.o $(CORE_OBJS)
 ifeq ($(UNAME_S),Darwin)
-	$(CC) $(CFLAGS) -o $@ ds4_test.o rax.o $(CORE_OBJS) $(METAL_LDLIBS)
+	$(CC) $(CFLAGS) -o $@ ds4_test.o ds4_help.o ds4_kvstore.o rax.o $(CORE_OBJS) $(METAL_LDLIBS)
 else
-	$(NVCC) $(NVCCFLAGS) -o $@ ds4_test.o rax.o $(CORE_OBJS) $(CUDA_LDLIBS)
+	$(NVCC) $(NVCCFLAGS) -o $@ ds4_test.o ds4_help.o ds4_kvstore.o rax.o $(CORE_OBJS) $(CUDA_LDLIBS)
 endif
 
-test: ds4_test
+ds4_agent_test: ds4_agent_test.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS)
+ifeq ($(UNAME_S),Darwin)
+	$(CC) $(CFLAGS) -o $@ ds4_agent_test.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS) $(METAL_LDLIBS)
+else
+	$(NVCC) $(NVCCFLAGS) -o $@ ds4_agent_test.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS) $(CUDA_LDLIBS)
+endif
+
+test: ds4_test ds4_agent_test ds4-eval q4k-dot-test
+	./ds4-eval --self-test-extractors
+	./ds4_agent_test
 	./ds4_test
 
+q4k-dot-test: tests/test_q4k_dot.c
+	$(CC) -O2 -Wall -Wextra -std=c99 -o tests/test_q4k_dot tests/test_q4k_dot.c -lm -pthread
+	./tests/test_q4k_dot
+
 clean:
-	rm -f ds4 ds4-server ds4-bench ds4_cpu ds4_native ds4_server_test ds4_test *.o tests/cuda_long_context_smoke tests/cuda_long_context_smoke.o
+	rm -f ds4 ds4-server ds4-bench ds4-eval ds4-agent ds4_cpu ds4_native ds4_server_test ds4_test ds4_agent_test gguf-tools/quality-testing/score_official tests/test_q4k_dot *.o tests/cuda_long_context_smoke tests/cuda_long_context_smoke.o

QA_BEFORE_RELEASES.md

@@ -0,0 +1,290 @@
+# QA Before Releases
+
+This is the release gate for DwarfStar.  Run it before tagging or pushing a
+release build.  The goal is not to prove every code path exhaustively; it is to
+exercise the paths that have historically regressed: Metal graph inference,
+CUDA, ROCm, SSD streaming, distributed execution, disk KV cache, server APIs, and the
+agent TUI/tool state machine.
+
+Do not run multiple huge model processes at the same time.  Record the commit,
+hardware, GGUF file, context size, and any non-default flags for every manual
+run.
+
+Preferred release test hosts:
+
+- CUDA / DGX Spark: `toor@192.168.0.180`.
+- Metal / distributed Mac testing: `mac-m5max-it` and `mac-m5max-us`.
+- ROCm: The Strix Halo system at antirez@strixhalo (Framework Desktop).
+
+The Mac hosts have DNS entries and are reached through an internet VPN.  They
+are connected to each other over WiFi and also through a Thunderbolt 5
+point-to-point link.  The TB5 route is the preferred distributed-inference
+network when it is available, but it can be fragile and sometimes only works
+when `ds4` is executed in the foreground.  Prefer these machines for release
+testing, especially distributed inference.  Local fallback testing on this
+machine is acceptable when needed; it is an M3 Max with 128 GB RAM.
+The Strix Halo system is reachable via the VPN as well and has a local WiFi
+address in the same lan of the M5 Max systems. The CUDA hosts are in a
+different remote lan and are accessible via a different VPN active
+in this system.
+
+## 1. Repository And Build Sanity
+
+- Start from a clean tree except intentional release notes:
+  `git status --short`.
+- Build the normal local target:
+  `make clean && make`.
+- Build CPU-only binaries as a compile check only:
+  `make clean && make cpu`.
+- Run whitespace checks before committing:
+  `git diff --check`.
+- Confirm `./ds4 --help`, `./ds4-server --help`, and `./ds4-agent --help` render
+  cleanly, with readable section colors and no broken wrapping.
+
+## 2. Core Regression Tests
+
+- Run the default suite:
+  `make test`.
+- Run the vector checks explicitly after any tokenizer, template, KV, kernel,
+  quantization, or prompt-rendering change:
+  `./ds4_test --logprob-vectors`
+  and `./ds4_test --local-golden-vectors`.
+- Run server tests when HTTP, SSE, prompt rendering, cache policy, or tool-call
+  replay changed:
+  `./ds4_test --server`.
+- Run `./ds4-eval --self-test-extractors`.
+
+## 3. Official Continuation Quality Gates
+
+These tests are release-blocking after tokenizer, template, KV-cache, attention,
+MoE routing, quantization, logit, or model-graph changes.  They are
+teacher-forced continuation checks against hosted-model output and API
+top-logprob slices, so do not replace them with one sampled chat answer.
+
+- Build the scorer:
+  `make -C gguf-tools quality-score`.
+- Run the tracked DeepSeek V4 Flash smoke vectors:
+  `./ds4_test --logprob-vectors`.
+  This covers the official Flash API vectors in `tests/test-vectors/`, including
+  short prompts and long-prompt attention cases.
+- Run the 100-case DeepSeek V4 Flash fixture for every released Flash GGUF:
+  `gguf-tools/quality-testing/score_official /path/to/deepseek-v4-flash.gguf gguf-tools/quality-testing/data/flash/manifest.tsv /tmp/flash.tsv 4096`.
+- Run the 100-case GLM 5.2 OpenRouter fixture for every released GLM GGUF:
+  `gguf-tools/quality-testing/score_official models/GLM-5.2-UD-Q4_K_XL.gguf gguf-tools/quality-testing/data/glm52-openrouter-100/manifest.tsv /tmp/glm52-q4.tsv 4096`.
+  Current Q4 XL reference band: first-token match `95/100`, API top-1 agreement
+  about `0.942`, and API pair-order agreement about `0.880`.
+- Run the same GLM fixture for reduced-precision GLM release files.  The Q2
+  routed reference is lower quality but should stay near first-token match
+  `92/100`, API top-1 agreement about `0.890`, and API pair-order agreement
+  about `0.800` unless the quantization changed deliberately.
+- Run the 100-case DeepSeek V4 PRO fixture for every released PRO GGUF:
+  `gguf-tools/quality-testing/score_official /path/to/deepseek-v4-pro.gguf gguf-tools/quality-testing/data/pro/manifest.tsv /tmp/pro.tsv 4096`.
+- For SSD streaming, run the same official-continuation scorer once with full
+  residency and once with `--ssd-streaming` for the release model.  The summary
+  and API agreement should stay in the same quality band.
+- Compare any candidate against the previous release or last-known-good output:
+  `python3 gguf-tools/quality-testing/compare_scores.py /tmp/old.tsv /tmp/new.tsv`.
+  Treat a large first-token-match drop, a clear NLL regression, or a material
+  API top-1/pair-order regression as a blocker unless the release notes call out
+  an intentional quality tradeoff.
+- Keep the raw `summary` and `api_summary` lines in the release notes or QA log.
+  Do not use stale manifests from `misc/` as release evidence.
+
+## 4. Metal Flash Path
+
+Use the normal Flash GGUF that 128 GB users run.
+
+- One-shot CLI:
+  `./ds4 -m ds4flash.gguf --ctx 32768 --nothink -p "Explain C pointers in one paragraph."`
+- Thinking and max-thinking prompts:
+  run one short coding prompt with default thinking and one with max thinking.
+- Long-context recall:
+  run the long name/number or archive recall test used for catching attention
+  and MoE routing drift.
+- Logprob sanity:
+  `./ds4 --nothink --temp 0 --dump-logprobs /tmp/ds4-logprobs.json --logprobs-top-k 20 -p "..."`
+  and inspect that the continuation is sane.
+- Speed sanity:
+  run `ds4-bench` with `speed-bench/promessi_sposi.txt` and compare prefill,
+  generation speed, and KV bytes with the last known good numbers for the same
+  machine.
+
+## 5. Metal PRO Path
+
+PRO support is experimental, but release builds must not break it silently.
+
+- If a PRO-capable machine is available, run a short PRO q2 prompt and verify
+  the correct template, thinking behavior, and endpoint aliases.
+- For PRO Q4 distributed builds, test only on the intended high-memory machines.
+- If PRO cannot be run locally, at least build all binaries and review changes
+  touching model shape, tensor lookup, routed expert mapping, template logic,
+  and KV payload compatibility.
+
+## 6. SSD Streaming
+
+SSD streaming is a capacity path, so test both correctness and user experience.
+
+- Flash q2/q2-q4 streaming:
+  `./ds4 -m ds4flash.gguf --ssd-streaming --ssd-streaming-cache-experts 32GB -p "..."`
+- Regression test mixed-quant Flash SSD streaming. Use the mixed q2/q4 GGUF
+  with boosted Q4 routed-expert layers and a prompt long enough to exercise the
+  selected-address prefill path; it must not fail with "model range is not
+  covered by mapped model views":
+  `./ds4 -m gguf/DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf --ssd-streaming --ssd-streaming-cache-experts 16GB --ctx 4096 --tokens 1 --nothink --prompt-file /tmp/ds4_600tok_prompt.txt`.
+- Cold streaming measurement:
+  run once with `--ssd-streaming-cold` and verify no deadlock, missing expert,
+  or impossible slowdown.
+- Confirm startup reports cache budget and that generation does not stall on
+  repeated expert misses for a small interactive prompt.
+- If streaming cache internals changed, test the same prompt twice and compare
+  first-token/logprob sanity between runs.
+
+## 7. CUDA / DGX Spark
+
+Before a release, ask the user for CUDA access if it is not already configured.
+Use the DGX Spark / GB10 host `toor@192.168.0.180`.  Do not claim CUDA is
+release-ready without this pass.
+
+- Fetch or push the exact release commit to the CUDA machine.
+- Build:
+  `make clean && make cuda-spark`.
+- Run:
+  `make cuda-regression`.
+- Run a short CLI prompt with the Flash GGUF and record generation t/s.
+- Run a longer prompt that exercises routed experts past a few thousand tokens.
+- If CUDA Q4, distributed, streaming hooks, tensor span loading, or model cache
+  code changed, test the specific GGUF and split mode that uses that path.
+- Verify that any CUDA-only warning fixes are also clean on macOS and do not
+  change Metal behavior.
+
+## 8. ROCm / Strix Halo
+
+Use the Strix Halo Framework Desktop via the VPN hostname `strixhalo`
+(`antirez@strixhalo`).  This host validates the ROCm backend; do not use it as
+a substitute for CUDA or Metal release testing.
+
+- Fetch or push the exact release commit to the Strix Halo machine.
+- Build:
+  `make clean && make strix-halo`.
+- Use the q2 Flash imatrix GGUF for release smoke tests:
+  `DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf`.
+- Do not use the mixed q2-q4 or Q4 Flash GGUFs for routine Strix Halo QA yet.
+  They are dangerous on this machine for now because the ROCm path can hit
+  system OOM instead of failing cleanly.
+- Run a short CLI prompt:
+  `./ds4 -m gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf --ctx 4096 --nothink -p "Reply with exactly: OK"`.
+- Run one longer prompt if ROCm kernels, backend hooks, tensor loading, model
+  cache, KV cache, or graph prefill code changed.
+- Record startup memory/cache messages, prefill speed, generation speed, and
+  whether the backend reports `ROCm backend initialized`.
+
+## 9. Distributed Inference
+
+Distributed code has regressed around route setup, KV snapshots, request IDs,
+and split model loading.  Test it whenever distributed, KV, session, or model
+loading code changes.
+
+- Prefer `mac-m5max-it` and `mac-m5max-us` for Metal distributed tests.  Use the
+  TB5 point-to-point link when it is working; otherwise note that the run used
+  WiFi/VPN routing.
+- Start workers first, then the coordinator.
+- Test a small prompt and a longer prompt.
+- Verify the coordinator waits for a complete route and exits cleanly.
+- Verify `Ctrl+C` returns control after the current distributed token or chunk
+  drains.
+- Save and restore a distributed KV snapshot if that code changed.
+- If CUDA distributed is relevant, test across the CUDA hosts and record
+  generation speed, not just "it works".
+
+## 10. Disk KV Cache
+
+Disk KV cache bugs are high impact for server users.
+
+- Start the server with:
+  `./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192`.
+- Run the same request twice and verify the second request hits cache.
+- Fill the cache enough to trigger eviction; verify the newly-written entry is
+  not evicted and useful anchors are retained.
+- Test rejection of incompatible checkpoints when model, quantization, context,
+  or raw/compressed KV layout changes.
+- Test stripped agent sessions: `/strip <id>` then `/switch <id>` should rebuild
+  by prefill and render sane history.
+
+## 11. Server APIs
+
+The server must keep compatibility across OpenAI, Responses, and Anthropic
+clients.
+
+- `GET /v1/models/deepseek-v4-flash` and `GET /v1/models/deepseek-v4-pro`
+  should both serve whichever GGUF is loaded.
+- Test OpenAI chat completion, OpenAI Responses, and Anthropic messages.
+- Test SSE streaming with thinking enabled and disabled.
+- Test keepalive during long prefill and confirm clients do not time out.
+- Test `--trace` and confirm rendered prompts, cache decisions, generated text,
+  and tool-parser events are useful without leaking unrelated state.
+
+## 12. ds4-agent
+
+The agent is the most stateful component.  Test it manually, not only by build.
+
+- Startup banner, status bar, help, `/power`, `/save`, `/list`, `/switch`,
+  `/history`, `/compact`, `/new`, `/del`, and `/strip`.
+- Ctrl+C during generation, during prefill, during a web fetch, and during a
+  long tool call.  After `Stopped by user`, typing a new prompt must work.
+- Queue messages while the model is busy.  Queued messages must not skip tool
+  execution; after tool results, the queued user text must be provided.
+- Read/search/edit/write tools:
+  create a temp project, ask for edits, verify old/new and `[upto]` anchored
+  edits fail safely on ambiguous matches and do not require retyping whole files.
+- Real coding edit loop:
+  delete `/tmp/mymandel`, ask ds4-agent to create a small C ASCII Mandelbrot
+  program there, build and run it, then in a second user turn ask for a small
+  modification that should naturally use the edit tool, such as changing the
+  ASCII character ramp or output dimensions.  Verify the agent edits the
+  existing file instead of rewriting the whole project, and that the final
+  program still builds and runs.
+- Bash tools:
+  test short output, large output truncation, non-zero exit output, long-running
+  jobs, `bash_status`, and `bash_stop`.
+- Web tools:
+  `google_search` and `visit_page` should ask for visible Chrome approval with
+  timeout, open pages without stealing focus when possible, extract Markdown,
+  close tabs, and handle consent/privacy walls as tool errors the model can see.
+- TUI:
+  test multiline prompt editing, history navigation, queued prompt display,
+  status bar fill to terminal width, syntax highlighting in Markdown/code blocks,
+  and SSH/remote terminal flicker.
+
+## 13. Download Script And Model Files
+
+- Test `download_model.sh` in a temporary directory so local weights are not
+  overwritten.
+- Test one Flash target and one PRO target enough to verify URL, resume, Hugging
+  Face CLI/curl behavior, file naming, and symlink policy.
+- Verify legacy removed targets fail clearly.
+- Verify README model names match the script and Hugging Face repository.
+
+## 14. Performance And Power
+
+- Run `ds4-bench` on the release machine and compare with tracked CSV baselines.
+- Test `--power 100` is not throttled.
+- Test `--power 50` visibly reduces duty cycle in CLI, server, agent, eval, and
+  bench where practical.
+- Confirm context buffer size, raw KV rows, compressed KV rows, and mmap behavior
+  match expectations for 32k, 100k, and any release-advertised context size.
+
+## 15. Release Sign-off
+
+Do not sign off until:
+
+- macOS Metal Flash passed.
+- Official continuation quality gates passed for every released model family.
+- CUDA was tested on the CUDA machine or the release notes explicitly say CUDA
+  was not validated.
+- ROCm was tested on Strix Halo or the release notes explicitly say ROCm was
+  not validated.
+- Disk KV cache was exercised.
+- Server API streaming was exercised.
+- Agent interruption and tool loops were exercised manually.
+- Speed is within expected variance for the same hardware and model.
+- Any skipped item is written down with the reason.

README.md

@@ -1,44 +1,46 @@
-# DwarfStar 4
+# DwarfStar
 
-DrawfStar 4 is a small native inference engine specific for **DeepSeek V4 Flash**. It is
+**DwarfStar** is a small native inference engine optimized first for
+**DeepSeek V4 Flash**, with support for **DeepSeek V4 PRO** on very high-memory
+machines. It is
 intentionally narrow: not a generic GGUF runner, not a wrapper around another
 runtime: it is completely self-contained. Other than running the model in a
-correct and fast way, the project goal is to provide DS4 specific loading,
-prompt rendering, tool calling, KV state handling (RAM and on-disk), and server
-API, all ready to work with coding agents or with the provided CLI interface.
-There are also tools for GGUF and imatrix generation, and for quality and
-speed testing.
+correct and fast way, the project goal is to provide DeepSeek specific loading,
+prompt rendering, tool calling, KV state handling (RAM and on-disk), server
+API and integrated coding agent, all ready to work with coding agents or with
+the provided CLI interface. There are also tools for GGUF and imatrix generation,
+and for quality and speed testing.
 
 We support the following backends:
-* **Metal** is our primary target. Starting from MacBooks with 96GB of RAM.
-* **NVIDIA CUDA** with special care for the DGX Spark.
-* **AMD ROCm** is only supported in the [rocm](https://github.com/antirez/ds4/tree/rocm) branch. It is kept separate from main since I (antirez) don't have direct hardware access, so the community rebases the branch as needed.
+* **Metal** is our primary target. Starting from MacBooks with 96GB of RAM (or less, using SSD streaming).
+* **NVIDIA CUDA / DGX Spark**, CUDA with special care for the DGX Spark.
+* **Strix Halo (ROCm)**, systems like the Framework Desktop and other systems based on the same GPU and unified RAM design.
 
 This project would not exist without **llama.cpp and GGML**, make sure to read
 the acknowledgements section, a big thank you to Georgi Gerganov and all the
 other contributors.
 
+**Note that DeepSeek v4** is not our only target. Right now Flash and PRO are the
+perfect fit because of capabilities, size, KV cache efficiency. If tomorrow a
+better open weight model is released for the 128GB size, we could switch, the same
+for other important size classes like 512GB of RAM. The project is stictly
+opportunistic depending on what open weight models exist in a given moment.
+If a new model will be supported, the old one may be removed completely and
+no longer supported, unless there is some kind of overlap of abilities.
+
 ## Motivations
 
-Now, back at this project. Why we believe DeepSeek v4 Flash to be a pretty special
-model deserving a stand alone engine? Because after comparing it with powerful smaller
-dense models, we can report that:
-
-1. DeepSeek v4 Flash is faster because of less active parameters.
-2. In thinking mode, if you avoid *max thinking*, it produces a thinking section that is a lot shorter than other models, even 1/5 of other models in many cases, and crucially, the thinking section length is **proportional to the problem complexity**. This makes DeepSeek v4 Flash usable with thinking enabled when other models are practically impossible to use in the same conditions.
-3. The model features a context window of **1 million tokens**.
-4. Being so large, it knows more things if you go sampling at the edge of knowledge. For instance asking about Italian show or political questions soon uncovers that 284B parameters are a lot more than 27B or 35B parameters.
-5. It writes much better English and Italian. It *feels* a quasi-frontier model.
-6. The KV cache is incredibly compressed, allowing long context inference on local computers and **on disk KV cache persistence**.
-7. It works well with 2-bit quantization, if quantized in a special way (read later). This allows to run it in MacBooks with 128GB of RAM (and many people reported it working with 96GB as well, even at 250k context window!).
-8. We expect DeepSeek to release **updated versions of v4 Flash** in the future, even better than the current one.
+* Very capable open weight models finally exist. DeepSeek v4 Flash feels quasi-frontier. The PRO is even better. Both resist 2 bit quantization very well.
+* Very capable computers like MacBooks, the DGX Spark now exist.
+* DeepSeek v4 kv cache design makes it pratical to run very big contexts. Other vendors are using this approach.
+* This few hundred billions models are strictly better than smaller (even if dense) models, regardless of what benchmarks say.
 
 That said, a few important things about this project:
 
 * The local inference landscape contains many excellent projects, but new models are released continuously, and the attention immediately gets captured by the next model to implement. This project takes a deliberately narrow bet: one model at a time, official-vector validation (logits obtained with the official implementation), long-context tests, and enough agent integration to know if it really works. The exact model may change as the landscape evolves, but the constraint remains: local inference credible on high end personal machines or Mac Studios, starting from 96/128GB of memory.
 * This software is developed with **strong assistance from GPT 5.5** and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without `llama.cpp` and GGML, largely written by hand.
-* This implementation is based on the idea that compressed KV caches like the one of DeepSeek v4 and the fast SSD disks of modern MacBooks should change our idea that KV cache belongs to RAM. **The KV cache is actually a first-class disk citizen**.
-* Our vision is that local inference should be a set of three things working well together, out of the box: A) inference engine with HTTP API + B) GGUF specially crafted to run well under a given engine and given assumptions + C) testing and validation with coding agents implementations. This inference engine only runs with the GGUF files provided. It gets tested against officially obtained logits at different context sizes. This project exists because we wanted to make one local model feel finished end to end, not just runnable. However this is just alpha quality code, so probably we are not still there.
+* This implementation is based on the idea that compressed KV caches like the one of DeepSeek v4 and the fast SSD disks of modern MacBooks should change our idea that KV cache belongs to RAM. **The KV cache is actually a first-class disk citizen**. Fast SSD disks also changed the inference game from the point of view of "model needs to fit RAM": while having more RAM the the model size is still preferred, SSD streaming allows to turn the available amount of RAM from a hard cutoff (can I run this model or not?) to continuous spectrum of speed levels.
+* Our vision is that local inference should be a set of three things working well together, out of the box: A) inference engine with HTTP API + B) GGUF specially crafted to run well under a given engine and given assumptions + C) testing and validation with coding agents implementations. D) Purpose built agents for specific models and execution environments. DwarfStar only runs with the GGUF files provided. It gets tested against officially obtained logits at different context sizes. This project exists because we wanted to make one local model feel finished end to end, not just runnable. However this is beta quality code, so probably we are not still there, especially since recently we introduced large new features: distributed inference, SSD streaming, and other minor improvements.
 * The optimized graph path targets **Metal on macOS** and **CUDA on Linux**. The CPU path is only for correctness checks and model/tokenizer diagnostics. For CPU-only Linux builds, use `make cpu`; it builds the normal `./ds4` and `./ds4-server` binaries without CUDA or Metal. On macOS, **warning: current macOS versions have a bug in the virtual memory implementation that will crash the kernel** if you try to run the CPU code. Remember? Software sucks. It was not possible to fix the CPU inference to avoid crashing, since each time you have to restart the computer, which is not funny. Help us, if you have the guts.
 
 ## Acknowledgements to llama.cpp and GGML
@@ -48,7 +50,7 @@ llama.cpp project and the kernels, quantization formats, GGUF ecosystem, and har
 engineering knowledge developed there**.
 We are thankful and indebted to [`llama.cpp`](https://github.com/ggml-org/llama.cpp)
 and its contributors. Their implementation, kernels, tests, and design choices were
-an essential reference while building this DeepSeek V4 Flash-specific inference path.
+an essential reference while building this DeepSeek V4 specific inference path.
 Some source-level pieces are retained or adapted here under the MIT license: GGUF
 quant layouts and tables, CPU quant/dot logic, and certain kernels. For this
 reason, and because we are genuinely grateful, we keep the GGML authors copyright
@@ -56,13 +58,15 @@ notice in our `LICENSE` file.
 
 ## Status
 
-The code and GGUF files are to be considered of **alpha quality** because
+The code and GGUF files are to be considered of **beta quality** because
 inference and model serving is a complicated matter and all this exists
 only for a few days. It will take months to reach a more stable form.
 However, we try to keep the project in a usable state, and we are making
-progresses. If you have issues, make sure to use `--trace` to log the
+progress. If you have issues, make sure to use `--trace` to log the
 sessions, and open issues including the full trace.
 
+The `ds4-agent` is alpha quality, the project was later added.
+
 ## More Documentation
 
 If you are looking for very specific things, we have other
@@ -78,17 +82,17 @@ next sections.
 - [gguf-tools/imatrix/dataset/README.md](gguf-tools/imatrix/dataset/README.md):
   how the calibration prompt corpus is generated.
 - [gguf-tools/quality-testing/README.md](gguf-tools/quality-testing/README.md):
-  how local GGUFs are scored against official DeepSeek V4 Flash continuations.
+  how local GGUFs are scored against official DeepSeek V4 Flash/PRO continuations.
 - [dir-steering/README.md](dir-steering/README.md): directional steering data,
   vector generation, and usage.
-- [speed-bench/README.md](speed-bench/README.md): benchmark CSV files and graph
-  generation.
+- [speed-bench/README.md](speed-bench/README.md): benchmark commands, charts,
+  and CSV generation.
 - [tests/test-vectors/README.md](tests/test-vectors/README.md): official
   continuation vectors used for regression checks.
 
 ## Model Weights
 
-This implementation only works with the DeepSeek V4 Flash GGUFs published for
+This implementation only works with the DeepSeek V4 Flash and PRO GGUFs published for
 this project. It is not a general GGUF loader, and arbitrary DeepSeek/GGUF files
 will not have the tensor layout, quantization mix, metadata, or optional MTP
 state expected by the engine. The 2 bit quantizations provided here are not
@@ -102,35 +106,52 @@ Download one main model. **Prefer the imatrix versions.**
 
 ```sh
 ./download_model.sh q2-imatrix   # 96/128 GB RAM machines, imatrix-tuned q2
+./download_model.sh q2-q4-imatrix  # 96/128 GB RAM machines, q2 with last 6 layers q4
 ./download_model.sh q4-imatrix   # >= 256 GB RAM machines, imatrix-tuned q4
+./download_model.sh pro-q2-imatrix  # 512 GB RAM machines, PRO q2 imatrix quant
 ```
 
-Legacy GGUF files are still available if you specifically need the older
-non-imatrix quants:
+For the full PRO Q4 distributed run, download one half on each machine:
 
 ```sh
-./download_model.sh q2           # 96/128 GB RAM machines, legacy non-imatrix
-./download_model.sh q4           # >= 256 GB RAM machines, legacy non-imatrix
+./download_model.sh pro-q4-layers00-30      # first half of PRO Q4 split
+./download_model.sh pro-q4-layers31-output  # second half of PRO Q4 split
 ```
 
 The script downloads from `https://huggingface.co/antirez/deepseek-v4-gguf`,
 stores files under `./gguf/`, resumes partial downloads with `curl -C -`, and
-updates `./ds4flash.gguf` to point at the selected q2-imatrix/q4-imatrix/q2/q4
-model. The plain q2 XXS weights are produced with the weights importance vector
-only, without an imatrix. The imatrix variants are preferred.
+updates `./ds4flash.gguf` to point at the selected main model.
+The `pro-q4-layers00-30`, `pro-q4-layers31-output`, and `pro-q4-split` targets
+download distributed PRO Q4 pieces and do not update `./ds4flash.gguf`.
 Authentication is optional for public downloads, but `--token TOKEN`,
 `HF_TOKEN`, or the local Hugging Face token cache are used when present.
 
 If you want to regenerate GGUF files or collect a new imatrix, see
 [gguf-tools/README.md](gguf-tools/README.md). Those tools are meant for offline
 model-building work and can take a long time on the full DeepSeek V4 Flash
-weights.
+weights. Flash GGUF generation is supported by the local tools. PRO GGUF
+production currently still depends on the external `llama.cpp`-based workflow;
+native tooling can be added later.
 
 `./download_model.sh mtp` fetches the optional speculative decoding support
-GGUF. It can be used with q2-imatrix, q4-imatrix, q2, and q4, but must be
-enabled explicitly with `--mtp`. The current MTP/speculative decoding path is
-still experimental: it is correctness-gated and currently provides at most a
-slight speedup, not a meaningful generation-speed win.
+GGUF for Flash. It can be used with q2-imatrix, q2-q4-imatrix, and q4-imatrix,
+but must be enabled explicitly with `--mtp`. The current MTP/speculative
+decoding path is still experimental: it is correctness-gated and currently
+provides at most a slight speedup, not a meaningful generation-speed win.
+
+GLM 5.2 support is currently limited to the GGUF files tested by this branch:
+
+```sh
+./download_model.sh glm-unsloth-q4  # Unsloth UD-Q4_K_XL, 11 shards
+./download_model.sh glm-antirez-q2  # antirez routed Q2_K single-file GGUF
+./download_model.sh glm-antirez-q4  # antirez routed Q4_K single-file GGUF
+```
+
+The supported GLM layout keeps dense/model-control tensors in the existing
+Q8/F32 paths and supports routed expert gate/up tensors in `Q2_K`, `Q4_K`, or
+`Q5_K`; routed expert down tensors are supported in `Q2_K`, `Q4_K`, `Q5_K`, or
+`Q6_K`. Other GLM GGUF quant layouts should be treated as unsupported until they
+are added deliberately and scored against the official 100-case fixture.
 
 Then build:
 
@@ -158,13 +179,379 @@ Q4 requires the larger-memory machine class, so M3 Max Q4 numbers are `N/A`.
 | MacBook Pro M3 Max, 128 GB | q2 | 11709 tokens | 250.11 t/s | 21.47 t/s |
 | MacBook Pro M3 Max, 128 GB | q4 | short | N/A | N/A |
 | MacBook Pro M3 Max, 128 GB | q4 | long | N/A | N/A |
+| MacBook Pro M5 Max, 128 GB | q2 | short | 87.25 t/s | 34.27 t/s |
+| MacBook Pro M5 Max, 128 GB | q2 | 11707 tokens | 463.44 t/s | 25.90 t/s |
 | Mac Studio M3 Ultra, 512 GB | q2 | short | 84.43 t/s | 36.86 t/s |
 | Mac Studio M3 Ultra, 512 GB | q2 | 11709 tokens | 468.03 t/s | 27.39 t/s |
 | Mac Studio M3 Ultra, 512 GB | q4 | short | 78.95 t/s | 35.50 t/s |
 | Mac Studio M3 Ultra, 512 GB | q4 | 12018 tokens | 448.82 t/s | 26.62 t/s |
+| Mac Studio M3 Ultra, 512 GB | PRO q2 | 32768 tokens | 138.82 t/s | 9.56 t/s |
 | DGX Spark GB10, 128 GB | q2 | 7047 tokens | 343.81 t/s | 13.75 t/s |
 
 ![M3 Max t/s](speed-bench/m3_max_ts.svg)
+![PRO model M3 Ultra t/s](speed-bench/pro_model_m3_ultra_ts.svg)
+
+## Running models larger than RAM
+
+The normal Metal path tries to make the model resident in GPU-addressable
+memory. This is the fastest path and should remain your default when the model
+fits. When it does not fit, DwarfStar also has a Metal-only **SSD streaming**
+capacity mode. In this mode the non-routed model weights stay resident, while
+routed MoE experts are kept in an in-memory cache and loaded from the GGUF file
+on cache misses.
+
+Streaming is not as fast as fitting the full model in RAM. It still needs memory
+for non-routed weights, KV cache, graph scratch, activations, and the routed
+expert cache. It is useful because routed experts dominate model size and modern
+Mac SSDs are fast enough to make cache misses tolerable. Long prefills can still
+be fast; generation is more sensitive to cache misses because every new token
+routes through experts again.
+
+Start with the automatic cache budget:
+
+```sh
+./ds4 -m ./ds4flash.gguf --ssd-streaming
+```
+
+If startup reports that the expert cache is too large, or if you want to reserve
+more memory for context, set the routed expert cache explicitly:
+
+```sh
+./ds4 -m ./ds4flash.gguf --ssd-streaming --ssd-streaming-cache-experts 32GB
+```
+
+The `32GB` value is a routed-expert memory budget, not a generic byte cache.
+DwarfStar first reserves headroom for the two full routed layers used by
+overlapped streaming prefill, then converts the remaining bytes to the number of
+dynamic cached experts that fit for the current GGUF. Explicit `NGB` budgets may
+also be capped after context/KV accounting so the Metal working set stays out of
+the slow pressure zone. A plain number such as
+`--ssd-streaming-cache-experts 4000` is different: it means exactly 4000 dynamic
+expert slots, with no extra accounting. Non-routed weights, KV cache, graph
+scratch, and activations need additional memory. The automatic cache budget takes
+80% of the Metal recommended working set, subtracts non-routed weights, then
+applies the same routed-prefill headroom before sizing the dynamic cache. Leave
+the hot expert preload enabled for normal use; use `--ssd-streaming-cold` and
+`--ssd-streaming-preload-experts N` only for measurements.
+
+### Practical SSD streaming examples
+
+On 64GB MacBooks, start with the 2-bit Flash GGUF and a moderate expert cache:
+
+```sh
+./download_model.sh q2-imatrix
+
+./ds4 \
+  -m ./ds4flash.gguf \
+  --ssd-streaming \
+  --ssd-streaming-cache-experts 32GB \
+  --ctx 32768 \
+  --nothink
+```
+
+On 128GB MacBooks, PRO q2 streaming is experimental but usable for inspection
+and occasional work when you accept slow generation. Start with `--nothink`:
+
+```sh
+./download_model.sh pro-q2-imatrix
+
+./ds4 \
+  -m gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf \
+  --ssd-streaming \
+  --ctx 32768 \
+  --nothink
+```
+
+On an M5 Max with 128GB of RAM, a short PRO q2 streaming decode benchmark found
+the automatic budget best: it selected about `59GB` of routed expert cache.
+Manual `64GB` to `75GB` caches were close on that machine. Prefer the automatic
+budget; if setting the cache manually on this class of machine, start around
+`48GB` to `64GB`, then increase only while the machine remains responsive and
+the startup log shows the requested dynamic cache. Once the machine is stable,
+re-enable thinking with a conservative generation limit:
+
+```sh
+./ds4 \
+  -m gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf \
+  --ssd-streaming \
+  --ctx 32768 \
+  --think \
+  --tokens 1500
+```
+
+The important startup line is the cache report. Start conservative, then
+increase the cache if the machine has headroom.
+
+## Distributed Inference
+
+Distributed inference lets DwarfStar **run a model that is too large for one machine** by
+splitting transformer layers across multiple machines. The main example is the
+full 4-bit Flash quant across two 128 GB MacBooks: each process maps only its
+own layer slice, activations are sent over TCP, and the coordinator keeps normal
+CLI/API behavior.
+
+Distributed inference also allows to **speed up prefill** by
+using multiple GPUs at the same time to process different micro-batches at
+different layers, like in an assembly line. Only prefill can be accelerated this
+way. Generation is purely autoregressive: each token must finish across the
+route before the next token can start. The model work is the same as a single
+process, plus coordination latency, so distributed generation is slower.
+
+To build an initial mental model, here are the high level concepts:
+
+1. You put the GGUF on every machine, but each one loads just a subset. `--layers` controls which tensors are mapped, so a worker with `--layers 20:output` does not load the earlier layers.
+2. Layer ranges are inclusive: `10:20` means layers 10, 11, ..., 20. `N:output` means layer `N` through the final layer plus the output head.
+3. You assign one of the machines the role of `coordinator`, the others the roles of `workers`. Workers will connect to the coordinator and will tell they are there and which layers they are able to process.
+4. Each worker keeps its slice of the KV cache.
+5. Communication is worker-to-worker, there is no need to use the coordinator as relay, so if your coordinator is `A`, and you make a request, activations will flow in `A -> B -> C -> back to A`.
+
+### How it works and how to configure it
+
+The prefill path is pipelined (this is why it can go faster than in a single machine).
+For large prompts the coordinator can run its
+slice on chunk N+1 while the worker is running its slice on chunk N. The
+distributed rows below were measured with two M5 Max 128 GB MacBooks connected
+by Thunderbolt 5, using the Q4 Flash GGUF and the default 4096-token
+distributed prefill chunk. The single-process column is a reference run with
+the Q2 GGUF on a single machine, so it actually is a bit faster since
+the routed MoEs are smaller.
+
+| Prompt | Single-process reference | Two MacBooks | Speedup |
+| ---: | ---: | ---: | ---: |
+| 9421 tokens | 421.70 t/s | 582.22 t/s | 1.38x |
+| 28684 tokens | 405.30 t/s | 674.16 t/s | 1.66x |
+| 63819 tokens | 353.62 t/s | 654.79 t/s | 1.85x |
+
+Generation is different. **It is strictly autoregressive**: token N+1 cannot start
+until token N has produced logits and sampling has selected the next token. That
+means distributed generation cannot use the long prefill pipeline. It pays at
+least one cross-machine activation hop per generated token, so generation is
+slower than a single local process. On the same two-Mac Thunderbolt setup, a
+12k-context control run with the 91 GB Flash quant went from 30.59 t/s
+single-process to 24.67 t/s distributed, a 19.4% loss. Distributed inference is
+therefore mainly for fitting larger models and speeding up long prefills, not
+for making decode faster.
+
+### Full DeepSeek V4 PRO Q4 on two Mac Studios
+
+The full-size PRO Q4 GGUF can be run across two 512 GB Mac Studio M3 Ultra
+machines by giving the coordinator layers `0:30` and the worker
+`31:output`. Use the split GGUF files so each side maps only the tensors it
+needs:
+
+```sh
+# Coordinator machine.
+./download_model.sh pro-q4-layers00-30
+
+# Worker machine.
+./download_model.sh pro-q4-layers31-output
+```
+
+The two files are:
+
+```text
+gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf
+gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf
+```
+
+This is a capacity use case: each process maps only its own half of the model,
+while the worker owns the output head and returns logits.
+
+The current PRO Q4 Metal path uses queue-resident exact expert tables for the
+large routed experts. This avoids the broad multi-GiB routed-tensor bindings
+that made early distributed PRO Q4 attempts either run very slowly or hit Metal
+memory accounting limits. In a short greedy smoke test over the direct
+`192.168.0.182` / `192.168.0.183` link, the model generated coherent text and
+measured 11.47 t/s generation after startup. Per-token telemetry was balanced:
+local layers were around 39-43 ms, remote layers around 44-49 ms, for total
+token times around 84-92 ms. Expect a slow startup while each side maps and
+makes its half of the model resident. Long-context PRO Q4 prefill and decode
+performance still needs separate benchmarking.
+
+The measurements above use a Thunderbolt 5 cable. The implementation is plain
+TCP and also works over slower links, including WiFi, but fast Ethernet or
+Thunderbolt networking is strongly recommended. Slow links mostly hurt
+generation latency and short prefills; large prefills can still benefit when
+the layer split is balanced. In the normal performance path, the last worker
+owns the output head and returns logits directly.
+
+Minimal two-host configuration:
+
+```sh
+# Machine A: coordinator, owns tokenization, sampling, the prompt, and layers 0..30.
+./ds4 \
+  -m gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf \
+  --role coordinator \
+  --layers 0:30 \
+  --listen 169.254.43.68 1234
+
+# Machine B: worker, connects to A and owns layers 31..output.
+./ds4 \
+  -m gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf \
+  --role worker \
+  --layers 31:output \
+  --coordinator 169.254.43.68 1234
+```
+
+Normally the final worker should own the output head too, for example
+`--layers 20:output`. This avoids returning a full final hidden-state batch
+after prefill and lets the final worker produce the logits directly. On very
+slow or metered links, `--layers 20:42` is also supported: the coordinator will
+load the output head and compute logits locally, trading extra coordinator work
+for smaller per-token replies.
+
+### Network Link Comparison
+
+The table below shows the same two M5 Max hosts, the same 91 GB Flash quant,
+coordinator `--layers 0:19`, worker `--layers 20:output`, an 8192-token prompt
+from `speed-bench/promessi_sposi.txt`, and 128 generated tokens. WiFi and
+Internet numbers vary with local conditions, but the shape is the important
+part: high latency hurts generation directly, while lower bandwidth also pulls
+down long-prefill speed.
+
+| Link | Addresses | Ping avg | Prefill | Generation |
+| --- | --- | ---: | ---: | ---: |
+| Thunderbolt 5 | `169.254.43.68` -> `169.254.12.245` | 0.45 ms | 582.99 t/s | 25.09 t/s |
+| WiFi | `192.168.1.57` -> `192.168.1.95` | 77.20 ms | 250.70 t/s | 10.70 t/s |
+| Internet / VPN | `10.77.0.4` -> `10.77.0.3` | 152.10 ms | 114.88 t/s | 3.63 t/s |
+
+The Internet/VPN case is not meant to be a good interactive experience. It is
+still useful for collective testing: multiple people can temporarily combine
+machines to run a larger model that would not fit on any single host, accepting
+slow decode in exchange for being able to inspect the model at all.
+
+Use the coordinator exactly like normal `./ds4`: interactive chat, `/read`,
+and ordinary generation go through the same high-level session API. The same
+distributed options are also wired into `ds4-agent`, `ds4-eval`, and
+`ds4-bench`. For benchmarks, workers should already be running; `ds4-bench`
+waits until a complete route is available.
+
+Useful tuning and diagnostics:
+
+```sh
+./ds4-bench \
+  -m gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf \
+  --prompt-file speed-bench/promessi_sposi.txt \
+  --ctx-start 32768 \
+  --ctx-max 65536 \
+  --step-incr 32768 \
+  --gen-tokens 0 \
+  --role coordinator \
+  --layers 0:19 \
+  --listen 169.254.43.68 1234 \
+  --debug
+```
+
+`--debug` on the coordinator prints route formation and per-hop telemetry:
+layer range, token span, local evaluation time, downstream wait time, socket
+send time, and input/output byte counts. This is the current profiling tool for
+deciding whether a split is balanced. `--dist-prefill-window N` controls how
+many prefill chunks may be in flight end-to-end; the default is conservative
+and bounded. `--dist-prefill-chunk N` exists for experiments, but the default
+4096-token chunk is the canonical setting and should be used unless you are
+explicitly validating a different chunk size.
+
+By default DwarfStar sends hidden-state activations as 32-bit floats. To reduce
+traffic, pass `--dist-activation-bits 16` or `--dist-activation-bits 8` on the
+coordinator. This changes only the transport format between machines, not the
+model weights or KV cache. 16-bit transport halves activation traffic and is the
+first option to try on Ethernet or WiFi. 8-bit transport is more aggressive and
+should be treated as an approximate/experimental mode unless you have validated
+the output for your use case. However experimentally reduction activation
+size didn't provide a significant improvement, so this option may be removed
+in the future.
+
+**If a worker disconnects, the coordinator removes that worker from the active
+route**. The request already in flight can fail, and later calls report an
+incomplete route until a compatible worker reconnects and sends a new
+registration. For live sessions, the coordinator keeps the token history and can
+rebuild worker KV state by replaying the prefix when the route is available
+again. Workers also validate a rolling 64-bit token-prefix hash on every work
+item, so a restarted worker at position 0 cannot silently accept work for
+position N; it reports the mismatch and the coordinator replays the current
+transcript. Ctrl+C in the CLI and agent is cooperative: DwarfStar waits for the
+current distributed token or prefill chunk to drain before returning control,
+which avoids coordinator-caused KV splits. Saved agent/server sessions use the
+same KV file format as single-machine sessions: during save the coordinator
+fetches worker-owned layer tensors and serializes one normal payload; during
+load it splits that payload over the currently registered route.
+
+### Distributed protocol overview
+
+At the protocol level there are two kinds of connections. Workers keep a
+control TCP connection open to the coordinator and send a `HELLO` with their
+model ID, model family, quant profile, layer slice, context capacity, and data
+port. The coordinator uses these registrations to build a route that covers all
+layers. Work then moves over low-latency TCP data connections: the coordinator
+computes the first slice, sends a `WORK` frame with session ID, token positions,
+rolling token-prefix hashes before and after the span, route information, and
+hidden-state payload, and each worker computes its slice. Middle workers can
+forward directly to the next worker. The final worker returns logits to the
+coordinator, or ACKs for non-final prefill chunks so the prefill pipeline can
+stay full. `RESULT` frames echo the request ID and the post-span hash. A worker
+status error is handled differently from a socket failure: KV/hash mismatch can
+be recovered by replaying the token history on the same route, while transport
+failure drops the route and waits for a replacement worker. For persistent KV,
+the coordinator opens worker data connections and sends snapshot save/load
+messages for each worker-owned layer range; the disk payload remains a single
+agent/server cache file. The protocol has no
+encryption or authentication, and is not release-stable yet; coordinator and
+workers should be built from the same commit and used on trusted machines and
+trusted networks.
+
+## Reducing heat, power usage and fan noise
+
+Long local inference runs can keep the GPU busy for extended periods. If you
+care more about heat, fan noise, battery life on MacBooks, or reducing thermal
+stress on the hardware than about maximum throughput, use `--power N`.
+
+`--power 100` is the default and means full speed. Lower values ask DwarfStar to target
+that percentage of GPU usage: `--power 70` targets about 70%, `--power 50`
+targets about half usage, and so forth. DwarfStar does this by measuring GPU work time
+and inserting small sleeps between work units: during prefill it sleeps between
+layers, and during generation it sleeps between decoded tokens. This reduces
+sustained load without changing model output.
+
+The option is available on the CLI, server, agent, eval, and benchmark tools,
+for example:
+
+```sh
+./ds4 --power 50
+./ds4-agent --power 70
+./ds4-server --power 40 --ctx 100000
+```
+
+## Native agent
+
+DwarfStar features a native coding agent that works in a different way
+than most other systems: the inference is controlled from within the agent
+itself, without socket/API boundaries, so the session is represented
+by the on-disk KV cache itself. Moreover the tools and the system prompt
+are all designed vertically for DeepSeek v4 Flash and PRO. This provides a
+few advantages:
+
+* Low latency experience, bounded mainly by the prefill speed limits. Displaying of generated text, tool calling, start of a new session are always instantaneous.
+* Live progress bar during prefill time.
+* No DSML tool calling conversion, the tools are handled natively in the LLM format.
+* KV cache mismatch are impossible by construction, the current state is always the truth.
+* Everything is tuned for this model.
+* Ability to switch saved sessions with `/list` and `/switch`; full KV sessions resume without a prefill stage.
+
+Agent sessions are stored in `~/.ds4/kvcache`. Use `/save` to persist the
+current session, `/list` to show saved sessions sorted by recent update time,
+and `/switch <sha>` to resume one of them. The session ID is stable across
+future saves and is derived from the first user prompt and creation time.
+`/del <sha>` removes a saved session. `/strip <sha>` keeps the rendered
+conversation text and title but removes the heavy KV payload; switching to a
+stripped session rebuilds the KV cache by prefilling the saved text.
+
+Use `--chdir /path/to/ds4` when launching `ds4-agent` from another directory,
+so relative runtime files such as `metal/*.metal` resolve from the project tree.
+
+However while the system already works, there is a lot of work to do
+in order to make it ready for prime time. When finally the agent will reach
+the wanted shape, we will *likely* split the server and the client creating a stateful
+session-based protocol that can recreate all that in a client-server way.
 
 ## Benchmarking
 
@@ -194,6 +581,107 @@ exponential sweeps. Output is CSV with one row per frontier: latest prefill
 interval tokens/sec, generation tokens/sec at that frontier, and
 `kvcache_bytes`.
 
+Sessions prefill long prompts in 4096-token chunks by default. Set
+`DS4_METAL_PREFILL_CHUNK=N` to compare another chunk size, for example `2048`
+to match the strict official-vector checkpoint path, or
+`DS4_METAL_PREFILL_CHUNK=0` to prefill a prompt as one whole batch when memory
+allows. Changing the chunk changes the KV checkpoint/logit path, so compare it
+as an explicit run configuration.
+Chunked Metal prefill reuses the same range-capable layer-major graph for each
+chunk, preserving absolute compressor/indexer boundaries while avoiding the old
+per-layer chunk dispatch path.
+
+## Capability Evaluation
+
+`ds4-eval` is a small real-model integration benchmark. It is not a leaderboard
+runner and should not be reported as an official GPQA, SuperGPQA, AIME, or
+security benchmark score: the questions are an embedded 92-item subset chosen
+to make local regression testing useful and visually inspectable. The program
+loads the real GGUF, renders DeepSeek chat prompts, streams sampled tokens in a split-screen TUI, grades
+the final answer, and prints a per-question report with prompt tokens,
+generated tokens, pass/fail state, the model answer, and the correct answer.
+
+```sh
+./ds4-eval -m ds4flash.gguf --trace /tmp/ds4-eval.txt
+```
+
+The default run uses `--tokens 16000`, thinking mode enabled, and a soft/hard
+`</think>` budget cutoff so the model has room to produce a visible answer.
+`ds4-eval` sizes the context internally from the largest selected prompt plus
+the generation budget, and refuses runs that would need more than 1M context
+tokens. Press `p` to pause, `q` to exit and print the report, Up/Down to
+inspect or select another question, and Enter to run the selected question next.
+`--plain` disables the TUI.
+
+Use `--regrade-trace /path/to/trace.txt` to replay the current answer
+extractor and scorer against a prior `--trace` file without loading the model
+or regenerating tokens. This is useful when auditing evaluator changes: it
+shows which cases changed, the old picked answer, the new picked answer, and a
+pass/fail summary.
+
+For inference changes that can affect generation drift, keep this deterministic
+q1..q4 token-count gate in the test plan:
+
+```sh
+./ds4-eval \
+  -m ds4flash.gguf \
+  --plain \
+  --questions 4 \
+  --tokens 2048 \
+  --temp 0 \
+  --seed 1
+```
+
+The generated-token counts must stay aligned with the baseline:
+
+| Question | Expected state | Expected generated tokens | Expected given/correct |
+|---:|---|---:|---|
+| 1 | `PASSED` | 2048 | `B` / `B` |
+| 2 | `PASSED` | 438 | `C` / `C` |
+| 3 | `PASSED` | 666 | `70` / `70` |
+| 4 | `FAILED` | 2048 | `A` / `C` |
+
+The first 75 embedded questions are interleaved as 25 GPQA Diamond, 25 audited
+SuperGPQA, and 25 AIME 2025 problems. The final 17 are an audited COMPSEC
+subset of reduced single-function C/C++ vulnerability-localization questions.
+The model is asked for the single best source line, or the smallest exact line
+set only when the bug cannot be localized to one line; the scorer accepts small
+audited ranges only when adjacent lines are equivalent locations for the same
+bug. The order is
+intentionally progressive: early questions are useful smoke tests, while later
+questions are hard enough that a strong reasoning model should still miss some
+of them. The SuperGPQA slice is curated rather than blind: upstream rows with
+wrong keys, missing figures, or underspecified prompts are replaced with cleaner
+rows.
+
+The set should be treated as a hard capability regression suite rather than
+a pass/fail unit test.
+
+- **GPQA Diamond** contributes graduate-level science questions with
+  multiple-choice answers. DeepSeek's model card reports strong results
+  on full GPQA Diamond in thinking mode, but individual items still require
+  careful physics, chemistry, or biology reasoning and are easy to lose with a
+  small prompt/rendering or sampling regression.
+- **SuperGPQA** contributes broad specialist knowledge and domain-transfer
+  questions. The model-card SuperGPQA number is much lower than GPQA Diamond,
+  so these items are expected to be uneven: some look mundane, others require
+  niche professional knowledge or exact interpretation of a translated-style
+  exam question.
+- **AIME 2025** contributes exact-answer contest math. These are often the most
+  unforgiving items in the set: no multiple-choice prior, no partial credit, and
+  a single arithmetic or algebraic slip changes the grade.
+- **COMPSEC** contributes single-function C/C++ security reasoning items
+  reduced from public CVE writeups. These are not exploit prompts: the task is
+  to identify the best source line where the defensive code flaw is introduced,
+  or return `0` for a safe function.
+
+In practice this means `ds4-eval` should not be expected to produce a perfect
+92/92 run. It is meant to answer a more useful engineering question: after a
+kernel, quantization, prompt-rendering, KV-cache, or tool-streaming change, does
+DeepSeek V4 Flash still solve a representative mix of hard science, broad
+knowledge, exact math, and security-code problems while using the same inference
+path users run?
+
 ## CLI
 
 One-shot prompt:
@@ -209,7 +697,7 @@ No `-p` starts the interactive prompt:
 ds4>
 ```
 
-The interactive CLI is a real multi-turn DS4 chat. It keeps the rendered chat
+The interactive CLI is a real multi-turn chat. It keeps the rendered chat
 transcript and the live graph KV checkpoint, so each turn extends the previous
 conversation. Useful commands are `/help`, `/think`, `/think-max`, `/nothink`,
 `/ctx N`, `/read FILE`, and `/quit`. Ctrl+C interrupts the current generation
@@ -229,6 +717,9 @@ Start a local OpenAI/Anthropic-compatible server:
 ./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
 ```
 
+Use `--chdir /path/to/ds4` when launching `ds4-server` from another directory,
+so relative runtime files such as `metal/*.metal` resolve from the project tree.
+
 The server keeps one mutable backend/KV checkpoint in memory,
 so stateless clients that resend a longer version of the same prompt can reuse
 the shared prefix instead of pre-filling from token zero.
@@ -242,11 +733,16 @@ Supported endpoints:
 
 - `GET /v1/models`
 - `GET /v1/models/deepseek-v4-flash`
+- `GET /v1/models/deepseek-v4-pro`
 - `POST /v1/chat/completions`
 - `POST /v1/responses`
 - `POST /v1/completions`
 - `POST /v1/messages`
 
+The Flash and PRO model endpoints are compatibility aliases. They both report
+the model currently loaded from the GGUF passed with `-m`; the endpoint name does
+not select a different model.
+
 `/v1/chat/completions` accepts the usual OpenAI-style `messages`,
 `max_tokens`/`max_completion_tokens`, `temperature`, `top_p`, `top_k`, `min_p`,
 `seed`, `stream`, `stream_options.include_usage`, `tools`, and `tool_choice`.
@@ -255,17 +751,23 @@ tool calls are mapped back to OpenAI tool calls.
 
 `/v1/responses` accepts OpenAI Responses-style `input`, `instructions`,
 `tools`, `tool_choice`, `max_output_tokens`, `temperature`, `top_p`, `stream`,
-and `reasoning`. It is the preferred endpoint for Codex CLI. The server still
-renders the same DSML prompt internally, so exact tool-call replay and KV cache
-prefix reuse follow the same path as chat completions.
+and `reasoning`. It is the preferred endpoint for Codex CLI. The server keeps
+Responses continuations bound to live state when possible, and can fall back to
+the same DSML rendering and KV prefix reuse used by chat completions.
 
 `/v1/messages` is the Anthropic-compatible endpoint used by Claude Code style
 clients. It accepts `system`, `messages`, `tools`, `tool_choice`, `max_tokens`,
 `temperature`, `top_p`, `top_k`, `stream`, `stop_sequences`, and thinking
 controls. Tool uses are returned as Anthropic `tool_use` blocks.
 
-Both APIs support SSE streaming. In thinking mode, reasoning is streamed in the
-native API shape instead of being mixed into final text. OpenAI chat streaming
+Default sampled API generation uses `temperature=1`, `top_p=1`, and
+`min_p=0.05`, so the default filter is relative probability rather than
+nucleus mass. In thinking mode DwarfStar uses those fixed sampling defaults and
+ignores client sampling knobs, matching DeepSeek's fixed-thinking API behavior.
+
+The chat, Responses, and Anthropic endpoints support SSE streaming. In thinking
+mode, reasoning is streamed in the native API shape instead of being mixed into
+final text. OpenAI chat streaming
 also streams tool calls as soon as the DSML invocation is recognized: the tool
 header is sent first, then parameter bytes are forwarded as
 `tool_calls[].function.arguments` deltas while generation continues. The
@@ -276,9 +778,14 @@ including `response.output_text.delta`, function-call argument events, and
 terminal `response.completed` / `response.incomplete` / `response.failed`
 events.
 
+For browser JavaScript clients served from another origin, start the server with
+`--cors` to emit `Access-Control-Allow-*` headers. This only changes HTTP
+headers; it does not expose the server on the LAN. Use `--host 0.0.0.0`
+explicitly when remote machines should be able to connect.
+
 ### Tool call handling and canonicalization
 
-DeepSeek V4 Flash emits tool calls as [DSML text](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/encoding/README.md). Agent clients do not send that
+DeepSeek V4 emits tool calls as [DSML text](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/encoding/README.md). Agent clients do not send that
 same text back on the next request: they send normalized OpenAI/Anthropic JSON
 tool-call objects. **If the server re-rendered those objects slightly
 differently, the rendered byte prefix would no longer match the live KV
@@ -600,7 +1107,7 @@ The DS4 session payload starts with thirteen little-endian `u32` fields:
 
 ```text
 0   magic = "DSV4"
-1   payload version = 1
+1   payload version = 2
 2   saved context size
 3   prefill chunk size
 4   raw KV ring capacity
@@ -634,6 +1141,12 @@ running one extra decode step. MTP draft logits/state are not persisted; after
 loading a disk checkpoint the draft state is invalidated and rebuilt by normal
 generation.
 
+Distributed coordinator sessions use the same `DSV4` payload. Worker-owned
+layer tensors are pulled during save and merged into the normal layer-ordered
+tensor stream; during load the coordinator splits that stream into the current
+route and pushes the relevant layer tensors back to the workers. The saved file
+does not retain the distributed topology.
+
 The tensor payload is DS4-specific KV/session state, not a generic inference
 graph dump. It is expected to be portable only across compatible `ds4.c`
 builds for this model layout.
@@ -725,12 +1238,15 @@ captured from the official DeepSeek V4 Flash API. The requests use
 `deepseek-v4-flash`, greedy decoding, thinking disabled, and the maximum
 `top_logprobs` slice exposed by the API. Local vectors are generated with
 `./ds4 --dump-logprobs` and compared by token bytes, so tokenizer/template or
-attention regressions show up before they become long generation failures.
+attention regressions show up before they become long generation failures. The
+C runner pins `DS4_METAL_PREFILL_CHUNK=2048` for this strict API-vector
+comparison.
 
-All project tests are driven by the C runner:
+All project tests are driven by the C runner, with a small `ds4-eval`
+extractor self-test run first:
 
 ```sh
-make test                  # ./ds4_test --all
+make test                  # ./ds4-eval --self-test-extractors && ./ds4_test --all
 ./ds4_test --logprob-vectors
 ./ds4_test --server
 ```
@@ -743,6 +1259,7 @@ first answer:
 ```sh
 ./ds4 --dump-tokens -p "..."
 ./ds4 --dump-logprobs /tmp/out.json --logprobs-top-k 20 --temp 0 -p "..."
+./ds4 --dump-logits /tmp/logits.json --metal --nothink --prompt-file prompt.txt
 ./ds4-server --trace /tmp/ds4-trace.txt ...
 ```
 

STRIXHALO.md

@@ -0,0 +1,136 @@
+# DS4 on Strix Halo
+
+This is the minimal setup for DS4 ROCm inference on a
+Strix Halo machine with 128 GB RAM and Radeon 8060S (`gfx1151`).
+
+## 1. Install ROCm
+
+On Ubuntu 26.04 LTS, install the ROCm compiler/runtime and libraries used by the Strix Halo backend:
+
+```sh
+sudo apt-get update
+sudo apt-get install -y \
+  hipcc rocminfo rocm-smi \
+  libamdhip64-dev \
+  libhipblas-dev libhipblaslt-dev \
+  librocblas-dev \
+  librocwmma-dev \
+  libhipcub-dev
+```
+
+The backend uses rocWMMA. On this Ubuntu 26.04 setup, `librocwmma-dev`
+installs the top-level rocWMMA headers but misses `rocwmma/internal/`.
+No Ubuntu package currently provides those internal headers. Install a complete
+matching rocWMMA header tree:
+
+```sh
+git clone --depth 1 --branch rocm-7.1.0 https://github.com/ROCm/rocWMMA.git /tmp/rocWMMA-rocm-7.1.0
+sudo mkdir -p /usr/local/include
+sudo cp -a /tmp/rocWMMA-rocm-7.1.0/library/include/rocwmma /usr/local/include/
+```
+
+If ROCm is installed under `/usr` but tooling expects `/opt/rocm`, add these
+compatibility links:
+
+```sh
+sudo mkdir -p /opt/rocm/bin
+sudo ln -sf /usr/bin/hipcc /opt/rocm/bin/hipcc
+sudo ln -sfn /usr/lib/x86_64-linux-gnu /opt/rocm/lib
+sudo ln -sfn /usr/include /opt/rocm/include
+```
+
+## 2. Enable ROCm access
+
+The user running DS4 must be able to open `/dev/kfd` and the DRM render node:
+
+```sh
+sudo usermod -aG render,video "$USER"
+```
+
+Log out and back in, or reboot. Verify:
+
+```sh
+rocminfo | grep -A80 'Name:                    gfx1151'
+```
+
+If DS4 says `no ROCm-capable device is detected`, check that `rocminfo` can open
+`/dev/kfd` and that `groups` includes `render`.
+
+## 3. Increase GPU-visible memory
+
+A 128 GB Strix Halo system may initially expose only about 62 GB of GPU-visible
+memory. DS4 needs the larger GTT aperture for the 80.76 GiB model plus runtime
+buffers.
+
+Use these kernel parameters:
+
+```text
+amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856
+```
+
+On Ubuntu with GRUB:
+
+```sh
+sudo cp /etc/default/grub /etc/default/grub.bak
+sudoedit /etc/default/grub
+```
+
+Set:
+
+```text
+GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856"
+```
+
+Then:
+
+```sh
+sudo update-grub
+sudo reboot
+```
+
+After reboot, verify:
+
+```sh
+cat /proc/cmdline
+sudo dmesg | grep -Ei 'GTT|gttsize|TTM|VRAM'
+rocminfo | grep -A80 'Name:                    gfx1151'
+```
+
+Expected signs:
+
+```text
+amdgpu:  126976M of GTT memory ready
+rocminfo gfx1151 pool: 130023424 KB
+```
+
+## 4. Build DS4
+
+Use the normal Strix Halo target. It builds the standard binary names:
+
+```sh
+make strix-halo -j"$(nproc)"
+```
+
+`make rocm` is an alias for `make strix-halo`.
+
+## 5. Use the right GGUF
+
+Use the standard IQ2XXS/Q2K/Q8 imatrix GGUF:
+
+```text
+DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
+```
+
+Avoid the mixed IQ2/IQ4 or IQ2/Q4 GGUFs on this machine for now. They put much
+more memory pressure on the ROCm path and can trigger system OOM instead of a
+clean DS4 failure.
+
+## 6. Run DS4
+
+Run it normally:
+
+```sh
+./ds4 -m gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
+```
+
+The ROCm build uses the Strix Halo backend automatically.

download_model.sh

@@ -1,12 +1,23 @@
 #!/bin/sh
 set -e
 
-REPO="antirez/deepseek-v4-gguf"
-Q2_FILE="DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf"
+DEEPSEEK_REPO="antirez/deepseek-v4-gguf"
+GLM_UNSLOTH_REPO="unsloth/GLM-5.2-GGUF"
+GLM_ANTIREZ_REPO="antirez/GLM-5.2-GGUF"
+REPO=$DEEPSEEK_REPO
 Q2_IMATRIX_FILE="DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf"
-Q4_FILE="DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf"
 Q4_IMATRIX_FILE="DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf"
+Q2_Q4_IMATRIX_FILE="DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf"
+PRO_Q2_IMATRIX_FILE="DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf"
+PRO_Q4_LAYERS00_30_FILE="DeepSeek-V4-Pro-Q4K-Layers00-30.gguf"
+PRO_Q4_LAYERS31_OUTPUT_FILE="DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf"
 MTP_FILE="DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf"
+GLM_UNSLOTH_Q4_REMOTE_BASE="UD-Q4_K_XL/GLM-5.2-UD-Q4_K_XL"
+GLM_UNSLOTH_Q4_LOCAL_BASE="GLM-5.2-UD-Q4_K_XL"
+GLM_UNSLOTH_Q4_FIRST_FILE="$GLM_UNSLOTH_Q4_LOCAL_BASE-00001-of-00011.gguf"
+GLM_ANTIREZ_IQ2XXS_FILE="GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf"
+GLM_ANTIREZ_Q2_FILE="GLM-5.2-UD-Q2_K_RoutedQ2K.gguf"
+GLM_ANTIREZ_Q4_FILE="GLM-5.2-UD-Q4_K_RoutedQ4K.gguf"
 
 ROOT=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
 OUT_DIR=${DS4_GGUF_DIR:-"$ROOT/gguf"}
@@ -18,40 +29,75 @@ TOKEN=${HF_TOKEN:-}
 
 usage() {
     cat <<EOF
-DeepSeek V4 Flash GGUF downloader
+DwarfStar GGUF downloader
 
 Usage:
   ./download_model.sh q2-imatrix [--token TOKEN]
+  ./download_model.sh q2-q4-imatrix [--token TOKEN]
   ./download_model.sh q4-imatrix [--token TOKEN]
-  ./download_model.sh q2 [--token TOKEN]
-  ./download_model.sh q4 [--token TOKEN]
+  ./download_model.sh pro-q2-imatrix [--token TOKEN]
+  ./download_model.sh pro-q4-layers00-30 [--token TOKEN]
+  ./download_model.sh pro-q4-layers31-output [--token TOKEN]
+  ./download_model.sh pro-q4-split [--token TOKEN]
   ./download_model.sh mtp [--token TOKEN]
+  ./download_model.sh glm-unsloth-q4 [--token TOKEN]
+  ./download_model.sh glm-antirez-iq2xxs [--token TOKEN]
+  ./download_model.sh glm-antirez-q2 [--token TOKEN]
+  ./download_model.sh glm-antirez-q4 [--token TOKEN]
 
 Targets:
-  *** PREFERRED GGUF FILES: USE THE IMATRIX VERSIONS BELOW ***
 
   q2-imatrix
        2-bit routed experts, about 81 GB on disk.
        Recommended model for 96 and 128 GB RAM machines.
 
+  q2-q4-imatrix
+       Mixed Flash quant: mostly q2 routed experts, with the last 6 layers
+       using q4 routed experts. About 98 GB on disk. Good for higher
+       quality inference for 128 GB MacBooks. Works on DGX Spark but loading
+       may struggle compared to q2-imatrix.
+
   q4-imatrix
        4-bit routed experts, about 153 GB on disk.
        Recommended model for machines with 256 GB RAM or more.
 
-  Legacy GGUF files:
+  pro-q2-imatrix
+       DeepSeek V4 PRO q2 imatrix quant, as a single GGUF file. About 430 GB
+       on disk; intended for 512 GB RAM machines.
 
-  q2   2-bit routed experts, about 81 GB on disk.
-       Older non-imatrix model for 96 and 128 GB RAM machines. Prefer
-       q2-imatrix unless you specifically need the legacy quant.
+  pro-q4-layers00-30
+       First half of the DeepSeek V4 PRO Q4 routed-expert quant, layers 0..30.
+       Use on the coordinator in a two-Mac-Studio distributed run. About 426 GB.
 
-  q4   4-bit routed experts, about 153 GB on disk.
-       Older non-imatrix model for machines with 256 GB RAM or more. Prefer
-       q4-imatrix unless you specifically need the legacy quant.
+  pro-q4-layers31-output
+       Second half of the DeepSeek V4 PRO Q4 routed-expert quant, layers
+       31..output. Use on the worker in a two-Mac-Studio distributed run.
+       About 412 GB.
+
+  pro-q4-split
+       Downloads both PRO Q4 split files into the download directory. About
+       838 GB total. This target does not update ./ds4flash.gguf.
 
   mtp  Optional speculative decoding component, about 3.5 GB on disk.
-       It is useful with q2-imatrix, q4-imatrix, q2, and q4, but must be
+       It is useful with q2-imatrix, q2-q4-imatrix, and q4-imatrix, but must be
        enabled explicitly with --mtp when running ds4 or ds4-server.
 
+  glm-unsloth-q4
+       GLM 5.2 Unsloth UD-Q4_K_XL quant from unsloth/GLM-5.2-GGUF.
+       Downloads all 11 shards and links ./ds4flash.gguf to the first shard.
+
+  glm-antirez-iq2xxs
+       GLM 5.2 antirez routed IQ2_XXS GGUF from antirez/GLM-5.2-GGUF.
+       Includes Q2_K block 78 and is intended for reduced-memory testing.
+
+  glm-antirez-q2
+       GLM 5.2 antirez routed Q2_K GGUF from antirez/GLM-5.2-GGUF.
+       About 262 GB on disk.
+
+  glm-antirez-q4
+       GLM 5.2 antirez routed Q4_K GGUF from antirez/GLM-5.2-GGUF.
+       About 434 GB on disk.
+
 Options:
   --token TOKEN  Hugging Face token. Otherwise HF_TOKEN or the local HF token
                  cache is used if present.
@@ -60,7 +106,7 @@ Environment:
   DS4_GGUF_DIR   Directory used for downloaded GGUF files.
                  Default: ./gguf
 
-After q2-imatrix/q4-imatrix/q2/q4 downloads the script updates:
+After main-model downloads the script updates:
   ./ds4flash.gguf -> <download directory>/<selected model>
 
 Then the default commands work:
@@ -69,6 +115,10 @@ Then the default commands work:
 
 After downloading mtp, enable it explicitly, for example:
   ./ds4 --mtp <download directory>/$MTP_FILE --mtp-draft 2
+
+PRO and GLM files are downloaded with the official Hugging Face downloader
+because they are too large, sharded, or nested for the curl path used by the
+smaller DeepSeek Flash GGUF files.
 EOF
 }
 
@@ -79,13 +129,48 @@ fi
 
 MODEL=$1
 shift
+MODEL_FILES=
+LINK_MODEL=1
+FORCE_HF_DOWNLOAD=0
+FLATTEN_DOWNLOADS=0
 
 case "$MODEL" in
     q2-imatrix) MODEL_FILE=$Q2_IMATRIX_FILE ;;
+    q2-q4-imatrix) MODEL_FILE=$Q2_Q4_IMATRIX_FILE ;;
     q4-imatrix) MODEL_FILE=$Q4_IMATRIX_FILE ;;
-    q2) MODEL_FILE=$Q2_FILE ;;
-    q4) MODEL_FILE=$Q4_FILE ;;
-    mtp) MODEL_FILE=$MTP_FILE ;;
+    pro-q2-imatrix) MODEL_FILE=$PRO_Q2_IMATRIX_FILE ;;
+    pro-q4-layers00-30) MODEL_FILE=$PRO_Q4_LAYERS00_30_FILE; LINK_MODEL=0 ;;
+    pro-q4-layers31-output) MODEL_FILE=$PRO_Q4_LAYERS31_OUTPUT_FILE; LINK_MODEL=0 ;;
+    pro-q4-split)
+        MODEL_FILES="$PRO_Q4_LAYERS00_30_FILE $PRO_Q4_LAYERS31_OUTPUT_FILE"
+        LINK_MODEL=0
+        ;;
+    mtp) MODEL_FILE=$MTP_FILE; LINK_MODEL=0 ;;
+    glm-unsloth-q4)
+        REPO=$GLM_UNSLOTH_REPO
+        MODEL_FILE=$GLM_UNSLOTH_Q4_FIRST_FILE
+        MODEL_FILES=
+        for part in 00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011; do
+            MODEL_FILES="$MODEL_FILES $GLM_UNSLOTH_Q4_REMOTE_BASE-${part}-of-00011.gguf"
+        done
+        FORCE_HF_DOWNLOAD=1
+        FLATTEN_DOWNLOADS=1
+        ;;
+    glm-antirez-q2)
+        REPO=$GLM_ANTIREZ_REPO
+        MODEL_FILE=$GLM_ANTIREZ_Q2_FILE
+        FORCE_HF_DOWNLOAD=1
+        ;;
+    glm-antirez-iq2xxs)
+        REPO=$GLM_ANTIREZ_REPO
+        MODEL_FILE=$GLM_ANTIREZ_IQ2XXS_FILE
+        FORCE_HF_DOWNLOAD=1
+        ;;
+    glm-antirez-q4)
+        REPO=$GLM_ANTIREZ_REPO
+        MODEL_FILE=$GLM_ANTIREZ_Q4_FILE
+        FORCE_HF_DOWNLOAD=1
+        ;;
     -h|--help|help)
         usage
         exit 0
@@ -120,14 +205,101 @@ if [ -z "$TOKEN" ] && [ -s "$HOME/.cache/huggingface/token" ]; then
     TOKEN=$(cat "$HOME/.cache/huggingface/token")
 fi
 
+needs_hf_download() {
+    if [ "${FORCE_HF_DOWNLOAD:-0}" -eq 1 ]; then
+        return 0
+    fi
+    case "$1" in
+        "$PRO_Q2_IMATRIX_FILE"|"$PRO_Q4_LAYERS00_30_FILE"|"$PRO_Q4_LAYERS31_OUTPUT_FILE")
+            return 0
+            ;;
+        *)
+            return 1
+            ;;
+    esac
+}
+
+find_hf_command() {
+    if command -v hf >/dev/null 2>&1; then
+        printf '%s\n' hf
+        return 0
+    fi
+    return 1
+}
+
+local_download_name() {
+    if [ "${FLATTEN_DOWNLOADS:-0}" -eq 1 ]; then
+        basename "$1"
+    else
+        printf '%s\n' "$1"
+    fi
+}
+
+download_one_hf() {
+    file=$1
+    local_file=$(local_download_name "$file")
+    out="$OUT_DIR/$local_file"
+    hf_out="$OUT_DIR/$file"
+    part="$out.part"
+
+    mkdir -p "$(dirname "$out")"
+
+    if [ -s "$out" ]; then
+        echo "Already downloaded: $out"
+        return
+    fi
+
+    if [ -e "$part" ]; then
+        echo "Found curl partial download: $part" >&2
+        echo "The Hugging Face downloader cannot resume curl .part files." >&2
+        echo "Move or remove that partial download before retrying this PRO target." >&2
+        exit 1
+    fi
+
+    HF_CMD=$(find_hf_command || true)
+    if [ -z "$HF_CMD" ]; then
+        echo "Large GGUF downloads require the official Hugging Face CLI." >&2
+        echo "Install it with:" >&2
+        echo "  python3 -m pip install -U huggingface_hub hf_xet" >&2
+        exit 1
+    fi
+
+    echo "Downloading $file"
+    echo "from https://huggingface.co/$REPO"
+    echo "using $HF_CMD download"
+    echo "If the download stops, run the same command again to resume it."
+
+    if [ -n "$TOKEN" ]; then
+        "$HF_CMD" download "$REPO" "$file" --repo-type model --local-dir "$OUT_DIR" --token "$TOKEN"
+    else
+        "$HF_CMD" download "$REPO" "$file" --repo-type model --local-dir "$OUT_DIR"
+    fi
+
+    if [ "$hf_out" != "$out" ] && [ -s "$hf_out" ]; then
+        mv "$hf_out" "$out"
+        rmdir "$(dirname "$hf_out")" 2>/dev/null || true
+    fi
+
+    if [ ! -s "$out" ]; then
+        echo "Hugging Face download finished but expected file is missing: $out" >&2
+        exit 1
+    fi
+}
+
 download_one() {
     file=$1
-    out="$OUT_DIR/$file"
+    local_file=$(local_download_name "$file")
+    out="$OUT_DIR/$local_file"
     part="$out.part"
     aria2_part="$out.aria2"
     url="https://huggingface.co/$REPO/resolve/main/$file"
 
-    mkdir -p "$OUT_DIR"
+    if needs_hf_download "$file"; then
+        download_one_hf "$file"
+        return
+    fi
+
+    mkdir -p "$(dirname "$out")"
 
     if [ -e "$aria2_part" ]; then
         echo "Found incomplete aria2 download sidecar: $aria2_part" >&2
@@ -153,14 +325,24 @@ download_one() {
     mv "$part" "$out"
 }
 
-download_one "$MODEL_FILE"
+if [ -n "$MODEL_FILES" ]; then
+    for file in $MODEL_FILES; do
+        download_one "$file"
+    done
+else
+    download_one "$MODEL_FILE"
+fi
 
 if [ "$MODEL" = "mtp" ]; then
     echo
-    echo "MTP is an optional component for q2-imatrix, q4-imatrix, q2, and q4."
+    echo "MTP is an optional component for q2-imatrix, q2-q4-imatrix, and q4-imatrix."
     echo "Enable it explicitly, for example:"
     echo "  ./ds4 --mtp $OUT_DIR/$MTP_FILE --mtp-draft 2"
-else
+elif [ "$MODEL" = "pro-q4-layers00-30" ] || [ "$MODEL" = "pro-q4-layers31-output" ] || [ "$MODEL" = "pro-q4-split" ]; then
+    echo
+    echo "Downloaded PRO Q4 distributed split file(s). Use them with --layers,"
+    echo "for example coordinator layers 0:30 and worker layers 31:output."
+elif [ "$LINK_MODEL" -eq 1 ]; then
     cd "$ROOT"
     ln -sfn "$OUT_DIR/$MODEL_FILE" ds4flash.gguf
     echo "Linked ./ds4flash.gguf -> $OUT_DIR/$MODEL_FILE"

ds4.h

@@ -6,6 +6,8 @@
 #include <stdint.h>
 #include <stdio.h>
 
+#include "ds4_ssd.h"
+
 /* Public engine boundary.
  *
  * The CLI and server should treat ds4_engine as the loaded model and
@@ -50,23 +52,77 @@ typedef struct {
     float logprob;
 } ds4_token_score;
 
+#define DS4_DEFAULT_TEMPERATURE 1.0f
+#define DS4_DEFAULT_TOP_P 1.0f
+#define DS4_DEFAULT_MIN_P 0.05f
+
 typedef struct ds4_engine ds4_engine;
 typedef struct ds4_session ds4_session;
 
 typedef void (*ds4_session_progress_fn)(void *ud, const char *event, int current, int total);
+typedef bool (*ds4_session_cancel_fn)(void *ud);
+
+#define DS4_SESSION_SYNC_INTERRUPTED 2
+
+typedef enum {
+    DS4_DISTRIBUTED_NONE = 0,
+    DS4_DISTRIBUTED_COORDINATOR,
+    DS4_DISTRIBUTED_WORKER,
+} ds4_distributed_role;
+
+typedef struct {
+    uint32_t start;
+    uint32_t end;
+    bool has_output;
+    bool set;
+} ds4_distributed_layers;
+
+typedef struct {
+    ds4_distributed_role role;
+    ds4_distributed_layers layers;
+    const char *listen_host;
+    int listen_port;
+    const char *coordinator_host;
+    int coordinator_port;
+    uint32_t prefill_chunk;
+    uint32_t prefill_window;
+    uint32_t activation_bits;
+    bool replay_check;
+    bool debug;
+} ds4_distributed_options;
 
 typedef struct {
     const char *model_path;
     const char *mtp_path;
     ds4_backend backend;
     int n_threads;
+    int context_size;
+    uint32_t prefill_chunk;
     int mtp_draft_tokens;
     float mtp_margin;
     const char *directional_steering_file;
+    const char *expert_profile_path;
     float directional_steering_attn;
     float directional_steering_ffn;
+    int power_percent;
+    uint32_t ssd_streaming_cache_experts;
+    uint64_t ssd_streaming_cache_bytes;
+    uint32_t ssd_streaming_full_layers;
+    uint32_t ssd_streaming_preload_experts;
+    uint64_t simulate_used_memory_bytes;
     bool warm_weights;
     bool quality;
+    bool ssd_streaming;
+    bool ssd_streaming_cold;
+    bool ssd_streaming_full_layers_set;
+    bool inspect_only;
+    bool first_token_test;
+    bool metal_graph_test;
+    bool load_slice;
+    uint32_t load_layer_start;
+    uint32_t load_layer_end;
+    bool load_output;
+    ds4_distributed_options distributed;
 } ds4_engine_options;
 
 typedef void (*ds4_token_emit_fn)(void *ud, int token);
@@ -88,16 +144,53 @@ typedef struct {
     uint64_t cap;
 } ds4_session_snapshot;
 
+typedef struct {
+    char *path;
+    uint64_t bytes;
+} ds4_session_payload_file;
+
 int ds4_engine_open(ds4_engine **out, const ds4_engine_options *opt);
 void ds4_engine_close(ds4_engine *e);
 void ds4_engine_summary(ds4_engine *e);
+int ds4_engine_vocab_size(ds4_engine *e);
+int ds4_engine_power(ds4_engine *e);
+int ds4_engine_set_power(ds4_engine *e, int power_percent);
+const char *ds4_engine_model_name(ds4_engine *e);
+int ds4_engine_layer_count(ds4_engine *e);
+uint32_t ds4_engine_layer_compress_ratio(ds4_engine *e, uint32_t layer);
+uint64_t ds4_engine_hidden_f32_values(ds4_engine *e);
+bool ds4_engine_glm_layer_payload_bytes(ds4_engine *e,
+                                        uint32_t layer,
+                                        uint32_t full_live,
+                                        uint32_t key_dim,
+                                        uint32_t value_dim,
+                                        uint32_t compact_live,
+                                        uint32_t index_live,
+                                        uint64_t *out);
+/* Stable id for cache compatibility.  0 is the original Flash shape, so old
+ * KV files with the previously-zero reserved byte remain Flash-compatible;
+ * Pro and later shapes must use nonzero ids. */
+int ds4_engine_model_id(ds4_engine *e);
+bool ds4_engine_is_glm_dsa(ds4_engine *e);
 const char *ds4_backend_name(ds4_backend backend);
 bool ds4_think_mode_enabled(ds4_think_mode mode);
 const char *ds4_think_mode_name(ds4_think_mode mode);
 const char *ds4_think_max_prefix(void);
+const char *ds4_glm_reasoning_effort_text(ds4_think_mode mode);
 uint32_t ds4_think_max_min_context(void);
 ds4_think_mode ds4_think_mode_for_context(ds4_think_mode mode, int ctx_size);
+/* Uses the active model shape selected by ds4_engine_open(); call after opening
+ * the GGUF so Flash/Pro dimensions are known. */
 ds4_context_memory ds4_context_memory_estimate(ds4_backend backend, int ctx_size);
+ds4_context_memory ds4_context_memory_estimate_with_prefill(
+        ds4_backend backend,
+        int ctx_size,
+        uint32_t prefill_chunk);
+ds4_context_memory ds4_context_memory_estimate_with_prefill_mode(
+        ds4_backend backend,
+        int ctx_size,
+        uint32_t prefill_chunk,
+        bool ssd_streaming);
 bool ds4_log_is_tty(FILE *fp);
 void ds4_log(FILE *fp, ds4_log_type type, const char *fmt, ...);
 int ds4_engine_generate_argmax(ds4_engine *e, const ds4_tokens *prompt,
@@ -141,10 +234,31 @@ void ds4_chat_append_assistant_prefix(ds4_engine *e, ds4_tokens *tokens, ds4_thi
 
 char *ds4_token_text(ds4_engine *e, int token, size_t *len);
 int ds4_token_eos(ds4_engine *e);
+bool ds4_token_is_stop(ds4_engine *e, int token);
+bool ds4_token_is_thinking_control(ds4_engine *e, int token);
+bool ds4_token_is_stop_for_think_mode(ds4_engine *e,
+                                      int token,
+                                      ds4_think_mode mode);
+int ds4_token_user(ds4_engine *e);
+int ds4_token_assistant(ds4_engine *e);
 
 int ds4_session_create(ds4_session **out, ds4_engine *e, int ctx_size);
 void ds4_session_free(ds4_session *s);
+int ds4_session_power(ds4_session *s);
+int ds4_session_set_power(ds4_session *s, int power_percent);
+bool ds4_session_is_distributed(ds4_session *s);
 void ds4_session_set_progress(ds4_session *s, ds4_session_progress_fn fn, void *ud);
+/* UI-only progress. It may report fine-grained progress inside a prefill chunk;
+ * callers must not treat it as a durable KV checkpoint boundary. */
+void ds4_session_set_display_progress(ds4_session *s, ds4_session_progress_fn fn, void *ud);
+/* Optional cooperative cancellation.  ds4_session_sync() checks it only at
+ * safe boundaries where the live checkpoint is either unchanged or represents a
+ * valid token prefix, and returns DS4_SESSION_SYNC_INTERRUPTED when it stops. */
+void ds4_session_set_cancel(ds4_session *s, ds4_session_cancel_fn fn, void *ud);
+void ds4_session_report_progress(ds4_session *s, const char *event, int current, int total);
+/* Distributed coordinator sessions return 1 when the full layer route is
+ * available, 0 when it is still incomplete, and -1 for a local API error. */
+int ds4_session_distributed_route_ready(ds4_session *s, char *err, size_t errlen);
 
 typedef enum {
     DS4_SESSION_REWRITE_ERROR = -1,
@@ -165,9 +279,13 @@ ds4_session_rewrite_result ds4_session_rewrite_from_common(
 int ds4_session_common_prefix(ds4_session *s, const ds4_tokens *prompt);
 int ds4_session_argmax(ds4_session *s);
 int ds4_session_argmax_excluding(ds4_session *s, int excluded_id);
+int ds4_sample_logits(const float *logits, int n_vocab, float temperature,
+                      int top_k, float top_p, float min_p, uint64_t *rng);
 int ds4_session_sample(ds4_session *s, float temperature, int top_k, float top_p, float min_p, uint64_t *rng);
 int ds4_session_top_logprobs(ds4_session *s, ds4_token_score *out, int k);
 int ds4_session_token_logprob(ds4_session *s, int token, ds4_token_score *out);
+int ds4_session_copy_logits(ds4_session *s, float *out, int cap);
+int ds4_session_set_logits(ds4_session *s, const float *logits, int n);
 int ds4_session_eval(ds4_session *s, int token, char *err, size_t errlen);
 int ds4_session_eval_speculative_argmax(ds4_session *s, int first_token,
                                         int max_tokens, int eos_token,
@@ -177,18 +295,66 @@ void ds4_session_invalidate(ds4_session *s);
 void ds4_session_rewind(ds4_session *s, int pos);
 int ds4_session_pos(ds4_session *s);
 int ds4_session_ctx(ds4_session *s);
+int ds4_session_prefill_cap(ds4_session *s);
 int ds4_engine_routed_quant_bits(ds4_engine *e);
+bool ds4_engine_has_output_head(ds4_engine *e);
 bool ds4_engine_has_mtp(ds4_engine *e);
 int ds4_engine_mtp_draft_tokens(ds4_engine *e);
 const ds4_tokens *ds4_session_tokens(ds4_session *s);
 
-/* Disk KV cache payload helpers.  The server owns the outer file header and
- * policy; the engine owns the DS4-specific serialized graph state. */
+/* Low-level graph slice entry points used by distributed inference.  The
+ * transport/session routing logic lives in ds4_distributed.c. */
+int ds4_session_layer_slice_reset(ds4_session *s, char *err, size_t errlen);
+int ds4_session_eval_layer_slice(ds4_session *s,
+                                 const int *tokens,
+                                 uint32_t n_tokens,
+                                 uint32_t pos0,
+                                 uint32_t layer_start,
+                                 uint32_t layer_end,
+                                 const float *input_hc,
+                                 float *output_hc,
+                                 bool output_logits,
+                                 float *logits,
+                                 char *err,
+                                 size_t errlen);
+int ds4_session_eval_output_head_from_hc(ds4_session *s,
+                                         const float *hidden_hc,
+                                         uint32_t n_tokens,
+                                         float *logits,
+                                         char *err,
+                                         size_t errlen);
+
+/* Disk KV payload helpers.  HTTP/agent code owns the outer file header and
+ * persistence policy; the engine owns the DS4-specific serialized graph state. */
+#define DS4_SESSION_PAYLOAD_MAGIC UINT32_C(0x34565344) /* "DSV4" */
+#define DS4_SESSION_PAYLOAD_VERSION UINT32_C(2)
+#define DS4_SESSION_PAYLOAD_U32_FIELDS 13u
+#define DS4_SESSION_LAYER_PAYLOAD_MAGIC UINT32_C(0x4c565344) /* "DSVL" */
+#define DS4_SESSION_LAYER_PAYLOAD_VERSION UINT32_C(1)
+#define DS4_SESSION_LAYER_PAYLOAD_U32_FIELDS 14u
+
 uint64_t ds4_session_payload_bytes(ds4_session *s);
+int ds4_session_stage_payload(ds4_session *s, ds4_session_payload_file *out,
+                              char *err, size_t errlen);
+int ds4_session_write_staged_payload(const ds4_session_payload_file *payload,
+                                     FILE *fp, char *err, size_t errlen);
+void ds4_session_payload_file_free(ds4_session_payload_file *payload);
 int ds4_session_save_payload(ds4_session *s, FILE *fp, char *err, size_t errlen);
 int ds4_session_load_payload(ds4_session *s, FILE *fp, uint64_t payload_bytes, char *err, size_t errlen);
 int ds4_session_save_snapshot(ds4_session *s, ds4_session_snapshot *snap, char *err, size_t errlen);
 int ds4_session_load_snapshot(ds4_session *s, const ds4_session_snapshot *snap, char *err, size_t errlen);
 void ds4_session_snapshot_free(ds4_session_snapshot *snap);
 
+uint64_t ds4_session_layer_payload_bytes(ds4_session *s,
+                                         uint32_t layer_start,
+                                         uint32_t layer_end);
+int ds4_session_save_layer_payload(ds4_session *s, FILE *fp,
+                                   uint32_t layer_start, uint32_t layer_end,
+                                   char *err, size_t errlen);
+int ds4_session_load_layer_payload(ds4_session *s, FILE *fp,
+                                   uint64_t payload_bytes,
+                                   const int *tokens, uint32_t n_tokens,
+                                   uint32_t layer_start, uint32_t layer_end,
+                                   char *err, size_t errlen);
+
 #endif

ds4_bench.c

@@ -1,13 +1,15 @@
 #include "ds4.h"
+#include "ds4_distributed.h"
+#include "ds4_help.h"
 
 /* Purpose-built throughput benchmark.
  *
  * The benchmark walks one fixed token sequence to configurable context
  * frontiers, measuring only the newest prefill interval at each frontier.  It
- * then snapshots the live session in memory, performs a fixed greedy decode
- * run without allowing EOS, restores the snapshot, and continues to the next
- * frontier.  Snapshot save/restore time is intentionally outside both timing
- * windows.
+ * then snapshots the live session in memory when the payload is small enough,
+ * performs a fixed greedy decode run without allowing EOS, restores the
+ * snapshot or replays the prefix, and continues to the next frontier.  Snapshot
+ * save/restore time is intentionally outside both timing windows.
  */
 
 #include <errno.h>
@@ -20,12 +22,15 @@
 #include <string.h>
 #include <time.h>
 
+#define DS4_BENCH_DEFAULT_SNAPSHOT_MAX_BYTES (UINT64_C(1) << 30)
+
 typedef struct {
     const char *model_path;
     const char *prompt_path;
     const char *chat_prompt_path;
     const char *system;
     const char *csv_path;
+    const char *expert_profile_path;
     ds4_backend backend;
     int threads;
     int ctx_start;
@@ -33,9 +38,21 @@ typedef struct {
     int ctx_alloc;
     int step_incr;
     int gen_tokens;
+    int power_percent;
+    uint32_t prefill_chunk;
+    uint32_t ssd_streaming_cache_experts;
+    uint64_t ssd_streaming_cache_bytes;
+    uint32_t ssd_streaming_full_layers;
+    uint32_t ssd_streaming_preload_experts;
+    uint64_t simulate_used_memory_bytes;
     double step_mul;
+    const char *dump_frontier_logits_dir;
+    ds4_dist_options dist;
     bool warm_weights;
     bool quality;
+    bool ssd_streaming;
+    bool ssd_streaming_cold;
+    bool ssd_streaming_full_layers_set;
 } bench_config;
 
 static double bench_now_sec(void) {
@@ -44,42 +61,31 @@ static double bench_now_sec(void) {
     return (double)ts.tv_sec + (double)ts.tv_nsec / 1000000000.0;
 }
 
-static void usage(FILE *fp) {
-    fprintf(fp,
-        "Usage: ds4-bench --prompt-file FILE [options]\n"
-        "\n"
-        "Benchmarks instantaneous prefill and generation throughput at context\n"
-        "frontiers such as 2048, 4096, 6144, ... . Generation is always greedy,\n"
-        "runs for exactly --gen-tokens tokens, and skips EOS so every row is\n"
-        "comparable.\n"
-        "\n"
-        "Input:\n"
-        "  --prompt-file FILE\n"
-        "      Raw benchmark text. The fixed token sequence is sliced at each frontier.\n"
-        "  --chat-prompt-file FILE\n"
-        "      Render FILE as one no-thinking chat user message, then slice that sequence.\n"
-        "  -sys, --system TEXT\n"
-        "      System prompt used only with --chat-prompt-file.\n"
-        "\n"
-        "Model and backend:\n"
-        "  -m, --model FILE       GGUF model path. Default: ds4flash.gguf\n"
-        "  --metal | --cuda | --cpu | --backend NAME\n"
-        "      Select backend explicitly. Defaults to Metal on macOS, CUDA elsewhere.\n"
-        "  -t, --threads N        CPU helper threads.\n"
-        "  --quality              Prefer exact kernels where applicable.\n"
-        "  --warm-weights         Touch mapped tensor pages before benchmarking.\n"
-        "\n"
-        "Sweep:\n"
-        "  --ctx-start N          First measured frontier. Default: 2048\n"
-        "  --ctx-max N            Last measured frontier. Default: 32768\n"
-        "  --ctx-alloc N          Allocated context. Default: ctx-max + gen-tokens + 1\n"
-        "  --step-mul F           Multiplicative step. Default: 1\n"
-        "  --step-incr N          Linear step when --step-mul is 1. Default: 2048\n"
-        "  --gen-tokens N         Greedy decode tokens per frontier. Default: 128\n"
-        "\n"
-        "Output:\n"
-        "  --csv FILE             Write CSV there instead of stdout.\n"
-        "  -h, --help             Show this help.\n");
+static uint64_t bench_snapshot_max_bytes(void) {
+    const char *env = getenv("DS4_BENCH_SNAPSHOT_MAX_BYTES");
+    if (!env || env[0] == '\0') return DS4_BENCH_DEFAULT_SNAPSHOT_MAX_BYTES;
+    if (!strcmp(env, "unlimited") || !strcmp(env, "UNLIMITED") ||
+        !strcmp(env, "inf") || !strcmp(env, "INF")) {
+        return UINT64_MAX;
+    }
+    char *end = NULL;
+    unsigned long long v = strtoull(env, &end, 10);
+    if (env[0] == '\0' || !end || *end != '\0') {
+        fprintf(stderr,
+                "ds4-bench: invalid DS4_BENCH_SNAPSHOT_MAX_BYTES=%s; using default %llu\n",
+                env,
+                (unsigned long long)DS4_BENCH_DEFAULT_SNAPSHOT_MAX_BYTES);
+        return DS4_BENCH_DEFAULT_SNAPSHOT_MAX_BYTES;
+    }
+    return (uint64_t)v;
+}
+
+static double bytes_to_gib(uint64_t bytes) {
+    return (double)bytes / (1024.0 * 1024.0 * 1024.0);
+}
+
+static void usage(FILE *fp, const char *topic) {
+    ds4_help_print(fp, DS4_HELP_BENCH, topic);
 }
 
 static int parse_int(const char *s, const char *opt) {
@@ -92,6 +98,16 @@ static int parse_int(const char *s, const char *opt) {
     return (int)v;
 }
 
+static int parse_nonnegative_int(const char *s, const char *opt) {
+    char *end = NULL;
+    long v = strtol(s, &end, 10);
+    if (s[0] == '\0' || *end != '\0' || v < 0 || v > INT_MAX) {
+        fprintf(stderr, "ds4-bench: invalid value for %s: %s\n", opt, s);
+        exit(2);
+    }
+    return (int)v;
+}
+
 static double parse_double_arg(const char *s, const char *opt) {
     char *end = NULL;
     double v = strtod(s, &end);
@@ -112,10 +128,18 @@ static const char *need_arg(int *i, int argc, char **argv, const char *opt) {
 
 static ds4_backend parse_backend(const char *s, const char *opt) {
     if (!strcmp(s, "metal")) return DS4_BACKEND_METAL;
+#ifdef DS4_ROCM_BUILD
+    if (!strcmp(s, "rocm")) return DS4_BACKEND_CUDA;
+#else
     if (!strcmp(s, "cuda")) return DS4_BACKEND_CUDA;
+#endif
     if (!strcmp(s, "cpu")) return DS4_BACKEND_CPU;
     fprintf(stderr, "ds4-bench: invalid value for %s: %s\n", opt, s);
+#ifdef DS4_ROCM_BUILD
+    fprintf(stderr, "ds4-bench: valid backends are: metal, rocm, cpu\n");
+#else
     fprintf(stderr, "ds4-bench: valid backends are: metal, cuda, cpu\n");
+#endif
     exit(2);
 }
 
@@ -183,9 +207,29 @@ static bench_config parse_options(int argc, char **argv) {
     for (int i = 1; i < argc; i++) {
         const char *arg = argv[i];
         if (!strcmp(arg, "-h") || !strcmp(arg, "--help")) {
-            usage(stdout);
+            const char *topic = (i + 1 < argc && argv[i + 1][0] != '-') ?
+                argv[i + 1] : NULL;
+            usage(stdout, topic);
             exit(0);
-        } else if (!strcmp(arg, "-m") || !strcmp(arg, "--model")) {
+        }
+        char dist_parse_err[256] = {0};
+        ds4_dist_cli_parse_result dist_parse =
+            ds4_dist_parse_cli_arg(arg,
+                                   &i,
+                                   argc,
+                                   argv,
+                                   &c.dist,
+                                   dist_parse_err,
+                                   sizeof(dist_parse_err));
+        if (dist_parse == DS4_DIST_CLI_ERROR) {
+            fprintf(stderr,
+                    "ds4-bench: %s\n",
+                    dist_parse_err[0] ? dist_parse_err : "invalid distributed option");
+            exit(2);
+        }
+        if (dist_parse == DS4_DIST_CLI_MATCHED) continue;
+
+        if (!strcmp(arg, "-m") || !strcmp(arg, "--model")) {
             c.model_path = need_arg(&i, argc, argv, arg);
         } else if (!strcmp(arg, "--prompt-file")) {
             c.prompt_path = need_arg(&i, argc, argv, arg);
@@ -204,26 +248,76 @@ static bench_config parse_options(int argc, char **argv) {
         } else if (!strcmp(arg, "--step-mul")) {
             c.step_mul = parse_double_arg(need_arg(&i, argc, argv, arg), arg);
         } else if (!strcmp(arg, "--gen-tokens") || !strcmp(arg, "--tokens") || !strcmp(arg, "-n")) {
-            c.gen_tokens = parse_int(need_arg(&i, argc, argv, arg), arg);
+            c.gen_tokens = parse_nonnegative_int(need_arg(&i, argc, argv, arg), arg);
         } else if (!strcmp(arg, "--csv")) {
             c.csv_path = need_arg(&i, argc, argv, arg);
+        } else if (!strcmp(arg, "--dump-frontier-logits-dir")) {
+            c.dump_frontier_logits_dir = need_arg(&i, argc, argv, arg);
+        } else if (!strcmp(arg, "--expert-profile")) {
+            c.expert_profile_path = need_arg(&i, argc, argv, arg);
         } else if (!strcmp(arg, "-t") || !strcmp(arg, "--threads")) {
             c.threads = parse_int(need_arg(&i, argc, argv, arg), arg);
         } else if (!strcmp(arg, "--backend")) {
             c.backend = parse_backend(need_arg(&i, argc, argv, arg), arg);
         } else if (!strcmp(arg, "--metal")) {
             c.backend = DS4_BACKEND_METAL;
+#ifdef DS4_ROCM_BUILD
+        } else if (!strcmp(arg, "--rocm")) {
+            c.backend = DS4_BACKEND_CUDA;
+#else
         } else if (!strcmp(arg, "--cuda")) {
             c.backend = DS4_BACKEND_CUDA;
+#endif
         } else if (!strcmp(arg, "--cpu")) {
             c.backend = DS4_BACKEND_CPU;
         } else if (!strcmp(arg, "--quality")) {
             c.quality = true;
+        } else if (!strcmp(arg, "--ssd-streaming")) {
+            c.ssd_streaming = true;
+        } else if (!strcmp(arg, "--ssd-streaming-cold")) {
+            c.ssd_streaming_cold = true;
+        } else if (!strcmp(arg, "--ssd-streaming-cache-experts")) {
+            uint32_t experts = 0;
+            uint64_t bytes = 0;
+            if (!ds4_parse_streaming_cache_experts_arg(
+                    need_arg(&i, argc, argv, arg), &experts, &bytes)) {
+                fprintf(stderr,
+                        "ds4-bench: --ssd-streaming-cache-experts must be a positive count or <number>GB\n");
+                exit(2);
+            }
+            c.ssd_streaming_cache_experts = experts;
+            c.ssd_streaming_cache_bytes = bytes;
+        } else if (!strcmp(arg, "--ssd-streaming-full-layers")) {
+            int v = parse_nonnegative_int(need_arg(&i, argc, argv, arg), arg);
+            c.ssd_streaming_full_layers = (uint32_t)v;
+            c.ssd_streaming_full_layers_set = true;
+        } else if (!strcmp(arg, "--ssd-streaming-preload-experts")) {
+            int v = parse_int(need_arg(&i, argc, argv, arg), arg);
+            if (v <= 0) {
+                fprintf(stderr, "ds4-bench: --ssd-streaming-preload-experts must be positive\n");
+                exit(2);
+            }
+            c.ssd_streaming_preload_experts = (uint32_t)v;
+        } else if (!strcmp(arg, "--simulate-used-memory")) {
+            if (!ds4_parse_gib_arg(need_arg(&i, argc, argv, arg),
+                                   &c.simulate_used_memory_bytes)) {
+                fprintf(stderr,
+                        "ds4-bench: --simulate-used-memory must be a positive GiB value, e.g. 64GB\n");
+                exit(2);
+            }
+        } else if (!strcmp(arg, "--prefill-chunk")) {
+            c.prefill_chunk = (uint32_t)parse_int(need_arg(&i, argc, argv, arg), arg);
+        } else if (!strcmp(arg, "--power")) {
+            c.power_percent = parse_int(need_arg(&i, argc, argv, arg), arg);
+            if (c.power_percent < 1 || c.power_percent > 100) {
+                fprintf(stderr, "ds4-bench: --power must be between 1 and 100\n");
+                exit(2);
+            }
         } else if (!strcmp(arg, "--warm-weights")) {
             c.warm_weights = true;
         } else {
             fprintf(stderr, "ds4-bench: unknown option: %s\n", arg);
-            usage(stderr);
+            usage(stderr, NULL);
             exit(2);
         }
     }
@@ -253,9 +347,114 @@ static bench_config parse_options(int argc, char **argv) {
         fprintf(stderr, "ds4-bench: --ctx-alloc must be greater than ctx-max + gen-tokens\n");
         exit(2);
     }
+    char dist_err[256];
+    if (ds4_dist_prepare_engine_options(&c.dist, NULL, dist_err, sizeof(dist_err)) != 0) {
+        fprintf(stderr, "ds4-bench: %s\n", dist_err);
+        exit(2);
+    }
+    if (c.dist.role == DS4_DISTRIBUTED_WORKER) {
+        fprintf(stderr, "ds4-bench: --role worker is a serving mode; start workers with ./ds4\n");
+        exit(2);
+    }
     return c;
 }
 
+static void json_write_string(FILE *fp, const char *s) {
+    fputc('"', fp);
+    if (s) {
+        for (const unsigned char *p = (const unsigned char *)s; *p; p++) {
+            switch (*p) {
+            case '"':  fputs("\\\"", fp); break;
+            case '\\': fputs("\\\\", fp); break;
+            case '\b': fputs("\\b", fp); break;
+            case '\f': fputs("\\f", fp); break;
+            case '\n': fputs("\\n", fp); break;
+            case '\r': fputs("\\r", fp); break;
+            case '\t': fputs("\\t", fp); break;
+            default:
+                if (*p < 0x20) fprintf(fp, "\\u%04x", (unsigned)*p);
+                else fputc((char)*p, fp);
+                break;
+            }
+        }
+    }
+    fputc('"', fp);
+}
+
+static int write_frontier_logits_json(
+        const bench_config *cfg,
+        ds4_engine         *engine,
+        ds4_session        *session,
+        int                 frontier,
+        int                 previous) {
+    if (!cfg->dump_frontier_logits_dir) return 0;
+
+    const int vocab = ds4_engine_vocab_size(engine);
+    float *logits = malloc((size_t)vocab * sizeof(logits[0]));
+    if (!logits) {
+        fprintf(stderr, "ds4-bench: out of memory copying frontier logits\n");
+        return 1;
+    }
+    if (ds4_session_copy_logits(session, logits, vocab) != vocab) {
+        fprintf(stderr, "ds4-bench: failed to copy frontier logits at %d\n", frontier);
+        free(logits);
+        return 1;
+    }
+
+    char path[PATH_MAX];
+    const int n = snprintf(path,
+                           sizeof(path),
+                           "%s/frontier_%06d.logits.json",
+                           cfg->dump_frontier_logits_dir,
+                           frontier);
+    if (n <= 0 || (size_t)n >= sizeof(path)) {
+        fprintf(stderr, "ds4-bench: frontier logits path is too long\n");
+        free(logits);
+        return 1;
+    }
+
+    FILE *fp = fopen(path, "wb");
+    if (!fp) {
+        fprintf(stderr, "ds4-bench: failed to open %s: %s\n", path, strerror(errno));
+        free(logits);
+        return 1;
+    }
+
+    const int argmax = ds4_session_argmax(session);
+    fprintf(fp, "{\n  \"source\":\"ds4-bench\",\n  \"model\":");
+    json_write_string(fp, cfg->model_path);
+    fprintf(fp,
+            ",\n  \"backend\":\"%s\",\n  \"quality\":%s,\n"
+            "  \"quant_bits\":%d,\n  \"prompt_tokens\":%d,\n"
+            "  \"frontier_tokens\":%d,\n  \"prefill_tokens\":%d,\n"
+            "  \"ctx\":%d,\n  \"vocab\":%d,\n"
+            "  \"argmax_id\":%d,\n  \"argmax_logit\":%.9g,\n  \"logits\":[",
+            ds4_backend_name(cfg->backend),
+            cfg->quality ? "true" : "false",
+            ds4_engine_routed_quant_bits(engine),
+            frontier,
+            frontier,
+            frontier - previous,
+            cfg->ctx_alloc,
+            vocab,
+            argmax,
+            logits[argmax]);
+    for (int i = 0; i < vocab; i++) {
+        if (i) fputc(',', fp);
+        if ((i % 8) == 0) fputs("\n    ", fp);
+        if (isfinite(logits[i])) fprintf(fp, "%.9g", logits[i]);
+        else fputs("null", fp);
+    }
+    fputs("\n  ]\n}\n", fp);
+    if (fclose(fp) != 0) {
+        fprintf(stderr, "ds4-bench: failed to close %s\n", path);
+        free(logits);
+        return 1;
+    }
+    free(logits);
+    return 0;
+}
+
 static int next_frontier(const bench_config *c, int cur) {
     if (cur >= c->ctx_max) return c->ctx_max;
     int next;
@@ -271,8 +470,15 @@ static int next_frontier(const bench_config *c, int cur) {
     return next;
 }
 
-static void log_context_memory(ds4_backend backend, int ctx_size) {
-    ds4_context_memory m = ds4_context_memory_estimate(backend, ctx_size);
+static void log_context_memory(ds4_backend backend,
+                               int         ctx_size,
+                               uint32_t    prefill_chunk,
+                               bool        ssd_streaming) {
+    ds4_context_memory m =
+        ds4_context_memory_estimate_with_prefill_mode(backend,
+                                                      ctx_size,
+                                                      prefill_chunk,
+                                                      ssd_streaming);
     fprintf(stderr,
             "ds4-bench: context buffers %.2f MiB (ctx=%d, backend=%s, prefill_chunk=%u, raw_kv_rows=%u, compressed_kv_rows=%u)\n",
             (double)m.total_bytes / (1024.0 * 1024.0),
@@ -283,19 +489,89 @@ static void log_context_memory(ds4_backend backend, int ctx_size) {
             m.comp_cap);
 }
 
+static int wait_distributed_route(ds4_session *session) {
+    char err[256] = {0};
+    char last[256] = {0};
+    unsigned ticks = 0;
+    const struct timespec delay = {0, 250000000L};
+
+    for (;;) {
+        int ready = ds4_session_distributed_route_ready(session, err, sizeof(err));
+        if (ready > 0) {
+            if (ticks) fprintf(stderr, "ds4-bench: distributed route ready\n");
+            return 0;
+        }
+        if (ready < 0) {
+            fprintf(stderr,
+                    "ds4-bench: distributed route readiness failed: %s\n",
+                    err[0] ? err : "unknown error");
+            return 1;
+        }
+        const char *why = err[0] ? err : "route incomplete";
+        if (strcmp(last, why) != 0 || (ticks % 20u) == 0) {
+            fprintf(stderr, "ds4-bench: waiting for distributed route: %s\n", why);
+            snprintf(last, sizeof(last), "%s", why);
+        }
+        nanosleep(&delay, NULL);
+        ticks++;
+    }
+}
+
+static void maybe_warn_distributed_step_shape(const bench_config *cfg, ds4_session *session) {
+    if (!cfg || !session || cfg->dist.role != DS4_DISTRIBUTED_COORDINATOR) return;
+    uint32_t chunk = cfg->dist.prefill_chunk;
+    if (chunk == 0) {
+        const int cap = ds4_session_prefill_cap(session);
+        if (cap > 0) chunk = (uint32_t)cap;
+    }
+    if (chunk == 0) return;
+    if (cfg->step_mul == 1.0 &&
+        cfg->step_incr > 0 &&
+        (uint32_t)cfg->step_incr < chunk &&
+        cfg->ctx_start < cfg->ctx_max)
+    {
+        fprintf(stderr,
+                "ds4-bench: note: --step-incr=%d is smaller than distributed prefill chunk %u; "
+                "suffix rows will not show multi-chunk pipeline overlap\n",
+                cfg->step_incr,
+                chunk);
+    }
+}
+
 int main(int argc, char **argv) {
     bench_config cfg = parse_options(argc, argv);
-    log_context_memory(cfg.backend, cfg.ctx_alloc);
 
     ds4_engine_options opt = {
         .model_path = cfg.model_path,
         .backend = cfg.backend,
         .n_threads = cfg.threads,
+        .context_size = cfg.ctx_alloc,
+        .prefill_chunk = cfg.prefill_chunk,
+        .ssd_streaming_cache_experts = cfg.ssd_streaming_cache_experts,
+        .ssd_streaming_cache_bytes = cfg.ssd_streaming_cache_bytes,
+        .ssd_streaming_full_layers = cfg.ssd_streaming_full_layers,
+        .ssd_streaming_preload_experts = cfg.ssd_streaming_preload_experts,
+        .simulate_used_memory_bytes = cfg.simulate_used_memory_bytes,
+        .power_percent = cfg.power_percent,
         .warm_weights = cfg.warm_weights,
         .quality = cfg.quality,
+        .ssd_streaming = cfg.ssd_streaming,
+        .ssd_streaming_cold = cfg.ssd_streaming_cold,
+        .ssd_streaming_full_layers_set = cfg.ssd_streaming_full_layers_set,
+        .expert_profile_path = cfg.expert_profile_path,
+        .distributed = cfg.dist,
     };
+    char dist_err[256];
+    if (ds4_dist_prepare_engine_options(&cfg.dist, &opt, dist_err, sizeof(dist_err)) != 0) {
+        fprintf(stderr, "ds4-bench: %s\n", dist_err);
+        return 2;
+    }
     ds4_engine *engine = NULL;
     if (ds4_engine_open(&engine, &opt) != 0) return 1;
+    log_context_memory(cfg.backend,
+                       cfg.ctx_alloc,
+                       cfg.prefill_chunk,
+                       cfg.ssd_streaming);
 
     char *text = read_file(cfg.prompt_path ? cfg.prompt_path : cfg.chat_prompt_path);
     ds4_tokens prompt = {0};
@@ -323,6 +599,15 @@ int main(int argc, char **argv) {
         ds4_engine_close(engine);
         return 1;
     }
+    if (cfg.dist.role == DS4_DISTRIBUTED_COORDINATOR &&
+        wait_distributed_route(session) != 0)
+    {
+        ds4_session_free(session);
+        ds4_tokens_free(&prompt);
+        ds4_engine_close(engine);
+        return 1;
+    }
+    maybe_warn_distributed_step_shape(&cfg, session);
 
     FILE *out = stdout;
     if (cfg.csv_path) {
@@ -335,11 +620,14 @@ int main(int argc, char **argv) {
             return 1;
         }
     }
-    fprintf(out, "ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes\n");
+    fprintf(out, "ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes\n");
     fflush(out);
 
     const int eos = ds4_token_eos(engine);
+    const bool distributed = cfg.dist.role == DS4_DISTRIBUTED_COORDINATOR;
     ds4_session_snapshot snap = {0};
+    const uint64_t snapshot_max_bytes = bench_snapshot_max_bytes();
+    bool warned_large_snapshot = false;
     char err[256];
     int previous = 0;
     int rc = 0;
@@ -361,13 +649,42 @@ int main(int argc, char **argv) {
         const double prefill_sec = prefill_t1 - prefill_t0;
         const int prefill_tokens = frontier - previous;
 
-        if (ds4_session_save_snapshot(session, &snap, err, sizeof(err)) != 0) {
-            fprintf(stderr, "ds4-bench: snapshot at %d failed: %s\n", frontier, err);
+        if (write_frontier_logits_json(&cfg, engine, session, frontier, previous) != 0) {
             rc = 1;
             break;
         }
 
+        const bool need_restore_after_generation =
+            cfg.gen_tokens > 0 && frontier < cfg.ctx_max;
+        bool have_snapshot = false;
+        if (need_restore_after_generation && !distributed &&
+            getenv("DS4_BENCH_DISABLE_SNAPSHOT") == NULL) {
+            const uint64_t payload_bytes = ds4_session_payload_bytes(session);
+            const bool large_snapshot_forced =
+                getenv("DS4_BENCH_FORCE_SNAPSHOT") != NULL;
+            if (payload_bytes > snapshot_max_bytes && !large_snapshot_forced) {
+                if (!warned_large_snapshot) {
+                    fprintf(stderr,
+                            "ds4-bench: session payload snapshot is %.2f GiB, above the %.2f GiB benchmark limit; "
+                            "replaying prefixes instead (set DS4_BENCH_FORCE_SNAPSHOT=1 to force snapshots)\n",
+                            bytes_to_gib(payload_bytes),
+                            bytes_to_gib(snapshot_max_bytes));
+                    warned_large_snapshot = true;
+                }
+            } else if (payload_bytes > 0) {
+                if (ds4_session_save_snapshot(session, &snap, err, sizeof(err)) != 0) {
+                    fprintf(stderr, "ds4-bench: snapshot at %d failed: %s\n", frontier, err);
+                    rc = 1;
+                    break;
+                }
+                have_snapshot = true;
+            }
+        }
+
         const double gen_t0 = bench_now_sec();
+        double gen_first_sec = 0.0;
+        double gen_steady_sec = 0.0;
+        int gen_done = 0;
         for (int i = 0; i < cfg.gen_tokens; i++) {
             if (ds4_session_pos(session) + 1 >= ds4_session_ctx(session)) {
                 fprintf(stderr, "ds4-bench: generation would exceed allocated context at frontier %d\n", frontier);
@@ -380,30 +697,49 @@ int main(int argc, char **argv) {
                 rc = 1;
                 break;
             }
+            const double token_t0 = bench_now_sec();
             if (ds4_session_eval(session, token, err, sizeof(err)) != 0) {
                 fprintf(stderr, "ds4-bench: decode at frontier %d failed: %s\n", frontier, err);
                 rc = 1;
                 break;
             }
+            const double token_t1 = bench_now_sec();
+            if (i == 0) gen_first_sec = token_t1 - token_t0;
+            else gen_steady_sec += token_t1 - token_t0;
+            gen_done++;
         }
         const double gen_t1 = bench_now_sec();
         if (rc != 0) break;
 
-        if (ds4_session_load_snapshot(session, &snap, err, sizeof(err)) != 0) {
-            fprintf(stderr, "ds4-bench: restore at %d failed: %s\n", frontier, err);
-            rc = 1;
-            break;
+        if (!need_restore_after_generation) {
+            /* Nothing later depends on the frontier state. */
+        } else if (distributed || !have_snapshot) {
+            if (ds4_session_sync(session, &prefix, err, sizeof(err)) != 0) {
+                fprintf(stderr, "ds4-bench: replay restore at %d failed: %s\n", frontier, err);
+                rc = 1;
+                break;
+            }
+        } else {
+            if (ds4_session_load_snapshot(session, &snap, err, sizeof(err)) != 0) {
+                fprintf(stderr, "ds4-bench: restore at %d failed: %s\n", frontier, err);
+                rc = 1;
+                break;
+            }
         }
 
         const double gen_sec = gen_t1 - gen_t0;
+        const int gen_steady_tokens = gen_done > 1 ? gen_done - 1 : 0;
         fprintf(out,
-                "%d,%d,%.2f,%d,%.2f,%llu\n",
+                "%d,%d,%.2f,%d,%.2f,%.3f,%d,%.2f,%llu\n",
                 frontier,
                 prefill_tokens,
                 prefill_sec > 0.0 ? (double)prefill_tokens / prefill_sec : 0.0,
-                cfg.gen_tokens,
-                gen_sec > 0.0 ? (double)cfg.gen_tokens / gen_sec : 0.0,
-                (unsigned long long)snap.len);
+                gen_done,
+                gen_sec > 0.0 ? (double)gen_done / gen_sec : 0.0,
+                gen_first_sec * 1000.0,
+                gen_steady_tokens,
+                gen_steady_sec > 0.0 ? (double)gen_steady_tokens / gen_steady_sec : 0.0,
+                (unsigned long long)(have_snapshot ? snap.len : 0));
         fflush(out);
 
         previous = frontier;

ds4_distributed.h

@@ -0,0 +1,125 @@
+#ifndef DS4_DISTRIBUTED_H
+#define DS4_DISTRIBUTED_H
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#include "ds4.h"
+
+/* Distributed inference is an engine backend, not a separate frontend API.
+ * Programs parse distributed options here, then keep using the normal
+ * ds4_session_* calls. Only worker/coordinator serving modes call
+ * ds4_dist_run() directly.
+ */
+typedef ds4_distributed_options ds4_dist_options;
+typedef struct ds4_dist_session ds4_dist_session;
+
+/* Options used by standalone `./ds4 --role coordinator -p ...` generation.
+ * Interactive tools and the server go through the normal ds4_session API.
+ */
+typedef struct {
+    const char *prompt;
+    const char *system;
+    const char *dump_logits_path;
+    const char *dump_logprobs_path;
+    int dump_logprobs_top_k;
+    int n_predict;
+    int ctx_size;
+    float temperature;
+    float top_p;
+    float min_p;
+    uint64_t seed;
+    ds4_think_mode think_mode;
+} ds4_dist_generation_options;
+
+typedef enum {
+    DS4_DIST_CLI_ERROR = -1,
+    DS4_DIST_CLI_NOT_MATCHED = 0,
+    DS4_DIST_CLI_MATCHED = 1,
+} ds4_dist_cli_parse_result;
+
+/* Shared option parsing. */
+bool ds4_dist_enabled(const ds4_dist_options *opt);
+ds4_dist_options *ds4_dist_options_create(void);
+void ds4_dist_options_free(ds4_dist_options *opt);
+void ds4_dist_usage(FILE *fp);
+ds4_dist_cli_parse_result ds4_dist_parse_cli_arg(
+        const char *arg,
+        int *index,
+        int argc,
+        char **argv,
+        ds4_dist_options *opt,
+        char *err,
+        size_t errlen);
+
+/* Applies distributed layer-loading choices to the engine options before the
+ * model is loaded.
+ */
+int ds4_dist_prepare_engine_options(
+        const ds4_dist_options *opt,
+        ds4_engine_options *engine,
+        char *err,
+        size_t errlen);
+
+/* Coordinator session backend used by ds4.c. These mirror the normal session
+ * operations; callers outside the engine should not need to call them directly.
+ */
+int ds4_dist_session_create(
+        ds4_dist_session **out,
+        ds4_engine *engine,
+        const ds4_dist_options *opt,
+        ds4_session *owner,
+        int ctx_size,
+        char *err,
+        size_t errlen);
+void ds4_dist_session_free(ds4_dist_session *d);
+
+/* Returns 1 when the coordinator has full layer coverage, 0 when workers are
+ * still missing, and -1 for configuration or internal errors.
+ */
+int ds4_dist_session_route_ready(ds4_dist_session *d, char *err, size_t errlen);
+
+/* Synchronize the distributed KV state to the requested prompt timeline. */
+int ds4_dist_session_sync(
+        ds4_dist_session *d,
+        ds4_session *owner,
+        const ds4_tokens *checkpoint,
+        const ds4_tokens *prompt,
+        float *logits,
+        char *err,
+        size_t errlen);
+
+/* Evaluate one additional token across the current distributed route. */
+int ds4_dist_session_eval(
+        ds4_dist_session *d,
+        ds4_session *owner,
+        const ds4_tokens *checkpoint,
+        int token,
+        float *logits,
+        char *err,
+        size_t errlen);
+
+/* Save/load use the normal DSV4 payload format. The coordinator gathers or
+ * pushes remote layer shards internally so saved files are topology-neutral.
+ */
+int ds4_dist_session_save_payload(
+        ds4_dist_session *d,
+        ds4_session *owner,
+        FILE *fp,
+        char *err,
+        size_t errlen);
+int ds4_dist_session_load_payload(
+        ds4_dist_session *d,
+        ds4_session *owner,
+        FILE *fp,
+        uint64_t payload_bytes,
+        char *err,
+        size_t errlen);
+
+/* Standalone distributed mode. Workers stay in this loop; coordinator one-shot
+ * mode uses it for `./ds4 --role coordinator -p ...`.
+ */
+int ds4_dist_run(ds4_engine *engine, const ds4_dist_options *opt, const ds4_dist_generation_options *gen);
+
+#endif

ds4_help.c

@@ -0,0 +1,564 @@
+#include "ds4_help.h"
+
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+
+typedef struct {
+    const char *off;
+    const char *cyan;
+    const char *title;
+    const char *yellow;
+    const char *grey;
+    const char *red;
+    const char *white;
+    const char *bright;
+} help_colors;
+
+static help_colors help_make_colors(FILE *fp) {
+    bool color = isatty(fileno(fp));
+    help_colors c = {0};
+    if (!color) return c;
+    c.off = "\x1b[0m";
+    c.cyan = "\x1b[38;5;81m";
+    c.title = "\x1b[1;38;5;250m";
+    c.yellow = "\x1b[38;5;179m";
+    c.grey = "\x1b[38;5;240m";
+    c.red = "\x1b[38;5;203m";
+    c.white = "\x1b[38;5;252m";
+    c.bright = "\x1b[1;38;5;231m";
+    return c;
+}
+
+static void title(FILE *fp, const help_colors *c, const char *s) {
+    fprintf(fp, "%s%s%s\n", c->title ? c->title : "", s, c->off ? c->off : "");
+}
+
+static void title_red(FILE *fp, const help_colors *c, const char *s) {
+    fprintf(fp, "%s%s%s\n", c->red ? c->red : "", s, c->off ? c->off : "");
+}
+
+static bool option_name_has_switch(const char *name) {
+    bool word_start = true;
+    while (*name) {
+        if (word_start && (*name == '-' || *name == '/')) return true;
+        word_start = (*name == ' ');
+        name++;
+    }
+    return false;
+}
+
+static void print_colored_option_name(FILE *fp, const help_colors *c, const char *name) {
+    bool has_switch = option_name_has_switch(name);
+    bool word_start = true;
+    while (*name) {
+        const char *start = name;
+        while (*name && *name != ' ') name++;
+        bool is_option = !has_switch || *start == '-' || *start == '/' ||
+                         (word_start && has_switch && *start != '[');
+        const char *color = is_option ? c->cyan : c->bright;
+        if (color) fputs(color, fp);
+        fwrite(start, 1, (size_t)(name - start), fp);
+        if (color && c->off) fputs(c->off, fp);
+        if (*name == ' ') {
+            fputc(*name++, fp);
+            word_start = false;
+        }
+    }
+}
+
+static void opt(FILE *fp, const help_colors *c, const char *name, const char *desc) {
+    if (c->cyan) {
+        fputs("  ", fp);
+        print_colored_option_name(fp, c, name);
+        fprintf(fp, " %s|%s ", c->grey ? c->grey : "", c->grey ? c->off : "");
+        fprintf(fp, "%s%s%s\n", c->white ? c->white : "", desc,
+                c->white ? c->off : "");
+        return;
+    }
+
+    const int col = 30;
+    int n = (int)strlen(name);
+    if (n > col) {
+        fprintf(fp, "  %s\n      %s\n", name, desc);
+    } else {
+        fprintf(fp, "  %-30s %s\n", name, desc);
+    }
+}
+
+static void para(FILE *fp, const help_colors *c, const char *s) {
+    fprintf(fp, "%s%s%s\n",
+            c->yellow ? c->yellow : "", s, c->yellow ? c->off : "");
+}
+
+static bool streq(const char *a, const char *b) {
+    return a && b && strcmp(a, b) == 0;
+}
+
+static bool topic_is(const char *topic, const char *name) {
+    return topic && strcmp(topic, name) == 0;
+}
+
+static const char *tool_name(ds4_help_tool tool) {
+    switch (tool) {
+    case DS4_HELP_DS4: return "ds4";
+    case DS4_HELP_SERVER: return "ds4-server";
+    case DS4_HELP_AGENT: return "ds4-agent";
+    case DS4_HELP_BENCH: return "ds4-bench";
+    case DS4_HELP_EVAL: return "ds4-eval";
+    }
+    return "ds4";
+}
+
+static const char *tool_usage(ds4_help_tool tool) {
+    switch (tool) {
+    case DS4_HELP_DS4:
+        return "Usage: ds4 [(-p PROMPT | --prompt-file FILE)] [options]";
+    case DS4_HELP_SERVER:
+        return "Usage: ds4-server [options]";
+    case DS4_HELP_AGENT:
+        return "Usage: ds4-agent [options]";
+    case DS4_HELP_BENCH:
+        return "Usage: ds4-bench (--prompt-file FILE | --chat-prompt-file FILE) [options]";
+    case DS4_HELP_EVAL:
+        return "Usage: ds4-eval [options]";
+    }
+    return "Usage: ds4 [options]";
+}
+
+static const char *tool_summary(ds4_help_tool tool) {
+    switch (tool) {
+    case DS4_HELP_DS4:
+        return "Chat with a local DwarfStar model, run one-shot prompts, inspect models, or coordinate distributed inference.";
+    case DS4_HELP_SERVER:
+        return "Serve one loaded DwarfStar model through OpenAI, Responses, Anthropic, and completion-compatible HTTP APIs.";
+    case DS4_HELP_AGENT:
+        return "Run the native terminal coding agent with live tools, session save/restore, and a responsive prompt while the model works.";
+    case DS4_HELP_BENCH:
+        return "Measure prefill, decode, context growth, and KV-cache size across repeatable context frontiers.";
+    case DS4_HELP_EVAL:
+        return "Run the built-in reasoning, math, science, and security evaluation harness with a live terminal UI.";
+    }
+    return "";
+}
+
+static void print_model_runtime(FILE *fp, const help_colors *c,
+                                ds4_help_tool tool, bool full) {
+    title(fp, c, "Model And Runtime");
+    opt(fp, c, "-m, --model FILE", "GGUF model path. Default: ds4flash.gguf");
+#ifdef DS4_ROCM_BUILD
+    opt(fp, c, "--metal | --rocm | --cpu", "Select the backend explicitly.");
+    opt(fp, c, "--backend NAME", "Backend name: metal, rocm, or cpu.");
+#else
+    opt(fp, c, "--metal | --cuda | --cpu", "Select the backend explicitly.");
+    opt(fp, c, "--backend NAME", "Backend name: metal, cuda, or cpu.");
+#endif
+    if (tool != DS4_HELP_BENCH) {
+        opt(fp, c, "-c, --ctx N", "Allocated context tokens.");
+    }
+    if (tool == DS4_HELP_SERVER) {
+        opt(fp, c, "-n, --tokens N", "Default max output tokens when clients omit a limit.");
+    }
+    opt(fp, c, "-t, --threads N", "CPU helper threads for host-side/reference work.");
+    opt(fp, c, "--power N", "GPU duty-cycle target, 1..100. Default: 100");
+    opt(fp, c, "--ssd-streaming", "Metal/CUDA/ROCm: opt in to SSD-backed model streaming instead of full residency.");
+    opt(fp, c, "--ssd-streaming-cold", "SSD streaming: skip default popularity-based expert-cache preload.");
+    opt(fp, c, "--ssd-streaming-cache-experts N|NGB", "SSD streaming: N is an exact dynamic expert count; NGB is a routed memory budget that also reserves two full prefill layers. Auto: 80% working set minus non-routed weights; GLM Metal caps lower.");
+    opt(fp, c, "--ssd-streaming-full-layers N", "GLM Metal streaming: keep the first N routed layers fully resident. Default: auto from NGB expert budget; use 0 to disable.");
+    opt(fp, c, "--ssd-streaming-preload-experts N", "SSD streaming: upfront popularity preload count. DeepSeek auto-seeds by default; GLM demand-fills unless N is explicit.");
+    opt(fp, c, "--simulate-used-memory NGB", "Diagnostic: lock N GiB before model load to simulate a smaller-memory machine.");
+    opt(fp, c, "--prefill-chunk N", "Metal graph prefill chunk size. Default: auto (PRO long prompts use 8192; others use 4096).");
+    if (full) {
+        if (tool != DS4_HELP_BENCH) {
+            opt(fp, c, "--mtp FILE", "Optional MTP support GGUF used for draft-token probes.");
+        }
+        if (tool == DS4_HELP_DS4 || tool == DS4_HELP_AGENT || tool == DS4_HELP_SERVER) {
+            opt(fp, c, "--mtp-draft N", "Maximum autoregressive MTP draft tokens. Default: 1");
+            opt(fp, c, "--mtp-margin F", "Verifier confidence margin for fast MTP acceptance. Default: 3");
+        }
+        opt(fp, c, "--quality", "Prefer exact kernels where faster approximate paths exist.");
+        opt(fp, c, "--warm-weights", "Touch mapped tensor pages at startup to reduce first-use stalls.");
+        if (tool == DS4_HELP_DS4 || tool == DS4_HELP_BENCH) {
+            opt(fp, c, "--expert-profile FILE", "Metal-only: write routed expert locality/cache simulation JSON.");
+        }
+    }
+    fputc('\n', fp);
+}
+
+static void print_sampling(FILE *fp, const help_colors *c, bool full) {
+    title(fp, c, "Prompt And Sampling");
+    opt(fp, c, "-n, --tokens N", "Maximum generated tokens.");
+    opt(fp, c, "--temp F", "Sampling temperature. 0 is greedy/deterministic.");
+    opt(fp, c, "--top-p F", "Nucleus sampling probability.");
+    opt(fp, c, "--min-p F", "Keep tokens scoring at least F times the top token.");
+    opt(fp, c, "--seed N", "Sampling seed for reproducible non-greedy runs.");
+    para(fp, c, "GLM CLI and agent runs default to temperature 1.0, top-p 0.95, and min-p 0 unless those options are set explicitly.");
+    opt(fp, c, "--think", "Use normal thinking mode.");
+    opt(fp, c, "--think-max", "Use Think Max when context is large enough.");
+    opt(fp, c, "--nothink", "Disable thinking and ask for direct replies.");
+    if (full) {
+        opt(fp, c, "-sys, --system TEXT", "System prompt. Empty string disables the default where supported.");
+        opt(fp, c, "-p, --prompt TEXT", "One-shot prompt text.");
+        opt(fp, c, "--prompt-file FILE", "Read one-shot prompt text from FILE.");
+        opt(fp, c, "--raw-prompt", "Tokenize the one-shot prompt without chat markers.");
+    }
+    fputc('\n', fp);
+}
+
+static void print_steering(FILE *fp, const help_colors *c) {
+    title(fp, c, "Directional Steering");
+    opt(fp, c, "--dir-steering-file FILE", "Load one f32 direction vector per layer.");
+    opt(fp, c, "--dir-steering-ffn F", "Apply steering after FFN outputs. Default with file: 1");
+    opt(fp, c, "--dir-steering-attn F", "Apply steering after attention outputs. Default: 0");
+    fputc('\n', fp);
+}
+
+static void print_distributed(FILE *fp, const help_colors *c) {
+    title(fp, c, "Distributed Inference");
+    fputc('\n', fp);
+    para(fp, c, "Distributed mode runs one logical session across several machines by assigning contiguous model layer ranges to workers. Workers own their layer slice and KV-cache shard; the coordinator owns the prompt, sampling loop, and client/API flow. Start workers first, then start the coordinator. The coordinator waits for a complete route and streams hidden states through the workers.");
+    fputc('\n', fp);
+    opt(fp, c, "--role ROLE", "Distributed role: coordinator or worker.");
+    opt(fp, c, "--layers A:B", "Inclusive layer slice, e.g. 0:20 or 21:output.");
+    opt(fp, c, "--listen HOST PORT", "Coordinator listen address; workers may use it for their data listener.");
+    opt(fp, c, "--coordinator HOST PORT", "Coordinator address for --role worker.");
+    opt(fp, c, "--dist-prefill-chunk N", "Coordinator prefill pipeline chunk size. Default: session cap.");
+    opt(fp, c, "--dist-prefill-window N", "Max prefill chunks in flight. Default: workers+2, capped at 8.");
+    opt(fp, c, "--dist-activation-bits N", "Hidden-state transport width: 32, 16, or 8. Default: 32");
+    opt(fp, c, "--dist-replay-check", "Diagnostic: reset and replay prompt, then compare logits.");
+    opt(fp, c, "--debug", "Print coordinator route/debug logs.");
+    fputc('\n', fp);
+}
+
+static void print_cli_diagnostics(FILE *fp, const help_colors *c);
+
+static void print_cli_specific(FILE *fp, const help_colors *c, bool full) {
+    title(fp, c, "CLI Modes");
+    opt(fp, c, "ds4", "Start the interactive prompt.");
+    opt(fp, c, "ds4 -p TEXT", "Run one prompt and exit.");
+    opt(fp, c, "ds4 --prompt-file FILE", "Run a long prompt from a file and exit.");
+    fputc('\n', fp);
+    if (full) {
+        print_cli_diagnostics(fp, c);
+    }
+}
+
+static void print_cli_diagnostics(FILE *fp, const help_colors *c) {
+    title(fp, c, "Diagnostics And Data Collection");
+    opt(fp, c, "--inspect", "Load the model and print a summary only.");
+    opt(fp, c, "--dump-tokens", "Tokenize the prompt exactly as written, then exit.");
+    opt(fp, c, "--dump-logits FILE", "Write full next-token logits as JSON.");
+    opt(fp, c, "--dump-logprobs FILE", "Write greedy continuation top-logprobs as JSON.");
+    opt(fp, c, "--logprobs-top-k N", "Alternatives stored by --dump-logprobs. Default: 20");
+    opt(fp, c, "--decode-consistency N", "Compare N-token decode logits with a fresh full prefill.");
+    opt(fp, c, "--expert-profile FILE", "Metal-only: write routed expert locality/cache simulation JSON.");
+    opt(fp, c, "--perplexity-file FILE", "Score raw text with teacher-forced NLL.");
+    opt(fp, c, "--imatrix-dataset FILE", "Rendered prompt dataset for imatrix collection.");
+    opt(fp, c, "--imatrix-out FILE", "Write llama-compatible routed-MoE imatrix .dat.");
+    opt(fp, c, "--imatrix-max-prompts N", "Stop imatrix collection after N prompts.");
+    opt(fp, c, "--imatrix-max-tokens N", "Stop imatrix collection after N prompt tokens.");
+    opt(fp, c, "--head-test", "Run the output HC/logits head after the native slice.");
+    opt(fp, c, "--first-token-test", "Run exact CPU whole-model pass for the first prompt token.");
+    opt(fp, c, "--metal-graph-test", "Compare first GPU-resident graph stages with CPU.");
+    opt(fp, c, "--metal-graph-full-test", "Run the GPU-resident self-token graph across all layers.");
+    opt(fp, c, "--metal-graph-prompt-test", "Compare CPU and GPU graph logits for the full prompt.");
+    fputc('\n', fp);
+}
+
+static void print_cli_commands(FILE *fp, const help_colors *c) {
+    title_red(fp, c, "Interactive Commands");
+    opt(fp, c, "/help", "Show interactive commands.");
+    opt(fp, c, "/think, /think-max, /nothink", "Switch thinking mode.");
+    opt(fp, c, "/ctx N", "Restart the interactive session with a new context size.");
+    opt(fp, c, "/power N", "Set GPU duty cycle percentage, 1..100.");
+    opt(fp, c, "/read FILE", "Read FILE and submit it as the next user message.");
+    opt(fp, c, "/quit, /exit", "Leave the prompt.");
+    opt(fp, c, "Ctrl+C", "Stop current generation and return to ds4>.");
+    fputc('\n', fp);
+}
+
+static void print_agent_specific(FILE *fp, const help_colors *c) {
+    title(fp, c, "Agent Options");
+    opt(fp, c, "-p, --prompt TEXT", "Submit an initial prompt after startup.");
+    opt(fp, c, "--non-interactive", "Run without TUI. With -p: one turn; without -p: repeated stdin prompts.");
+    opt(fp, c, "--raw-prompt", "Non-interactive -p only: tokenize prompt without agent chat/tool text.");
+    opt(fp, c, "-sys, --system TEXT", "Extra system prompt. Empty disables extra text.");
+    opt(fp, c, "--trace FILE", "Write prompt, token, and DSML debug trace.");
+    opt(fp, c, "--chdir DIR", "Change working directory before loading runtime assets.");
+    fputc('\n', fp);
+}
+
+static void print_agent_sessions(FILE *fp, const help_colors *c) {
+    title(fp, c, "Agent Runtime Commands");
+    opt(fp, c, "/save", "Save the current session in ~/.ds4/kvcache.");
+    opt(fp, c, "/compact", "Compact the current session context now.");
+    opt(fp, c, "/list", "List saved sessions, sorted by recent update time.");
+    opt(fp, c, "/switch ID", "Load a saved session and show recent history.");
+    opt(fp, c, "/del ID", "Delete a saved session.");
+    opt(fp, c, "/strip ID", "Remove KV payload; the text history can be rebuilt later.");
+    opt(fp, c, "/history [N]", "Show N recent user turns from the current session.");
+    opt(fp, c, "/power N", "Set GPU duty cycle percentage, 1..100.");
+    opt(fp, c, "/new", "Start a fresh session from the system prompt.");
+    opt(fp, c, "/quit, /exit", "Exit.");
+    fputc('\n', fp);
+}
+
+static void print_server_api(FILE *fp, const help_colors *c) {
+    title(fp, c, "HTTP API");
+    opt(fp, c, "--host HOST", "Bind address. Default: 127.0.0.1");
+    opt(fp, c, "--port N", "Bind port. Default: 8000");
+    opt(fp, c, "--cors", "Add Access-Control-Allow-* headers for browser JS clients.");
+    opt(fp, c, "--trace FILE", "Write prompts, cache decisions, output, and tool calls.");
+    para(fp, c, "Endpoints: /v1/chat/completions, /v1/responses, /v1/completions, and /v1/messages.");
+    para(fp, c, "Model endpoint aliases include deepseek-v4-flash and deepseek-v4-pro; both serve the loaded GGUF.");
+    fputc('\n', fp);
+}
+
+static void print_server_thinking(FILE *fp, const help_colors *c) {
+    title(fp, c, "Server Thinking Defaults");
+    para(fp, c, "DeepSeek-compatible chat requests default to high-effort thinking.");
+    para(fp, c, "reasoning_effort=max or output_config.effort=max requests Think Max.");
+    para(fp, c, "Think Max requires --ctx >= 393216; smaller contexts use high.");
+    para(fp, c, "thinking={type:disabled}, think=false, or model=deepseek-chat selects non-thinking mode.");
+    para(fp, c, "In thinking mode, client sampling knobs are ignored like the official API.");
+    fputc('\n', fp);
+}
+
+static void print_kv_cache(FILE *fp, const help_colors *c) {
+    title(fp, c, "Disk KV Cache");
+    opt(fp, c, "--kv-disk-dir DIR", "Enable disk KV checkpoints in DIR.");
+    opt(fp, c, "--kv-disk-space-mb N", "Disk budget. Default when enabled: 4096");
+    opt(fp, c, "--kv-cache-min-tokens N", "Do not save/load checkpoints shorter than N. Default: 512");
+    opt(fp, c, "--kv-cache-cold-max-tokens N", "Save cold first prompts up to N tokens. 0 disables. Default: 30000");
+    opt(fp, c, "--kv-cache-continued-interval-tokens N", "Save aligned continued frontiers. 0 disables. Default: 10000");
+    opt(fp, c, "--kv-cache-boundary-trim-tokens N", "Trim tail tokens for cold boundary saves. Default: 32");
+    opt(fp, c, "--kv-cache-boundary-align-tokens N", "Align cold boundary saves to this multiple. Default: 2048");
+    opt(fp, c, "--kv-cache-reject-different-quant", "Reject checkpoints written with different routed-expert quantization.");
+    opt(fp, c, "--disable-exact-dsml-tool-replay", "Disable exact sampled DSML tool replay map.");
+    opt(fp, c, "--tool-memory-max-ids N", "Exact tool-call IDs kept in RAM. Default: 100000");
+    fputc('\n', fp);
+}
+
+static void print_bench_specific(FILE *fp, const help_colors *c) {
+    title(fp, c, "Benchmark Input");
+    opt(fp, c, "--prompt-file FILE", "Raw benchmark text; token sequence is sliced at each frontier.");
+    opt(fp, c, "--chat-prompt-file FILE", "Render FILE as one no-thinking chat user message.");
+    opt(fp, c, "-sys, --system TEXT", "System prompt used only with --chat-prompt-file.");
+    fputc('\n', fp);
+    title(fp, c, "Benchmark Sweep");
+    opt(fp, c, "--ctx-start N", "First measured frontier. Default: 2048");
+    opt(fp, c, "--ctx-max N", "Last measured frontier. Default: 32768");
+    opt(fp, c, "--ctx-alloc N", "Allocated context. Default: ctx-max + gen-tokens + 1");
+    opt(fp, c, "--step-mul F", "Multiplicative step. Default: 1");
+    opt(fp, c, "--step-incr N", "Linear step when --step-mul is 1. Default: 2048");
+    opt(fp, c, "--gen-tokens N", "Greedy decode tokens per frontier. 0 for pure prefill. Default: 128");
+    opt(fp, c, "--csv FILE", "Write CSV there instead of stdout.");
+    opt(fp, c, "--dump-frontier-logits-dir DIR", "Write one full-logit JSON file per frontier.");
+    fputc('\n', fp);
+}
+
+static void print_eval_specific(FILE *fp, const help_colors *c) {
+    title(fp, c, "Evaluation");
+    opt(fp, c, "-n, --tokens N", "Max generated tokens per question. Default: 16000");
+    opt(fp, c, "--questions N", "Run only the first N embedded questions.");
+    opt(fp, c, "--case-sequence LIST", "Run 1-based case numbers in this comma-separated order.");
+    opt(fp, c, "--trace FILE", "Write questions, outputs, and grading decisions.");
+    opt(fp, c, "--regrade-trace FILE", "Regrade a prior trace without loading the model.");
+    opt(fp, c, "--soft-limit-reply-budget N", "Soft close thinking near the end of reply budget. Default: 1024");
+    opt(fp, c, "--hard-limit-reply-budget N", "Force </think> with N tokens left. Default: 512");
+    opt(fp, c, "--soft-limit-think-close-rank N", "Soft-close when </think> is in top N tokens. Default: 3");
+    opt(fp, c, "--pause-ms N", "Pause after each result in the TTY UI. Default: 350");
+    opt(fp, c, "--plain", "Disable split-screen ANSI UI.");
+    opt(fp, c, "--self-test-extractors", "Run answer-extractor self-tests and exit.");
+    fputc('\n', fp);
+}
+
+static bool tool_has_topic(ds4_help_tool tool, const char *topic) {
+    if (!topic) return true;
+    if (streq(topic, "all")) return true;
+    if (streq(topic, "runtime") || streq(topic, "distributed")) return true;
+    if (streq(topic, "sampling"))
+        return tool == DS4_HELP_DS4 || tool == DS4_HELP_AGENT || tool == DS4_HELP_EVAL;
+    if (streq(topic, "steering"))
+        return tool == DS4_HELP_DS4 || tool == DS4_HELP_SERVER || tool == DS4_HELP_AGENT;
+    switch (tool) {
+    case DS4_HELP_DS4:
+        return streq(topic, "diagnostics") || streq(topic, "commands");
+    case DS4_HELP_SERVER:
+        return streq(topic, "api") || streq(topic, "kv-cache") || streq(topic, "thinking");
+    case DS4_HELP_AGENT:
+        return streq(topic, "sessions") || streq(topic, "commands") || streq(topic, "tools");
+    case DS4_HELP_BENCH:
+        return streq(topic, "benchmark");
+    case DS4_HELP_EVAL:
+        return streq(topic, "evaluation");
+    }
+    return false;
+}
+
+static void more_line(FILE *fp, const help_colors *c, const char *label, const char *topic) {
+    static const char *colors[] = {
+        "\x1b[38;5;81m", "\x1b[38;5;114m", "\x1b[38;5;179m",
+        "\x1b[38;5;141m", "\x1b[38;5;147m"
+    };
+    static size_t idx;
+    const char *on = c->cyan ? colors[idx++ % (sizeof(colors) / sizeof(colors[0]))] : "";
+    if (streq(label, "Interactive commands:") && c->red) on = c->red;
+    const char *off = c->off ? c->off : "";
+    fprintf(fp, "    %s%-26s%s --help %s\n", on, label, off, topic);
+}
+
+static void print_more_info(FILE *fp, const help_colors *c, ds4_help_tool tool) {
+    title(fp, c, "More Info");
+    more_line(fp, c, "Runtime full info:", "runtime");
+    if (tool_has_topic(tool, "sampling"))
+        more_line(fp, c, "Sampling full info:", "sampling");
+    more_line(fp, c, "Distributed inference:", "distributed");
+    if (tool_has_topic(tool, "steering"))
+        more_line(fp, c, "Steering full info:", "steering");
+    if (tool == DS4_HELP_DS4) {
+        more_line(fp, c, "Interactive commands:", "commands");
+        more_line(fp, c, "Diagnostics:", "diagnostics");
+    } else if (tool == DS4_HELP_SERVER) {
+        more_line(fp, c, "HTTP API:", "api");
+        more_line(fp, c, "Disk KV cache:", "kv-cache");
+        more_line(fp, c, "Thinking behavior:", "thinking");
+    } else if (tool == DS4_HELP_AGENT) {
+        more_line(fp, c, "Agent sessions:", "sessions");
+        more_line(fp, c, "Agent commands:", "commands");
+        more_line(fp, c, "Agent tool system:", "tools");
+    } else if (tool == DS4_HELP_BENCH) {
+        more_line(fp, c, "Benchmark sweep:", "benchmark");
+    } else if (tool == DS4_HELP_EVAL) {
+        more_line(fp, c, "Evaluation options:", "evaluation");
+    }
+    fputc('\n', fp);
+}
+
+static void print_examples(FILE *fp, const help_colors *c, ds4_help_tool tool, const char *topic) {
+    title(fp, c, "Examples");
+    if (topic_is(topic, "distributed")) {
+        opt(fp, c, "worker", "./ds4 --role worker --layers 21:output --coordinator 192.168.0.181 9000 -m ds4flash.gguf");
+        opt(fp, c, "coordinator", "./ds4 --role coordinator --layers 0:20 --listen 0.0.0.0 9000 -p \"Hello\" -m ds4flash.gguf");
+    } else if (topic_is(topic, "runtime")) {
+        if (tool == DS4_HELP_SERVER) {
+            opt(fp, c, "Metal API", "./ds4-server -m ds4flash.gguf --metal --ctx 100000");
+            opt(fp, c, "quiet API", "./ds4-server --power 60 --host 127.0.0.1 --port 8000");
+        } else if (tool == DS4_HELP_AGENT) {
+            opt(fp, c, "agent", "./ds4-agent -m ds4flash.gguf --ctx 100000");
+            opt(fp, c, "quiet agent", "./ds4-agent --power 50");
+        } else if (tool == DS4_HELP_BENCH) {
+            opt(fp, c, "bench", "./ds4-bench --prompt-file long.txt --ctx-max 32768");
+            opt(fp, c, "quiet bench", "./ds4-bench --prompt-file long.txt --power 70");
+        } else if (tool == DS4_HELP_EVAL) {
+            opt(fp, c, "eval", "./ds4-eval --questions 10 --ctx 100000");
+            opt(fp, c, "CPU debug", "./ds4-eval --cpu --questions 1 --tokens 32");
+        } else {
+            opt(fp, c, "Metal", "./ds4 -m ds4flash.gguf --metal -c 100000");
+            opt(fp, c, "quiet thermals", "./ds4 -p \"Summarize README\" --power 50");
+        }
+    } else if (topic_is(topic, "steering")) {
+        opt(fp, c, "steer FFN", "./ds4 -p \"Write tersely\" --dir-steering-file dir.bin --dir-steering-ffn 0.8");
+    } else if (tool == DS4_HELP_SERVER || topic_is(topic, "api") || topic_is(topic, "kv-cache")) {
+        opt(fp, c, "local API", "./ds4-server --ctx 100000 --kv-disk-dir ~/.ds4/server-kv --kv-disk-space-mb 8192");
+        opt(fp, c, "curl", "curl http://127.0.0.1:8000/v1/models");
+    } else if (tool == DS4_HELP_AGENT || topic_is(topic, "sessions") || topic_is(topic, "tools")) {
+        opt(fp, c, "interactive", "./ds4-agent");
+        opt(fp, c, "one shot", "./ds4-agent --non-interactive -p \"Create /tmp/hello.c\"");
+    } else if (tool == DS4_HELP_BENCH || topic_is(topic, "benchmark")) {
+        opt(fp, c, "csv", "./ds4-bench --prompt-file long.txt --ctx-max 32768 --csv speed.csv");
+        opt(fp, c, "prefill only", "./ds4-bench --prompt-file long.txt --gen-tokens 0");
+    } else if (tool == DS4_HELP_EVAL || topic_is(topic, "evaluation")) {
+        opt(fp, c, "first 10", "./ds4-eval --questions 10 --trace eval.trace");
+        opt(fp, c, "plain", "./ds4-eval --plain --nothink --tokens 512");
+    } else {
+        opt(fp, c, "chat", "./ds4");
+        opt(fp, c, "one shot", "./ds4 -p \"Explain mmap in C\"");
+        opt(fp, c, "long prompt", "./ds4 --think-max --prompt-file prompt.txt --ctx 393216");
+    }
+    fputc('\n', fp);
+}
+
+static void print_topic(FILE *fp, const help_colors *c, ds4_help_tool tool, const char *topic) {
+    if (streq(topic, "all")) {
+        print_model_runtime(fp, c, tool, true);
+        if (tool_has_topic(tool, "sampling")) print_sampling(fp, c, true);
+        if (tool_has_topic(tool, "steering")) print_steering(fp, c);
+        print_distributed(fp, c);
+        if (tool == DS4_HELP_DS4) {
+            print_cli_specific(fp, c, true);
+            print_cli_commands(fp, c);
+        } else if (tool == DS4_HELP_SERVER) {
+            print_server_api(fp, c);
+            print_server_thinking(fp, c);
+            print_kv_cache(fp, c);
+        } else if (tool == DS4_HELP_AGENT) {
+            print_agent_specific(fp, c);
+            print_agent_sessions(fp, c);
+        } else if (tool == DS4_HELP_BENCH) {
+            print_bench_specific(fp, c);
+        } else if (tool == DS4_HELP_EVAL) {
+            print_eval_specific(fp, c);
+        }
+        return;
+    }
+
+    if (streq(topic, "runtime")) print_model_runtime(fp, c, tool, true);
+    else if (streq(topic, "sampling")) print_sampling(fp, c, true);
+    else if (streq(topic, "steering")) print_steering(fp, c);
+    else if (streq(topic, "distributed")) print_distributed(fp, c);
+    else if (tool == DS4_HELP_DS4 && streq(topic, "diagnostics")) print_cli_diagnostics(fp, c);
+    else if (tool == DS4_HELP_DS4 && streq(topic, "commands")) print_cli_commands(fp, c);
+    else if (tool == DS4_HELP_SERVER && streq(topic, "api")) print_server_api(fp, c);
+    else if (tool == DS4_HELP_SERVER && streq(topic, "kv-cache")) print_kv_cache(fp, c);
+    else if (tool == DS4_HELP_SERVER && streq(topic, "thinking")) print_server_thinking(fp, c);
+    else if (tool == DS4_HELP_AGENT && streq(topic, "sessions")) print_agent_sessions(fp, c);
+    else if (tool == DS4_HELP_AGENT && streq(topic, "commands")) print_agent_sessions(fp, c);
+    else if (tool == DS4_HELP_AGENT && streq(topic, "tools")) {
+        title(fp, c, "Agent Tool System");
+        para(fp, c, "The agent can read, search, write, edit, run bash, and browse through Chrome-backed web tools.");
+        para(fp, c, "DeepSeek-family models emit DSML tool calls; GLM models use native <tool_call> syntax. Both are rendered live in the terminal.");
+        para(fp, c, "Edit uses exact old/new replacement; [upto] can bridge a unique head and tail for large anchored edits.");
+        fputc('\n', fp);
+    } else if (tool == DS4_HELP_BENCH && streq(topic, "benchmark")) print_bench_specific(fp, c);
+    else if (tool == DS4_HELP_EVAL && streq(topic, "evaluation")) print_eval_specific(fp, c);
+}
+
+static void print_default(FILE *fp, const help_colors *c, ds4_help_tool tool) {
+    print_model_runtime(fp, c, tool, false);
+
+    if (tool == DS4_HELP_DS4) {
+        print_cli_specific(fp, c, true);
+        print_sampling(fp, c, false);
+    } else if (tool == DS4_HELP_SERVER) {
+        print_server_api(fp, c);
+        print_kv_cache(fp, c);
+    } else if (tool == DS4_HELP_AGENT) {
+        print_agent_specific(fp, c);
+        print_agent_sessions(fp, c);
+    } else if (tool == DS4_HELP_BENCH) {
+        print_bench_specific(fp, c);
+    } else if (tool == DS4_HELP_EVAL) {
+        print_eval_specific(fp, c);
+    }
+}
+
+void ds4_help_print(FILE *fp, ds4_help_tool tool, const char *topic) {
+    help_colors c = help_make_colors(fp);
+    if (topic && !tool_has_topic(tool, topic)) {
+        fprintf(fp, "%s: unknown help topic '%s'\n\n", tool_name(tool), topic);
+        topic = NULL;
+    }
+
+    fprintf(fp, "%s%s%s\n", c.bright ? c.bright : "", tool_name(tool), c.off ? c.off : "");
+    fprintf(fp, "%s\n\n", tool_summary(tool));
+    fprintf(fp, "%s\n\n", tool_usage(tool));
+
+    if (topic) print_topic(fp, &c, tool, topic);
+    else {
+        print_default(fp, &c, tool);
+        print_more_info(fp, &c, tool);
+    }
+    print_examples(fp, &c, tool, topic);
+}

ds4_help.h

@@ -0,0 +1,16 @@
+#ifndef DS4_HELP_H
+#define DS4_HELP_H
+
+#include <stdio.h>
+
+typedef enum {
+    DS4_HELP_DS4,
+    DS4_HELP_SERVER,
+    DS4_HELP_AGENT,
+    DS4_HELP_BENCH,
+    DS4_HELP_EVAL,
+} ds4_help_tool;
+
+void ds4_help_print(FILE *fp, ds4_help_tool tool, const char *topic);
+
+#endif

ds4_kvstore.h

@@ -0,0 +1,218 @@
+#ifndef DS4_KVSTORE_H
+#define DS4_KVSTORE_H
+
+#include "ds4.h"
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+
+#define DS4_KVSTORE_FIXED_HEADER 48u
+#define DS4_KVSTORE_DEFAULT_MB 4096
+#define DS4_KVSTORE_HIT_HALF_LIFE_SECONDS (6ull * 60ull * 60ull)
+
+#define DS4_KVSTORE_EXT_TOOL_MAP          (1u << 0)
+#define DS4_KVSTORE_EXT_RESPONSES_VISIBLE (1u << 1)
+#define DS4_KVSTORE_EXT_THINKING_VISIBLE  (1u << 2)
+#define DS4_KVSTORE_EXT_SESSION_TITLE     (1u << 3)
+
+typedef enum {
+    DS4_KVSTORE_REASON_UNKNOWN   = 0,
+    DS4_KVSTORE_REASON_COLD      = 1,
+    DS4_KVSTORE_REASON_CONTINUED = 2,
+    DS4_KVSTORE_REASON_EVICT     = 3,
+    DS4_KVSTORE_REASON_SHUTDOWN  = 4,
+    DS4_KVSTORE_REASON_AGENT_SYSTEM  = 5,
+    DS4_KVSTORE_REASON_AGENT_SESSION = 6,
+} ds4_kvstore_reason;
+
+typedef enum {
+    DS4_KVSTORE_LOG_DEFAULT,
+    DS4_KVSTORE_LOG_KVCACHE,
+    DS4_KVSTORE_LOG_WARNING,
+} ds4_kvstore_log_type;
+
+typedef struct {
+    /* The file name is the rendered byte prefix, not the token sequence. The
+     * payload still carries the exact tokens and graph state; the hash only
+     * answers "does this checkpoint represent the bytes at the front of the
+     * incoming prompt?" */
+    char sha[41];
+    char *path;
+    uint8_t quant_bits;
+    /* Stored in header byte 7.  Flash is 0 for backward compatibility with
+     * older cache files where this reserved byte was always written as zero. */
+    uint8_t model_id;
+    uint8_t reason;
+    uint32_t tokens;
+    uint32_t hits;
+    uint32_t ctx_size;
+    uint8_t ext_flags;
+    uint64_t created_at;
+    uint64_t last_used;
+    uint64_t payload_bytes;
+    uint64_t text_bytes;
+    uint64_t file_size;
+} ds4_kvstore_entry;
+
+typedef struct {
+    int min_tokens;
+    int cold_max_tokens;
+    int continued_interval_tokens;
+    int boundary_trim_tokens;
+    int boundary_align_tokens;
+} ds4_kvstore_options;
+
+typedef struct {
+    bool enabled;
+    char *dir;
+    uint64_t budget_bytes;
+    bool reject_different_quant;
+    ds4_kvstore_options opt;
+    int continued_last_store_tokens;
+    ds4_kvstore_entry *entry;
+    int len;
+    int cap;
+    const char *log_name;
+    void *log_ud;
+    void (*log)(void *ud, ds4_kvstore_log_type type, const char *msg);
+} ds4_kvstore;
+
+typedef struct {
+    const char *text;
+    size_t text_len;
+    uint8_t model_id;
+    uint8_t quant_bits;
+    uint32_t ctx_size;
+    bool reject_different_quant;
+} ds4_kvstore_eviction_context;
+
+typedef struct {
+    void *ud;
+    uint8_t ext_flag;
+    bool (*serialized_size)(void *ud, const char *text, uint64_t *bytes_out);
+    bool (*write)(void *ud, FILE *fp, const char *text, uint64_t *written_bytes);
+    int (*load)(void *ud, FILE *fp, const void *wanted);
+    const void *load_wanted;
+} ds4_kvstore_trailer_hooks;
+
+typedef struct {
+    int tokens;
+    uint32_t text_bytes;
+    uint8_t quant_bits;
+    uint8_t ext_flags;
+    double load_ms;
+    bool consumed;
+    char *path;
+} ds4_kvstore_load_result;
+
+ds4_kvstore_options ds4_kvstore_default_options(void);
+uint8_t ds4_kvstore_reason_code(const char *reason);
+const char *ds4_kvstore_key_kind(uint8_t ext_flags);
+
+bool ds4_kvstore_open(ds4_kvstore *kc, const char *dir, uint64_t budget_mb,
+                      bool reject_different_quant, ds4_kvstore_options opt,
+                      const char *log_name,
+                      void (*log)(void *ud, ds4_kvstore_log_type type, const char *msg),
+                      void *log_ud);
+void ds4_kvstore_close(ds4_kvstore *kc);
+void ds4_kvstore_clear(ds4_kvstore *kc);
+void ds4_kvstore_entry_free(ds4_kvstore_entry *e);
+
+char *ds4_kvstore_render_tokens_text(ds4_engine *engine,
+                                     const ds4_tokens *tokens,
+                                     size_t *out_len);
+bool ds4_kvstore_byte_prefix_match(const char *text, size_t text_len,
+                                   const char *prefix, size_t prefix_len);
+void ds4_kvstore_tokens_copy_prefix(ds4_tokens *dst, const ds4_tokens *src, int n);
+void ds4_kvstore_build_prompt_from_exact_prefix_and_text_suffix(
+        ds4_engine *engine,
+        const ds4_tokens *exact_prefix,
+        const char *suffix_text,
+        ds4_tokens *out);
+
+int ds4_kvstore_store_len(const ds4_kvstore *kc, int tokens);
+int ds4_kvstore_chat_anchor_pos(const ds4_kvstore *kc,
+                                const ds4_tokens *prompt,
+                                int user_token_id,
+                                int assistant_token_id);
+int ds4_kvstore_continued_store_target(const ds4_kvstore *kc, int live_tokens);
+void ds4_kvstore_note_store(ds4_kvstore *kc, int tokens);
+int ds4_kvstore_suppress_continued_store(ds4_kvstore *kc, int tokens);
+void ds4_kvstore_restore_suppressed_continued(ds4_kvstore *kc,
+                                              int old_tokens,
+                                              int suppressed_tokens);
+
+bool ds4_kvstore_file_size_fits(const ds4_kvstore *kc,
+                                uint64_t text_bytes,
+                                uint64_t payload_bytes,
+                                uint64_t trailer_bytes,
+                                uint64_t *file_bytes_out,
+                                uint64_t *required_bytes_out);
+double ds4_kvstore_entry_eviction_score(const ds4_kvstore_entry *e,
+                                        const ds4_tokens *live,
+                                        uint64_t now,
+                                        const ds4_kvstore_eviction_context *incoming);
+void ds4_kvstore_evict(ds4_kvstore *kc, const ds4_tokens *live,
+                       uint64_t extra_bytes,
+                       const ds4_kvstore_eviction_context *incoming);
+int ds4_kvstore_find_text_prefix(ds4_kvstore *kc, const char *prompt_text,
+                                 int model_id, int quant_bits, int ctx_size);
+
+bool ds4_kvstore_store_live_prefix_text(ds4_kvstore *kc,
+                                        ds4_engine *engine,
+                                        ds4_session *session,
+                                        const ds4_tokens *tokens,
+                                        int store_len,
+                                        const char *reason,
+                                        const char *cache_text_override,
+                                        uint8_t cache_text_ext,
+                                        const char *cache_text_key,
+                                        const ds4_kvstore_trailer_hooks *hooks,
+                                        char *err,
+                                        size_t err_len);
+bool ds4_kvstore_store_live_prefix(ds4_kvstore *kc,
+                                   ds4_engine *engine,
+                                   ds4_session *session,
+                                   const ds4_tokens *tokens,
+                                   int store_len,
+                                   const char *reason,
+                                   const ds4_kvstore_trailer_hooks *hooks,
+                                   char *err,
+                                   size_t err_len);
+bool ds4_kvstore_maybe_store_continued(ds4_kvstore *kc,
+                                       ds4_engine *engine,
+                                       ds4_session *session,
+                                       const ds4_kvstore_trailer_hooks *hooks,
+                                       char *err,
+                                       size_t err_len);
+int ds4_kvstore_try_load_text(ds4_kvstore *kc,
+                              ds4_engine *engine,
+                              ds4_session *session,
+                              const char *prompt_text,
+                              ds4_tokens *effective_prompt,
+                              ds4_kvstore_load_result *result,
+                              const ds4_kvstore_trailer_hooks *hooks,
+                              bool responses_protocol);
+void ds4_kvstore_load_result_free(ds4_kvstore_load_result *result);
+
+bool ds4_kvstore_read_header(FILE *fp, ds4_kvstore_entry *e,
+                             uint32_t *text_bytes);
+bool ds4_kvstore_read_entry_file(const char *path, const char sha[41],
+                                 ds4_kvstore_entry *out);
+void ds4_kvstore_fill_header(uint8_t h[DS4_KVSTORE_FIXED_HEADER],
+                             uint8_t model_id, uint8_t quant_bits,
+                             uint8_t reason, uint8_t ext_flags,
+                             uint32_t tokens, uint32_t hits, uint32_t ctx_size,
+                             uint64_t created_at, uint64_t last_used,
+                             uint64_t payload_bytes);
+bool ds4_kvstore_touch_file(const char *path, uint32_t hits);
+bool ds4_kvstore_sha_hex_name(const char *name, char sha[41]);
+void ds4_kvstore_sha1_bytes_hex(const void *ptr, size_t len, char out[41]);
+char *ds4_kvstore_path_join(const char *dir, const char *name);
+char *ds4_kvstore_path_for_sha(ds4_kvstore *kc, const char sha[41]);
+void ds4_kvstore_le_put32(uint8_t *p, uint32_t v);
+uint32_t ds4_kvstore_le_get32(const uint8_t *p);
+
+#endif

ds4_rocm.cu

@@ -0,0 +1,131 @@
+#ifdef __HIP_PLATFORM_AMD__
+#include "ds4_rocm.h"
+#include <hipblaslt/hipblaslt.h>
+
+#define FULL_WARP_MASK 0xFFFFFFFFFFFFFFFFULL
+#define MASK_T uint64_t
+#define DS4_GPU_BACKEND_NAME "ROCm"
+#define DS4_GPU_LOG_PREFIX "ds4: ROCm "
+#define DS4_GPU_BLAS_NAME "hipBLAS"
+#else
+#include <cuda_runtime.h>
+#include <cuda_fp16.h>
+#include <mma.h>
+#include <cublas_v2.h>
+#include <cub/block/block_radix_sort.cuh>
+
+#define FULL_WARP_MASK 0xFFFFFFFFu
+#define MASK_T uint32_t
+#define DS4_GPU_BACKEND_NAME "CUDA"
+#define DS4_GPU_LOG_PREFIX "ds4: CUDA "
+#define DS4_GPU_BLAS_NAME "cuBLAS"
+#endif
+
+#include <stdint.h>
+#include <ctype.h>
+#include <errno.h>
+#include <limits.h>
+#include <math.h>
+#include <fcntl.h>
+#include <pthread.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <time.h>
+#include <unistd.h>
+#include <unordered_map>
+#include <vector>
+
+#include "ds4_gpu.h"
+
+#ifndef M_PI
+#define M_PI 3.14159265358979323846
+#endif
+
+#define CUDA_QK_K 256
+#define DS4_ROCM_UNUSED __attribute__((unused))
+
+enum {
+    /* attention_decode_mixed_kernel stores raw-window scores plus visible
+     * compressed scores in shared memory.  The host routes larger unmasked
+     * decode calls to the online attention kernel so this fixed buffer never
+     * becomes an out-of-bounds write at long context. */
+    DS4_ROCM_ATTENTION_SCORE_CAP = 8192u,
+    DS4_ROCM_ATTENTION_RAW_SCORE_CAP = 256u,
+    DS4_ROCM_TOPK_MERGE_GROUP = 8u
+};
+
+struct ds4_gpu_tensor {
+    void *ptr;
+    uint64_t bytes;
+    int owner;
+};
+
+typedef struct {
+    uint8_t scales[CUDA_QK_K / 16];
+    uint8_t qs[CUDA_QK_K / 4];
+    uint16_t d;
+    uint16_t dmin;
+} cuda_block_q2_K;
+
+typedef struct {
+    uint16_t d;
+    uint16_t dmin;
+    uint8_t scales[12];
+    uint8_t qs[CUDA_QK_K / 2];
+} cuda_block_q4_K;
+
+typedef struct {
+    float d;
+    int8_t qs[CUDA_QK_K];
+    int16_t bsums[CUDA_QK_K / 16];
+} cuda_block_q8_K;
+
+typedef struct {
+    uint16_t d;
+    uint16_t qs[CUDA_QK_K / 8];
+} cuda_block_iq2_xxs;
+
+#include "ds4_iq2_tables_cuda.inc"
+
+#include "rocm/ds4_rocm_runtime.cuh"
+
+#include "rocm/ds4_rocm_common.cuh"
+
+#include "rocm/ds4_rocm_q8.cuh"
+
+#include "rocm/ds4_rocm_norm_rope.cuh"
+
+#include "rocm/ds4_rocm_fp8_kv.cuh"
+
+#include "rocm/ds4_rocm_attention.cuh"
+
+#include "rocm/ds4_rocm_hc.cuh"
+
+#include "rocm/ds4_rocm_output.cuh"
+
+#include "rocm/ds4_rocm_indexer.cuh"
+
+#include "rocm/ds4_rocm_embedding_launch.cuh"
+
+#include "rocm/ds4_rocm_matmul.cuh"
+
+#include "rocm/ds4_rocm_fp8_kv_launch.cuh"
+
+#include "rocm/ds4_rocm_compressor.cuh"
+
+#include "rocm/ds4_rocm_attention_launch.cuh"
+
+#include "rocm/ds4_rocm_shared_expert.cuh"
+
+#include "rocm/ds4_rocm_misc_launch.cuh"
+#include "rocm/ds4_rocm_router.cuh"
+
+#include "rocm/ds4_rocm_moe.cuh"
+
+#include "rocm/ds4_rocm_moe_launch.cuh"
+
+#include "rocm/ds4_rocm_hc_output_launch.cuh"
+
+#include "rocm/ds4_rocm_current_api_compat.cuh"

ds4_rocm.h

@@ -0,0 +1,142 @@
+#pragma once
+
+#include <hip/hip_runtime.h>
+#include <hipblas/hipblas.h>
+#include <hip/hip_fp16.h>
+#include <hipcub/hipcub.hpp>
+#include <rocwmma/rocwmma-version.hpp>
+#include <rocwmma/rocwmma.hpp>
+
+#define cudaError_t hipError_t
+#define cudaStream_t hipStream_t
+#define cudaEvent_t hipEvent_t
+#define cudaDeviceProp hipDeviceProp_t
+#define cudaMemLocation hipMemLocation
+
+#define cudaSuccess hipSuccess
+#define cudaErrorNotSupported hipErrorNotSupported
+#define cudaErrorInvalidValue hipErrorInvalidValue
+#define cudaGetLastError hipGetLastError
+#define cudaGetErrorString hipGetErrorString
+
+#define cudaGetDevice hipGetDevice
+#define cudaSetDevice hipSetDevice
+#define cudaDeviceSynchronize hipDeviceSynchronize
+#define cudaDeviceGetAttribute hipDeviceGetAttribute
+#define cudaGetDeviceProperties hipGetDeviceProperties
+#define cudaDevAttrPageableMemoryAccess hipDeviceAttributePageableMemoryAccess
+#define cudaDevAttrMaxSharedMemoryPerBlockOptin hipDeviceAttributeSharedMemPerBlockOptin
+#define cudaFuncAttributeMaxDynamicSharedMemorySize hipFuncAttributeMaxDynamicSharedMemorySize
+#define cudaFuncSetAttribute(func, attr, value) hipFuncSetAttribute((const void *)(func), (attr), (value))
+#define cudaMemLocationTypeDevice hipMemLocationTypeDevice
+
+#define cudaMalloc hipMalloc
+#define cudaMallocHost hipHostMalloc
+#define cudaMallocManaged hipMallocManaged
+#define cudaFree hipFree
+#define cudaFreeHost hipFreeHost
+#define cudaMemset hipMemset
+#define cudaMemcpy hipMemcpy
+#define cudaMemcpyAsync hipMemcpyAsync
+#define cudaMemcpyHostToDevice hipMemcpyHostToDevice
+#define cudaMemcpyDeviceToHost hipMemcpyDeviceToHost
+#define cudaMemcpyDeviceToDevice hipMemcpyDeviceToDevice
+#define cudaMemGetInfo hipMemGetInfo
+#define cudaMemsetAsync hipMemsetAsync
+
+#define cudaHostRegister hipHostRegister
+#define cudaHostUnregister hipHostUnregister
+#define cudaHostGetDevicePointer hipHostGetDevicePointer
+#define cudaHostRegisterMapped hipHostRegisterMapped
+#define cudaHostRegisterReadOnly hipHostRegisterReadOnly
+
+#define cudaMemAdvise(p1, p2, p3, p4) hipMemAdvise(p1, p2, p3, p4.id)
+#define cudaMemPrefetchAsync(devPtr, count, location, flags, stream) hipMemPrefetchAsync(devPtr, count, location.id, stream)
+#define cudaMemAdviseSetReadMostly hipMemAdviseSetReadMostly
+#define cudaMemAdviseSetPreferredLocation hipMemAdviseSetPreferredLocation
+
+#define cudaStreamCreateWithFlags hipStreamCreateWithFlags
+#define cudaStreamSynchronize hipStreamSynchronize
+#define cudaStreamDestroy hipStreamDestroy
+#define cudaStreamNonBlocking hipStreamNonBlocking
+
+#define cudaEventCreate hipEventCreate
+#define cudaEventCreateWithFlags hipEventCreateWithFlags
+#define cudaEventDestroy hipEventDestroy
+#define cudaEventRecord hipEventRecord
+#define cudaEventSynchronize hipEventSynchronize
+#define cudaEventElapsedTime hipEventElapsedTime
+#define cudaEventDisableTiming hipEventDisableTiming
+
+#define cublasHandle_t hipblasHandle_t
+#define cublasStatus_t hipblasStatus_t
+#define cublasMath_t hipblasMath_t
+
+#define CUBLAS_STATUS_SUCCESS HIPBLAS_STATUS_SUCCESS
+#define CUBLAS_OP_N HIPBLAS_OP_N
+#define CUBLAS_OP_T HIPBLAS_OP_T
+#define CUBLAS_GEMM_DEFAULT HIPBLAS_GEMM_DEFAULT
+#define CUBLAS_DEFAULT_MATH HIPBLAS_DEFAULT_MATH
+#define CUBLAS_COMPUTE_32F HIPBLAS_COMPUTE_32F
+#define CUBLAS_TF32_TENSOR_OP_MATH HIPBLAS_TF32_TENSOR_OP_MATH
+#define CUDA_R_16F HIPBLAS_R_16F
+#define CUDA_R_32F HIPBLAS_R_32F
+
+#define cublasCreate hipblasCreate
+#define cublasDestroy hipblasDestroy
+#define cublasSetMathMode hipblasSetMathMode
+#define cublasSgemm hipblasSgemm
+#define cublasSgemmStridedBatched hipblasSgemmStridedBatched
+#define cublasGemmEx hipblasGemmEx
+#define cublasGemmStridedBatchedEx hipblasGemmStridedBatchedEx
+
+namespace cub = hipcub;
+
+static __device__ __forceinline__ int32_t __vcmpne4(uint32_t a, uint32_t b) {
+    // For each byte: 0xFF if a != b, 0x00 if a == b
+    uint32_t diff = a ^ b;
+    // Spread any set bit in each byte to fill the whole byte
+    diff |= (diff >> 1); diff |= (diff >> 2); diff |= (diff >> 4);
+    diff &= 0x01010101u;
+    diff *= 0xFFu; // 0x01 -> 0xFF per byte
+    return (int32_t)diff;
+}
+
+static __device__ __forceinline__ int32_t __vsub4(int32_t a, int32_t b) {
+    // Per-byte subtraction (wrapping, not saturating)
+    uint32_t ua = (uint32_t)a, ub = (uint32_t)b;
+    // Trick: subtract bytes in parallel avoiding cross-byte borrows
+    uint32_t diff = ((ua | 0x80808080u) - (ub & 0x7F7F7F7Fu)) ^ ((ua ^ ~ub) & 0x80808080u);
+    return (int32_t)diff;
+}
+
+// __dp4a: dot product of 4 signed int8s packed in an int32.
+// gfx11-class AMD GPUs expose this as a single v_dot4_i32_i8 instruction;
+// using the clang builtin avoids expanding every Q8/Q8_K dot into scalar byte
+// multiplies in the ROCm compatibility layer.
+static __device__ __forceinline__ int32_t __dp4a(int32_t a, int32_t b, int32_t c) {
+#if defined(__HIP_PLATFORM_AMD__) || defined(__HIPCC__)
+    union ds4_i8x4_bits { int32_t i; char4 v; } av, bv;
+    av.i = a;
+    bv.i = b;
+    return amd_mixed_dot(av.v, bv.v, c, false);
+#else
+    const int8_t *a_bytes = reinterpret_cast<const int8_t*>(&a);
+    const int8_t *b_bytes = reinterpret_cast<const int8_t*>(&b);
+    return c + (int32_t)a_bytes[0] * b_bytes[0]
+             + (int32_t)a_bytes[1] * b_bytes[1]
+             + (int32_t)a_bytes[2] * b_bytes[2]
+             + (int32_t)a_bytes[3] * b_bytes[3];
+#endif
+}
+
+// Precise transcendentals for the MoE router top-k scores, immune to -fapprox-func.
+// These functions are to be used on paths where small error can be translated to
+// some macro effect - like expert selection kernels
+extern "C" __device__ __attribute__((pure))  float __ocml_exp_f32(float);
+extern "C" __device__ __attribute__((pure))  float __ocml_log1p_f32(float);
+extern "C" __device__ __attribute__((const)) float __ocml_sqrt_f32(float);
+
+static __device__ __forceinline__ float ds4_precise_expf(float x)   { return __ocml_exp_f32(x); }
+static __device__ __forceinline__ float ds4_precise_log1pf(float x) { return __ocml_log1p_f32(x); }
+static __device__ __forceinline__ float ds4_precise_sqrtf(float x)  { return __ocml_sqrt_f32(x); }

ds4_ssd.c

@@ -0,0 +1,181 @@
+#include "ds4_ssd.h"
+
+#include <ctype.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+#ifndef MAP_ANONYMOUS
+#define MAP_ANONYMOUS MAP_ANON
+#endif
+
+static const uint64_t DS4_GIB = 1024ull * 1024ull * 1024ull;
+
+bool ds4_parse_gib_arg(const char *s, uint64_t *bytes) {
+    if (bytes) *bytes = 0;
+    if (!s || !s[0] || !bytes) return false;
+
+    size_t len = strlen(s);
+    if (len > 2 &&
+        (s[len - 2] == 'g' || s[len - 2] == 'G') &&
+        (s[len - 1] == 'b' || s[len - 1] == 'B')) {
+        len -= 2;
+    }
+    if (len == 0) return false;
+    for (size_t i = 0; i < len; i++) {
+        if (!isdigit((unsigned char)s[i])) return false;
+    }
+
+    char numbuf[32];
+    if (len >= sizeof(numbuf)) return false;
+    memcpy(numbuf, s, len);
+    numbuf[len] = '\0';
+
+    errno = 0;
+    unsigned long long v = strtoull(numbuf, NULL, 10);
+    if (errno != 0 || v == 0 || v > UINT64_MAX / DS4_GIB) return false;
+
+    *bytes = (uint64_t)v * DS4_GIB;
+    return true;
+}
+
+bool ds4_parse_streaming_cache_experts_arg(const char *s,
+                                           uint32_t   *experts,
+                                           uint64_t   *bytes) {
+    if (experts) *experts = 0;
+    if (bytes) *bytes = 0;
+    if (!s || !s[0] || !experts || !bytes) return false;
+
+    const size_t len = strlen(s);
+    if (len > 2 &&
+        (s[len - 2] == 'g' || s[len - 2] == 'G') &&
+        (s[len - 1] == 'b' || s[len - 1] == 'B')) {
+        return ds4_parse_gib_arg(s, bytes);
+    }
+
+    for (size_t i = 0; i < len; i++) {
+        if (!isdigit((unsigned char)s[i])) return false;
+    }
+
+    errno = 0;
+    unsigned long v = strtoul(s, NULL, 10);
+    if (errno != 0 || v == 0 || v > UINT32_MAX) return false;
+
+    *experts = (uint32_t)v;
+    return true;
+}
+
+uint32_t ds4_ssd_cache_experts_for_byte_budget(uint64_t bytes,
+                                               uint64_t per_expert_bytes) {
+    if (bytes == 0 || per_expert_bytes == 0) return 0;
+    const uint64_t experts = bytes / per_expert_bytes;
+    if (experts == 0 || experts > UINT32_MAX) return 0;
+    return (uint32_t)experts;
+}
+
+bool ds4_ssd_auto_cache_plan(uint64_t            recommended_bytes,
+                             uint64_t            non_routed_bytes,
+                             uint64_t            per_expert_bytes,
+                             uint64_t            max_model_experts,
+                             ds4_ssd_cache_plan *out) {
+    if (!out) return false;
+    memset(out, 0, sizeof(*out));
+    if (recommended_bytes == 0 || per_expert_bytes == 0) return false;
+
+    out->model_target_bytes =
+        recommended_bytes > UINT64_MAX / 4ull ?
+            UINT64_MAX : (recommended_bytes * 4ull) / 5ull;
+    if (out->model_target_bytes > non_routed_bytes) {
+        out->cache_bytes = out->model_target_bytes - non_routed_bytes;
+    }
+
+    uint64_t cache_experts = out->cache_bytes / per_expert_bytes;
+    if (cache_experts == 0) cache_experts = 1;
+    if (max_model_experts != 0 && cache_experts > max_model_experts) {
+        cache_experts = max_model_experts;
+    }
+    if (cache_experts > UINT32_MAX) cache_experts = UINT32_MAX;
+
+    out->cache_experts = (uint32_t)cache_experts;
+    out->effective_cache_bytes = cache_experts * per_expert_bytes;
+    return out->cache_experts != 0;
+}
+
+bool ds4_ssd_memory_lock_acquire(ds4_ssd_memory_lock *lock,
+                                 uint64_t             bytes) {
+    if (!lock) return false;
+    lock->ptr = NULL;
+    lock->bytes = 0;
+    if (bytes == 0) return true;
+    if (bytes > (uint64_t)SIZE_MAX) {
+        fprintf(stderr,
+                "ds4: --simulate-used-memory is too large for this process\n");
+        return false;
+    }
+
+    void *ptr = mmap(NULL,
+                     (size_t)bytes,
+                     PROT_READ | PROT_WRITE,
+                     MAP_PRIVATE | MAP_ANONYMOUS,
+                     -1,
+                     0);
+    if (ptr == MAP_FAILED) {
+        fprintf(stderr,
+                "ds4: --simulate-used-memory mmap %.2f GiB failed: %s\n",
+                (double)bytes / (double)DS4_GIB,
+                strerror(errno));
+        return false;
+    }
+
+    const long page_long = sysconf(_SC_PAGESIZE);
+    const uint64_t page = page_long > 0 ? (uint64_t)page_long : 4096ull;
+    const uint64_t chunk_bytes = 256ull * 1024ull * 1024ull;
+    volatile unsigned char *p = (volatile unsigned char *)ptr;
+
+    /*
+     * Touch and lock in bounded chunks.  A single very large mlock() is harder
+     * to diagnose when it fails and can create long uninterruptible VM work on
+     * macOS; chunking mirrors the standalone diagnostic utility.
+     */
+    uint64_t locked = 0;
+    for (uint64_t off = 0; off < bytes; off += chunk_bytes) {
+        uint64_t len = bytes - off;
+        if (len > chunk_bytes) len = chunk_bytes;
+
+        for (uint64_t pos = off; pos < off + len; pos += page) {
+            p[pos] = (unsigned char)(pos / page);
+        }
+        if (len != 0) p[off + len - 1u] = 1;
+
+        if (mlock((void *)(p + off), (size_t)len) != 0) {
+            fprintf(stderr,
+                    "ds4: --simulate-used-memory mlock failed after %.2f/%.2f GiB: %s\n",
+                    (double)locked / (double)DS4_GIB,
+                    (double)bytes / (double)DS4_GIB,
+                    strerror(errno));
+            if (locked != 0) munlock(ptr, (size_t)locked);
+            munmap(ptr, (size_t)bytes);
+            return false;
+        }
+        locked += len;
+    }
+
+    lock->ptr = ptr;
+    lock->bytes = bytes;
+    fprintf(stderr,
+            "ds4: simulated used memory: locked %.2f GiB before model load\n",
+            (double)bytes / (double)DS4_GIB);
+    return true;
+}
+
+void ds4_ssd_memory_lock_release(ds4_ssd_memory_lock *lock) {
+    if (!lock || !lock->ptr || lock->bytes == 0) return;
+    munlock(lock->ptr, (size_t)lock->bytes);
+    munmap(lock->ptr, (size_t)lock->bytes);
+    lock->ptr = NULL;
+    lock->bytes = 0;
+}

ds4_ssd.h

@@ -0,0 +1,36 @@
+#ifndef DS4_SSD_H
+#define DS4_SSD_H
+
+#include <stdbool.h>
+#include <stdint.h>
+
+typedef struct {
+    void *ptr;
+    uint64_t bytes;
+} ds4_ssd_memory_lock;
+
+typedef struct {
+    uint64_t model_target_bytes;
+    uint64_t cache_bytes;
+    uint64_t effective_cache_bytes;
+    uint32_t cache_experts;
+} ds4_ssd_cache_plan;
+
+bool ds4_parse_gib_arg(const char *s, uint64_t *bytes);
+bool ds4_parse_streaming_cache_experts_arg(const char *s,
+                                           uint32_t   *experts,
+                                           uint64_t   *bytes);
+
+uint32_t ds4_ssd_cache_experts_for_byte_budget(uint64_t bytes,
+                                               uint64_t per_expert_bytes);
+bool ds4_ssd_auto_cache_plan(uint64_t            recommended_bytes,
+                             uint64_t            non_routed_bytes,
+                             uint64_t            per_expert_bytes,
+                             uint64_t            max_model_experts,
+                             ds4_ssd_cache_plan *out);
+
+bool ds4_ssd_memory_lock_acquire(ds4_ssd_memory_lock *lock,
+                                 uint64_t             bytes);
+void ds4_ssd_memory_lock_release(ds4_ssd_memory_lock *lock);
+
+#endif

ds4_web.h

@@ -0,0 +1,33 @@
+#ifndef DS4_WEB_H
+#define DS4_WEB_H
+
+#include <stddef.h>
+#include <stdbool.h>
+
+typedef int (*ds4_web_confirm_fn)(void *privdata, const char *message,
+                                  char *err, size_t err_len);
+typedef void (*ds4_web_log_fn)(void *privdata, const char *message);
+typedef bool (*ds4_web_cancel_fn)(void *privdata);
+
+typedef struct {
+    const char *home_dir;
+    int port;
+    ds4_web_confirm_fn confirm;
+    void *confirm_privdata;
+    ds4_web_log_fn log;
+    void *log_privdata;
+    ds4_web_cancel_fn cancel;
+    void *cancel_privdata;
+} ds4_web_config;
+
+typedef struct ds4_web ds4_web;
+
+ds4_web *ds4_web_create(const ds4_web_config *cfg);
+void ds4_web_free(ds4_web *web);
+
+char *ds4_web_google_search(ds4_web *web, const char *query,
+                            char *err, size_t err_len);
+char *ds4_web_visit_page(ds4_web *web, const char *url,
+                         char *err, size_t err_len);
+
+#endif

gguf-tools/.gitignore

@@ -1,3 +1,11 @@
 deepseek4-quantize
 quality-testing/score_official
-quality-testing/data/
+quality-testing/score_llama
+quality-testing/data/*
+!quality-testing/data/README.md
+!quality-testing/data/glm52-openrouter-100/
+!quality-testing/data/glm52-openrouter-100/**
+!quality-testing/data/pro/
+!quality-testing/data/pro/**
+!quality-testing/data/flash/
+!quality-testing/data/flash/**

gguf-tools/Makefile

@@ -1,12 +1,21 @@
 CC ?= cc
+CXX ?= c++
 UNAME_S := $(shell uname -s)
+LLAMA_CPP_DIR ?= ../../deepseek-v4-quants/llama.cpp
+LLAMA_CPP_BUILD ?= $(LLAMA_CPP_DIR)/build-remote
+LLAMA_CPP_BIN ?= $(LLAMA_CPP_BUILD)/bin
+LLAMA_CPP_CXXFLAGS ?= -O3 -Wall -Wextra -std=c++17 \
+	-I$(LLAMA_CPP_DIR)/include -I$(LLAMA_CPP_DIR)/ggml/include
 
 ifeq ($(UNAME_S),Darwin)
 NATIVE_CPU_FLAG ?= -mcpu=native
 QUALITY_LDLIBS := -lm -pthread -framework Foundation -framework Metal
-QUALITY_TARGETS := ds4.o ds4_metal.o
-QUALITY_OBJS := ../ds4.o ../ds4_metal.o
+QUALITY_TARGETS := ds4.o ds4_metal.o ds4_distributed.o ds4_ssd.o rax.o
+QUALITY_OBJS := ../ds4.o ../ds4_metal.o ../ds4_distributed.o ../ds4_ssd.o ../rax.o
 QUALITY_LINK := $(CC) $(CFLAGS) -I.. -o
+LLAMA_CPP_LDLIBS ?= -L$(LLAMA_CPP_BIN) -Wl,-rpath,$(abspath $(LLAMA_CPP_BIN)) \
+	-lllama -lggml -lggml-base -lggml-cpu -lggml-blas -lggml-metal \
+	-framework Foundation -framework Metal -framework Accelerate
 else
 NATIVE_CPU_FLAG ?= -march=native
 CUDA_HOME ?= /usr/local/cuda
@@ -17,15 +26,17 @@ NVCC_ARCH_FLAGS := -arch=$(CUDA_ARCH)
 endif
 NVCCFLAGS ?= -O3 --use_fast_math $(NVCC_ARCH_FLAGS) -Xcompiler $(NATIVE_CPU_FLAG) -Xcompiler -pthread
 QUALITY_LDLIBS ?= -lm -Xcompiler -pthread -L$(CUDA_HOME)/targets/sbsa-linux/lib -L$(CUDA_HOME)/lib64 -lcudart -lcublas
-QUALITY_TARGETS := ds4.o ds4_cuda.o
-QUALITY_OBJS := ../ds4.o ../ds4_cuda.o
+QUALITY_TARGETS := ds4.o ds4_cuda.o ds4_distributed.o ds4_ssd.o rax.o
+QUALITY_OBJS := ../ds4.o ../ds4_cuda.o ../ds4_distributed.o ../ds4_ssd.o ../rax.o
 QUALITY_LINK := $(NVCC) $(NVCCFLAGS) -I.. -o
+LLAMA_CPP_LDLIBS ?= -L$(LLAMA_CPP_BIN) -Wl,-rpath,$(abspath $(LLAMA_CPP_BIN)) \
+	-lllama -lggml -lggml-base -lggml-cpu -ldl -lpthread
 endif
 
 CFLAGS ?= -O3 -Wall -Wextra -std=c11 $(NATIVE_CPU_FLAG)
 CPPFLAGS ?= -D_GNU_SOURCE
 
-.PHONY: all clean quality-score
+.PHONY: all clean quality-score quality-llama-score
 
 all: deepseek4-quantize
 
@@ -36,5 +47,8 @@ quality-score:
 	$(MAKE) -C .. $(QUALITY_TARGETS)
 	$(QUALITY_LINK) quality-testing/score_official quality-testing/score_official.c $(QUALITY_OBJS) $(QUALITY_LDLIBS)
 
+quality-llama-score:
+	$(CXX) $(LLAMA_CPP_CXXFLAGS) -o quality-testing/score_llama quality-testing/score_llama.cpp $(LLAMA_CPP_LDLIBS)
+
 clean:
-	rm -f deepseek4-quantize quality-testing/score_official
+	rm -f deepseek4-quantize quality-testing/score_official quality-testing/score_llama

gguf-tools/deepseek4-quantize.c

@@ -1,8 +1,9 @@
 /*
- * DeepSeek V4 Flash HF -> GGUF quantizer.
+ * DeepSeek V4 Flash/Pro HF -> GGUF quantizer.
  *
  * This is a plain C, model-specific version of the DS4 quantization pipeline.
- * It deliberately keeps only the pieces needed by DeepSeek V4 Flash:
+ * It deliberately keeps only the pieces needed by the DeepSeek V4 Flash and
+ * Pro GGUF recipes used by this repository:
  *
  * - safetensors index/header loading;
  * - FP8 E4M3 + E8M0 dequantization for dense tensors;
@@ -27,6 +28,7 @@
 #include <ctype.h>
 #include <errno.h>
 #include <inttypes.h>
+#include <limits.h>
 #include <math.h>
 #include <stdbool.h>
 #include <stdint.h>
@@ -1323,6 +1325,7 @@ typedef struct {
     uint8_t *kv_raw;
     size_t kv_raw_len;
     size_t alignment;
+    int n_experts;
     size_t data_offset;
     tensor_meta *tensors;
     hmap tensor_map;
@@ -1478,6 +1481,12 @@ static gguf_file load_gguf_metadata(const char *path) {
         if (strcmp(key, "general.alignment") == 0 && type == GGUF_TYPE_UINT32) {
             uint32_t a = read_u32_le_fp(fp, "GGUF alignment");
             if (a) g.alignment = a;
+        } else if (strcmp(key, "deepseek4.expert_count") == 0 && type == GGUF_TYPE_UINT32) {
+            uint32_t n = read_u32_le_fp(fp, "GGUF expert count");
+            if (n <= (uint32_t)INT_MAX) g.n_experts = (int)n;
+        } else if (strcmp(key, "deepseek4.expert_count") == 0 && type == GGUF_TYPE_UINT64) {
+            uint64_t n = read_u64_le_fp(fp, "GGUF expert count");
+            if (n <= (uint64_t)INT_MAX) g.n_experts = (int)n;
         } else {
             skip_gguf_value_fp(fp, type);
         }
@@ -1686,7 +1695,7 @@ typedef struct {
 
 static void usage(const char *argv0) {
     printf("usage: %s --hf DIR --template MODEL.gguf --out OUT.gguf [options]\n", argv0);
-    printf("\nDeepSeek V4 Flash safetensors -> GGUF quantizer in plain C.\n\n");
+    printf("\nDeepSeek V4 Flash/Pro safetensors -> GGUF quantizer in plain C.\n\n");
     printf("options:\n");
     printf("  --hf DIR               Hugging Face model directory with model.safetensors.index.json\n");
     printf("  --template FILE        existing DS4 GGUF used for metadata, tensor order, shapes\n");
@@ -1708,7 +1717,7 @@ static void usage(const char *argv0) {
     printf("  --output TYPE          output.* tensor type\n");
     printf("  --dense TYPE           remaining 2D+ non-routed tensor type\n");
     printf("  --tensor-type PFX=TYPE exact tensor-name or prefix override; may repeat\n");
-    printf("  --n-experts N          routed expert count, default 256\n");
+    printf("  --n-experts N          routed expert count, default template metadata\n");
     printf("  --threads N            expert worker count, default 8\n");
     printf("\nTYPE examples: f16, f32, bf16, q8_0, q4_k, q2_k, iq2_xxs\n");
 }
@@ -1733,7 +1742,7 @@ static params parse_args(int argc, char **argv) {
     p.policy.routed_w1 = p.policy.routed_w2 = p.policy.routed_w3 = DS4Q_TYPE_COUNT;
     p.policy.attention_proj = p.policy.attention = p.policy.shared = DS4Q_TYPE_COUNT;
     p.policy.embedding = p.policy.output = p.policy.dense = DS4Q_TYPE_COUNT;
-    p.n_experts = 256;
+    p.n_experts = 0;
     p.n_threads = 8;
 
     for (int i = 1; i < argc; i++) {
@@ -1861,6 +1870,17 @@ int main(int argc, char **argv) {
     if (p.imatrix_file) imatrix_load(&imatrix, p.imatrix_file, p.imatrix_strict);
 
     gguf_file tmpl = load_gguf_metadata(p.template_gguf);
+    if (p.n_experts <= 0) {
+        if (tmpl.n_experts > 0) {
+            p.n_experts = tmpl.n_experts;
+            fprintf(stderr, "using %d routed experts from template metadata\n", p.n_experts);
+        } else {
+            p.n_experts = 256;
+            fprintf(stderr, "warning: template has no deepseek4.expert_count; using Flash default %d routed experts\n", p.n_experts);
+        }
+    } else {
+        fprintf(stderr, "using %d routed experts from --n-experts\n", p.n_experts);
+    }
     output_context out_ctx = build_output_context(&tmpl, &p.policy, &imatrix);
     print_plan(&tmpl, &out_ctx);
     if (p.dry_run) return 0;

gguf-tools/imatrix/README.md

@@ -1,10 +1,12 @@
 # DS4 Imatrix Pipeline
 
 This directory contains the calibration dataset and instructions used to build
-activation importance matrices for DeepSeek V4 Flash GGUF quantization.
+activation importance matrices for DeepSeek V4 Flash and Pro GGUF
+quantization.
 
-The current imatrix target is the routed MoE path.  DS4 has 43 layers, 256
-routed experts per layer, and three routed expert tensors per layer:
+The current imatrix target is the routed MoE path.  Flash has 43 layers and
+256 routed experts per layer.  Pro has 61 layers and 384 routed experts per
+layer.  Both variants expose three routed expert tensors per layer:
 
 - `blk.N.ffn_gate_exps.weight`
 - `blk.N.ffn_up_exps.weight`
@@ -36,17 +38,22 @@ It contains DS4-rendered chat prompts, separated by visible
 - C/Metal source-review prompts from this repository.
 - Long-context snippets.
 - Agent/tool-call prompts using DS4's DSML syntax.
-- English and Italian prompts.
+- Language/prose rewriting, summarization, extraction, and translation prompts.
+- `ds4-eval` GPQA Diamond, SuperGPQA, and AIME2025 benchmark prompts.
 - Both thinking and non-thinking assistant prefixes.
 
-The current tracked dataset has 1952 rendered prompts and roughly 1.44M tokens
+The current tracked dataset has 4682 rendered prompts and roughly 2.91M tokens
 by the coarse bytes/4 estimate.  Check
 `gguf-tools/imatrix/dataset/manifest.json` for the exact generated-file
 summary.
 
 ## 2. Collect The Imatrix
 
-Use the DS4 runtime itself to collect routed MoE activation statistics:
+Use the DS4 runtime itself to collect routed MoE activation statistics.  The
+collector uses the loaded GGUF metadata, so the same command shape works for
+Flash and Pro.
+
+Flash example:
 
 ```sh
 ./ds4 \
@@ -56,6 +63,18 @@ Use the DS4 runtime itself to collect routed MoE activation statistics:
   --ctx 32768
 ```
 
+Pro example with a smaller calibration budget:
+
+```sh
+./ds4 \
+  -m ../deepseek-v4-quants/gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct.gguf \
+  --imatrix-dataset gguf-tools/imatrix/dataset/rendered_prompts.txt \
+  --imatrix-out ../deepseek-v4-quants/imatrix/DeepSeek-V4-Pro-Instruct-routed-moe-ds4-small.dat \
+  --imatrix-max-prompts 16 \
+  --imatrix-max-tokens 32768 \
+  --ctx 32768
+```
+
 Useful smoke-test limits:
 
 ```sh
@@ -79,6 +98,8 @@ entry length = n_expert * n_columns
 ```
 
 The quantizer slices the right expert's segment when quantizing each expert.
+For Pro, that means each routed tensor entry contains 384 vectors instead of
+Flash's 256.
 
 ## 3. Generate GGUF Files With The Imatrix
 
@@ -109,6 +130,21 @@ gguf-tools/deepseek4-quantize \
   --imatrix ../deepseek-v4-quants/imatrix/DeepSeek-V4-Flash-chat-v2-routed-moe-ds4-1p5m.dat
 ```
 
+Example Pro Q2 regeneration from a Pro template:
+
+```sh
+gguf-tools/deepseek4-quantize \
+  --hf ../deepseek-v4-quants/hf/DeepSeek-V4-Pro \
+  --template ../deepseek-v4-quants/gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct.gguf \
+  --out ../deepseek-v4-quants/gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix-small.gguf \
+  --imatrix ../deepseek-v4-quants/imatrix/DeepSeek-V4-Pro-Instruct-routed-moe-ds4-small.dat \
+  --threads 24
+```
+
+`deepseek4-quantize` reads the routed expert count from the template GGUF.  Use
+`--n-experts` only when working with an older template that lacks
+`deepseek4.expert_count`.
+
 For Q4, the imatrix does not change the runtime tensor type: routed experts
 remain `Q4_K`.  It changes how quantization error is weighted while choosing
 scales and codes.  For Q2, it replaces the previous synthetic weight-energy

gguf-tools/imatrix/dataset/README.md

@@ -26,11 +26,12 @@ The renderer mirrors the server prompt shape:
 
 Some records include DSML tool schemas, sampled DSML tool calls, and tool-result
 turns so the imatrix sees the same special-token patterns used by agent clients.
-The corpus is provider-neutral and also includes multi-language programming
-prompts, Bash scripting, algorithm recall, English-to-European-language
-translation, English-to-Chinese translation, reverse translation to English,
-long-context source excerpts, Metal/C code review tasks, and
-inference-specific debugging tasks.
+The corpus is provider-neutral and also includes language/prose rewriting,
+summarization, copy-editing, extraction, multilingual translation, programming
+prompts, Bash scripting, algorithm recall, `ds4-eval` benchmark-reasoning
+prompts, long-context code synthesis, agent transcript replay, log diagnosis,
+prose fact recovery, delayed-constraint and small needle tasks, Metal/C code
+review tasks, and inference-specific debugging tasks.
 
 For normal imatrix collection, use `rendered_prompts.txt` so calibration covers
 both thinking and non-thinking modes. Split files are provided for ablations.

gguf-tools/imatrix/dataset/build_ds4_imatrix_dataset.py

@@ -3,8 +3,8 @@
 
 The quantizer needs activation statistics from prompts that look like the real
 workload.  This script creates deterministic DS4-rendered chat prompts from the
-repo itself, agent/tool conversations, multilingual translation tasks,
-programming prompts, and long-context code reviews.  The output is
+repo itself, agent/tool conversations, language and translation tasks,
+benchmark-reasoning prompts, programming prompts, and long-context code reviews.  The output is
 intentionally plain JSONL/text so the imatrix collector can consume it without
 depending on this script.
 
@@ -18,6 +18,7 @@ provider-neutral rather than tuned around a single client brand.
 from __future__ import annotations
 
 import argparse
+import ast
 import hashlib
 import json
 import random
@@ -99,13 +100,9 @@ class Record:
 
 
 def escape_tool_result(text: str) -> str:
-    """Match ``append_dsml_text_escaped`` for tool result bodies."""
+    """Match ``append_tool_result_text`` for tool result bodies."""
 
-    return (
-        text.replace("&", "&")
-        .replace("<", "&lt;")
-        .replace(">", "&gt;")
-    )
+    return text.replace("</tool_result>", "&lt;/tool_result>")
 
 
 def escape_dsml_parameter(text: str) -> str:
@@ -234,11 +231,22 @@ def stable_id(category: str, source: str, mode: str, text: str) -> str:
     return f"{category}-{h.hexdigest()[:12]}"
 
 
+def normalize_rendered_text(text: str) -> str:
+    """Keep generated calibration files free of accidental trailing spaces.
+
+    Some imported benchmark questions include LaTeX/TikZ lines with trailing
+    blanks.  Those bytes do not carry useful calibration signal, and removing
+    them avoids churny generated files and whitespace-check failures.
+    """
+
+    return "\n".join(line.rstrip() for line in text.split("\n"))
+
+
 def add_record(records: list[Record], category: str, source: str,
                messages: list[dict], *, tools_schema: list[dict] | None = None,
                modes: Iterable[str] = ("nothink", "think")) -> None:
     for mode in modes:
-        rendered = render(messages, mode, tools_schema=tools_schema)
+        rendered = normalize_rendered_text(render(messages, mode, tools_schema=tools_schema))
         rid = stable_id(category, source, mode, rendered)
         records.append(Record(rid, category, mode, source, messages, rendered))
 
@@ -482,22 +490,792 @@ def make_translation_records(records: list[Record]) -> None:
         ])
 
 
+LANGUAGE_PASSAGES = [
+    (
+        "harbor-library",
+        "At seven each morning, Mira unlocked the little harbor library before "
+        "the fish market opened. The room smelled of salt, paper, and wet rope. "
+        "Retired captains came in to read newspapers, apprentices borrowed repair "
+        "manuals, and schoolchildren used the atlas to trace routes they hoped to "
+        "sail one day. When a storm damaged the pier, the library became the only "
+        "quiet place with a working radio, a kettle, and a table large enough for "
+        "the volunteers to spread their maps.",
+    ),
+    (
+        "city-heat",
+        "The city planted two thousand street trees after a summer in which the "
+        "hospital recorded an unusual rise in heat exhaustion. Sensors showed "
+        "that shaded sidewalks were often eight degrees cooler than exposed ones. "
+        "Shop owners first worried that construction would block their windows, "
+        "but many later said people lingered longer outside their stores. The "
+        "largest challenge was not planting the trees; it was watering them "
+        "during their first three dry seasons.",
+    ),
+    (
+        "museum-letter",
+        "The museum received a letter from a visitor who had recognized her "
+        "grandmother in the background of a 1934 photograph. Until then, the "
+        "archive listed the woman only as an unnamed station worker. The visitor "
+        "sent a copy of a diary page, a payroll stub, and a recipe card written "
+        "in the same hand. The curator updated the catalog entry, then wrote a "
+        "short exhibit label about how public memory can repair an incomplete "
+        "institutional record.",
+    ),
+    (
+        "clinic-schedule",
+        "A rural clinic changed its appointment schedule after noticing that "
+        "patients who worked in the fields missed morning visits twice as often "
+        "as afternoon ones. The new calendar reserved early slots for school "
+        "vaccinations and late slots for adults who could not leave work before "
+        "sunset. Missed appointments fell within a month, but the nurses also "
+        "found that reminder calls mattered most when they came from someone "
+        "who spoke the patient's first language.",
+    ),
+    (
+        "mountain-train",
+        "The mountain train was built to carry ore, not tourists. Its narrow "
+        "tracks followed the river, crossed three timber bridges, and climbed "
+        "through tunnels cut by hand. After the mine closed, the town nearly "
+        "removed the line, but teachers argued it was the clearest way to show "
+        "students how geography shaped local industry. Today the train runs "
+        "only on weekends, slowly enough for passengers to see the old signal "
+        "posts and the terraces where workers once kept vegetable gardens.",
+    ),
+    (
+        "software-release",
+        "The release note sounded simple: reduce startup time by warming fewer "
+        "buffers. In practice the change required a careful audit of which pages "
+        "were touched by the first request, which buffers were only needed after "
+        "a tool call, and which allocations could be delayed without moving a "
+        "latency spike into the user's first visible token. The team kept the "
+        "patch small, but the test plan became longer than the code.",
+    ),
+    (
+        "school-garden",
+        "The school garden began as a science project and became a language "
+        "project by accident. Students labeled the herbs in Arabic, Italian, "
+        "Mandarin, Spanish, and English because those were the languages spoken "
+        "at home. Parents started leaving recipes in a notebook by the gate. By "
+        "spring, the teacher used the garden to discuss migration, measurement, "
+        "soil chemistry, and why a word can carry a memory as well as a meaning.",
+    ),
+    (
+        "warehouse-robot",
+        "The warehouse robot failed only on rainy days. Engineers first blamed "
+        "the wheel encoders, then the floor markers, then the routing software. "
+        "A night-shift operator finally noticed that wet umbrellas left small "
+        "puddles near the entrance, and the robot slowed down whenever its front "
+        "sensor saw reflected ceiling lights. The fix was a mat, a hood over the "
+        "sensor, and a dashboard note that credited the operator by name.",
+    ),
+    (
+        "river-council",
+        "At the council meeting, the argument was not whether the river should "
+        "be cleaned, but how to measure success. Anglers wanted fish counts, "
+        "parents wanted safer banks, farmers wanted predictable irrigation, and "
+        "the budget office wanted milestones it could audit. The final plan used "
+        "four measures and a public map updated every quarter, which made the "
+        "project slower to announce but easier to defend.",
+    ),
+    (
+        "bakery-queue",
+        "The bakery changed its queue after the owner realized that customers "
+        "waiting for coffee blocked the bread shelves. A painted line on the "
+        "floor helped for a week, then failed during the Saturday rush. The best "
+        "solution was less decorative: move the sugar, cups, and lids away from "
+        "the register. Average waiting time fell, and the staff stopped having "
+        "to apologize for a problem that had never been caused by impatience.",
+    ),
+    (
+        "radio-weather",
+        "The weather bulletin was read in three languages because the fishing "
+        "boats did not all share the same crew language. The broadcaster learned "
+        "to keep numbers in the same order every time: wind direction, wind "
+        "speed, wave height, visibility, and warnings. Crews said the repeated "
+        "structure helped even when reception was poor, because a missing word "
+        "could often be inferred from its position in the sequence.",
+    ),
+    (
+        "archive-index",
+        "When the archive digitized its index cards, the scanner captured every "
+        "coffee stain and pencil mark. Some volunteers wanted to clean the "
+        "images automatically, but the archivist objected. The stains sometimes "
+        "showed which boxes had been handled during a flood, and the pencil "
+        "marks showed corrections made before the typed catalog existed. The "
+        "project therefore stored both a clean transcription and the original "
+        "image side by side.",
+    ),
+    (
+        "neighborhood-map",
+        "A neighborhood map drawn by residents looked inaccurate to professional "
+        "surveyors because distances were distorted. The bus stop, pharmacy, "
+        "laundromat, and playground appeared much larger than the courthouse or "
+        "bank. The planner kept the distortion because it revealed what official "
+        "maps often hide: the city experienced by walking with a stroller, "
+        "carrying groceries, or looking for shade is not the same city measured "
+        "from above.",
+    ),
+    (
+        "repair-cafe",
+        "At the repair cafe, people brought lamps, radios, torn jackets, and a "
+        "clock that had stopped after forty years. The volunteers did not promise "
+        "success. They promised to explain what they were trying, label the parts "
+        "they removed, and say when replacement would cost more than repair. The "
+        "most popular table was not electronics or sewing, but the one where "
+        "children took apart broken toys to see how gears and springs worked.",
+    ),
+    (
+        "forest-trail",
+        "The forest trail reopened with fewer signs than before. Rangers had "
+        "learned that visitors ignored crowded notice boards but noticed a single "
+        "clear instruction at the exact point where a decision was needed. The "
+        "new signs named the next landmark, the distance, and one safety warning. "
+        "Complaints about being lost dropped, and the trail looked less like a "
+        "corridor of rules.",
+    ),
+    (
+        "market-receipt",
+        "A market cooperative printed receipts that showed the farmer's name, "
+        "the distance traveled, and the date of harvest. Some customers found it "
+        "charming; others found it distracting. Sales data later showed that the "
+        "extra information mattered most for items people could not inspect "
+        "directly, such as flour, oil, and honey. For tomatoes and peaches, smell "
+        "still outweighed every sentence on the receipt.",
+    ),
+    (
+        "night-ferry",
+        "The night ferry crossed the lake even when the road around it was clear. "
+        "Commuters liked the twenty quiet minutes between phone calls and office "
+        "lights, and nurses used the benches to rest after long shifts. When the "
+        "transport agency proposed canceling the route, its spreadsheet counted "
+        "only tickets sold, not the ambulance trips avoided during winter, the "
+        "students who reached evening classes, or the small shops that opened "
+        "early because dock workers bought breakfast there.",
+    ),
+    (
+        "orchard-ledger",
+        "In the orchard ledger, every tree had a row for pruning, pests, harvest "
+        "weight, and weather notes. The oldest entries were written by a farmer "
+        "who never used abbreviations; later entries became shorter as the farm "
+        "grew. A new manager wanted to replace the ledger with a tablet form, but "
+        "kept one blank field labeled 'odd observations' after discovering that "
+        "several good decisions had started as remarks that did not fit any box.",
+    ),
+    (
+        "community-theater",
+        "The community theater sold out after it moved rehearsals into the front "
+        "room where passersby could hear the music. People who would never read a "
+        "poster stopped to ask about costumes, borrowed folding chairs, or offered "
+        "old curtains for the set. The director said the change worked because it "
+        "made preparation visible. The audience felt invited before anyone asked "
+        "them to buy a ticket.",
+    ),
+    (
+        "desert-well",
+        "The desert well was marked on maps as unreliable, but the older guides "
+        "still visited it after winter rains. They knew the water rose slowly "
+        "through cracked stone and disappeared before summer. A research team "
+        "installed a sensor and confirmed the pattern, yet the guides' calendar "
+        "was more useful than the graph because it included wind, animal tracks, "
+        "and stories about years when the rains arrived late.",
+    ),
+    (
+        "newspaper-correction",
+        "The newspaper correction was longer than the original mistake. It named "
+        "the incorrect statistic, explained how the reporter had misread the "
+        "table, printed the correct number, and described how future data stories "
+        "would be checked. Some readers mocked the length, but subscriptions rose "
+        "after the editor began treating corrections as public maintenance rather "
+        "than embarrassment.",
+    ),
+    (
+        "apartment-elevator",
+        "Residents complained that the apartment elevator was slow, but the repair "
+        "company found no mechanical fault. The building manager placed a mirror "
+        "beside the doors, then added a board with local notices and lost keys. "
+        "Complaints dropped because waiting felt shorter when people had something "
+        "useful to read. The elevator had not changed; the experience of waiting "
+        "had.",
+    ),
+    (
+        "field-notebook",
+        "A biologist's field notebook used color more than prose: blue dots for "
+        "water, green lines for moss, orange circles for nests. Her assistant "
+        "wanted to transcribe everything into sentences, but she kept the diagrams "
+        "because they showed proximity at a glance. Years later, the sketches "
+        "helped another team see that the birds nested near temporary pools only "
+        "after a particular moss appeared.",
+    ),
+    (
+        "town-clock",
+        "The town clock gained four minutes every week. For decades no one fixed "
+        "it because train schedules had moved to phones, and the clock remained "
+        "mostly ceremonial. Then the school used it for a lesson about measurement "
+        "error. Students logged the drift, predicted when it would be wrong by an "
+        "hour, and convinced the council that preservation sometimes means making "
+        "an old thing accurate again.",
+    ),
+    (
+        "kitchen-radio",
+        "In the restaurant kitchen, the radio was tuned low enough that customers "
+        "could not hear it and high enough that cooks could recognize the song. "
+        "The owner thought it was a distraction until a busy night when the radio "
+        "failed and the staff lost their rhythm. The music had been acting as a "
+        "shared clock, marking prep, service, cleanup, and the moment everyone "
+        "could finally sit down.",
+    ),
+    (
+        "wetland-boardwalk",
+        "The wetland boardwalk was raised by only twelve centimeters, but that was "
+        "enough to keep spring water from covering the planks. Visitors noticed "
+        "the new railings and benches, not the height. The maintenance crew "
+        "noticed something else: fewer people stepped off the path, so seedlings "
+        "returned near the edges. A small engineering change had become a habitat "
+        "change.",
+    ),
+    (
+        "festival-tickets",
+        "The festival changed ticket prices from three tiers to one suggested "
+        "price with a quiet option to pay less. Revenue stayed almost the same, "
+        "but attendance became more mixed by age and neighborhood. Volunteers "
+        "said the new wording mattered: people did not feel they were asking for "
+        "a discount; they felt they were choosing the price that let them attend "
+        "and still leave space for someone else.",
+    ),
+    (
+        "sewing-manual",
+        "The sewing manual used drawings instead of photographs because thread "
+        "and shadows confused beginners. Each drawing exaggerated the needle, "
+        "the fold, and the direction of tension. Advanced tailors found the pages "
+        "plain, but new learners made fewer mistakes. The designer concluded that "
+        "realism is not always the friend of instruction; sometimes the useful "
+        "picture is the one that lies about scale.",
+    ),
+    (
+        "island-power",
+        "The island power station added batteries before it added more solar "
+        "panels. Tourists expected the opposite, but engineers explained that "
+        "the island already produced enough midday electricity and lost too much "
+        "after sunset. The batteries did not make the photographs prettier, yet "
+        "they reduced diesel use more than another row of panels would have. The "
+        "best improvement was the least visible one.",
+    ),
+    (
+        "language-class",
+        "The language class stopped beginning with grammar charts and started "
+        "with lost-and-found objects: a glove, a ticket, a key, a cracked phone "
+        "case. Students described who might own each item and what had happened "
+        "before it was lost. Grammar still appeared, but as a tool for making "
+        "the story precise. Attendance improved because every lesson began with "
+        "a small mystery.",
+    ),
+    (
+        "bridge-inspection",
+        "The bridge inspection report listed no urgent danger, but it changed the "
+        "maintenance plan anyway. Hairline cracks near the drainage holes showed "
+        "that water was collecting where the original drawings assumed it would "
+        "flow away. The fix was inexpensive: clean the channels twice a year and "
+        "seal one joint. The important discovery was not damage, but the pattern "
+        "that would eventually create it.",
+    ),
+    (
+        "bookshop-window",
+        "The bookshop window once displayed bestsellers in perfect rows. A new "
+        "bookseller replaced them with pairs: a cookbook beside a travel memoir, "
+        "a history beside a novel, a children's atlas beside a book about birds. "
+        "Sales became less predictable but more varied. Customers said the window "
+        "felt like a conversation instead of a ranking.",
+    ),
+]
+
+
+LANGUAGE_TASKS = [
+    ("summary", "Summarize the passage in three concise bullet points."),
+    ("plain", "Rewrite the passage in plain English for a careful twelve-year-old reader."),
+    ("formal", "Rewrite the passage as a formal memo while preserving every factual detail."),
+    ("copyedit", "Copy-edit the passage for clarity. Keep the meaning, but improve flow and sentence rhythm."),
+    ("entities", "Extract people, places, organizations, problems, actions, and outcomes as compact JSON."),
+    ("tone", "Identify the tone, intended audience, and two implied assumptions in the passage."),
+    ("headline", "Write a headline, a one-sentence deck, and five search keywords for the passage."),
+    ("faq", "Turn the passage into a short FAQ with four questions and direct answers."),
+    ("compress", "Compress the passage to at most 90 words without losing the causal chain."),
+    ("compare", "Compare the passage with a similar situation from another field, making the analogy explicit."),
+    ("dialogue", "Turn the passage into a short dialogue between two people who disagree but listen carefully."),
+    ("critique", "Write a constructive critique of the decision or design described in the passage."),
+    ("abstract", "Write an academic-style abstract for the passage in 120 words or fewer."),
+    ("it", "Translate the passage into natural Italian, preserving names and concrete details."),
+    ("es", "Translate the passage into natural Spanish, preserving names and concrete details."),
+    ("zh", "Translate the passage into Simplified Chinese, preserving names and concrete details."),
+]
+
+
+def make_language_records(records: list[Record]) -> None:
+    """Add non-code prose tasks.
+
+    The passages above are original public-domain-compatible calibration text
+    written for this repository.  They cover summarization, rewriting, editing,
+    information extraction, and multilingual prose so the imatrix is not shaped
+    almost entirely by source-code and agent transcripts.
+    """
+
+    system = (
+        "You are a careful language assistant. Preserve meaning, names, numbers, "
+        "and causal relationships. Do not invent facts not supported by the text."
+    )
+    for source, passage in LANGUAGE_PASSAGES:
+        for label, instruction in LANGUAGE_TASKS:
+            add_record(records, "language", f"{source}:{label}", [
+                {"role": "system", "content": system},
+                {"role": "user", "content": f"{instruction}\n\nPassage:\n{passage}"},
+            ])
+
+
+def c_scan_string_end(text: str, pos: int) -> int:
+    """Return the index after a C string literal starting at ``pos``."""
+
+    assert text[pos] == '"'
+    pos += 1
+    while pos < len(text):
+        ch = text[pos]
+        if ch == "\\":
+            pos += 2
+            continue
+        if ch == '"':
+            return pos + 1
+        pos += 1
+    raise ValueError("unterminated C string literal")
+
+
+def c_string_value_at(text: str, pos: int) -> tuple[str, int]:
+    """Parse one or more adjacent C string literals.
+
+    ``ds4_eval.c`` uses ordinary escaped C strings for embedded questions.  We
+    reuse those definitions instead of duplicating 75 benchmark prompts in this
+    generator.  Python's literal parser handles the escaped newlines, quotes,
+    and backslashes used by these string initializers.
+    """
+
+    parts: list[str] = []
+    while True:
+        while pos < len(text) and text[pos].isspace():
+            pos += 1
+        if pos >= len(text) or text[pos] != '"':
+            break
+        end = c_scan_string_end(text, pos)
+        parts.append(ast.literal_eval(text[pos:end]))
+        pos = end
+    if not parts:
+        raise ValueError("expected C string literal")
+    return "".join(parts), pos
+
+
+def eval_case_blocks(eval_source: str) -> list[str]:
+    marker = "static const eval_case eval_cases[]"
+    start = eval_source.find(marker)
+    if start < 0:
+        raise ValueError("missing eval_cases array")
+    pos = eval_source.find("{", start)
+    if pos < 0:
+        raise ValueError("missing eval_cases initializer")
+
+    blocks: list[str] = []
+    depth = 1
+    case_start: int | None = None
+    pos += 1
+    in_string = False
+    in_char = False
+    line_comment = False
+    block_comment = False
+    escape = False
+    while pos < len(eval_source):
+        ch = eval_source[pos]
+        nxt = eval_source[pos + 1] if pos + 1 < len(eval_source) else ""
+
+        if line_comment:
+            if ch == "\n":
+                line_comment = False
+            pos += 1
+            continue
+        if block_comment:
+            if ch == "*" and nxt == "/":
+                block_comment = False
+                pos += 2
+            else:
+                pos += 1
+            continue
+        if in_string:
+            if escape:
+                escape = False
+            elif ch == "\\":
+                escape = True
+            elif ch == '"':
+                in_string = False
+            pos += 1
+            continue
+        if in_char:
+            if escape:
+                escape = False
+            elif ch == "\\":
+                escape = True
+            elif ch == "'":
+                in_char = False
+            pos += 1
+            continue
+
+        if ch == "/" and nxt == "/":
+            line_comment = True
+            pos += 2
+            continue
+        if ch == "/" and nxt == "*":
+            block_comment = True
+            pos += 2
+            continue
+        if ch == '"':
+            in_string = True
+            pos += 1
+            continue
+        if ch == "'":
+            in_char = True
+            pos += 1
+            continue
+        if ch == "{":
+            depth += 1
+            if depth == 2:
+                case_start = pos
+        elif ch == "}":
+            if depth == 2 and case_start is not None:
+                blocks.append(eval_source[case_start:pos + 1])
+                case_start = None
+            depth -= 1
+            if depth == 0:
+                break
+        pos += 1
+
+    return blocks
+
+
+def c_field_string(block: str, field: str) -> str:
+    m = re.search(rf"\.{re.escape(field)}\s*=", block)
+    if not m:
+        return ""
+    value, _ = c_string_value_at(block, m.end())
+    return value
+
+
+def c_choice_strings(block: str) -> list[str]:
+    choices: dict[int, str] = {}
+    for m in re.finditer(r"\.choice\[(\d+)\]\s*=", block):
+        value, _ = c_string_value_at(block, m.end())
+        choices[int(m.group(1))] = value
+    return [choices[i] for i in sorted(choices)]
+
+
+def build_eval_imatrix_prompt(case: dict) -> str:
+    out = [case["question"], ""]
+    choices = case["choices"]
+    if choices:
+        out.append("Choices:")
+        for i, choice in enumerate(choices):
+            out.append(f"{chr(ord('A') + i)}. {choice}")
+        out.append("")
+        out.append(
+            "Solve the question. At the end, write exactly one final line in "
+            "this format and do not write anything after it:"
+        )
+        out.append("Answer: <letter>")
+    else:
+        out.append(
+            "Solve the problem. At the end, write exactly one final line in "
+            "this format and do not write anything after it:"
+        )
+        out.append("Answer: <integer>")
+    return "\n".join(out)
+
+
+def make_eval_reasoning_records(root: Path, records: list[Record]) -> None:
+    """Include the exact benchmark prompt family used by ``ds4-eval``.
+
+    The benchmark cases are already embedded in ``ds4_eval.c`` with source
+    licensing notes.  This parser keeps the imatrix corpus synchronized with the
+    integration harness while excluding answer keys from model-visible text.
+    """
+
+    text = read_text(root / "ds4_eval.c")
+    system = (
+        "You are solving a hard benchmark question. Reason carefully. "
+        "The final answer must follow the requested format exactly."
+    )
+    for idx, block in enumerate(eval_case_blocks(text)):
+        case = {
+            "source": c_field_string(block, "source"),
+            "id": c_field_string(block, "id"),
+            "domain": c_field_string(block, "domain"),
+            "title": c_field_string(block, "title"),
+            "question": c_field_string(block, "question"),
+            "choices": c_choice_strings(block),
+        }
+        prompt = build_eval_imatrix_prompt(case)
+        source = f"{case['source']}:{case['id'] or idx}"
+        title = f"{case['source']} / {case['domain']} / {case['title']}"
+        add_record(records, "eval_reasoning", source, [
+            {"role": "system", "content": system},
+            {"role": "user", "content": f"{title}\n\n{prompt}"},
+        ])
+
+
 def make_long_context_records(root: Path, records: list[Record]) -> None:
+    """Add realistic long-context prompts.
+
+    These records are meant to expose routed FFNs to activations produced after
+    attention has integrated distant evidence.  They are not pure length padding:
+    every prompt asks the model to recover, compare, diagnose, or obey details
+    that occur far apart in the context.
+    """
+
     sources = [
         ("README.md", read_text(root / "README.md")),
         ("AGENT.md", read_text(root / "AGENT.md")),
         ("METAL.md", read_text(root / "METAL.md")),
+        ("gguf-tools/imatrix/README.md", read_text(root / "gguf-tools/imatrix/README.md")),
+        ("gguf-tools/imatrix/dataset/README.md", read_text(root / "gguf-tools/imatrix/dataset/README.md")),
+        ("gguf-tools/quality-testing/README.md", read_text(root / "gguf-tools/quality-testing/README.md")),
         ("ds4_server.c", read_text(root / "ds4_server.c")),
         ("ds4_metal.m", read_text(root / "ds4_metal.m")),
         ("metal/dsv4_hc.metal", read_text(root / "metal/dsv4_hc.metal")),
         ("metal/moe.metal", read_text(root / "metal/moe.metal")),
+        ("metal/dense.metal", read_text(root / "metal/dense.metal")),
     ]
-    blocks = []
+    by_name: dict[str, list[str]] = {}
     for name, text in sources:
-        for i, chunk in enumerate(chunk_text(text, size=5200, overlap=120)[:6]):
-            blocks.append(f"### {name} chunk {i}\n{chunk}")
+        chunks = chunk_text(text, size=3600, overlap=120)[:8]
+        if chunks:
+            by_name[name] = [f"### {name} chunk {i}\n{chunk}" for i, chunk in enumerate(chunks)]
+
+    def block(name: str, idx: int) -> str:
+        chunks = by_name.get(name) or []
+        if not chunks:
+            return f"### {name}\n(unavailable)"
+        return chunks[idx % len(chunks)]
+
+    def add_long(label: str, instruction: str, parts: list[str]) -> None:
+        body = "\n\n".join(p for p in parts if p)
+        add_record(records, "long_context", label, [
+            {"role": "system", "content": DEFAULT_SYSTEM},
+            {"role": "user", "content": instruction + "\n\n" + body},
+        ])
+
+    # Codebase synthesis: several distant implementation excerpts that require
+    # understanding cross-file contracts rather than a single local snippet.
+    add_long("codebase:sampling-cache", (
+        "Read these repository excerpts as one long context. Explain how prompt "
+        "rendering, KV disk-cache lookup, and tool replay interact. Identify two "
+        "ways a future patch could accidentally cause a useless re-prefill."
+    ), [
+        block("ds4_server.c", 1),
+        block("ds4_server.c", 4),
+        block("ds4.c", 2),
+        block("README.md", 1),
+    ])
+    add_long("codebase:metal-moe", (
+        "Read these long code excerpts and summarize the dataflow from routed "
+        "expert selection through MoE execution. Name the tensors whose value "
+        "distribution matters for imatrix collection."
+    ), [
+        block("ds4.c", 5),
+        block("ds4_metal.m", 3),
+        block("metal/moe.metal", 0),
+        block("metal/dsv4_hc.metal", 1),
+    ])
+    add_long("codebase:server-protocols", (
+        "Compare the protocol and server excerpts. What state must survive across "
+        "tool calls, process restarts, and client replay? Return a concise list of "
+        "invariants and the first log line you would add when debugging."
+    ), [
+        block("README.md", 0),
+        block("ds4_server.c", 0),
+        block("ds4_server.c", 3),
+        block("AGENT.md", 0),
+    ])
+    add_long("codebase:eval-imatrix", (
+        "Use the following excerpts to explain how the evaluation harness and the "
+        "imatrix dataset differ in purpose. Mention which parts affect model "
+        "quality measurement and which parts affect quantization calibration."
+    ), [
+        block("gguf-tools/imatrix/README.md", 0),
+        block("gguf-tools/imatrix/dataset/README.md", 0),
+        block("ds4.c", 6),
+        block("README.md", 2),
+    ])
+
+    # Agent transcript replay: the model must connect early user intent, later
+    # tool outputs, and repeated status/error text before deciding the next step.
+    transcript_a = "\n".join([
+        "USER: The server resumed an old coding session and got slower after a tool call.",
+        "ASSISTANT TOOL_CALL bash: rg -n \"canonicalization|responses replay|kv cache hit\" ds4_server.c",
+        "TOOL_RESULT:",
+        TOOL_RESULTS["grep-hit"],
+        "ASSISTANT: The first grep hit points to the renderer; I need logs next.",
+        "USER: Here is the trace excerpt. Do not change protocol semantics, only avoid useless rebuilds.",
+        "TRACE:",
+        "\n".join(f"0514 00:{30+i:02d}:14 ds4-server: chat ctx={12288+i*257}..{13000+i*257}:712 RESPPROTO TOOLS prefill chunk {i*128}/712 ({i*18}.0%) chunk=151.{i} t/s avg=149.{i} t/s" for i in range(6)),
+        "0514 00:36:37 ds4-server: tool checkpoint canonicalized ctx=13257..16846:3589 common=16846 live=16994 canonical=16937 via=rebuild",
+        "USER: What minimal next patch would you inspect first?",
+    ])
+    add_long("agent-transcript:responses-rebuild", (
+        "Read the long agent transcript and trace. Diagnose the likely source of "
+        "the wasted work and propose the next file/function to inspect first."
+    ), [transcript_a, block("ds4_server.c", 2), block("ds4_server.c", 5)])
+
+    transcript_b = "\n".join([
+        "USER: Please test CUDA on the remote machine, then check whether Metal has the same bug.",
+        "ASSISTANT TOOL_CALL bash: ssh 0.180 'cd ds4.c && make clean && make cuda && ./tests/cuda_long_context_smoke'",
+        "TOOL_RESULT:",
+        "nvcc -O3 --use_fast_math -o ds4-server ...\nlong_context_smoke: PASS tokens=81920 prefill=412.5s gen=128\n",
+        "ASSISTANT TOOL_CALL read_file: ds4_cuda.c lines 500..680",
+        "TOOL_RESULT:",
+        "cuda kernel uses kv_row_kind to branch raw/compressed writes; host path checks ctx rows before launch.\n",
+        "USER: Do not assume. Compare with Metal and tell me whether the issue exists there too.",
+        "ASSISTANT TOOL_CALL read_file: ds4_metal.m lines 900..1120",
+        "TOOL_RESULT:",
+        "Metal path stores raw and compressed KV through distinct command encoders and asserts the row range before dispatch.\n",
+        "USER: Summarize the conclusion and what regression test would catch this later.",
+    ])
+    add_long("agent-transcript:cuda-metal", (
+        "Read the long tool transcript. Produce a conclusion that distinguishes "
+        "confirmed facts from inferences, and name the regression test that would "
+        "make the issue findable in the future."
+    ), [transcript_b, block("ds4_metal.m", 2), block("metal/dense.metal", 0)])
+
+    transcript_c = "\n".join([
+        "USER: The benchmark TUI flickers after I press arrows.",
+        "ASSISTANT TOOL_CALL bash: git show 5b92e9b:ds4_eval.c | sed -n '1100,1260p'",
+        "TOOL_RESULT:",
+        "old renderer writes left panel only at safe boundaries and streams sampled bytes only in the right panel.\n",
+        "USER: Return to that model. Add raw mode and a stdin thread, but do not interleave writes.",
+        "ASSISTANT TOOL_CALL bash: ./ds4-eval --questions 2 --trace /tmp/eval.trace",
+        "TOOL_RESULT:",
+        "screen ok; arrows move selection; running case continues without left-panel corruption.\n",
+        "USER: Commit.",
+    ])
+    add_long("agent-transcript:tui-state", (
+        "Given this transcript and the code excerpts, explain the rendering state "
+        "model that avoids flicker and why input handling must be consumed only "
+        "at safe points."
+    ), [transcript_c, block("ds4.c", 0), block("ds4_server.c", 6)])
+
+    # Log/trace diagnosis: repeated operational logs with relevant details
+    # buried among normal progress lines.
+    cache_trace = "\n".join(
+        [f"0515 19:{10+i//6:02d}:{10+i%6*8:02d} ds4-server: chat ctx={20480+i*256}..{22528+i*256}:2048 TOOLS prefill chunk {min(2048, i*96)}/2048 ({min(100, i*4)}.0%) chunk={180-i%7*3}.12 t/s avg={176-i%5*2}.44 t/s" for i in range(28)] +
+        [
+            "0515 19:49:03 ds4-server: kv cache stored tokens=32768 trimmed=0 reason=continued key=token-text size=475.01 MiB save=83.4 ms",
+            "0515 19:49:04 ds4-server: kv cache evicted reason=disk-cache-full tokens=10240 hits=31 size=157.34 MiB file=/tmp/kvcache/old-hot.kv",
+            "0515 19:49:04 ds4-server: kv cache hit text tokens=32768 text=129332 quant=2 key=token-text load=41.7 ms file=/tmp/kvcache/new-frontier.kv",
+        ]
+    )
+    add_long("trace:kv-eviction", (
+        "Analyze this long cache trace. Explain whether eviction favored a useful "
+        "frontier or a stale once-hot file, and state which signal in the log "
+        "supports your conclusion."
+    ), [cache_trace])
+
+    tool_trace = "\n".join(
+        [f"0514 22:{16+i//10:02d}:{10+i%10*5:02d} ds4-server: chat ctx=13695..22840:9145 gen={5000+i*25} TOOLS DSML_START decoding chunk=16.{i%9}6 t/s avg=17.{i%5}0 t/s" for i in range(36)] +
+        [
+            "0514 22:17:06 ds4-server: chat ctx=13695..22840:9145 gen=6080 TOOLS DSML_START DSML_END finish=tool_calls",
+            "0514 22:17:06 ds4-server: tool parse recovered loose nested parameter name=write",
+            "0514 22:17:06 ds4-server: tool calls ctx=13695..22840:9145 n=1 names=[write]",
+        ]
+    )
+    add_long("trace:tool-recovery", (
+        "Read the long tool-call trace. Explain why returning a recoverable tool "
+        "result is better than failing the server request, and identify the first "
+        "line showing the model intended a tool call."
+    ), [tool_trace])
+
+    speed_trace = "\n".join(
+        [f"ctx_tokens={2048+i*2048},prefill_tps={232.4-i*2.1:.2f},gen_tps={19.0-i*0.08:.2f},kvcache_bytes={52184460+i*28188672}" for i in range(32)] +
+        ["note: one outlier at ctx=69632 has prefill_tps=124.32 and gen_tps=14.47"]
+    )
+    add_long("trace:speed-curve", (
+        "Given this long benchmark CSV excerpt, describe the trend without "
+        "overfitting to single-row noise. Name one plausible reason prefill and "
+        "generation slopes differ."
+    ), [speed_trace])
+
+    # Prose fact recovery: natural long story, not an artificial isolated
+    # needle, with facts spelled in words and requested later as structured data.
+    people = [
+        ("Alice", "fifty-two", 52, "kept the ferry accounts"),
+        ("Bruno", "seventeen", 17, "repaired the school roof"),
+        ("Caterina", "eighty-one", 81, "organized the archive"),
+        ("Diego", "thirty-four", 34, "mapped the old orchard"),
+        ("Elena", "sixty-eight", 68, "ran the clinic calendar"),
+        ("Farid", "twenty-nine", 29, "maintained the radio tower"),
+        ("Giulia", "ninety-three", 93, "cataloged the harbor tools"),
+        ("Hana", "forty-six", 46, "translated the notices"),
+        ("Ivan", "seventy-five", 75, "surveyed the river path"),
+        ("Marta", "eleven", 11, "tracked the bakery deliveries"),
+    ]
+    story_parts = []
+    for idx, (name, word, _num, job) in enumerate(people):
+        story_parts.append(
+            f"During week {idx + 1}, {name} {job}. The committee did not write "
+            f"{name}'s reference as digits; in the minutes it was spelled out as "
+            f"{word}, because the secretary wanted all assignments to look uniform."
+        )
+        story_parts.append(LANGUAGE_PASSAGES[idx % len(LANGUAGE_PASSAGES)][1])
+    add_long("story:assigned-numbers", (
+        "Read the long community story. Extract only the assignment references "
+        "for named people and return them as Name=number pairs, one per line."
+    ), ["\n\n".join(story_parts)])
+
+    # Document comparison: long policy/spec style context from multiple docs.
+    add_long("document-comparison:cache-eval-imatrix", (
+        "Compare these documents. Identify three tensions between benchmark "
+        "quality, imatrix representativeness, and disk-cache behavior, then "
+        "propose a practical compromise for each."
+    ), [
+        block("README.md", 3),
+        block("gguf-tools/imatrix/README.md", 0),
+        block("gguf-tools/quality-testing/README.md", 0),
+        block("AGENT.md", 1),
+    ])
+
+    # Needle-in-haystack: keep one small slice. The useful signal is the long
+    # context plus exact retrieval, not the artificial format by itself.
+    hay = []
+    for i in range(24):
+        hay.append(LANGUAGE_PASSAGES[(i * 5) % len(LANGUAGE_PASSAGES)][1])
+        if i == 17:
+            hay.append("Inspection note: the emergency calibration phrase is LANTERN-4827-CEDAR.")
+    add_long("needle:calibration-phrase", (
+        "Most of the following context is ordinary prose. Find the emergency "
+        "calibration phrase and quote it exactly, then explain where it appeared."
+    ), ["\n\n".join(hay)])
+
+    # Delayed constraint following: early requirements must survive a large body
+    # of intervening realistic context.
+    early_constraints = (
+        "Initial requirements, written before the long context: answer in exactly "
+        "four bullets; each bullet must start with SAFE:, FAST:, CLEAR:, or TEST:; "
+        "do not mention implementation details unless they appear in the context."
+    )
+    add_long("delayed-constraints:server-plan", (
+        "Follow the initial requirements after reading all context. Produce the "
+        "requested four-bullet plan and preserve the required prefixes."
+    ), [
+        early_constraints,
+        block("ds4_server.c", 1),
+        block("ds4_server.c", 5),
+        block("README.md", 4),
+        block("metal/moe.metal", 1),
+    ])
+
+    # Keep the older shuffled source-excerpt shape as a small background slice.
+    blocks = [chunk for chunks in by_name.values() for chunk in chunks[:4]]
     random.Random(7).shuffle(blocks)
-    for i in range(0, len(blocks), 4):
+    for i in range(0, min(len(blocks), 16), 4):
         group = blocks[i:i + 4]
         if len(group) < 2:
             continue
@@ -1392,7 +2170,7 @@ def write_outputs(outdir: Path, records: list[Record]) -> None:
 
     rendered_bytes = sum(len(r.rendered.encode("utf-8")) for r in records)
     manifest = {
-        "version": 3,
+        "version": 4,
         "purpose": "DeepSeek V4 Flash imatrix calibration prompts",
         "record_count": len(records),
         "rendered_utf8_bytes": rendered_bytes,
@@ -1426,7 +2204,9 @@ def main() -> None:
     make_general_records(records)
     make_programming_records(records)
     make_algorithm_records(records)
+    make_language_records(records)
     make_translation_records(records)
+    make_eval_reasoning_records(root, records)
     make_long_context_records(root, records)
     write_outputs(outdir, records)
 

gguf-tools/imatrix/dataset/manifest.json

@@ -1,29 +1,33 @@
 {
-  "version": 3,
+  "version": 4,
   "purpose": "DeepSeek V4 Flash imatrix calibration prompts",
-  "record_count": 3310,
-  "rendered_utf8_bytes": 10167295,
-  "rough_token_estimate_bytes_div_4": 2541823,
+  "record_count": 4690,
+  "rendered_utf8_bytes": 11670191,
+  "rough_token_estimate_bytes_div_4": 2917547,
   "categories": {
     "agent": 1106,
     "algorithms": 32,
+    "eval_reasoning": 150,
     "general": 40,
-    "long_context": 12,
+    "language": 1024,
+    "long_context": 36,
     "programming": 48,
-    "source": 1892,
+    "source": 2074,
     "translation": 180
   },
   "modes": {
-    "nothink": 1655,
-    "think": 1655
+    "nothink": 2345,
+    "think": 2345
   },
   "bytes_by_category": {
-    "agent": 4432105,
+    "agent": 4428393,
     "algorithms": 9388,
+    "eval_reasoning": 144853,
     "general": 11902,
-    "long_context": 248628,
+    "language": 725152,
+    "long_context": 357114,
     "programming": 14320,
-    "source": 5394802,
+    "source": 5922919,
     "translation": 56150
   },
   "files": {

gguf-tools/mixed/README.md

@@ -0,0 +1,40 @@
+# Mixed GGUF Splicing
+
+This directory contains a small GGUF splicer for experiments where only some
+routed-expert layers are upgraded from one quantization to another.
+
+The tool does not dequantize or requantize weights. It reads two compatible
+GGUF files, uses the first one as the base, and copies selected routed-expert
+tensors from the second one. All other tensors remain byte-for-byte identical to
+the base file.
+
+## Last Six Layers Q4 Experiment
+
+The best mixed Flash experiment so far keeps the fixed-imatrix 2-bit GGUF as the
+base, then replaces only the routed experts in layers `37-42` with the same
+tensors from the fixed-imatrix Q4 GGUF.
+
+This produces a file around 91 GB instead of the full Q4 file around 153 GB. In
+the local output-agreement checks, this last-six-layers file was statistically
+closer to the full Q4 GGUF than the plain 2-bit GGUF. This is not the same as a
+benchmark score, but it is a useful signal that the mixed file behaves more like
+the stronger Q4 model while keeping most of the 2-bit size.
+
+## Usage
+
+```bash
+python3 gguf-tools/mixed/splice_mixed_expert_layers_gguf.py \
+  --base /path/to/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf \
+  --donor /path/to/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-fixed.gguf \
+  --q4-layers 37-42 \
+  --out /path/to/DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf
+```
+
+Use `--dry-run` first to check the selected tensors and expected size delta.
+Use `--force` only when you really want to overwrite the output file.
+
+Layer ranges can be changed without editing the script:
+
+```bash
+--q4-layers 0-2,40-42
+```

gguf-tools/mixed/splice_mixed_expert_layers_gguf.py

@@ -0,0 +1,404 @@
+#!/usr/bin/env python3
+"""Build a mixed DeepSeek V4 Flash GGUF by splicing routed-expert layers.
+
+The base GGUF supplies metadata and all tensors by default.  For selected layer
+IDs, this copies the routed expert tensors from a donor GGUF, rewriting the GGUF
+tensor directory and streaming tensor payloads without dequantizing or
+requantizing.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import re
+import struct
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+from typing import BinaryIO
+
+
+GGUF_HEADER_SIZE = 24
+GGUF_DEFAULT_ALIGNMENT = 32
+
+GGUF_VALUE_UINT8 = 0
+GGUF_VALUE_INT8 = 1
+GGUF_VALUE_UINT16 = 2
+GGUF_VALUE_INT16 = 3
+GGUF_VALUE_UINT32 = 4
+GGUF_VALUE_INT32 = 5
+GGUF_VALUE_FLOAT32 = 6
+GGUF_VALUE_BOOL = 7
+GGUF_VALUE_STRING = 8
+GGUF_VALUE_ARRAY = 9
+GGUF_VALUE_UINT64 = 10
+GGUF_VALUE_INT64 = 11
+GGUF_VALUE_FLOAT64 = 12
+
+GGUF_SCALAR_SIZES = {
+    GGUF_VALUE_UINT8: 1,
+    GGUF_VALUE_INT8: 1,
+    GGUF_VALUE_UINT16: 2,
+    GGUF_VALUE_INT16: 2,
+    GGUF_VALUE_UINT32: 4,
+    GGUF_VALUE_INT32: 4,
+    GGUF_VALUE_FLOAT32: 4,
+    GGUF_VALUE_BOOL: 1,
+    GGUF_VALUE_UINT64: 8,
+    GGUF_VALUE_INT64: 8,
+    GGUF_VALUE_FLOAT64: 8,
+}
+
+# GGML quant type -> (block elements, bytes per block, display name).
+# This intentionally includes the formats used by the current DeepSeek V4 Flash
+# GGUFs.  Add entries here if future recipes introduce new tensor types.
+GGML_QUANT_SIZES = {
+    0: (1, 4, "F32"),
+    1: (1, 2, "F16"),
+    8: (32, 34, "Q8_0"),
+    10: (256, 84, "Q2_K"),
+    12: (256, 144, "Q4_K"),
+    16: (256, 66, "IQ2_XXS"),
+    26: (1, 4, "I32"),
+}
+
+EXPERT_TENSOR_RE = re.compile(r"^blk\.(\d+)\.ffn_(gate|up|down)_exps\.weight$")
+
+
+@dataclass(frozen=True)
+class TensorInfo:
+    name: str
+    dims: tuple[int, ...]
+    ggml_type: int
+    rel_offset: int
+    data_offset: int
+    n_bytes: int
+
+
+@dataclass(frozen=True)
+class GGUFInfo:
+    path: Path
+    version: int
+    tensor_count: int
+    kv_count: int
+    kv_blob: bytes
+    alignment: int
+    tensors: list[TensorInfo]
+    tensor_by_name: dict[str, TensorInfo]
+
+
+@dataclass(frozen=True)
+class SplicePlan:
+    name: str
+    source: str
+    tensor: TensorInfo
+    new_rel_offset: int
+
+
+def read_u32(data: bytes, offset: int) -> int:
+    return struct.unpack_from("<I", data, offset)[0]
+
+
+def read_u64(data: bytes, offset: int) -> int:
+    return struct.unpack_from("<Q", data, offset)[0]
+
+
+def pad_to(value: int, alignment: int) -> int:
+    return ((value + alignment - 1) // alignment) * alignment
+
+
+def read_exact(src: BinaryIO, n_bytes: int) -> bytes:
+    data = src.read(n_bytes)
+    if len(data) != n_bytes:
+        raise EOFError("short read while parsing GGUF")
+    return data
+
+
+def read_u32_file(src: BinaryIO) -> int:
+    return struct.unpack("<I", read_exact(src, 4))[0]
+
+
+def read_u64_file(src: BinaryIO) -> int:
+    return struct.unpack("<Q", read_exact(src, 8))[0]
+
+
+def skip_gguf_string_file(src: BinaryIO) -> None:
+    n = read_u64_file(src)
+    src.seek(n, os.SEEK_CUR)
+
+
+def read_gguf_string_file(src: BinaryIO) -> str:
+    n = read_u64_file(src)
+    return read_exact(src, n).decode("utf-8")
+
+
+def skip_value_payload_file(src: BinaryIO, value_type: int) -> None:
+    if value_type == GGUF_VALUE_STRING:
+        skip_gguf_string_file(src)
+        return
+    if value_type == GGUF_VALUE_ARRAY:
+        subtype = read_u32_file(src)
+        count = read_u64_file(src)
+        if subtype == GGUF_VALUE_STRING:
+            for _ in range(count):
+                skip_gguf_string_file(src)
+            return
+        if subtype == GGUF_VALUE_ARRAY:
+            for _ in range(count):
+                skip_value_payload_file(src, subtype)
+            return
+        size = GGUF_SCALAR_SIZES.get(subtype)
+        if size is None:
+            raise ValueError(f"unsupported GGUF array subtype {subtype}")
+        src.seek(count * size, os.SEEK_CUR)
+        return
+    size = GGUF_SCALAR_SIZES.get(value_type)
+    if size is None:
+        raise ValueError(f"unsupported GGUF value type {value_type}")
+    src.seek(size, os.SEEK_CUR)
+
+
+def tensor_nbytes(dims: tuple[int, ...], ggml_type: int) -> int:
+    q = GGML_QUANT_SIZES.get(ggml_type)
+    if q is None:
+        raise ValueError(f"unsupported GGML tensor type {ggml_type}; add it to GGML_QUANT_SIZES")
+    block_elems, block_bytes, _name = q
+    n_elems = 1
+    for dim in dims:
+        n_elems *= dim
+    if n_elems % block_elems != 0:
+        raise ValueError(f"tensor element count {n_elems} is not divisible by block size {block_elems}")
+    return (n_elems // block_elems) * block_bytes
+
+
+def parse_gguf(path: Path) -> GGUFInfo:
+    with path.open("rb") as src:
+        if read_exact(src, 4) != b"GGUF":
+            raise ValueError(f"{path} is not a GGUF file")
+        version = read_u32_file(src)
+        tensor_count = read_u64_file(src)
+        kv_count = read_u64_file(src)
+
+        alignment = GGUF_DEFAULT_ALIGNMENT
+        for _ in range(kv_count):
+            key = read_gguf_string_file(src)
+            value_type = read_u32_file(src)
+            value_start = src.tell()
+            skip_value_payload_file(src, value_type)
+            value_end = src.tell()
+            if key == "general.alignment" and value_type == GGUF_VALUE_UINT32:
+                src.seek(value_start)
+                alignment = read_u32_file(src)
+                src.seek(value_end)
+
+        kv_end = src.tell()
+        src.seek(GGUF_HEADER_SIZE)
+        kv_blob = read_exact(src, kv_end - GGUF_HEADER_SIZE)
+        src.seek(kv_end)
+
+        raw_tensors: list[tuple[str, tuple[int, ...], int, int]] = []
+        for _ in range(tensor_count):
+            name = read_gguf_string_file(src)
+            n_dims = read_u32_file(src)
+            dims = tuple(read_u64_file(src) for _ in range(n_dims))
+            ggml_type = read_u32_file(src)
+            rel_offset = read_u64_file(src)
+            raw_tensors.append((name, dims, ggml_type, rel_offset))
+
+        data_start = pad_to(src.tell(), alignment)
+    tensors: list[TensorInfo] = []
+    for name, dims, ggml_type, rel_offset in raw_tensors:
+        n_bytes = tensor_nbytes(dims, ggml_type)
+        tensors.append(TensorInfo(name, dims, ggml_type, rel_offset, data_start + rel_offset, n_bytes))
+
+    return GGUFInfo(
+        path=path,
+        version=version,
+        tensor_count=tensor_count,
+        kv_count=kv_count,
+        kv_blob=kv_blob,
+        alignment=alignment,
+        tensors=tensors,
+        tensor_by_name={t.name: t for t in tensors},
+    )
+
+
+def parse_layer_set(spec: str) -> set[int]:
+    layers: set[int] = set()
+    for part in spec.split(","):
+        part = part.strip()
+        if not part:
+            continue
+        if "-" in part:
+            lo_s, hi_s = part.split("-", 1)
+            lo = int(lo_s)
+            hi = int(hi_s)
+            if hi < lo:
+                raise ValueError(f"bad descending range {part!r}")
+            layers.update(range(lo, hi + 1))
+        else:
+            layers.add(int(part))
+    if not layers:
+        raise ValueError("no layers selected")
+    return layers
+
+
+def should_take_donor(name: str, q4_layers: set[int]) -> bool:
+    match = EXPERT_TENSOR_RE.match(name)
+    return match is not None and int(match.group(1)) in q4_layers
+
+
+def qtype_name(ggml_type: int) -> str:
+    return GGML_QUANT_SIZES.get(ggml_type, (0, 0, f"type_{ggml_type}"))[2]
+
+
+def build_plan(base: GGUFInfo, donor: GGUFInfo, q4_layers: set[int]) -> list[SplicePlan]:
+    if base.version != donor.version:
+        raise ValueError(f"GGUF version mismatch: base={base.version} donor={donor.version}")
+    if base.tensor_count != donor.tensor_count:
+        raise ValueError(f"tensor count mismatch: base={base.tensor_count} donor={donor.tensor_count}")
+    if base.alignment != donor.alignment:
+        raise ValueError(f"alignment mismatch: base={base.alignment} donor={donor.alignment}")
+
+    plan: list[SplicePlan] = []
+    next_rel = 0
+    for base_tensor in base.tensors:
+        donor_tensor = donor.tensor_by_name.get(base_tensor.name)
+        if donor_tensor is None:
+            raise ValueError(f"donor is missing tensor {base_tensor.name}")
+        if base_tensor.dims != donor_tensor.dims:
+            raise ValueError(f"shape mismatch for {base_tensor.name}: {base_tensor.dims} vs {donor_tensor.dims}")
+        use_donor = should_take_donor(base_tensor.name, q4_layers)
+        source_tensor = donor_tensor if use_donor else base_tensor
+        plan.append(SplicePlan(
+            name=base_tensor.name,
+            source="donor" if use_donor else "base",
+            tensor=source_tensor,
+            new_rel_offset=next_rel,
+        ))
+        next_rel += pad_to(source_tensor.n_bytes, base.alignment)
+    return plan
+
+
+def pack_string(value: str) -> bytes:
+    raw = value.encode("utf-8")
+    return struct.pack("<Q", len(raw)) + raw
+
+
+def write_tensor_info(out: BinaryIO, item: SplicePlan) -> None:
+    t = item.tensor
+    out.write(pack_string(item.name))
+    out.write(struct.pack("<I", len(t.dims)))
+    for dim in t.dims:
+        out.write(struct.pack("<Q", dim))
+    out.write(struct.pack("<I", t.ggml_type))
+    out.write(struct.pack("<Q", item.new_rel_offset))
+
+
+def copy_exact(src: BinaryIO, dst: BinaryIO, n_bytes: int, bufsize: int = 64 * 1024 * 1024) -> None:
+    remaining = n_bytes
+    while remaining:
+        chunk = src.read(min(bufsize, remaining))
+        if not chunk:
+            raise EOFError("short read while copying tensor payload")
+        dst.write(chunk)
+        remaining -= len(chunk)
+
+
+def write_padding(out: BinaryIO, n_bytes: int, alignment: int) -> None:
+    pad = pad_to(n_bytes, alignment) - n_bytes
+    if pad:
+        out.write(b"\0" * pad)
+
+
+def write_mixed(base: GGUFInfo, donor: GGUFInfo, plan: list[SplicePlan], out_path: Path, force: bool) -> None:
+    tmp_path = out_path.with_name(out_path.name + ".tmp")
+    if out_path.exists() and not force:
+        raise FileExistsError(f"{out_path} already exists; pass --force to overwrite")
+    if tmp_path.exists():
+        raise FileExistsError(f"temporary file already exists: {tmp_path}")
+
+    total_payload = sum(item.tensor.n_bytes for item in plan)
+    copied = 0
+    next_report = 0
+
+    with base.path.open("rb") as base_file, donor.path.open("rb") as donor_file, tmp_path.open("wb") as out:
+        out.write(b"GGUF")
+        out.write(struct.pack("<I", base.version))
+        out.write(struct.pack("<Q", base.tensor_count))
+        out.write(struct.pack("<Q", base.kv_count))
+        out.write(base.kv_blob)
+        for item in plan:
+            write_tensor_info(out, item)
+        write_padding(out, out.tell(), base.alignment)
+
+        for item in plan:
+            src = donor_file if item.source == "donor" else base_file
+            src.seek(item.tensor.data_offset)
+            copy_exact(src, out, item.tensor.n_bytes)
+            write_padding(out, item.tensor.n_bytes, base.alignment)
+            copied += item.tensor.n_bytes
+            pct = int(copied * 100 / total_payload) if total_payload else 100
+            if pct >= next_report:
+                print(f"copied {copied / (1024 ** 3):.2f} GiB / {total_payload / (1024 ** 3):.2f} GiB ({pct}%)", flush=True)
+                next_report += 5
+
+        out.flush()
+        os.fsync(out.fileno())
+
+    os.replace(tmp_path, out_path)
+
+
+def summarize(base: GGUFInfo, donor: GGUFInfo, plan: list[SplicePlan]) -> None:
+    selected = [item for item in plan if item.source == "donor"]
+    base_bytes = sum(t.n_bytes for t in base.tensors)
+    out_bytes = sum(item.tensor.n_bytes for item in plan)
+    delta = out_bytes - base_bytes
+
+    print(f"base:  {base.path}")
+    print(f"donor: {donor.path}")
+    print(f"selected donor tensors: {len(selected)}")
+    by_type: dict[str, int] = {}
+    for item in selected:
+        by_type[qtype_name(item.tensor.ggml_type)] = by_type.get(qtype_name(item.tensor.ggml_type), 0) + 1
+    print("selected donor types:", ", ".join(f"{name}:{count}" for name, count in sorted(by_type.items())) or "none")
+    print(f"base tensor payload: {base_bytes:,} bytes ({base_bytes / (1024 ** 3):.2f} GiB)")
+    print(f"mixed tensor payload: {out_bytes:,} bytes ({out_bytes / (1024 ** 3):.2f} GiB)")
+    print(f"payload delta: {delta:,} bytes ({delta / (1024 ** 3):.2f} GiB)")
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Splice selected DeepSeek V4 Flash routed-expert layers from a donor GGUF.")
+    parser.add_argument("--base", required=True, type=Path, help="base GGUF used for metadata and default tensors")
+    parser.add_argument("--donor", required=True, type=Path, help="donor GGUF used for selected routed expert layers")
+    parser.add_argument("--out", required=True, type=Path, help="output mixed GGUF")
+    parser.add_argument("--q4-layers", required=True, help="comma-separated layer IDs/ranges to take from donor, e.g. 37-42")
+    parser.add_argument("--dry-run", action="store_true", help="print the plan without writing the output")
+    parser.add_argument("--force", action="store_true", help="overwrite --out if it already exists")
+    args = parser.parse_args()
+
+    q4_layers = parse_layer_set(args.q4_layers)
+    print("q4 layers:", ",".join(str(x) for x in sorted(q4_layers)))
+
+    base = parse_gguf(args.base)
+    donor = parse_gguf(args.donor)
+    plan = build_plan(base, donor, q4_layers)
+    summarize(base, donor, plan)
+
+    if args.dry_run:
+        return 0
+
+    write_mixed(base, donor, plan, args.out, args.force)
+    final_size = args.out.stat().st_size
+    print(f"wrote: {args.out}")
+    print(f"file size: {final_size:,} bytes ({final_size / (1024 ** 3):.2f} GiB)")
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        raise SystemExit(main())
+    except Exception as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        raise SystemExit(1)

gguf-tools/quality-testing/README.md

@@ -1,35 +1,88 @@
 # Official-Continuation Quality Testing
 
-This directory contains the 100 prompts and scripts used to compare local GGUF
-variants against official DeepSeek V4 Flash continuations.
+This directory contains the prompts, tracked official fixtures, and scripts used
+to compare local GGUF variants against hosted-model continuations.
 
 The metric is target-token negative log likelihood: collect a deterministic
 official continuation, then ask each local GGUF how much probability it assigns
 to that exact continuation token by token.  This avoids judging quality from one
 sampled answer.
 
-## 1. Collect Official Continuations
+## 1. Tracked Fixture Sets
+
+Curated fixtures are kept in the repository so release QA can run without
+calling hosted APIs:
+
+- `data/glm52-openrouter-100`: 100 GLM 5.2 continuations collected through
+  OpenRouter `z-ai/glm-5.2` with `top_logprobs=20`.
+- `data/flash`: 100 DeepSeek V4 Flash continuations collected from the official
+  DeepSeek API with `top_logprobs=20`.
+- `data/pro`: 100 DeepSeek V4 PRO continuations collected from the official
+  DeepSeek API with `top_logprobs=20`.
+
+DeepSeek V4 Flash also has tracked official smoke vectors in
+`tests/test-vectors/`.  Those vectors drive `./ds4_test --logprob-vectors` and
+include short prompts plus long-prompt attention cases.
+
+The hosted APIs expose output-token logprobs and top-logprob alternatives, not
+full vocabulary logits.
+
+## 2. Collect Official Continuations
 
 ```sh
 export DEEPSEEK_API_KEY=...
 python3 gguf-tools/quality-testing/collect_official.py \
   --prompts gguf-tools/quality-testing/prompts.jsonl \
-  --out gguf-tools/quality-testing/data \
+  --out gguf-tools/quality-testing/data/flash \
   --count 100 \
   --max-tokens 24
 ```
 
+For GLM 5.2 through OpenRouter:
+
+```sh
+export OPENROUTER_API_KEY=...
+python3 gguf-tools/quality-testing/collect_official.py \
+  --model z-ai/glm-5.2 \
+  --endpoint https://openrouter.ai/api/v1/chat/completions \
+  --api-key-env OPENROUTER_API_KEY \
+  --prompts gguf-tools/quality-testing/prompts.jsonl \
+  --out gguf-tools/quality-testing/data/glm52-openrouter \
+  --count 100 \
+  --max-tokens 24 \
+  --top-logprobs 20 \
+  --token-limit-field max_tokens \
+  --provider-order parasail/fp8 \
+  --require-parameters \
+  --thinking omit \
+  --reasoning-effort none
+```
+
+Use one output directory per official model.  The default model is Flash, so
+`data/flash` is the recommended path for Flash continuations.  For PRO:
+
+```sh
+python3 gguf-tools/quality-testing/collect_official.py \
+  --model deepseek-v4-pro \
+  --prompts gguf-tools/quality-testing/prompts.jsonl \
+  --out gguf-tools/quality-testing/data/pro \
+  --count 100 \
+  --max-tokens 24 \
+  --top-logprobs 20
+```
+
 The script writes:
 
-- `data/prompts/case_*.txt`
-- `data/continuations/case_*.txt`
-- `data/responses/case_*.json`
-- `data/manifest.tsv`
+- `data/<model>/prompts/case_*.txt`
+- `data/<model>/continuations/case_*.txt`
+- `data/<model>/responses/case_*.json`
+- `data/<model>/manifest.tsv`
 
-The prompt list is tracked in `prompts.jsonl`; the official responses are not
-tracked because they are derived from an external API.
+The prompt list is tracked in `prompts.jsonl`.  Curated fixture directories are
+also tracked after review; ad-hoc API collection directories should stay
+untracked until they are intentionally promoted into the release QA set.
 
-## 2. Build The Local Scorer
+## 3. Build The Local Scorer
 
 ```sh
 make -C gguf-tools quality-score
@@ -37,23 +90,41 @@ make -C gguf-tools quality-score
 
 The scorer links against the DS4 runtime and uses Metal by default.
 
-## 3. Score GGUF Variants
+## 4. Score GGUF Variants
 
 ```sh
 gguf-tools/quality-testing/score_official \
   ../deepseek-v4-quants/gguf/OLD.gguf \
-  gguf-tools/quality-testing/data/manifest.tsv \
+  gguf-tools/quality-testing/data/pro/manifest.tsv \
   /tmp/old.tsv \
   4096
 
 gguf-tools/quality-testing/score_official \
   ../deepseek-v4-quants/gguf/NEW.gguf \
-  gguf-tools/quality-testing/data/manifest.tsv \
+  gguf-tools/quality-testing/data/pro/manifest.tsv \
   /tmp/new.tsv \
   4096
 ```
 
-## 4. Compare
+Use `data/flash/manifest.tsv` for Flash GGUFs,
+`data/glm52-openrouter-100/manifest.tsv` for GLM 5.2 GGUFs, and
+`data/pro/manifest.tsv` for PRO GGUFs.  The scorer and comparator do not care
+which model produced the manifest; the manifest path selects the continuation
+set.
+
+For a full-residency vs SSD-streaming comparison, score the same model twice and
+add the streaming flags to one run:
+
+```sh
+gguf-tools/quality-testing/score_official \
+  /path/to/model.gguf \
+  gguf-tools/quality-testing/data/glm52-openrouter-100/manifest.tsv \
+  /tmp/streaming.tsv \
+  4096 \
+  --ssd-streaming
+```
+
+## 5. Compare
 
 ```sh
 python3 gguf-tools/quality-testing/compare_scores.py /tmp/old.tsv /tmp/new.tsv
@@ -69,3 +140,13 @@ Output fields:
   official first token.
 - `avg_greedy_lcp`: average greedy longest common prefix against the official
   continuation.
+- `api_target_mae`: when the manifest includes `response_file`, absolute
+  local-vs-API logprob delta for aligned official output tokens.
+- `api_top_coverage`: fraction of API top-logprob alternatives that map exactly
+  to one local tokenizer token.
+- `api_top1_rate`: how often the API top alternative equals the local greedy
+  token.
+- `api_topn_recall`: fraction of mapped API top-N alternatives found in the
+  local top-N for the same position.
+- `api_top_mae`: local-vs-API logprob MAE over mapped API top alternatives.
+- `api_pair_rate`: pairwise ordering agreement among mapped API alternatives.

gguf-tools/quality-testing/collect_official.py

@@ -1,11 +1,12 @@
 #!/usr/bin/env python3
-"""Collect official DeepSeek V4 Flash continuations for local quant scoring."""
+"""Collect hosted-model continuations for local quant scoring."""
 
 from __future__ import annotations
 
 import argparse
 import json
 import os
+import sys
 import time
 import urllib.error
 import urllib.request
@@ -120,19 +121,41 @@ PROMPTS = [
 ]
 
 
-def request_one(api_key: str, prompt: str, max_tokens: int) -> dict:
+def request_one(
+    api_key: str,
+    endpoint: str,
+    model: str,
+    prompt: str,
+    max_tokens: int,
+    top_logprobs: int,
+    thinking: str,
+    reasoning_effort: str,
+    token_limit_field: str,
+    provider_order: list[str],
+    provider_allow_fallbacks: bool,
+    provider_require_parameters: bool,
+) -> dict:
     payload = {
-        "model": MODEL,
+        "model": model,
         "messages": [{"role": "user", "content": prompt}],
         "temperature": 0,
-        "max_tokens": max_tokens,
         "logprobs": True,
-        "top_logprobs": 5,
-        "thinking": {"type": "disabled"},
+        "top_logprobs": top_logprobs,
         "stream": False,
     }
+    payload[token_limit_field] = max_tokens
+    if thinking != "omit":
+        payload["thinking"] = {"type": thinking}
+    if reasoning_effort != "omit":
+        payload["reasoning"] = {"effort": reasoning_effort}
+    if provider_order or provider_require_parameters:
+        provider = {"require_parameters": provider_require_parameters}
+        if provider_order:
+            provider["order"] = provider_order
+            provider["allow_fallbacks"] = provider_allow_fallbacks
+        payload["provider"] = provider
     req = urllib.request.Request(
-        ENDPOINT,
+        endpoint,
         data=json.dumps(payload).encode("utf-8"),
         headers={
             "Authorization": f"Bearer {api_key}",
@@ -144,11 +167,37 @@ def request_one(api_key: str, prompt: str, max_tokens: int) -> dict:
         return json.loads(fp.read().decode("utf-8"))
 
 
-def fetch_with_retry(api_key: str, prompt: str, max_tokens: int) -> dict:
+def fetch_with_retry(
+    api_key: str,
+    endpoint: str,
+    model: str,
+    prompt: str,
+    max_tokens: int,
+    top_logprobs: int,
+    thinking: str,
+    reasoning_effort: str,
+    token_limit_field: str,
+    provider_order: list[str],
+    provider_allow_fallbacks: bool,
+    provider_require_parameters: bool,
+) -> dict:
     delay = 1.0
     for attempt in range(6):
         try:
-            return request_one(api_key, prompt, max_tokens)
+            return request_one(
+                api_key,
+                endpoint,
+                model,
+                prompt,
+                max_tokens,
+                top_logprobs,
+                thinking,
+                reasoning_effort,
+                token_limit_field,
+                provider_order,
+                provider_allow_fallbacks,
+                provider_require_parameters,
+            )
         except urllib.error.HTTPError as e:
             body = e.read().decode("utf-8", "replace")
             if e.code < 500 and e.code != 429:
@@ -167,13 +216,46 @@ def main() -> int:
     ap = argparse.ArgumentParser()
     ap.add_argument("--out", default="gguf-tools/quality-testing/data")
     ap.add_argument("--prompts", default="gguf-tools/quality-testing/prompts.jsonl")
+    ap.add_argument("--model", default=MODEL)
+    ap.add_argument("--endpoint", default=ENDPOINT)
+    ap.add_argument("--api-key-env", default=None)
     ap.add_argument("--count", type=int, default=100)
     ap.add_argument("--max-tokens", type=int, default=24)
+    ap.add_argument("--top-logprobs", type=int, default=5)
+    ap.add_argument("--thinking", choices=("disabled", "enabled", "omit"), default="disabled")
+    ap.add_argument("--reasoning-effort",
+                    choices=("xhigh", "high", "medium", "low", "minimal", "none", "omit"),
+                    default=None)
+    ap.add_argument("--token-limit-field",
+                    choices=("auto", "max_tokens", "max_completion_tokens"),
+                    default="auto")
+    ap.add_argument("--provider-order",
+                    help="comma-separated OpenRouter provider slugs, for example ambient/fp8")
+    ap.add_argument("--allow-provider-fallbacks", action="store_true")
+    ap.add_argument("--require-parameters", action="store_true",
+                    help="for OpenRouter, route only to endpoints advertising the requested parameters")
     args = ap.parse_args()
+    if args.top_logprobs < 0 or args.top_logprobs > 20:
+        raise SystemExit("--top-logprobs must be between 0 and 20")
 
-    api_key = os.environ.get("DEEPSEEK_API_KEY")
+    openrouter = "openrouter.ai" in args.endpoint
+    api_key_env = args.api_key_env or ("OPENROUTER_API_KEY" if openrouter else "DEEPSEEK_API_KEY")
+    api_key = os.environ.get(api_key_env)
     if not api_key:
-        raise SystemExit("DEEPSEEK_API_KEY is not set")
+        raise SystemExit(f"{api_key_env} is not set")
+    thinking = args.thinking
+    reasoning_effort = args.reasoning_effort
+    if reasoning_effort is None:
+        reasoning_effort = "none" if openrouter else "omit"
+    if openrouter and args.thinking == "disabled":
+        thinking = "omit"
+    token_limit_field = args.token_limit_field
+    if token_limit_field == "auto":
+        token_limit_field = "max_tokens"
+    provider_order = []
+    if args.provider_order:
+        provider_order = [item.strip() for item in args.provider_order.split(",") if item.strip()]
+    provider_require_parameters = args.require_parameters
     prompts = load_prompts(Path(args.prompts))
 
     out = Path(args.out)
@@ -184,14 +266,37 @@ def main() -> int:
     manifest = out / "manifest.tsv"
     rows = []
     total = min(args.count, len(prompts))
+    print(f"model={args.model} endpoint={args.endpoint}", file=sys.stderr)
+    print(f"key_env={api_key_env} token_field={token_limit_field} thinking={thinking} reasoning={reasoning_effort}",
+          file=sys.stderr)
+    if provider_order or provider_require_parameters:
+        print(f"provider_order={','.join(provider_order) or '-'} "
+              f"provider_fallbacks={args.allow_provider_fallbacks} "
+              f"require_parameters={provider_require_parameters}", file=sys.stderr)
     for i, prompt in enumerate(prompts[: args.count]):
         case_id = f"case_{i:03d}"
         print(f"official {i + 1}/{total}: {case_id}", file=sys.stderr, flush=True)
-        response = fetch_with_retry(api_key, prompt, args.max_tokens)
+        response = fetch_with_retry(
+            api_key,
+            args.endpoint,
+            args.model,
+            prompt,
+            args.max_tokens,
+            args.top_logprobs,
+            thinking,
+            reasoning_effort,
+            token_limit_field,
+            provider_order,
+            args.allow_provider_fallbacks,
+            provider_require_parameters,
+        )
         choice = response["choices"][0]
         content = choice.get("message", {}).get("content", "")
         if not content:
             print(f"warning: empty continuation for {case_id}", file=sys.stderr)
+        logprob_items = (choice.get("logprobs") or {}).get("content", []) or []
+        if args.top_logprobs > 0 and not logprob_items:
+            raise RuntimeError(f"{case_id}: response did not include output-token logprobs")
 
         prompt_path = out / "prompts" / f"{case_id}.txt"
         cont_path = out / "continuations" / f"{case_id}.txt"

gguf-tools/quality-testing/compare_scores.py

@@ -8,20 +8,89 @@ import sys
 from pathlib import Path
 
 
-def load(path: Path) -> dict[str, dict[str, float]]:
+def parse_cell(value: str) -> float | int:
+    v = float(value)
+    if v.is_integer():
+        return int(v)
+    return v
+
+
+def load(path: Path) -> dict[str, dict[str, float | int]]:
     with path.open(newline="", encoding="utf-8") as fp:
         rows = {}
         for row in csv.DictReader(fp, delimiter="\t"):
-            rows[row["id"]] = {
+            parsed = {
                 "target_tokens": int(row["target_tokens"]),
                 "nll": float(row["nll"]),
                 "avg_nll": float(row["avg_nll"]),
                 "first_match": int(row["first_match"]),
                 "greedy_lcp": int(row["greedy_lcp"]),
             }
+            for key, value in row.items():
+                if key is None or key == "id" or key in parsed or value is None or value == "":
+                    continue
+                parsed[key] = parse_cell(value)
+            rows[row["id"]] = parsed
         return rows
 
 
+def has_api(rows: dict[str, dict[str, float | int]], ids: list[str]) -> bool:
+    return all(
+        "api_target_tokens" in rows[case_id] and "api_top_items" in rows[case_id]
+        for case_id in ids
+    )
+
+
+def sum_key(rows: dict[str, dict[str, float | int]], ids: list[str], key: str) -> float:
+    return sum(float(rows[case_id].get(key, 0.0)) for case_id in ids)
+
+
+def weighted_avg(
+    rows: dict[str, dict[str, float | int]], ids: list[str], value_key: str, count_key: str
+) -> float:
+    count = sum_key(rows, ids, count_key)
+    if count == 0:
+        return 0.0
+    total = sum(
+        float(rows[case_id].get(value_key, 0.0)) * float(rows[case_id].get(count_key, 0.0))
+        for case_id in ids
+    )
+    return total / count
+
+
+def ratio(num: float, den: float) -> float:
+    return num / den if den else 0.0
+
+
+def print_api_summary(label: str, rows: dict[str, dict[str, float | int]], ids: list[str]) -> None:
+    target_tokens = sum_key(rows, ids, "api_target_tokens")
+    top_items = sum_key(rows, ids, "api_top_items")
+    top_mapped = sum_key(rows, ids, "api_top_mapped")
+    top1_count = sum_key(rows, ids, "api_top1_count")
+    top1_match = sum_key(rows, ids, "api_top1_match")
+    topn_ref = sum_key(rows, ids, "api_topn_ref")
+    topn_hit = sum_key(rows, ids, "api_topn_hit")
+    top_logprob_count = sum_key(rows, ids, "api_top_logprob_count")
+    pair_total = sum_key(rows, ids, "api_pair_total")
+    pair_agree = sum_key(rows, ids, "api_pair_agree")
+
+    print(f"{label}_api_target_tokens\t{int(target_tokens)}")
+    print(f"{label}_api_target_mae\t{weighted_avg(rows, ids, 'api_target_mae', 'api_target_tokens'):.9f}")
+    print(
+        f"{label}_api_target_mean_delta\t"
+        f"{weighted_avg(rows, ids, 'api_target_mean_delta', 'api_target_tokens'):.9f}"
+    )
+    print(f"{label}_api_top_coverage\t{ratio(top_mapped, top_items):.9f}")
+    print(f"{label}_api_top1_rate\t{ratio(top1_match, top1_count):.9f}")
+    print(f"{label}_api_topn_recall\t{ratio(topn_hit, topn_ref):.9f}")
+    print(f"{label}_api_top_mae\t{weighted_avg(rows, ids, 'api_top_mae', 'api_top_logprob_count'):.9f}")
+    print(
+        f"{label}_api_top_mean_delta\t"
+        f"{weighted_avg(rows, ids, 'api_top_mean_delta', 'api_top_logprob_count'):.9f}"
+    )
+    print(f"{label}_api_pair_rate\t{ratio(pair_agree, pair_total):.9f}")
+
+
 def main() -> int:
     if len(sys.argv) != 3:
         print(f"usage: {sys.argv[0]} OLD.tsv NEW.tsv", file=sys.stderr)
@@ -73,6 +142,9 @@ def main() -> int:
     print(f"case_wins_new_old_ties\t{new_case_wins}\t{old_case_wins}\t{ties}")
     print(f"first_token_matches_old_new\t{old_first}\t{new_first}")
     print(f"avg_greedy_lcp_old_new\t{old_lcp / len(ids):.3f}\t{new_lcp / len(ids):.3f}")
+    if has_api(old, ids) and has_api(new, ids):
+        print_api_summary("old", old, ids)
+        print_api_summary("new", new, ids)
 
     print("\nnew best cases:")
     for delta, case_id, t, old_avg, new_avg in sorted(deltas)[:8]:

gguf-tools/quality-testing/data/README.md

@@ -0,0 +1,21 @@
+# Official Quality Fixtures
+
+This directory contains curated hosted-model continuation fixtures that are
+safe to commit and use in release QA.
+
+- `glm52-openrouter-100`: 100 GLM 5.2 OpenRouter continuations with API
+  top-logprob slices.
+- `flash`: 100 DeepSeek V4 Flash official continuations with API top-logprob
+  slices.
+- `pro`: 100 DeepSeek V4 PRO official continuations with API top-logprob
+  slices.
+
+Each fixture directory contains:
+
+- `prompts/case_*.txt`: exact user prompts.
+- `continuations/case_*.txt`: deterministic hosted-model continuations.
+- `responses/case_*.json`: raw hosted responses, including logprob slices.
+- `manifest.tsv`: paths consumed by `score_official`.
+
+DeepSeek V4 Flash smoke vectors are also tracked in `tests/test-vectors/` and
+are run by `./ds4_test --logprob-vectors`.

gguf-tools/quality-testing/data/flash/continuations/case_000.txt

@@ -0,0 +1 @@
+Here is a detailed explanation of B-tree insertion, covering the standard algorithm, node splits, and the special case of splitting

gguf-tools/quality-testing/data/flash/continuations/case_001.txt

@@ -0,0 +1,5 @@
+Here's a concise design for a TCP echo server that handles slow clients:
+
+## Architecture
+
+**Event-driven, non-block

gguf-tools/quality-testing/data/flash/continuations/case_002.txt

@@ -0,0 +1 @@
+Here's a comprehensive comparison of memory-mapped (mmap) model weights versus copying all weights into private buffers on macOS

gguf-tools/quality-testing/data/flash/continuations/case_003.txt

@@ -0,0 +1 @@
+This is an excellent question because it gets to the heart of why **RMSNorm** (Root Mean Square Normalization)

gguf-tools/quality-testing/data/flash/continuations/case_004.txt

@@ -0,0 +1 @@
+This is an excellent question that gets at a core optimization challenge in modern LLM deployment. Let's break down why **

gguf-tools/quality-testing/data/flash/continuations/case_005.txt

@@ -0,0 +1,5 @@
+Here's pseudocode for a binary heap push (insert) operation:
+
+```
+function push(heap, value):
+    //

gguf-tools/quality-testing/data/flash/continuations/case_006.txt

@@ -0,0 +1 @@
+This is a great question that gets at the core of how modern LLMs generate text. Both **temperature** and **

gguf-tools/quality-testing/data/flash/continuations/case_007.txt

@@ -0,0 +1 @@
+Here is a summary of how speculative decoding preserves exact greedy output (i.e., the same tokens that would be generated by

gguf-tools/quality-testing/data/flash/continuations/case_008.txt

@@ -0,0 +1 @@
+This is an excellent question that gets to the heart of how Large Language Models (LLMs) are served efficiently. The

gguf-tools/quality-testing/data/flash/continuations/case_009.txt

@@ -0,0 +1,5 @@
+Here are three invariants for a special-token table used in a tokenizer:
+
+---
+
+### 1. **Un

gguf-tools/quality-testing/data/flash/continuations/case_010.txt

@@ -0,0 +1 @@
+Ecco una spiegazione dettagliata del funzionamento dell'inserimento in un **B-tree**, con

gguf-tools/quality-testing/data/flash/continuations/case_011.txt

@@ -0,0 +1,3 @@
+Ecco un progetto conciso per un server TCP echo con gestione di client lenti:
+
+## Architettura

gguf-tools/quality-testing/data/flash/continuations/case_012.txt

@@ -0,0 +1 @@
+Ecco un confronto dettagliato tra **mmap** (memory-mapped file) e **copia

gguf-tools/quality-testing/data/flash/continuations/case_013.txt

@@ -0,0 +1 @@
+Certamente. Ecco una derivazione dettagliata del perché **RMSNorm** (Root Mean Square Normalization

gguf-tools/quality-testing/data/flash/continuations/case_014.txt

@@ -0,0 +1 @@
+La cache KV (Key-Value) su disco è particolarmente utile nelle **sessioni agentiche** (d

gguf-tools/quality-testing/data/flash/continuations/case_015.txt

@@ -0,0 +1 @@
+Ecco lo pseudocodice per inserire un elemento in un heap binario (min-heap o max-he

gguf-tools/quality-testing/data/flash/continuations/case_016.txt

@@ -0,0 +1 @@
+Questa è un'ottima domanda che tocca il cuore del controllo della generazione nei modelli linguistici

gguf-tools/quality-testing/data/flash/continuations/case_017.txt

@@ -0,0 +1 @@
+La decodifica speculativa mantiene l'output esatto perché utilizza un **meccanismo di ver

gguf-tools/quality-testing/data/flash/continuations/case_018.txt

@@ -0,0 +1 @@
+Questa è un'ottima domanda che tocca un aspetto fondamentale dell'ottimizzazione dell'inferenza

gguf-tools/quality-testing/data/flash/continuations/case_019.txt

@@ -0,0 +1 @@
+Ecco tre invarianti per una tabella di token speciali (ad esempio in un parser o in un sistema di

gguf-tools/quality-testing/data/flash/continuations/case_020.txt

@@ -0,0 +1,3 @@
+## Binary Search for Insertion Point
+
+Binary search finds the insertion point for a target value in a sorted array by maintaining

gguf-tools/quality-testing/data/flash/continuations/case_021.txt

@@ -0,0 +1,4 @@
+Here's a short C function that clamps an integer to a range:
+
+```c
+int clamp(int value, int

gguf-tools/quality-testing/data/flash/continuations/case_022.txt

@@ -0,0 +1 @@
+This is a fundamental question in concurrent programming. The core difference lies in **what they protect** and **how they coordinate

gguf-tools/quality-testing/data/flash/continuations/case_023.txt

@@ -0,0 +1 @@
+In the context of a network server, **backpressure** is a mechanism for **flow control**. It refers to the

gguf-tools/quality-testing/data/flash/continuations/case_024.txt

@@ -0,0 +1 @@
+Here is a safe and robust format for storing a model checkpoint header, designed for clarity, extensibility, and error prevention

gguf-tools/quality-testing/data/flash/continuations/case_025.txt

@@ -0,0 +1 @@
+A **ring buffer** (also called a circular buffer) is a fixed-size data structure that uses a contiguous block of

gguf-tools/quality-testing/data/flash/continuations/case_026.txt

@@ -0,0 +1,4 @@
+Here is a brief, structured plan for testing long-context prompt chunking.
+
+### Goal
+Validate that chunking a

gguf-tools/quality-testing/data/flash/continuations/case_027.txt

@@ -0,0 +1 @@
+This is an excellent question that gets to the heart of why modern low-bit quantization (like 4-bit and 3

gguf-tools/quality-testing/data/flash/continuations/case_028.txt

@@ -0,0 +1 @@
+This is a great question that gets at a key performance optimization in modern large language model (LLM) inference, particularly

gguf-tools/quality-testing/data/flash/continuations/case_029.txt

@@ -0,0 +1 @@
+This is an excellent question because comparing quantizations is rarely a simple "A is better than B" scenario. The "

gguf-tools/quality-testing/data/flash/continuations/case_030.txt

@@ -0,0 +1,3 @@
+## Ricerca Binaria del Punto di Inserimento
+
+La ricerca binaria del punto di inserimento è una

gguf-tools/quality-testing/data/flash/continuations/case_031.txt

@@ -0,0 +1,4 @@
+Ecco una breve funzione C che limita un intero a un intervallo specificato:
+
+```c
+int clamp

gguf-tools/quality-testing/data/flash/continuations/case_032.txt

@@ -0,0 +1 @@
+Certamente. La differenza fondamentale tra un **mutex** (mutual exclusion) e un **contatore atom

gguf-tools/quality-testing/data/flash/continuations/case_033.txt

@@ -0,0 +1 @@
+In un server di rete, **backpressure** (contro-pressione o contropressione) è un me

gguf-tools/quality-testing/data/flash/continuations/case_034.txt

@@ -0,0 +1 @@
+Ecco un formato sicuro per salvare l'header di un checkpoint modello, progettato per garantire integ

gguf-tools/quality-testing/data/flash/continuations/case_035.txt

@@ -0,0 +1 @@
+Ti spiegherò il funzionamento di un **ring buffer** (o buffer circolare) e le tec

gguf-tools/quality-testing/data/flash/continuations/case_036.txt

@@ -0,0 +1 @@
+Ecco un piano breve e strutturato per testare il **chunking** (frammentazione) di

gguf-tools/quality-testing/data/flash/continuations/case_037.txt

@@ -0,0 +1 @@
+Nella quantizzazione low-bit (a bassa precisione) dei modelli di deep learning, una **matrice

gguf-tools/quality-testing/data/flash/continuations/case_038.txt

@@ -0,0 +1 @@
+Questa è un'ottima domanda che tocca un aspetto cruciale dell'ottimizzazione dei modelli

gguf-tools/quality-testing/data/flash/continuations/case_039.txt

@@ -0,0 +1 @@
+Per confrontare due quantizzazioni (ad esempio, due modelli di compressione di un segnale, due

gguf-tools/quality-testing/data/flash/continuations/case_040.txt

@@ -0,0 +1,3 @@
+Here are the first five measurements to take when diagnosing generation slowdowns at long context:
+
+1.  **Time-to-F

gguf-tools/quality-testing/data/flash/continuations/case_041.txt

@@ -0,0 +1 @@
+This is an excellent and fundamental question about how language models actually "think." The short answer is: **because the model

gguf-tools/quality-testing/data/flash/continuations/case_042.txt

@@ -0,0 +1,3 @@
+Let's break down **online softmax** for attention using simple variables.
+
+## The Problem: Standard Softmax in Attention

gguf-tools/quality-testing/data/flash/continuations/case_043.txt

@@ -0,0 +1,5 @@
+Here's a minimal JSON schema for a tool call with name and arguments:
+
+```json
+{
+  "type": "

gguf-tools/quality-testing/data/flash/continuations/case_044.txt

@@ -0,0 +1 @@
+To test that a file-backed mmap is not repeatedly remapped, you need to verify that the kernel is reusing

gguf-tools/quality-testing/data/flash/continuations/case_045.txt

@@ -0,0 +1 @@
+This is a fascinating question that gets to the heart of how modern graphics APIs like Metal, Vulkan, and DirectX

gguf-tools/quality-testing/data/flash/continuations/case_046.txt

@@ -0,0 +1 @@
+In a Mixture-of-Experts (MoE) layer, the **router** (also called a **gating

gguf-tools/quality-testing/data/flash/continuations/case_047.txt

@@ -0,0 +1 @@
+Here is the difference between **raw KV rows** and **compressed KV rows** in the context of storage engines (

gguf-tools/quality-testing/data/flash/continuations/case_048.txt

@@ -0,0 +1 @@
+Here is a comprehensive checklist for validating that a quantized model still follows instructions effectively. This checklist is designed to catch both

gguf-tools/quality-testing/data/flash/continuations/case_049.txt

@@ -0,0 +1 @@
+This is an excellent question that gets to the heart of how to properly test and understand a language model's capabilities. Evaluating