Guard native cross-session attention, grouped expert, and Q8 batch GEMM tests with __APPLE__. CUDA retains its supported scalar, prefill, and recurrent kernel oracles instead of failing to link against Metal-only entry points.
Use a four-term log1p series for exp(x) below 1/32 in all four DeepSeek router transform sites. Clamp the polynomial input so its unselected branch stays finite, and compare Metal probabilities against the shared accurate reference without widening the tolerance.
Refs antirez/ds4#1039. Validated on M5 Max with a fresh default build, test_deepseek41_metal, test_deepseek41_graph --vision-routing, and full Metal test runs with DS4_METAL_DISABLE_PRE_M5_ROUTER_TRANSFORM_FINALIZE_FUSION=1 and DS4_METAL_MATH_SAFE=1. No model required. Pre-M5 validation and model-level speed/SSD checks remain outstanding.
(cherry picked from commit 7f5f1b77ae)
Negotiate 512-byte TCP segments on Linux before connecting. Otherwise shrinking loopback sockets to 1 KiB receive buffers leaves 32 KiB segments, making zero-window probing dominate large test transfers and sometimes trip the idle timeout. Retain the payload sizes, bidirectional backpressure, descriptor-mode checks and timeout/EOF cases. Linux and macOS transport tests pass; production transport is unchanged.
Keep fingerprint-only matching for authenticated visible histories that rebase spans, but require exact historical image positions for token and rewind reuse. Reproduce the old false hit with a synthetic checkpoint. Fix the live eviction oracle to include all competing slots. Four-slot Qwen and GLM vision/tool/eviction and concurrent-request tests pass. Follow-up to #765.
jpeg_prog_decode_ac_refine treated ZRL (run=15, size=0) as run=16 and
reused the skip loop, which kept consuming correction bits for non-zero
coefficients after the 16th zero-history coefficient. Those bits belong
after the next Huffman symbol (ITU T.81 G.1.2.3, libjpeg
decode_mcu_AC_refine), so the bitstream desynchronised: many progressive
JPEGs, including plain `cjpeg -progressive` output, failed to load or
decoded with wrong pixels.
Skip `run` zeros while refining non-zeros on the way, write the new value
(0 for ZRL) at the next zero, and stop, matching libjpeg's structure.
Add tests/test_image_decode with two cjpeg-generated fixtures that fail
on the old decoder (wrong pixels for grayscale, NULL for 4:2:0) and are
bit-exact with djpeg after the fix. Wire it into `make test`.
(cherry picked from commit 50ed0a4c73)
Cover all pool tail lengths after 2048 selected rows. Repeat the split Metal decode with padded IDs and compare its value projection to a double-precision attention reference. Follow-up to #1051.
Upstream 81a7658 ("Preserve live vision context across server tool calls")
rewrote the live-continuation helpers this branch replaced with
slot_probe_reuse(): image markers carry request-local nonces and are
normalized for visible replay, image tokens are spliced into continuations
instead of being tokenized as text, and image appends are validated by
fingerprint and row count rather than by exact span equality. The merge
keeps the probe architecture and folds the new conditions into it, so slot
routing and the execution ladder still cannot disagree:
- The vision gate is fingerprint-prefix (new
ds4_session_vision_fingerprint_prefix_matches): historical images must
match by row count and fingerprint, positions may shift because
build_live_prompt_suffix() rebases them, and appended images are allowed.
An exact-state gate rejected valid appends, and a position-sensitive one
rejected replays that omit hidden reasoning.
- Every live tier materializes through build_live_prompt_suffix(), so new
images are spliced into the continuation and a failed rebase is a safe
miss rather than a malformed prompt.
- responses-visible and thinking-visible compare in visible_prompt_key
space, so a replay with fresh marker nonces still binds while a changed or
shifted image does not.
- Tool-output-only continuations keep upstream's contract: if the selected
live tier cannot be materialized, the request is answered with 409
(live_continuation_unavailable) instead of being generated from a
context-free prompt built from tool outputs alone.
- Single-session mode refreshes the slot's rendered-text cache like the
batched worker does, so the memory-text tier is probe-authoritative on
both worker paths; the redundant ladder fallback and the four superseded
upstream helpers are removed.
Tests: test_slot_probe_vision_tiers covers image appends, pixel mismatches
and marker-nonce visible replay at the probe/routing level;
test_live_continuation_contract pins the 409 predicate;
tests/test_session_state.c gains fingerprint-prefix cases; and the new live
suite tests/test_server_vision_live_state.py checks slot routing, eviction
victims and the tool-result-only 409 against a real vision model.
Verified on DeepSeek V4 Flash Vision Experimental (M5 Max): ./ds4_test
--server, ds4_agent_test, test-session-state, live test_server_vision_cache.py
(1- and 4-session), test_server_story.py, test_server_batching.py and the new
vision live-state suite all pass.
At sixteen rows the per-token Q8 matvec read every matrix once per four
rows and streamed the batch's activations once per four weight rows: a
third of the step for 3.4 GB of weights. The batch GEMM gives each
simdgroup 32 weight rows and every token: each lane stages one q8_0 block
of its row as floats (the matvec's d * q values) into the simdgroup's own
threadgroup memory, so simdgroups never wait on each other, and the rows
multiply on fp32 simdgroup matrices against token tiles read straight from
the activations. Narrow projections split k across the grid and sum their
planes with the tile reduce. The sums stay within 2e-6 of the matvec's;
tests pin the kernel against a double reference and the matvec.
+10% on the C=16 step of an M5 Max (paired A/B). Every simdgroup matrix
index is a compile-time constant: with the token-tile count a runtime
value the same kernel ran twice slower. DS4_QWEN4_NO_BATCH_MM keeps the
matvec for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Sixteen rows pick ten experts each but only about a third as many
distinct ones, and the per-row kernels read an expert once per row that
chose it. The grouped kernels give each expert to the threadgroup of the
first (row, slot) pair in its list, which walks the expert's rows once per
four pairs with a named accumulator per pair, so every row's block walk and
chain order are the per-row kernel's and its outputs match bit for bit
(tests/test_qwen4_kernels.c pins mid and down under heavy reuse). The
shared expert, read once per row as an extra slot, becomes dense
projections over the batch as in prefill.
Together +7% on the C=16 step of an M5 Max (paired A/B); single rows and
verify rows keep the per-row kernels. DS4_QWEN4_MOE_NO_GROUP and
DS4_QWEN4_MOE_SHARED_DENSE_MIN keep the old paths for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The decode batch ran attention as sixteen chains of per-row dispatches:
prep, block key, score, select, expand, decode and merge for each session
in turn. The rows kernels take the row as a grid axis and reach each
session's caches through a staged table of GPU addresses, so a layer costs
one dispatch per stage. Every kernel keeps the single-row arithmetic, the
row's own split count included, and the single-row kernels call the same
bodies; tests/test_qwen4_kernels.c pins the rows path byte-exact against
the per-row dispatches for dense, block-completing and sparse rows on both
selector widths.
On an M5 Max at C=16, ctx 4096 the step is unchanged (1.00x): those grids
were not latency-bound after all, the attention stages take ~6.5 ms of a
110 ms step. DS4_QWEN4_NO_BATCH_ATTN keeps the per-row path for A/B.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Accumulate dense component products directly in the output and fuse short F16 projections while retaining FP32 correction sums and cuBLAS fallbacks. Tune expert tile sizes by weight format and unpack IQ2 signs in packed bytes without changing represented weights. Keep BF16 n-grams disk-only.
Extend independent kernel references for large outputs, partial tiles, alignment, strides, repeated writes and output guards. Refresh Spark performance references and repeatable sanitizer checks.
Paired Spark runs improve Q2/Q4 large prefill from 685/670 to 745/755 tokens/s, Q2 decoding from 21.0 to 22.6 tokens/s, and Q2 English MTP from 26.8 to 30.3 tokens/s. Official continuation scores remain unchanged; kernel references, sanitizers, session/MTP, mixed-prefill and vision checks pass.
Use tensor-core products with scaled residual correction for FP32 queries and probabilities. Keep scalar and split attention fallbacks, and extend coverage with independent double-precision references, sparse masks, empty rows and head-group tails.
Keep the decode reduction order and verify byte-exact normalized rows and injection partials against one-token controls, including long batches and guarded tensor views.
Collect references from Alibaba through OpenRouter with an explicit empty system message and no provider fallback. Validate token byte alignment and reject lossy Unicode metadata for API agreement while retaining continuation likelihood scoring.
Track 100 short and 12 long reference cases, Q2/Q4 default and quality controls, continued-prefill results, and independent long-context answers. Include collector and scorer regressions and document the hosted-checkpoint and precision limitations.
Share the Metal session graph with CUDA for text and vision, MTP and exact sampling, steering, checkpoints, and disk-only BF16 n-grams. Add bounded concurrent Linux n-gram reads, vectorized dense and expert kernels, register-held recurrent prefill state, and efficient small speculative batches.
Keep Qwen out of DeepSeek batch dispatch and exclude disk n-grams from resident memory accounting. Use normal file-backed weight loading in diagnostics and link n-gram support into the CUDA quality scorer.
Include independent kernel and memory-safety tests, mixed-prefill and recovery checks, long-context server and agent coverage, and user documentation with measured DGX Spark performance.
Use one verified GGUF per quantization, remove sidecar setup, and document disk versus resident size. Record native-reader, model, agent/server, steering, fallback, and continued-prefill QA with explicit limits.
Pass the requested vector and scales into the standalone graph, as the session runner already does. The native-BF16 QA exposed the missing load because ordinary generation silently ignored attention steering. Require the enabled diagnostic and retain exact ordinary/MTP text checks for both Q2 and Q4.
The native-BF16 Q4 fixture exposed half-query rounding amplified by nearly tied MoE routing scores at a four-row tail. CPU double-precision attention checks confirmed the matrix formula, but using decode arithmetic for at most eight rows keeps this boundary stable without changing large-prefill kernels. Extend kernel and full-logit coverage through the eight-row tail.
Keep the table outside model mappings and Metal residency. Use bounded explicit reads, reject failed lookups before GPU consumption, and remove the runtime sidecar option and paging controls. Preserve Qwen hash and speculative state handling; cover malformed files and exact recovery after prefill, decode, and MTP read failures.
Restore matching logits on verifier rewinds, replay stale recurrent state, discard old snapshots on restore, and invalidate partial payload reads. Keep lazy snapshots all-or-nothing and respect small prefill buffers and the unfused diagnostic path.
Preserve schema-declared string arguments. Repair all-row reference dumps and teacher-forced chunking. Add model-backed regressions and fix test-fixture leaks exposed by ASAN.
Reuse the carry workspace for medium appends with a substantial final tile. Preserve the existing 2048-row arithmetic partitions instead of visiting every layer twice. Retain the old dispatch for tiny tails and explicitly smaller prefill chunks.
Extend dispatch and mixed-state controls. Full logits, caches, snapshots and next decode remain exact through 60K context and fresh 49K deferred prefill. Together with read-ahead, the 3241-token README prefill improves from about 55 to 98 tokens/s with a 64 GiB cache hint.
Read one next layer into reserved slots of the admitted expert cache using two 8 MiB staging buffers. Protect current inputs, publish only completed copies, keep unused read-ahead cold, and join before cancellation or cache/model teardown.
Enable read-ahead from 2K tokens; smaller prefills were slower with full-layer reads. Add eviction, truncated-source, buffered-I/O, cancellation and early-exit tests. Validate exact logits and saved state on the Sparks, including Compute Sanitizer.
Add bounded SSD streaming, exact short prefills, accelerated large prefills, and native session batching. Support resident tensor parallel inference across two DGX Sparks over RoCE, with aligned expert shards, split vocabulary projection, decode graphs, and shared-expert overlap.
Include primitive, state, transport, and model regression tests, plus usage documentation and measured QA results.
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars,
Metal text and vision graphs, built-in MTP, and recurrent KV persistence.
Integrate the CLI, agent, server, download targets, and conversion tools.
Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared
quantizer build target. Keep exact-MTP rewind and vision-cache row-width
fixes. Persist valid logits and recurrent state at completed prefill chunks,
and reuse Qwen text-answer history when tools clients omit reasoning.
Retain exact reasoning replay and test continued checkpoints, cancellation,
cache eviction, and restart reuse with and without reasoning_content. Include
reproducible charts and measurements comparing throughput before and after
the checkpoint fixes on M3 Ultra.
Add Qwen trunk steering and prompt activation capture, including GDN layers.
Keep the MTP predictor outside the 48-row bank and preserve prompt dumps
through one-token prefill tails and multi-row speculative verification.
Recover Chrome target creation and retain its last page target. Preserve
full agent transcripts in stripped-session format when KV no longer fits,
while retaining the newer automatic compaction and continuation logic.
Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload
regressions. Adapt the intended changes from fork PRs #13 and #7.
Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs,
tool syntax, session state, and Metal kernel dispatch. Exclude superseded
engine code and prototypes.
Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box>
Co-authored-by: david <contact@davidberard.fr>
Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com>
Co-authored-by: Ogilthorp3 <bertrand@nepveu.name>
Co-authored-by: anemll <realanemll@gmail.com>
Render the existing hint marker with a white-on-teal badge and a slim left rule. Keep body text, redirected output and the model transcript unchanged.
Test every stream split, unfinished markers, quoted continuations, literal code and both tool syntaxes. Verify terminal colors and UTF-8 at multiple widths, including redraws between fragments.
Reserve summary space, recover already-full conversations by summarizing a bounded prefix, and retain every unsummarized token. Reject oversized input without corrupting the current conversation.
Resume unfinished responses in the same assistant message while preserving thinking state and the output budget. Discard incomplete tool calls before compacting and retry without executing partial arguments.
Cover real-model compaction, generated-code correctness, budget accounting, and recovery in a PTY regression; add deterministic boundary tests.
Reuse bounded image embeddings, preserve exact image identities while appending new images, and splice image tokens into authenticated live continuations. Handle image tool results in Responses and Anthropic requests as well as Chat Completions. Normalize request-local image markers for visible replay without trusting text placeholders as image identities.
Add prefix and encoder-cache unit coverage plus real-model image replay and Pi coding-client regressions. Validated GLM 5.3 Flash and DeepSeek Vision-Exp on M5 Max, including 53K/56K prefixes, batched sessions, all three API formats, fresh logit replay, and frontend ASan/UBSan.
Adapt teaching-mode PR #391 to /hints on|off. Append the full
instruction once per context and short state changes at safe
assistant boundaries. New and resumed sessions start off;
compaction reinstates the current policy.
Use ordinary Markdown asides and the existing renderer, without
a learner profile, new tool, or persistent preference.
Tested on M5 Max IT with DeepSeek Flash and GLM Flash coding sessions,
prefix reuse, queued changes, compaction, save/switch/new, frontend
and PTY tests, and agent ASAN/UBSAN checks.
Closes#391
Co-authored-by: Salvatore Sanfilippo <antirez@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Reserve a separate byte for the terminator written by fmemopen, without changing the serialized payload length. Add model-free grow/reuse coverage and compare real-model snapshots against file serialization.
Use a macro for the scheduler policy mock so historical verifier extraction does not fail -Werror on an unused helper. Check the current verifier with sanitizers and confirm that the pre-fix source reproduces the excess committed EOS row.
Mock the existing short-fallback policy referenced by the extracted verifier. Keep scheduler behavior outside this state-ownership fixture, and retain compatibility with pre-fix negative controls.
Preserve all five original commits by Donato Capitella. Resolve the DSpark fallback conflict by retaining the existing Metal/DGX Spark short-proposal behavior and adding the ROCm reference-alignment condition.
Closes#992
Tarjei Mandt
Salvatore Sanfilippo
Emilian Bold
korale77
Mikhail Korobov
davidandClaude Fable 5.1
