The CPU-only placement tests also define DS4_ROCM_BUILD, but never use the GPU memory baseline. Exclude its declaration there, matching the existing guards around its uses.
Verified warning-free ROCm applications and test builds on Strix Halo, with placement, memory, session, server and agent tests passing. The production ds4.o is byte-identical after stripping debug metadata.
Retain the measured Metal win without imposing per-token pthread creation on CUDA and ROCm. Tables remain disk-only. Warm M5 Max SSD decode improved from 12.62 to 13.39 tokens/s with byte-identical output; row and error tests also pass under ASAN/UBSAN.
ds4_engram_read_batch() already fans row reads across ENGRAM_READERS
workers, with the comment "Fixed concurrency hides random-read latency
without caching the table" -- but only above a 256-row threshold.
ds41_graph_step() reads DS4_ENGRAM_COLS rows per table (48 rows for the
two tables) through the serial ds4_engram_read(), so ordinary decode
never reached that path. Those 48 preads run in series, and they run
before ds4_gpu_begin_commands(), so the GPU is idle for all of them.
Only bulk prefill ever cleared 256.
This routes the decode read through the batched entry point and lowers
the threshold to 8 so decode-sized batches engage the existing workers.
Same rows, same order, same values -- only the fetch is concurrent.
Measured on a Mac Studio M3 Ultra (80 GPU cores, 256 GB, macOS 26.6.2),
DeepSeek-V4.1-Flash-Q2, resident (no --ssd-streaming), --ctx 131072,
temperature 0, 200 generated tokens, this branch vs its merge-base:
main 19.18, 19.14 tok/s median 19.16
main + patch 20.42, 21.00, 21.11, 21.18 tok/s median 21.06
+9.9%, about 4.7 ms/token. Generated text was byte-identical between
the two builds. (The 20.42 was a cold first run after rebuild.)
The underlying cost is storage latency, not bandwidth -- a standalone
pread benchmark against the same GGUF, 48 random 264-byte rows with
F_NOCACHE and readahead disabled, 150 batches:
serial 5.177 ms/batch (107.9 us/row)
parallel 0.614 ms/batch ( 12.8 us/row)
The saving is roughly constant per token, so it is worth more on a
faster tree: on ivanfioravanti's ds41f-optimizations branch, where this
was originally profiled, the same change measured 26.8 -> 30.5 tok/s
(Engram wall 5.55 -> 1.12 ms/token) over three alternating pairs, also
byte-identical.
Credit where it is due:
- ivanfioravanti, whose V4.1 optimization work is where this was
profiled. His ds41f-dspark branch already carries a better version
of this fix, ds41_engram_parallel(), which reads both tables in a
single dispatch -- but gated to speculative sessions, so ordinary
decode still falls through to the serial loop there too. Ungating
that is probably the better long-term shape of this change.
- The oMLX / drowzeys V4.1-on-Mac work, which is what pointed at
Engram as a decode bottleneck at all. Their approach was a RAM
cache of hot rows; measured row reuse here was only 22.5% over a
600-token coding generation, worth roughly +1 tok/s, so the win
turned out to be concurrency rather than caching. Wrong idea,
right place to look.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 92a61612a2)
Track the pooled selection’s guaranteed prefix instead of promising that all selected rows are valid. Padded selections use the existing bounds-checked kernel.
Adapted from 101d20c23f3a27456bdc61e0da35e2bea97eb04e.
Validation: make -j1 and git diff --check passed on macOS M3 Ultra (Metal build). No model was loaded; GPU runtime and performance tests were not rerun.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 4e166b1fb4)
Upstream 81a7658 ("Preserve live vision context across server tool calls")
rewrote the live-continuation helpers this branch replaced with
slot_probe_reuse(): image markers carry request-local nonces and are
normalized for visible replay, image tokens are spliced into continuations
instead of being tokenized as text, and image appends are validated by
fingerprint and row count rather than by exact span equality. The merge
keeps the probe architecture and folds the new conditions into it, so slot
routing and the execution ladder still cannot disagree:
- The vision gate is fingerprint-prefix (new
ds4_session_vision_fingerprint_prefix_matches): historical images must
match by row count and fingerprint, positions may shift because
build_live_prompt_suffix() rebases them, and appended images are allowed.
An exact-state gate rejected valid appends, and a position-sensitive one
rejected replays that omit hidden reasoning.
- Every live tier materializes through build_live_prompt_suffix(), so new
images are spliced into the continuation and a failed rebase is a safe
miss rather than a malformed prompt.
- responses-visible and thinking-visible compare in visible_prompt_key
space, so a replay with fresh marker nonces still binds while a changed or
shifted image does not.
- Tool-output-only continuations keep upstream's contract: if the selected
live tier cannot be materialized, the request is answered with 409
(live_continuation_unavailable) instead of being generated from a
context-free prompt built from tool outputs alone.
- Single-session mode refreshes the slot's rendered-text cache like the
batched worker does, so the memory-text tier is probe-authoritative on
both worker paths; the redundant ladder fallback and the four superseded
upstream helpers are removed.
Tests: test_slot_probe_vision_tiers covers image appends, pixel mismatches
and marker-nonce visible replay at the probe/routing level;
test_live_continuation_contract pins the 409 predicate;
tests/test_session_state.c gains fingerprint-prefix cases; and the new live
suite tests/test_server_vision_live_state.py checks slot routing, eviction
victims and the tool-result-only 409 against a real vision model.
Verified on DeepSeek V4 Flash Vision Experimental (M5 Max): ./ds4_test
--server, ds4_agent_test, test-session-state, live test_server_vision_cache.py
(1- and 4-session), test_server_story.py, test_server_batching.py and the new
vision live-state suite all pass.
Sixteen speculative cycles before the first plain one made a short
generation on prose pay the losing kind for most of its length (C=16 at
32 steps: 174 tok/s against ~195 once the policy has settled). Eight
cycles give p over 128 draft rows at sixteen streams, enough to price.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
A draft row's cost is not per row: the seventeenth row of a batched cycle
opens a second tile of every dense projection, after which rows are
nearly free (16 rows 80 ms, 23 rows 126, 32 rows 149), so gating streams
by their draft's confidence bought nothing (prose 184.2 vs 183.9 at a
margin of 2, code 210.7 vs 213.6). A batch speculates for everyone or
for nobody, and it pays when (1 + p) c_plain > c_spec.
The engine keeps p and both cycle times as running averages (the same
shape as the session's depth policy), starts by pricing both kinds and
then probes the other kind for two cycles, every thirty-two while plain
and every hundred and twenty-eight while speculating. DS4_QWEN4_SPEC_TRACE
prints the state per cycle.
C=16 prose 184 -> 195 tok/s (plain ~200, mostly plain cycles now), code
213.6 -> 214.3; C=8 prose 131 plain -> 158, code 131 -> 179 (the drafts
fill the one tile a C=8 batch has). C=2 verifies lossless.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The batched cycle drafted with one predictor pass per session, sixteen
passes of the nextn layer over one or two rows each, every one reading
the layer's weights again. One pass now covers every row of the batch:
the trunk streams sit in the batch's R at the positions the trunk rows
had, so the layer's attention takes the same row table as the trunk's,
the staged inputs and residual go where the prompt priming puts them,
and the sessions' last rows are gathered for one pass of the head mixer
before the batched head and the per-session argmax. A rejected draft's
row runs along with its parent as the token that follows; the next pass
rewrites its cache entry before anything reads it.
C=16 code 204 -> 213.5 tok/s cumulative (cycle 156 -> 148 ms), prose
176 -> 183; C=2 verifies lossless against plain greedy.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The speculative batch ran the GDN convolution, scan and the attention of
each session through the single-session kernels, one to two rows at a
time: sixteen sessions, forty-eight layers, ~2300 short dispatches a
cycle. The GDN rows kernels now come in a variant addressed through a
table (state, history and the verify snapshots of each session by GPU
address, its first row and its row count), which walks a session's one or
two tokens in order and writes the snapshot after the first, with the
per-token arithmetic of the verify scan; the attention rows kernels take
one entry per token, so a draft row is simply a second entry at pos + 1
that sees the first token's keys. The trunk of a cycle with no drafts now
costs what a plain step does (89 ms against 92).
With drafts the cycle is 158 ms for 1.50 tokens per stream (152 tok/s at
C=16 on the prose corpus): every draft row costs its own MoE pass, so the
marginal row (~4.4 ms) is close to a plain token (5.6 ms) and greedy
speculation cannot pay at this width on this model unless acceptance is
high. Lossless against plain greedy decode (harness --spec --verify).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
ds4_sessions_eval_batch_speculative_argmax runs one MTP cycle for a batch:
every session contributes its next token and, when the predictor left it a
draft for that token, the draft as a second row. The dense work of the
trunk runs once over all the rows; the recurrent, n-gram and attention
steps run per session over its one or two rows with the single-session
verify kernels, snapshots included, so a session's own arithmetic is that
of the C=1 speculative cycle. A draft stands when it is the target's
argmax for its row; a rejected one restores the row-0 snapshot. The
predictor then runs its layer per session over the committed rows and
every session's head as one product. The Q8 batch GEMM serves 9..32 rows
in sixteen-row halves.
The harness gains --spec (with --verify checking the committed tokens
against plain greedy decode, which speculation must reproduce). On an M5
Max at C=16 the first cut commits 1.49 tokens per stream and cycle on the
prose corpus but the cycle costs 171 ms against 92 for a plain step: the
per-session kernels and the sequential predictor passes are the next
thing to batch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The fp32 tile for F16 batches was meant for decode batches, where the
generic tile's half-staged activations carried rounding into the logits;
its rule also caught every prefill chunk's narrow hyper-connection
projections. There it is slower (1305 vs 1410 tok/s at 4k) and it moved
a single stream's tokens off the reference's: with the generic tile back
for chunks past 64 rows, a C=1 stream generates the same 32 tokens as
upstream main, and prefill returns to 1408 tok/s at 4k, 1455 at 16k.
Decode batches keep the fp32 tile.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The shared transients arena took the shape of the first session, and a
later session needing a wider one (a larger context, hence more indexer
blocks) silently fell back to private transients, ~4-5 GiB each: sixteen
of them at the next context size of a sweep exceeded memory and the
process died without a word. The engine now counts the sessions
borrowing the arena and, when a wider session arrives while none does,
frees the arena so that session builds the wider one; while sessions
borrow it, a wider one keeps private transients as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
The plain batched decode sized its logits scratch to the batch width, so
the Q8 tile's 32-row padding did not fit and the output head, 248320 rows
read once per four tokens by the per-token matvec, cost 3.9 ms of a
76 ms step. The scratch now holds the tile's rows, which the speculative
path already did, and the head takes the tile like every other Q8
projection.
C=16 plain prose 196.9 -> 203 tok/s cumulative; batched vs sequential
unchanged (same worst logit deviation, tokens agree).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Tarjei Mandt
Salvatore Sanfilippo
Jake Maness
Mark Shank
Mikhail Korobov
davidandClaude Fable 5.1