Qwen3.8 sessions were excluded from native session batching: each decode
step ran the whole graph once per session, so sixteen sessions cost
sixteen times the weight traffic. The batched encoder now runs the dense
work (hyper-connection mixers, projections, MoE, output head) once over
the stacked rows and keeps only the recurrent state, the caches and the
PLE history per session, with batched GDN scan and conv kernels reading
the pooled state through per-row slots.
Session transients (prefill scratch) move to one shared arena sized for
the largest session, so each further session costs its caches only
(memory per session divided by ~3.6 at ctx 4096). The dense GEMM gains a
split-K path for the narrow-batch regime, and the dense/MoE/mixer GEMM
thresholds, tuned for T=1 or T>=8192, are moved to serve decode batches
of 4..64 rows. Every step keeps the tokens of the sequential path;
float reduction order changes where the batched kernels do.
C=16 at ctx 4096 on an M5 Max goes from ~50 to ~143 tok/s aggregate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxdJvkAEqcDT2o7tqKMw6d
Use the model-defined 2051-token dense boundary, handle padded selections, and synchronize reused reduction storage in Metal, CUDA and ROCm, including vision normalization.
Enable bounded ROCm selected-attention GEMM for GLM 5.3 without RoPE. Add independent attention references, boundary tests and finite, repeatable vision checks. Focused Strix continued prefill improves from about 21 to 68 tokens per second; stored official scores are unchanged.
Keep the introduction and everyday usage in the README, with setup guides near the top and detailed references at the end. Document Ada/L40S support explicitly and replace stale model, memory, caching and performance claims.
Retain the root Strix Halo page as a link to its new guide.
Tarjei Mandt
Salvatore Sanfilippo
davidandClaude Fable 5.1