Commit Graph
8 Commits
Author SHA1 Message Date
Salvatore Sanfilippo df6eedd972 Add optimized CUDA inference for Qwen3.8 Flash Next
Share the Metal session graph with CUDA for text and vision, MTP and exact sampling, steering, checkpoints, and disk-only BF16 n-grams. Add bounded concurrent Linux n-gram reads, vectorized dense and expert kernels, register-held recurrent prefill state, and efficient small speculative batches.

Keep Qwen out of DeepSeek batch dispatch and exclude disk n-grams from resident memory accounting. Use normal file-backed weight loading in diagnostics and link n-gram support into the CUDA quality scorer.

Include independent kernel and memory-safety tests, mixed-prefill and recovery checks, long-context server and agent coverage, and user documentation with measured DGX Spark performance.
2026-09-15 15:00:11 +02:00
antirez ab06d196da Speed up GLM 5.3 decode on ROCm 2026-08-31 20:14:12 +02:00
antirez b0c31af367 Improve GLM 5.3 attention memory and batching 2026-08-29 02:15:31 +02:00
antirez 6537ba089b Account for batched GLM session memory during placement 2026-08-03 14:48:05 +02:00
antirez a185c36dc4 Stabilize CUDA and Metal session batching 2026-07-18 18:50:59 +02:00
antirez ef8d923ad7 Add ROCm GLM 5.2 support 2026-07-18 18:50:30 +02:00
antirez 66cbce5289 Unify distributed inference CLI and documentation 2026-07-18 18:50:24 +02:00
antirez 36ee8c1407 Add CUDA tensor parallelism and session batching 2026-07-18 18:50:03 +02:00