Merge antirez/main at 8db1d1d into qwen-kernel-opt
Integrate upstream Qwen CUDA support, Metal multi-session decode and speculative batching, shared prefill workspaces and recurrent-state pools, grouped resident MoE kernels, dense/Q8 batch projections, and server state handling. Retain the Qwen SSD-streaming and Metal kernel optimizations from this branch. Resolve the host/shader MoE argument layout by retaining the compact SSD slot masks and appending the upstream grouped-list fields. Update layout assertions and direct gate/up and down regression fixtures to the resulting 72-byte ABI. Keep SSD address resources and in-flight buffer lifetimes in the common dispatch path alongside the upstream resident row cache. Keep SSD sessions on the ordered ordinary/speculative fallback with private streaming workspaces. Resident sessions use the new upstream shared arena. Preserve the existing SSD prefill tile threshold and guard Metal-only SSD, cache-only attention preparation, and MTP projection calls from CUDA builds. Reconcile retained MTP tail rows and snapshot ownership with native batching: prepare pending predictor boundaries, copy per-session tails and rejection snapshots, maintain tail metadata after commit, and complete the historical cache row after an accepted two-token verification when next-draft creation is skipped. Preserve checkpoint payload compatibility within this branch. Retain both upstream and branch-specific build/test targets. Extend the SSD failure/recovery test to reject native resident dispatch explicitly and verify exact logits across ordered ordinary and speculative batches. Clarify resident versus SSD batching in the Qwen and server documentation. Validation on Apple M1 Max, 32 GiB: - Build ds4, ds4-server, and all selected test binaries without warnings. - Pass Qwen memory planning and compact-MoE CPU tests with ASan/UBSan. - Pass the complete Qwen kernel suite with Metal API validation, including upstream grouped MoE, batch projections, and attention paths. - Pass direct MoE mid (180 fixtures, 1,119,600 comparisons), MoE down (1,650 fixtures, 19,955,180 comparisons), HC (92 fixtures), and Q8 reduction (3,072 cases) regression suites. - Pass SSD expert cache/read/retry tests and MoE specialization tests. - Pass real Qwen Q2 SSD n-gram failure/recovery, ordered batch parity, MTP prefill, snapshot/payload replay, and continuation checks. - Pass a 50-token greedy SSD generation for 'narrami la storia di roma'. CUDA execution and full resident-model multi-session integration were not run on this 32 GiB Mac. This merge does not make new performance claims.
This commit is contained in:
GiorgioOppo committed
commit
f47db4fb83
402 files changed
+15731
-1445
No files matched your search
@@ -72,7 +72,7 @@ endif
|
||||
.PHONY: all help clean test test-rocm test-glm53-kda-rocm test-metal-session-batch test-mxfp4-cuda test-mxfp4-rocm test-cuda-session-batch test-cuda-mixed-batch dspark-acceptance dspark-verify-depth mtp-verify-depth cpu cuda cuda-spark cuda-generic cuda-regression strix-halo rocm
|
||||
|
||||
ifeq ($(UNAME_S),Darwin)
|
||||
.PHONY: metal-decode-schedule-bench metal-prefill-variant-bench check-mxfp4-half-lut
|
||||
.PHONY: metal-decode-schedule-bench metal-prefill-variant-bench session-concurrency-bench check-mxfp4-half-lut
|
||||
.PHONY: test-metal-moe-prefill test-metal-dense-mpp
|
||||
|
||||
all: ds4 ds4-server ds4-bench ds4-eval ds4-agent
|
||||
@@ -83,6 +83,7 @@ help:
|
||||
@echo " make cpu Build CPU-only ./ds4, ./ds4-server, ./ds4-bench, ./ds4-eval, and ./ds4-agent"
|
||||
@echo " make test Build and run tests"
|
||||
@echo " make metal-decode-schedule-bench Build the balanced Metal decode schedule benchmark"
|
||||
@echo " make session-concurrency-bench Build the concurrency x context serving benchmark"
|
||||
@echo " make metal-prefill-variant-bench Build the balanced Metal prefill variant benchmark"
|
||||
@echo " make check-mxfp4-half-lut Verify the checked-in MXFP4 half LUT matches the generator"
|
||||
@echo " make test-mxfp4-metal Check the MXFP4 half LUT, then run Metal MXFP4 exactness tests"
|
||||
@@ -149,6 +150,14 @@ speed-bench/metal_prefill_variant_bench: speed-bench/metal_prefill_variant_bench
|
||||
|
||||
metal-prefill-variant-bench: speed-bench/metal_prefill_variant_bench
|
||||
|
||||
speed-bench/session_concurrency_bench.o: speed-bench/session_concurrency_bench.c ds4.h
|
||||
$(CC) $(CFLAGS) -I. -c -o $@ $<
|
||||
|
||||
speed-bench/session_concurrency_bench: speed-bench/session_concurrency_bench.o $(CORE_OBJS)
|
||||
$(CC) $(CFLAGS) -o $@ $^ $(METAL_LDLIBS)
|
||||
|
||||
session-concurrency-bench: speed-bench/session_concurrency_bench
|
||||
|
||||
tests/test_mxfp4_metal.o: tests/test_mxfp4_metal.c ds4_gpu.h
|
||||
$(CC) $(CFLAGS) -I. -c -o $@ $<
|
||||
|
||||
@@ -227,7 +236,7 @@ check-metal-qwen4-moe-down: tests/test_metal_qwen4_moe_down
|
||||
tests/test_qwen4_ssd_experts.o: tests/test_qwen4_ssd_experts.c ds4_gpu.h
|
||||
$(CC) $(CFLAGS) -fno-fast-math -I. -c -o $@ $<
|
||||
|
||||
tests/ds4_metal_qwen_ssd.o: ds4_metal.m ds4_gpu.h ds4_gpu_tp.h ds4_deepseek41_gpu.h $(METAL_SRCS) tests/qwen4_ssd_pread_probe.h
|
||||
tests/ds4_metal_qwen_ssd.o: ds4_metal.m ds4_gpu.h ds4_gpu_tp.h ds4_deepseek41_gpu.h ds4_qwen4_vision.h $(METAL_SRCS) tests/qwen4_ssd_pread_probe.h
|
||||
$(CC) $(OBJCFLAGS) -include tests/qwen4_ssd_pread_probe.h -c -o $@ ds4_metal.m
|
||||
|
||||
tests/test_qwen4_ssd_experts: tests/test_qwen4_ssd_experts.o $(filter-out ds4_metal.o,$(CORE_OBJS)) tests/ds4_metal_qwen_ssd.o
|
||||
@@ -539,6 +548,16 @@ tests/test_deepseek41_cuda: tests/test_deepseek41_cuda.o ds4_cuda.o ds4_image.o
|
||||
.PHONY: test-deepseek41-cuda
|
||||
test-deepseek41-cuda: tests/test_deepseek41_cuda
|
||||
./tests/test_deepseek41_cuda
|
||||
|
||||
tests/test_qwen4_cuda.o: tests/test_qwen4_kernels.c ds4_gpu.h ds4.h
|
||||
$(CC) $(QUALITY_CFLAGS) -D_GNU_SOURCE -I. -c -o $@ $<
|
||||
|
||||
tests/test_qwen4_cuda: tests/test_qwen4_cuda.o ds4_cuda.o ds4_image.o $(MMQ_OBJS)
|
||||
$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
|
||||
|
||||
.PHONY: test-qwen4-cuda
|
||||
test-qwen4-cuda: tests/test_qwen4_cuda
|
||||
./tests/test_qwen4_cuda
|
||||
endif
|
||||
|
||||
ds4.o: ds4.c ds4.h ds4_ssd.h ds4_distributed.h ds4_gpu.h ds4_gpu_tp.h ds4_deepseek41_gpu.h ds4_linux_memory.h ds4_engram.h
|
||||
@@ -679,11 +698,16 @@ tests/test_qwen4_kernels.o: tests/test_qwen4_kernels.c ds4_gpu.h ds4.h
|
||||
$(QWEN4_KERNEL_TEST): tests/test_qwen4_kernels.o ds4_metal.o ds4_image.o
|
||||
$(CC) $(CFLAGS) -o $@ $^ $(METAL_LDLIBS)
|
||||
|
||||
endif
|
||||
|
||||
tests/test_qwen4_vision.o: tests/test_qwen4_vision.c ds4.h
|
||||
$(CC) $(CFLAGS) -I. -c -o $@ tests/test_qwen4_vision.c
|
||||
|
||||
tests/test_qwen4_vision: tests/test_qwen4_vision.o $(CORE_OBJS)
|
||||
ifeq ($(UNAME_S),Darwin)
|
||||
$(CC) $(CFLAGS) -o $@ $^ $(METAL_LDLIBS)
|
||||
else
|
||||
$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
|
||||
endif
|
||||
|
||||
.PHONY: test-qwen4-kernels test-qwen4-q2 test-qwen4-vision
|
||||
@@ -796,7 +820,11 @@ tests/test_qwen4_ngram_state.o: tests/test_qwen4_ngram_state.c ds4.c ds4.h
|
||||
$(CC) $(filter-out -ffast-math,$(CFLAGS)) -Wno-unused-function -I. -c -o $@ $<
|
||||
|
||||
tests/test_qwen4_ngram_state: tests/test_qwen4_ngram_state.o $(filter-out ds4.o,$(CORE_OBJS))
|
||||
ifeq ($(UNAME_S),Darwin)
|
||||
$(CC) $(filter-out -ffast-math,$(CFLAGS)) -o $@ $^ $(METAL_LDLIBS)
|
||||
else
|
||||
$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
|
||||
endif
|
||||
|
||||
tests/test_qwen4_mtp_prefill.o: tests/test_qwen4_mtp_prefill.c ds4.c ds4.h
|
||||
$(CC) $(filter-out -ffast-math,$(CFLAGS)) -Wno-unused-function -I. -c -o $@ $<
|
||||
@@ -804,7 +832,19 @@ tests/test_qwen4_mtp_prefill.o: tests/test_qwen4_mtp_prefill.c ds4.c ds4.h
|
||||
tests/test_qwen4_mtp_prefill: tests/test_qwen4_mtp_prefill.o $(filter-out ds4.o,$(CORE_OBJS))
|
||||
$(CC) $(filter-out -ffast-math,$(CFLAGS)) -o $@ $^ $(METAL_LDLIBS)
|
||||
|
||||
ds4_cuda.o: ds4_cuda.cu ds4_gpu.h ds4_gpu_tp.h ds4_gpu_mgpu.h ds4_linux_memory.h ds4_deepseek41_gpu.h ds4_deepseek41_cuda.cuh ds4_glm53_vision_gpu.cuh ds4_deepseek4_vision_gpu.cuh ds4_image.h ds4_iq2_tables_cuda.inc cuda/mmq/ds4_mmq.h
|
||||
tests/test_qwen4_prefill.o: tests/test_qwen4_prefill.c ds4.h
|
||||
$(CC) $(QUALITY_CFLAGS) -I. -c -o $@ $<
|
||||
|
||||
tests/test_qwen4_prefill: tests/test_qwen4_prefill.o $(CORE_OBJS)
|
||||
ifeq ($(UNAME_S),Darwin)
|
||||
$(CC) $(CFLAGS) -o $@ $^ $(METAL_LDLIBS)
|
||||
else
|
||||
$(DS4_LINK) -o $@ $^ $(DS4_LINK_LIBS)
|
||||
endif
|
||||
|
||||
ds4.o ds4_cpu.o ds4_cpu_test_hooks.o ds4_cuda_test_hooks.o ds4_metal.o ds4_cuda.o ds4_rocm.o tests/test_qwen4_cuda.o tests/test_qwen4_kernels.o tests/test_qwen4_ngram_state.o tests/test_qwen4_mtp_prefill.o tests/test_qwen4_memory.o: ds4_qwen4_vision.h
|
||||
|
||||
ds4_cuda.o: ds4_cuda.cu ds4_gpu.h ds4_gpu_tp.h ds4_gpu_mgpu.h ds4_linux_memory.h ds4_deepseek41_gpu.h ds4_deepseek41_cuda.cuh ds4_glm53_vision_gpu.cuh ds4_deepseek4_vision_gpu.cuh ds4_qwen4_cuda.cuh ds4_image.h ds4_iq2_tables_cuda.inc cuda/mmq/ds4_mmq.h
|
||||
$(NVCC) $(NVCCFLAGS) -c -o $@ ds4_cuda.cu
|
||||
|
||||
# Vendored mmq pieces (see cuda/mmq/VENDOR.md). ds4_mmq.cu transitively
|
||||
@@ -1139,7 +1179,8 @@ clean:
|
||||
rm -f tests/test_metal_tp_spec
|
||||
rm -f tests/test_metal_tp_cancel
|
||||
rm -f ds4 ds4-server ds4-bench ds4-eval ds4-agent ds4_cpu ds4_native ds4_server_test ds4_test ds4_agent_test gguf-tools/quality-testing/score_official gguf-tools/quality-testing/score_official.o speed-bench/metal_decode_schedule_bench speed-bench/metal_prefill_variant_bench speed-bench/*.o tests/test_q4k_dot tests/test_mxfp4_dot tests/test_mxfp4_metal tests/test_mxfp4_rocm tests/test_mxfp4_cuda tests/test_metal_session_batch tests/test_metal_moe_prefill tests/test_qwen4_moe_mm_specialize tests/test_qwen4_conv_parallel tests/test_q8_prefill_variants tests/test_metal_dense_mpp tests/test_glm53_kda tests/test_glm53_kda_rocm tests/test_glm53_vision_engine tests/test_glm53_vision_prompt tests/test_deepseek4_vision_image tests/test_prompt_prefix tests/test_gpu_xdev tests/test_gpu_model_cache tests/test_gpu_lookup_cache_strict tests/test_engine_mgpu_refusal tests/test_engine_mgpu_runtime tests/test_engine_correctness tests/test_sampling tests/test_cuda_session_batch tests/test_cuda_mixed_batch tests/*.o *.o tests/cuda_long_context_smoke tests/cuda_long_context_smoke.o
|
||||
rm -f tests/test_qwen4_kernels tests/test_qwen4_vision
|
||||
rm -f tests/test_qwen4_kernels tests/test_qwen4_cuda tests/test_qwen4_vision tests/test_qwen4_prefill
|
||||
rm -f speed-bench/session_concurrency_bench
|
||||
rm -f tests/test_metal_q8_reduction
|
||||
rm -f tests/test_metal_qwen4_hc
|
||||
rm -f tests/test_metal_qwen4_moe_mid
|
||||
|
||||
+351
-1066
File diff suppressed because it is too large.
Load diff
@@ -9,7 +9,7 @@ a small native inference engine optimized first for
|
||||
**DeepSeek V4 Flash** (including the experimental vision model),
|
||||
**DeepSeek V4.1 Flash** (Metal, and text inference on CUDA),
|
||||
and additionally **GLM 5.2 and 5.3**, **GLM 5.3 Flash** and
|
||||
**DeepSeek V4 PRO**, and **Qwen3.8 Flash Next** (Metal). The code is self-contained and
|
||||
**DeepSeek V4 PRO**, and **Qwen3.8 Flash Next** (Metal and CUDA). The code is self-contained and
|
||||
deliberately narrow, not a general GGUF runner: you need to use the
|
||||
GGUF files the project produces, that are part of the project
|
||||
itself.
|
||||
|
||||
@@ -60,6 +60,27 @@ runs measured about 400 t/s for a 32K prefill and 290 t/s for an 8K append. The
|
||||
[network TP QA record](../QA_BEFORE_RELEASES.md#cuda-network-tensor-parallelism)
|
||||
has the timing controls and memory limits.
|
||||
|
||||
## Qwen3.8 Flash Next
|
||||
|
||||
Both Q2 and Q4 fit on one Spark. After building:
|
||||
|
||||
```sh
|
||||
./download_model.sh qwen38-q2
|
||||
./ds4 --mtp --ctx 32768
|
||||
```
|
||||
|
||||
Q2 keeps 41.73 GiB of weights resident; Q4 keeps 69.74 GiB. Both GGUFs also
|
||||
contain 95.37 GiB of BF16 n-grams, read directly from the SSD. Leave room
|
||||
for context and runtime buffers. Vision, the native agent, server APIs and
|
||||
text-session checkpoints work on CUDA; see [Qwen setup](QWEN38_FLASH_NEXT.md).
|
||||
|
||||
September 15, 2026, Q2 on one Spark: a 16K prefill reached 250 t/s, extending
|
||||
it to 30K reached 243 t/s, and ordinary decode stayed around 17.3 t/s.
|
||||
These are single runs with 8,192-token prefill chunks and 32 decode tokens.
|
||||
A separate 256-token prose run measured 17.1 t/s ordinarily and 19.9 t/s with
|
||||
MTP, at temperature zero and 8K allocated context. Speculation is opt-in;
|
||||
its benefit depends on the prompt.
|
||||
|
||||
## GLM 5.3 Flash
|
||||
|
||||
Q2 is the resident target for one Spark:
|
||||
|
||||
+1
-1
@@ -128,7 +128,7 @@ allocations add to the main weights, but the n-gram table is not mapped or
|
||||
preloaded. Keep the GGUF on a fast local SSD.
|
||||
The larger `qwen38-q4k` target uses 165.11 GiB on disk and 69.74 GiB for
|
||||
resident weights, before runtime buffers.
|
||||
This model runs on Metal.
|
||||
This model runs on Metal and single-GPU CUDA, including DGX Spark.
|
||||
The script links `ds4flash.gguf` to the combined GGUF:
|
||||
|
||||
```sh
|
||||
|
||||
+91
-13
@@ -3,7 +3,7 @@
|
||||
[Back to README](../README.md)
|
||||
|
||||
Qwen3.8-Flash-Next uses the `qwen4exp` GGUF architecture and a dedicated
|
||||
Metal graph with gated delta-net, gated GQA, block-sparse attention,
|
||||
Metal/CUDA graph with gated delta-net, gated GQA, block-sparse attention,
|
||||
hyper-connections, n-gram embeddings, MoE, and MTP.
|
||||
|
||||
## Download and run
|
||||
@@ -84,18 +84,62 @@ Qwen fills its expert cache on demand, so `--ssd-streaming-cold` does not
|
||||
change its preload behavior. Popularity preloading and the GLM-specific
|
||||
`--ssd-streaming-full-layers` option are not supported for Qwen.
|
||||
|
||||
Use the same model options with `ds4-agent`, `ds4-server`, or
|
||||
`ds4-bench`. Add `--mtp` for speculative decoding using the built-in MTP
|
||||
weights. `--nothink` disables thinking. The server exposes
|
||||
On a DGX Spark, build with `make cuda-spark` instead of `make`. Both Q2 and
|
||||
Q4 fit resident on an otherwise idle 128 GB Spark; the n-grams stay on SSD.
|
||||
The same commands support text, vision, MTP, steering and text-session
|
||||
checkpoints in `ds4`, `ds4-agent` and `ds4-server`. No CUDA-specific model
|
||||
conversion or feature flags are needed.
|
||||
Leave `--prefill-chunk` unset on the Spark to use the faster 8192-token
|
||||
default. The 1024-token example above saves memory on smaller Macs.
|
||||
|
||||
Use the same model options with `ds4-agent` or `ds4-server`.
|
||||
Add `--mtp` for speculative decoding using the built-in MTP weights.
|
||||
`ds4-bench` benchmarks ordinary decoding; it does not accept `--mtp`.
|
||||
`--nothink` disables thinking. The server exposes
|
||||
`qwen3.8-flash-next`, `qwen3.8-flash-next-chat`, and
|
||||
`qwen3.8-flash-next-reasoner` aliases. Tool calls use the native
|
||||
`<tool_call><function=...><parameter=...>` format.
|
||||
|
||||
With resident weights on Metal, `ds4-server --batched-session N` decodes
|
||||
the slots together. Work that
|
||||
only reads weights runs once for the whole batch, while the delta-net
|
||||
recurrence, the attention caches and the n-gram convolution stay per session.
|
||||
The slots also share one prefill workspace, so an extra slot costs its caches
|
||||
rather than another few GiB of transients. As with the other natively batched
|
||||
models, grouping changes the order of floating point reductions, so a batched
|
||||
reply can differ from the same prompt decoded alone.
|
||||
Add `--mtp` to use batched speculative decoding as well. The server accepts
|
||||
matching greedy drafts by default, including at nonzero temperature. With
|
||||
`--mtp-exact-sampling`, sampled requests use ordinary batched decoding;
|
||||
temperature-zero requests can still speculate. Default greedy acceptance also
|
||||
means a seed need not reproduce a reply under a different batching schedule.
|
||||
Speculative batches above 16 sessions, images and steering use the ordered
|
||||
fallback. Metal SSD expert streaming also uses the ordered fallback and
|
||||
keeps each session's streaming workspace; it does not use the resident batch
|
||||
arena. CUDA currently decodes sessions in order.
|
||||
|
||||
Disk KV checkpoints include recurrent state. Rewinding to an earlier position
|
||||
replays the retained prefix on the next evaluation. The native context is
|
||||
262144 tokens; `DS4_QWEN4_YARN_FACTOR=2` or `=4` enables static YaRN for
|
||||
longer contexts, with a possible quality cost on shorter prompts.
|
||||
|
||||
## DGX Spark performance
|
||||
|
||||
Measured on a single Spark with resident weights and disk-only n-grams:
|
||||
|
||||
| Model | First 1024 tokens | Next 7168 tokens | Decode at 8192 tokens |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| Q2 | 516 t/s | 745 t/s | 22.6 t/s |
|
||||
| Q4 | 513 t/s | 755 t/s | 21.2 t/s |
|
||||
|
||||
These are averages of two runs using `speed-bench/promessi_sposi.txt`,
|
||||
8192-token prefill chunks and 128 teacher-forced decode tokens, without MTP.
|
||||
Loading the model is excluded. Q2 also reached about 771 t/s on a fresh
|
||||
32K-token prefix with `--prefill-chunk 32768`. Small continuations have lower
|
||||
throughput: adding 32 tokens after an 8K Q4 prefix took about 273 ms.
|
||||
MTP speed depends on how often drafts are accepted; the table measures
|
||||
ordinary decoding.
|
||||
|
||||
## Conversion
|
||||
|
||||
See [GGUF conversion](../gguf-tools/README.md) for conversion from the HF
|
||||
@@ -107,17 +151,16 @@ Expanding an old quantized table to BF16 does not restore its lost precision.
|
||||
|
||||
## Vision
|
||||
|
||||
Images go through the model's Qwen3-VL vision tower. Download llama.cpp's
|
||||
mmproj file with `./download_model.sh qwen38-vision` (it comes from
|
||||
`ggml-org/Qwen3.8-Flash-Next-GGUF`; alternatively run
|
||||
`convert_hf_to_gguf.py --mmproj --outtype f16` on the checkpoint) with
|
||||
`--vision` and send `image_url` parts as usual. Each image is resized to
|
||||
Images go through the model's Qwen3-VL vision tower. Download the encoder
|
||||
with `./download_model.sh qwen38-vision`, then use
|
||||
`--vision gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf`. The CLI accepts
|
||||
`/read image.png`; the server accepts `image_url` parts. Each image is resized to
|
||||
multiples of 32 pixels within 64 to 1024 tokens (`DS4_QWEN4_IMAGE_MAX_TOKENS`
|
||||
raises the cap), encoded on the GPU, and takes the model's 3D rope positions;
|
||||
live KV reuse keys on the image fingerprints. `make test-qwen4-vision`
|
||||
compares the tower with the Hugging Face implementation using the same GGUF
|
||||
weights, dequantized to float32. It separately reports GGUF-versus-original
|
||||
checkpoint quality and Metal-versus-original agreement. Both comparisons use
|
||||
checkpoint quality and GPU-versus-original agreement. Both comparisons use
|
||||
the unchanged minimum per-token cosine threshold of 0.99; implementation
|
||||
parity gates the default exit status. Add `--require-quality` to also fail on
|
||||
GGUF-versus-original quality loss. A passing implementation check alone does
|
||||
@@ -162,11 +205,12 @@ The test checks that all four turns complete in each mode, including errors
|
||||
that the interactive CLI can report without a nonzero process exit. It saves
|
||||
the responses and diagnostics for inspection; it does not grade image content.
|
||||
|
||||
Metal only for now. The Metal graph accepts Q8_0, Q4_0, F16, BF16 and F32
|
||||
The Metal and CUDA graphs accept Q8_0, Q4_0, F16, BF16 and F32
|
||||
dense weights, Q8_0/MXFP4/Q4_0/Q4_K/Q2_K/IQ2_XXS experts, F16/F32/Q8_0
|
||||
hyper-connection mixers and the original BF16 n-gram table.
|
||||
Tensor parallelism and pipeline execution are not implemented for this model
|
||||
yet. CPU code is a correctness reference, not a general inference backend.
|
||||
yet. SSD expert streaming is supported on Metal only. ROCm is not supported.
|
||||
CPU code is a correctness reference, not a general inference backend.
|
||||
|
||||
|
||||
## Validation
|
||||
@@ -197,7 +241,10 @@ optional shared experts and all 16 routed slots. A failed SSD read must leave
|
||||
the completed cached/shared gate/up slots exact, preserve the untouched slots,
|
||||
and allow an exact retry after draining GPU work.
|
||||
The memory-estimate test runs without a GPU or model.
|
||||
Run `tests/test_qwen4_ngram_state MODEL.gguf` under Metal validation to check
|
||||
On CUDA, use `make test-qwen4-cuda` for the kernel tests. They compare the
|
||||
active kernels with independent CPU references, without model weights.
|
||||
Vision and end-to-end checks additionally require the checkpoints above.
|
||||
Run `tests/test_qwen4_ngram_state MODEL.gguf` on either backend to check
|
||||
failed disk reads during prefill, decode and MTP, then exact recovery.
|
||||
For a smaller memory footprint, run
|
||||
`tests/test_qwen4_ngram_state MODEL.gguf --ssd-streaming`. This uses a
|
||||
@@ -217,6 +264,22 @@ and exact-sampling continuation checks. It uses a 128-token context and a
|
||||
1024-expert cache request. The small-buffer MTP CLI regression also accepts
|
||||
`--ssd-streaming`.
|
||||
|
||||
Official Alibaba continuations are tracked for 100 short prompts and 12
|
||||
archive/code prompts from 2K to 24K tokens. Build the quality scorer, then run
|
||||
from the repository root:
|
||||
|
||||
```sh
|
||||
gguf-tools/quality-testing/score_official MODEL.gguf \
|
||||
gguf-tools/quality-testing/data/qwen38-flash-alibaba-100/manifest.tsv /tmp/qwen-short.tsv 4096
|
||||
gguf-tools/quality-testing/score_official MODEL.gguf \
|
||||
gguf-tools/quality-testing/data/qwen38-flash-alibaba-long/manifest.tsv /tmp/qwen-long.tsv 32768
|
||||
```
|
||||
|
||||
Repeat with `--quality` and, for the long set, `--continued-prefill 1` and
|
||||
`--continued-prefill 256`. These fixtures match the no-thinking template;
|
||||
no rendered-prompt flag is needed. See [quality testing](../gguf-tools/quality-testing/README.md)
|
||||
for collection settings, measurements and the hosted-checkpoint limitations.
|
||||
|
||||
[Checkpoint-fix benchmark charts and measurements](../speed-bench/qwen38-checkpoints/README.md)
|
||||
compare prefill, ordinary decode, and MTP decode against the preceding PR head.
|
||||
|
||||
@@ -249,6 +312,21 @@ prefill with teacher-forced decode across a chunk boundary. For official
|
||||
continuation scoring, use `--rendered-prompt`
|
||||
when a fixture already contains the complete model chat template.
|
||||
|
||||
`tests/test_qwen4_prefill MODEL PROMPT 8192` checks mixed prefill sizes,
|
||||
progress callbacks and exact replay through the sparse-attention boundary.
|
||||
It also reports differences against a fresh prefill followed by individual
|
||||
decodes. Those schedules can round differently and exchange nearly tied
|
||||
experts; their full logits need not match. Use model-quality checks as well,
|
||||
not only state replay:
|
||||
|
||||
```sh
|
||||
python3 tests/test_server_story.py --url http://127.0.0.1:8000 \
|
||||
--model qwen3.8-flash-next --output /tmp/qwen-story
|
||||
```
|
||||
|
||||
Start the server with at least `--ctx 49152`. This checks all sixteen facts
|
||||
in a 31K-token story, then a correction turn that must reuse the prefix.
|
||||
|
||||
Qwen directional steering and activation capture support all 48 trunk layers;
|
||||
see [directional steering](../dir-steering/README.md). Model-backed regressions:
|
||||
|
||||
|
||||
+13
-1
@@ -60,6 +60,12 @@ preallocates independent KV states and queues requests when all slots are busy.
|
||||
Choose context and slot count together: a context that fits once may not fit
|
||||
four times. Idle slots can be cached before reuse; active requests are not evicted.
|
||||
|
||||
Where the model supports it, the slots share one prefill workspace instead of
|
||||
each keeping its own, so an extra slot costs only its caches. That matters most
|
||||
for Qwen3.8 Flash Next, whose transients are sized by the prefill chunk rather
|
||||
than by the context: at the default chunk they run to several GiB per session.
|
||||
The startup line reports both figures.
|
||||
|
||||
| Backend/model | Decode execution |
|
||||
| --- | --- |
|
||||
| Metal, resident Flash | Native shared-expert/QKV batching where supported |
|
||||
@@ -68,7 +74,9 @@ four times. Idle slots can be cached before reuse; active requests are not evict
|
||||
| Metal SSD streaming, V4.1 Flash | Ordered fallback |
|
||||
| Metal, GLM 5.2 | Ordered fallback |
|
||||
| Metal, GLM 5.3 | Native batching through 2051 visible tokens; ordered fallback afterward |
|
||||
| Metal, Qwen3.8 Flash Next with resident weights | Native batching of the shared work; recurrent state, caches and PLE history stay per session |
|
||||
| CUDA, supported multi-GPU Flash TP layout | Native grouped decode and mixed prefill/decode |
|
||||
| Metal, Qwen3.8 Flash Next with SSD expert streaming | Ordered fallback with per-session streaming workspaces |
|
||||
| Single-GPU CUDA, including Spark | Ordered fallback |
|
||||
|
||||
Fallback executes the rows separately. It provides concurrency and scheduling
|
||||
@@ -78,7 +86,11 @@ use the ordered fallback.
|
||||
|
||||
Long prefills yield to active decoders in bounded intervals, normally 128
|
||||
tokens. `--mixed-prefill-quantum N` changes that interval for testing.
|
||||
Session-batched serving uses ordinary target decoding, not MTP/DSpark.
|
||||
Session-batched serving uses ordinary target decoding, except Qwen3.8 on
|
||||
Metal, where `--mtp` also batches speculative decoding with resident weights
|
||||
and uses ordered speculative cycles with SSD expert streaming. Its
|
||||
`--mtp-exact-sampling` mode uses ordinary batches for nonzero-temperature
|
||||
requests. Other models do not use MTP/DSpark while session batching is active.
|
||||
For the eight-L40S example, see [CUDA GPUs](CUDA_MULTI_GPU.md#serve-multiple-users).
|
||||
|
||||
## Images
|
||||
|
||||
@@ -68,7 +68,7 @@ Both Qwen downloads include MTP and native BF16 n-grams in the main GGUF:
|
||||
|
||||
Ordinary decode uses the same file with `--mtp` omitted. For non-zero
|
||||
temperature, add `--mtp-exact-sampling` to preserve the target sampling
|
||||
distribution. See [Qwen setup](QWEN38_FLASH_NEXT.md) for the Metal runtime.
|
||||
distribution. See [Qwen setup](QWEN38_FLASH_NEXT.md) for Metal and CUDA.
|
||||
|
||||
For the Q2 model with Metal SSD streaming:
|
||||
|
||||
|
||||
@@ -526,6 +526,16 @@ typedef struct {
|
||||
* sequential fallback. */
|
||||
int ds4_sessions_eval_batch(ds4_decode_item *items, int count,
|
||||
char *err, size_t errlen);
|
||||
/* One speculative cycle for a batch of sessions (greedy acceptance, Qwen3.8
|
||||
* with --mtp): each item feeds its token; a pending draft rides along as a
|
||||
* second row and is committed when it is the target's argmax. accepted[i]
|
||||
* lists the tokens committed for item i (the fed token, then the draft) and
|
||||
* n_accepted[i] how many; the session's logits then follow its last
|
||||
* committed token. Engines without native batching run one cycle per
|
||||
* session in turn. */
|
||||
int ds4_sessions_eval_batch_speculative_argmax(ds4_decode_item *items, int count,
|
||||
int (*accepted)[2], int *n_accepted,
|
||||
char *err, size_t errlen);
|
||||
/* Advance one resumed prefill suffix and an independent decode batch as one
|
||||
* scheduling step. Unsupported combinations use the ordinary serialized
|
||||
* session operations. */
|
||||
|
||||
@@ -34036,3 +34036,4 @@ extern "C" int ds4_gpu_tp_big_gate_wait(uint64_t seq) {
|
||||
#include "ds4_glm53_vision_gpu.cuh"
|
||||
#include "ds4_deepseek4_vision_gpu.cuh"
|
||||
#include "ds4_deepseek41_cuda.cuh"
|
||||
#include "ds4_qwen4_cuda.cuh"
|
||||
@@ -3,6 +3,7 @@
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include "ds4_qwen4_vision.h"
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
@@ -267,6 +268,82 @@ static inline int ds4_gpu_device_is_m5_apple_silicon(void) { return 0; }
|
||||
#endif
|
||||
void ds4_gpu_set_streaming_expert_cache_budget(uint32_t experts);
|
||||
void ds4_gpu_set_streaming_expert_cache_expert_bytes(uint64_t bytes);
|
||||
int ds4_gpu_qwen4_conv_stream_rows_tensor(
|
||||
ds4_gpu_tensor *x, ds4_gpu_tensor *hist_pool,
|
||||
const void *model_map, uint64_t model_size, uint64_t weight_offset,
|
||||
const uint32_t *slots, uint32_t n_rows, uint32_t n_channels,
|
||||
uint32_t conv_kernel, uint32_t state_stride, uint32_t x_stride, int apply_silu);
|
||||
int ds4_gpu_qwen4_gdn_scan_rows_tensor(
|
||||
ds4_gpu_tensor *out, ds4_gpu_tensor *state_pool,
|
||||
const ds4_gpu_tensor *qkv, const ds4_gpu_tensor *ga, const ds4_gpu_tensor *gb,
|
||||
const uint32_t *slots, uint32_t n_rows,
|
||||
uint32_t n_k_head, uint32_t n_v_head, uint32_t head_dim,
|
||||
uint32_t state_stride, uint32_t qkv_stride, uint32_t out_stride);
|
||||
/* One row of a Qwen3.8 decode batch for the attention rows kernels: the
|
||||
* session's caches for the layer, its rope positions, its token position and
|
||||
* whether it attends through the block selection or densely. */
|
||||
typedef struct {
|
||||
ds4_gpu_tensor *k_cache, *v_cache, *ik_cache, *block_key;
|
||||
const ds4_gpu_tensor *pos3;
|
||||
uint32_t pos;
|
||||
int use_sel;
|
||||
} ds4_gpu_qwen4_attn_row;
|
||||
#define DS4_GPU_QWEN4_ATTN_ROW_BYTES 64u /* one staged table entry */
|
||||
/* One session of a decode batch for the GDN rows kernels: its state and
|
||||
* history, optional snapshots of both after its first token, its first row
|
||||
* and its row count (one or two). */
|
||||
typedef struct {
|
||||
ds4_gpu_tensor *state, *hist, *snap_state, *snap_hist;
|
||||
uint32_t row0, n_tok;
|
||||
} ds4_gpu_qwen4_gdn_row;
|
||||
#define DS4_GPU_QWEN4_GDN_ROW_BYTES 48u
|
||||
int ds4_gpu_qwen4_gdn_rows_stage(ds4_gpu_tensor *table, uint64_t entry0,
|
||||
const ds4_gpu_qwen4_gdn_row *rows, uint32_t n_rows);
|
||||
int ds4_gpu_qwen4_gdn_scan_rows2_tensor(
|
||||
ds4_gpu_tensor *out, const ds4_gpu_tensor *qkv, const ds4_gpu_tensor *ga, const ds4_gpu_tensor *gb,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_gdn_row *rows, uint32_t n_rows,
|
||||
uint32_t n_batch_rows, uint32_t n_k_head, uint32_t n_v_head, uint32_t head_dim,
|
||||
uint32_t qkv_stride, uint32_t out_stride);
|
||||
int ds4_gpu_qwen4_conv_stream_rows2_tensor(
|
||||
ds4_gpu_tensor *x, const void *model_map, uint64_t model_size, uint64_t weight_offset,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_gdn_row *rows, uint32_t n_rows,
|
||||
uint32_t n_batch_rows, uint32_t n_channels, uint32_t conv_kernel, uint32_t x_stride, int apply_silu);
|
||||
int ds4_gpu_qwen4_attn_rows_stage(ds4_gpu_tensor *table, uint64_t entry0,
|
||||
const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows, uint32_t ratio);
|
||||
int ds4_gpu_qwen4_attn_prep_rows_tensor(
|
||||
ds4_gpu_tensor *q_out, ds4_gpu_tensor *gate_out, ds4_gpu_tensor *iq_out,
|
||||
const ds4_gpu_tensor *qg, const ds4_gpu_tensor *kproj, const ds4_gpu_tensor *vproj,
|
||||
const ds4_gpu_tensor *iq, const ds4_gpu_tensor *ik,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows,
|
||||
const void *model_map, uint64_t model_size,
|
||||
uint64_t g_q_offset, uint64_t g_k_offset, uint64_t g_iq_offset,
|
||||
uint32_t n_head, uint32_t n_head_kv, uint32_t head_dim, uint32_t n_rot,
|
||||
uint32_t n_idx_head, uint32_t idx_dim, float rope_base, float eps);
|
||||
int ds4_gpu_qwen4_idx_block_key_rows_tensor(
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows,
|
||||
const void *model_map, uint64_t model_size, uint64_t g_ik_offset,
|
||||
uint32_t ratio, uint32_t idx_dim, uint32_t n_rot, float rope_base, float eps);
|
||||
int ds4_gpu_qwen4_idx_score_rows_tensor(
|
||||
ds4_gpu_tensor *score, ds4_gpu_tensor *tile_max, const ds4_gpu_tensor *iq,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows,
|
||||
uint32_t n_block_stride, uint32_t n_idx_head, uint32_t idx_dim, uint32_t ratio);
|
||||
int ds4_gpu_qwen4_idx_select_rows_tensor(
|
||||
ds4_gpu_tensor *sel, const ds4_gpu_tensor *score, const ds4_gpu_tensor *tile_max,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows,
|
||||
uint32_t n_block_stride, uint32_t top_k);
|
||||
int ds4_gpu_qwen4_idx_expand_rows_tensor(
|
||||
ds4_gpu_tensor *sel_tokens, ds4_gpu_tensor *n_sel, const ds4_gpu_tensor *sel_blocks,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, uint32_t n_rows,
|
||||
uint32_t n_sel_blocks, uint32_t ratio, uint32_t sel_stride);
|
||||
int ds4_gpu_qwen4_attn_decode_rows_tensor(
|
||||
ds4_gpu_tensor *out, const ds4_gpu_tensor *q, const ds4_gpu_tensor *gate,
|
||||
const ds4_gpu_tensor *sel_tokens, const ds4_gpu_tensor *n_sel, ds4_gpu_tensor *part,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows,
|
||||
uint32_t n_head, uint32_t n_head_kv, uint32_t head_dim, uint32_t sel_stride, float scale);
|
||||
int ds4_gpu_qwen4_batch_mm_q8_tensor(
|
||||
ds4_gpu_tensor *out, const ds4_gpu_tensor *x,
|
||||
const void *model_map, uint64_t model_size, uint64_t weight_offset,
|
||||
uint32_t n_tokens, uint32_t in_dim, uint32_t out_rows);
|
||||
uint64_t ds4_gpu_recommended_working_set_size(void);
|
||||
uint32_t ds4_gpu_stream_expert_cache_configured_count(void);
|
||||
uint32_t ds4_gpu_stream_expert_cache_current_count(void);
|
||||
@@ -594,8 +671,8 @@ int ds4_gpu_matmul_q8_0_tensor(
|
||||
uint64_t out_dim,
|
||||
const ds4_gpu_tensor *x,
|
||||
uint64_t n_tok);
|
||||
#if defined(__APPLE__)
|
||||
/* Qwen prefill: bounded Q8 unpack defaults on for M3 Ultra. */
|
||||
#if !defined(DS4_ROCM_BUILD)
|
||||
/* Qwen projections preserve FP32 activations on both GPU backends. */
|
||||
int ds4_gpu_qwen4_matmul_q8_0_tensor(
|
||||
ds4_gpu_tensor *out,
|
||||
const void *model_map,
|
||||
@@ -3406,6 +3483,18 @@ int ds4_gpu_qwen4_moe_down_tensor(
|
||||
uint32_t weight_type, uint32_t n_total_expert, uint32_t n_tokens, uint32_t n_slots,
|
||||
uint32_t ff_dim, uint32_t out_dim,
|
||||
uint64_t shared_down_offset, uint32_t shared_type);
|
||||
int ds4_gpu_qwen4_moe_mid_grouped_tensor(
|
||||
ds4_gpu_tensor *mid, const ds4_gpu_tensor *x, const ds4_gpu_tensor *selected,
|
||||
const ds4_gpu_tensor *lists, const ds4_gpu_tensor *counts, uint32_t list_cap,
|
||||
const void *model_map, uint64_t model_size, uint64_t gate_offset, uint64_t up_offset,
|
||||
uint32_t weight_type, uint32_t n_total_expert, uint32_t n_tokens, uint32_t n_slots,
|
||||
uint32_t in_dim, uint32_t ff_dim);
|
||||
int ds4_gpu_qwen4_moe_down_grouped_tensor(
|
||||
ds4_gpu_tensor *part, const ds4_gpu_tensor *mid, const ds4_gpu_tensor *selected,
|
||||
const ds4_gpu_tensor *lists, const ds4_gpu_tensor *counts, uint32_t list_cap,
|
||||
const void *model_map, uint64_t model_size, uint64_t down_offset,
|
||||
uint32_t weight_type, uint32_t n_total_expert, uint32_t n_tokens, uint32_t n_slots,
|
||||
uint32_t ff_dim, uint32_t out_dim);
|
||||
/* shared_gate NULL: no shared expert; shared NULL: the shared output is part
|
||||
* slot n_slots, otherwise `shared` [T][dim] holds it. part_stride = slots per
|
||||
* token in part. */
|
||||
@@ -3455,18 +3544,6 @@ int ds4_gpu_qwen4_gdn_front_tensor(
|
||||
uint32_t weight_type, uint32_t n_tokens, uint32_t n_k_head, uint32_t n_v_head, uint32_t head_dim,
|
||||
uint32_t conv_kernel, uint32_t in_dim, ds4_gpu_tensor *snap_state, uint32_t snap_tok,
|
||||
ds4_gpu_tensor *snap2_state, uint32_t snap2_tok);
|
||||
#define DS4_QWEN4_VISION_LAYERS 27
|
||||
typedef struct {
|
||||
uint64_t ln1_w, ln1_b, qkv_w, qkv_b, out_w, out_b, ln2_w, ln2_b, up_w, up_b, down_w, down_b;
|
||||
uint32_t qkv_type, out_type, up_type, down_type;
|
||||
} ds4_qwen4_vision_layer_weights;
|
||||
typedef struct {
|
||||
uint64_t patch_w0, patch_w1, patch_b, pos_embd, post_ln_w, post_ln_b, mm0_w, mm0_b, mm2_w, mm2_b;
|
||||
uint32_t mm0_type, mm2_type, patch_type;
|
||||
uint32_t n_embd, n_ff, n_head, n_patch, n_merge, n_pos_side, n_out;
|
||||
float eps;
|
||||
ds4_qwen4_vision_layer_weights layer[DS4_QWEN4_VISION_LAYERS];
|
||||
} ds4_qwen4_vision_weights;
|
||||
/* Encode one image: patches [n_patches][3*P*P] in 2x2 window order plus the
|
||||
* resampled position embedding [n_patches][n_embd]; out receives
|
||||
* [n_patches/4][n_out]. Weights are read from the mapped mmproj GGUF. */
|
||||
|
||||
+638
-16
@@ -11534,6 +11534,7 @@ int ds4_gpu_synchronize(void) {
|
||||
}
|
||||
|
||||
static void qwen4_nax_release_scratch(void);
|
||||
static void qwen4_batch_release_scratch(void);
|
||||
|
||||
void ds4_gpu_cleanup(void) {
|
||||
if (!g_initialized) return;
|
||||
@@ -11824,6 +11825,7 @@ void ds4_gpu_cleanup(void) {
|
||||
g_model_mapped_size = 0;
|
||||
g_model_mapped_max_tensor_bytes = 0;
|
||||
qwen4_nax_release_scratch();
|
||||
qwen4_batch_release_scratch();
|
||||
ds4_gpu_tensor_tracking_reset();
|
||||
g_flash_attn_mask_bytes = 0;
|
||||
g_flash_attn_zero_mask_bytes = 0;
|
||||
@@ -48402,6 +48404,8 @@ enum {
|
||||
QWEN4_K_MOE_DOWN,
|
||||
QWEN4_K_MOE_DOWN_Q2K,
|
||||
QWEN4_K_MOE_DOWN_MXFP4_PF,
|
||||
QWEN4_K_MOE_MID_Q4K_GROUPED,
|
||||
QWEN4_K_MOE_DOWN_MXFP4_GROUPED,
|
||||
QWEN4_K_MOE_REDUCE,
|
||||
QWEN4_K_HC_COMBINE_NORM,
|
||||
QWEN4_K_ARGMAX,
|
||||
@@ -48431,6 +48435,22 @@ enum {
|
||||
QWEN4_K_MOE_MM_DOWN_NAXC64,
|
||||
QWEN4_K_ROWS_F32_TO_F16,
|
||||
QWEN4_K_DENSE_MM,
|
||||
QWEN4_K_DENSE_MM_REDUCE,
|
||||
QWEN4_K_BATCH_MM_Q8_T1,
|
||||
QWEN4_K_BATCH_MM_Q8_T2,
|
||||
QWEN4_K_GDN_SCAN_ROWS,
|
||||
QWEN4_K_CONV_STREAM_ROWS,
|
||||
QWEN4_K_GDN_SCAN_ROWS2,
|
||||
QWEN4_K_CONV_STREAM_ROWS2,
|
||||
QWEN4_K_ATTN_PREP_ROWS,
|
||||
QWEN4_K_IDX_BLOCK_KEY_ROWS,
|
||||
QWEN4_K_IDX_SCORE_ROWS,
|
||||
QWEN4_K_IDX_SELECT_ROWS,
|
||||
QWEN4_K_IDX_EXPAND_ROWS,
|
||||
QWEN4_K_ATTN_DECODE_ROWS_NPT8,
|
||||
QWEN4_K_ATTN_DECODE_ROWS_NPT4,
|
||||
QWEN4_K_ATTN_MERGE_ROWS_NPT8,
|
||||
QWEN4_K_ATTN_MERGE_ROWS_NPT4,
|
||||
QWEN4_K_HC_LO_ACT,
|
||||
QWEN4_K_HC_MIX_ROWS,
|
||||
QWEN4_K_VIS_PATCH_FINISH,
|
||||
@@ -48494,6 +48514,8 @@ static const char *const qwen4_kernel_names[QWEN4_K_COUNT] = {
|
||||
"kernel_qwen4_moe_down",
|
||||
"kernel_qwen4_moe_down_q2k",
|
||||
"kernel_qwen4_moe_down_mxfp4_pf",
|
||||
"kernel_qwen4_moe_mid_q4k_grouped",
|
||||
"kernel_qwen4_moe_down_mxfp4_grouped",
|
||||
"kernel_qwen4_moe_reduce",
|
||||
"kernel_qwen4_hc_combine_norm_f16",
|
||||
"kernel_qwen4_argmax",
|
||||
@@ -48523,6 +48545,22 @@ static const char *const qwen4_kernel_names[QWEN4_K_COUNT] = {
|
||||
"kernel_qwen4_moe_mm_down_naxc64",
|
||||
"kernel_qwen4_rows_f32_to_f16",
|
||||
"kernel_qwen4_dense_mm",
|
||||
"kernel_qwen4_dense_mm_reduce",
|
||||
"kernel_qwen4_batch_mm_q8_t1",
|
||||
"kernel_qwen4_batch_mm_q8_t2",
|
||||
"kernel_qwen4_gdn_scan_rows",
|
||||
"kernel_qwen4_conv_stream_rows",
|
||||
"kernel_qwen4_gdn_scan_rows2",
|
||||
"kernel_qwen4_conv_stream_rows2",
|
||||
"kernel_qwen4_attn_prep_rows",
|
||||
"kernel_qwen4_idx_block_key_rows",
|
||||
"kernel_qwen4_idx_score_rows",
|
||||
"kernel_qwen4_idx_select_rows",
|
||||
"kernel_qwen4_idx_expand_rows",
|
||||
"kernel_qwen4_attn_decode_rows_npt8",
|
||||
"kernel_qwen4_attn_decode_rows_npt4",
|
||||
"kernel_qwen4_attn_merge_rows_npt8",
|
||||
"kernel_qwen4_attn_merge_rows_npt4",
|
||||
"kernel_qwen4_hc_lo_act",
|
||||
"kernel_qwen4_hc_mix_rows",
|
||||
"kernel_qwen4_vis_patch_finish",
|
||||
@@ -48588,13 +48626,19 @@ static uint32_t qwen4_moe_mm_dispatch_experts(qwen4_moe_mm_args *args) {
|
||||
static id<MTLComputePipelineState> g_qwen4_pipelines[QWEN4_K_COUNT];
|
||||
#define QWEN4_ATTN_NSG 4
|
||||
#define QWEN4_ATTN_MAX_SPLITS 64
|
||||
#define QWEN4_ATTN_ROWS_MAX 64 /* decode batch rows the attention rows kernels take */
|
||||
|
||||
typedef struct {
|
||||
uint32_t n_tokens, n_slots, in_dim, out_rows, weight_type, row_bytes;
|
||||
uint64_t expert_bytes;
|
||||
uint32_t has_shared, shared_type, shared_row_bytes, n_total_expert;
|
||||
uint32_t slot_mask[3], masked_tokens;
|
||||
uint32_t list_cap, pad0;
|
||||
} qwen4_moe_args;
|
||||
_Static_assert(sizeof(qwen4_moe_args) == 72u, "Qwen MoE host/Metal argument layout");
|
||||
_Static_assert(offsetof(qwen4_moe_args, slot_mask) == 48u, "Qwen MoE slot-mask offset");
|
||||
_Static_assert(offsetof(qwen4_moe_args, masked_tokens) == 60u, "Qwen MoE masked-token offset");
|
||||
_Static_assert(offsetof(qwen4_moe_args, list_cap) == 64u, "Qwen MoE grouped-list offset");
|
||||
|
||||
/* A zero mask skips that token when masking is enabled. The rectangular
|
||||
* grid uses the widest row; narrower rows exit before accessing weights. */
|
||||
@@ -48634,9 +48678,13 @@ static uint32_t qwen4_moe_mv_groups(uint32_t type) {
|
||||
return (uint32_t)ds4_gpu_env_u64("DS4_QWEN4_MOE_MV_NSG", default_nsg, 1u, 16u);
|
||||
}
|
||||
|
||||
static int qwen4_dispatch(int kernel, const void *args, size_t args_len,
|
||||
const qwen4_bind *binds, int n_binds,
|
||||
MTLSize grid, MTLSize tg, NSUInteger tg_mem) {
|
||||
/* resident: buffers the kernel reaches through GPU addresses rather than
|
||||
* bindings (the decode batch's per-session caches); they are made resident
|
||||
* for this dispatch. */
|
||||
static int qwen4_dispatch_resident(int kernel, const void *args, size_t args_len,
|
||||
const qwen4_bind *binds, int n_binds,
|
||||
MTLSize grid, MTLSize tg, NSUInteger tg_mem,
|
||||
const qwen4_bind *resident, uint32_t n_resident) {
|
||||
if (!g_initialized && !ds4_gpu_init()) return 0;
|
||||
@autoreleasepool {
|
||||
const bool addresses = g_qwen4_stream_weights && g_qwen4_stream_weights->addresses;
|
||||
@@ -48747,12 +48795,24 @@ static int qwen4_dispatch(int kernel, const void *args, size_t args_len,
|
||||
[enc setBuffer:binds[i].buf offset:binds[i].off atIndex:(NSUInteger)(i + 1)];
|
||||
}
|
||||
if (tg_mem) [enc setThreadgroupMemoryLength:tg_mem atIndex:0];
|
||||
if (n_resident) {
|
||||
id<MTLResource> res[QWEN4_ATTN_ROWS_MAX * 5u];
|
||||
if (n_resident > sizeof(res) / sizeof(res[0])) return 0;
|
||||
for (uint32_t i = 0; i < n_resident; i++) res[i] = resident[i].buf;
|
||||
[enc useResources:res count:n_resident usage:MTLResourceUsageRead | MTLResourceUsageWrite];
|
||||
}
|
||||
[enc dispatchThreadgroups:grid threadsPerThreadgroup:tg];
|
||||
ds4_gpu_end_compute_encoder(cb, enc);
|
||||
return ds4_gpu_finish_command_buffer(cb, owned, qwen4_kernel_names[kernel]);
|
||||
}
|
||||
}
|
||||
|
||||
static int qwen4_dispatch(int kernel, const void *args, size_t args_len,
|
||||
const qwen4_bind *binds, int n_binds,
|
||||
MTLSize grid, MTLSize tg, NSUInteger tg_mem) {
|
||||
return qwen4_dispatch_resident(kernel, args, args_len, binds, n_binds, grid, tg, tg_mem, NULL, 0u);
|
||||
}
|
||||
|
||||
/* bytes of one hc mixer row of n elements: f16, f32 or q8_0 */
|
||||
static uint64_t qwen4_hc_row_bytes(uint32_t weight_type, uint64_t n) {
|
||||
return weight_type == 1u ? n * 2u : weight_type == 0u ? n * 4u : weight_type == 8u ? (n / 32u) * 34u : 0u;
|
||||
@@ -49055,6 +49115,179 @@ int ds4_gpu_qwen4_gdn_scan_tensor(
|
||||
MTLSizeMake(head_dim, n_v_head, 1), MTLSizeMake(32, 1, 1), 0);
|
||||
}
|
||||
|
||||
static ds4_gpu_tensor *g_qwen4_gdn_slots;
|
||||
|
||||
/* One decode step for every row of a batch against its own slot of the state
|
||||
* pool. The grid keeps the per-row kernel's shape and simd-group grouping, so
|
||||
* each row's arithmetic is unchanged and the result is bit-identical; the rows
|
||||
* merely share a dispatch long enough to reach steady state. */
|
||||
int ds4_gpu_qwen4_gdn_scan_rows_tensor(
|
||||
ds4_gpu_tensor *out, ds4_gpu_tensor *state_pool,
|
||||
const ds4_gpu_tensor *qkv, const ds4_gpu_tensor *ga, const ds4_gpu_tensor *gb,
|
||||
const uint32_t *slots, uint32_t n_rows,
|
||||
uint32_t n_k_head, uint32_t n_v_head, uint32_t head_dim,
|
||||
uint32_t state_stride, uint32_t qkv_stride, uint32_t out_stride) {
|
||||
if (!g_initialized && !ds4_gpu_init()) return 0;
|
||||
if (head_dim != 128u || n_rows == 0u || n_rows > 64u) return 0;
|
||||
|
||||
struct { uint32_t n_rows, n_k_head, n_v_head, head_dim, state_stride, qkv_stride, out_stride, pad0; } args =
|
||||
{ n_rows, n_k_head, n_v_head, head_dim, state_stride, qkv_stride, out_stride, 0 };
|
||||
|
||||
const uint64_t slot_bytes = (uint64_t)n_rows * sizeof(uint32_t);
|
||||
if (!g_qwen4_gdn_slots) {
|
||||
g_qwen4_gdn_slots = ds4_gpu_tensor_alloc(64u * sizeof(uint32_t));
|
||||
if (!g_qwen4_gdn_slots) return 0;
|
||||
}
|
||||
if (ds4_gpu_tensor_write(g_qwen4_gdn_slots, 0, slots, slot_bytes) == 0) return 0;
|
||||
|
||||
qwen4_bind b[6];
|
||||
if (!qwen4_bind_tensor(&b[0], qkv, (uint64_t)n_rows * qkv_stride * sizeof(float), "gdn rows qkv") ||
|
||||
!qwen4_bind_tensor(&b[1], ga, (uint64_t)n_rows * n_v_head * sizeof(float), "gdn rows decay") ||
|
||||
!qwen4_bind_tensor(&b[2], gb, (uint64_t)n_rows * n_v_head * sizeof(float), "gdn rows beta") ||
|
||||
!qwen4_bind_tensor(&b[3], state_pool, 0, "gdn rows state pool") ||
|
||||
!qwen4_bind_tensor(&b[4], out, (uint64_t)n_rows * out_stride * sizeof(float), "gdn rows output") ||
|
||||
!qwen4_bind_tensor(&b[5], g_qwen4_gdn_slots, slot_bytes, "gdn rows slots")) {
|
||||
return 0;
|
||||
}
|
||||
/* Simd groups own independent value rows, so their grouping changes the
|
||||
* dispatch shape without touching a row's recurrent arithmetic. */
|
||||
const uint32_t nsg = 1u;
|
||||
return qwen4_dispatch(QWEN4_K_GDN_SCAN_ROWS, &args, sizeof(args), b, 6,
|
||||
MTLSizeMake((head_dim + 4u * nsg - 1u) / (4u * nsg), n_v_head, n_rows),
|
||||
MTLSizeMake(32u * nsg, 1, 1), 0);
|
||||
}
|
||||
|
||||
/* One decode token per batch row, each against its own slot of the history
|
||||
* pool; see kernel_qwen4_conv_stream_rows. */
|
||||
int ds4_gpu_qwen4_conv_stream_rows_tensor(
|
||||
ds4_gpu_tensor *x, ds4_gpu_tensor *hist_pool,
|
||||
const void *model_map, uint64_t model_size, uint64_t weight_offset,
|
||||
const uint32_t *slots, uint32_t n_rows, uint32_t n_channels,
|
||||
uint32_t conv_kernel, uint32_t state_stride, uint32_t x_stride, int apply_silu) {
|
||||
if (!g_initialized && !ds4_gpu_init()) return 0;
|
||||
if (n_rows == 0u || n_rows > 64u || conv_kernel < 2u || conv_kernel > 4u) return 0;
|
||||
|
||||
struct { uint32_t n_rows, n_channels, conv_kernel, apply_silu, state_stride, x_stride, pad0, pad1; } args =
|
||||
{ n_rows, n_channels, conv_kernel, apply_silu ? 1u : 0u, state_stride, x_stride, 0, 0 };
|
||||
|
||||
const uint64_t slot_bytes = (uint64_t)n_rows * sizeof(uint32_t);
|
||||
if (!g_qwen4_gdn_slots) {
|
||||
g_qwen4_gdn_slots = ds4_gpu_tensor_alloc(64u * sizeof(uint32_t));
|
||||
if (!g_qwen4_gdn_slots) return 0;
|
||||
}
|
||||
if (ds4_gpu_tensor_write(g_qwen4_gdn_slots, 0, slots, slot_bytes) == 0) return 0;
|
||||
|
||||
qwen4_bind b[4];
|
||||
if (!qwen4_bind_tensor(&b[0], x, (uint64_t)n_rows * x_stride * sizeof(float), "conv rows x") ||
|
||||
!qwen4_bind_tensor(&b[1], hist_pool, 0, "conv rows history pool") ||
|
||||
!qwen4_bind_weight(&b[2], model_map, model_size, weight_offset,
|
||||
(uint64_t)n_channels * conv_kernel * sizeof(float), "conv rows weight") ||
|
||||
!qwen4_bind_tensor(&b[3], g_qwen4_gdn_slots, slot_bytes, "conv rows slots")) {
|
||||
return 0;
|
||||
}
|
||||
return qwen4_dispatch(QWEN4_K_CONV_STREAM_ROWS, &args, sizeof(args), b, 4,
|
||||
MTLSizeMake((n_channels + 255u) / 256u, n_rows, 1), MTLSizeMake(256, 1, 1), 0);
|
||||
}
|
||||
|
||||
/* ---- GDN rows by address: one or two tokens per session ---------------- */
|
||||
|
||||
static uint64_t qwen4_tensor_address(const ds4_gpu_tensor *t);
|
||||
|
||||
typedef struct {
|
||||
uint64_t state, hist, snap_state, snap_hist;
|
||||
uint32_t row0, n_tok, pad0, pad1;
|
||||
} qwen4_gdn_row_entry;
|
||||
|
||||
int ds4_gpu_qwen4_gdn_rows_stage(ds4_gpu_tensor *table, uint64_t entry0,
|
||||
const ds4_gpu_qwen4_gdn_row *rows, uint32_t n_rows) {
|
||||
if (!g_initialized && !ds4_gpu_init()) return 0;
|
||||
if (!rows || n_rows == 0u || n_rows > QWEN4_ATTN_ROWS_MAX) return 0;
|
||||
qwen4_gdn_row_entry e[QWEN4_ATTN_ROWS_MAX];
|
||||
for (uint32_t i = 0; i < n_rows; i++) {
|
||||
e[i].state = qwen4_tensor_address(rows[i].state);
|
||||
e[i].hist = qwen4_tensor_address(rows[i].hist);
|
||||
e[i].snap_state = qwen4_tensor_address(rows[i].snap_state);
|
||||
e[i].snap_hist = qwen4_tensor_address(rows[i].snap_hist);
|
||||
if (!e[i].state || !e[i].hist || (rows[i].snap_state && !e[i].snap_state) ||
|
||||
(rows[i].snap_hist && !e[i].snap_hist) || rows[i].n_tok == 0u || rows[i].n_tok > 2u) {
|
||||
return 0;
|
||||
}
|
||||
e[i].row0 = rows[i].row0;
|
||||
e[i].n_tok = rows[i].n_tok;
|
||||
e[i].pad0 = e[i].pad1 = 0u;
|
||||
}
|
||||
return ds4_gpu_tensor_write(table, entry0 * sizeof(e[0]), e, (uint64_t)n_rows * sizeof(e[0]));
|
||||
}
|
||||
|
||||
static uint32_t qwen4_gdn_rows_resident(qwen4_bind *out, const ds4_gpu_qwen4_gdn_row *rows, uint32_t n_rows) {
|
||||
uint32_t n = 0;
|
||||
for (uint32_t i = 0; i < n_rows; i++) {
|
||||
const ds4_gpu_tensor *ts[4] = { rows[i].state, rows[i].hist, rows[i].snap_state, rows[i].snap_hist };
|
||||
for (int j = 0; j < 4; j++) {
|
||||
if (!ts[j]) continue;
|
||||
out[n].buf = ds4_gpu_tensor_buffer(ts[j]);
|
||||
out[n].off = 0;
|
||||
n++;
|
||||
}
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
static bool qwen4_bind_gdn_rows(qwen4_bind *b, const ds4_gpu_tensor *table, uint64_t entry0, uint32_t n_rows) {
|
||||
const uint64_t off = entry0 * sizeof(qwen4_gdn_row_entry);
|
||||
if (!table || !ds4_gpu_tensor_buffer(table) ||
|
||||
ds4_gpu_tensor_bytes(table) < off + (uint64_t)n_rows * sizeof(qwen4_gdn_row_entry)) {
|
||||
fprintf(stderr, "ds4: Qwen3.8 GDN row table is missing or undersized\n");
|
||||
return false;
|
||||
}
|
||||
b->buf = ds4_gpu_tensor_buffer(table);
|
||||
b->off = ds4_gpu_tensor_offset(table) + (NSUInteger)off;
|
||||
return true;
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_gdn_scan_rows2_tensor(
|
||||
ds4_gpu_tensor *out, const ds4_gpu_tensor *qkv, const ds4_gpu_tensor *ga, const ds4_gpu_tensor *gb,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_gdn_row *rows, uint32_t n_rows,
|
||||
uint32_t n_batch_rows, uint32_t n_k_head, uint32_t n_v_head, uint32_t head_dim,
|
||||
uint32_t qkv_stride, uint32_t out_stride) {
|
||||
if (!g_initialized && !ds4_gpu_init()) return 0;
|
||||
if (head_dim != 128u || n_rows == 0u || n_rows > QWEN4_ATTN_ROWS_MAX) return 0;
|
||||
struct { uint32_t n_rows, n_k_head, n_v_head, head_dim, state_stride, qkv_stride, out_stride, pad0; } args =
|
||||
{ n_rows, n_k_head, n_v_head, head_dim, 0u, qkv_stride, out_stride, 0 };
|
||||
qwen4_bind b[5], res[QWEN4_ATTN_ROWS_MAX * 4u];
|
||||
if (!qwen4_bind_tensor(&b[0], qkv, (uint64_t)n_batch_rows * qkv_stride * sizeof(float), "gdn rows qkv") ||
|
||||
!qwen4_bind_tensor(&b[1], ga, (uint64_t)n_batch_rows * n_v_head * sizeof(float), "gdn rows decay") ||
|
||||
!qwen4_bind_tensor(&b[2], gb, (uint64_t)n_batch_rows * n_v_head * sizeof(float), "gdn rows beta") ||
|
||||
!qwen4_bind_tensor(&b[3], out, (uint64_t)n_batch_rows * out_stride * sizeof(float), "gdn rows output") ||
|
||||
!qwen4_bind_gdn_rows(&b[4], table, entry0, n_rows)) {
|
||||
return 0;
|
||||
}
|
||||
const uint32_t nsg = 1u;
|
||||
return qwen4_dispatch_resident(QWEN4_K_GDN_SCAN_ROWS2, &args, sizeof(args), b, 5,
|
||||
MTLSizeMake((head_dim + 4u * nsg - 1u) / (4u * nsg), n_v_head, n_rows),
|
||||
MTLSizeMake(32u * nsg, 1, 1), 0, res, qwen4_gdn_rows_resident(res, rows, n_rows));
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_conv_stream_rows2_tensor(
|
||||
ds4_gpu_tensor *x, const void *model_map, uint64_t model_size, uint64_t weight_offset,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_gdn_row *rows, uint32_t n_rows,
|
||||
uint32_t n_batch_rows, uint32_t n_channels, uint32_t conv_kernel, uint32_t x_stride, int apply_silu) {
|
||||
if (!g_initialized && !ds4_gpu_init()) return 0;
|
||||
if (n_rows == 0u || n_rows > QWEN4_ATTN_ROWS_MAX || conv_kernel < 2u || conv_kernel > 4u) return 0;
|
||||
struct { uint32_t n_rows, n_channels, conv_kernel, apply_silu, state_stride, x_stride, pad0, pad1; } args =
|
||||
{ n_rows, n_channels, conv_kernel, apply_silu ? 1u : 0u, 0u, x_stride, 0, 0 };
|
||||
qwen4_bind b[3], res[QWEN4_ATTN_ROWS_MAX * 4u];
|
||||
if (!qwen4_bind_tensor(&b[0], x, (uint64_t)n_batch_rows * x_stride * sizeof(float), "conv rows x") ||
|
||||
!qwen4_bind_weight(&b[1], model_map, model_size, weight_offset,
|
||||
(uint64_t)n_channels * conv_kernel * sizeof(float), "conv rows weight") ||
|
||||
!qwen4_bind_gdn_rows(&b[2], table, entry0, n_rows)) {
|
||||
return 0;
|
||||
}
|
||||
return qwen4_dispatch_resident(QWEN4_K_CONV_STREAM_ROWS2, &args, sizeof(args), b, 3,
|
||||
MTLSizeMake((n_channels + 255u) / 256u, n_rows, 1), MTLSizeMake(256, 1, 1), 0,
|
||||
res, qwen4_gdn_rows_resident(res, rows, n_rows));
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_gdn_out_tensor(
|
||||
ds4_gpu_tensor *o, const ds4_gpu_tensor *z,
|
||||
const void *model_map, uint64_t model_size, uint64_t weight_offset,
|
||||
@@ -49387,14 +49620,11 @@ int ds4_gpu_qwen4_idx_expand_tensor(
|
||||
}
|
||||
|
||||
/* keys per split of the partial-softmax decode path (DS4_QWEN4_ATTN_SPLIT_KEYS overrides) */
|
||||
/* Read per call, not cached, so an A/B harness can switch it per step. */
|
||||
static uint32_t qwen4_attn_split_keys(void) {
|
||||
static uint32_t keys;
|
||||
if (!keys) {
|
||||
const char *env = getenv("DS4_QWEN4_ATTN_SPLIT_KEYS");
|
||||
const int v = env ? atoi(env) : 0;
|
||||
keys = v > 0 ? (uint32_t)v : 32u;
|
||||
}
|
||||
return keys;
|
||||
const char *env = getenv("DS4_QWEN4_ATTN_SPLIT_KEYS");
|
||||
const int v = env ? atoi(env) : 0;
|
||||
return v > 0 ? (uint32_t)v : 32u;
|
||||
}
|
||||
|
||||
uint64_t ds4_gpu_qwen4_attn_part_floats(uint32_t n_tokens, uint32_t n_head, uint32_t head_dim) {
|
||||
@@ -49479,6 +49709,229 @@ int ds4_gpu_qwen4_attn_decode_tensor(
|
||||
MTLSizeMake(n_head, n_tokens, 1), MTLSizeMake(32, 1, 1), 0);
|
||||
}
|
||||
|
||||
/* ---- decode batch: attention over rows ----
|
||||
*
|
||||
* The caches stay per session, so the rows kernels find each row's through
|
||||
* a table of GPU addresses (one entry per row, staged per layer) and the
|
||||
* dispatch makes those buffers resident. Every kernel keeps the single-row
|
||||
* arithmetic, so a row's outputs match the per-row dispatches bit for bit. */
|
||||
|
||||
typedef struct {
|
||||
uint64_t k_cache, v_cache, ik_cache, block_key, pos3;
|
||||
uint32_t pos, n_blocks, use_sel, pad0;
|
||||
} qwen4_attn_row_entry;
|
||||
|
||||
static uint64_t qwen4_tensor_address(const ds4_gpu_tensor *t) {
|
||||
return t ? ds4_gpu_buffer_address(ds4_gpu_tensor_buffer(t), ds4_gpu_tensor_offset(t)) : 0u;
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_attn_rows_stage(ds4_gpu_tensor *table, uint64_t entry0,
|
||||
const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows, uint32_t ratio) {
|
||||
if (!g_initialized && !ds4_gpu_init()) return 0;
|
||||
if (!rows || n_rows == 0u || n_rows > QWEN4_ATTN_ROWS_MAX || ratio == 0u) return 0;
|
||||
qwen4_attn_row_entry e[QWEN4_ATTN_ROWS_MAX];
|
||||
for (uint32_t i = 0; i < n_rows; i++) {
|
||||
e[i].k_cache = qwen4_tensor_address(rows[i].k_cache);
|
||||
e[i].v_cache = qwen4_tensor_address(rows[i].v_cache);
|
||||
e[i].ik_cache = qwen4_tensor_address(rows[i].ik_cache);
|
||||
e[i].block_key = qwen4_tensor_address(rows[i].block_key);
|
||||
e[i].pos3 = qwen4_tensor_address(rows[i].pos3);
|
||||
if (!e[i].k_cache || !e[i].v_cache || !e[i].ik_cache || !e[i].block_key || !e[i].pos3) {
|
||||
fprintf(stderr, "ds4: Qwen3.8 attention rows need GPU buffer addresses (macOS 13+)\n");
|
||||
return 0;
|
||||
}
|
||||
e[i].pos = rows[i].pos;
|
||||
e[i].n_blocks = (rows[i].pos + 1u) / ratio;
|
||||
e[i].use_sel = rows[i].use_sel ? 1u : 0u;
|
||||
e[i].pad0 = 0u;
|
||||
}
|
||||
return ds4_gpu_tensor_write(table, entry0 * sizeof(e[0]), e, (uint64_t)n_rows * sizeof(e[0]));
|
||||
}
|
||||
|
||||
static bool qwen4_bind_rows(qwen4_bind *b, const ds4_gpu_tensor *table, uint64_t entry0, uint32_t n_rows) {
|
||||
const uint64_t off = entry0 * sizeof(qwen4_attn_row_entry);
|
||||
if (!table || !ds4_gpu_tensor_buffer(table) ||
|
||||
ds4_gpu_tensor_bytes(table) < off + (uint64_t)n_rows * sizeof(qwen4_attn_row_entry)) {
|
||||
fprintf(stderr, "ds4: Qwen3.8 attention row table is missing or undersized\n");
|
||||
return false;
|
||||
}
|
||||
b->buf = ds4_gpu_tensor_buffer(table);
|
||||
b->off = ds4_gpu_tensor_offset(table) + (NSUInteger)off;
|
||||
return true;
|
||||
}
|
||||
|
||||
static uint32_t qwen4_rows_resident(qwen4_bind *out, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows) {
|
||||
uint32_t n = 0;
|
||||
for (uint32_t i = 0; i < n_rows; i++) {
|
||||
const ds4_gpu_tensor *ts[5] = { rows[i].k_cache, rows[i].v_cache, rows[i].ik_cache,
|
||||
rows[i].block_key, rows[i].pos3 };
|
||||
for (int j = 0; j < 5; j++) { out[n].buf = ds4_gpu_tensor_buffer(ts[j]); out[n].off = 0; n++; }
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_attn_prep_rows_tensor(
|
||||
ds4_gpu_tensor *q_out, ds4_gpu_tensor *gate_out, ds4_gpu_tensor *iq_out,
|
||||
const ds4_gpu_tensor *qg, const ds4_gpu_tensor *kproj, const ds4_gpu_tensor *vproj,
|
||||
const ds4_gpu_tensor *iq, const ds4_gpu_tensor *ik,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows,
|
||||
const void *model_map, uint64_t model_size,
|
||||
uint64_t g_q_offset, uint64_t g_k_offset, uint64_t g_iq_offset,
|
||||
uint32_t n_head, uint32_t n_head_kv, uint32_t head_dim, uint32_t n_rot,
|
||||
uint32_t n_idx_head, uint32_t idx_dim, float rope_base, float eps) {
|
||||
/* The caller checks each row's position against its own cache; the
|
||||
* kernel does not read cache_cap. */
|
||||
struct {
|
||||
uint32_t n_tokens, n_head, n_head_kv, head_dim, n_rot, n_idx_head, idx_dim, pos0, cache_cap;
|
||||
float rope_base, eps; uint32_t pad0; float rope_mscale; float rope_freq[32];
|
||||
} args = { n_rows, n_head, n_head_kv, head_dim, n_rot, n_idx_head, idx_dim, 0u, 0u,
|
||||
rope_base, eps, 0, 1.0f, { 0 } };
|
||||
qwen4_rope_fill(args.rope_freq, &args.rope_mscale, n_rot, rope_base);
|
||||
const uint64_t q_bytes = (uint64_t)n_rows * n_head * head_dim * sizeof(float);
|
||||
const uint64_t kv_bytes = (uint64_t)n_rows * n_head_kv * head_dim * sizeof(float);
|
||||
const uint64_t iq_bytes = (uint64_t)n_rows * n_idx_head * idx_dim * sizeof(float);
|
||||
qwen4_bind b[12], res[QWEN4_ATTN_ROWS_MAX * 5u];
|
||||
if (n_rows == 0 || n_rows > QWEN4_ATTN_ROWS_MAX || head_dim < 32 || head_dim > 256 || (head_dim % 32) != 0 ||
|
||||
idx_dim < 32 || idx_dim > 128 || (idx_dim % 32) != 0 || n_rot > 64 || (n_rot % 2) != 0 ||
|
||||
n_rot > idx_dim || n_head_kv == 0 || (n_head % n_head_kv) != 0) {
|
||||
return 0;
|
||||
}
|
||||
if (!qwen4_bind_tensor(&b[0], qg, 2u * q_bytes, "attn rows q/gate projection") ||
|
||||
!qwen4_bind_tensor(&b[1], kproj, kv_bytes, "attn rows k projection") ||
|
||||
!qwen4_bind_tensor(&b[2], vproj, kv_bytes, "attn rows v projection") ||
|
||||
!qwen4_bind_tensor(&b[3], iq, iq_bytes, "indexer rows q projection") ||
|
||||
!qwen4_bind_tensor(&b[4], ik, (uint64_t)n_rows * idx_dim * sizeof(float), "indexer rows k projection") ||
|
||||
!qwen4_bind_weight(&b[5], model_map, model_size, g_q_offset, (uint64_t)head_dim * sizeof(float), "attn q_norm") ||
|
||||
!qwen4_bind_weight(&b[6], model_map, model_size, g_k_offset, (uint64_t)head_dim * sizeof(float), "attn k_norm") ||
|
||||
!qwen4_bind_weight(&b[7], model_map, model_size, g_iq_offset, (uint64_t)idx_dim * sizeof(float), "indexer q_norm") ||
|
||||
!qwen4_bind_tensor(&b[8], q_out, q_bytes, "attn rows q") ||
|
||||
!qwen4_bind_tensor(&b[9], gate_out, q_bytes, "attn rows gate") ||
|
||||
!qwen4_bind_tensor(&b[10], iq_out, iq_bytes, "indexer rows q") ||
|
||||
!qwen4_bind_rows(&b[11], table, entry0, n_rows)) {
|
||||
return 0;
|
||||
}
|
||||
return qwen4_dispatch_resident(QWEN4_K_ATTN_PREP_ROWS, &args, sizeof(args), b, 12,
|
||||
MTLSizeMake(n_head + n_head_kv + n_idx_head + 1u, n_rows, 1), MTLSizeMake(32, 1, 1), 0,
|
||||
res, qwen4_rows_resident(res, rows, n_rows));
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_idx_block_key_rows_tensor(
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows,
|
||||
const void *model_map, uint64_t model_size, uint64_t g_ik_offset,
|
||||
uint32_t ratio, uint32_t idx_dim, uint32_t n_rot, float rope_base, float eps) {
|
||||
struct { uint32_t block0, n_blocks, ratio, idx_dim, n_rot; float rope_base, eps; uint32_t pad0;
|
||||
float rope_mscale; float rope_freq[32]; } args =
|
||||
{ 0u, n_rows, ratio, idx_dim, n_rot, rope_base, eps, 0, 1.0f, { 0 } };
|
||||
qwen4_rope_fill(args.rope_freq, &args.rope_mscale, n_rot, rope_base);
|
||||
qwen4_bind b[2], res[QWEN4_ATTN_ROWS_MAX * 5u];
|
||||
if (n_rows == 0 || n_rows > QWEN4_ATTN_ROWS_MAX || ratio == 0 || idx_dim < 32 || idx_dim > 128 ||
|
||||
(idx_dim % 32) != 0 || n_rot > idx_dim ||
|
||||
!qwen4_bind_weight(&b[0], model_map, model_size, g_ik_offset, (uint64_t)idx_dim * sizeof(float), "indexer k_norm") ||
|
||||
!qwen4_bind_rows(&b[1], table, entry0, n_rows)) {
|
||||
return 0;
|
||||
}
|
||||
return qwen4_dispatch_resident(QWEN4_K_IDX_BLOCK_KEY_ROWS, &args, sizeof(args), b, 2,
|
||||
MTLSizeMake(n_rows, 1, 1), MTLSizeMake(32, 1, 1), 0,
|
||||
res, qwen4_rows_resident(res, rows, n_rows));
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_idx_score_rows_tensor(
|
||||
ds4_gpu_tensor *score, ds4_gpu_tensor *tile_max, const ds4_gpu_tensor *iq,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows,
|
||||
uint32_t n_block_stride, uint32_t n_idx_head, uint32_t idx_dim, uint32_t ratio) {
|
||||
struct { uint32_t n_tokens, n_blocks, n_idx_head, idx_dim, pos0, ratio, pad0, pad1; } args =
|
||||
{ n_rows, n_block_stride, n_idx_head, idx_dim, 0u, ratio, 0, 0 };
|
||||
const uint32_t n_tiles = (n_block_stride + 7u) / 8u;
|
||||
qwen4_bind b[4], res[QWEN4_ATTN_ROWS_MAX * 5u];
|
||||
if (n_rows == 0 || n_rows > QWEN4_ATTN_ROWS_MAX || n_block_stride == 0 || ratio == 0 ||
|
||||
n_idx_head * idx_dim > 512u || (idx_dim & 3u) != 0u ||
|
||||
!qwen4_bind_tensor(&b[0], iq, (uint64_t)n_rows * n_idx_head * idx_dim * sizeof(float), "indexer rows q") ||
|
||||
!qwen4_bind_tensor(&b[1], score, (uint64_t)n_rows * n_block_stride * sizeof(float), "indexer rows scores") ||
|
||||
!qwen4_bind_tensor(&b[2], tile_max, (uint64_t)n_rows * n_tiles * sizeof(uint32_t), "indexer rows tile maxima") ||
|
||||
!qwen4_bind_rows(&b[3], table, entry0, n_rows)) {
|
||||
return 0;
|
||||
}
|
||||
return qwen4_dispatch_resident(QWEN4_K_IDX_SCORE_ROWS, &args, sizeof(args), b, 4,
|
||||
MTLSizeMake((n_block_stride + 127) / 128, n_rows, 1), MTLSizeMake(128, 1, 1), 0,
|
||||
res, qwen4_rows_resident(res, rows, n_rows));
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_idx_select_rows_tensor(
|
||||
ds4_gpu_tensor *sel, const ds4_gpu_tensor *score, const ds4_gpu_tensor *tile_max,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows,
|
||||
uint32_t n_block_stride, uint32_t top_k) {
|
||||
struct { uint32_t n_tokens, n_blocks, top_k, pad0; } args = { n_rows, n_block_stride, top_k, 0 };
|
||||
const uint32_t n_tiles = (n_block_stride + 7u) / 8u;
|
||||
qwen4_bind b[4];
|
||||
if (n_rows == 0 || n_rows > QWEN4_ATTN_ROWS_MAX || top_k == 0 || top_k > n_block_stride ||
|
||||
!qwen4_bind_tensor(&b[0], score, (uint64_t)n_rows * n_block_stride * sizeof(float), "indexer rows scores") ||
|
||||
!qwen4_bind_tensor(&b[1], tile_max, (uint64_t)n_rows * n_tiles * sizeof(uint32_t), "indexer rows tile maxima") ||
|
||||
!qwen4_bind_tensor(&b[2], sel, (uint64_t)n_rows * top_k * sizeof(int32_t), "rows selected blocks") ||
|
||||
!qwen4_bind_rows(&b[3], table, entry0, n_rows)) {
|
||||
return 0;
|
||||
}
|
||||
(void)rows;
|
||||
return qwen4_dispatch(QWEN4_K_IDX_SELECT_ROWS, &args, sizeof(args), b, 4,
|
||||
MTLSizeMake(n_rows, 1, 1), MTLSizeMake(1024, 1, 1), 0);
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_idx_expand_rows_tensor(
|
||||
ds4_gpu_tensor *sel_tokens, ds4_gpu_tensor *n_sel, const ds4_gpu_tensor *sel_blocks,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, uint32_t n_rows,
|
||||
uint32_t n_sel_blocks, uint32_t ratio, uint32_t sel_stride) {
|
||||
struct { uint32_t n_tokens, n_sel_blocks, ratio, pos0, sel_stride, pad0, pad1, pad2; } args =
|
||||
{ n_rows, n_sel_blocks, ratio, 0u, sel_stride, 0, 0, 0 };
|
||||
qwen4_bind b[4];
|
||||
if (n_rows == 0 || n_rows > QWEN4_ATTN_ROWS_MAX || ratio == 0 || sel_stride < n_sel_blocks * ratio + ratio - 1u ||
|
||||
!qwen4_bind_tensor(&b[0], sel_blocks, (uint64_t)n_rows * n_sel_blocks * sizeof(int32_t), "rows selected blocks") ||
|
||||
!qwen4_bind_tensor(&b[1], sel_tokens, (uint64_t)n_rows * sel_stride * sizeof(int32_t), "rows selected tokens") ||
|
||||
!qwen4_bind_tensor(&b[2], n_sel, (uint64_t)n_rows * sizeof(uint32_t), "rows selected counts") ||
|
||||
!qwen4_bind_rows(&b[3], table, entry0, n_rows)) {
|
||||
return 0;
|
||||
}
|
||||
return qwen4_dispatch(QWEN4_K_IDX_EXPAND_ROWS, &args, sizeof(args), b, 4,
|
||||
MTLSizeMake(n_rows, 1, 1), MTLSizeMake(256, 1, 1), 0);
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_attn_decode_rows_tensor(
|
||||
ds4_gpu_tensor *out, const ds4_gpu_tensor *q, const ds4_gpu_tensor *gate,
|
||||
const ds4_gpu_tensor *sel_tokens, const ds4_gpu_tensor *n_sel, ds4_gpu_tensor *part,
|
||||
const ds4_gpu_tensor *table, uint64_t entry0, const ds4_gpu_qwen4_attn_row *rows, uint32_t n_rows,
|
||||
uint32_t n_head, uint32_t n_head_kv, uint32_t head_dim, uint32_t sel_stride, float scale) {
|
||||
/* Each row takes the splits the single-row dispatch gives its key count
|
||||
* (see the kernel); the partials use the split cap as their stride. */
|
||||
struct { uint32_t n_tokens, n_head, n_head_kv, head_dim, pos0, use_sel, sel_stride; float scale;
|
||||
uint32_t n_splits, keys_per_split, pad0, pad1; } args =
|
||||
{ n_rows, n_head, n_head_kv, head_dim, 0u, 0u, sel_stride, scale,
|
||||
QWEN4_ATTN_MAX_SPLITS, qwen4_attn_split_keys(), 0, 0 };
|
||||
const uint64_t q_bytes = (uint64_t)n_rows * n_head * head_dim * sizeof(float);
|
||||
const uint64_t part_bytes = ds4_gpu_qwen4_attn_part_floats(n_rows, n_head, head_dim) * sizeof(float);
|
||||
qwen4_bind b[7], res[QWEN4_ATTN_ROWS_MAX * 5u];
|
||||
if (n_rows == 0 || n_rows > QWEN4_ATTN_ROWS_MAX || n_head_kv == 0 || (n_head % n_head_kv) != 0 ||
|
||||
n_head / n_head_kv > 12 || (head_dim != 128 && head_dim != 256) ||
|
||||
!qwen4_bind_tensor(&b[0], q, q_bytes, "attn rows q") ||
|
||||
!qwen4_bind_tensor(&b[1], gate, q_bytes, "attn rows gate") ||
|
||||
!qwen4_bind_tensor(&b[2], sel_tokens, (uint64_t)n_rows * sel_stride * sizeof(int32_t), "rows selected tokens") ||
|
||||
!qwen4_bind_tensor(&b[3], n_sel, (uint64_t)n_rows * sizeof(uint32_t), "rows selected counts") ||
|
||||
!qwen4_bind_tensor(&b[4], out, q_bytes, "attn rows out") ||
|
||||
!qwen4_bind_tensor(&b[5], part, part_bytes, "attn rows partials") ||
|
||||
!qwen4_bind_rows(&b[6], table, entry0, n_rows)) {
|
||||
return 0;
|
||||
}
|
||||
const bool npt8 = head_dim == 256u;
|
||||
if (!qwen4_dispatch_resident(npt8 ? QWEN4_K_ATTN_DECODE_ROWS_NPT8 : QWEN4_K_ATTN_DECODE_ROWS_NPT4,
|
||||
&args, sizeof(args), b, 7,
|
||||
MTLSizeMake(QWEN4_ATTN_MAX_SPLITS, n_head_kv, n_rows),
|
||||
MTLSizeMake(32 * QWEN4_ATTN_NSG, 1, 1), 0,
|
||||
res, qwen4_rows_resident(res, rows, n_rows))) {
|
||||
return 0;
|
||||
}
|
||||
qwen4_bind mb[4] = { b[5], b[1], b[4], b[6] };
|
||||
return qwen4_dispatch(npt8 ? QWEN4_K_ATTN_MERGE_ROWS_NPT8 : QWEN4_K_ATTN_MERGE_ROWS_NPT4,
|
||||
&args, sizeof(args), mb, 4,
|
||||
MTLSizeMake(n_head, n_rows, 1), MTLSizeMake(head_dim, 1, 1), 0);
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_moe_mid_tensor(
|
||||
ds4_gpu_tensor *mid, const ds4_gpu_tensor *x, const ds4_gpu_tensor *selected,
|
||||
const void *model_map, uint64_t model_size, uint64_t gate_offset, uint64_t up_offset,
|
||||
@@ -49494,7 +49947,7 @@ int ds4_gpu_qwen4_moe_mid_tensor(
|
||||
const uint64_t shared_bytes = (uint64_t)sh_row_bytes * ff_dim;
|
||||
qwen4_moe_args args = { n_tokens, n_slots, in_dim, ff_dim, weight_type, row_bytes, expert_bytes,
|
||||
has_shared ? 1u : 0u, has_shared ? shared_type : 0u, sh_row_bytes, n_total_expert,
|
||||
{0}, 0 };
|
||||
{0}, 0u, 0u, 0u };
|
||||
qwen4_bind b[7];
|
||||
if (n_tokens == 0 || n_slots == 0 || row_bytes == 0 || ff_dim == 0 || (has_shared && sh_row_bytes == 0) ||
|
||||
!qwen4_bind_experts(&b[0], model_map, model_size, gate_offset, experts_bytes, "moe gate experts") ||
|
||||
@@ -49569,7 +50022,8 @@ int ds4_gpu_qwen4_moe_down_tensor(
|
||||
const uint64_t experts_bytes = expert_bytes * n_total_expert;
|
||||
const uint64_t shared_bytes = (uint64_t)sh_row_bytes * out_dim;
|
||||
qwen4_moe_args args = { n_tokens, n_slots, ff_dim, out_dim, weight_type, row_bytes, expert_bytes,
|
||||
has_shared ? 1u : 0u, has_shared ? shared_type : 0u, sh_row_bytes, 0, {0}, 0 };
|
||||
has_shared ? 1u : 0u, has_shared ? shared_type : 0u, sh_row_bytes, 0u,
|
||||
{0}, 0u, 0u, 0u };
|
||||
qwen4_bind b[5];
|
||||
if (n_tokens == 0 || n_slots == 0 || row_bytes == 0 || (ff_dim % 32u) != 0 ||
|
||||
out_dim == 0 || (has_shared && sh_row_bytes == 0) ||
|
||||
@@ -49604,6 +50058,66 @@ int ds4_gpu_qwen4_moe_down_tensor(
|
||||
MTLSizeMake(32u * nsg, 1, 1), 0);
|
||||
}
|
||||
|
||||
/* Decode batch over its distinct experts (kernel_qwen4_moe_mid_q4k_grouped):
|
||||
* Q4_K gate/up, no shared slot, per-expert pair lists from
|
||||
* ds4_gpu_qwen4_moe_build_lists_tensor. Same grid as the per-token kernel;
|
||||
* the pairs that do not own their expert return. */
|
||||
int ds4_gpu_qwen4_moe_mid_grouped_tensor(
|
||||
ds4_gpu_tensor *mid, const ds4_gpu_tensor *x, const ds4_gpu_tensor *selected,
|
||||
const ds4_gpu_tensor *lists, const ds4_gpu_tensor *counts, uint32_t list_cap,
|
||||
const void *model_map, uint64_t model_size, uint64_t gate_offset, uint64_t up_offset,
|
||||
uint32_t weight_type, uint32_t n_total_expert, uint32_t n_tokens, uint32_t n_slots,
|
||||
uint32_t in_dim, uint32_t ff_dim) {
|
||||
if (weight_type != 12u || (in_dim % 256u) != 0) return 0;
|
||||
const uint32_t row_bytes = qwen4_expert_row_bytes(weight_type, in_dim);
|
||||
const uint64_t expert_bytes = (uint64_t)row_bytes * ff_dim;
|
||||
qwen4_moe_args args = { n_tokens, n_slots, in_dim, ff_dim, weight_type, row_bytes, expert_bytes,
|
||||
0u, 0u, 0u, n_total_expert, {0}, 0u, list_cap, 0u };
|
||||
qwen4_bind b[7];
|
||||
if (n_tokens == 0 || n_slots == 0 || row_bytes == 0 || ff_dim == 0 || list_cap == 0 ||
|
||||
!qwen4_bind_weight(&b[0], model_map, model_size, gate_offset, expert_bytes * n_total_expert, "moe gate experts") ||
|
||||
!qwen4_bind_weight(&b[1], model_map, model_size, up_offset, expert_bytes * n_total_expert, "moe up experts") ||
|
||||
!qwen4_bind_tensor(&b[2], selected, (uint64_t)n_tokens * n_slots * sizeof(int32_t), "moe selected") ||
|
||||
!qwen4_bind_tensor(&b[3], x, (uint64_t)n_tokens * in_dim * sizeof(float), "moe input") ||
|
||||
!qwen4_bind_tensor(&b[4], mid, (uint64_t)n_tokens * n_slots * ff_dim * sizeof(float), "moe mid") ||
|
||||
!qwen4_bind_tensor(&b[5], lists, (uint64_t)n_total_expert * list_cap * sizeof(int32_t), "moe lists") ||
|
||||
!qwen4_bind_tensor(&b[6], counts, (uint64_t)n_total_expert * sizeof(int32_t), "moe counts")) {
|
||||
return 0;
|
||||
}
|
||||
const uint32_t nsg = 2u, rows_per_tg = 2u * nsg;
|
||||
return qwen4_dispatch(QWEN4_K_MOE_MID_Q4K_GROUPED, &args, sizeof(args), b, 7,
|
||||
MTLSizeMake((ff_dim + rows_per_tg - 1) / rows_per_tg, n_slots, n_tokens),
|
||||
MTLSizeMake(32u * nsg, 1, 1), 0);
|
||||
}
|
||||
|
||||
/* The MXFP4 down rows of the same batch, on the same lists. */
|
||||
int ds4_gpu_qwen4_moe_down_grouped_tensor(
|
||||
ds4_gpu_tensor *part, const ds4_gpu_tensor *mid, const ds4_gpu_tensor *selected,
|
||||
const ds4_gpu_tensor *lists, const ds4_gpu_tensor *counts, uint32_t list_cap,
|
||||
const void *model_map, uint64_t model_size, uint64_t down_offset,
|
||||
uint32_t weight_type, uint32_t n_total_expert, uint32_t n_tokens, uint32_t n_slots,
|
||||
uint32_t ff_dim, uint32_t out_dim) {
|
||||
if (weight_type != 39u || (ff_dim % 32u) != 0) return 0;
|
||||
const uint32_t row_bytes = qwen4_expert_row_bytes(weight_type, ff_dim);
|
||||
const uint64_t expert_bytes = (uint64_t)row_bytes * out_dim;
|
||||
qwen4_moe_args args = { n_tokens, n_slots, ff_dim, out_dim, weight_type, row_bytes, expert_bytes,
|
||||
0u, 0u, 0u, n_total_expert, {0}, 0u, list_cap, 0u };
|
||||
qwen4_bind b[6];
|
||||
if (n_tokens == 0 || n_slots == 0 || row_bytes == 0 || out_dim == 0 || list_cap == 0 ||
|
||||
!qwen4_bind_weight(&b[0], model_map, model_size, down_offset, expert_bytes * n_total_expert, "moe down experts") ||
|
||||
!qwen4_bind_tensor(&b[1], selected, (uint64_t)n_tokens * n_slots * sizeof(int32_t), "moe selected") ||
|
||||
!qwen4_bind_tensor(&b[2], mid, (uint64_t)n_tokens * n_slots * ff_dim * sizeof(float), "moe mid") ||
|
||||
!qwen4_bind_tensor(&b[3], part, (uint64_t)n_tokens * n_slots * out_dim * sizeof(float), "moe partial") ||
|
||||
!qwen4_bind_tensor(&b[4], lists, (uint64_t)n_total_expert * list_cap * sizeof(int32_t), "moe lists") ||
|
||||
!qwen4_bind_tensor(&b[5], counts, (uint64_t)n_total_expert * sizeof(int32_t), "moe counts")) {
|
||||
return 0;
|
||||
}
|
||||
const uint32_t nsg = 4u, rows_per_tg = 2u * nsg;
|
||||
return qwen4_dispatch(QWEN4_K_MOE_DOWN_MXFP4_GROUPED, &args, sizeof(args), b, 6,
|
||||
MTLSizeMake((out_dim + rows_per_tg - 1) / rows_per_tg, n_slots, n_tokens),
|
||||
MTLSizeMake(32u * nsg, 1, 1), 0);
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_moe_reduce_tensor(
|
||||
ds4_gpu_tensor *out, const ds4_gpu_tensor *part, const ds4_gpu_tensor *weights,
|
||||
const ds4_gpu_tensor *shared_gate, const ds4_gpu_tensor *shared, ds4_gpu_tensor *R, const ds4_gpu_tensor *inj,
|
||||
@@ -50743,14 +51257,43 @@ int ds4_gpu_qwen4_multi_gemv_tensor(
|
||||
MTLSizeMake((total + 7) / 8, n_tokens, 1), MTLSizeMake(128, 1, 1), 0);
|
||||
}
|
||||
|
||||
static ds4_gpu_tensor *g_qwen4_dense_mm_partials;
|
||||
static uint64_t g_qwen4_dense_mm_partials_bytes;
|
||||
|
||||
static void qwen4_batch_release_scratch(void) {
|
||||
ds4_gpu_tensor_free(g_qwen4_dense_mm_partials);
|
||||
ds4_gpu_tensor_free(g_qwen4_gdn_slots);
|
||||
g_qwen4_dense_mm_partials = g_qwen4_gdn_slots = NULL;
|
||||
g_qwen4_dense_mm_partials_bytes = 0;
|
||||
for (unsigned i = 0; i < QWEN4_K_COUNT; i++) g_qwen4_pipelines[i] = nil;
|
||||
}
|
||||
|
||||
/* Scratch for the k-split planes, allocated once and generously: the
|
||||
* command buffer holds unretained references, so a scratch replaced while
|
||||
* earlier dispatches of the same buffer still point at it is read after its
|
||||
* release. Growing it per request was what made wide splits look wrong. */
|
||||
static bool qwen4_dense_mm_partials_ensure(uint64_t bytes) {
|
||||
if (g_qwen4_dense_mm_partials && g_qwen4_dense_mm_partials_bytes >= bytes) return true;
|
||||
if (g_qwen4_dense_mm_partials) {
|
||||
fprintf(stderr, "ds4: Qwen3.8 k-split scratch of %.1f MiB cannot grow to %.1f MiB mid-batch\n",
|
||||
ds4_gpu_mib(g_qwen4_dense_mm_partials_bytes), ds4_gpu_mib(bytes));
|
||||
return false;
|
||||
}
|
||||
const uint64_t grow = bytes > (64u << 20) ? bytes : (64u << 20);
|
||||
g_qwen4_dense_mm_partials = ds4_gpu_tensor_alloc(grow);
|
||||
if (!g_qwen4_dense_mm_partials) return false;
|
||||
g_qwen4_dense_mm_partials_bytes = grow;
|
||||
return true;
|
||||
}
|
||||
|
||||
int ds4_gpu_qwen4_dense_mm_tensor(
|
||||
ds4_gpu_tensor *out, const ds4_gpu_tensor *x,
|
||||
const void *model_map, uint64_t model_size, uint64_t weight_offset, uint32_t weight_type,
|
||||
uint32_t n_tokens, uint32_t in_dim, uint32_t out_rows) {
|
||||
const uint32_t row_bytes = weight_type == 0u ? in_dim * 4u : weight_type == 1u ? in_dim * 2u :
|
||||
qwen4_expert_row_bytes(weight_type, in_dim);
|
||||
struct { uint32_t n_tokens, in_dim, out_rows, weight_type, row_bytes, pad0, pad1, pad2; } args =
|
||||
{ n_tokens, in_dim, out_rows, weight_type, row_bytes, 0, 0, 0 };
|
||||
struct { uint32_t n_tokens, in_dim, out_rows, weight_type, row_bytes, n_split, pad1, pad2; } args =
|
||||
{ n_tokens, in_dim, out_rows, weight_type, row_bytes, 1, 0, 0 };
|
||||
qwen4_bind b[3];
|
||||
if (n_tokens == 0 || row_bytes == 0 || (weight_type != 0u && weight_type != 1u && weight_type != 8u) ||
|
||||
(weight_type == 8u && (in_dim % 32) != 0) || (in_dim % 8) != 0 || out_rows == 0) {
|
||||
@@ -50764,8 +51307,87 @@ int ds4_gpu_qwen4_dense_mm_tensor(
|
||||
!qwen4_bind_tensor(&b[2], out, (uint64_t)n_tokens * out_rows * sizeof(float), "dense mm output")) {
|
||||
return 0;
|
||||
}
|
||||
return qwen4_dispatch(QWEN4_K_DENSE_MM, &args, sizeof(args), b, 3,
|
||||
MTLSizeMake((out_rows + 31) / 32, (n_tokens + 31) / 32, 1), MTLSizeMake(128, 1, 1), 0);
|
||||
/* A projection whose output is narrow gives this grid only a few
|
||||
* threadgroups when the batch is a decode step; split k until the
|
||||
* machine has work, then sum the planes. A prefill chunk's token tiles
|
||||
* already fill it, and its planes would not fit the scratch anyway.
|
||||
* The split is chosen so the partials stay small and every threadgroup
|
||||
* still walks enough k to amortize its staging. */
|
||||
const uint32_t tiles = (out_rows + 31u) / 32u;
|
||||
const uint32_t threadgroups = tiles * ((n_tokens + 31u) / 32u);
|
||||
uint32_t n_split = 1u;
|
||||
if (!ds4_gpu_env_u64("DS4_QWEN4_NO_DENSE_MM_KSPLIT", 0u, 0u, 1u)) {
|
||||
const uint32_t nk = (in_dim + 31u) / 32u;
|
||||
const uint32_t target = 128u;
|
||||
const uint32_t want = threadgroups >= target ? 1u : (target + threadgroups - 1u) / threadgroups;
|
||||
n_split = want > nk / 4u ? (nk / 4u ? nk / 4u : 1u) : want;
|
||||
if (n_split > 64u) n_split = 64u;
|
||||
}
|
||||
args.n_split = n_split;
|
||||
if (n_split <= 1u) {
|
||||
return qwen4_dispatch(QWEN4_K_DENSE_MM, &args, sizeof(args), b, 3,
|
||||
MTLSizeMake(tiles, (n_tokens + 31) / 32, 1), MTLSizeMake(128, 1, 1), 0);
|
||||
}
|
||||
|
||||
const uint64_t plane = (uint64_t)n_tokens * out_rows * sizeof(float);
|
||||
if (!qwen4_dense_mm_partials_ensure(plane * n_split)) return 0;
|
||||
qwen4_bind bp[3];
|
||||
bp[0] = b[0];
|
||||
bp[1] = b[1];
|
||||
if (!qwen4_bind_tensor(&bp[2], g_qwen4_dense_mm_partials, plane * n_split, "dense mm partials")) {
|
||||
return 0;
|
||||
}
|
||||
if (!qwen4_dispatch(QWEN4_K_DENSE_MM, &args, sizeof(args), bp, 3,
|
||||
MTLSizeMake(tiles, (n_tokens + 31) / 32, n_split), MTLSizeMake(128, 1, 1), 0)) {
|
||||
return 0;
|
||||
}
|
||||
qwen4_bind br[2];
|
||||
br[0] = bp[2];
|
||||
br[1] = b[2];
|
||||
const uint64_t n = (uint64_t)n_tokens * out_rows;
|
||||
return qwen4_dispatch(QWEN4_K_DENSE_MM_REDUCE, &args, sizeof(args), br, 2,
|
||||
MTLSizeMake((NSUInteger)((n + 255) / 256), 1, 1), MTLSizeMake(256, 1, 1), 0);
|
||||
}
|
||||
|
||||
/* Decode batch of 8 or 16 rows against a Q8 matrix (kernel_qwen4_batch_mm_q8):
|
||||
* the k-split gives narrow projections enough simdgroups, the partial planes
|
||||
* are summed by kernel_qwen4_dense_mm_reduce. */
|
||||
int ds4_gpu_qwen4_batch_mm_q8_tensor(
|
||||
ds4_gpu_tensor *out, const ds4_gpu_tensor *x,
|
||||
const void *model_map, uint64_t model_size, uint64_t weight_offset,
|
||||
uint32_t n_tokens, uint32_t in_dim, uint32_t out_rows) {
|
||||
if ((n_tokens != 8u && n_tokens != 16u) || (in_dim % 32u) != 0 || (out_rows % 32u) != 0 || in_dim == 0) return 0;
|
||||
const uint32_t row_bytes = (in_dim / 32u) * 34u;
|
||||
const uint32_t nk = in_dim / 32u;
|
||||
const uint32_t simdgroups = out_rows / 32u;
|
||||
/* enough simdgroups to cover the machine several times over, no split
|
||||
* shorter than four blocks */
|
||||
const uint32_t target = 640u;
|
||||
uint32_t n_split = simdgroups >= target ? 1u : (target + simdgroups - 1u) / simdgroups;
|
||||
if (n_split > nk / 4u) n_split = nk / 4u ? nk / 4u : 1u;
|
||||
if (n_split > 16u) n_split = 16u;
|
||||
struct { uint32_t n_tokens, in_dim, out_rows, weight_type, row_bytes, n_split, pad1, pad2; } args =
|
||||
{ n_tokens, in_dim, out_rows, 8u, row_bytes, n_split, 0, 0 };
|
||||
qwen4_bind b[3];
|
||||
if (!qwen4_bind_weight(&b[0], model_map, model_size, weight_offset, (uint64_t)row_bytes * out_rows, "batch mm weight") ||
|
||||
!qwen4_bind_tensor(&b[1], x, (uint64_t)n_tokens * in_dim * sizeof(float), "batch mm input") ||
|
||||
!qwen4_bind_tensor(&b[2], out, (uint64_t)n_tokens * out_rows * sizeof(float), "batch mm output")) {
|
||||
return 0;
|
||||
}
|
||||
const MTLSize grid = MTLSizeMake((out_rows + 127u) / 128u, 1, n_split);
|
||||
const int kernel = n_tokens == 16u ? QWEN4_K_BATCH_MM_Q8_T2 : QWEN4_K_BATCH_MM_Q8_T1;
|
||||
if (n_split <= 1u) {
|
||||
return qwen4_dispatch(kernel, &args, sizeof(args), b, 3, grid, MTLSizeMake(128, 1, 1), 0);
|
||||
}
|
||||
const uint64_t plane = (uint64_t)n_tokens * out_rows * sizeof(float);
|
||||
if (!qwen4_dense_mm_partials_ensure(plane * n_split)) return 0;
|
||||
qwen4_bind bp[3] = { b[0], b[1] };
|
||||
if (!qwen4_bind_tensor(&bp[2], g_qwen4_dense_mm_partials, plane * n_split, "batch mm partials")) return 0;
|
||||
if (!qwen4_dispatch(kernel, &args, sizeof(args), bp, 3, grid, MTLSizeMake(128, 1, 1), 0)) return 0;
|
||||
qwen4_bind br[2] = { bp[2], b[2] };
|
||||
const uint64_t n = (uint64_t)n_tokens * out_rows;
|
||||
return qwen4_dispatch(QWEN4_K_DENSE_MM_REDUCE, &args, sizeof(args), br, 2,
|
||||
MTLSizeMake((NSUInteger)((n + 255) / 256), 1, 1), MTLSizeMake(256, 1, 1), 0);
|
||||
}
|
||||
|
||||
/* Vision tower (metal/qwen4_vision.metal). */
|
||||
|
||||
+2344
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,19 @@
|
||||
#ifndef DS4_QWEN4_VISION_H
|
||||
#define DS4_QWEN4_VISION_H
|
||||
|
||||
#include <stdint.h>
|
||||
|
||||
#define DS4_QWEN4_VISION_LAYERS 27
|
||||
typedef struct {
|
||||
uint64_t ln1_w, ln1_b, qkv_w, qkv_b, out_w, out_b, ln2_w, ln2_b, up_w, up_b, down_w, down_b;
|
||||
uint32_t qkv_type, out_type, up_type, down_type;
|
||||
} ds4_qwen4_vision_layer_weights;
|
||||
typedef struct {
|
||||
uint64_t patch_w0, patch_w1, patch_b, pos_embd, post_ln_w, post_ln_b, mm0_w, mm0_b, mm2_w, mm2_b;
|
||||
uint32_t mm0_type, mm2_type, patch_type;
|
||||
uint32_t n_embd, n_ff, n_head, n_patch, n_merge, n_pos_side, n_out;
|
||||
float eps;
|
||||
ds4_qwen4_vision_layer_weights layer[DS4_QWEN4_VISION_LAYERS];
|
||||
} ds4_qwen4_vision_weights;
|
||||
|
||||
#endif
|
||||
+75
-19
@@ -10019,6 +10019,9 @@ struct server_slot {
|
||||
bool decode_in_flight;
|
||||
bool decode_done;
|
||||
int decode_token;
|
||||
bool decode_speculative;
|
||||
int decode_accepted[2];
|
||||
int decode_accepted_count;
|
||||
int decode_rc;
|
||||
char decode_err[160];
|
||||
};
|
||||
@@ -10142,6 +10145,7 @@ struct server {
|
||||
pthread_mutex_t model_mu;
|
||||
pthread_cond_t model_cv;
|
||||
bool model_busy;
|
||||
bool qwen4_batch_mtp;
|
||||
bool model_stopping;
|
||||
int decode_pending;
|
||||
int active_generations;
|
||||
@@ -13079,8 +13083,10 @@ static bool server_cancel_pending_decode_locked(server *s, server_slot *slot) {
|
||||
return true;
|
||||
}
|
||||
|
||||
static int server_eval_token(server *s, server_slot *slot, int token,
|
||||
char *err, size_t errlen) {
|
||||
static int server_eval_tokens(server *s, server_slot *slot, int token,
|
||||
bool speculative, int accepted[2], int *n_accepted,
|
||||
char *err, size_t errlen) {
|
||||
*n_accepted = 0;
|
||||
if (!s || !slot) return 1;
|
||||
if (!s->batched_mode) {
|
||||
if (g_stop_requested || slot_job_cancelled(slot)) {
|
||||
@@ -13092,6 +13098,7 @@ static int server_eval_token(server *s, server_slot *slot, int token,
|
||||
pthread_mutex_lock(&s->inference_mu);
|
||||
int rc = ds4_session_eval(slot->session, token, err, errlen);
|
||||
pthread_mutex_unlock(&s->inference_mu);
|
||||
if (rc == 0) { accepted[0] = token; *n_accepted = 1; }
|
||||
return rc;
|
||||
}
|
||||
|
||||
@@ -13109,6 +13116,8 @@ static int server_eval_token(server *s, server_slot *slot, int token,
|
||||
return 1;
|
||||
}
|
||||
slot->decode_token = token;
|
||||
slot->decode_speculative = speculative;
|
||||
slot->decode_accepted_count = 0;
|
||||
slot->decode_rc = 1;
|
||||
slot->decode_err[0] = '\0';
|
||||
slot->decode_done = false;
|
||||
@@ -13138,10 +13147,20 @@ static int server_eval_token(server *s, server_slot *slot, int token,
|
||||
(slot->decode_err[0] ? slot->decode_err : "decode interrupted"));
|
||||
}
|
||||
slot->decode_done = false;
|
||||
if (rc == 0) {
|
||||
*n_accepted = slot->decode_accepted_count;
|
||||
memcpy(accepted, slot->decode_accepted, (size_t)*n_accepted * sizeof(int));
|
||||
}
|
||||
pthread_mutex_unlock(&s->model_mu);
|
||||
return rc;
|
||||
}
|
||||
|
||||
static int server_eval_token(server *s, server_slot *slot, int token,
|
||||
char *err, size_t errlen) {
|
||||
int accepted[2], count;
|
||||
return server_eval_tokens(s, slot, token, false, accepted, &count, err, errlen);
|
||||
}
|
||||
|
||||
static long server_decode_coalesce_us(void) {
|
||||
long us = 2000;
|
||||
const char *env = getenv("DS4_SERVER_DECODE_COALESCE_US");
|
||||
@@ -13164,6 +13183,8 @@ static void *decode_worker_main(void *arg) {
|
||||
server *s = arg;
|
||||
ds4_decode_item *items = xmalloc((size_t)s->slot_count * sizeof(*items));
|
||||
server_slot **members = xmalloc((size_t)s->slot_count * sizeof(*members));
|
||||
int (*accepted)[2] = xmalloc((size_t)s->slot_count * sizeof(*accepted));
|
||||
int *n_accepted = xmalloc((size_t)s->slot_count * sizeof(*n_accepted));
|
||||
const long coalesce_us = server_decode_coalesce_us();
|
||||
const bool log_batches = getenv("DS4_SERVER_BATCH_LOG") != NULL;
|
||||
|
||||
@@ -13194,17 +13215,21 @@ static void *decode_worker_main(void *arg) {
|
||||
}
|
||||
if (s->model_stopping && s->decode_pending == 0) break;
|
||||
|
||||
int count = 0;
|
||||
for (int i = 0; i < s->slot_count; i++) {
|
||||
server_slot *slot = &s->slots[i];
|
||||
if (!slot->decode_pending) continue;
|
||||
slot->decode_pending = false;
|
||||
slot->decode_in_flight = true;
|
||||
s->decode_pending--;
|
||||
members[count] = slot;
|
||||
items[count].session = slot->session;
|
||||
items[count].token = slot->decode_token;
|
||||
count++;
|
||||
int count = 0, plain_count = 0;
|
||||
/* Keep sampled/exact requests out of the greedy speculative group. */
|
||||
for (int speculative = 0; speculative <= 1; speculative++) {
|
||||
for (int i = 0; i < s->slot_count; i++) {
|
||||
server_slot *slot = &s->slots[i];
|
||||
if (!slot->decode_pending || slot->decode_speculative != (bool)speculative) continue;
|
||||
slot->decode_pending = false;
|
||||
slot->decode_in_flight = true;
|
||||
s->decode_pending--;
|
||||
members[count] = slot;
|
||||
items[count].session = slot->session;
|
||||
items[count].token = slot->decode_token;
|
||||
count++;
|
||||
}
|
||||
if (!speculative) plain_count = count;
|
||||
}
|
||||
if (count == 0) continue;
|
||||
s->model_busy = true;
|
||||
@@ -13213,8 +13238,17 @@ static void *decode_worker_main(void *arg) {
|
||||
char batch_err[160] = {0};
|
||||
const double batch_t0 = log_batches ? now_sec() : 0.0;
|
||||
pthread_mutex_lock(&s->inference_mu);
|
||||
int rc = ds4_sessions_eval_batch(items, count,
|
||||
batch_err, sizeof(batch_err));
|
||||
int rc = plain_count ? ds4_sessions_eval_batch(items, plain_count,
|
||||
batch_err, sizeof(batch_err)) : 0;
|
||||
for (int i = 0; i < plain_count; i++) {
|
||||
accepted[i][0] = items[i].token;
|
||||
n_accepted[i] = 1;
|
||||
}
|
||||
if (rc == 0 && count > plain_count)
|
||||
rc = ds4_sessions_eval_batch_speculative_argmax(items + plain_count, count - plain_count,
|
||||
accepted + plain_count, n_accepted + plain_count, batch_err, sizeof(batch_err));
|
||||
if (rc != 0)
|
||||
for (int i = 0; i < count; i++) ds4_session_invalidate(items[i].session);
|
||||
pthread_mutex_unlock(&s->inference_mu);
|
||||
if (log_batches) {
|
||||
server_log(DS4_LOG_DEFAULT,
|
||||
@@ -13229,6 +13263,10 @@ static void *decode_worker_main(void *arg) {
|
||||
server_slot *slot = members[i];
|
||||
slot->decode_in_flight = false;
|
||||
slot->decode_rc = rc;
|
||||
if (rc == 0) {
|
||||
slot->decode_accepted_count = n_accepted[i];
|
||||
memcpy(slot->decode_accepted, accepted[i], (size_t)n_accepted[i] * sizeof(int));
|
||||
}
|
||||
if (rc != 0) {
|
||||
snprintf(slot->decode_err, sizeof(slot->decode_err), "%s",
|
||||
batch_err[0] ? batch_err : "batched decode failed");
|
||||
@@ -13240,6 +13278,8 @@ static void *decode_worker_main(void *arg) {
|
||||
pthread_mutex_unlock(&s->model_mu);
|
||||
free(members);
|
||||
free(items);
|
||||
free(accepted);
|
||||
free(n_accepted);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
@@ -13810,6 +13850,14 @@ decode_again:
|
||||
finish = "error";
|
||||
break;
|
||||
}
|
||||
} else if (s->batched_mode && s->qwen4_batch_mtp &&
|
||||
max_tokens - completion >= 2 && !j->req.ignore_eos &&
|
||||
(!ds4_engine_mtp_exact_sampling(s->engine) || temperature == 0.0f) &&
|
||||
getenv("DS4_MTP_SPEC_DISABLE") == NULL) {
|
||||
if (server_eval_tokens(s, slot, token, true, toks, &ntok, err, sizeof(err)) != 0) {
|
||||
finish = "error";
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
if (server_eval_token(s, slot, token, err, sizeof(err)) != 0) {
|
||||
finish = "error";
|
||||
@@ -15237,11 +15285,17 @@ static void log_context_memory(ds4_backend backend, int ctx_size,
|
||||
m.raw_cap,
|
||||
m.comp_cap);
|
||||
if (session_count > 1) {
|
||||
/* Only the caches repeat per slot. Where the model shares one prefill
|
||||
* workspace across slots, the transients are paid once instead. */
|
||||
const double gib = 1024.0 * 1024.0 * 1024.0;
|
||||
const double caches = (double)(m.raw_bytes + m.compressed_bytes);
|
||||
server_log(DS4_LOG_DEFAULT,
|
||||
"ds4-server: %d resident sessions request at least %.2f GiB of context buffers",
|
||||
"ds4-server: %d resident sessions request at least %.2f GiB of "
|
||||
"context buffers, or %.2f GiB when the model shares one prefill "
|
||||
"workspace",
|
||||
session_count,
|
||||
(double)m.total_bytes * (double)session_count /
|
||||
(1024.0 * 1024.0 * 1024.0));
|
||||
(double)m.total_bytes * (double)session_count / gib,
|
||||
(caches * (double)session_count + (double)m.scratch_bytes) / gib);
|
||||
}
|
||||
}
|
||||
static void server_close_resources(server *s) {
|
||||
@@ -15680,6 +15734,8 @@ int main(int argc, char **argv) {
|
||||
s.ctx_size = cfg.ctx_size;
|
||||
s.slot_count = slot_count;
|
||||
s.batched_mode = cfg.batched_sessions > 0;
|
||||
s.qwen4_batch_mtp = cfg.engine.backend == DS4_BACKEND_METAL &&
|
||||
ds4_engine_is_qwen4(engine) && ds4_engine_mtp_draft_tokens(engine) > 1;
|
||||
s.mixed_prefill_quantum = cfg.mixed_prefill_quantum;
|
||||
s.last_prefill_slot = slot_count - 1;
|
||||
s.default_tokens = cfg.default_tokens;
|
||||
@@ -15735,7 +15791,7 @@ int main(int argc, char **argv) {
|
||||
server_prefill_quantum_for(&s, false),
|
||||
server_prefill_quantum_for(&s, true),
|
||||
server_decode_coalesce_us());
|
||||
if (ds4_engine_has_mtp(engine)) {
|
||||
if (ds4_engine_mtp_draft_tokens(engine) > 1 && !s.qwen4_batch_mtp) {
|
||||
server_log(DS4_LOG_DEFAULT,
|
||||
"ds4-server: MTP speculative decoding is disabled while native session batching is active");
|
||||
}
|
||||
|
||||
@@ -14,3 +14,7 @@ quality-testing/data/*
|
||||
!quality-testing/data/glm53-flash-openrouter-zai-fp8-100/**
|
||||
!quality-testing/data/flash-vision-exp-openrouter-novita-100/
|
||||
!quality-testing/data/flash-vision-exp-openrouter-novita-100/**
|
||||
!quality-testing/data/qwen38-flash-alibaba-100/
|
||||
!quality-testing/data/qwen38-flash-alibaba-100/**
|
||||
!quality-testing/data/qwen38-flash-alibaba-long/
|
||||
!quality-testing/data/qwen38-flash-alibaba-long/**
|
||||
+2
-2
@@ -31,10 +31,10 @@ NVCC_ARCH_FLAGS := -arch=$(CUDA_ARCH)
|
||||
endif
|
||||
NVCCFLAGS ?= -O3 --use_fast_math $(NVCC_ARCH_FLAGS) -Xcompiler $(NATIVE_CPU_FLAG) -Xcompiler -pthread
|
||||
QUALITY_LDLIBS ?= -lm -Xcompiler -pthread -L$(CUDA_HOME)/targets/sbsa-linux/lib -L$(CUDA_HOME)/lib64 -lcudart -lcublas
|
||||
QUALITY_TARGETS := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o ds4_image.o rax.o ds4_gpu_args.o \
|
||||
QUALITY_TARGETS := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o ds4_engram.o ds4_image.o rax.o ds4_gpu_args.o \
|
||||
cuda/mmq/ds4_ggml_stubs.o cuda/mmq/ds4_mmq.o cuda/mmq/ds4_mmq_d2r.o \
|
||||
cuda/mmq/quantize.o cuda/mmq/mmid.o cuda/mmq/mmvq.o cuda/mmq/ds4_repack.o
|
||||
QUALITY_OBJS := ../ds4.o ../ds4_cuda.o ../ds4_distributed.o ../ds4_tp.o ../ds4_ssd.o ../ds4_layer_pack.o ../ds4_image.o ../rax.o ../ds4_gpu_args.o \
|
||||
QUALITY_OBJS := ../ds4.o ../ds4_cuda.o ../ds4_distributed.o ../ds4_tp.o ../ds4_ssd.o ../ds4_layer_pack.o ../ds4_engram.o ../ds4_image.o ../rax.o ../ds4_gpu_args.o \
|
||||
../cuda/mmq/ds4_ggml_stubs.o ../cuda/mmq/ds4_mmq.o ../cuda/mmq/ds4_mmq_d2r.o \
|
||||
../cuda/mmq/quantize.o ../cuda/mmq/mmid.o ../cuda/mmq/mmvq.o ../cuda/mmq/ds4_repack.o
|
||||
QUALITY_LINK := $(NVCC) $(NVCCFLAGS) -I.. -o
|
||||
|
||||
@@ -27,6 +27,10 @@ calling hosted APIs:
|
||||
official DeepSeek API with `top_logprobs=20`.
|
||||
- `data/pro-0813`: 100 DeepSeek V4 PRO 0813 continuations collected from the
|
||||
official DeepSeek API with `top_logprobs=20`.
|
||||
- `data/qwen38-flash-alibaba-100`: 100 non-thinking Qwen3.8 Flash
|
||||
continuations from Alibaba through OpenRouter, with top-five logprobs.
|
||||
- `data/qwen38-flash-alibaba-long`: 12 archive/code continuations from the
|
||||
same endpoint, with prompts from 2K to 24K tokens.
|
||||
|
||||
DeepSeek V4 Flash also has tracked official smoke vectors in
|
||||
`tests/test-vectors/`. Those vectors drive `./ds4_test --logprob-vectors` and
|
||||
@@ -35,6 +39,12 @@ include short prompts plus long-prompt attention cases.
|
||||
The hosted APIs expose output-token logprobs and top-logprob alternatives, not
|
||||
full vocabulary logits.
|
||||
|
||||
The scorer verifies each API token's bytes against the local token boundaries,
|
||||
not just the number of tokens. If they differ, it still scores the continuation
|
||||
text but skips API logprob comparisons for that case. Alternatives containing
|
||||
Unicode replacement characters are excluded because the original token bytes
|
||||
may have been lost by the provider.
|
||||
|
||||
## 2. Collect Official Continuations
|
||||
|
||||
For the tracked DeepSeek V4 Flash 0731 fixture:
|
||||
@@ -116,13 +126,45 @@ The prompt list is tracked in `prompts.jsonl`. Curated fixture directories are
|
||||
also tracked after review; ad-hoc API collection directories should stay
|
||||
untracked until they are intentionally promoted into the release QA set.
|
||||
|
||||
Use `--system TEXT` when the reference needs an explicit system message.
|
||||
`--system ''` sends an empty message, which is different from omitting it:
|
||||
some providers insert a default system prompt when none is supplied. The
|
||||
collector records this setting and rejects a changed system message on resume.
|
||||
For a nonempty system message, render the matching local chat prompt and use
|
||||
`--rendered-prompt`; the scorer's ordinary prompt mode adds no system message.
|
||||
|
||||
For Qwen, use the following settings in a new output directory:
|
||||
|
||||
```sh
|
||||
python3 gguf-tools/quality-testing/collect_official.py \
|
||||
--out /tmp/qwen-official \
|
||||
--model qwen/qwen3.8-flash \
|
||||
--endpoint https://openrouter.ai/api/v1/chat/completions \
|
||||
--provider-order alibaba --require-parameters \
|
||||
--require-response-provider Alibaba --require-response-model qwen/qwen3.8-flash \
|
||||
--system '' --thinking omit --reasoning-effort none \
|
||||
--count 100 --max-tokens 64 --top-logprobs 5
|
||||
```
|
||||
|
||||
This reads `OPENROUTER_API_KEY` from the environment and disables provider
|
||||
fallbacks. Omitting `--system ''` caused Alibaba to add 13 prompt tokens in
|
||||
the September 2026 checks. The empty message matches Qwen's published
|
||||
no-thinking template and the scorer's ordinary prompt mode.
|
||||
|
||||
Alibaba does not disclose this endpoint's precision. The
|
||||
[official model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) describes
|
||||
hosted Flash as based on Flash Next, not as a byte-identical checkpoint.
|
||||
Agreement with this service is an external quality check, not proof of exact
|
||||
native-weight inference.
|
||||
|
||||
## 3. Build The Local Scorer
|
||||
|
||||
```sh
|
||||
make -C gguf-tools quality-score
|
||||
```
|
||||
|
||||
The scorer links against the DS4 runtime and uses Metal by default.
|
||||
The scorer links against the DS4 runtime, using Metal on macOS and CUDA on
|
||||
Linux. On a DGX Spark, pass `CUDA_ARCH=sm_121` to the build command.
|
||||
|
||||
Build the optional llama.cpp control scorer with:
|
||||
|
||||
@@ -320,3 +362,39 @@ Run the host-only positive and negative controls:
|
||||
```sh
|
||||
python3 -m unittest discover -s gguf-tools/quality-testing/tests -v
|
||||
```
|
||||
|
||||
## 7. Qwen3.8 Spark Baseline
|
||||
|
||||
Measured on 2026-09-15, using one DGX Spark per quant. The short set has 100
|
||||
cases and 5,696 target tokens; the long set has 12 cases and 766 target tokens,
|
||||
with prefixes from 1,971 to 23,985 tokens. These are ordinary decoding and
|
||||
prefill checks, without MTP or vision.
|
||||
|
||||
| Suite | GGUF | API top-token agreement | NLL, default | NLL, `--quality` |
|
||||
| --- | --- | ---: | ---: | ---: |
|
||||
| Short | Q2 | 88.90% | 0.352980 | 0.352915 |
|
||||
| Short | Q4 | 92.08% | 0.290297 | 0.290538 |
|
||||
| Long | Q2 | 93.99% | 0.154199 | 0.154106 |
|
||||
| Long | Q4 | 97.26% | 0.124611 | 0.125546 |
|
||||
|
||||
Agreement is measured after feeding the same official continuation prefix,
|
||||
not as a percentage of correct answers. The two short cases with damaged
|
||||
emoji metadata contribute to NLL but not API agreement, which covers 5,568
|
||||
short-set tokens. All 766 long-set tokens align. Every local prompt and target
|
||||
token ID was also checked against Qwen's published tokenizer.
|
||||
|
||||
The default fast path shows no material loss relative to `--quality` in these
|
||||
sets. The paired case-bootstrap intervals include zero for both quants and
|
||||
both suites. Q4 is closer to the hosted reference than Q2 on average.
|
||||
These measurements do not prove identical native weights or correctness at
|
||||
every context length.
|
||||
|
||||
Leaving the last 1 or 256 prompt tokens for a separate continued prefill
|
||||
changed aggregate NLL by less than 0.0008 for either quant. Independent CLI
|
||||
answers at about 2K and 24K context also retrieved the correct archive owner
|
||||
and revision, and gave the correct clamp outputs and boundary tests, for
|
||||
both Q2 and Q4.
|
||||
|
||||
Raw per-case scores, weight/source hashes and comparison details are retained
|
||||
in [short results](data/qwen38-flash-alibaba-100/cuda-spark-20260915/results.json)
|
||||
and [long results](data/qwen38-flash-alibaba-long/cuda-spark-20260915/results.json).
|
||||
@@ -136,6 +136,7 @@ def request_one(
|
||||
provider_allow_fallbacks: bool,
|
||||
provider_require_parameters: bool,
|
||||
temperature: float = 0,
|
||||
system: str | None = None,
|
||||
) -> dict:
|
||||
payload = {
|
||||
"model": model,
|
||||
@@ -143,6 +144,8 @@ def request_one(
|
||||
"temperature": temperature,
|
||||
"stream": False,
|
||||
}
|
||||
if system is not None:
|
||||
payload["messages"].insert(0, {"role": "system", "content": system})
|
||||
if top_logprobs > 0:
|
||||
payload["logprobs"] = True
|
||||
payload["top_logprobs"] = top_logprobs
|
||||
@@ -184,6 +187,7 @@ def fetch_with_retry(
|
||||
provider_allow_fallbacks: bool,
|
||||
provider_require_parameters: bool,
|
||||
temperature: float = 0,
|
||||
system: str | None = None,
|
||||
) -> dict:
|
||||
delay = 1.0
|
||||
for attempt in range(6):
|
||||
@@ -202,6 +206,7 @@ def fetch_with_retry(
|
||||
provider_allow_fallbacks,
|
||||
provider_require_parameters,
|
||||
temperature,
|
||||
system,
|
||||
)
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", "replace")
|
||||
@@ -222,6 +227,9 @@ def main() -> int:
|
||||
ap.add_argument("--out", default="gguf-tools/quality-testing/data")
|
||||
ap.add_argument("--prompts", default="gguf-tools/quality-testing/prompts.jsonl")
|
||||
ap.add_argument("--model", default=MODEL)
|
||||
ap.add_argument("--system", default=None,
|
||||
help="explicit system message, including an empty string; "
|
||||
"some providers insert a default when this is omitted")
|
||||
ap.add_argument("--endpoint", default=ENDPOINT)
|
||||
ap.add_argument("--api-key-env", default=None)
|
||||
ap.add_argument("--count", type=int, default=100)
|
||||
@@ -278,6 +286,12 @@ def main() -> int:
|
||||
(out / "prompts").mkdir(parents=True, exist_ok=True)
|
||||
(out / "continuations").mkdir(parents=True, exist_ok=True)
|
||||
(out / "responses").mkdir(parents=True, exist_ok=True)
|
||||
system_path = out / "request_system.json"
|
||||
if args.resume and (system_path.exists() or any((out / "responses").glob("*.json"))):
|
||||
saved_system = json.loads(system_path.read_text()) if system_path.exists() else None
|
||||
if saved_system != args.system:
|
||||
raise RuntimeError("saved system message differs from --system; use a new output directory")
|
||||
system_path.write_text(json.dumps(args.system, ensure_ascii=False) + "\n", encoding="utf-8")
|
||||
|
||||
manifest = out / "manifest.tsv"
|
||||
rows = []
|
||||
@@ -322,6 +336,7 @@ def main() -> int:
|
||||
args.allow_provider_fallbacks,
|
||||
provider_require_parameters,
|
||||
args.temperature,
|
||||
args.system,
|
||||
)
|
||||
choice = response["choices"][0]
|
||||
content = choice.get("message", {}).get("content")
|
||||
@@ -369,6 +384,7 @@ def main() -> int:
|
||||
"schema": "ds4-official-continuations-v1",
|
||||
"created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"model": args.model,
|
||||
"system": args.system,
|
||||
"endpoint": args.endpoint,
|
||||
"observed_models": sorted(observed_models),
|
||||
"observed_providers": sorted(observed_providers),
|
||||
|
||||
@@ -13,13 +13,26 @@ safe to commit and use in release QA.
|
||||
- `pro`: 100 DeepSeek V4 PRO preview continuations with API top-logprob slices.
|
||||
- `pro-0813`: 100 DeepSeek V4 PRO 0813 continuations with API top-logprob
|
||||
slices.
|
||||
- `qwen38-flash-alibaba-100`: 100 Qwen3.8 Flash continuations from Alibaba,
|
||||
pinned through OpenRouter, with top-five logprobs and no thinking.
|
||||
- `qwen38-flash-alibaba-long`: 12 continuations from the same endpoint, with
|
||||
archive and code prompts from 2K to 24K tokens, crossing sparse attention.
|
||||
|
||||
Each fixture directory contains:
|
||||
|
||||
- `prompts/case_*.txt`: exact user prompts.
|
||||
- `continuations/case_*.txt`: deterministic hosted-model continuations.
|
||||
- `responses/case_*.json`: raw hosted responses, including logprob slices.
|
||||
- `responses/case_*.json`: hosted responses, including logprob slices.
|
||||
Some curated sets omit unrelated response IDs and billing fields.
|
||||
- `manifest.tsv`: paths consumed by `score_official`.
|
||||
|
||||
DeepSeek V4 Flash smoke vectors are also tracked in `tests/test-vectors/` and
|
||||
are run by `./ds4_test --logprob-vectors`.
|
||||
|
||||
Qwen's `collection.json` records the explicit empty system message, provider,
|
||||
template hashes and token-alignment checks. All prompt lengths match the
|
||||
published tokenizer. Short cases 058 and 069 contain damaged emoji bytes in
|
||||
the API logprob metadata: keep their text likelihood, but exclude their API
|
||||
agreement. `manifest-aligned.tsv` selects the cases with intact token bytes.
|
||||
Alibaba does not disclose this endpoint's precision, and hosted Flash is
|
||||
based on Flash Next rather than guaranteed to use identical checkpoint weights.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,4 @@
|
||||
Here is a step-by-step explanation of B-tree insertion, covering the general algorithm, the mechanics of node splitting, and the specific handling of the root node.
|
||||
|
||||
### 1. Prerequisites: The B-Tree Structure
|
||||
To understand insertion, we must first define the constraints of a B-Tree of **Order
|
||||
@@ -0,0 +1,5 @@
|
||||
Here is a concise design for a TCP echo server optimized for handling slow clients (the "C10K" or "Slowloris" problem).
|
||||
|
||||
### Core Principle
|
||||
**Non-blocking I/O + Event-Driven Architecture.**
|
||||
Never block the main thread waiting for data from a slow client. Instead, register
|
||||
@@ -0,0 +1,5 @@
|
||||
Comparing **memory-mapped (mmaped) model weights** versus **copying all weights into private buffers** on macOS involves trade-offs between memory efficiency, startup latency, runtime performance, and system resource management. Below is a detailed comparison across key dimensions:
|
||||
|
||||
---
|
||||
|
||||
### 1. **Memory Usage**
|
||||
@@ -0,0 +1,5 @@
|
||||
To derive why RMSNorm (Root Mean Square Normalization) requires a **sum of squares** and a **scaling pass**, we must look at the mathematical goal of normalization in neural networks: stabilizing the distribution of activations to facilitate faster and more stable training.
|
||||
|
||||
Here is the step-by-step logical derivation.
|
||||
|
||||
---
|
||||
@@ -0,0 +1,6 @@
|
||||
KV cache checkpointing helps agentic sessions primarily by **reducing latency, saving computational resources, and enabling efficient state management** in long-running, multi-step AI workflows.
|
||||
|
||||
Here’s a detailed breakdown of why it is critical for agents:
|
||||
|
||||
### 1. What is the KV Cache?
|
||||
In transformer-based
|
||||
@@ -0,0 +1,5 @@
|
||||
Here is the pseudocode for the **Push** (also known as `Insert` or `Enqueue`) operation on a Binary Heap.
|
||||
|
||||
### Logic Explanation
|
||||
1. **Add to End**: The new element is placed at the very end of the underlying array (the next available leaf position).
|
||||
2.
|
||||
@@ -0,0 +1,3 @@
|
||||
To understand the tradeoffs between **Top-p sampling** (also known as Nucleus Sampling) and **Temperature Zero decoding** (Greedy Decoding), we must first clarify what each method does to the probability distribution of the next token generated by a Large Language Model (LLM).
|
||||
|
||||
### 1. Definitions
|
||||
@@ -0,0 +1,4 @@
|
||||
Speculative decoding preserves exact greedy output by using a **verification step** that ensures the final sequence is identical to what would have been generated by the target model alone, token-by-token. Here’s how it works:
|
||||
|
||||
### Core Mechanism
|
||||
1. **Draft Phase**: A smaller, faster "draft" model generates
|
||||
@@ -0,0 +1,3 @@
|
||||
The premise of your question contains a fundamental misunderstanding: **A GPU prefill path is not faster than single-token decode in terms of latency per token.** In fact, the opposite is true.
|
||||
|
||||
However, I believe you are asking about **throughput** or **computational efficiency**, where the prefill phase *is
|
||||
@@ -0,0 +1,5 @@
|
||||
Here are three critical invariants for a tokenizer special-token table, designed to ensure correctness, determinism, and safety during tokenization:
|
||||
|
||||
### 1. **Uniqueness of Special Token Strings**
|
||||
- **Invariant**: No two distinct special tokens may share the same string representation.
|
||||
- **R
|
||||
@@ -0,0 +1,3 @@
|
||||
L'inserimento in un **B-tree** è un'operazione fondamentale che mantiene l'albero bilanciato, garantendo che tutte le foglie si trovino alla stessa profondità. Questo assicura operazioni di ricerca, inserimento e cancellazione in tempo logaritmico $O(\log n)$.
|
||||
|
||||
Ecco
|
||||
@@ -0,0 +1,6 @@
|
||||
Ecco un progetto conciso per un server TCP Echo progettato specificamente per gestire client lenti (slow clients), evitando che un singolo client lento blocchi gli altri.
|
||||
|
||||
### **Obiettivo**
|
||||
Creare un server TCP che:
|
||||
1. Accetti connessioni multiple.
|
||||
2. Risponda
|
||||
@@ -0,0 +1,3 @@
|
||||
Confrontare `mmap` (memory-mapped file) con la copia completa dei pesi in memoria su macOS è una questione cruciale per l'ottimizzazione delle prestazioni, specialmente nel contesto di modelli di machine learning di grandi dimensioni.
|
||||
|
||||
Ecco un'analisi dettagliata dei due approcci, con
|
||||
@@ -0,0 +1,3 @@
|
||||
La derivazione di **RMSNorm** (Root Mean Square Normalization) parte dall'idea di normalizzare i vettori di attivazione in modo più efficiente rispetto a LayerNorm, eliminando la sottrazione della media e la divisione per la varianza.
|
||||
|
||||
Ecco la derivazione passo-passo
|
||||
@@ -0,0 +1,3 @@
|
||||
La cache KV (Key-Value) su disco è una tecnica di ottimizzazione fondamentale per le **sessioni agentiche** (agent-based sessions), dove un modello linguistico (LLM) deve mantenere uno stato coerente attraverso molti passaggi, chiamate a strumenti e interazioni con l'ambiente esterno.
|
||||
|
||||
E
|
||||
@@ -0,0 +1,4 @@
|
||||
Ecco lo pseudocodice per inserire un elemento in un **heap binario** (assumendo un **max-heap**, dove il padre è sempre maggiore o uguale ai figli).
|
||||
|
||||
### Concetti chiave:
|
||||
1. L'elemento viene inserito alla fine dell'array (ultimo livello,
|
||||
@@ -0,0 +1,3 @@
|
||||
Questa è una domanda fondamentale nel campo dell'ingegneria dei prompt (prompt engineering) e dello sviluppo di applicazioni basate su Large Language Models (LLM).
|
||||
|
||||
**Top-p (Nucleus Sampling)** e **Temperatura 0 (Greedy Decoding)** sono due strategie diverse per selezionare il token
|
||||
@@ -0,0 +1,5 @@
|
||||
La decodifica speculativa (Speculative Decoding) mantiene l'output esatto grazie a un meccanismo di **verifica rigorosa** che garantisce la distribuzione di probabilità identica a quella del modello target originale.
|
||||
|
||||
Ecco il riassunto dei principi chiave:
|
||||
|
||||
### 1. Il Principio Fond
|
||||
@@ -0,0 +1,3 @@
|
||||
In realtà, c'è un malinteso fondamentale nella premessa della tua domanda: **il prefill non è più veloce del decode per singolo token**. Anzi, il contrario è vero.
|
||||
|
||||
Tuttavia, la confusione nasce spesso dal confronto tra **throughput totale** (token al secondo) e
|
||||
@@ -0,0 +1,3 @@
|
||||
Ecco tre invarianti fondamentali per una tabella di token speciali (ad esempio, in un contesto di elaborazione del linguaggio naturale o di compilatori):
|
||||
|
||||
1. **Unicità dei Token**: Ogni token speciale nella tabella deve essere unico. Non devono esistere duplicati dello stesso token (es. `[CLS
|
||||
@@ -0,0 +1,3 @@
|
||||
To describe how binary search finds the insertion point in a sorted array, we must look at the specific variant of binary search often referred to as **`bisect_left`** (in Python) or **`lower_bound`** (in C++).
|
||||
|
||||
The goal is not just to find if an element exists,
|
||||
@@ -0,0 +1,7 @@
|
||||
```c
|
||||
int clamp(int value, int min, int max) {
|
||||
if (value < min) return min;
|
||||
if (value > max) return max;
|
||||
return value;
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,8 @@
|
||||
# Mutex vs. Atomic Counter
|
||||
|
||||
## Core Conceptual Difference
|
||||
|
||||
| Aspect | Mutex | Atomic Counter |
|
||||
|--------|-------|----------------|
|
||||
| **Purpose** | Mutual exclusion (protects a *critical section*) | Lock-free synchronization for a *single value* |
|
||||
| **Mechanism** | Blocks
|
||||
@@ -0,0 +1,5 @@
|
||||
**Backpressure** in a network server is a flow control mechanism that prevents a fast producer (sender) from overwhelming a slow consumer (receiver).
|
||||
|
||||
In simple terms, it is the process of **signaling upstream components to slow down or stop sending data when downstream components cannot keep up with processing it.**
|
||||
|
||||
###
|
||||
@@ -0,0 +1 @@
|
||||
Storing a model checkpoint header safely is critical because the header often contains metadata (e.g., architecture, hyperparameters, versioning) that may be parsed or executed during loading. If not handled properly, it can lead to security vulnerabilities such as **arbitrary code execution**, **denial of service**, or **data
|
||||
@@ -0,0 +1,7 @@
|
||||
# Ring Buffer (Circular Buffer) Mechanics
|
||||
|
||||
## How a Ring Buffer Wraps
|
||||
|
||||
A ring buffer is a fixed-size array used as a FIFO queue, where the end of the array logically connects back to the beginning. Two pointers (or indices) track positions:
|
||||
|
||||
- **Head (write index)**: Where the
|
||||
@@ -0,0 +1,9 @@
|
||||
Here is a brief, structured plan for testing long-context prompt chunking strategies.
|
||||
|
||||
### **Objective**
|
||||
Evaluate the effectiveness of different chunking methods in preserving semantic integrity, retrieval accuracy, and response quality when processing documents exceeding model context limits.
|
||||
|
||||
---
|
||||
|
||||
### **1. Test Dataset Preparation**
|
||||
- **Source
|
||||
@@ -0,0 +1 @@
|
||||
In the context of **Low-Bit Quantization** (specifically for Large Language Models or deep neural networks), an **Importance Matrix** is a data structure used to determine which weights in a model are most critical to preserve with high precision, and which can be aggressively quantized (compressed) without significantly degrading
|
||||
@@ -0,0 +1,5 @@
|
||||
Grouped MoE (Mixture of Experts) expert execution improves prefill performance primarily by **optimizing memory bandwidth utilization and reducing kernel launch overhead through batched, contiguous computation**.
|
||||
|
||||
Here’s a detailed breakdown of why this is critical for the prefill phase:
|
||||
|
||||
### 1. Prefill vs. Decode:
|
||||
@@ -0,0 +1 @@
|
||||
Comparing two model quantizations (e.g., INT8 vs. FP16, or different calibration methods for the same bit-width) requires a multi-dimensional evaluation framework. Relying on a single metric (like top-1 accuracy) is insufficient because quantization can introduce subtle biases, latency trade-offs, and
|
||||
@@ -0,0 +1 @@
|
||||
La **ricerca binaria del punto di inserimento** (spesso implementata come `bisect_left` o `bisect_right` in Python, o `lower_bound`/`upper_bound` in C++) è un algoritmo che determina l'indice corretto per inserire un nuovo elemento in una lista ordin
|
||||
@@ -0,0 +1,7 @@
|
||||
Ecco una breve funzione C che limita (clamp) un intero a un intervallo specificato:
|
||||
|
||||
```c
|
||||
int clamp(int value, int min, int max) {
|
||||
if (value < min) return min;
|
||||
if (value > max) return max;
|
||||
return value;
|
||||
@@ -0,0 +1,5 @@
|
||||
La differenza fondamentale tra un **mutex** e un **contatore atomico** risiede nel loro scopo, nella granularità dell'operazione che proteggono e nel meccanismo con cui garantiscono la sicurezza in ambienti multi-thread.
|
||||
|
||||
Ecco una spiegazione dettagliata:
|
||||
|
||||
### 1. Contatore Atomico
|
||||
@@ -0,0 +1 @@
|
||||
In un server di rete, il **backpressure** (o "retropressione") è un meccanismo di controllo del flusso che consente al ricevitore (il server) di segnalare al mittente (il client o un altro nodo della rete) di rallentare o fermare temporaneamente l'inv
|
||||
@@ -0,0 +1,3 @@
|
||||
Salvare l'header di un checkpoint di modello in modo sicuro è fondamentale per prevenire attacchi come **deserializzazione non sicura** (es. `pickle` in Python), che possono portare all'esecuzione arbitraria di codice.
|
||||
|
||||
Ecco un formato sicuro e best practice, basato su standard aperti e validazione
|
||||
@@ -0,0 +1,3 @@
|
||||
Un **ring buffer** (o buffer circolare) è una struttura dati fondamentale in informatica, particolarmente utile nei sistemi embedded, nell'elaborazione di segnali audio/video e nella comunicazione tra processi.
|
||||
|
||||
Ecco una spiegazione dettagliata del suo funzionamento e delle strategie per gestire le sovrascritture.
|
||||
@@ -0,0 +1,4 @@
|
||||
Ecco un piano conciso per testare l'efficacia del chunking (frammentazione) su prompt lunghi:
|
||||
|
||||
### 1. Definizione degli Obiettivi
|
||||
* **Metriche chiave:** Accuratezza della risposta, coerenza logica tra i chunk, latenza totale e costo token.
|
||||
@@ -0,0 +1 @@
|
||||
Una **matrice di importanza** (o *importance matrix*) nella quantizzazione low-bit è una struttura dati che assegna un peso numerico a ciascun elemento o gruppo di elementi di una matrice di pesi (ad esempio, in una rete neurale) per indicare quanto sia critico preservare la precisione di
|
||||
@@ -0,0 +1 @@
|
||||
Raggruppare gli esperti (Expert Grouping) nei modelli Mixture-of-Experts (MoE) accelera la fase di **prefill** principalmente ottimizzando l'accesso alla memoria e migliorando l'efficienza del calcolo parallelo. Ecco una spiegazione dettagliata dei meccanismi coinvolti:
|
||||
@@ -0,0 +1 @@
|
||||
Confrontare due quantizzazioni (ad esempio, un modello a 8-bit vs. uno a 4-bit, o INT8 vs. FP16) senza fidarsi di una singola metrica o risposta richiede un approccio **multidimensionale e robusto**. L'obiettivo è valutare non solo la "
|
||||
@@ -0,0 +1,5 @@
|
||||
When investigating performance degradation during generation with long contexts, the goal is to isolate whether the bottleneck is in the **prefill** (processing the input), the **decode** (generating tokens), or system-level resource constraints.
|
||||
|
||||
Here are the first five critical measurements to take, ordered by diagnostic priority:
|
||||
|
||||
###
|
||||
@@ -0,0 +1 @@
|
||||
In the context of Large Language Models (LLMs) and autoregressive generation, **greedy decoding** selects the token with the highest probability at each step. The "logit" is the raw output score from the final layer of the neural network before it is converted into a probability distribution via the Softmax function.
|
||||
@@ -0,0 +1,5 @@
|
||||
Online Softmax is a technique used in attention mechanisms (like FlashAttention) to compute the softmax function **without needing to see all the values at once**. This allows for memory-efficient and faster computation, especially when dealing with long sequences.
|
||||
|
||||
Here’s an explanation using simple variables:
|
||||
|
||||
### The Problem with Standard Softmax
|
||||
@@ -0,0 +1,12 @@
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"arguments": {
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"required": ["name", "
|
||||
@@ -0,0 +1,3 @@
|
||||
To test that a file-backed `mmap` is not repeatedly remapped (i.e., to ensure the mapping persists and isn't being torn down/recreated, which would cause performance issues or data loss), you can use several approaches depending on your specific concern.
|
||||
|
||||
## Key Concerns with Repeated Remapping
|
||||
@@ -0,0 +1,5 @@
|
||||
# Why One Large Metal Buffer Can Be Worse Than Multiple Overlapping Views
|
||||
|
||||
## The Core Issue: Memory Coherency and Cache Behavior
|
||||
|
||||
At first glance, a single large buffer seems simpler — one allocation, one binding point. But in practice, **multiple overlapping views** (or separate buffers) often outperform a
|
||||
@@ -0,0 +1,3 @@
|
||||
In a Mixture-of-Experts (MoE) layer, the **Router** (also known as the Gating Network or Switch) acts as the intelligent dispatcher. Its primary role is to determine which specific "Expert" sub-networks should process a given input token.
|
||||
|
||||
Here is a detailed breakdown of its functions
|
||||
@@ -0,0 +1,3 @@
|
||||
In the context of Large Language Models (LLMs) and Transformer architectures, **KV Cache** (Key-Value Cache) is a mechanism used to speed up inference by storing the Key and Value vectors from previous tokens so they don’t need to be recomputed.
|
||||
|
||||
The difference between **Raw KV Rows** and **
|
||||
@@ -0,0 +1,3 @@
|
||||
Validating that a quantized model (e.g., INT8, FP16, or mixed-precision) still follows instructions is critical because quantization can introduce subtle numerical errors that degrade reasoning capabilities, instruction adherence, and output quality.
|
||||
|
||||
Here’s a comprehensive checklist for validating instruction-following in quantized models:
|
||||
@@ -0,0 +1,3 @@
|
||||
Evaluating 50 prompts is significantly more informative than relying on a single "favorite" prompt because it shifts the assessment from **anecdotal evidence** to **statistical reliability**.
|
||||
|
||||
Here’s a breakdown of why this approach yields better insights, particularly in fields like AI model evaluation, user experience research, or
|
||||
@@ -0,0 +1,4 @@
|
||||
Ecco cinque misure iniziali da adottare per diagnosticare e mitigare la lentezza del decode in contesti lunghi:
|
||||
|
||||
1. **Verificare l’uso della KV Cache (Key-Value Cache)**
|
||||
Assicurarsi che la cache delle chiavi e dei valori sia abilitata e correttamente configurata
|
||||
@@ -0,0 +1,5 @@
|
||||
La ragione per cui piccole differenze nei logit possono cambiare drasticamente una continuazione greedy risiede nella natura **discreta** e **non continua** della funzione di argmax, combinata con la sensibilità numerica dei modelli linguistici.
|
||||
|
||||
Ecco una spiegazione dettagliata:
|
||||
|
||||
### 1. La
|
||||
@@ -0,0 +1,6 @@
|
||||
Spiegare l'**Online Softmax** (o Softmax Incrementale) è fondamentale per capire come funzionano modelli moderni come FlashAttention, che permettono di gestire sequenze molto lunghe senza saturare la memoria.
|
||||
|
||||
Ecco una spiegazione passo-passo usando variabili semplici e logica intuitiva.
|
||||
|
||||
---
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
Ecco uno schema JSON minimo per una tool call, basato sul formato standard (come quello utilizzato da OpenAI e altri LLM):
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "nome_della_tool",
|
||||
"arguments": {
|
||||
"parametro1": "valore1",
|
||||
"
|
||||
@@ -0,0 +1 @@
|
||||
Per verificare che un `mmap` su file non venga rimappato ripetutamente (ovvero, per assicurarsi che la mappatura sia stabile e non causi overhead di sistema o errori), è necessario combinare **monitoraggio delle chiamate di sistema**, **analisi della memoria** e **test funzionali**.
|
||||
@@ -0,0 +1 @@
|
||||
La premessa della tua domanda contiene un'assunzione tecnica che va corretta: **un grande buffer Metal non è intrinsecamente "peggiore" delle viste sovrapposte (overlapping views)**. In realtà, in molti casi, l'uso di un singolo buffer Metal ben gestito è *super
|
||||
@@ -0,0 +1,3 @@
|
||||
In un'architettura **Mixture-of-Experts (MoE)**, il **router** (o gate) svolge un ruolo fondamentale e critico: è il componente responsabile della **selezione dinamica degli esperti** più adatti per elaborare ogni singolo token o input.
|
||||
|
||||
### Ruolo dettagliato del Router
|
||||
@@ -0,0 +1 @@
|
||||
La differenza fondamentale tra **righe KV raw** (grezze) e **righe KV compresse** risiede nel modo in cui i dati vengono archiviati fisicamente su disco o in memoria, con implicazioni dirette sulle prestazioni di lettura/scrittura, sull'uso dello spazio e sulla complessità
|
||||
@@ -0,0 +1,3 @@
|
||||
Ecco una checklist strutturata per validare che un modello quantizzato (es. INT8, INT4, FP16) mantenga la capacità di seguire le istruzioni (Instruction Following), confrontandolo con il modello originale non quantizzato o con un benchmark standard.
|
||||
|
||||
### 📋 Checklist di Validazione:
|
||||
@@ -0,0 +1,5 @@
|
||||
Valutare 50 prompt (o un numero significativo di varianti) è generalmente più utile che affidarsi a un singolo "prompt preferito" per diverse ragioni fondamentali legate alla natura dei modelli linguistici (LLM) e al processo di ottimizzazione.
|
||||
|
||||
Ecco i motivi principali:
|
||||
|
||||
### 1. **
|
||||
@@ -0,0 +1,4 @@
|
||||
```python
|
||||
def median_of_three(a, b, c):
|
||||
return sorted([a, b, c])[1]
|
||||
```
|
||||
@@ -0,0 +1,5 @@
|
||||
Sorted arrays make membership tests faster with binary search because they allow the algorithm to **eliminate half of the remaining search space** at each step, reducing the time complexity from linear $O(n)$ to logarithmic $O(\log n)$.
|
||||
|
||||
Here is a detailed explanation of why this works:
|
||||
|
||||
### 1
|
||||
@@ -0,0 +1,6 @@
|
||||
**Eventual consistency** is a consistency model used in distributed computing and databases that guarantees that, given enough time without new updates, all replicas of the data will eventually converge to the same state.
|
||||
|
||||
### Key Concepts
|
||||
|
||||
1. **No Immediate Guarantee**:
|
||||
- When a write operation occurs, it does not
|
||||
@@ -0,0 +1,7 @@
|
||||
Here is a short, classic example of a race condition in C using two threads incrementing a shared counter.
|
||||
|
||||
### The Problem
|
||||
The operation `counter++` is **not atomic**. It consists of three steps:
|
||||
1. Read the value from memory.
|
||||
2. Increment the value.
|
||||
3. Write
|
||||
@@ -0,0 +1,6 @@
|
||||
**Latency** and **throughput** are two fundamental metrics used to measure the performance of computer systems, networks, and storage devices. While they are related, they measure different aspects of performance.
|
||||
|
||||
### 1. Definitions
|
||||
|
||||
#### **Latency**
|
||||
- **Definition**: The time delay between the cause and
|
||||
@@ -0,0 +1 @@
|
||||
A **Trie** (also known as a digital tree, prefix tree, or radix tree) represents a set of strings by storing the characters of the strings in the nodes of a tree structure. Unlike a binary search tree where keys are stored in nodes, in a Trie, the **path from the root to a
|
||||
@@ -0,0 +1,11 @@
|
||||
# Testing a Command-Line Parser with Edge Cases
|
||||
|
||||
## Overview
|
||||
|
||||
A robust command-line parser must handle not just typical inputs, but also malformed, ambiguous, and adversarial inputs. Here's a systematic approach to testing edge cases.
|
||||
|
||||
---
|
||||
|
||||
## 1. Categorize Edge Case Types
|
||||
|
||||
### A. Empty &
|
||||
@@ -0,0 +1,6 @@
|
||||
The difference between **mmap page residency** and **virtual address space size** arises from the fundamental design of virtual memory systems, specifically the concept of **lazy allocation** (or demand paging).
|
||||
|
||||
Here’s a detailed breakdown:
|
||||
|
||||
### 1. Virtual Address Space Size
|
||||
- **Definition**: The total amount of
|
||||
@@ -0,0 +1,5 @@
|
||||
To explain why quantization error affects rare experts more than common experts, we must look at the mechanics of **Mixture-of-Experts (MoE)** architectures and how **quantization** interacts with statistical frequency.
|
||||
|
||||
### 1. Context: MoE and Quantization Basics
|
||||
|
||||
* **Mixture-of-
|
||||
@@ -0,0 +1,6 @@
|
||||
Here is a concise checklist for reviewing pull requests that involve memory lifetimes (e.g., in Rust, C++, or other systems languages):
|
||||
|
||||
### 🧠 Memory Lifetime Review Checklist
|
||||
|
||||
#### 1. **Ownership & Borrowing Clarity**
|
||||
- [ ] Is ownership clearly transferred or borrowed? Avoid ambiguous `
|
||||
@@ -0,0 +1,5 @@
|
||||
```python
|
||||
def median_of_three(a: float, b: float, c: float) -> float:
|
||||
"""Restituisce la mediana di tre numeri."""
|
||||
return sorted([a, b, c])[1]
|
||||
```
|
||||
@@ -0,0 +1,2 @@
|
||||
Gli array ordinati rendono più veloce la ricerca di appartenenza (cioè verificare se un elemento esiste nell'array) perché permettono l'utilizzo dell'algoritmo di **ricerca binaria** (*binary search*), che ha una complessità temporale logaritmica, molto inferiore alla ricerca lineare.
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
La **consistenza eventuale** (in inglese *eventual consistency*) è un modello di consistenza utilizzato nei database distribuiti che garantisce che, se non vengono apportate nuove modifiche a un dato, tutti i nodi del sistema alla fine convergeranno verso lo stesso valore.
|
||||
|
||||
### Spiegazione dettagli
|
||||
@@ -0,0 +1,10 @@
|
||||
Ecco un esempio classico e conciso di **race condition** in C, utilizzando due thread che incrementano una variabile condivisa senza protezione.
|
||||
|
||||
### Esempio di Race Condition
|
||||
|
||||
```c
|
||||
#include <stdio.h>
|
||||
#include <pthread.h>
|
||||
#include <unistd.h>
|
||||
|
||||
int shared_counter =
|
||||
@@ -0,0 +1,7 @@
|
||||
La differenza tra **latenza** e **throughput** è fondamentale in informatica, networking e ingegneria dei sistemi. Sebbene siano spesso correlate, misurano aspetti diversi delle prestazioni di un sistema.
|
||||
|
||||
Ecco una spiegazione chiara con analogie e definizioni tecniche:
|
||||
|
||||
---
|
||||
|
||||
### 1
|
||||
@@ -0,0 +1,6 @@
|
||||
Un **trie** (pronunciato "try", da *retrieval*) è una struttura dati ad albero ordinata che rappresenta un insieme di stringhe in modo efficiente, sfruttando i prefissi comuni.
|
||||
|
||||
### Come funziona la rappresentazione:
|
||||
|
||||
1. **Radice vuota**:
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
Testare un parser da riga di comando (CLI) con casi limite è fondamentale per garantire robustezza, sicurezza e usabilità. Un parser mal progettato può causare crash, interpretazioni errate o vulnerabilità di sicurezza (es. injection).
|
||||
|
||||
Ecco una guida strutturata su come farlo, suddivisa
|
||||
@@ -0,0 +1 @@
|
||||
La differenza tra la **dimensione virtuale** (Virtual Size) e la **residenza** (Resident Set Size, RSS) delle pagine mappate in memoria (`mmap`) è un concetto fondamentale nella gestione della memoria dei sistemi operativi moderni. Ecco una spiegazione dettagliata del perché queste due grandez
|
||||
Loaded 100 of 402 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user