ds4/docs/DGX_SPARK.md

DGX Spark

README | Getting started

Use the NVIDIA driver and CUDA development toolkit for the machine; the build needs nvcc and cuBLAS. Check that nvidia-smi sees the GPU before building.

Build and run

make cuda-spark
./download_model.sh ds4f-q2
./ds4 --cuda

The build selects sm_121 and enables the Blackwell-specific kernels. Do not use --cuda-tensor-parallel on a single Spark. Stop other inference services before loading a model so they do not compete for memory. Restore any services you stopped when finished.

DeepSeek V4.1 Flash

Q2 needs SSD streaming on one 128 GB Spark:

./download_model.sh ds41f-q2
./ds4 --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
  --ssd-streaming --ctx 32768

The file is 341 GiB, including 189 GiB of disk-only Engram tables. Keep it on a fast local SSD. The expert cache is sized automatically; leave memory for other programs and the context. Use the same options with ds4-agent or ds4-server.

Two Sparks can instead keep half the experts each, using network tensor parallelism. Both need the complete GGUF on their local SSD. V4.1 CUDA vision and DSpark are not supported. The model guide covers thinking levels and the separate Metal configurations.

September 13-14, 2026, Q2 SSD with automatic cache sizing: a 3,241-token /read README.md prefill reached 93-96 t/s, up from 49-54 t/s, with 32K allocated context. No extra flag is needed. A separate 64 GiB cache-budget run with 64K allocated context reached 384 t/s for a 32K initial prefill and 88 t/s for a 3.2K append. These are SSD-streamed V4.1 results, not resident V4 Flash numbers; speed depends on prompt length and the expert cache. The automatic-cache 256-token reply test decoded about 5% slower, but prefill plus the reply fell from about 87 to 59 seconds. See the QA record for conditions and longer-context measurements.

On two Sparks over RoCE, September 13 measurements averaged 21.9 t/s for one session: Q2, a 1K prompt, 64K allocated context and 2,048 teacher-forced decode tokens. This does not use speculative decoding. Separately, eight ready sessions reached about 28 aggregate t/s, not 28 t/s per client. Decode batching starts at five ready sessions; smaller groups run in order. Separate long runs measured about 400 t/s for a 32K prefill and 290 t/s for an 8K append. The network TP QA record has the timing controls and memory limits.

Qwen3.8 Flash Next

Both Q2 and Q4 fit on one Spark. After building:

./download_model.sh qwen38-q2
./ds4 --mtp --ctx 32768

Q2 keeps 41.73 GiB of weights resident; Q4 keeps 69.74 GiB. Both GGUFs also contain 95.37 GiB of BF16 n-grams, read directly from the SSD. Leave room for context and runtime buffers. Vision, the native agent, server APIs and text-session checkpoints work on CUDA; see Qwen setup.

September 15, 2026, Q2 on one Spark: a 16K prefill reached 250 t/s, extending it to 30K reached 243 t/s, and ordinary decode stayed around 17.3 t/s. These are single runs with 8,192-token prefill chunks and 32 decode tokens. A separate 256-token prose run measured 17.1 t/s ordinarily and 19.9 t/s with MTP, at temperature zero and 8K allocated context. Speculation is opt-in; its benefit depends on the prompt.

GLM 5.3 Flash

Q2 is the resident target for one Spark:

./download_model.sh glm53-q2
./ds4 --cuda -m gguf/GLM-5.3-Flash-Q2.gguf --ctx 16384

Q4 does not fit resident. GLM Spark-to-Spark tensor parallelism is not implemented; the two-Mac RDMA instructions do not apply to GLM on CUDA.

Vision and speculative decoding

DeepSeek Vision Experimental needs its matching text model and encoder:

./download_model.sh ds4f-vision-q2
./ds4 --cuda --vision gguf/DeepSeek-V4-Flash-Vision-Encoder.gguf

Use /read image.png in the CLI. GLM vision also works on this backend; see models and vision.

For Flash 0731, DSpark uses the separate 0731 support file. Vision Experimental has a different drafter. GLM uses its built-in MTP block with --mtp. None is enabled by default.

Flash Q2 with DSpark

September 6, 2026, fully resident Flash 0731: median generation speed from three runs of 256 tokens, 4K allocated context and a 512-token prefill chunk.

Prompt	Temperature	Ordinary decode	DSpark
C hash table	0	19.72 t/s	31.41 t/s
C hash table	1	19.53 t/s	29.98 t/s
Unpredictable prose	1	19.53 t/s	18.81 t/s

Longer coding runs reached about 33 t/s. Poor draft acceptance can still make DSpark slower. Temperature 1 uses the default opportunistic policy; these are not exact-sampling results. DSpark does not accelerate prefill.

To reproduce the greedy coding case after downloading ds4f-q2 and ds4f-dspark:

./ds4 --cuda -m ds4flash.gguf \
  --dspark --mtp-model gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
  --ctx 4096 --prefill-chunk 512 --nothink --temp 0 --seed 12345 -n 256 \
  -p 'Write a complete C hash table implementation with string keys, insert, find, delete, and a test main. Output only C code.'

For the temperature-1 rows, use --temp 1 --top-p 0.95 --min-p 0.05. The QA record includes the previous implementation, continued-context checks and quality comparisons.

Larger models and serving

CUDA also has SSD streaming paths for larger weights. Memory fit and speed depend on the model layout; start with Q2 for normal use.

V4.1 Q2 SSD serving batches up to eight decode rows together. An eight-session test reached 11.0 aggregate t/s versus 8.4 with ordered execution; this is total throughput, not speed per client. Other single-Spark model paths retain their existing scheduling. See serving and the recorded benchmarks.