mtfyun/ds4
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Enable DeepSeek V4.1 DSpark over Metal and CUDA TP

Browse Source
This commit is contained in:
antirez committed 2026-10-10 13:10:32 +02:00
1 parent 792e7e6aad
commit 97079d7dcf
9 files changed
+265 -57
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

QA_BEFORE_RELEASES.md
+17 -1
View File
@@ -1688,7 +1688,8 @@ the long sparse-boundary tests; neither substitutes for the other.
Use `DeepSeek-V4.1-Flash-DSpark.gguf`, not a V4.0 drafter. The native support
model is 8.04 GiB and remains resident while the target can use SSD streaming.
On 128 GiB hosts, leave room for both models and context buffers. This path
supports one Metal, CUDA or ROCm host; TP plus DSpark must fail explicitly.
supports one Metal, CUDA or ROCm host and two-rank Metal/CUDA TP. For TP,
pass the matching `--dspark --mtp-model DSPARK` options on both peers.
- Run `python3 tests/test_deepseek41_dspark_conversion.py` and audit every
real tensor with `python3 gguf-tools/deepseek41_dspark.py --hf SOURCE
@@ -1709,6 +1710,12 @@ supports one Metal, CUDA or ROCm host; TP plus DSpark must fail explicitly.
after verification. Exercise opportunistic and exact sampling. Loading an
older target snapshot must warm missing draft features rather than reuse
stale ones. Repeat the backend primitives, including 128-expert/top-3 routing.
- Repeat the runtime test on physical Metal RDMA and CUDA RoCE pairs:
`tests/test_deepseek41_dspark TARGET DSPARK --runtime-tp HOST PORT DEVICE GID`.
Start the normal `ds4` TP worker with context 2048 and the matching drafter.
Require all prefix commits, correction rewinds, both sampling modes and
cancellation to finish without a lost peer. Also reject mismatched DSpark
settings at startup (`DS4_TEST_TP_MODE_MISMATCH=3` in `test_tp_link`).
- Build `gguf-tools/quality-testing/score_official_verify`. Compare the matching
official manifest with `--mtp-model DSPARK` against the same command plus
`--verify-batch 6`. Keep `--ssd-streaming --ssd-streaming-cache-experts 40gb`
@@ -1717,6 +1724,9 @@ supports one Metal, CUDA or ROCm host; TP plus DSpark must fail explicitly.
34816; also repeat its `manifest-long.tsv` with `--continued-prefill 32`.
Report paired teacher-forced NLL and API top-token agreement. Greedy sampled
prefix length and scalar-versus-verifier text equality are not quality gates.
Repeat the scalar/verifier comparison on both physical TP pairs using the
normal coordinator flags instead of SSD streaming; start a matching worker
for each run. Verify the complete vocabulary, not just the local half.
- Run real CLI and agent code/prose requests at temperature zero and above
zero, including `--mtp-exact-sampling`. Against an otherwise idle speculative
server, run `tests/test_server_speculative_stop.py --model deepseek-v4.1-flash
@@ -1725,6 +1735,12 @@ supports one Metal, CUDA or ROCm host; TP plus DSpark must fail explicitly.
coding client's edit/build/test loop and cancellation near context capacity.
Record acceptance and end-to-end timing; working speculation does not by
itself establish a speedup.
- On ROCm, repeat at least ten actual server cold starts outside a debugger,
alternating default and exact sampling. Send a request, check the answer,
and shut down cleanly each time. Watch usable host memory, excluding CMA.
If startup crashes, retain the binary and core or backtrace plus kernel
logs; distinguish a userspace HIP failure from OOM or a GPU reset. A clean
rerun is evidence against recurrence, not an explanation of a prior crash.
### Metal Reference Checks
docs/SPECULATIVE_DECODING.md
+6 -3
View File
@@ -65,14 +65,17 @@ See [conversion](../gguf-tools/README.md#convert-v41-dspark) to build it.
The same options work with `ds4-agent` and `ds4-server`. Metal, CUDA and ROCm
support single-host execution, with either resident target weights or SSD
streaming. The drafter stays resident, so leave another 8 GiB for it. Network
TP with V4.1 DSpark is not implemented; ordinary V4.1 TP is unchanged.
streaming. The drafter stays resident, so leave another 8 GiB for it.
Two-host tensor parallelism works on Metal and CUDA: add the same `--dspark
--mtp-model FILE` options to both the coordinator and worker. Each peer keeps
the drafter resident; the target and its verification run across both GPUs.
Up to five drafts are checked together. The sampling choices below apply;
`--mtp-exact-sampling` uses sampled proposals and the full acceptance/correction
rule for V4.1. Restored sessions warm the draft window with 128 ordinary tokens
before speculation resumes. Measure speed on your prompts: acceptance and SSD
cache size both matter.
cache size both matter. V4.1 DSpark can still be slower than ordinary decoding,
including in TP mode.
## GLM: built-in MTP
ds4.c
+150 -13
View File
@@ -71877,13 +71877,12 @@ static int ds4_engine_open_internal(ds4_engine **out,
#endif
opt->distributed.role == DS4_DISTRIBUTED_NONE &&
!load_slice && !opt->glm_mtp &&
(!(opt->dspark || (opt->mtp_path && opt->mtp_path[0])) || opt->tp.role == DS4_TP_NONE) &&
!opt->first_token_test && !opt->metal_graph_test &&
(!opt->directional_steering_file || !opt->directional_steering_file[0]) &&
e->power_percent == 100 && opt->context_size <= 1048576;
if (!supported) {
fprintf(stderr, "ds4: V4.1 requires Metal, single-GPU CUDA or ROCm (network TP on Metal/CUDA); "
"DSpark currently requires one rank; steering and legacy diagnostics "
"steering and legacy diagnostics "
"are not supported (maximum context 1048576)\n");
ds4_engine_close(e);
*out = NULL;
@@ -73858,6 +73857,8 @@ int ds4_engine_tp_bind(ds4_engine *e, struct ds4_tp *tp, char *err, size_t errle
}
uint32_t execution_mode = e->cuda_tp_expert_split ?
DS4_TP_MODE_EXPERT_INTERMEDIATE_SPLIT : 0u;
if (DS4_MODEL_FAMILY == DS4_MODEL_FAMILY_DEEPSEEK41 && e->dspark)
execution_mode |= DS4_TP_MODE_V41_DSPARK;
if (DS4_MODEL_FAMILY == DS4_MODEL_FAMILY_DEEPSEEK41 &&
!getenv("DS4_METAL_DISABLE_V41_TP_SHARED_OWNER")) {
execution_mode |= DS4_TP_MODE_V41_SHARED_OWNER;
@@ -78925,6 +78926,9 @@ static int ds4_session_eval_internal(ds4_session *s, int token, bool probe_mtp,
ds4_engine *e = s->engine;
#ifdef DS4_HAS_DEEPSEEK41_GPU
if (ds4_session_is_ds41(s)) {
ds41_dspark *d = s->ds41_graph.dspark;
const bool rewindable = d && d->rewind_end == s->ds41_graph.pos &&
d->rewind_end && d->rewind_end < d->verify_start + d->verify_count;
if (!s->ds41_graph_ready ||
(!s->checkpoint_valid && s->checkpoint.len != 0) ||
s->ds41_graph.pos != (uint32_t)s->checkpoint.len ||
@@ -78935,6 +78939,7 @@ static int ds4_session_eval_internal(ds4_session *s, int token, bool probe_mtp,
}
token_vec_push(&s->checkpoint, token);
s->checkpoint_valid = true;
if (rewindable) d->rewind_end = s->ds41_graph.pos;
return 0;
}
#endif
@@ -82869,6 +82874,100 @@ static int ds4_session_eval_dspark_speculative_stochastic(
}
#endif
#ifdef DS4_HAS_DEEPSEEK41_GPU
/* The target vocabulary is sharded on CUDA. Both ranks keep the small native
* drafter resident, then exchange its head rows before the leader applies the
* sequential Markov bias. Metal currently keeps the complete target head. */
static bool ds41_session_draft(ds4_session *s, int token) {
ds4_engine *e = s->engine;
ds41_gpu_graph *g = &s->ds41_graph;
ds41_dspark *d = g->dspark;
const bool split = e->tp.active && e->tp.vocab_split;
if (!d || (split && e->tp.rank == 0 &&
!ds4_tp_send_dspark_draft(e->tp.ctx, s->tp_session_id, token,
(int)g->pos, DS4_TP_DSPARK_DRAFT))) return false;
if (!ds41_dspark_forward(d, &e->model, &e->weights, &e->mtp_model, g->pos, token, g))
return false;
if (!split) return true;
const uint32_t half = DS4_N_VOCAB / 2u;
/* draft_probs is not live until Markov sampling; borrow it for the packed
* local half rows so expansion cannot overwrite the next row. */
if (!ds4_gpu_tensor_read(d->logits, 0, d->draft_probs,
(uint64_t)DS41_DRAFT_ROWS * half * sizeof(float))) return false;
for (uint32_t i = 0; i < DS41_DRAFT_ROWS; i++) {
float *local = d->draft_probs + (uint64_t)i * half;
float *row = d->host_logits + (uint64_t)i * DS4_N_VOCAB;
if (e->tp.rank == 1) {
if (!ds4_tp_send_logits_half(e->tp.ctx, local, half)) return false;
} else {
memcpy(row, local, half * sizeof(float));
if (!ds4_tp_recv_logits_half(e->tp.ctx, row + half, half)) return false;
}
}
return e->tp.rank == 1 || ds4_gpu_tensor_write(d->logits, 0, d->host_logits,
(uint64_t)DS41_DRAFT_ROWS * DS4_N_VOCAB * sizeof(float));
}
static bool ds41_session_verify(ds4_session *s, const int *tokens, uint32_t count) {
ds4_engine *e = s->engine;
ds41_gpu_graph *g = &s->ds41_graph;
ds41_dspark *d = g->dspark;
if (ds4_session_tp_leader(s) &&
!ds4_tp_send_verify(e->tp.ctx, s->tp_session_id, tokens, count)) return false;
/* Scalar TP merges its head on the host. Preserve that complete frontier
* for a zero-prefix rewind, not the stale other half on the device. */
if (!ds4_gpu_tensor_write(g->logits, 0, s->logits, (uint64_t)DS4_N_VOCAB * 4u) ||
!ds41_dspark_verify(d, g, &e->model, &e->weights, tokens, count)) return false;
if (!e->tp.active || !e->tp.vocab_split) return true;
const uint32_t half = DS4_N_VOCAB / 2u;
for (uint32_t i = 0; i < count; i++) {
float *row = d->host_verified + (uint64_t)i * DS4_N_VOCAB;
if (e->tp.rank == 1) {
if (!ds4_tp_send_logits_half(e->tp.ctx, row + half, half)) return false;
} else if (!ds4_tp_recv_logits_half(e->tp.ctx, row + half, half)) return false;
}
return e->tp.rank == 1 || ds4_gpu_tensor_write(d->verify_logits, 0, d->host_verified,
(uint64_t)count * DS4_N_VOCAB * sizeof(float));
}
static bool ds41_session_commit(ds4_session *s, const int *tokens,
uint32_t count, uint32_t keep) {
ds4_engine *e = s->engine;
if (keep > count || (ds4_session_tp_leader(s) &&
!ds4_tp_send_verify_commit(e->tp.ctx,
keep == count ? DS4_TP_VERIFY_COMMIT_FULL : DS4_TP_VERIFY_COMMIT_PREFIX,
(int32_t)keep))) return false;
if (!ds41_dspark_commit(s->ds41_graph.dspark, &s->ds41_graph, keep, s->logits))
return false;
char err[128];
if (ds4_session_tp_leader(s) && !ds4_tp_wait_command_ack(e->tp.ctx,
s->tp_session_id, "V4.1 verify commit", err, sizeof(err))) return false;
for (uint32_t i = 0; i < keep; i++) token_vec_push(&s->checkpoint, tokens[i]);
return true;
}
static int ds41_session_tp_verify(ds4_session *s, const int *tokens, uint32_t count,
uint32_t flags, char *err, size_t errlen) {
ds4_engine *e = s->engine;
int32_t mode, keep;
bool ok = !flags && s->checkpoint_valid && s->ds41_graph.dspark &&
s->ds41_graph.pos == (uint32_t)s->checkpoint.len &&
ds41_session_verify(s, tokens, count) &&
ds4_tp_recv_verify_commit(e->tp.ctx, &mode, &keep);
if (ok) ok = keep >= 0 && (uint32_t)keep <= count &&
((mode == DS4_TP_VERIFY_COMMIT_FULL && (uint32_t)keep == count) ||
(mode == DS4_TP_VERIFY_COMMIT_PREFIX && (uint32_t)keep < count));
if (ok) ok = ds41_dspark_commit(s->ds41_graph.dspark, &s->ds41_graph,
(uint32_t)keep, s->logits);
if (ok) for (int32_t i = 0; i < keep; i++) token_vec_push(&s->checkpoint, tokens[i]);
if (!ok) {
s->checkpoint_valid = s->ds41_graph.valid = false;
snprintf(err, errlen, "tp: V4.1 verification or prefix commit failed");
}
return ok ? 0 : 1;
}
#endif
/* TP worker side of a mirrored speculative-verify block. Runs its half of the
* same batch verify as the leader, including per-layer combine gates, purely
* for KV, compressor, and indexer side effects; then it obeys the commit
@@ -82900,6 +82999,10 @@ int ds4_session_tp_spec_cycle(ds4_session *s, const int *drafts, int draft_n,
snprintf(err, errlen, "tp: verify block beyond context");
return 1;
}
#ifdef DS4_HAS_DEEPSEEK41_GPU
if (ds4_session_is_ds41(s))
return ds41_session_tp_verify(s, drafts, (uint32_t)draft_n, flags, err, errlen);
#endif
ds4_spec_frontier frontier;
memset(&frontier, 0, sizeof(frontier));
const int start = s->checkpoint.len;
@@ -83088,6 +83191,17 @@ int ds4_session_tp_spec_cycle(ds4_session *s, const int *drafts, int draft_n,
int ds4_session_tp_dspark_draft(ds4_session *s, int token, int pos,
uint32_t mode, char *err, size_t errlen) {
#ifdef DS4_HAS_DEEPSEEK41_GPU
if (s && ds4_session_is_ds41(s)) {
ds4_engine *e = s->engine;
if (e->tp.active && e->tp.rank == 1 && e->tp.vocab_split &&
mode == DS4_TP_DSPARK_DRAFT && pos > 0 &&
s->checkpoint_valid && pos == s->checkpoint.len &&
(uint32_t)pos == s->ds41_graph.pos && ds41_session_draft(s, token)) return 0;
snprintf(err, errlen, "tp: V4.1 draft failed or frontier differs");
return 1;
}
#endif
#if !defined(DS4_NO_GPU) && !defined(__APPLE__) && !defined(DS4_ROCM_BUILD) && \
defined(DS4_HAS_DEEPSEEK41_GPU)
ds4_engine *e = s ? s->engine : NULL;
@@ -85912,7 +86026,7 @@ static int ds4_sessions_eval_batch_with_prefill_cuda(
int ds4_test_ds41_verify_tokens(ds4_session *s, const int *tokens, int count,
float *logits, char *err, size_t errlen) {
if (!s || !ds4_session_is_ds41(s) || !tokens || !logits || count < 1 ||
count > (int)DS41_VERIFY_ROWS || !s->checkpoint_valid || s->engine->tp.active ||
count > (int)DS41_VERIFY_ROWS || !s->checkpoint_valid ||
!s->ds41_graph.dspark || s->checkpoint.len + count > s->ctx_size) {
snprintf(err, errlen, "invalid V4.1 verification test block");
return 1;
@@ -85927,14 +86041,13 @@ int ds4_test_ds41_verify_tokens(ds4_session *s, const int *tokens, int count,
return 0;
}
ds41_dspark *d = s->ds41_graph.dspark;
if (!ds41_dspark_verify(d, &s->ds41_graph, &s->engine->model, &s->engine->weights, tokens, count) ||
!ds41_dspark_commit(d, &s->ds41_graph, count, s->logits)) {
if (!ds41_session_verify(s, tokens, count) ||
!ds41_session_commit(s, tokens, count, count)) {
ds4_session_invalidate(s);
snprintf(err, errlen, "V4.1 verification test failed");
return 1;
}
memcpy(logits, d->host_verified, (uint64_t)count * DS4_N_VOCAB * sizeof(float));
for (int i = 0; i < count; i++) token_vec_push(&s->checkpoint, tokens[i]);
return 0;
}
#endif
@@ -85955,7 +86068,7 @@ static int ds41_session_speculate(ds4_session *s, int first_token, int cap, int
if (stats) { s->dspark_stats.cycles++; s->dspark_stats.first_tokens++; }
bool ready = d && !e->quality && !e->dspark_strict && cap > 1 &&
first_token != eos && g->pos && g->pos <= 1048576u - DS41_DRAFT_ROWS &&
!e->tp.active;
ds41_tp_batch_enabled(g);
/* Old session files contain target caches, not draft features. Warm the
* window with ordinary decoding instead of inventing missing features. */
if (ready) for (uint32_t pos = g->pos > 128u ? g->pos - 128u : 0; pos < g->pos; pos++)
@@ -85969,7 +86082,7 @@ static int ds41_session_speculate(ds4_session *s, int first_token, int cap, int
}
const bool exact = temperature > 0 && e->dspark_exact_sampling;
const double started = now_sec();
if (!ds41_dspark_forward(d, &e->model, &e->weights, &e->mtp_model, g->pos, first_token))
if (!ds41_session_draft(s, first_token))
goto fail;
int tokens[DS41_VERIFY_ROWS] = {first_token};
uint32_t count = 1;
@@ -86008,7 +86121,7 @@ static int ds41_session_speculate(ds4_session *s, int first_token, int cap, int
s->dspark_stats.proposed_tokens += count - 1u;
ds4_dspark_stats_note_len(s->dspark_stats.draft_len_hist, count - 1u);
}
if (!ds41_dspark_verify(d, g, &e->model, &e->weights, tokens, count)) goto fail;
if (!ds41_session_verify(s, tokens, count)) goto fail;
uint32_t keep = 1;
int correction = -1;
for (; keep < count; keep++) {
@@ -86026,9 +86139,8 @@ static int ds41_session_speculate(ds4_session *s, int first_token, int cap, int
}
}
}
if (!ds41_dspark_commit(d, g, keep, s->logits)) goto fail;
if (!ds41_session_commit(s, tokens, count, keep)) goto fail;
for (uint32_t i = 0; i < keep; i++) {
token_vec_push(&s->checkpoint, tokens[i]);
accepted[i] = tokens[i];
}
const uint32_t accepted_drafts = keep - 1u;
@@ -86061,6 +86173,7 @@ fail:
g->valid = s->checkpoint_valid = false;
if (stats) s->dspark_stats.verifier_errors++;
snprintf(err, errlen, "V4.1 DSpark draft/verification failed at position %u", g->pos);
if (e->tp.active) ds4_tp_mark_failed(e->tp.ctx);
return -1;
}
#endif
@@ -87117,6 +87230,21 @@ int ds4_session_restore_speculative_prefix(ds4_session *s, int pos) {
(void)s; (void)pos;
return 1;
#else
#ifdef DS4_HAS_DEEPSEEK41_GPU
if (s && ds4_session_is_ds41(s)) {
ds41_gpu_graph *g = &s->ds41_graph;
ds41_dspark *d = g->dspark;
if (!s->checkpoint_valid || !g->valid || !d || pos < 0 ||
d->rewind_end != (uint32_t)s->checkpoint.len ||
(uint32_t)pos < d->verify_start || (uint32_t)pos >= d->rewind_end) return 1;
if (!ds41_dspark_commit(d, g, (uint32_t)pos - d->verify_start, s->logits)) {
s->checkpoint_valid = false;
return -1;
}
s->checkpoint.len = pos;
return 0;
}
#endif
if (!s || !s->checkpoint_valid || !s->dspark_rewind_logits ||
s->dspark_rewind_end <= s->dspark_rewind_start ||
s->dspark_rewind_end != s->checkpoint.len ||
@@ -87213,11 +87341,20 @@ bool ds4_session_rewind_speculative(ds4_session *s, int pos) {
d->rewind_end != (uint32_t)s->checkpoint.len ||
(uint32_t)pos < d->verify_start || (uint32_t)pos >= d->rewind_end)
return false;
if (!ds41_dspark_commit(d, g, (uint32_t)pos - d->verify_start, s->logits)) {
const bool mirror = ds4_session_tp_leader(s);
ds4_engine *e = s->engine;
if (mirror && !ds4_tp_send_spec_restore(e->tp.ctx, s->tp_session_id, pos))
return false;
const int local = ds4_session_restore_speculative_prefix(s, pos);
int peer = 0;
char err[128];
if (mirror && !ds4_tp_wait_command_status(e->tp.ctx, s->tp_session_id,
&peer, "V4.1 speculative restore", err, sizeof(err))) peer = -1;
if (local || peer) {
s->checkpoint_valid = false;
if (mirror) ds4_session_invalidate(s);
return false;
}
s->checkpoint.len = pos;
return true;
}
#endif
ds4_deepseek41_dspark.h
+17 -3
View File
@@ -53,6 +53,7 @@ static void ds41_dspark_free(ds41_dspark *d) {
ds41_gpu_graph *g = &d->verified[i];
ds4_gpu_tensor_free(g->residual); ds4_gpu_tensor_free(g->pre);
ds4_gpu_tensor_free(g->logits);
ds4_gpu_tensor_free(g->tp_logits_half);
for (unsigned j = 0; j < 4; j++) {
ds4_gpu_tensor_free(g->verify_previous_kv[j]);
ds4_gpu_tensor_free(g->verify_previous_score[j]);
@@ -270,9 +271,11 @@ static bool ds41_dspark_stage(ds41_dspark *d, const ds4_model *m, uint32_t stage
* GPU. Markov sampling is sequential and performed after this stage chain. */
static DS4_MAYBE_UNUSED bool ds41_dspark_forward(ds41_dspark *d, const ds4_model *target,
const ds4_weights *tw, const ds4_model *draft,
uint32_t pos, int token) {
uint32_t pos, int token, ds41_gpu_graph *target_graph) {
if (!d || !pos || token < 0 || (uint32_t)token >= DS4_N_VOCAB) return false;
ds41_prefill_row *b = &d->batch;
const bool local_experts = target_graph && target_graph->tp_world == 2;
if (local_experts) ds4_gpu_tp_suspend_expert_sharding(1);
bool ok = ds4_gpu_begin_commands() && ds41_dspark_seed(d, draft, pos);
if (!ok) fprintf(stderr, "ds4: V4.1 DSpark seed failed\n");
const float pre[DS41_DRAFT_ROWS][4] = {{1, 0, 0, 0}, {1, 0, 0, 0}, {1, 0, 0, 0}, {1, 0, 0, 0}, {1, 0, 0, 0}};
@@ -292,9 +295,12 @@ static DS4_MAYBE_UNUSED bool ds41_dspark_forward(ds41_dspark *d, const ds4_model
if (ok) ok = ds4_gpu_hc_weighted_sum_split_tensor(b->x, b->residual,
b->ffn_split, DS4_N_EMBD, DS4_N_HC) &&
ds4_gpu_dsv41_quantize(b->x, DS4_N_EMBD, DS41_DRAFT_ROWS, DS4_V41_BF16) &&
ds41_norm_batch(b->norm, b->x, draft, d->weights->stage[2].norm, DS41_DRAFT_ROWS) &&
ds41_norm_batch(b->norm, b->x, draft, d->weights->stage[2].norm, DS41_DRAFT_ROWS);
if (ok) ok = target_graph ?
ds41_output_projection(target_graph, d->logits, target, tw, b->norm, DS41_DRAFT_ROWS) :
ds41_matmul_batch(d->logits, target, tw->output, b->norm, DS41_DRAFT_ROWS, false);
if (ds4_gpu_commands_active() && !ds4_gpu_end_commands()) ok = false;
if (local_experts) ds4_gpu_tp_suspend_expert_sharding(0);
if (!ok) d->kv_valid = false;
return ok;
}
@@ -332,7 +338,7 @@ static DS4_MAYBE_UNUSED bool ds41_dspark_markov(ds41_dspark *d, const ds4_model
static DS4_MAYBE_UNUSED bool ds41_dspark_verify(ds41_dspark *d, ds41_gpu_graph *g,
const ds4_model *m, const ds4_weights *w, const int *tokens, uint32_t count) {
if (!d || !g->valid || count < 2 || count > DS41_VERIFY_ROWS ||
g->pos > g->ctx || count > g->ctx - g->pos || g->tp_logits_half) return false;
g->pos > g->ctx || count > g->ctx - g->pos || !ds41_tp_batch_enabled(g)) return false;
const uint64_t kv = DS4_N_HEAD_DIM * 4u, feature = DS4_N_EMBD * 4u;
ds41_gpu_graph *graphs[DS41_VERIFY_ROWS];
bool ok = ds4_gpu_begin_commands();
@@ -352,6 +358,7 @@ static DS4_MAYBE_UNUSED bool ds41_dspark_verify(ds41_dspark *d, ds41_gpu_graph *
}
ds41_gpu_graph *row = &d->verified[i];
ds4_gpu_tensor *res = row->residual, *pre = row->pre, *logits = row->logits;
ds4_gpu_tensor *half = row->tp_logits_half;
ds4_gpu_tensor *pool_k[4], *pool_s[4];
memcpy(pool_k, row->verify_previous_kv, sizeof(pool_k));
memcpy(pool_s, row->verify_previous_score, sizeof(pool_s));
@@ -360,6 +367,13 @@ static DS4_MAYBE_UNUSED bool ds41_dspark_verify(ds41_dspark *d, ds41_gpu_graph *
row->pre = pre ? pre : ds4_gpu_tensor_view(d->verify_pre, i * DS4_N_HC * 4u, DS4_N_HC * 4u);
row->logits = logits ? logits : ds4_gpu_tensor_view(d->verify_logits,
(uint64_t)i * DS4_N_VOCAB * 4u, (uint64_t)DS4_N_VOCAB * 4u);
row->tp_logits_half = NULL;
if (g->tp_logits_half) {
const uint64_t bytes = (uint64_t)DS4_N_VOCAB * 2u;
row->tp_logits_half = half ? half : ds4_gpu_tensor_view(row->logits,
g->tp_rank * bytes, bytes);
ok = ok && row->tp_logits_half;
}
row->verify_prefix = true;
for (uint32_t owner = 0; owner < 4; owner++) {
const uint64_t off = (i * 8u + owner * 2u) * kv;
ds4_tp.c
+5 -1
View File
@@ -2522,7 +2522,8 @@ int ds4_tp_agree_execution_mode(ds4_tp *tp, uint32_t flags, char *err, size_t er
return 0;
}
const uint32_t known = DS4_TP_MODE_EXPERT_INTERMEDIATE_SPLIT |
DS4_TP_MODE_V41_SHARED_OWNER | DS4_TP_MODE_V41_SHARED_BALANCE;
DS4_TP_MODE_V41_SHARED_OWNER | DS4_TP_MODE_V41_SHARED_BALANCE |
DS4_TP_MODE_V41_DSPARK;
if (peer[0] != mine[0] || peer[1] != flags || (flags & ~known) != 0) {
ds4_tp_mark_failed(tp);
tp_set_err(err, errlen, "tp execution mode mismatch (peer 0x%x, local 0x%x); "
@@ -3854,6 +3855,9 @@ int ds4_tp_worker_run(ds4_engine *engine, const ds4_tp_options *opt) {
(int)command.n_tokens,
(uint32_t)command.value,
err, sizeof(err));
if (ds4_engine_is_deepseek41(engine) &&
!ds4_tp_send_command_ack(tp, command.session_id, spec_rc ? -1 : 0))
spec_rc = 1;
if (spec_rc != 0) {
ds4_log(stderr, DS4_LOG_ERROR, "tp worker verify: %s", err);
rc = 1;
ds4_tp.h
+1
View File
@@ -112,6 +112,7 @@ enum {
DS4_TP_MODE_EXPERT_INTERMEDIATE_SPLIT = 1u,
DS4_TP_MODE_V41_SHARED_OWNER = 2u,
DS4_TP_MODE_V41_SHARED_BALANCE = 4u,
DS4_TP_MODE_V41_DSPARK = 8u,
};
/* Call on both engines before binding the GPU slab. Resident layouts and
* shared-expert ownership must match before either rank can reduce results. */
gguf-tools/quality-testing/score_official.c
+5 -6
View File
@@ -782,18 +782,17 @@ int main(int argc, char **argv) {
}
}
if (ctx_size < 1024) ctx_size = 1024;
#ifdef DS4_TEST_HOOKS
if (verify_batch && (!mtp_path || session_count != 1 || tp.role != DS4_TP_NONE ||
dist.role != DS4_DISTRIBUTED_NONE))
die("--verify-batch requires --mtp-model and one local session");
#endif
char tp_err[256] = {0};
if (!ds4_tp_adopt_distributed_options(&tp, &dist,
tp_err, sizeof(tp_err))) {
fprintf(stderr, "score_official: %s\n", tp_err);
return 2;
}
#ifdef DS4_TEST_HOOKS
if (verify_batch && (!mtp_path || session_count != 1 ||
dist.role != DS4_DISTRIBUTED_NONE))
die("--verify-batch requires --mtp-model and one session without pipeline execution");
#endif
if (tp.role == DS4_TP_WORKER || dist.role == DS4_DISTRIBUTED_WORKER) {
die("score_official: start workers with ./ds4");
}
tests/test_deepseek41_dspark.c
+59 -28
View File
@@ -73,9 +73,10 @@ done:
return rc;
}
static int check_runtime(const char *target, const char *draft) {
static int check_runtime(const char *target, const char *draft, const ds4_tp_options *tp_opt) {
int rc = 1;
ds4_engine *engine = NULL;
ds4_tp *tp = NULL;
ds4_session *a = NULL, *b = NULL;
ds4_tokens prompt = {0};
FILE *snapshot = NULL;
@@ -88,7 +89,18 @@ static int check_runtime(const char *target, const char *draft) {
#endif
.context_size = 2048, .power_percent = 100, .ssd_streaming = true,
.ssd_streaming_cache_bytes = UINT64_C(40) << 30, .dspark = true};
if (tp_opt) { opt.tp = *tp_opt; opt.ssd_streaming = false; }
REQUIRE(!ds4_engine_open(&engine, &opt));
if (tp_opt) {
ds4_tp_identity id = {.gguf_bytes = ds4_engine_model_bytes(engine),
.model_id = ds4_engine_model_id(engine), .n_layer = DS4_N_LAYER,
.n_embd = DS4_N_EMBD, .n_vocab = DS4_N_VOCAB,
.quant_bits = ds4_engine_routed_quant_bits(engine), .ctx_size = 2048};
ds4_engine_tp_gate_schedule(engine, &id.gate_slot_start, &id.gate_slot_step,
&id.gates_per_token, id.gate_slot_mask);
REQUIRE(ds4_tp_create(&tp, &opt.tp, &id, err, sizeof(err)));
REQUIRE(ds4_engine_tp_bind(engine, tp, err, sizeof(err)));
}
REQUIRE(!ds4_session_create(&a, engine, 2048));
REQUIRE(!ds4_session_create(&b, engine, 2048));
for (unsigned i = 0; i < 127; i++) ds4_tokens_push(&prompt, 300 + i % 23);
@@ -97,30 +109,32 @@ static int check_runtime(const char *target, const char *draft) {
REQUIRE(snapshot && !ds4_session_save_payload(a, snapshot, err, sizeof(err)));
const uint64_t bytes = ds4_session_payload_bytes(a);
for (unsigned keep = 0; keep <= 6; keep++) {
rewind(snapshot);
REQUIRE(!ds4_session_load_payload(a, snapshot, bytes, err, sizeof(err)));
rewind(snapshot);
REQUIRE(!ds4_session_load_payload(b, snapshot, bytes, err, sizeof(err)));
if (tp) {
ds4_session_invalidate(a); ds4_session_invalidate(b);
REQUIRE(!ds4_session_sync(a, &prompt, err, sizeof(err)));
REQUIRE(!ds4_session_sync(b, &prompt, err, sizeof(err)));
} else {
rewind(snapshot);
REQUIRE(!ds4_session_load_payload(a, snapshot, bytes, err, sizeof(err)));
rewind(snapshot);
REQUIRE(!ds4_session_load_payload(b, snapshot, bytes, err, sizeof(err)));
}
int ta[6], tb[6];
for (unsigned i = 0; i < 6; i++) {
ta[i] = 700 + i;
tb[i] = i < keep ? ta[i] : 900 + (int)i;
}
ds41_dspark *da = a->ds41_graph.dspark, *db = b->ds41_graph.dspark;
REQUIRE(ds41_dspark_verify(da, &a->ds41_graph, &engine->model, &engine->weights, ta, 6));
REQUIRE(ds41_dspark_verify(db, &b->ds41_graph, &engine->model, &engine->weights, tb, 6));
REQUIRE(ds41_session_verify(a, ta, 6));
REQUIRE(ds41_session_commit(a, ta, 6, 6));
REQUIRE(ds41_session_verify(b, tb, 6));
REQUIRE(ds41_session_commit(b, tb, 6, keep));
double worst = 0;
for (uint64_t i = 0; i < (uint64_t)keep * DS4_N_VOCAB; i++)
worst = fmax(worst, fabs(da->host_verified[i] - db->host_verified[i]));
fprintf(stderr, "verify prefix=%u causal max=%.8g\n", keep, worst);
REQUIRE(worst < 0.0001);
REQUIRE(ds41_dspark_commit(da, &a->ds41_graph, 6, a->logits));
REQUIRE(ds41_dspark_commit(db, &b->ds41_graph, keep, b->logits));
for (unsigned i = 0; i < 6; i++) ds4_tokens_push(&a->checkpoint, ta[i]);
if (keep < 6) REQUIRE(ds4_session_rewind_speculative(a, 127 + keep));
for (unsigned i = 0; i < keep; i++) {
ds4_tokens_push(&b->checkpoint, tb[i]);
}
REQUIRE(a->checkpoint.len == b->checkpoint.len && a->checkpoint_valid);
REQUIRE(!memcmp(a->logits, b->logits, DS4_N_VOCAB * sizeof(float)));
for (unsigned i = 0; i < 8; i++) {
@@ -158,18 +172,21 @@ static int check_runtime(const char *target, const char *draft) {
REQUIRE(a->checkpoint.len == before + count);
}
}
rewind(snapshot);
REQUIRE(!ds4_session_save_payload(a, snapshot, err, sizeof(err)));
const uint64_t saved_bytes = ds4_session_payload_bytes(a);
rewind(snapshot);
REQUIRE(!ds4_session_load_payload(b, snapshot, saved_bytes, err, sizeof(err)));
REQUIRE(!b->ds41_graph.dspark->kv_valid);
for (unsigned i = 0; i < 3; i++) {
REQUIRE(!ds4_session_eval(a, 1500 + i, err, sizeof(err)));
REQUIRE(!ds4_session_eval(b, 1500 + i, err, sizeof(err)));
REQUIRE(!memcmp(a->logits, b->logits, DS4_N_VOCAB * sizeof(float)));
if (!tp) {
rewind(snapshot);
REQUIRE(!ds4_session_save_payload(a, snapshot, err, sizeof(err)));
const uint64_t saved_bytes = ds4_session_payload_bytes(a);
rewind(snapshot);
REQUIRE(!ds4_session_load_payload(b, snapshot, saved_bytes, err, sizeof(err)));
REQUIRE(!b->ds41_graph.dspark->kv_valid);
for (unsigned i = 0; i < 3; i++) {
REQUIRE(!ds4_session_eval(a, 1500 + i, err, sizeof(err)));
REQUIRE(!ds4_session_eval(b, 1500 + i, err, sizeof(err)));
REQUIRE(!memcmp(a->logits, b->logits, DS4_N_VOCAB * sizeof(float)));
}
}
fprintf(stderr, "V4.1 DSpark exact=%u sampling, frontier and post-verify snapshot: PASS\n", exact);
fprintf(stderr, "V4.1 DSpark exact=%u sampling and frontier%s: PASS\n", exact,
tp ? " (physical TP)" : ", post-verify snapshot");
}
{
int accepted[6];
@@ -200,13 +217,27 @@ done:
if (err[0]) fprintf(stderr, "%s\n", err);
if (snapshot) fclose(snapshot);
ds4_tokens_free(&prompt);
ds4_session_free(b); ds4_session_free(a); ds4_engine_close(engine);
ds4_session_free(b); ds4_session_free(a);
if (tp) (void)ds4_tp_send_stop(tp);
ds4_engine_close(engine); ds4_tp_free(tp);
return rc;
}
int main(int argc, char **argv) {
if (argc == 2 && !strcmp(argv[1], "--candidate-mask")) return check_candidate_mask();
if (argc == 4 && !strcmp(argv[3], "--runtime")) return check_runtime(argv[1], argv[2]);
if (argc == 4 && !strcmp(argv[3], "--runtime")) return check_runtime(argv[1], argv[2], NULL);
if (argc == 8 && !strcmp(argv[3], "--runtime-tp")) {
char *end;
long port = strtol(argv[5], &end, 10);
if (*end || port < 1 || port > 65535) return 2;
long gid = strtol(argv[7], &end, 10);
if (*end || gid < 0 || gid > 255) return 2;
ds4_tp_options opt = {.role = DS4_TP_LEADER, .requested = true,
.listen_host = argv[4], .listen_port = (int)port,
.transport = DS4_TP_TRANSPORT_RDMA, .rdma_device = argv[6],
.rdma_gid_index = (int)gid, .rdma_gid_index_set = true};
return check_runtime(argv[1], argv[2], &opt);
}
if (argc != 4) {
fprintf(stderr, "usage: %s TARGET_GGUF DSPARK_GGUF OUTPUT_DIRECTORY\n", argv[0]);
return 2;
@@ -251,7 +282,7 @@ int main(int argc, char **argv) {
REQUIRE(ds41_dspark_capture(d, 37 + stage, 0, hc, ROWS));
REQUIRE(ds4_gpu_end_commands());
}
REQUIRE(ds41_dspark_forward(d, &target, &tw, &draft, ROWS, 100));
REQUIRE(ds41_dspark_forward(d, &target, &tw, &draft, ROWS, 100, NULL));
REQUIRE(d->kv_valid && d->kv_end == ROWS);
REQUIRE(dump_tensor(argv[3], "main_x", d->main_x, DS41_DRAFT_WINDOW * DS4_N_EMBD));
for (unsigned stage = 0; stage < 3; stage++) {
@@ -263,7 +294,7 @@ int main(int argc, char **argv) {
REQUIRE(dump_tensor(argv[3], "norm", d->batch.norm, DS41_DRAFT_ROWS * DS4_N_EMBD));
REQUIRE(dump_tensor(argv[3], "logits", d->logits, DS41_DRAFT_ROWS * DS4_N_VOCAB));
REQUIRE(ds4_gpu_tensor_read(d->logits, 0, d->host_logits, DS41_DRAFT_ROWS * DS4_N_VOCAB * sizeof(float)));
REQUIRE(ds41_dspark_forward(d, &target, &tw, &draft, ROWS, 100));
REQUIRE(ds41_dspark_forward(d, &target, &tw, &draft, ROWS, 100, NULL));
float *second = malloc(DS41_DRAFT_ROWS * DS4_N_VOCAB * sizeof(float));
REQUIRE(second);
bool repeated = ds4_gpu_tensor_read(d->logits, 0, second, DS41_DRAFT_ROWS * DS4_N_VOCAB * sizeof(float)) &&
tests/test_tp_link.c
+5 -2
View File
@@ -287,9 +287,12 @@ int main(int argc, char **argv) {
error, sizeof(error)));
CHECK(ds4_tp_agree_execution_mode(tp, DS4_TP_MODE_V41_SHARED_OWNER |
DS4_TP_MODE_V41_SHARED_BALANCE, error, sizeof(error)));
CHECK(ds4_tp_agree_execution_mode(tp, DS4_TP_MODE_V41_DSPARK |
DS4_TP_MODE_V41_SHARED_OWNER, error, sizeof(error)));
if (getenv("DS4_TEST_TP_MODE_MISMATCH")) {
const uint32_t mode = atoi(getenv("DS4_TEST_TP_MODE_MISMATCH")) == 2 ?
DS4_TP_MODE_V41_SHARED_BALANCE : DS4_TP_MODE_EXPERT_INTERMEDIATE_SPLIT;
const int test = atoi(getenv("DS4_TEST_TP_MODE_MISMATCH"));
const uint32_t mode = test == 3 ? DS4_TP_MODE_V41_DSPARK :
test == 2 ? DS4_TP_MODE_V41_SHARED_BALANCE : DS4_TP_MODE_EXPERT_INTERMEDIATE_SPLIT;
CHECK(!ds4_tp_agree_execution_mode(tp, rank ?
mode : 0, error, sizeof(error)));
CHECK(ds4_tp_failed(tp));