A cluster message whose topic is known on the receiving node but has no subscriber there was published to nobody and uncounted, and its Keepalive kept the topic from ever expiring: a passive node held every topic the active one published to. It is now counted in ntfy_cluster_messages_wasted_total and not published, so idle topics expire after topicExpungeAfter.
A node cut off from its database only showed in the logs. The health check loop now always
runs and pings the primary too; ntfy_db_up (1/0, per pool, next to ntfy_db_pool_*) is the
result of the last check, so an alert can fire on ntfy_db_up{role="primary"} == 0.
Move the base-url check out of the else-if chain instead of into a
helper, and test both cases with an invalid auth-default-access, which
fails right after the checks, instead of starting a server.
The WebSocket handler deleted every version of a message's sequence
before adding it, so a replayed message that was already stored came
back as unread and notified again, and a replayed older version
replaced a newer one. addNotification now replaces older versions of
the sequence itself, and skips a message if a newer version is stored.
Fixes#2090
Tier to tier changes keep spend (no tier to a tier never did), only a tier's daily email limit is a cap, only the read over the size limit is refunded, and the #2119 bullet describes the fix.
Claude-Session: https://claude.ai/code/session_01DeXuhSjq5Gta5qPJgSceFe
The server replays the whole second of a since=<id> marker, so a poll
returns messages the user deleted. Remember deleted notifications and
skip them (and older versions of their sequence), and don't move the
marker back when a poll only replayed known messages, which made each
poll replay a different part of that second.
Fixes#2098
Stale read the limiter key's last use, which every charged publish refreshes, so a topic kept
a vanished subscriber's limits (and its UnifiedPush subscription) for as long as anyone
published to it. The visitor now carries its own keepalive, set when it is built and by the
subscribe handlers' keepalive loops, like the shared visitor's seen was.
Also from the review: Prune returns and the manager logs the number of forgotten keys, the
unused visitorID wrapper goes, a stale comment and test name are updated, and the release
notes get the #1051 entry.
Claude-Session: https://claude.ai/code/session_01DeXuhSjq5Gta5qPJgSceFe
A tier with a daily email limit of zero sends no emails (the limit service reads a daily
limit of zero as "no cap"), a tier edited in place flushes the visitor's pending user stats
before resetting its local counts like a tier ID change does, and a bucket refill with a
timestamp older than its last one neither charges tokens nor moves the refill point back.
The tests pin what the rework changed and what it must keep: shared keys count once in the
user stats, an expunged visitor continues its counts, the daily reset keeps the buckets,
zero bursts admit nothing, seeds apply once, prefix keying, tier changes in all directions,
concurrency on a shared key, and the account page's remaining values matching admissions.
Claude-Session: https://claude.ai/code/session_01DeXuhSjq5Gta5qPJgSceFe
The Matrix gateway built its publish request from the push key without the
router's topic path check. Publishing sanitizes the topic after authorization
(it strips NUL bytes and invalid UTF-8), so an anonymous push key like
/sec%00ret was authorized against an unmatched topic and stored under the
protected topic "secret".
/v1/tiers is now the admin API for tiers (GET/POST/PUT/DELETE), instead
of the unreleased /v1/admin/tiers. The public price list for the upgrade
dialog moved to /v1/billing/tiers; GET /v1/tiers still returns it to
non-admins until 2026-11-22, for web apps from before v2.33.0.
topicSubscriber no longer stores the user ID next to the user (userID() derives it), which also brings the struct below its size before the user was added. The cancel loops assign the matching subscribers to a variable instead of calling subscribersMatching inside the for header. The periodic access cache reload runs every 63 s: access changes are rare and peers are told directly, so the interval only bounds a missed signal.
Canceling a subscriber logs the topic's context, which takes the topic lock. The cancels held that lock while they logged: CancelSubscribersExceptUser held the write lock (deadlock at once), the per-user cancels a read lock (deadlock as soon as a canceled stream's Unsubscribe waited for the write lock). The context is only evaluated with debug logging or log-level-overrides on. In the lab this froze the access cache reload loop after its first re-check. The cancels now collect the matching subscribers under the lock and cancel them after releasing it.
globalLevelWithOverride iterated the overrides map without the log lock while SetLevelOverride changes it in place; it now holds the read lock while iterating.
Access changes are rare and a full reload is cheap (about 15 ms on ntfy.sh's 5,500 entries), so the access cache no longer reloads per user. Every reload is full and serialized (which replaces the sequence-number race guard), and the diff after each reload re-checks the live subscribers on the changed topics.
An access write (reservation, allow, reset, user removal, role change) reloads the local cache and tells the other cluster nodes "access changed" (cluster.State.AccessChanged), which reload theirs; their diff ends the streams that lost access there. This replaces the relayed per-topic cancels and the targeted reloads. Only a deleted user's cancel is still relayed: a user without access entries (e.g. an admin) leaves nothing in the cache to compare. The periodic reload, the backstop for a lost signal and for writes by other processes (ntfy access CLI), now runs every 31 s instead of 87 s.
Other systems keep two things apart: draining before a shutdown (ephemeral, cleared by the restart: RabbitMQ drain, CockroachDB node drain) and an operator's maintenance flag (persistent cluster state: Kubernetes cordon, Nomad eligibility, Consul maint). This PR is the former, so it is named that way: Server.Drain (synchronous again; the on/off API is gone until the persistent maintenance flag exists), Cluster.SetDraining/Draining, the read-only phase flag readOnly, /v1/ready reports "draining": true, error 50301 says the node is shutting down, and docs/config.md has a "Draining on shutdown" section. A drain duration of 0 now closes all streams at once instead of disabling the drain.
Maintenance mode now has two phases. Entering (experimental-cluster-drain-delay, default 15s): /v1/ready fails and the node acts as passive in the cluster, but it still serves everything, so the load balancers can stop sending it requests without any request failing. Draining (experimental-cluster-drain-duration, default 1m): read-only, i.e. new subscriptions and all publishes get 503 (email gets SMTP 451, so mail servers retry instead of bouncing), and the streams are closed evenly. The defaults together fit systemd's 90s stop timeout. docs/config.md gets a maintenance mode section with the behavior table.
binwiederhier
Philipp C. Heckel