Maintenance mode: enter, then drain read-only

Maintenance mode now has two phases. Entering (experimental-cluster-drain-delay, default 15s): /v1/ready fails and the node acts as passive in the cluster, but it still serves everything, so the load balancers can stop sending it requests without any request failing. Draining (experimental-cluster-drain-duration, default 1m): read-only, i.e. new subscriptions and all publishes get 503 (email gets SMTP 451, so mail servers retry instead of bouncing), and the streams are closed evenly. The defaults together fit systemd's 90s stop timeout. docs/config.md gets a maintenance mode section with the behavior table.
This commit is contained in:
binwiederhier committed 2026-10-23 13:20:33 +00:00
1 parent b197d5d8ca
commit 30eb5db6d8
11 files changed
+209 -45

No files matched your search

+9 -6
View File
@@ -87,12 +87,14 @@
# jobs, and it never closes its subscribers to send them elsewhere. Active nodes do not count
# it as a place to send theirs either. The member list reports each node's role, so load
# balancer agents can route to active nodes only.
# - experimental-cluster-drain-duration is how long an active node takes, on shutdown (SIGTERM),
# to close its subscriber connections, so they reconnect to the other nodes gradually instead
# of all at once. It is in maintenance mode meanwhile: new subscriptions get 503 and /v1/ready
# fails, so the load balancer must retry on 503. A node drains only if another active node is
# up. Make sure the service manager waits long enough (systemd: TimeoutStopSec, default 90s).
# Set to 0 to disable.
# - experimental-cluster-drain-delay and experimental-cluster-drain-duration control maintenance
# mode, which an active node enters on shutdown (SIGTERM) if another active node is up, so its
# clients move to the other nodes gradually instead of all at once. For the drain delay, the node
# is not ready (/v1/ready fails) but still serves everything, so the load balancer can stop
# sending it requests. Then it drains: it is read-only (new subscriptions and publishes get 503)
# and closes its subscriber connections evenly over the drain duration. Make sure the service
# manager waits long enough (systemd: TimeoutStopSec, default 90s). A drain duration of 0
# turns maintenance mode off.
#
# Clustering also requires "database-url": the shared PostgreSQL database carries the durable
# state and the cluster's own coordination tables.
@@ -107,6 +109,7 @@
# experimental-cluster-secret: <secret>
# experimental-cluster-batch-linger: 500ms
# experimental-cluster-role: active
# experimental-cluster-drain-delay: 15s
# experimental-cluster-drain-duration: 1m
# If "cache-file" is set, messages are cached in a local SQLite database instead of only in-memory.