Drain cluster nodes on shutdown

On SIGTERM, an active cluster node with another healthy active node first drains: new subscriptions (HTTP streams and WebSockets) get 503, which a load balancer retries on another node, /v1/ready fails with "draining": true, and the open streams are closed evenly over experimental-cluster-drain-duration (default 1m, fits systemd's 90s stop timeout; 0 disables). Clients reconnect a few at a time and resume with since=, instead of all at once. Publishes and polls are still served, since a load balancer does not retry a POST. A second signal skips the rest of the drain.

A stream registry on the server tracks open subscriber connections once each, however many topics they subscribe to.

server/server.yml

@@ -84,6 +84,11 @@
 #   jobs, and it never closes its subscribers to send them elsewhere. Active nodes do not count
 #   it as a place to send theirs either. The member list reports each node's role, so load
 #   balancer agents can route to active nodes only.
+# - experimental-cluster-drain-duration is how long an active node takes, on shutdown (SIGTERM),
+#   to close its subscriber connections, so they reconnect to the other nodes gradually instead
+#   of all at once. While draining, new subscriptions get 503 and /v1/ready fails, so the load
+#   balancer must retry on 503. A node drains only if another active node is up. Make sure the
+#   service manager waits long enough (systemd: TimeoutStopSec, default 90s). Set to 0 to disable.
 #
 # Clustering also requires "database-url": the shared PostgreSQL database carries the durable
 # state and the cluster's own coordination tables.
@@ -98,6 +103,7 @@
 # experimental-cluster-secret: <secret>
 # experimental-cluster-batch-linger: 500ms
 # experimental-cluster-role: active
+# experimental-cluster-drain-duration: 1m
 
 # If "cache-file" is set, messages are cached in a local SQLite database instead of only in-memory.
 # This allows for service restarts without losing messages in support of the since= parameter.