On cron bursts (top of the hour), the single batch writer fell behind: it wrote one INSERT round
trip per message, so against a busy remote Postgres it managed ~100 rows/s while ~300/s arrived.
New messages took up to ~40s to reach the database (polls missed them), and publishes blocked on
the unbuffered queue for up to ~25s.
- Postgres: write each batch with one multi-row INSERT (up to 1000 rows per statement) instead of
one statement per row. SQLite keeps the row-by-row path.
- Skip DeleteScheduledBySequenceID when the client did not set a sequence ID: the sequence ID is
then the message's own fresh ID, so no scheduled message can match. This saves a transaction on
the primary for almost every publish.
- Buffer up to 100 batches in the BatchingQueue, so a briefly slow writer does not stall publishes.
- dbtest: add a latency proxy (CreateTestPostgresWithLatency, WithLatency) to test round trips.
Five option names and their validation, ahead of the implementation, so the
config surface is settled and the cluster change is only code. The
experimental- prefix is there because the names may still change.
The options parse into server.Config and the cross-field rules are enforced:
a listen address needs database-url, a secret and a node id; a secret without
a listen address is an error rather than clustering being silently off; a
wildcard bind needs an explicit advertise URL; a negative linger is rejected.
Setting experimental-cluster-listen then refuses to start the server, since
accepting the options and running single-node would look like a working
cluster. The cluster change deletes that one line.
The Content-Type header was compared to "text/markdown" as a whole string, so any media type parameter (most commonly charset) made ntfy treat the message as plain text. Parse the header with mime.ParseMediaType and compare only the media type.
Fixes#1992
Now that the signal handler waits for Stop, two holes in Stop matter:
- Stop waits for every store to close, and those waits have no deadline of
their own: attachment.Store.Close waits for the sync loop, whose
AttachmentsWithSizes query runs against the database (and S3 deletes are in
there too). A wedged database parked Stop before it ever reached the message
cache, whose own 5s bound was then useless. Stop now gives up after 10s.
- A signal can arrive before Run has set up its listeners. Stop closed the
stores and consumed stopOnce, and Run then started listeners against closed
stores and blocked on the listener error, so the process neither served nor
exited and a second signal could not stop it. Run bails out when the server
is already stopped, outside the lock (a stuck shutdown holds it) and again
under it.
On Postgres all stores share one pool, and closeDatabases closed the user
manager first, which closed the pool under the message cache's final batch
write ("Cannot write message batch: sql: database is closed").
ensureUser checks v.User(), the handler re-reads it, and a concurrent
anonymous request on the same shared IP-keyed visitor can SetUser(nil) in
between -> nil dereference (~8x/day on ntfy.sh). Return 401 instead. The
regression test makes the race deterministic.
With cache-batch-size/-timeout set (ntfy.sh: 100 / 1500ms), accepted messages
wait in a BatchingQueue before they are written. Nothing drained that queue on
shutdown, and `ntfy serve` did not handle SIGTERM at all (only SIGHUP), so
every restart dropped up to one batch timeout's worth of accepted messages
from the cache: live subscribers got them, pollers and since= replays never
did.
- util.BatchingQueue.Close emits the pending elements and closes the output
channel (Enqueue after Close is a no-op)
- message.Cache.Close closes the queue and waits for the batch writer before
closing the database
- serve stops the server on SIGTERM/SIGINT and waits for Stop to finish (Run
returns as soon as the listeners close); Stop is idempotent now, since the
signal handler and serve both call it
binwiederhier
Philipp C. Heckel
cipherprofessor
Bee mines
Aditya
jvoisin