On cron bursts (top of the hour), the single batch writer fell behind: it wrote one INSERT round
trip per message, so against a busy remote Postgres it managed ~100 rows/s while ~300/s arrived.
New messages took up to ~40s to reach the database (polls missed them), and publishes blocked on
the unbuffered queue for up to ~25s.
- Postgres: write each batch with one multi-row INSERT (up to 1000 rows per statement) instead of
one statement per row. SQLite keeps the row-by-row path.
- Skip DeleteScheduledBySequenceID when the client did not set a sequence ID: the sequence ID is
then the message's own fresh ID, so no scheduled message can match. This saves a transaction on
the primary for almost every publish.
- Buffer up to 100 batches in the BatchingQueue, so a briefly slow writer does not stall publishes.
- dbtest: add a latency proxy (CreateTestPostgresWithLatency, WithLatency) to test round trips.
The Content-Type header was compared to "text/markdown" as a whole string, so any media type parameter (most commonly charset) made ntfy treat the message as plain text. Parse the header with mime.ParseMediaType and compare only the media type.
Fixes#1992
Now that the signal handler waits for Stop, two holes in Stop matter:
- Stop waits for every store to close, and those waits have no deadline of
their own: attachment.Store.Close waits for the sync loop, whose
AttachmentsWithSizes query runs against the database (and S3 deletes are in
there too). A wedged database parked Stop before it ever reached the message
cache, whose own 5s bound was then useless. Stop now gives up after 10s.
- A signal can arrive before Run has set up its listeners. Stop closed the
stores and consumed stopOnce, and Run then started listeners against closed
stores and blocked on the listener error, so the process neither served nor
exited and a second signal could not stop it. Run bails out when the server
is already stopped, outside the lock (a stuck shutdown holds it) and again
under it.
On Postgres all stores share one pool, and closeDatabases closed the user
manager first, which closed the pool under the message cache's final batch
write ("Cannot write message batch: sql: database is closed").
With cache-batch-size/-timeout set (ntfy.sh: 100 / 1500ms), accepted messages
wait in a BatchingQueue before they are written. Nothing drained that queue on
shutdown, and `ntfy serve` did not handle SIGTERM at all (only SIGHUP), so
every restart dropped up to one batch timeout's worth of accepted messages
from the cache: live subscribers got them, pollers and since= replays never
did.
- util.BatchingQueue.Close emits the pending elements and closes the output
channel (Enqueue after Close is a no-op)
- message.Cache.Close closes the queue and waits for the batch writer before
closing the database
- serve stops the server on SIGTERM/SIGINT and waits for Stop to finish (Run
returns as soon as the listeners close); Stop is idempotent now, since the
signal handler and serve both call it
binwiederhier
Philipp C. Heckel
cipherprofessor
Aditya
Hunter Kehoe