The wrapper only exposed the context-free database/sql methods, so callers
that need a deadline (or cancellation on shutdown) had to reach for
Primary() directly. Delegates to the primary, like Exec and Query.
The since marker was resolved by an inline subquery on the replica. A message
ID the replica had not seen yet (a client reconnecting right after receiving
a message, ahead of replication) resolved to 'id > 0' and replayed the topic's
whole retained history to that client.
Resolution is now its own lookup: on the replica, and only if the ID is
unknown there, on the primary. At most two point lookups before the replay,
and none on the primary unless a replica is configured. An ID unknown to the
primary as well (expired) keeps the full-replay fallback.
Now that the signal handler waits for Stop, two holes in Stop matter:
- Stop waits for every store to close, and those waits have no deadline of
their own: attachment.Store.Close waits for the sync loop, whose
AttachmentsWithSizes query runs against the database (and S3 deletes are in
there too). A wedged database parked Stop before it ever reached the message
cache, whose own 5s bound was then useless. Stop now gives up after 10s.
- A signal can arrive before Run has set up its listeners. Stop closed the
stores and consumed stopOnce, and Run then started listeners against closed
stores and blocked on the listener error, so the process neither served nor
exited and a second signal could not stop it. Run bails out when the server
is already stopped, outside the lock (a stuck shutdown holds it) and again
under it.
The final batch write has no deadline (no statement_timeout, and database/sql
is used without a context), so an unresponsive database would hold Stop until
systemd's stop timeout fires and SIGKILLs us. Cache.Close now gives up after
5 seconds, both on handing the pending messages to the batch writer and on
waiting for it, and logs that it did.
Without pool_max_idle_conns, database/sql keeps only 2 idle connections, so
any burst above that closed connections after use and re-opened them on the
next query: a DNS lookup, TCP + TLS handshake and SCRAM auth per query. Under
load this shows up as HTTP 500s whenever the resolver hiccups ("hostname
resolving error"). ntfy.sh runs pool_max_conns=30 without an idle setting, so
it churns connections the same way.
pool_max_idle_conns now defaults to pool_max_conns.
On Postgres all stores share one pool, and closeDatabases closed the user
manager first, which closed the pool under the message cache's final batch
write ("Cannot write message batch: sql: database is closed").
ensureUser checks v.User(), the handler re-reads it, and a concurrent
anonymous request on the same shared IP-keyed visitor can SetUser(nil) in
between -> nil dereference (~8x/day on ntfy.sh). Return 401 instead. The
regression test makes the race deterministic.
With cache-batch-size/-timeout set (ntfy.sh: 100 / 1500ms), accepted messages
wait in a BatchingQueue before they are written. Nothing drained that queue on
shutdown, and `ntfy serve` did not handle SIGTERM at all (only SIGHUP), so
every restart dropped up to one batch timeout's worth of accepted messages
from the cache: live subscribers got them, pollers and since= replays never
did.
- util.BatchingQueue.Close emits the pending elements and closes the output
channel (Enqueue after Close is a no-op)
- message.Cache.Close closes the queue and waits for the batch writer before
closing the database
- serve stops the server on SIGTERM/SIGINT and waits for Stop to finish (Run
returns as soon as the listeners close); Stop is idempotent now, since the
signal handler and serve both call it
binwiederhier