Notes

Backpressure in a homegrown job queue

Had a worker pool that looked fine in every load test until a burst of ~5k jobs landed in one second. Memory climbed until the process got OOM-killed. The pool used an unbounded Go channel as the queue — producers never blocked, so nothing ever signalled "slow down."

Fix was boring: bound the channel, and expose a gauge for how full it is. Producers now block (or shed load, depending on the endpoint) once the buffer is 80% full, and the gauge turned a silent OOM into a visible, alertable metric weeks before it would have mattered again.

Systemd template units for per-tenant workers

Used to run one supervisor process that forked a worker per tenant. Replaced it with a systemd template unit, worker@.service, instantiated per tenant id via systemctl enable worker@42.service.

Gained: independent restart policy per tenant, proper journald tagging (journalctl -u worker@42), and resource limits via MemoryMax= in a drop-in instead of hand-rolled cgroup code. Lost: nothing I've missed yet.

Postgres partition pruning gotchas

Range-partitioned a large events table by created_at, expecting query times to drop. They didn't — until I noticed the application was filtering on a derived, app-side-computed timestamp column instead of the partition key directly. Postgres can't prune what it can't prove is monotonic with the partition expression.

EXPLAIN (ANALYZE, BUFFERS) showed every partition being scanned. Rewriting the filter to use the partition key directly dropped the query from ~40 partitions scanned to 2.