Worker pool sizing when CPU is idle but lag keeps climbing
On-call sees queue depth rising and asks for more workers. CPU graphs stay flat. The instinct is wrong more often than right—usually something else is the bottleneck.
Check broker prefetch and visibility
Consumers may be holding messages they cannot finish because visibility timeouts are too short, causing endless redelivery loops that look like lag without burning CPU.
External rate limits
Transcoding, OCR, and partner APIs cap throughput per token. Adding workers only increases 429 responses and retry storms. Measure end-to-end latency per stage before scaling horizontally.
Database connection ceilings
Workers block on pool acquisition. Postgres shows low CPU while jobs wait on checkout. Sometimes the fix is fewer workers with larger batch sizes, not more pods.
Partition skew
Keyed queues can leave one hot partition while others idle. Rebalance keys or add dedicated consumers for the hot shard.
When you are mid-incident, a tuning session walks through these checks live with your metrics open.