QueueHub System Status

Increased latency on API endpoint /api/v1/queues and /api/v1/jobs due to a database connection pool exhaustion. Root cause identified as a connection leak following a deployment of the job inspection service. Pool size increased and fix deployed.

Some self-hosted agents experienced intermittent disconnections from the relay service. Traced to a WebSocket frame parsing issue after the v1.8.0 agent update. A hotfix was rolled out within 45 minutes.

Queue dashboard took 10–15 seconds to load for workspaces with 200+ queues. Caused by an unoptimized query in the queue aggregation endpoint. Query was rewritten and a Redis cache layer added, reducing load time to under 2 seconds.

Amazon SQS queue metrics were delayed by up to 15 minutes for some FIFO queues. Root cause was a rate-limiting issue with the AWS API polling. Adjusted polling strategy to use batch requests and added adaptive rate limiting.

Email and Slack alert notifications were delayed by 5–10 minutes. Caused by a backlog in the notification worker queue due to a high volume of concurrent alert triggers. Worker pool was scaled up and alert deduplication logic improved.

Users experienced sporadic 502 errors when loading the QueueHub web app. Traced to an upstream DNS resolution timeout affecting our CDN origin. DNS caching configuration was updated and a secondary origin pool was added for failover.