Ingestion delay in app.scalyr.com

Incident Report for DataSet

Resolved

This incident has been resolved.

Posted Oct 24, 2024 - 23:49 UTC

Monitoring

The UI should now be loading as expected. We have increased the database connection limit to accommodate more concurrent connections. The queue is in the process of recovering and is gradually processing the backlog of events.

Posted Oct 24, 2024 - 19:11 UTC

Update

The aggressive scaling out of servers led to a 500 error when loading the page due to hitting the database connection limit. We are currently in the process of scaling the servers back in, and the error should be resolved shortly.

Posted Oct 24, 2024 - 18:11 UTC

Identified

A misconfiguration deployed this morning prevented the servers from scaling up correctly. We are currently in the process of manually scaling up the servers to manage the ingestion volume effectively.

Posted Oct 24, 2024 - 17:13 UTC

Investigating

We are currently investigating the issue.

Posted Oct 24, 2024 - 16:54 UTC