All Systems Operational
Main Site
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Jun 5, 2020
No incidents reported today.
Jun 4, 2020
No incidents reported.
Jun 3, 2020
No incidents reported.
Jun 2, 2020
Resolved -
We rolled a certificate update that allowed the old agents to use an alternate trust chain to connect to Scalyr. Most traffic is back. We are calling this issue resolved for now. If you are still seeing connect errors to Scalyr endpoints, please reach out to Scalyr support via email support at scalyr dot com.
We are working on adding monitoring for root and intermediate cert expiration among other things as learnings from this incident.
Jun 2, 04:33 UTC
We are working on adding monitoring for root and intermediate cert expiration among other things as learnings from this incident.
Jun 2, 04:33 UTC
Monitoring -
We have released release Scalyr Agent version 2.1.5 which addresses an issue with Windows. Windows users must apply this update.
May 30, 20:48 UTC
May 30, 20:48 UTC
Update -
We are working on release 2.1.5 to address an issue with Windows, and expect to release that within approximately an hour. Windows users must apply this update once it is available.
May 30, 19:51 UTC
May 30, 19:51 UTC
Identified -
We just rolled out a new Scalyr agent release (2.1.4) with new CA certs
bundled with the agent. Customers are highly encouraged to upgrade to this new
version of Scalyr agent (2.1.4) if they are seeing connection errors from the agent.
Scalyr agent upgrade instructions are documented at
https://app.scalyr.com/help/scalyr-agent#upgrades
You may want to run
sudo apt-get update
(or)
sudo yum update
based on your distro to see the new packages. Please also ensure you’re running the 2.1.4 version using distro commands
sudo apt-cache policy scalyr-agent-2
(or)
sudo yum info scalyr-agent-2
Docker images are also updated.
https://hub.docker.com/layers/scalyr/scalyr-k8s-agent/2.1.4/images/sha256-e010d29ad3950c9b5289a28b512cb8771ca797e194f4b1558298a950f48e0791?context=explore
If you do not wish to upgrade the agent at this time, you can refresh your CA certs
using the instructions available at
https://gist.github.com/lakshmi-kannan/1db93ede48745b1397acef40867fa2f5
May 30, 17:38 UTC
bundled with the agent. Customers are highly encouraged to upgrade to this new
version of Scalyr agent (2.1.4) if they are seeing connection errors from the agent.
Scalyr agent upgrade instructions are documented at
https://app.scalyr.com/help/scalyr-agent#upgrades
You may want to run
sudo apt-get update
(or)
sudo yum update
based on your distro to see the new packages. Please also ensure you’re running the 2.1.4 version using distro commands
sudo apt-cache policy scalyr-agent-2
(or)
sudo yum info scalyr-agent-2
Docker images are also updated.
https://hub.docker.com/layers/scalyr/scalyr-k8s-agent/2.1.4/images/sha256-e010d29ad3950c9b5289a28b512cb8771ca797e194f4b1558298a950f48e0791?context=explore
If you do not wish to upgrade the agent at this time, you can refresh your CA certs
using the instructions available at
https://gist.github.com/lakshmi-kannan/1db93ede48745b1397acef40867fa2f5
May 30, 17:38 UTC
Update -
We ship CA bundles along with the scalyr agent and in some specific installations, we rely on using the agent CA bundle to validate the cert as opposed to system shipped CA bundles. We believe customers experiencing the connection issues are in this category (using scalyr agent CA bundles in /usr/share/scalyr-agent-2/certs/ca_certs.crt).
If you are still experiencing connection errors from the agent, run following commands to update expired CA bundle shipped with the agent:
https://gist.github.com/lakshmi-kannan/1db93ede48745b1397acef40867fa2f5
We are working on rolling out a new release of scalyr agent with updated CA certs. It should be available shortly.
May 30, 16:01 UTC
If you are still experiencing connection errors from the agent, run following commands to update expired CA bundle shipped with the agent:
https://gist.github.com/lakshmi-kannan/1db93ede48745b1397acef40867fa2f5
We are working on rolling out a new release of scalyr agent with updated CA certs. It should be available shortly.
May 30, 16:01 UTC
Update -
We identified expired cert in our cert chain and replaced them with new ones. The impact is on a subset of customers. We are further investigating the partial nature of the impact.
May 30, 14:43 UTC
May 30, 14:43 UTC
Investigating -
We are looking into reports of connection errors to a few API endpoints ((https://agent.scalyr.com, https://upload.eu.scalyr.com) in both US and EU regions.
May 30, 13:20 UTC
May 30, 13:20 UTC
Jun 1, 2020
No incidents reported.
May 31, 2020
No incidents reported.
May 30, 2020
May 29, 2020
No incidents reported.
May 28, 2020
Resolved -
Error rates are back to normal. This incident is now resolved.
May 28, 19:12 UTC
May 28, 19:12 UTC
Monitoring -
An instance failure resulted in query errors and therefore impacted alert evaluations for a subset of customers in our US region. We have replaced the failed instance and are monitoring recovery.
May 28, 19:01 UTC
May 28, 19:01 UTC
May 27, 2020
No incidents reported.
May 26, 2020
No incidents reported.
May 25, 2020
No incidents reported.
May 24, 2020
No incidents reported.
May 23, 2020
No incidents reported.
May 22, 2020
No incidents reported.