We rolled a certificate update that allowed the old agents to use an alternate trust chain to connect to Scalyr. Most traffic is back. We are calling this issue resolved for now. If you are still seeing connect errors to Scalyr endpoints, please reach out to Scalyr support via email support at scalyr dot com.
We are working on adding monitoring for root and intermediate cert expiration among other things as learnings from this incident.
Jun 2, 04:33 UTC
We have released release Scalyr Agent version 2.1.5 which addresses an issue with Windows. Windows users must apply this update.
May 30, 20:48 UTC
We are working on release 2.1.5 to address an issue with Windows, and expect to release that within approximately an hour. Windows users must apply this update once it is available.
May 30, 19:51 UTC
We ship CA bundles along with the scalyr agent and in some specific installations, we rely on using the agent CA bundle to validate the cert as opposed to system shipped CA bundles. We believe customers experiencing the connection issues are in this category (using scalyr agent CA bundles in /usr/share/scalyr-agent-2/certs/ca_certs.crt).
If you are still experiencing connection errors from the agent, run following commands to update expired CA bundle shipped with the agent:https://gist.github.com/lakshmi-kannan/1db93ede48745b1397acef40867fa2f5
We are working on rolling out a new release of scalyr agent with updated CA certs. It should be available shortly.
May 30, 16:01 UTC
We identified expired cert in our cert chain and replaced them with new ones. The impact is on a subset of customers. We are further investigating the partial nature of the impact.
May 30, 14:43 UTC