We rolled a certificate update that allowed the old agents to use an alternate trust chain to connect to Scalyr. Most traffic is back. We are calling this issue resolved for now. If you are still seeing connect errors to Scalyr endpoints, please reach out to Scalyr support via email support at scalyr dot com.
We are working on adding monitoring for root and intermediate cert expiration among other things as learnings from this incident.
Posted Jun 02, 2020 - 04:33 UTC
Monitoring
We have released release Scalyr Agent version 2.1.5 which addresses an issue with Windows. Windows users must apply this update.
Posted May 30, 2020 - 20:48 UTC
Update
We are working on release 2.1.5 to address an issue with Windows, and expect to release that within approximately an hour. Windows users must apply this update once it is available.
Posted May 30, 2020 - 19:51 UTC
Identified
We just rolled out a new Scalyr agent release (2.1.4) with new CA certs bundled with the agent. Customers are highly encouraged to upgrade to this new version of Scalyr agent (2.1.4) if they are seeing connection errors from the agent.
We ship CA bundles along with the scalyr agent and in some specific installations, we rely on using the agent CA bundle to validate the cert as opposed to system shipped CA bundles. We believe customers experiencing the connection issues are in this category (using scalyr agent CA bundles in /usr/share/scalyr-agent-2/certs/ca_certs.crt).
If you are still experiencing connection errors from the agent, run following commands to update expired CA bundle shipped with the agent:
We are working on rolling out a new release of scalyr agent with updated CA certs. It should be available shortly.
Posted May 30, 2020 - 16:01 UTC
Update
We identified expired cert in our cert chain and replaced them with new ones. The impact is on a subset of customers. We are further investigating the partial nature of the impact.