Every once in a while, security vulnerabilities are discovered on the most secure platforms, and when that happens, the IT community worldwide becomes abuzz with information, warnings, and security advice within hours. This is the time when vulnerabilities - small cracks in the thick mantle of security software protecting the integrity of a cloud server - are called zero-day vulnerabilities, indicating the time period in which the crack has been found, but a fix is still pending.

Luckily, this fix is usually available very fast. Most zero-day vulnerabilities have a patch available by the time the general excitement starts. Patches generally are full, definite fixes of the problem, they close the "crack" and prevent any future exploitation.

 

Acting fast is important

The recent GHOST and Heartbleed vulnerabilities were an example of this. The problem is, once a risk has been disclosed to the general public, so will explanations on how to exploit it. Within a very short time, tools are available for download that can easily be morphed into automatic hack-bots, scanning the Internet for unpatched servers.

Thus, the real exposure is in these hours right after the announcement. During this time, system administrators, hosting providers, devops and client service managers need to work together to patch systems as quickly as possible.

How the WebPal Cloud is protected

Of course, as with many providers, our team has a standard procedure for these cases. When GHOST was discovered, we followed our standard risk mitigation checklist:

1. Are we affected at all?
   Check whether in fact WebPal Servers were affected and if so, which ones. In the case of Heartbleed for example, our strategy of late adoption of OS version paid off and we were not affected at all.
2. Is a patch available and applied?
   WebPal Servers automatically update nightly with latest security patches provided by our OS distributors. Nonetheless, we make sure that the latest patch (if available) is applied to all servers, in all data centres. If a patch is not available yet, we weigh the benefits of disabling affected software on the servers. 
3. Has any data been compromised already?
   All attacks leave traces. Depending on the vulnerability, there are patterns in network traffic, process history and server behavior that would let us ascertain whether a server has already been attacked or, worse, compromised. A thorough scan is performed to ensure that this is not the case.
4. Downtime required and when?
   Some security patches are to subsystems so widely used in a cloud server such that a full reboot is recommended to ensure that no un-patched code is still running on the server. GHOST was an example of this. As system restarts are very disruptive, we generally schedule these as rolling overnight reboots.
5. Have actions been communicated?
   At all times, we communicate with our clients over our blog, twitter and email alerts to disclose risks and notify of any anticipated downtimes.

We don’t cry wolf

We appreciate that WebPal clients generally are business-level users and care not so much about the technical details of the vulnerability, but for the disruptions the fix causes, versus the potential risk of complacency.  WebPal clients rely on us to fend off serious threats, but forcing unnecessary reboots for every security update (there are dozens a month) would merely be a method of offloading risk management onto them and thus not provide any added value. Our advantage is that we know the business use of WebPal Cloud Servers very well, and with this knowledge, can make risk mitigation decisions that are in line with our client's needs.

 

References

https://access.redhat.com/articles/1332213

http://arstechnica.com/security/2015/01/highly-critical-ghost-allowing-code-execution-affects-most-linux-systems/