What is Rackspace?
Rackspace Technologies is a tech company that provides cloud-based servers, data storage and data backup services.
What Happened?
On December 2, 2022, at 2:49 a.m. EST, Rackspace posted a message stating that customers that used their hosted exchange email servers did not have email access. The Hosted Exchange services include mailboxes (up to 100GB), Microsoft Outlook, Outlook Web Access, mobile device synchronization, anti-spam and anti-virus protection.
On 12/6, Rackspace indicated that they suffered a ransomware attack.
Rackspace has not yet indicated when email service will be restored to their clients. In the meantime, email accounts and domains are being migrated to Microsoft 365. This temporary solution only provides access to new emails. Clients currently have no access to existing emails.
Rackspace has not reported the number of impacted customers. It has been speculated that the number of small and medium sized customers may be in the thousands.
Why is this Important?
In the old days, Microsoft Outlook and Office programs were installed on your company’s server. Email Exchange Servers were also physically located within your company. All emails, email attachments, documents, and spreadsheets were also stored on the server or on your desktop. Today, companies like Rackspace and Microsoft provide these applications with data storage in the cloud.
The Rackspace incident provides a sobering example that cloud applications and cloud stored data are not as safe as you think. Rackspace customers lost the ability to receive and send emails. According to news reports, many customers have email after Rackspace moved them over to Microsoft 365. But there is an ongoing concern of archived email data loss once email service is restored. Think about the impact to your organization and your job tasks if you lost the ability to send and receive emails, plus access to all of your old emails, both sent and received. My guess is that you will come to the same conclusion as me – the impact would be significant if not catastrophic.
Impact?
Loss of email typically means lost revenue. What is your organization’s tolerance to downtime? In other words, how long can you go without email? These are questions that need to be posed to each department. The loss of access to the EHR is the #1 issue, but that can be handled by going old school with paper documentation. The impact on other departments must be reviewed in detail.
Let’s start with the business office. Is there enough cash if billing Medicare, Medicare Advantage, Medicaid and private pay stops or takes longer than normal? What about follow-up of unpaid claims? Referrals? Communication with referring hospitals is typically handled by email. How do you review payor eligibility? How will you recruit staff for open positions without receiving email notifications from recruiting websites? Background checks and review of exclusion lists? The list goes on and on.
All of us are heavily dependent on emails to do our daily tasks. The temporary loss of being able to send or receive emails for a week or two is tolerable, but the tipping point may well be the possible loss of old emails and attachments.
What to do?
I reached out to Scott Wolff, President and Director of IT Operations at LanServ, Inc., a managed service provider (MSP) in St. Louis, and asked him: What do companies need to do to limit their email downtime and prevent the loss of archived (old) emails and attachments? Here is a list of recommendations from Scott W:
Cloud MDR Protection – MDR providers are now offering their 24/7/365 Cyber Security protection that not only monitors your company’s network but also directly ties into Google and Microsoft Cloud offerings. This provides you with a whole team of cyber security experts that will make recommended security changes to your cloud environment, monitor for malicious activity, and potentially stop hackers in the course of taking over your users email accounts to reduce further damage.
Cloud Backups – It is recommended that customers who have cloud email and file storage use their own third party backup service that they control. In the event the cloud provider is attacked, you accidentally delete an email or file, the customer can restore their own data. There are third party providers now offering backup services for customers to use that directly tie into Google and Microsoft Cloud offerings.
Email Archiving – Email archiving services are another good option to make sure every sent and received email is stored securely on a completely different provider network. In the case of the Rackspace incident, customers are being pushed to Office365 email with no historical email data available. Archiving or backing up your email data provides almost immediate access to emails saved to the archive system. Some email archive providers also have the option to allow you to switch to using their email archive service as a live user email access service in the event of a disaster.
Scott W. also suggests asking your cloud provider questions regarding software patching and network segmentation.
Patching servers for vulnerabilities is key – It has been mentioned that some of the RackSpace hosted exchange servers were not all consistently patched to protect them against the “ProxyNotShell” zero day vulnerability that was reported in September of 2022.
While there is no official patch from Microsoft, mitigations have been released. Mitigations provide temporary security protection measures to prevent the vulnerability from being used by hackers. This may not be what the hackers used to gain initial access to the Rackspace systems, but it points to a bigger potential issue: How seriously does your cloud provider take patching vulnerable systems against known zero-day vulnerabilities? And are they consistent in applying them to all systems? You are only as secure as your weakest link.
Network Segmentation – Cloud providers should segment the servers they host into separate internal and isolated networks that separate each server from the others. Should one server be compromised it will not allow hackers to access the other servers, or allow ransomware to be deployed across multiple servers. The goal is to reduce the overall damage if a hacker gets in.
Next Steps – Update your HIPAA Security Risk Assessment and Management Plan
Ransomware attacks are unfortunately inevitable. Shifting applications and data storage to the cloud has been looked at a way to reduce risks. The ongoing Rackspace incident is evidence that loss of email and data is still a possibility. Pull out your security risk analysis (SRA). Use the Rackspace ransomware incident as an opportunity to review existing email security measures with your IT personnel or managed service provider.
Much thanks to Scott Wolff. He has provided strategies to review that will minimize the impact of email disruption.
