Rackspace hosted Exchange suffered a devastating failure starting December 2, 2022 and is still ongoing since 12:37 AM December fourth. Initially described as connection and login issues, the assistance was eventually updated to announce that they were handling a security incident.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the early morning hours of December 2, 2022. At first there was no word from Rackspace about what the problem was, much less an ETA of when it would be solved.
Consumers on Buy Twitter Verification reported that Rackspace was not reacting to support e-mails.
This has been quite the day with #Rackspace. Every hosted exchange client has been down for 14 hours approximately. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they succumbed to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer independently messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the past 16 hours.
Unsure the number of companies that is, however it’s considerable.
They’re serving a 554 long delay bounce so individuals emailing in aren’t knowledgeable about the bounce for several hours.”
The main Rackspace status page provided a running update of the interruption however the initial posts had no info aside from there was an outage and it was being examined.
The first official upgrade was on December 2nd at 2:49 AM:
“We are investigating a concern that is impacting our Hosted Exchange environments. More information will be posted as they become available.”
Thirteen minutes later Rackspace began calling it a “connection issue.”
“We are examining reports of connectivity problems to our Exchange environments.
Users may experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their e-mail client(s).”
By 6:36 AM the Rackspace updates described the ongoing issue as “connection and login issues” then later on that afternoon at 1:54 PM Rackspace announced they were still in the “investigation stage” of the blackout, still trying to find out what went wrong.
And they were still calling it “connection and login problems” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
4 hours later on Rackspace described the scenario as a “substantial failure”and began using their consumers totally free Microsoft Exchange Strategy 1 licenses on Microsoft 365 as a workaround until they comprehended the issue and might bring the system back online.
The official assistance mentioned:
“We experienced a considerable failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any more issues while we continue work to restore service. As we continue to overcome the root cause of the issue, we have an alternate option that will re-activate your capability to send out and receive e-mails.
At no cost to you, we will be supplying you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 till further notification.”
Rackspace Hosted Exchange Security Occurrence
It was not until nearly 24 hr later on at 1:57 AM on December 3rd that Rackspace officially announced that their hosted Exchange service was struggling with a security incident.
The statement further revealed that the Rackspace technicians had actually powered down and disconnected the Exchange environment.
“After further analysis, we have actually figured out that this is a security occurrence.
The recognized impact is separated to a part of our Hosted Exchange platform. We are taking necessary actions to assess and safeguard our environments.”
Twelve hours later that afternoon they upgraded the status page with more info that their security team and outside experts were still working on resolving the blackout.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has not launched information of the security event.
A security occasion generally includes a vulnerability and there are 2 severe vulnerabilities presently in the wile that were patched in November 2022.
These are the two most present vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to read and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an enemy has the ability to run destructive code on a server.
An advisory released in October 2022 explained the impact of the vulnerabilities:
“A confirmed remote assailant can carry out SSRF attacks to intensify advantages and execute arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mail box server, the assailant can possibly gain access to other resources by means of lateral movement into Exchange and Active Directory site environments.”
The Rackspace blackout updates have not shown what the particular problem was, just that it was a security event.
The most existing status upgrade as of December 4th stated that the service is still down and consumers are motivated to move to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in addressing the event. The schedule of your service and security of your data is of high significance.
We have actually devoted extensive internal resources and engaged world-class external proficiency in our efforts to reduce negative effects to clients.”
It’s possible that the above kept in mind vulnerabilities are related to the security incident affecting the Rackspace Hosted Exchange service.
There has been no announcement of whether customer details has been jeopardized. This occasion is still continuous.
Included image by Best SMM Panel/Orn Rin