Rackspace hosted Exchange suffered a catastrophic failure beginning December 2, 2022 and is still ongoing since 12:37 AM December 4th. At first referred to as connectivity and login problems, the assistance was ultimately updated to announce that they were dealing with a security incident.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be resolved.
Clients on Buy Twitter Verified reported that Rackspace was not responding to support e-mails.
This has been quite the day with #Rackspace. Every hosted exchange client has been down for 14 hours or two. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they came down with something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer privately messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Uncertain the number of business that is, but it’s considerable.
They’re serving a 554 long hold-up bounce so people emailing in aren’t familiar with the bounce for several hours.”
The official Rackspace status page used a running upgrade of the interruption however the initial posts had no info besides there was a blackout and it was being examined.
The first authorities update was on December 2nd at 2:49 AM:
“We are investigating a concern that is affecting our Hosted Exchange environments. More information will be published as they become available.”
Thirteen minutes later on Rackspace started calling it a “connection problem.”
“We are examining reports of connectivity concerns to our Exchange environments.
Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their e-mail client(s).”
By 6:36 AM the Rackspace updates described the ongoing issue as “connection and login concerns” then later on that afternoon at 1:54 PM Rackspace announced they were still in the “examination stage” of the outage, still trying to find out what went wrong.
And they were still calling it “connection and login concerns” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
Four hours later on Rackspace described the situation as a “substantial failure”and started using their clients complimentary Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the problem and could bring the system back online.
The main guidance mentioned:
“We experienced a significant failure in our Hosted Exchange environment. We proactively closed down the environment to prevent any further concerns while we continue work to restore service. As we continue to overcome the source of the issue, we have an alternate option that will re-activate your capability to send out and get e-mails.
At no charge to you, we will be offering you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 till more notice.”
Rackspace Hosted Exchange Security Occurrence
It was not till nearly 24 hours later at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was experiencing a security event.
The announcement further revealed that the Rackspace technicians had powered down and disconnected the Exchange environment.
“After further analysis, we have actually identified that this is a security incident.
The known effect is separated to a portion of our Hosted Exchange platform. We are taking needed actions to examine and protect our environments.”
Twelve hours later that afternoon they updated the status page with more details that their security team and outdoors specialists were still working on resolving the interruption.
Was Rackspace Service Affected by a Vulnerability?
Rackspace has not released information of the security occasion.
A security event normally includes a vulnerability and there are 2 serious vulnerabilities presently in the wile that were covered in November 2022.
These are the two most present vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to check out and change information on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an aggressor is able to run destructive code on a server.
An advisory released in October 2022 explained the effect of the vulnerabilities:
“A validated remote enemy can perform SSRF attacks to escalate advantages and carry out arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the aggressor can possibly gain access to other resources by means of lateral motion into Exchange and Active Directory environments.”
The Rackspace interruption updates have actually not shown what the specific issue was, just that it was a security incident.
The most present status update since December fourth specified that the service is still down and clients are motivated to migrate to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in resolving the occurrence. The schedule of your service and security of your data is of high value.
We have actually dedicated comprehensive internal resources and engaged world-class external proficiency in our efforts to minimize unfavorable effects to consumers.”
It’s possible that the above noted vulnerabilities belong to the security incident affecting the Rackspace Hosted Exchange service.
There has been no statement of whether client details has been jeopardized. This occasion is still ongoing.
Included image by Best SMM Panel/Orn Rin