Distributed-Denial-of-Service attacks on mailbox.org

There has been a steep increase in Distributed-Denial-of-Service (DDoS) attacks against e-mail providers worldwide in the past few weeks, and more recently these attacks have become even more frequent. Attacks like this are essentially a form of blackmail, as they have the objective of bringing services to a standstill first to then offer relief in exchange for money. As it happened, mailbox.org was also targeted, on Thursday night and Friday afternoon. At the same time, we received a communication in which the attackers demanded we pay them money in Bitcoin. - Of course, we will do no such thing.

We are expecting more attacks in the coming days, though. This blog article has information for our users, including some more background about DDoS attacks and their possible impact. We hope any disruption can be avoided or kept to a minimum. Thank you in advance for your understanding should any services become temporarily unavailable during the next few days.

Please note:

• Should any services become temporarily unavailable, for example if you are unable to retrieve your e-mails as usual, please try again later.
• Within the next few days, try not to issue any unnecessary support requests concerning, or notify us of any temporary downtime of services. We are expecting more attacks and will do everything required to restore services as quickly as possible.
• Any e-mails sent to you won't be lost, even if the service is temporarily unavailable. In the current circumstances, these may just be delivered a little later than usual.
• If you are experiencing any difficulties with using our services, please let us know on Twitter, if possible. Twitter is also a good place to look first in case something isn't working.

Background:

DDoS attacks aim to knock out their targets

A Distributed-Denial-of-Service (DDoS) attack requires a large number of computers to get infected by viruses first. Then, the attackers control these machines to send large quantities of data to a shared target in a coordinated fashion, with the goal of disabling the target's ability to respond to other requests. What we experienced was quite extraordinary: About 147.000 different computers had been infiltrated to be involved in the attack, and our servers received data through the Internet at a rate of 150 million data packets per second. As an analogy, imagine 147.000 people dialling the same phone number, every second, in order to knock out the telephone network of a city, a business, or a local authority.

The data that is sent in DDoS attacks has usually normal or even meaningless contents, because the primary goal is to impact the function of systems, firewalls, servers and networks through the huge amounts of data that are being sent. For example, doing so can overload a server, such that it is kept busy processing the DDoS data, while regular user requests suffer severe delays or even time-outs. It is important to point out, though, that at no point do the attackers actually have access to the targeted network. That is why we are confident that all customer data has always been secure and protected during the attacks.

For any internet host, it is not really possible to avoid DDoS attacks when they come. However, it is possible to create a resilient infrastructure that provides protection and a level of performance which is more difficult for attackers to knock out. However, the idea that someone could achieve full protection is an illusion: The opponent that fields more capable hardware and software will win eventually. On the attacking side, these are the number of systems that are coordinated to produce and send the data, and on the defending side, it's the systems that receive and process the data as well as those that detect and counter such attacks.

mailbox.org is well-prepared for DDoS attacks

We at mailbox.org have previously monitored occasional smaller-scale DDoS attacks, which usually go unnoticed by customers. We have also worked to prepare against larger efforts that would target our systems directly and invested considerably in a robust infrastructure that provides appropriate defensive measures. For example, we operate our servers redundantly across two separate data centres and have four different communication lines for data exchange with other providers on the Internet. Also, our network infrastructure has been designed to have a considerable amount of resources to spare during normal operations, which provides our systems with the performance to scale rapidly, if required.

Furthermore, we are working together with MyraSecurity, a German company that specializes in protection against DDoS attacks. MyraSecurity maintain a powerful infrastructure for the detection and filtering of DDoS data traffic and their systems provide additional protection by blocking a large amount of the incoming malicious traffic that is caused by DDoS attacks. During regular, everyday operations, these systems just sit there and don't need to do much. However, once an attack has been detected, our mailbox.org systems will switch over their regular data lines to filter the incoming traffic through this “DDoS shield”. As a result, less traffic reaches our critical infrastructure, which minimizes the potential of disruption.

It is important to note that MyraSecurity has been set up such that they have no actual access to the encrypted data traffic. Any DDoS protection measures will only look at external characteristics of the data packets they filter, to decide which ones will be blocked and which ones will be let through.

The first attack on Thursday

The difference between theory and practice: You will only know if your preparations were sufficient once the worst is actually happening. Last Thursday, we had a few teething problems at first but were able to defend ourselves effectively against the first large DDoS attack. At the beginning, we saw some disruption to services. However, we were able to find and disable the root causes quickly and adapt our infrastructure to better withstand any future attacks.

After about 30 to 60 minutes had passed, hardly any impact was noticeable anymore by our users despite the attack continuing at full strength. Some of our websites may have responded less quickly than usual but we were able to fix this as well by changing some configuration options. At 10pm it was all over – the attack stopped, after services had been running close to normal for some time already. All in all, we were reasonably happy with how things went down that night and considered our systems better prepared for whatever might be coming next.

The second attack on Friday

The team spent some time on Friday (22 October) to analyse what had happened the previous day, and we decided to put additional measures in place to further improve resilience. Yet, another attack came at 4pm, at the very second when we temporarily disabled some of our new firewall improvements to adjust some of the settings. That's hard luck! We were able to counter the attack within minutes just by enabling our firewall settings. However, due to a brief but large spike in traffic, one of our internal load balancers had crashed and for some reason, the automatic takeover by the replacement unit didn't work as smoothly as we had hoped. We needed a technician to actually go to the data centre and fix the problem on location, and for a duration of about 60 minutes there were some problems with logging in and accessing services through IPv6. We have now put further measures in place that will prevent a similar situation from arising in the future and also improved our ability to react more quickly.

Outlook

Attacks against e-mail providers that have the intent to extort money will likely continue in the future. We at mailbox.org expect more attacks to follow in the coming days, and it is the same for our colleagues who are with other organizations that are facing the same DDoS attack wave. As we are writing this blog, we are also extensively networking and communicating with other providers worldwide, to exchange experiences and best practices about how to deal effectively with these kinds of attacks.

It's hard to say what will happen in the future. There may well be that one attack that is so massive that it might overwhelm our infrastructure, but it's also possible that we will remain as resilient as we have so far, and that future DDoS attacks will continue to fail to have any noticeable impact on our operations.

What we can say is that we are prepared. With our knowledge and experience, we did our best to create good defensive measures. Yet, every network and every data line has physical limits, and that is something we cannot change. However, we know that it is also expensive for the attackers to actually run a coordinated DDoS operation that requires many resources. That means they have limits, too, and cannot push on forever.

We ask for your understanding

In light of the situation, we ask our users for their patience and understanding should they experience any disruption to services within the next few days. We will be doing our best to deflect any incoming attacks as quickly as possible.

We are asking in particular that general inquiries are kept to a minimum at the moment. Please accept our apologies that while an attack is ongoing, we may not be able to communicate with you as fully, speedily, and individually as you may have come to expect from us under normal circumstances. There is a chance that blog articles, user forum, and disruption banners may be affected as well. If things get sticky, watch out for the announcements and status updates the team will post on our Twitter channel.

Lastly, everyone can rest assured that under no circumstances will mailbox.org be blackmailed or submit to paying any money to any attacker.

---
Our admin and network teams at Heinlein Hosting and mailbox.org would like to take the opportunity to also thank our colleagues at MyraSecurity for their excellent support and friendly cooperation.