: our full report
on this incident is now available (in German)
No, the Internet is not breaking down, we did not have a doomsday scenario over the last week.
We did have an interesting situation, there were some disruption in some parts of the Internet, and there were a good number of overtime hours being put in to mitigate these disruptions.
Here are some links:
There is some discussion in international network security forums on what lessons should be taken from those events. This is all pretty preliminary, but boils down to the following:Control Plane Protection
After the attackers failed to bring down the anycasted webservers of CloudFlare (that part is described at the CloudFlare blog), they aimed at the network equipment supporting those nodes.
For one view on how to protect your own routers from attacks, see https://www.box.com/s/osk4po8ietn1zrjjmn8b
There are also some lessons for the operators of Internet Exchange Points. Network Hygiene
The attacks were mainly done using amplification via DNS servers. In order to reduce these attacks, the following three points need to be taken care of:
Forgery tracking capability
- Implement BCP38. The attackers need to send out forged packets;
restricting their ability to do so from a hacked box inside *your* network
- Recursive nameservers should not be open to the word. See
RFC5358. There are a few projects starting up
which scan the Internet for such open recursors in order to get them all fixed. One is http://openresolverproject.org.
Warning: the data-quality from that service is not optimal yet.If you want to scan your own netblock for open recursors, have a look at Aaron's software.
- Authoritative nameservers can also be abused as traffic amplifiers.
There are patches out which implement rate-limiting for the common
implementations. See e.g. http://www.redbarn.org/dns/ratelimits
In the case when someone is abusing ISP customers as amplificators to attack a.b.c.d, then he is forging the packets claiming to originate from a.b.c.d. As we need to track down the forgers, it is necessary that all networks are capable of investigating where these forgeries originate from
(inside your own network, from a peering, or from your upstream).
This needs to be a fast process.
It does not matter how this is achieved; the most obvious tool to do this is a decent netflow coverage of your network. For smaller networks, this can be done with open source tools like nfsen
Author: Otmar Lendl