2013/03/28

Update: our full report on this incident is now available (in German)

No, the Internet is not breaking down, we did not have a doomsday scenario over the last week.

We did have an interesting situation, there were some disruption in some parts of the Internet, and there were a good number of overtime hours being put in to mitigate these disruptions.

Here are some links:

• http://www.bbc.co.uk/news/technology-21954636
• https://greenhost.nl/2013/03/21/spam-not-spam-tracking-hijacked-spamhaus-ip/
• https://www.nytimes.com/2013/03/27/technology/internet/online-dispute-becomes-internet-snarling-attack.html?pagewanted=all
• http://rt.com/news/spamhaus-threat-cyberbunker-ddos-attack-956/
• http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet
• http://gizmodo.com/5992652
• http://cluepon.net/ras/gizmodo

There is some discussion in international network security forums on what lessons should be taken from those events. This is all pretty preliminary, but boils down to the following:

Control Plane Protection

After the attackers failed to bring down the anycasted webservers of CloudFlare (that part is described at the CloudFlare blog), they aimed at the network equipment supporting those nodes.

For one view on how to protect your own routers from attacks, see https://www.box.com/s/osk4po8ietn1zrjjmn8b

There are also some lessons for the operators of Internet Exchange Points.

Network Hygiene

The attacks were mainly done using amplification via DNS servers. In order to reduce these attacks, the following three points need to be taken care of:

1. Implement BCP38. The attackers need to send out forged packets; restricting their ability to do so from a hacked box inside *your* network helps.
2. Recursive nameservers should not be open to the word. See RFC5358. There are a few projects starting up which scan the Internet for such open recursors in order to get them all fixed. One is http://openresolverproject.org. Warning: the data-quality from that service is not optimal yet.

   If you want to scan your own netblock for open recursors, have a look at Aaron's software.

3. Authoritative nameservers can also be abused as traffic amplifiers. There are patches out which implement rate-limiting for the common implementations. See e.g. http://www.redbarn.org/dns/ratelimits

Forgery tracking capability

In the case when someone is abusing ISP customers as amplificators to attack a.b.c.d, then he is forging the packets claiming to originate from a.b.c.d. As we need to track down the forgers, it is necessary that all networks are capable of investigating where these forgeries originate from (inside your own network, from a peering, or from your upstream).

This needs to be a fast process.

It does not matter how this is achieved; the most obvious tool to do this is a decent netflow coverage of your network. For smaller networks, this can be done with open source tools like nfsen.

Author: Otmar Lendl

2013/03/19

Dear like-minded people,

I'm very proud to announce that our latest contribution to the malware analysis community is finally available as open beta.

It's called ProcDOT - I already gave a preview of the alpha version some months ago at SANS Forensics Summit in Prague - and it is an absolute must have tool for everyone's lab, at least in my humble opinion ;-)

It correlates Procmon logfiles and PCAPs to an interactively investigateable graph. Besides that ProcDOT is now also capable of animating the whole infection evolution based on a timeline of activities. This feature lets you even quickly find out which server or which requests were responsible that specific data/code got on the underlying system, by which process it was written, how often, who injected what, which autostart registry key was set, what happened when, and so forth ...

ProcDOT's approach of correlating Procmon logs and PCAPs to a directed animateable graph has the potential to reduce one's efforts to behavioral analyze a malicious situation to an absolute minimum. => Find out if there's something malicious going on under the hood with one quick glance. => Find out what it does in minutes.

To let users get the most out of ProcDOT I took the effort to create 50 minutes of tutorial videos which show the use of ProcDOT in a case of a typical drive-by-infection.

Get it for free from our website: http://www.cert.at/downloads/software/procdot_en.html ... and be sure to follow the included readme file!

Still there are so many things that I have planned to add, but from my experience regardless how many of them will be implemented this situation won't change ;-)

As always, feedback - good and bad - is very much appreciated.

Cheers,

Christian

PS: There's an upcoming (April) ProcDOT presentation at First Symposium in Amsterdam. See you there, maybe.

Author: Christian Wojner

2012/11/29

This blog post evolved over time - initially it was a mere scratchpad for notes during our initial research between 2012/11/29 and 11/30. Later, after Syria was back online again, I added a summary and some potential explanations of what might have happened at the end of this blog post.

 

The blackout

2012/11/29: as Renesys and others pointed out, Syria seems to be offline since 10:26 UTC. Michael Kafka and me verified this fact via looking at BGP routing tables and looked at the potential damage to the Internet as a global system. Here is what we can confirm / figured out:

AS29386 is the Syrian Telecom (STE). It seems to be the central hub for all connections to/from Syria. So, we started first with this network.

  This is how STE is connected:

Next, we looked at Renesys' claim that 92% of all announcements were offline. However, the situation seems to be worse: at the time of this writing we did not find a single BGP prefix which was announced via STE AS29386. There still might be some other carrier which announces Syrian IP spaces but none of them went through Syria Telecom.

 

Next, we looked at the list of known netblocks which are announced via AS29386 based on RIPE's stat.ripe.net service: https://stat.ripe.net/as29386#tabId=routing. This list was compared against our BGP table. We indeed could confirm that none of the networks were visible.

 

UPDATE 00:45 a.m.: Cloudflare has a good analysis of the Internet blackout. Summary: a fiber cable cut seems unlikely according to cloudflare.

UPDATE 2012/11/30, 11:00 a.m.: The last night was busy with figuring out some details. I could independently confirm that even the IP range, which was active in the Finfisher malware was not reachable anymore. This is interesting since this IP range was actually attributed to the Syrian government. So, even they are definitely offline. This is strange. If I were the Syrian government and I wanted to block rebels from communicating via the Internet, I wouldn't turn off my own connectivity as well. What happened?

 

UPDATE: Renesys has an updated Analysis. There seems to be some traffic flowing out, but via other carriers. We could not confirm this. However what we could find out was...

How does it look from inside of Syria?

I received a traceroute from within Syria via a friend of a friend of a friend. So, please be aware of this and treat it merely as a hint. However, to me it looks real. The first IP addresses are intentionally obfuscated to protect the individual who made the traceroute. It is interesting to see that the traceroute ends at:

% Information related to '82.137.192.0 - 82.137.199.255'
inetnum: 82.137.192.0 - 82.137.199.255
netname: SY-STE-PDN-NETWORK
descr: STE PDN Backbone Network
country: SY
admin-c: WS1833-RIPE
tech-c: MS4418-RIPE
tech-c: WS1833-RIPE
status: ASSIGNED PA
mnt-by: STEMNT-1
mnt-lower: STEMNT-1
remarks: -----------------------------------------------------------
remarks: INFRA-AW
remarks: ANY Problems Like SPAM, Hacking etc..
remarks: Send Email to wmsalem@gmail.com
remarks: -----------------------------------------------------------
source: RIPE # Filtered
person: Mostafa Sawan
address: Syria
phone: +963 21 5229803
fax-no: +963 21 52288992
nic-hdl: MS4418-RIPE
mnt-by: STEMNT-1
source: RIPE # Filtered
person: Weam Salem
address: PDN- STE
phone: +963-11-4461475
fax-no: +963-11-4466892
nic-hdl: WS1833-RIPE
mnt-by: WS-MNT
source: RIPE # Filtered
% Information related to '82.137.192.0/18AS29256'
route: 82.137.192.0/18
descr: STE Public Data Network Backbone and LIR
origin: AS29256
mnt-by: STEMNT-1
source: RIPE # Filtered
% Information related to '82.137.192.0/18AS29386'
route: 82.137.192.0/18
descr: STE Public Data Network Backbone and LIR
origin: AS29386
mnt-by: STEMNT-1
source: RIPE # Filtered

 

In other words, we are observing the path that data pakets would take from within Syria to google (8.8.8.8). However, the network where everything stops is STE (precisely the STE PDN Backbone Network). It will be very interesting to compare this traceroute later when the internet connectivity is restored again. Then we will be able to see exactly at which point data pakets were dropped.

Here is a screenshot of the traceroute:  

Why is this traceroute so special? It was smuggled out via a VSAT connection!! So this is to the best of my knowledge a unique view of the Internet blockage as seen from inside Syria at the time of when it happened. Indeed, there seem to be many VSATs in Syria. The New York times confirmed.

According to the same sources, regular landline phones still work in Damaskus. Also, accessing *.SY domains / servers from within Syria seems to still work (at least yesterday night). It's just the international wired links which are down. Somehow this proves my point that I made in an interview with "DerStandard". Still, VSATs and modems are slow - but at least some folks still had some connectivity.

Speaking of DNS:

$ dig -t ns sy
; <<>> DiG 9.7.6-P1 <<>> -t ns sy
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26318
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;sy. IN NS
;; ANSWER SECTION:
sy. 86400 IN NS ns1.tld.sy.
sy. 86400 IN NS ns2.tld.sy.
sy. 86400 IN NS pch.anycast.tld.sy.
sy. 86400 IN NS sy.cctld.authdns.ripe.net.
;; Query time: 18 msec
;; SERVER: x x x x #53(x x x x )
;; WHEN: Fri Nov 30 12:49:46 2012
;; MSG SIZE rcvd: 125

This means that ripe.net is still a secondary authoritative DNS server for *.sy domains. You can still query *.sy domains from there in case you know what you are searching for. This fits together with claims that *.sy domains which are hosted in the US are still reachable.

 

Physical connections

Syrian sea cables (source: Mikko Hipponen):

In addition there seem to be land cables to turkey - again according to Mikko.

UPDATE 2012/11/30 14:45: Otmar and me cross checked the BGP announcements via a different mechanism. Since our initial analysis only included any network which was announced via STE, we thought we should better double check again and look if we had forgotten any Syrian network which was not going through STE. Therefore, we took a different approach: we extracted the syrian IP ranges from the maxmind DB, fetched the ASNs, compared against the full BGP feed and found that our last hope, a traceroute path to Syria via the indian carrier TATA terminates in Paris. So even though we chose a different path for our analysis, we end up with the same result: Syria is indeed offline.

UPDATE 2012/12/2: it seems that the land line via Turkey mentioned above was planned but was not in operation during the Internet blackout. This makes it very hard to attribute the cause of the blackout to either the rebels or the government. As Renesys said in their updated blog:

The restoration was achieved just as quickly and neatly as the outage: like a switch being thrown. Does that mean that we believe the government (or the opposition) threw the switch? Frankly, the data available just don't support attribution at this point, despite all the speculation.

Where do the sea cables come in? Close to Tartous (Tartus) (and by the way, Tartus hosts the russian naval base).

Greg's cable map site has a nice list of lines arriving at Tartus:

1. BERYTAR
2. ALETAR
3. UGARIT

 

If there was some physical damage to these lines or the landing site, then most probably the carrier operators should know about that. Did someone ask them already what happened? If they don't know of any damage, then something must have happened on the software level or further inland.

Telecomix posted pictures of the Tarassul data center (note the Blue-coat (and the power cord that you could trip over ;-) ): . Again, just to be sure - this is some info from the Internet. No idea how much solid evidence this picture is.

 

Post-fact analysis - what actually happened?

Hard to tell. Personally I still believe it was caused by a "network maintenance" event. That is - someone got a phone call and had to disable BGP announcements. Or there was some upgrade to DPI systems. However, the fact that all networks were gone (including the government networks and the network range which was hosting the famous FinFisher command & control server) speaks exactly against this theory. If we apply reasoning (which might be the wrong thing in a war(?)), then we should assume that these government networks should have been up and running. But they were not as we could independently confirm. Also there are multiple sea cables going into Syria. But they seem to come together at one point in Tartus. A single point of failure.

 

Lessons learned

In summary I can only conclude that we don't know for sure what really happened, but we know for sure that it is really a bad idea to have a single point of failure.

More specifically, I believe we need:

physical redundancy

It makes sense to have many fiber lines. If we were to believe the official story that rebels blew up a communications cable, then redundant connections would have avoided any blackout.

organizational redundancy

Having one organization and one technician administering vital systems which are a single point of failure is a bad idea. Humans make errors. Or they can get bribed. Who knows what really happened in Syria? Maybe the technician had to upgrade the DPI system and the upgrade did not work? Having multiple organizations / ISPs minimizes these risks.

  Nevertheless...

they have guns!

All of the above is of course only a recommendation which works in a democratic nation at peace. If there are multiple SWAT teams with guns entering multiple ISPs' office at the same time and ordering a nationwide shutdown, there is little you can do but to shut down everything.

 

Nevertheless, I am very happy that Austria at least is well connected with multiple redundant links and with multiple ISPs. But even here we should learn the lesson from Syria: build redundancy! It's important! Srsly.

 

On a lighter note, I was wondering why I personally got so excited about the subject. Well, I guess xkcd answered it for me:

(source: http://xkcd.com/705/)

Author: L. Aaron Kaplan

2012/09/20

Microsoft's Security Intelligence Report 12 uses the computers cleaned per mille (CCM) metric to compare the infection rates over time and between countries.

This is, of course, no perfect measurement of the actual infection rates due to a number of factors, but nevertheless an interesting data-point.

Austria usually sports a quite low CCM score, but during Q4 2011 something strange happened: there was a clear upwards spike. So we wondered what happened to Austria (as well as to some of our neighbors):

There seem to be various factors playing together:

• The online banking gangs (using SpyEye, ZeuS, Ice-IX, Citadel, Torpig, ...) seem to be focussing on specific banks (and countries) at each point in time. Once they have the web-injects and the money-mules lined up, they use shady services to buy either installs or web-traffic on a by-country basis. Given the effectiveness of exploit-packs, web-traffic can be easily be turned into zombies.
• MSRT added detection for SpyEye in October 2011, causing spikes in CCM in those countries that experienced active SpyEye campaigns at that time.
• SpyEye has been supplanted by other banking-malware, MSRT might not be covering all of them.

We have thus on one side the criminal gangs which are changing both their targets and the malware they use, and on the other side Microsoft, which is also adapting their MSRT. If these factors intersect in the right way, the CCM for a country spikes.

(Mike Sandee from Fox-IT provided input towards this analysis.)

Author: Otmar Lendl