I was going through my Cyclops alerts yesterday when I noticed a  “More specific prefix” alert for an UCLA prefix I had configured. Specifically, I noticed AS30890 was announcing the prefix 131.179.244.122/32 which is usually announced as part 131.179.0.0/16 by UCLA (AS52). I further noticed this route was being propagated by several ASes upstream. AS30890 is a large ISP in Romania, with a degree of 358 and announcing on the order of 445 prefixes in a “normal day”. However, today was definetly not a “normal day” for AS30890,  and Cyclops was seeing on the order of 1345 /32 prefixes being announced and withdrawn in a time interval of less than 2 minutes, starting at arround 7:30 UTC on 2009-02-11.

BGP announcement from AS30890.

The announced prefixes were being propagated by several routers in Europe. Cyclops gave me a pretty good picture of what was happening through the “Prefix” menu:

http://cyclops.cs.ucla.edu/?vm=0&di=2009-02-12&de=2009-02-13&asn=30890&prfx=&loc=&ag=0&v=prfx&sub=1&p=91&s=0&o=8&d=1

At this point I could only do guess work about the causes of this incident. I know sometimes networks announce /32 to their providers to black-hole traffic to specific hosts under DoS attack, a feature called Remote Triggered Black Hole (RTBH), initially defined in RFC3882. This feature allows a network to send /32 routes upstream with a bogus next-hop so that traffic to specific hosts gets blocked before even reaching the destination network. Of course that this will “complete” the attack and block traffic to the specific end host. However, the advantage is that there will be more room for the other flows in the aggregate. A different technique was proposed more recently that blocks traffic based on source address using Unicast Reverse Path Forwarding (URPF) - check this I-draft. Basically a router will filter any traffic that does not match the interface where the source would be routed to.

I decided to contact directly Evolva Telecom to verify what went wrong. The folks at Evolva were very responsive and helpful, and provided all the details about what happened:

1. They started receiving traffic in one of their DE-CIX interfaces with destinations not announced by AS30890; apparently some other peer was forcing default routes to AS30890; traffic to these addresses was heavy, on the order of 2M packets/second
2. They start sending destination-based /32 filters to all their peers that were sending them traffic; and because these announcements did not have the NO_EXPORT community, they were further propagated for several hops and were caught by Cyclops; however, since they did not want to do destination based filtering on prefixes they were not announcing, they immediately withdraw these prefixes
3. They fired URPF in the router, which filtered traffic by the source addresses of the DDOS attack (probably fake addresses). Since these addresses had a void next-hop in the router,  traffic from these addresses start being dropped, and the attack was blocked
4. They changed the chain policy in all outgoing route-maps for /32s from
   accept to deny and stopped advertising any kind of /32 prefix

Several points can be taken from this incident. First, it was surprising how the /32 routes were propagated these many hops upstream, which I believe was not intended, and shows that there are still lots of networks out there that do not filter on prefix length. Second, additional care is needed inside IXPs, mostly because networks share a layer-2 cloud, and additional filtering is needed to make sure other peers are not default routing to your network. Here’re some suggestions on how to do that:

• Netflow analysis
• MAC accounting.  Some Internet exchanges will provide a
  traffic matrix which may indicate when this behavior is occuring. In
  some cases the operator may even notify you about this and “enforce” the
  policy when someone is attempting to route this way
• QPPB/DCU capabilities in Cisco/Juniper

The /32 routes displayed by Cyclops correspond actually to addresses that were being the target of a DDOS attack. Evolva Telecom is still investigating the origin of the attack, and I’ll follow this post as soon as I have more information.

–Ricardo