Several Cyclops users received “origin change” alerts like the following one:

————————————————–
Alert ID:                     7645041
Alert type:                   origin change
Monitored ASN,prefix:         128.97.0.0/16
Offending attribute:          128.97.0.0/16-9035
Date:                         2009-10-09 07:22:12 UTC
Duration:                     00:00:38 (hh:mm:ss)
No. monitors:                 3
Announced prefix:             128.97.0.0/16
Announced ASPATH:             12637 12637 3269 1267 9035
————————————————–

The alert clearly shows that only 3 monitors saw this alert which hints the event was very local, and when drilling down in the link we can see they reside in Italy:

Also, after inspecting the AS paths, it looks like only italian ISPs were involved. It seems this event was a result of an accidental misconfig of the router similar to the AS13214 also detected by Cyclops: http://cyclops.cs.ucla.edu/blog/?p=78. –Ricardo

For example, if I have “New Prefix” alert condition ON  for AS52, I will receive alerts every time AS52 announces a prefix that is not present in “My Prefixes” list. If I click on “Mark as False Alert” for a “New Prefix” alert, i’m implicitly adding the prefix that triggered the alert to “My Prefixes” list, so that alerts on this prefix will stop. So basically, “Mark as False Alert” feature changes the user configuration to avoid future alerts from being triggered from the same condition, and thus reduce the number of false positives. We are still researching ways of how to deal with false negatives, more to come soon. –Ricardo

————————————————–

Alert ID:                     3492061
Alert type:                   origin change
Monitored ASN,prefix:         192.35.225.0/24
Offending attribute:          192.35.225.0/24-13214
Date:                         2009-05-11 11:03:48 UTC
Duration:                     00:00:01 (hh:mm:ss)
No. monitors:                 1
Announced prefix:             192.35.225.0/24
Announced ASPATH:             48285 13214
————————————————–

As you can see from the text above, only a single monitor detected this incident, and that was a monitor belonging to AS48285 (ROBTEX) that have a BGP session with route-views4. Apparently AS48285 didn’t propagate the routes upstream, only to its other customers The customers started reaching the Internet using a much shorter path, and had their outbound traffic engineering completely disrupted. After looking at some neighbors of AS13214, it seems this router in the Caymans was the only one going  leaking the prefixes.  The first announcement of AS48285 was on 2009-05-11 11:03:11 UTC and the last on 2009-05-11 12:16:32 UTC, there were 266,289 prefixes leaked (they were withdrawn right afterwards).

This incident shows the advantage of having a wide set of peers for detection, it seems Cyclops was the only tool to detect this incident. Given the amount of banks and financial institutions in the Caymans, there would otherwise be a red flag, but it seems this case was an unintentional misconfiguration by AS13214. You can follow the NANOG thread here:
http://www.merit.edu/mail.archives/nanog/msg17928.html

–Ricardo

I’ve been receiving messages from several cyclopers inquiring about AS3130 containing a false AS adjacency to their own ASN. This is part of an experiment being developed by Randy Bush involving only prefixes 173.0.0.0/16 and 174.128.0.0/16, so it should NOT affect your network. If you want to know more about these experiments, please visit the site http://psg.com/173-174. You might also have seen that AS3130 has a very high degree (more than 25k), this is because by default Cyclops does not filter any BGP adjacency and takes into account every single BGP update for degree computation. However, given the false alerts AS3130 has raised, Cyclops will start filtering these announcements.

Sorry for any inconvinience this might have caused and keep cyclopying!

–Ricardo

BGP announcement from AS30890.

The announced prefixes were being propagated by several routers in Europe. Cyclops gave me a pretty good picture of what was happening through the “Prefix” menu:

http://cyclops.cs.ucla.edu/?vm=0&di=2009-02-12&de=2009-02-13&asn=30890&prfx=&loc=&ag=0&v=prfx&sub=1&p=91&s=0&o=8&d=1

At this point I could only do guess work about the causes of this incident. I know sometimes networks announce /32 to their providers to black-hole traffic to specific hosts under DoS attack, a feature called Remote Triggered Black Hole (RTBH), initially defined in RFC3882. This feature allows a network to send /32 routes upstream with a bogus next-hop so that traffic to specific hosts gets blocked before even reaching the destination network. Of course that this will “complete” the attack and block traffic to the specific end host. However, the advantage is that there will be more room for the other flows in the aggregate. A different technique was proposed more recently that blocks traffic based on source address using Unicast Reverse Path Forwarding (URPF) - check this I-draft. Basically a router will filter any traffic that does not match the interface where the source would be routed to.

I decided to contact directly Evolva Telecom to verify what went wrong. The folks at Evolva were very responsive and helpful, and provided all the details about what happened:

1. They started receiving traffic in one of their DE-CIX interfaces with destinations not announced by AS30890; apparently some other peer was forcing default routes to AS30890; traffic to these addresses was heavy, on the order of 2M packets/second
2. They start sending destination-based /32 filters to all their peers that were sending them traffic; and because these announcements did not have the NO_EXPORT community, they were further propagated for several hops and were caught by Cyclops; however, since they did not want to do destination based filtering on prefixes they were not announcing, they immediately withdraw these prefixes
3. They fired URPF in the router, which filtered traffic by the source addresses of the DDOS attack (probably fake addresses). Since these addresses had a void next-hop in the router,  traffic from these addresses start being dropped, and the attack was blocked
4. They changed the chain policy in all outgoing route-maps for /32s from
   accept to deny and stopped advertising any kind of /32 prefix

Several points can be taken from this incident. First, it was surprising how the /32 routes were propagated these many hops upstream, which I believe was not intended, and shows that there are still lots of networks out there that do not filter on prefix length. Second, additional care is needed inside IXPs, mostly because networks share a layer-2 cloud, and additional filtering is needed to make sure other peers are not default routing to your network. Here’re some suggestions on how to do that:

• Netflow analysis
• MAC accounting.  Some Internet exchanges will provide a
  traffic matrix which may indicate when this behavior is occuring. In
  some cases the operator may even notify you about this and “enforce” the
  policy when someone is attempting to route this way
• QPPB/DCU capabilities in Cisco/Juniper

The /32 routes displayed by Cyclops correspond actually to addresses that were being the target of a DDOS attack. Evolva Telecom is still investigating the origin of the attack, and I’ll follow this post as soon as I have more information.

–Ricardo

I’m pleased to announce Cyclops, a tool for real-time routing anomaly detection and alerting for service providers and enterprise networks. Cyclops uses real time data from thousands of vantage points of RouteViews, RIPE-RIS, Abilene, Packet Clearing House and University of Colorado Bgpmon, making it the widest and fastest free tool to assess how the rest of the world is reaching your network. Cyclops has been developed over the last half-year as part of the PhD work of UCLA students that have a track record of building monitoring tools. In fact, one of these students is first author of the very first paper on prefix hijack notification, PHAS published in USENIX Security 2006.

Cyclops features include:
- real-time alerting of prefix hijacks and misconfigured BGP announcements
- alerting of next-hop changes, AS in the middle (transit) and new prefixes
- alerting of new AS neighbor (false link announcement/leakages)
- AS connectivity assessment
- Prefix origin assessment
- Anomaly listings (transient prefix, anomalous depeerings, bogus ASNs, bogon prefixes, long/short prefixes)

To register for Cyclops please visit:
http://cyclops.cs.ucla.edu/?l=reg

To start configuring your network go to:
http://cyclops.cs.ucla.edu/?v=ma&tab=1
(need to be logged in)

You need to tell cyclops what are your prefixes, your ASNes and your neighbor ASNs. Then let Cyclops do its magic.

Please do not hesitate to contact cyclops@cs.ucla.edu in case you have questions/comments/bug reports.

During Cyclops development there were constant interactions with network operators and the Nanog community. We believe we have now a solid base to push the tool to the next level and we are currently looking for industry partners that might be interested in running Cyclops inside their networks.

Thanks for being one of the first Cycloper!

In name of the Cyclops team,

–Ricardo V Oliveira