Issues with 4-octet BGP AS and Akamai?

Mailman List Mirror (Read Only)
1

Hello BGP and/or Akamai experts,

Has anyone come across issues with using the new 4-octet BGP AS number format and reaching websites hosted by Akamai?

One of my customers currently uses the AS number of one of their partner companies, which is in the standard 2-octed AS format. They were recently assigned their own AS which is in the 4-octet AS format.

When they advertise their networks using the 2-octet AS number everything works fine, they can browse to all websites and all their outbound and inbound internet traffic works perfectly.

However when they stop advertising their networks using the 2-octet AS and advertise their networks using the new 4-octet AS number, they lose reachability to a number of websites, all of which seem to be hosted by Akamai, including www.cisco.com<http://www.cisco.com>, www.dell.com<http://www.dell.com>, and a number of others. Websites that are not hosted by Akamai work just fine.

We checked their route advertisements from several looking glass servers and the routes seem to be propagating properly, albeit there may be a number of older routers still out there that don't understand the new 4 octet BGP AS format.
For example, their AS shows up as AS_TRANS 23456 on some looking glass routers.

My understanding is the 4-octet ASN's have been out for quite some time now, so any interoperability issues should have been resolved by now. But could there be some old BGP speakers who don't understand the new format or the translated AS 23456 format and drop the routes?

Has anyone experienced similar issues when using the new format 4 byte BGP ASN's?

Any thoughts on who at Akamai we can speak to in order to troubleshoot this further?

Thanks,
Greg

[http://www.cisco.com/web/europe/images/email/signature/logo05.jpg]

Gregory Gombas
CCIE# 19649 - R&S
Network Consulting Engineer
Advanced Services
grgombas@cisco.com<mailto:grgombas@cisco.com>
Office: +1-212-714-4497
Mobile: +1-201-675-9457

Cisco Systems Limited
One Penn Plaza
6th & 9th Floors
New York, NY 10119
United States
Cisco.com

[Think before you print.]Think before you print.

This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

2

Hi,

What prefix and ASN is this about?

Are you sure you are advertising from an AS4 capable router?

Do you see the expected 4-byte ASN as origin in a aggregator looking
glass like http://lg.ring.nlnog.net/prefix_detail/lg01/ipv4?q=www.nlnog.net ?

Kind regards,

Job

3

Can you share details? Did you contact akamai?

Feel free to ping me offline.

- Jared

4

Thank you all for your assistance thus far. I wanted to confirm with my customer that it was okay to share more details and they said it was okay. We did just send an email to Akamai net support and awaiting their reply.

The customer is NYULH (AS 394666). They currently use NYU (AS 12) for internet connectivity. They advertise the following prefixes to NYU:

216.165.124.0/24
216.165.125.0/24
216.165.126.0/24
216.165.127.0/24

NYU aggregates the above prefixes, strips NYULH's AS number, and replaces it with their own AS number (AS 12).
The aggregates are as follows:

216.165.124.0/23
216.165.126.0/23

Below is a sample /23 route seen from one of the looking glass servers with origin AS 12:

216.165.124.0/23
[DIGITALOCEAN3 2017-11-11 from 162.243.188.2] * (100/-) [AS12i]
  Type: BGP unicast univ
  BGP.origin: IGP
  BGP.as_path: 393406 3630 12
  BGP.next_hop: 162.243.188.2
  BGP.local_pref: 100
  BGP.atomic_aggr:
  BGP.aggregator: 192.168.255.3 AS12
  BGP.community: (14061,2000) (14061,2002) (14061,3000) (14061,3001) (65363,714) (65363,2906) (65363,13335) (65363,13414) (65363,14061) (65363,20940) (65363,32934) (65363,41690) (65363,46489) (65363,65340)
  BGP.ext_community: (RPKI Origin Validation State: not-found)

With their routes originating from AS 12, all their internet connectivity works fine.

However when they failover to their secondary path which is F5 Silverline DDOS protection over Optimimum Lightpath, they are unable to connect to any Akamai hosted websites.
The difference between their primary path and secondary path is that the secondary path does not strip their origin AS 394666.

To answer Job's question, yes the originating router is AS4 capable. I checked the looking glass link you provided and see the correct origin AS 394666. See below:

216.165.124.0/24
[DIGITALOCEAN5 14:14:44 from 5.101.111.2] * (100/-) [AS394666i]
  Type: BGP unicast univ
  BGP.origin: IGP
  BGP.as_path: 202109 2914 55002 394666
  BGP.next_hop: 5.101.111.2
  BGP.local_pref: 100
  BGP.community: (2914,410) (2914,1203) (2914,2201) (2914,3200) (14061,2100) (14061,2101) (14061,4000) (14061,4001)
  BGP.ext_community: (RPKI Origin Validation State: not-found)

However we noticed some of Level 3's looking glass routers only see the AS_Trans 23456 as shown in the output below. I'm assuming that means some of Level3's routers are not AS4 capable, but does that mean they will drop the routes?

Report generated from: car1.jan1

Route results for 216.165.124.0/24 from Jackson, MS

BGP routing table entry for 216.165.124.0/24
Paths: (2 available, best #2)
  1299 55002 23456
  AS-path translation: { TELIANET DEFENSENET-1 AS23456 }
    ear3.Dallas1 (metric 43807)
      Origin IGP, metric 100000, localpref 86, valid, internal
      Community: North_America Lclprf_86 United_States Level3_Peer Dallas Level3:10497
      Originator: ear3.Dallas1
  1299 55002 23456
  AS-path translation: { TELIANET DEFENSENET-1 AS23456 }
    ear3.Dallas1 (metric 43807)
      Origin IGP, metric 100000, localpref 86, valid, internal, best
      Community: North_America Lclprf_86 United_States Level3_Peer Dallas Level3:10497
      Originator: ear3.Dallas1

Thanks,
Greg

Gregory Gombas
CCIE# 19649 - R&S
Network Consulting Engineer
Advanced Services
grgombas@cisco.com
Office: +1-212-714-4497
Mobile: +1-201-675-9457
Cisco Systems Limited
One Penn Plaza
6th & 9th Floors
New York, NY 10119
United States
Cisco.com

Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

5

Greg,

I have a 4 byte ASN and have not had any issues with reach ability,
including the 2 websites you have linked.

James

6

Are you advertising out multiple circuits? Check the pathing both
directions if you can. A lot of CDNs enforce uRPF strict.

7

Hi Tyler,

Unfortunately we had a limited window to test so could not check the reverse path.

During our failover testing we stopped advertising out the primary path and only advertised out the secondary path. Routes are advertised out the secondary path through a DDOS prevention company called F5 Silverline which is reached via a GRE tunnel running over the Optimum Lightpath network.

So outgoing traffic would go from NYULH going out the Optimum Lightpath circuit and return traffic coming in on F5 Silverline’s network then tunneled over Optimum Lightpath back to NYULH.
So traffic was definitely routing asymmetrically.

However F5 Silverline assured us they have many customers using a similar setup but have no issues with Akamai.

I would think that many customers using similar DDOS prevention services such as F5 Silverline and Prolexic are routing asymmetrically as well, wouldn’t uRPF be affecting them all?

Thanks,
Greg
[http://www.cisco.com/web/europe/images/email/signature/logo05.jpg]

Gregory Gombas
CCIE# 19649 – R&S
Network Consulting Engineer
Advanced Services
grgombas@cisco.com<mailto:grgombas@cisco.com>
Office: +1-212-714-4497
Mobile: +1-201-675-9457

Cisco Systems Limited
One Penn Plaza
6th & 9th Floors
New York, NY 10119
United States
Cisco.com

[Think before you print.]Think before you print.

This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

8

It should be noted that AS_TRANS aka 23456 shouldn’t be visible on the global internet and many people may filter that on AS4_PATH cable devices.

The fact that you’re seeing an AS_TRANS path from the Telia LG is likely an indication that route may be not fully internet visible.

It’s fairly suspicious that someone in 2017 is still having that issue.

There’s a chance that is being filtered if someone is putting AS_TRANS in the AS4_PATH, but it does appear the route is fully visible:

https://stat.ripe.net/widget/routing-history#w.resource=216.165.0.0/17&w.normalise_visibility=true

Note that the routes in red have low visibility, which includes some of the CIDR blocks you specified.

https://stat.ripe.net/216.165.124.0%2F23#tabId=routing

- Jared

9

About who to speak with at Akamai... please forgive me if any of this
contact info is out-of-date, as I'm pulling from my notes from an old
network diagram...

Akamai Customer Care
- 877-425-2832

Akamai NOCC
- 877-625-2624
- 877-6-akamai (same as above)
- 617-444-3007
- nocc-shift@akamai.com
- (if you do anything that would your cluster, give them at least 3 hours
notice and give them IP of cluster
- hardware issues and 24x7 contact: nocc-tix@akamai.com +1-877-6AKAMAI

Akamai Network Support
- traffic issues: netsupport-tix@akamai.com +1-888-421-1003

-Aaron

10

Greg,

I don't see a routing database object for your routes pointing too your
AS394666 /24's, I only see one for AS12 for the /23 and /24's. It is
possible (and probable) you are being filtered due to that.

james

route: 216.165.124.0/23
descr: NEW YORK UNIVERSITY (added by MAINT-AS6517)
origin: AS12
remarks: This route object was registered by
Global Cloud Xchange MAINT-AS6517
on behalf of their customer:
NEW YORK UNIVERSITY
notify: support@relianceglobalcom.com
mnt-by: MAINT-AS6517
changed: support@globalcloudxchange.com 20160506 #00:49:14Z
source: RADB

(125-127)
route: 216.165.127.0/24
descr: New York University Medical Center (maintained by NYU NOC)
origin: AS12
mnt-by: MAINT-AS12
changed: noc@nyu.edu 20121121 #16:23:31Z
source: RADB

11

Hi James,

I don't see a routing database object for your routes pointing too your
AS394666 /24's, I only see one for AS12 for the /23 and /24's. It is
possible (and probable) you are being filtered due to that.

This is a really good observation, and a likely explanation!

@ OP - during IP space migrations from AS A to AS B you should ensure that route
objects exist for both ASNs.

You may also want to double check with your upstream providers what
their AS path
filters look like for your circuits.

Kind regards,

Job