I’ve been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu’s cloud services/equivalent of Google Drive).
Easily verified through Verizon’s Looking Glass.
We all know Verizon’s BGP routing is a disaster, but does anyone have any ideas?
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
To follow up on this:I’ve engaged Verizon’s executive office to finally try to get to a network engineer (because I don’t have a contact myself). The (proxied) response from the engineer was that they aren’t receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon’s peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren’t internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I’ve heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
Unfortunately it seems like policy that Verizon pushes any issues that aren’t internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well.
Baidu is behind CT. If CT is not passing on routes to 701 , what exactly do you expect Verizon to do about it?
but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering >agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren’t internal >routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external >services as well.
Any thoughts?
You’re probably right they have a responsibility to maintain their peering and routes, but rather than move mountains to get a large network to do “the right thing” (either vzw or baidu), I’d think most of the time it’s much easier to pick a different provider to work with instead.
I would expect Verizon to be able to contact CT and figure out why they aren’t passing the correct routes just as I might expect Baidu to do the same thing, I suppose.
What defines ‘correct’? ASN’s routinely make traffic engineering or business decisions not to announce certain prefixes to certain other ASNs. This certainly does sometimes cause reachability issues for end users, but that’s a choice someone made. That’s part of why it’s generally good practice to get more than 1 upstream if you can; it gives you more control to mitigate the impact of these choices.
It is completely possible that Baidu or CT are intentionally not announcing prefixes to VZ. It is also completely possible that they are and VZ is not accepting it.
I would expect Verizon to be able to contact CT and figure out why they aren’t passing the correct routes just as I might expect Baidu to do the same thing, I suppose. Ultimately, whose responsibility is it other than CT? That is my question. Maybe in this instance, it is common in the industry for this to be the responsibility of Baidu. That seems counterintuitive to me as it is Verizon without the proper route ultimately, but I am not an expert.
Certainly, I think it is incorrect to ask the customer to try to resolve these issues after bringing it to the attention of these services. If Verizon couldn’t reach one of Google’s edge servers, would it be Verizon or Google’s responsibility to fix that if the issue were an intermediate peer failing to broadcase the correct route? I have the sneaking suspicion that Verizon might actually take some action in that instance…
In this instance, I would define “correct” behavior as VZ having any route to this subnet; after all, customers don’t pay VZ to access just their network or a subset of the internet. You make a good point, though I would expect that if it isn’t VZ’s business decision to not have this route, they would have some sort of vested interest in figuring out why CT is not advertising it to them.
From VZ engineer, I heard that no one is advertising it to AS701, so I assume it is not a case of VZ refusing to accept it (which is what I had initially assumed having been frustrated in the past with VZ on other matters + seeing all their BGP issues in the past few years).
I looked at this a little last night, but didn't have time to write an email about it. Verizon has a lookingglass:
which you can use to see that Verizon has no route covering 182.61.200.0. Looking at routeviews, I see routes for 182.61.200.0/22, and 182.61.200.0/21, but no path via Verizon.
Also looking at routeviews, there's ample evidence that Verizon and China Telecom peer, so the question is, does China Telecom not advertise these routes to Verizon, or is Verizon rejecting them for some reason? I suspect only engineers at CT and VZ can answer that.
In this instance, I would define “correct” behavior as VZ having any route to this subnet; after all, customers don’t pay VZ to access just their network or a subset of the internet.
https://en.wikipedia.org/wiki/Default-free_zone
701 is still (AFAIK) in the DFZ , meaning if they don’t get an advertisement for a prefix, they ain’t getting there.
holow29,
Usually decisions made about limiting route propagation outbound for
certain prefixes tend to happen along financial and operational
constraints.
Remember, outbound route announcements control inbound traffic
volumes. So, if you’ve got some full pipes across the ocean, for example,
one way to limit the amount of traffic trying to go across
them is to stop announcing some high-traffic prefixes across them.
Note that it is almost always the route announcer that is subtracting
out routes to control traffic volumes; very seldom does the receiver
completely subtract out prefixes heard. The more usual case is for
the receiver to de-preference the routes down, but to keep them in
their tables as a fallback, because they can control that behaviour.
From the route-exporting network’s perspective, however, they have
no reason to believe that the receiving network would listen to and
obey MEDs that they send, to shift traffic away from the congested
path. They might have tried adding some prepends along the path,
only to discover the receiving network was using LOCALPREF knob
adjustments to control traffic, which kinda shits all over attempts
to traffic engineer via prepending, at which point the only knob you
have left to turn in order to move traffic off the pathway is to stop
announcing prefixes entirely.
As a not-so-parenthetical aside, this is why I strongly encourage
networks that use LOCALPREF as a control knob to limit the depth
to which LOCALPREF adjustments are made; ie, allow overriding
of the default LOCALPREF only to paths that are $AVG_PATH_LENGTH+2
or shorter. That way, if a peer wants to signal a pathway is congested
and should not be used under normal circumstances, they can still do
that by triple-prepending their ASN. That way, peers can continue to
announce prefixes, but still have some control over traffic flows via
as-path prepending, versus being backed into the corner of having to
completely stop announcing prefixes entirely across certain peering
sessions.
It is very likely that in this case, CT is not announcing prefixes to AS701
in order to control inbound traffic volumes across certain connections,
because CT has determined that is the only control knob they have
available that works; they have probably already experimented and
found that AS701 squashes inbound MEDs, and uses LOCALPREF
to force traffic flows regardless of as-path prepending.
Your only recourse in a situation like this is likely to be getting
a second connection that bypasses the CT-VZ choke point.
Best of luck!
Matt
Hello,
I looked at this a little last night, but didn't have time to write an
email about it. Verizon has a lookingglass:Looking Glass | Verizon Enterprise Solutions
which you can use to see that Verizon has no route covering 182.61.200.0.
Looking at routeviews, I see routes for 182.61.200.0/22,
and 182.61.200.0/21, but no path via Verizon.
Just tested using:
Region: Asia Pacific
Location: HKG
Show BGP route
Net: 182.61.200.0
and the answer is:
182.61.200.0/22 (2 entries, 1 announced)
*BGP Preference: 170/-101
Age: 4w0d 22:32:07 Metric: 10 Metric2: 500502
Announcement bits (4): 0-KRT 3-RT 8-BGP_RT_Background 9-Resolve tree 4
AS path: (65336) 4134 23724 38365 I (Atomic Originator)
Communities:
Localpref: 100
Paul
Also looking at routeviews, there’s ample evidence that Verizon and China
Telecom peer, so the question is, does China Telecom not advertise these
routes to Verizon, or is Verizon rejecting them for some reason? I
suspect only engineers at CT and VZ can answer that.
I took a quick look this morning from our view at all prefixes we see with an origin of AS38365.
This would appear, at least externally, to be an intentional CT or Baidu decision.
Well that shows my assertion was probably wrong.
Given the geopolitical situation between the US and China, along with certain government orders, could likely infer this is intentional.
Oddly enough I do see this via Verizon-but-XO:
182.61.200.0/22 *[BGP/170] 3d 09:25:39, MED 100, localpref 100
AS path: 2828 4134 23724 38365 I, validation-state: unverified
Hello,
Well that shows my assertion was probably wrong.
Given the geopolitical situation between the US and China, along with
certain government orders, could likely infer this is intentional.
Well, I remember VZN when it was UUNet... and then, they had 3 main ASNs:
- 701 (US)
- 702 (EU)
- 703 (APAC)
Playing with the LG, it may seem that the route is visible in the "703
region", so that may be traffic engineering, geo-whatever reason, config
mistake, ...
Paul
I'd only looked using their Ashburn location. So, Verizon does receive the route, but doesn't appear to propagate it beyond the APAC region.
That could still be either network's choice. The fact that their looking glass doesn't show anything for "Communities:" is suspicious to me, because I seriously doubt VZ isn't community tagging all routes...so I'm assuming there's a filter that's removing those for display...but then why even include the Communities: line in the output?...or maybe they've migrated to large communities and their looking glass hasn't been updated to display those?
I would expect Verizon to be able to contact CT and figure out why they aren’t passing the correct routes just as I might expect Baidu to do the same thing, I suppose. Ultimately,
You seem to be misunderstanding the relationship here…
Baidu is a CUSTOMER of VZ/701
VZ’s only responsibility here is to provide reachabilty to the routes which Baidu announces to VZ.
There’s no reason VZ should believe anything other than the fact that Baidu is not sending the particular prefix (or covering routes) to VZ ‘today’… for reasons which are entirely Baidu’s.
whose responsibility is it other than CT? That is my question. Maybe in this instance, it is common in the industry for this to be the responsibility of Baidu. That seems counterintuitive to me as it is Verizon without the proper route ultimately, but I am not an expert.
The person announcing the routes (or not) is the one who makes the decision to do so…
There’s, of course, a possibility that VZ (or any isp instead of VZ in this particular case) has decided to filter certain prefixes from Baidu’s peering, but
that’s not generally the case for ISPs…
Note: there are no ROA for the /24 in question, but Baidu does claim they may announce: 182.61.200.0/22
according to IRR data…
Certainly, I think it is incorrect to ask the customer to try to resolve these issues after bringing it to the attention of these services. If Verizon couldn’t reach one of Google’s edge servers, would it be Verizon or Google’s responsibility to fix that if the issue were an intermediate peer failing to broadcase the correct route? I have the sneaking suspicion that Verizon might actually take some action in that instance…
you have misunderstood the customer/provider relationship I suspect.
Baido is NOT a customer of Verizon. They are a customer of China Telecom who has a peering relationship with Verizon.