Can someone from Verizon Business please contact me?
It appears that your network is losing traffic from Verizon Wireless (e.g. 63.59.39.232, 63.56.37.4, or 63.59.67.68) to me (AS33362, e.g. to 69.89.207.16). Note that 63.59.166.100 -> 69.89.207.16 was successfully (around 2023-11-27).
This breaks email between us and it's been MONTHS of VZW getting nowhere.
Based on some traceroutes (on 2023-11-27 and again just now), the working ones go through 140.222.234.223 (0.ae10.GW7.CHI13.ALTER.NET) while the broken ones stop at 140.222.234.221 (0.ae9.GW7.CHI13.ALTER.NET).
For a good long while (months) we have had similar issues with various Verizon destinations.
I observed it only happens when passing through certain geographic regions of the US. Other regions make it through without issue.
This is directly observable and repeatable using Cogent’s Looking glass website.
Do an IPv4 Trace to 63.59.67.68 using their US-Minneapollis router. It dies.
Do an IPv4 Trace to 63.59.67.68 using their US-Los Angeles router. It reaches the destination.
I went through a handful of Cogent’s looking glass locations and found some that work and some that don’t and concluded there must be one or more Verizon routers in a certain set geographic area that are having the problems.
Ultimately the issue is not resolved for me but I was able to BGP TE the traffic around the problem areas to facilitate reachability to the impacted destinations. This is obviously a tenuous band-aid.
Long story short: please, please, please, someone at Verizon or someone who has the ear of someone at Verizon, please, please, please, look into this.
For me it is some AS 6167 destinations.
WHOIS for that ASN says this is Verizon Business.
AS Number: 6167
Org Name: Verizon Business
I am not sure how I am supposed to accurately or authoritatively discern the differences in specific IP prefixes (or ASNs) as to whether they are are used in the Verizon Wireless, Verizon Business, Verizon XYZ, etc.
I am also not sure what the value would be understanding the difference as I have zero contacts at any Verizon entity: Wireless, Business, or any other.
I imagine at some level, there is a parent Verizon umbrella organization that is ultimately responsible for all underling organizations/divisions but I am not particularly interested in trying to pick apart the business silos of Verizon and then from there trying to chase down specific Verizon entity contacts to try and figure out who, might be the right contact to look into this. I have made efforts, prior to this NANOG thread even starting, to get this issue rectified but I have had zero luck so far getting any appropriate person at Verizon to take notice.
It kind of feels like trying to reach out to some company regarding a geolocation or IP-reputation type issue… just a lot of “Sorry, I don’t know. try this other group that you already talked to” or simply “piss off” type responses. Both of which I have received in sizable quantities. Now that my brain is on that tangent, my favorite geolocation response was when I was told “your ISP needs to set the correct bits in the IP packets to designate the traffic as coming from the correct geography.” I laughed and I cried at that one.
I see the route originated by two different ASNs. I agree that when I use the AS6167 path, it is broken (for the destinations where it is broken; 63.59.166.100 was working despite using the AS6167 path).
BGP routing table entry for 63.59.0.0/16
Paths: 2 available
6939 701 22394
184.105.34.254 from 184.105.34.254 (216.218.253.228)
Origin IGP, metric 0, localpref 60, IGP metric 0, weight 0, tag 0
Received 21d19h ago, valid, external, best
Rx SAFI: Unicast
6461 701 6167
69.89.205.202 from 69.89.205.202 (69.89.205.202)
Origin IGP, metric 887, localpref 60, IGP metric 40, weight 0, tag 0
Received 4d03h ago, valid, internal
Community: 6461:5997
Rx SAFI: Unicast
Based on the names in WHOIS, I would say that both AS6167 and AS22394 are Verizon Wireless.
Based on the ASName of both AS, including CELLCO which is the actual name of the corporate entity known as Verizon Wireless, I would agree that both are in fact Verizon Wireless. The contacts are just corporate standard entities.
To close the loop on this, Verizon Wireless reported to me that they fixed the issue (whatever it was).
They further said that 63.56.37.4 was a typo; all IPs should have been in 63.59.x.x.
I am able to reach the 63.59.0.0/16 IPs in question: 63.59.39.232 & 63.59.67.68.
Justin: Thanks for the detail that this was reproducible from Cogent’s looking glass. I think there’s a good chance that contributed to them being able to find it (i.e. having an easy way for them to test).