Short-circuited traceroutes on FIOS

Mailman List Mirror (Read Only)
1

Anyone have an idea why there are some destinations that on residential verizon fios here in NY area terminate right on first external hop?

There seems to be a CDN common denominator here. On other networks with more typical BGP paths and traceroutes, users are reporting issues accessing these sites.

C:\Users\Home>tracert www.usfoods.com

Tracing route to statics.usfoods.com [205.132.109.90]
over a maximum of 30 hops:

   1 3 ms <1 ms <1 ms 172.18.24.1
   2 4 ms 3 ms 3 ms 192.168.2.33
   3 17 ms 6 ms 3 ms statics.usfoods.com [205.132.109.90]

Trace complete.

C:\Users\Home>tracert atworkhp.americanexpress.com

Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87]
over a maximum of 30 hops:

   1 2 ms <1 ms <1 ms 172.18.24.1
   2 3 ms 4 ms 23 ms 192.168.2.33
   3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com [139.71.19.87]

Trace complete.

C:\Users\Home>tracert portal.discover.com

Tracing route to e14577.x.akamaiedge.net [23.51.172.254]
over a maximum of 30 hops:

   1 3 ms 1 ms 18 ms 172.18.24.1
   2 21 ms 7 ms 6 ms 192.168.2.33
   3 4 ms 2 ms 2 ms a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]

Trace complete.

2

Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes right on their gateways.

More internet breakage. Thanks for the information to all who responded.

Random control test.

C:\Users\Home>tracert -d 1.4.5.6

Tracing route to 1.4.5.6 over a maximum of 30 hops

   1 15 ms 5 ms <1 ms 172.18.24.1
   2 3 ms 23 ms 24 ms 192.168.2.33
   3 3 ms 6 ms 3 ms 1.4.5.6

Trace complete.

Joe

Joe Maimon wrote:

3

wasn't vz pursuing some 'get the a cdn in the central office' for a
time? :slight_smile: perhaps this is the manifestation of that? :slight_smile:
or perhaps jared arranged to get links back from each CO to his
network gear in akamai-land?

I love conspiracies!

4

Is that unique to the FiOS gateway device? I don’t use their router and my traces go right out.

5

I had this issue while looking at Ripe Atlas measurements.

Turns out these Verizon boxes spoof ICMP with TTL = 3 (or 2, I don't recall). Try doing a UDP or TCP based traceroute instead.

Maybe you're seeing the same problem.

Kind Regards,
Filip

6

This is not from a verizon CPE. Its happening on their CO internet gateway customer facing routers.

tcptraceroute looks more legit

Joe

Nimrod Levy wrote:

7

mtr -u 4.2.2.2 --report-wide
Start: 2019-12-10T21:26:20-0500
HOST: fedora-lenovo Loss% Snt Last Avg Best Wrst StDev
1.|-- _gateway 0.0% 10 1.3 1.4 1.1 2.3 0.3
2.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
3.|-- b3346.nwrknj-lcr-21.verizon-gni.net 0.0% 10 6.7 4.8 2.2 8.2 1.9
4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
5.|-- 0.ae1.br1.ewr6.alter.net 0.0% 10 18.9 6.2 3.6 18.9 4.6
6.|-- lag-12.ear3.newark1.level3.net 20.0% 10 3.9 3.3 2.2 4.4 1.0
7.|-- ae-2-3602.ear2.newyork1.level3.net 90.0% 10 5.6 5.6 5.6 5.6 0.0
8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0

Verizon FIOS
They are blocking ICMP and returning false responses right at their gateway at the CO, not the CPE (I’m using my own router)

You have to do it using UDP to get real results of a traceroute.

  • Javier
8

Is that unique to the FiOS gateway device? I don't use their router and my traces go right out.

I also don't use their device and:
$ traceroute 205.132.109.90
traceroute to 205.132.109.90 (205.132.109.90), 30 hops max, 60 byte packets
1 _gateway (192.168.100.1) 3.085 ms 2.990 ms 2.795 ms
....
15 * 12.250.138.90 (12.250.138.90) 65.970 ms *^C

-chris
(perhaps this is location specific? I'm in the ashburn-ish-area-ish)

9

Is that unique to the FiOS gateway device? I don't use their router and my
traces go right out.

I also don't use their device and:
$ traceroute 205.132.109.90
traceroute to 205.132.109.90 (205.132.109.90), 30 hops max, 60 byte packets
1 _gateway (192.168.100.1) 3.085 ms 2.990 ms 2.795 ms
....
15 * 12.250.138.90 (12.250.138.90) 65.970 ms *^C

-chris
(perhaps this is location specific? I'm in the ashburn-ish-area-ish)

It's protocol specific. Windows tracert uses icmp instead of udp.
On a linux box try
  ping -t 2 205.132.109.90

You should get a time to live exceeded but the Verizon router gives
you an echo reply instead.

Regards,
Lee

10

They're returning fake ICMP echo replies from their BNGs for echo packets with TTL=1, thus any ICMP traceroute (Windows and mtr by default, etc.) seems to terminate at their layer 3 edge. UDP/TCP traceroute are unaffected, ICMP works fine if you set the initial TTL to n+1 where n is the hop that's lying.

Support claims that it was a mistake, but it's also been 15+ months and it's pretty deliberate behavior. Draw your own conclusions...

-Rob

11

It's protocol specific. Windows tracert uses icmp instead of udp.
On a linux box try
  ping -t 2 205.132.109.90

You should get a time to live exceeded but the Verizon router gives
you an echo reply instead.

that's hilariously bad :frowning: I think this is the OLT really that's doing this...
$ ping -t 3 205.132.109.90
PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data.

From 130.81.32.236 icmp_seq=1 Time to live exceeded

$ ping -t 1 205.132.109.90
PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data.

From 192.168.100.1 icmp_seq=1 Time to live exceeded

$ ping -t 2 205.132.109.90
PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data.
64 bytes from 205.132.109.90: icmp_seq=1 ttl=254 time=3.38 ms

An outbound traceroute has:
1 _gateway (192.168.100.1) 2.537 ms 2.587 ms 2.703 ms
2 * * *
3 B3320.WASHDC-LCR-21.verizon-gni.net (130.81.32.236) 6.638 ms
B3320.WASHDC-LCR-22.verizon-gni.net (130.81.32.238) 6.223 ms 6.414
ms
...

and inbound that hop 2 is:
6 HundredGigE2-4-0-3.WASHDC-LCR-22.verizon-gni.NET (140.222.238.55)
5.504 ms HundredGigE2-6-0-3.WASHDC-LCR-21.verizon-gni.NET
(140.222.234.53) 9.261 ms 9.266 ms
7 ae203-0.WASHDC-VFTTP-320.verizon-gni.net () 7.955 ms 3.026 ms
ae204-0.WASHDC-VFTTP-320.verizon-gni.net (130.81.32.239) 2.347 ms

oh well, just wonky gpon again?

12

I’m in the same region as Chris and I still can’t make it fail. I wonder if it’s because I have static addressing?

13

If you have static addressing (biz account) then possibly different from what I have.

In North NJ, 3 different accounts I can verify have ICMP blocked as of sometime earlier this year or late last year so have to use udp to get a real traceroute.

Could not be deployed in all areas the same way.

  • Javier
14

How have we gone this long of a conversation without someone from FIOS stepping in and setting the record straight?

15

Greetings,

16

TTL decrement issues are fairly common across multiple vendors and hw,
can be sw can be hw limit. Common issues for example is if MPLS egress
PE receives explicit null labeled packet, it may not be able to
decrement TTL.
I may lack in imagination, but I struggle to envision a situation
where people decided to do this and then decided to be sneaky peaky
about it.

17

Yes, but you need to screw up gloriously on the decrement if you think that
"I decremented and it's zero now" means "therefor it must have been addressed
to me, so I'll send an ECHO REPLY instead of TTL EXCEEDED".

18

Traceroute is becoming more and more an expert’s tool because interpretation of its results isn’t straightforward.

I had written a paper last year and mentioned its misuse in academia in the context of estimating the number of energy-consuming devices between a source and a destination.
Traceroute was being used to count the number of physical router devices from the hop count, notwithstanding the use of MPLS in domain cores.
To an external observer, this results in significant underestimation of the energy consumption in the path from source to destination.

19

Yeah, and what do you do with a traceroute that looks like this…. (ip address intentionally changed)

C:>tracert -d -w 1 1.2.3.4

Tracing route to 1.2.3.4 over a maximum of 30 hops

1 8 ms 5 ms 5 ms 96.8.191.129

2 * * * Request timed out.

3 * * * Request timed out.

4 * * * Request timed out.

5 * * * Request timed out.

6 * * * Request timed out.

7 * * * Request timed out.

8 * * * Request timed out.

9 * * * Request timed out.

10 * * * Request timed out.

11 * * * Request timed out.

12 * * * Request timed out.

13 * * * Request timed out.

14 * * * Request timed out.

15 * * * Request timed out.

16 * * * Request timed out.

17 267 ms 202 ms * 1.2.3.4

18 205 ms 175 ms * 1.2.3.4

19 160 ms 233 ms * 1.2.3.4

20 199 ms 201 ms * 1.2.3.4

21 213 ms 206 ms * 1.2.3.4

22 165 ms 158 ms * 1.2.3.4

23 237 ms 158 ms * 1.2.3.4

24 158 ms 290 ms * 1.2.3.4

25 158 ms 160 ms 158 ms 1.2.3.4

Trace complete.

C:>

20

what do you do with a traceroute that looks like this

Tell you to not change IP addresses so that I can do a proper analysis on it?
Recommend you use something other than windows?
Give you a stock tip?
The possibilities are endless.

(I’m being sarcastic)

It is shitty and I have no clue why ISPs play these games.

Rumor is that with FiOS, that so many gamers or game software was sending out ICMP requests that it was enough traffic for them to say screw this and block it. I don’t buy that but whatever.
Just annoying.

Oh yeah, and why do I still to this day have to use a HE ipv6 tunnel?