Verizon: Extremely Strange CPE Routing in NYC/NJ Area

Can anyone from Verizon take a look at this behavior for us?

We’re having multiple Verizon FiOS users in the NYC/NJ area appear to teleport from their FiOS router to our IP in the Pittsburgh region. Users are seeing extreme slowness with TCP traffic, but ping times seem reasonable.

User 1:
1 fios_quantum_gateway (192.168.1.1) 1.575 ms 2.426 ms 3.193 ms
2 204.16.244.8 (204.16.244.8) 2.269 ms 3.055 ms 2.727 ms

User 2:
1 fios_quantum_gateway (192.168.1.1) 1.565 ms 1.048 ms 0.947 ms
2 204.16.244.8 (204.16.244.8) 2.162 ms 3.588 ms 3.048 ms

I can provide end-user NYC/NJ IPs off-list if desirable.

Here’s a normal looking trace from an FiOS line locally in the Pittsburgh region:

IP: 108.39.229.34

Tracing route to four.libsyn.com [204.16.244.8]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 5 ms 2 ms 7 ms lo0-100.PITBPA-VFTTP-301.verizon-gni.net [108.39.229.1]
3 5 ms 6 ms 6 ms B3301.PITBPA-LCR-22.verizon-gni.net [100.41.223.244]
4 * * * Request timed out.
5 * * * Request timed out.
6 13 ms 12 ms 13 ms 0.et-7-1-5.BR1.IAD8.ALTER.NET [140.222.226.17]
7 10 ms 10 ms 10 ms verizon.com.customer.alter.net [152.179.50.110]
8 12 ms 12 ms 13 ms be3084.ccr42.dca01.atlas.cogentco.com [154.54.30.65]
9 22 ms 22 ms 22 ms be2820.rcr21.pit02.atlas.cogentco.com [154.54.83.54]
10 22 ms 22 ms 21 ms 38.104.120.90
11 26 ms 21 ms 19 ms 204.16.241.133
12 * * * Request timed out.
13 21 ms 21 ms 21 ms 204.16.244.8

Is this a possible traffic engineering blip? I can’t say we’ve ever seen trace routes return such sparse results and actually make it to the destination.

Can anyone from Verizon take a look at this behavior for us?

We’re having multiple Verizon FiOS users in the NYC/NJ area appear to
teleport from their FiOS router to our IP in the Pittsburgh region.

Verizon is doing something seriously weird to windows traceroute:
C:\Users\Lee>tracert www.yahoo.com

Tracing route to atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232]
over a maximum of 30 hops:

  1 <1 ms <1 ms <1 ms fw.home.net
  2 1 ms <1 ms <1 ms vbz-router.home.net [192.168.1.1]
  3 8 ms 3 ms 6 ms
media-router-fp2.prod1.media.vip.ne1.yahoo.com [98.138.219.232]

Trace complete.

C:\Users\Lee>ping -i 12 www.yahoo.com.

Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.

Ping statistics for 98.138.219.232:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

C:\Users\Lee>ping -i 13 www.yahoo.com.

Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.219.232: bytes=32 time=32ms TTL=54
Reply from 98.138.219.232: bytes=32 time=31ms TTL=54
Reply from 98.138.219.232: bytes=32 time=33ms TTL=54
Reply from 98.138.219.232: bytes=32 time=33ms TTL=54

Ping statistics for 98.138.219.232:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 31ms, Maximum = 33ms, Average = 32ms

C:\Users\Lee>

traceroute from a linux box works as expected..

Lee

Windows traceroute uses ICMP by default, Unix based systems use UDP. For example from my Mac on Fios in Westchester, NY (and yes, they are doing something with ICMP traceroutes, possibly within their gateway box):

acpro:Documents mhuff$ sudo traceroute www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.9
traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.9), 64 hops max, 52 byte packets
1 firewall (10.1.1.1) 0.916 ms 0.419 ms 0.394 ms
2 * * *
3 b3369.nycmny-lcr-22.verizon-gni.net (130.81.16.102) 3.939 ms 10.172 ms
    b3369.nycmny-lcr-21.verizon-gni.net (130.81.16.100) 5.192 ms
4 * * *
5 0.ae4.xl2.buf4.alter.net (140.222.228.143) 13.500 ms * 12.970 ms
6 0.xe-5-3-0.gw1.buf4.alter.net (152.63.26.62) 12.390 ms 12.122 ms
    0.ae4.xl2.buf4.alter.net (140.222.228.143) 12.704 ms
7 0.xe-4-2-0.gw1.buf4.alter.net (152.63.26.46) 14.326 ms 13.069 ms
    verizon.com.customer.alter.net (204.148.47.162) 14.436 ms
8 et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103) 13.990 ms
    unknown-72-30-223-x.yahoo.com (72.30.223.3) 14.220 ms
    et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103) 13.711 ms
9 et-0-0-0.msr2.bf1.yahoo.com (74.6.227.137) 12.098 ms
    et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129) 13.388 ms
    et-18-0-1.msr1.bf1.yahoo.com (74.6.227.131) 14.457 ms
10 et-19-0-0.clr2-a-gdc.bf1.yahoo.com (74.6.122.37) 14.554 ms
    et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129) 14.810 ms
    et-1-1-1.msr1.bf1.yahoo.com (74.6.227.135) 13.502 ms
11 eth-18-3.bas2-1-flk.bf1.yahoo.com (98.139.128.75) 14.607 ms 16.608 ms
    eth-18-3-bas1-1-flk.bf1.yahoo.com (98.139.128.73) 15.161 ms
12 media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9) 13.470 ms
    eth-17-3.bas2-1-flk.bf1.yahoo.com (98.139.128.71) 13.582 ms
    media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9) 12.952 ms

macpro:Documents mhuff$ sudo traceroute -I www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.10
traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.10), 64 hops max, 72 byte packets
1 firewall (10.1.1.1) 0.675 ms 0.347 ms 0.322 ms
2 media-router-fp2.prod1.media.vip.bf1.yahoo.com (72.30.35.10) 2.456 ms 21.139 ms 12.834 ms

See this post for more info:
http://www.dslreports.com/forum/r32136909-Has-Vz-disabled-TTL-propagation