Dear NANOG
The Ecuadoran government has via the FLOK society hired Michel Bauwens of the P2p foundation to lead a two year long efforts to revision the ecudoran economy along the lines of a commons oriented collaborative society. I am very interested in the program yet i have NEVER been able to connect to their web site. At the end of two hours of trouble shooting with apple i was advised to call verizon. I am a FiOS customer on a two year contact. The traceroute below confirmed that the fault is in verizons network. The verizon tech agreed otherwise i never would have gotten the trouble ticket
my verizon trouble ticket is NJ DQ04PWR9.
Can someone tell me what number to call to pursue resolution of this trouble ticket?
as of 12:04 eastern time i still cannot connect
24 hours was the promise
14 of the 24 have elapsed
traceroute to floksociety.org (200.10.150.169), 64 hops max, 72 byte packets
1 192.168.1.1 (192.168.1.1) 0.759 ms 0.309 ms 0.357 ms
2 l100.cmdnnj-vfttp-26.verizon-gni.net (98.110.50.1) 36.778 ms 17.508 ms 7.316 ms
3 * g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net (130.81.184.119) 6.482 ms !N *
4 * * g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net (130.81.184.119) 7.101 ms !N
5 * g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net (130.81.184.119) 9.239 ms !N *
6 g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net (130.81.184.119) 6.823 ms !N * 8.846 ms !N
Traceroute has started…
traceroute to 200.10.150.169 (200.10.150.169), 64 hops max, 72 byte packets
1 192.168.1.1 (192.168.1.1) 0.622 ms 0.305 ms 0.361 ms
2 l100.cmdnnj-vfttp-26.verizon-gni.net (98.110.50.1) 6.633 ms 4.892 ms 4.809 ms
3 * * *
4 g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net (130.81.184.119) 10.441 ms !N * *
5 * * *
6 g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net (130.81.184.119) 9.932 ms !N * *
7 * * *
8 * * *
9 * * *
10 * * *
11 g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net (130.81.184.119) 9.765 ms !N * *
12 * * *
13 * * *=
Lookup has started…
Trying "floksociety.org"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13700
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;floksociety.org. IN ANY
;; ANSWER SECTION:
floksociety.org. 600 IN A 200.10.150.169
floksociety.org. 3600 IN MX 10 mail.floksociety.org.
floksociety.org. 3600 IN NS ns57.domaincontrol.com.
floksociety.org. 3600 IN NS ns58.domaincontrol.com.
floksociety.org. 3600 IN SOA ns57.domaincontrol.com. dns.jomax.net. 2013092400 28800 7200 604800 600
Received 174 bytes from 8.8.8.8#53 in 139 ms
My traceroute goes through, but we don't go through Verizon. However, the web server is returning an error that it is unavailable. It's possible that the destination web server has a geo location plug in that stops access from foreign locations, or that their server is down.
[root@lancaster ~]# traceroute -I 200.10.150.169
traceroute to 200.10.150.169 (200.10.150.169), 30 hops max, 40 byte packets
1 129.77.108.252 (129.77.108.252) 0.345 ms 0.384 ms 0.442 ms
2 switch-user1.ox.com (129.77.154.253) 0.408 ms 0.523 ms 0.585 ms
3 rtr-inet2.ox.com (129.77.1.252) 3.394 ms 3.437 ms 3.464 ms
4 129.77.3.254 (129.77.3.254) 0.515 ms 0.517 ms 0.541 ms
5 189d20f9.cst.lightpath.net (24.157.32.249) 4.909 ms 4.923 ms 4.922 ms
6 18267502.cst.lightpath.net (24.38.117.2) 7.318 ms 9.900 ms 9.889 ms
7 (69.74.203.201) 9.877 ms 9.444 ms 9.434 ms
8 * * *
9 adsl-065-015-003-181.sip.mia.bellsouth.net (65.15.3.181) 9.455 ms * *
10 * * *
11 xe-9-1-2.edge2.Newark1.Level3.net (4.31.45.173) 8.378 ms 14.395 ms 14.244 ms
12 ae-32-52.ebr2.Newark1.Level3.net (4.69.156.62) 39.992 ms 42.318 ms 42.303 ms
13 ae-4-4.ebr2.Washington1.Level3.net (4.69.132.101) 42.283 ms 42.284 ms 42.280 ms
14 ae-62-62.csw1.Washington1.Level3.net (4.69.134.146) 50.599 ms 50.594 ms 50.586 ms
15 ae-61-61.ebr1.washington1.level3.net (4.69.134.129) 40.769 ms 43.276 ms *
16 ae-2-2.ebr3.atlanta2.level3.net (4.69.132.85) 43.293 ms 39.230 ms 38.957 ms
17 ae-73-73.ebr2.Atlanta2.Level3.net (4.69.148.254) 38.942 ms 38.942 ms 38.501 ms
18 ae-2-2.ebr2.miami1.level3.net (4.69.140.141) 39.404 ms 37.772 ms 37.487 ms
19 ae-2-52.edge1.Miami2.Level3.net (4.69.138.107) 50.685 ms 50.674 ms 50.568 ms
20 telefonica.edge1.miami2.level3.net (4.71.212.118) 62.446 ms 60.038 ms 59.416 ms
21 176.52.251.189 (176.52.251.189) 57.850 ms 58.637 ms 58.541 ms
22 176.52.252.66 (176.52.252.66) 94.381 ms 97.548 ms 99.258 ms
23 * * *
24 * * *
25 * * *
26 host-186-5-116-193.telconet.net (186.5.116.193) 118.811 ms 118.803 ms 118.808 ms
27 host-186-101-89-42.telconet.net (186.101.89.42) 98.612 ms 98.589 ms 98.605 ms
28 200.10.150.169 (200.10.150.169) 98.534 ms 98.564 ms 98.505 ms
<root@newton> dig +short www.floksociety.org.
200.10.150.169
<root@newton> telnet 200.10.150.169 80
Trying 200.10.150.169...
Connected to 200.10.150.169.
Escape character is '^]'.
GET / HTTP/1.0
HTTP/1.1 503 Service Unavailable
Server: Varnish
Content-Type: text/html; charset=utf-8
Retry-After: 5
Content-Length: 418
Accept-Ranges: bytes
Connection: close
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<title>503 Service Unavailable</title>
</head>
<body>
<h1>Error 503 Service Unavailable</h1>
<p>Service Unavailable</p>
<h3>Guru Meditation:</h3>
<p>XID: 477990820</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
Connection to 200.10.150.169 closed by foreign host.
Site appears up and available, over Comcast Business fiber and Cogent from
Chicago (using Chrome 28).
Inaccessible via FIOS Washington DC too:
traceroute -T -p 80 200.10.150.169
traceroute to 200.10.150.169 (200.10.150.169), 30 hops max, 40 byte packets
1 L300.WASHDC-VFTTP-91.verizon-gni.net (173.73.47.1) 1.804 ms
1.595 ms 1.562 ms
2 G0-6-4-7.WASHDC-LCR-22.verizon-gni.net (130.81.216.250) 5.321 ms !N * *
Correctly accessible via Cox, Qwest, Sprint and others, but the
network path is really slow and really long.
The border is consistently with telefonica-wholesale.net and then
telconet.net. Beyond the border there are badly behaving routers,
including ones configured with RFC 1918 addresses. The addressable
routers are reachable via Verizon, just not the last hop.
traceroute -T -p 80 200.10.150.169
traceroute to 200.10.150.169 (200.10.150.169), 30 hops max, 60 byte packets
1 sark.dirtside.com (70.182.189.216) 0.708 ms 0.689 ms 0.569 ms
2 10.1.192.1 (10.1.192.1) 9.957 ms 9.874 ms 9.725 ms
3 ip68-100-3-49.dc.dc.cox.net (68.100.3.49) 9.631 ms 9.507 ms 9.424 ms
4 ip68-100-3-113.dc.dc.cox.net (68.100.3.113) 9.310 ms 9.226 ms 9.140 ms
5 mrfddsrj02gex070002.rd.dc.cox.net (68.100.0.145) 9.111 ms 9.019
ms 8.929 ms
6 68.1.4.139 (68.1.4.139) 8.791 ms * 5.981 ms
7 209.48.42.61 (209.48.42.61) 5.748 ms 11.361 ms 10.948 ms
8 vb2000d2.rar3.washington-dc.us.xo.net (207.88.13.66) 58.454 ms
52.415 ms 52.421 ms
9 te-3-0-0.rar3.atlanta-ga.us.xo.net (207.88.12.9) 60.543 ms
60.397 ms 60.378 ms
10 te-3-0-0.rar3.dallas-tx.us.xo.net (207.88.12.2) 58.211 ms 58.407
ms 58.392 ms
11 * * *
12 206.111.5.226.ptr.us.xo.net (206.111.5.226) 53.378 ms 49.080 ms 47.435 ms
13 Xe-8-1-0-0-grtmiabr3.red.telefonica-wholesale.net (94.142.125.54)
76.006 ms Xe8-0-2-0-grtmiabr4.red.telefonica-wholesale.net
(94.142.119.38) 60.181 ms
Xe13-1-4-0-grtmiabr4.red.telefonica-wholesale.net (213.140.43.109)
125.888 ms
14 Te-0-2-0-0-grtmiana4.red.telefonica-wholesale.net (94.142.119.233)
67.105 ms Te0-1-0-0-grtmiana4.red.telefonica-wholesale.net
(213.140.37.77) 63.435 ms
Xe5-1-8-0-grtmiana2.red.telefonica-wholesale.net (213.140.36.89)
141.873 ms
15 Xe9-3-0-0-gramiana4.red.telefonica-wholesale.net (94.142.126.197)
62.450 ms 176.52.249.245 (176.52.249.245) 66.665 ms 176.52.249.241
(176.52.249.241) 64.668 ms
16 176.52.252.66 (176.52.252.66) 118.619 ms 118.057 ms 117.934 ms
17 * * *
18 * * *
19 * * *
20 host-186-5-116-193.telconet.net (186.5.116.193) 122.586 ms
120.967 ms 115.040 ms
21 host-186-101-89-42.telconet.net (186.101.89.42) 122.801 ms
125.164 ms 119.520 ms
22 * * *
23 200.10.150.169 (200.10.150.169) 253.710 ms 246.684 ms 244.845 ms
200.10.150.169 is reachable from AS2828 and from AS20115, but not from
AS22394 (Verizon Wireless)
Thank you Bill
extremely helpful
if via FiOS from DC you cannot get through does his look like an intentional move by verizobn to black hole this site???
how to determine that?
would like to be sure before i start raising hell elsewhere
I have tried to connect all week long never once succeeded.
advise??
any verizon people here willing to help?
Also inaccessible from FIOS Boston:
new-host-2:~ mfidelman$ traceroute floksociety.org
traceroute to floksociety.org (200.10.150.169), 64 hops max, 52 byte packets
1 wireless_broadband_router (192.168.1.1) 1.534 ms 0.853 ms 0.724 ms
2 l100.bstnma-vfttp-84.verizon-gni.net (96.252.37.1) 7.619 ms 6.855 ms 7.304 ms
3 200.10.150.169 (200.10.150.169) 10.482 ms !N *^C
But just fine from our datacenter via xo.net. And the web server is up - at least to a text browser (Lynx).
Also via Verizion cell network (Boston area).
Some kind of routing table glitch or peering issue, perhaps?
William Herrin wrote:
My mistake - cell network is also blocked. Stupid autocorrect changed floksociety to folksociety. Sigh.... Other info is correct.
Miles Fidelman wrote:
err.. nothing in the /24 is reachable from 701's perspective (so it
seems)... so I'd suspect that there's a routing problem with the /24,
in fact the surrounding /24's also seem to be having the same problem.
esceula seems to announce a slew of /24's...
28027 | 200.10.149.0/24 | Escuela Superior Politecnica del Litoral
28027 | 200.10.150.0/24 | Escuela Superior Politecnica del Litoral
28027 | 200.10.151.0/24 | Escuela Superior Politecnica del Litoral
but not the covering /22:
200.10.148/22
looking at routeviews (who has a 701 peer):
route-views>sho ip bgp summ | in 157.130
157.130.10.233 4 701 12266262 1198223 968904820 0 0 3w2d
458803
route-views>
route-views>sho ip bgp neighbors 157.130.10.233 paths | in 28027
0x511FB188 11 0 701 12956 12956 12956 12956 12956 12956
12956 19169 27947 28027 i
0x47A5A4C0 2 0 701 3257 3257 19169 27947 28027 i
seems that not all of 28027's routes are heard by 701...
route-views>sho ip bgp neighbors 157.130.10.233 routes | in 28027
* 190.15.129.0/24 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
* 192.188.59.0 157.130.10.233 0 701 3257
3257 19169 27947 28027 i
* 200.9.176.0 157.130.10.233 0 701 3257
3257 19169 27947 28027 i
* 200.126.0.0/20 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
* 200.126.16.0/22 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
* 200.126.20.0/23 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
* 200.126.22.0 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
* 200.126.23.0 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
* 200.126.24.0/22 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
* 200.126.28.0 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
* 200.126.29.0 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
* 200.126.30.0 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
* 200.126.31.0 157.130.10.233 0 701 12956
12956 12956 12956 12956 12956 12956 19169 27947 28027 i
route-views>
this doesn't include the 200.10.148.0/22 networks at all though. I'd
look suspiciously at:
aut-num: as12956
as-name: Telefonica
descr: Telefonica Backbone Autonomous System
descr: Telefonica Wholesale Network
remarks: ===========================================================
remarks: http://www.telefonica-wholesale.com
remarks: =========================
who seems to be prepending like crazy...
Hi chris
really appreciate the help from ALL you guys
does what you just said mean that non reachability for version customer may mean a config problem for a small bloc and not something intentional??
I would expect that they're just not receiving the BGP advertisement
for 200.10.150.0/24 from telefonica-wholesale.net. No way to know from
the endpoints whether that's because Telefonica isn't sending or
because Verizon is rejecting.
Either way, it doesn't seem to be black holed. Black holed is when you
advertise a route to a destination and then dump the packets.
Regards,
Bill Herrin
Hi chris
really appreciate the help from ALL you guys
sure thing.
does what you just said mean that non reachability for version customer may mean a config problem for a small bloc and not something intentional??
that's probably hard to say... I do know that:
5 0.xe-9-2-0.GW9.IAD8.ALTER.NET (152.63.36.30) 6.457 ms 6.821 ms
0.xe-11-2-1.GW9.IAD8.ALTER.NET (152.63.42.2) 5.932 ms
6 telefonica-gw.customer.alter.net (152.179.50.114) 7.182 ms 5.433
ms 5.431 ms
7 Xe4-1-6-0-grtmiana2.red.telefonica-wholesale.net (94.142.123.145)
47.271 ms 48.381 ms Te0-7-0-5-grtmiana4.red.telefonica-wholesale.net
(94.142.126.182) 50.290 ms
telefonica is a 'customer' not a 'peer' of 701, based on their
connectivity. This means that telefonica has to tell 701: "Yo, I need
you to accept routes for x, y, z, ktnxbi!'
knowing the normal (well, 5yrs ago) config for customers of 701, I
don't expect there'd be special handling of this prefix either... so I
suspect either no one told 701 to accept this, or telefonica pouched
it at the handoff 
If they didn't do that, these routes wouldn't be accepted.
If telefonica botched some filter on their side (see the comment about
prepending).
It's fairly sure though that the prefix isn't 'blackholed'... since
it's everything in the /22 not just the local /32 or /24. Bill
Herrin's on target as well, it's really hard to say from here 
-chris
Working here on Bresnan (charter)
root@hq:~# traceroute floksociety.org
traceroute to floksociety.org (200.10.150.169), 30 hops max, 60 byte packets
1 * * *
2 cacco002dr9-GE-1-0-0-U0.int.bresnan.net (69.146.239.57) 38.721 ms
39.799 ms 39.899 m s
3 host-72-175-111-198.bln-mt.client.bresnan.net (72.175.111.198) 26.722
ms 27.671 ms 2 7.771 ms
4 chywyaT1cr5-XE-5-2-0-U0.int.bresnan.net (72.175.110.135) 27.791 ms
27.818 ms 27.820 ms
5 seawafh2cr5-XE-5-1-0-U0.int.bresnan.net (72.175.111.166) 67.666 ms
68.616 ms 68.641 ms
6 seawafh1tr5-XE-2-0-0-U0.int.bresnan.net (72.175.110.68) 51.856 ms
61.108 ms 61.768 m s
7 12.90.77.21 (12.90.77.21) 65.384 ms 66.327 ms 66.448 ms
8 cr2.st6wa.ip.att.net (12.123.46.130) 122.440 ms 112.765 ms 111.808 ms
9 cr2.dvmco.ip.att.net (12.122.1.77) 111.733 ms 102.272 ms 98.744 ms
10 cr2.dlstx.ip.att.net (12.122.31.89) 105.493 ms 106.421 ms 105.106 ms
11 dlstx02jt.ip.att.net (12.122.214.249) 104.780 ms 101.914 ms 106.405
ms
12 213.140.53.61 (213.140.53.61) 103.208 ms 192.205.35.250
(192.205.35.250) 100.279 ms 2 13.140.53.61 (213.140.53.61)
121.303 ms
13 Te0-5-0-6-grtmiabr5.red.telefonica-wholesale.net (94.142.125.58)
144.127 ms Xe8-0-2-0-
grtmiabr4.red.telefonica-wholesale.net(94.142.119.38) 159.790 ms
Xe3-1-2-0-grtmiabr4.red.
telefonica-wholesale.net (94.142.126.134) 139.640 ms
14 Te0-2-0-4-grtmiana4.red.telefonica-wholesale.net (94.142.123.5)
138.665 ms Te0-1-0-0-g
rtmiana4.red.telefonica-wholesale.net(213.140.37.77) 139.681 ms
Xe7-1-6-0-grtmiana2.red.t
elefonica-wholesale.net (94.142.125.154) 145.648 ms
15 176.52.249.245 (176.52.249.245) 163.222 ms 176.52.251.201
(176.52.251.201) 143.164 ms 176.52.249.241 (176.52.249.241)
140.896 ms
16 176.52.252.66 (176.52.252.66) 184.038 ms 188.723 ms 181.953 ms
17 * * *
18 * * *
19 * * *
20 host-186-5-116-193.telconet.net (186.5.116.193) 189.490 ms 188.537 ms
188.500 ms
21 host-186-101-89-42.telconet.net (186.101.89.42) 190.366 ms 188.294 ms
192.845 ms
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
Traceroute black holes but the site works.
Site works on VZW in Colorado.
Cheers
vzw gets transit from places OTHER than 701 in a bunch of places...
http://www.downforeveryoneorjustme.com/floksociety.org says it's up. My VZ
dsl says no.
I'm guessing either VZ is not carrying a default inside their network, or they are propagating whatever blackhole might exist.
From my FIOS connection at home, I never reach anything that looks like a
router that I would expect to have a complete view of the Internet routing table.
streiner@whammy ~ $ traceroute www.floksociety.org
traceroute to www.floksociety.org (200.10.150.169), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 0.815 ms 0.800 ms 0.789 ms
2 L300.PITBPA-VFTTP-29.verizon-gni.net (173.75.39.1) 4.095 ms 4.094 ms 4.720 ms
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * G0-6-0-6.PITBPA-LCR-22.verizon-gni.net (130.81.168.90) 11.303 ms !N
The below information is taken from an "internal" router directly connected
to 701
A little earlier, both FiOS and a uunet/701 connection yielded '!N', but
as of this writing both paths are reaching that website.
Really glad i posted take a boqand thank you again!