Facebook doesn't have a route to my ISP's (Cogeco) IPv6 space?

I've been trying to figure out why I can reach an IPv6 address at
Facebook (2a03:2880:f012:3:face:b00c:0:1) through (only) one of my two
Internet connections as well as via an HE IPv6 tunnel but not the other
of my two ISP connections

At one point in time a traceroute was dying inside of he.net:

Host Loss% Snt Last Avg Best Wrst StDev
1. 2001:1970:5261:d600::1 0.0% 7 2.1 1.3 0.7 2.9 0.8
2. 2001:1970:4000:82::1 0.0% 7 10.0 14.0 8.3 37.9 10.6
3. 2001:1970:0:1a6::1 16.7% 7 13.2 215.5 10.8 1031. 455.9
4. he.ip6.torontointernetxchange.net 0.0% 7 12.3 12.9 11.2 15.3 1.6
5. 100ge9-2.core2.chi1.he.net 0.0% 7 23.6 23.0 21.3 27.6 2.2
6. 100ge15-2.core1.chi1.he.net 0.0% 7 21.7 22.5 21.6 24.9 1.2
7. 100ge12-1.core1.atl1.he.net 0.0% 7 34.2 35.1 34.1 36.1 0.7
8. 100ge5-1.core1.tpa1.he.net 0.0% 7 49.1 46.6 44.8 49.1 1.5
9. 100ge12-1.core1.mia1.he.net 0.0% 7 51.6 54.5 50.5 73.3 8.3
10. ???

But I think it getting that far time was an anomaly and frankly it
usually dies even before exiting my ISP's (Cogeco) network like this:

Host Loss% Snt Last Avg Best Wrst StDev
1. 2001:1970:5261:d600::1 0.0% 33 0.6 0.7 0.6 1.0 0.1
2. 2001:1970:4000:82::1 0.0% 33 8.2 10.8 8.1 40.5 5.6
3. 2001:1970:0:1a7::1 15.2% 33 23.4 20.1 16.5 23.4 1.5
4. 2001:1970:0:61::1 33.3% 33 16.8 17.6 14.5 25.9 2.5
5. 2001:1978:1300::1 0.0% 33 16.0 17.5 14.2 29.6 3.1
6. 2001:1978:203::45 0.0% 33 30.7 30.7 28.4 35.1 1.7
7. ???

When I asked the kind folks at he.net for some advice about the problem
(i.e. in the first traceroute above) their diagnosis was that
Facebook's IPv6 router(s) likely didn't have a route back to my Cogeco
IPv6 address.

Trying to talk to my ISP (again, Cogeco) has been impossible. One
simply cannot reach the people who know more than how to reset your
router and configure your e-mail.

I wonder how I could go any further with this to confirm the diagnosis
that Facebook doesn't have a route to the Cogeco network's IPv6 address
space given that I only have access to my end of the path.

Cheers,
b.

It's problem from Cogentco, they do not have IPv6 peer with HE.net and Google

Google and HE don't have IPv6 connectivity with Cogent because Cogent's CEO has been in some decades long pissing match with them about free settlement free peering. That's the unfortunate reality of the situation; nothing you can do other than have another route to HE/Google IPv6 targets. We have some Cogent circuits that are effectively useless for IPv6 as our customer base has heavy traffic to/from Google cloud services, so they can't be used for a backup / DR scenario; their only real value is an optimal route to other Cogent customers. I'm slowly replacing our Cogent circuits when feasible because the reality is our customers reaching Google over IPv6 via all our upstreams is more valuable than Cogent's cost savings.

Cogent != Cogeco

Cogeco is not related to Cogent

On Thu, Dec 20, 2018, at 0:0.36666666666 AM, David Guo via NANOG写:

In other words, they’re on The Internet and you (and your transit provider) are not.

Cogent != Cogeco

Cogent - US Backbone Provider
Cogeco - Canadian Cable TV & Internet provider

Yikes, they should change their name rather than be mistaken for Cogent lol

    Cogent != Cogeco
    
    Cogent - US Backbone Provider
    Cogeco - Canadian Cable TV & Internet provider

At this moment it appears there are multiple rifts in the IPv6 default-free zone (that don’t exist in the IPv4 realm), between various organizations. Focusing on one particular partitioning may not help address the other issues.

Kind regards,

Job

They should probably just rebrand to CoGeCo, but that does look kinda
silly, so, they probably won't.

Cogeco is an acronym for Compagnie Générale de Communication ("General Communications Company").

Both Cogent Communications and Cogeco are in pretty different market
segments (for anyone in the know), so, they each probably don't really
care about the confusion in their respective target markets.

C.

Well known problem.

You can use our tunnel broker connection (tb.netassist.ua) as a workaround.

17.12.18 22:01, Brian J. Murrell пише:

Well known problem.

Interesting. As in a general problem across the Internet or a well
known problem with Cogeco specifically?

You can use our tunnel broker connection (tb.netassist.ua) as a
workaround.

Thanks. But I actually already have a tunnel as well as a(nother)
native IPv6 ISP (yes, I have two consumer ISP connections) which routes
to Facebook properly.

The problem is that for the clients behind this router receiving RAs
for all three upstream connections and plumbing IPv6 addresses on each
of those networks, I know of no way to prevent them from choosing their
Cogeco IP address among the 3 and thus trying to use the Cogeco route.

I can (and have) put a rule into that router to refuse connections to
Facebook when using the Cogeco source address which sends TCP clients a
TCP reset with the hope that (good at least) clients/IPv6 stacks will
try a different source address but the results on that seem spotty at
best.

b.

Job,

What other partitioning like this exists?

Jack

The most famous lack of IPv6 connectivity is between AS6939 (Hurricane
Electric LLC) and AS174 (Cogent Communication), not to be confused
with AS7992 (Cogeco Cable), which actually does peer with both
cogentco.com and he.net, including on IPv6.

The HE/Cogent issue is so widely known that, apparently, nearly
everyone in this thread is misattributing the well-known issue as the
reason for the problems described by the OP, but they're very
obviously not related, apart from the similarity in the brand names.

C.

Hi Brain

Let’s chat offlist to find why this is happening. The behaviour looks unusual and needs more troubleshooting.

Thanks

Anurag Bhatia
(Hurricane Electric)

Hi Brian,

With all the CoGeCo :canada: vs. CogentCo :us: confusion, I don't think
anyone asked the obvious question yet…

But what's exactly at 2a03:2880:f012:3:face:b00c:0:1?

I've tried reaching it on port http, and it doesn't answer:

% curl -6 -v --head --resolve
"www.facebook.com:80:2a03:2880:f012:3:face:b00c:0:1" www.facebook.com
* Added www.facebook.com:80:2a03:2880:f012:3:face:b00c:0:1 to DNS cache
* About to connect() to www.facebook.com port 80 (#0)
* Trying 2a03:2880:f012:3:face:b00c:0:1...
* Connection refused
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host
%

Meanwhile, regular IPv6 connectivity to Facebook works just fine, over
HE.net, no less.

Cheers,
Constantine.
http://cm.su/

Cogent started business in 1999 and Cogeco has been around since the 1950s. Who should change their name again?

(To OP: I believe that every last-mile provider in Canda is still offering IPv6 as a best-effort, unsupported service. As a former Canadian networking guy, this ... angers me. Good luck ...)

Yeah. I'm aware of this. But I want to give them the benefit of the
doubt that this problem is simply ignorance and something they'd like
to fix rather than any of the more depressing alternatives.

Cheers,
b.

Hi Brian,

Hi,

But what's exactly at 2a03:2880:f012:3:face:b00c:0:1?

It's one of the endpoints involved in Facebook's Messenger service.
IIRC it's "graph.facebook.com", although I note that that address is
currently answering as:

graph.facebook.com. 2068 IN CNAME api.facebook.com.
api.facebook.com. 2068 IN CNAME star.c10r.facebook.com.
star.c10r.facebook.com. 25 IN AAAA 2a03:2880:f00e:a:face:b00c:0:2

To be fair though, that one could just be what a load-balancing name
service is responding at the moment. Notice that the two addresses are
only off by one.

But even more interesting is that it's now working:

                                My traceroute [v0.92]
pc.interlinx.bc.ca (2001:1970:5261:d600:c5d9:3319:afbc:3bb6) 2018-12-20T23:07:14-0500
Keys: Help Display mode Restart statistics Order of fields quit
                                             Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2001:1970:5261:d600::1 0.0% 94 0.6 0.7 0.4 3.8 0.3
2. 2001:1970:4000:82::1 0.0% 94 9.0 20.2 7.3 889.2 91.0
3. 2001:1970:0:1a7::1 39.4% 94 19.8 19.5 17.9 26.3 1.8
4. 2001:1970:0:61::1 45.2% 93 18.1 16.5 14.3 22.5 1.9
5. ae7.pr03.yyz1.tfbnw.net 0.0% 93 19.6 19.4 14.9 76.8 9.4
6. po103.psw04.yyz1.tfbnw.net 0.0% 93 14.5 15.8 13.9 24.0 1.5
7. po4.msw1ab.01.yyz1.tfbnw.net 0.0% 93 15.3 15.7 14.2 24.0 1.3
8. 2a03:2880:f00e:a:face:b00c:0:1 0.0% 93 19.2 15.5 13.7 22.5 1.7

And even just a few minutes ago it was not as I was testing it for
another (off-list) query:

                                My traceroute [v0.92]
pc.interlinx.bc.ca (2001:1970:5261:d600:c5d9:3319:afbc:3bb6) 2018-12-20T22:47:51-0500
Keys: Help Display mode Restart statistics Order of fields quit
                                             Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2001:1970:5261:d600::1 0.0% 112 0.6 0.7 0.5 5.1 0.4
2. 2001:1970:4000:82::1 0.0% 112 9.2 15.8 7.3 374.4 36.2
3. 2001:1970:0:1a7::1 17.0% 112 18.3 19.1 17.6 21.9 0.8
4. 2001:1970:0:61::1 33.0% 112 15.9 16.0 14.8 27.9 1.8
5. 2001:1978:1300::1 0.0% 112 15.5 17.0 14.1 49.8 4.4
6. 2001:1978:203::45 0.0% 112 29.5 29.8 28.2 46.8 2.1
7. ???

Perhaps the bit of cage rattling that I have done here has knocked
something loose. :slight_smile:

Cheers,
b.

I think TFBNW folks definitely read these lists, even if they never
respond officially.

Their whole network was apparently down via IPv6 for like at least a
few days several years ago; they've then fixed it back in the day by
simply removing the AAAA records. :slight_smile:

https://puck.nether.net/pipermail/outages/2013-May/005570.html
https://puck.nether.net/pipermail/outages/2013-May/005579.html

It's kind of amazing that tech reporters never really pick up these
stories, TBH. I guess these outages don't really sell, especially if
noone but a few selected users can even detect it in the first place,
even if they do last for days or even weeks/years for certain users in
certain configurations.

Cheers,
Constantine.