Hi,
is anyone else receiving Facebooks /24 more specific from Level3 (AS3356)?
31.13.70.0/24 *[BGP/170] 1w5d 17:21:28, MED 0, localpref 100, from
a.b.c.d
AS path: 3356 32934 I, validation-state: unverified
This more specific is only visible via AS3356
Facebook isnt even
announcing it via direct peering.
Any Facebook admin on or off list available to get this debugged and tracked
down?
Thanks & best regards
Jürgen
I see that specific route both of my upstreams and not going through level
3.
Network Next Hop MED LocPrf Weight Path
*>x 31.13.70.0/24 x.x.x.x 0 80 0 6461 32934 i
*i 31.13.70.0/24 x.x.x.x 0 80 0 209 32934 i
* 31.13.70.0/24 x.x.x.x 10 80 0 209 32934 i
31.13.70.0/24
This more specific is only visible via AS3356 ... Facebook
isn’t even announcing it via direct peering.
This specific, and many others, are only announced to peers in the
metro they originate in. To receive this prefix directly you'll
need to peer with us in Los Angeles.
It appears you're in Austria though, which means there's likely
no use in you peering with us in LA. We globally load balance
people to the best point of presence. You shouldn't see much, if
any, traffic to or from the above prefix.
Any Facebook admin on or off list available to get this
debugged and tracked down?
Please reach out to noc@fb.com in the future.
Regards,
Hi Doug,
looks like this is also affecting other prefixes:
157.240.3.0/24 *[BGP/170] 18w5d 16:50:37, MED 0, localpref 150
AS path: 3356 32934 I, validation-state: unverified
> to 80.239.128.178 via ae9.0
I understand that FB is using some type of DNS geo-loadbalancing and other
mechanism to redirect users to (possibly) nearer mirrors. The used DNS is
directly requesting the root DNS and not any other public DNS (e.g. not
8.8.8.8). Running some requests within 3 minutes gives me the below results:
www.facebook.com => star-mini.c10r.facebook.com. => 31.13.77.36
www.facebook.com => star-mini.c10r.facebook.com. => 157.240.2.35
www.facebook.com => star-mini.c10r.facebook.com. => 31.13.93.36
www.facebook.com => star-mini.c10r.facebook.com. => 31.13.76.68
Trace to 31.13.77.36:
Packets
Pings
Host Loss% Snt Last Avg
Best Wrst StDev
[...]
6. win-b4-link.telia.net 0.0% 3 21.5 27.5
21.5 35.7 7.3
7. level-ic-1573273-wien-b4.c.telia.net 0.0% 3 21.7 20.8
20.0 21.7 0.8
8. ae-1-9.edge2.sanjose3.level3.net 0.0% 3 177.1 179.8
177.1 182.6 2.8
9. 4.53.210.78 0.0% 3 178.3 178.8
178.3 179.2 0.5
10. po131.asw04.sjc1.tfbnw.net 0.0% 3 184.6 180.0
177.6 184.6 4.0
11. po241.psw02.sjc2.tfbnw.net 0.0% 3 177.8 177.6
177.3 177.8 0.3
12. 173.252.67.79 0.0% 3 176.3 177.4
176.3 177.9 0.9
13. edge-star-mini-shv-01-sjc2.facebook.com 0.0% 3 177.6 177.4
177.1 177.6 0.3
Trace to 157.240.2.35:
Packets
Pings
Host Loss% Snt Last Avg
Best Wrst StDev
[...]
6. win-b4-link.telia.net 0.0% 3 20.9 25.8
20.8 35.8 8.6
7. level-ic-1573273-wien-b4.c.telia.net 0.0% 3 20.5 21.4
20.2 23.4 1.8
8. ???
9. 4.15.85.102 0.0% 3 142.9 138.5
135.8 142.9 3.8
10. po121.asw01.ord3.tfbnw.net 0.0% 3 135.5 135.7
135.5 136.0 0.4
11. po211.psw01c.ort2.tfbnw.net 0.0% 2 136.1 135.7
135.3 136.1 0.6
12. 173.252.67.127 0.0% 2 151.6 147.6
143.6 151.6 5.7
13. edge-star-mini-shv-01-ort2.facebook.com 0.0% 2 136.0 135.9
135.8 136.0 0.2
Trace to 31.13.93.36 => this one is going via the FB DE-CIX peering and
latency of 34ms is perfect:
Packets
Pings
Host Loss% Snt Last Avg
Best Wrst StDev
1. router 0.0% 5 0.5 0.5
0.5 0.6 0.0
2. 192.168.8.3 0.0% 4 0.7 0.7
0.7 0.8 0.1
3. 37.252.236.88 0.0% 4 37.8 38.0
31.8 46.1 6.0
4. er-03.00-09-23.anx04.vie.at.anexia-it.com 0.0% 4 25.3 25.2
23.1 29.2 2.9
5. cr-02.0v-08-72.anx03.vie.at.anexia-it.com 0.0% 4 21.3 30.4
20.6 40.1 10.9
6. cr-01.0v-08-73.anx25.fra.de.anexia-it.com 0.0% 4 45.7 39.1
33.1 45.7 5.3
7. ae1.br02.fra1.tfbnw.net 0.0% 4 33.0 34.6
32.1 40.3 3.8
8. po114.asw01.fra2.tfbnw.net 0.0% 4 33.6 35.6
33.3 41.7 4.0
9. po211.psw01d.fra3.tfbnw.net 0.0% 4 40.8 36.6
32.9 40.8 3.4
10. 173.252.67.171 0.0% 4 41.2 35.6
33.3 41.2 3.7
11. edge-star-mini-shv-01-fra3.facebook.com 0.0% 4 34.6 33.9
33.2 34.6 0.6
Trace to 31.13.76.68:
Packets
Pings
Host Loss% Snt Last Avg
Best Wrst StDev
[...]
6. win-b4-link.telia.net 0.0% 14 28.9 24.7
20.3 29.3 3.5
7. level-ic-1573273-wien-b4.c.telia.net 0.0% 14 22.8 22.5
20.0 30.9 3.7
8. ae-2-3613.edge1.seattle3.level3.net 0.0% 13 195.2 189.2
186.0 195.2 3.5
9. 4.59.232.46 0.0% 13 187.9 188.2
186.7 196.1 2.5
10. po101.psw03.sea1.tfbnw.net 0.0% 13 194.7 188.3
186.5 194.7 2.0
11. 173.252.67.125 0.0% 13 188.0 188.8
186.9 195.5 2.2
12. edge-star-mini-shv-01-sea1.facebook.com 0.0% 13 186.6 189.0
186.2 196.0 3.1
Best regards
Jürgen
looks like this is also affecting other prefixes:
Many of our prefixes are only announced to peers in the metro
they originate in. Please stop obsessing about this detail; it's
not the problem.
I understand that FB is using some type of DNS
geo-loadbalancing and other mechanism to redirect users to
(possibly) nearer mirrors.
We target traffic two ways. One is relatively traditional dns
global load balancing, using the resolver ip. The other
method---which steers the vast majority of our traffic---vends
urls that send people to a specific PoP based on their client ip.
It appears you're having a targeting problem. Please reach out
to our NOC to get support.
Hi, the load-balancing definitely doesn't choose the *nearest* mirror.
We are in France and unless we do dirty tricks, we *always* get directed
to US sites (as far as LA), with horrible performance. Everything since
end of December. As a consequence we let the dirty tricks in place
(query facebook.com and fbcdn.com on 8.8.8.8 instead of regular
recursive resolving) and we get directed to Frankfurt or Amsterdam
(never London or Paris).
Hi,
(query facebook.com and fbcdn.com on 8.8.8.8 instead of regular recursive resolving) and we get directed to Frankfurt or Amsterdam (never London or Paris).
This is exactly what I've implemented yesterday on my end :).
Best regards
Jürgen
Are your DNS resolvers on your network? No DNS forwarding?
Yes, DNS resolvers on our network. Forwarding only for facebook.com and
fbcdn.com, in order to eliminate bad performance associated with "direct
recursion".
Hi,
got the same from Kiev, Ukraine:
dig fbcdn.com
fbcdn.com. 300 IN A 31.13.74.1
which is slow and routed through USA
and
dig fbcdn.com @8.8.8.8
fbcdn.com. 299 IN A 31.13.93.3
which is fast and routed through Germany
Same is for IPv6.
Is there any solutions without dirty hacks?
got the same from Kiev, Ukraine:
dig fbcdn.com
fbcdn.com. 300 IN A 31.13.74.1
which is slow and routed through USAand
dig fbcdn.com @8.8.8.8
fbcdn.com. 299 IN A 31.13.93.3
which is fast and routed through GermanySame is for IPv6.
Is there any solutions without dirty hacks?
As mentioned in this thread, talk to the Facebook NOC:
If John is on the list, please have him contact me. Thanks.
Regards,
Roderick.
Sales
Cross Lake
SeaX-1 and SeaX-2