Well,
It happened. I have multiple subscribers calling in. They can not access Netflix.
Any contacts on list for Netflix that I can use to get my up blocks whitelisted?
John
geosupport@netflix.com has been very responsive for us. Best of luck, Netflix is always a hassle.
Phin
Is there some new DB that major CDNs are using?
We've been getting several reports of prefixes of ours being blocked, claiming to be VPNs, even though we've been using those subnets without incident for years.
HBO, Netflix, and Hulu appear to be common denominators. I have to wonder if they're all siphoning misinformation off of some new DB somewhere ...
+1 on Bryan’s message.
TL;DR
It seems lots of ISPs are struggling to figure out the why and the where of many IP addresses or blocks that are suddenly being blacklisted or flagged as VPNs or as out of service area.
I would really love to find, as Bryan said, if there is one particular IP reputation data provider who either got real aggressive recently or some (contaminated?) data was shared around. If there is I have no problem wading through their support processes to get it sorted but as it stands I just don’t know who to call. It just has been very difficult to glean any actionable info and of course the normal support teams at the respective streaming providers mostly just are telling customers to call their ISP… as if every random ISP has some special backdoor contact to every streaming provider where we can just get problems resolved quickly and easily while we all have a good laugh at people being able to watch their preferred movies and shows.
At least with email DNSBL filtering you usually get informed which DNSBL you are listed on and you can sort that out directly. In this case, the overall system of IP reputation based filtering seems still comparatively immature. The most I have gotten is after a very long phone call with someone at Hulu, they confirmed there is some issue affecting multiple networks and they are working on the issue and suggested I go through a whitelisting request process which may solve the problems but just for Hulu obviously.
I have published and tried to register our own geofeed data as defined in RFC8805 with as many IP geolocation providers as possible. I have checked around to as many IP geolocation and IP reputations sites as I can find and everything is either clean/accurate or there is no query method open to the public for troubleshooting that I can find. This is just yet another example to me of immaturity on dealing with geolocation problems: just spinning my wheels in the dark with mud spraying everywhere. There does not appear to be any consistency on handling issues by the content providers using IP geolocation and reputation to filter. If the content providers want to reject client connections they ought to provide more actionable information in their errors messages for ISPs since they are all just telling the users to call their ISPs. It just feels like a vicious circle.
So currently we are left with multiple video streaming providers that all started to flag many customers across many of our IP blocks all beginning earlier this month affecting customers, many of whom have been using the same IP address for years without issue until now. Do we try and decommission multiple IP subnets shuffle users over to new subnets and risk contaminating more subnets if this is an ongoing and regularly updated blacklist data set. This would further exacerbate the problem across yet more subnets that are getting scarcer. As a tangent, I am curious to see how IP geolocation and reputation systems are handling IPv6, I suppose they are just grouping larger and larger networks together into the same listings.
Someone who knows something concrete about this current issue, please throw us ISPs a bone.
With this email I feel like Leia recording a video plea for help addressed to Obi-Wan Kenobi… help me Nanog Community… you’re my only hope.
+1 on Bryan’s message.
TL;DR
It seems lots of ISPs are struggling to figure out the why and the where of many IP addresses or blocks that are suddenly being blacklisted or flagged as VPNs or as out of service area.
I would really love to find, as Bryan said, if there is one particular IP reputation data provider who either got real aggressive recently or some (contaminated?) data was shared around. If there is I have no problem wading through their support processes to get it sorted but as it stands I just don’t know who to call. It just has been very difficult to glean any actionable info and of course the normal support teams at the respective streaming providers mostly just are telling customers to call their ISP… as if every random ISP has some special backdoor contact to every streaming provider where we can just get problems resolved quickly and easily while we all have a good laugh at people being able to watch their preferred movies and shows.
At least with email DNSBL filtering you usually get informed which DNSBL you are listed on and you can sort that out directly. In this case, the overall system of IP reputation based filtering seems still comparatively immature. The most I have gotten is after a very long phone call with someone at Hulu, they confirmed there is some issue affecting multiple networks and they are working on the issue and suggested I go through a whitelisting request process which may solve the problems but just for Hulu obviously.
I have published and tried to register our own geofeed data as defined in RFC8805 with as many IP geolocation providers as possible.
So, RFC8805 is great and all, but it sure is annoying that you have to find webforms for a whole heap-o-geolocation providers, and figure out how to tell them where your geofeed file lives, etc.
Introducing RFC9092 - “Finding and Using Geofeed Data” ( https://datatracker.ietf.org/doc/rfc9092/ ). It slices, it dices…it even makes Julienne fries!..
Actually, nope, it just allows you to publish, in IRR records, the location of the RFC8805 format file. e.g:
$ whois -h whois.ripe.net 31.130.224.0 | egrep “inetnum|netname|remarks”
inetnum: 31.130.224.0 - 31.130.239.255
netname: ietf-meeting-network
remarks: Geofeed https://noc.ietf.org/geo/google.csv
The RFC has more examples, and also suggests an optional signature to strongly authenticate the data in the geofeed files…
W
Disclaimer: author
So I've made some progress, but not on the HBO front. (Hulu and Netflix have been responsive so far.)
Tried the e-mail address on Mike Hammett and Co.'s handy web-page, but got no response after several days. Ironically we were able to get through to the "closed-captioning" department, but this isn't particularly useful.
Does anyone have another possible contact for HBO folks to get some prefixes unflagged as "VPN"?
Try insulting them on Facebook. I did that several years ago in regards to wanting to be able to purchase HBO on-line without having to subscribe to it through a cable operator and shortly after, they launched a service to do just that.
(No, I’m not convinced that my insulting them on facebook had a causal effect, but it’s at least an amusing thought).
To be clear, this is not a geolocate issue. At least according to the error our users are getting.
Geolocate and VPN or Not are often kind of tied to the same kinds of reporting services and it may well be that whatever provider HBO is using for one is also being used for the other.
Owen
* warren@kumari.net (Warren Kumari) [Tue 31 Aug 2021, 21:04 CEST]:
So, RFC8805 is great and all, but it sure is annoying that you have to find
webforms for a whole heap-o-geolocation providers, and figure out how to
tell them where your geofeed file lives, etc.Introducing RFC9092 - "Finding and Using Geofeed Data" (
[..]
This won't help at all against geolocation vendors marking proxies and VPN endpoints as such.
-- Niels.
Indeed.
Let me be 100% clear:
We are having issues with prefixes flagged as VPNs. They are not.
We are NOT having issues with prefixes and geolocation.
So I’ve made some progress, but not on the HBO front. (Hulu and Netflix
have been responsive so far.)Tried the e-mail address on Mike Hammett and Co.'s handy web-page, but
got no response after several days. Ironically we were able to get
through to the “closed-captioning” department, but this isn’t
particularly useful.Does anyone have another possible contact for HBO folks to get some
prefixes unflagged as “VPN”?
I see a CDN at least in the path of their web server:
Thanks, Owen ... good point.
Now hearing reports for these same prefixes with Disney+ too.
So the common denominators are:
HBO
Hulu
Netflix
Amazon Prime
Disney+
... there has _got_ to be some new-fangled DB somewhere. This all started in the last month or so.
All of our RR objects, whois, DNS is solid ... dehr?
Fun times.
[snip]
I have customer subnet that does not work with Disney+ (pages never fully load), other streaming services are ok.
Sent multiple emails to Disney over the last few months using info on the Brothers WISP page. I got a response to the first email saying they would “pass it along for investigation” and silence since then.
What else can any of us do?
Force the traffic to these companies to use IPv6. Advise your customers that
you are doing this, why you are doing this and what steps they need to take
to enable IPv6 on their equipment. Your customers can’t be in a worse position.
"Dear customer,
if you want to reach … you will need to enable IPv6 support in
your home network. The world ran out of enough IPv4 for everyone several years
back and we have been sharing IPv4 between customers to allow you to reach IPv4
only sites. The afore mentioned companies are now blocking IPv4 connections from
ISPs that have to share IPv4 addresses. To give you a better service we are
blocking IPv4 connections to these companies so you will get a more reliable service
over IPv6.
For instructions on how to enable IPv6 connectivity on you home router see this
page ….
If your home router does not support IPv6 you will need to upgrade it to one that does."
You just broke 99% of the smart television sets in people’s homes, unfortunately.
That will resolve itself over time, of course, as sets are replaced, but anyone with
a set that is more than ~3 years old is mostly unlikely to have IPv6 support in it and
the vendors are ALL universally terrible about updating firmware.
As much as I like the idea (and that if a sufficient number of providers were willing
to do so, it might just serve as a forcing function to get firmware updates done),
I wouldn’t hold my breath and I suspect where there are competitive alternatives,
such a notice would be a boon to the competition.
Owen
You just broke 99% of the smart television sets in people’s homes, unfortunately.
If only everybody would not get a separate box, be that a AppleTV, a Playstation, a XBox, Chromecast, ... or many other options.
Fun part being that it is hard to get a Dumb TV... though that is primarily simply because of all the tracking non-sense in them that makes them 'cheaper'... (still wonder how well that tracking stuff complies with GDPR, I am thinking it does not ... Schrems anyone? 
)
That will resolve itself over time, of course, as sets are replaced, but anyone with
a set that is more than ~3 years old is mostly unlikely to have IPv6 support in it and
the vendors are ALL universally terrible about updating firmware.
Quite a bit of Android TV out there too.... and we all know how well that supports DHCPv6... 
Btw, geofeeds are getting fetched by some entities.
I've seen at least Dataprovider.com and DB-IP, others that fetch the CSV don't bother to set UA to something unique, thus one sees curl + axios coming by for instance, which does not tell much; but apparently we have to give up on UAs anyway, even though they are great for things like bots where one can have a wee bit of contact details in the line.
For instance DB-IP does regular updates of their code (rXXXX) and fetches quite often:
2a00:18a8:6:40:dcad:beff:feef:100 - - [23/Aug/2021:09:32:09 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6499"
2a00:18a8:6:40:dcad:beff:feef:100 - - [23/Aug/2021:09:02:14 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6499"
2a00:18a8:6:40:dcad:beff:feef:100 - - [24/Aug/2021:09:11:11 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6500"
2a00:18a8:6:40:dcad:beff:feef:100 - - [24/Aug/2021:09:42:15 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6500"
2a00:18a8:6:40:dcad:beff:feef:100 - - [24/Aug/2021:21:59:46 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6501"
2a00:18a8:6:40:dcad:beff:feef:100 - - [25/Aug/2021:01:24:28 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6501"
2a00:18a8:6:40:dcad:beff:feef:100 - - [25/Aug/2021:04:43:01 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6501"
2a00:18a8:6:40:dcad:beff:feef:100 - - [25/Aug/2021:05:11:05 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6501"
2a00:18a8:6:40:dcad:beff:feef:100 - - [26/Aug/2021:05:23:18 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6502"
2a00:18a8:6:40:dcad:beff:feef:100 - - [26/Aug/2021:02:49:59 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6502"
2a00:18a8:6:40:dcad:beff:feef:100 - - [27/Aug/2021:03:22:23 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6504"
2a00:18a8:6:40:dcad:beff:feef:100 - - [27/Aug/2021:03:55:04 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6504"
2a00:18a8:6:40:dcad:beff:feef:100 - - [28/Aug/2021:03:21:26 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6507"
2a00:18a8:6:40:dcad:beff:feef:100 - - [28/Aug/2021:03:51:20 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6507"
and looking up the IPs in DB-IP.com indeed nicely shows the locations configured in the geofeed, thus that is succesful.
But I am fairly sure that they will mark things as VPN if they get a sniff of that; though "VPN" seems to mean "Virtual Public Network", not the Private of days gone...
Greets,
Jeroen
“on you home router”
Is that still common anymore?
You just broke 99% of the smart television sets in people’s homes, unfortunately.
If only everybody would not get a separate box, be that a AppleTV, a Playstation, a XBox, Chromecast, ... or many other options.
Fun part being that it is hard to get a Dumb TV... though that is primarily simply because of all the tracking non-sense in them that makes them 'cheaper'... (still wonder how well that tracking stuff complies with GDPR, I am thinking it does not ... Schrems anyone?
Interestingly, no, it’s easy to get a “dumb TV” these days… We just call them “monitors”. I have two of them (one on either side) of my iMAC as I write this. (Makes for great X-Plane flying visuals.
On the other hand, the last time I went looking for a 27” monitor, I ended up buying a 44” smart television because it was a cheaper HDMI 4K monitor than the 27” alternatives that weren’t televisions. (It also ended up being cheaper than the 27” televisions which didn’t do 4K only 1080p, but I digress).
That will resolve itself over time, of course, as sets are replaced, but anyone with
a set that is more than ~3 years old is mostly unlikely to have IPv6 support in it and
the vendors are ALL universally terrible about updating firmware.Quite a bit of Android TV out there too.... and we all know how well that supports DHCPv6...
Does DHCPv6 really matter in a home? Really? I mean, I understand the NAC argument in the
corporate LAN environment, but the average household user can’t even spell NAC, let alone
implement an 802.1X stack.
Btw, geofeeds are getting fetched by some entities.
I presume geofeeds are getting fetched by many entities, but I’m not sure what the point of that is.
I've seen at least Dataprovider.com and DB-IP, others that fetch the CSV don't bother to set UA to something unique, thus one sees curl + axios coming by for instance, which does not tell much; but apparently we have to give up on UAs anyway, even though they are great for things like bots where one can have a wee bit of contact details in the line.
Yeah, Safari can now be trained to lie about it’s UA in developer mode easily. I presume this is true in Crome, Firefox, and just about anything else as well. It’s behind the drop-down panel to keep the adults out of the VCR, but it’s easily visible to any kid that would know how to program a VCR.
For instance DB-IP does regular updates of their code (rXXXX) and fetches quite often:
2a00:18a8:6:40:dcad:beff:feef:100 - - [23/Aug/2021:09:32:09 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6499"
2a00:18a8:6:40:dcad:beff:feef:100 - - [23/Aug/2021:09:02:14 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6499"
2a00:18a8:6:40:dcad:beff:feef:100 - - [24/Aug/2021:09:11:11 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6500"
2a00:18a8:6:40:dcad:beff:feef:100 - - [24/Aug/2021:09:42:15 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6500"
2a00:18a8:6:40:dcad:beff:feef:100 - - [24/Aug/2021:21:59:46 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6501"
2a00:18a8:6:40:dcad:beff:feef:100 - - [25/Aug/2021:01:24:28 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6501"
2a00:18a8:6:40:dcad:beff:feef:100 - - [25/Aug/2021:04:43:01 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6501"
2a00:18a8:6:40:dcad:beff:feef:100 - - [25/Aug/2021:05:11:05 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6501"
2a00:18a8:6:40:dcad:beff:feef:100 - - [26/Aug/2021:05:23:18 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6502"
2a00:18a8:6:40:dcad:beff:feef:100 - - [26/Aug/2021:02:49:59 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6502"
2a00:18a8:6:40:dcad:beff:feef:100 - - [27/Aug/2021:03:22:23 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6504"
2a00:18a8:6:40:dcad:beff:feef:100 - - [27/Aug/2021:03:55:04 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6504"
2a00:18a8:6:40:dcad:beff:feef:100 - - [28/Aug/2021:03:21:26 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6507"
2a00:18a8:6:40:dcad:beff:feef:100 - - [28/Aug/2021:03:51:20 +0000] "GET /geofeed.csv HTTP/1.0" 200 827 "-" "db-ip geofeed updater r6507"and looking up the IPs in DB-IP.com indeed nicely shows the locations configured in the geofeed, thus that is succesful.
I guess, but what do they do in terms of their “It’s a VPN” or “Not a VPN” service?
But I am fairly sure that they will mark things as VPN if they get a sniff of that; though "VPN" seems to mean "Virtual Public Network", not the Private of days gone...
A little of both these days.
I’m still holding out for DOHOTOROUDPOIPOGREOIPSECOIP for name resolution.
(I’m really not, just my twisted brand of cynical disgust at the everything->HTTPs trend)
Owen
If Netflix, et al. are not accepting connections from CGNs they are ALREADY obsolete.
Yes, I know it sucks to have to tell your customers that they just bought obsolete
equipment. Plug in Chromecast, Apple TV, and they can get back that functionality
with a product that does actually get upgraded.
Mark
Just get a "smart" TV, don't connect it to the Internet, and use its HDMI ports for your cable box, Apple TV, etc. and/or antenna input for local off-air reception.
Yeah, until TV manufacturers actually start incorporating, oh say, Google tv (which is just a form of Android) they are always going to be inferior. Having the TV just be a monitor is a feature, not a bug. It's a lot cheaper to upgrade a $50 hdmi based dongle than the whole TV, doubly so since manufacturers have a bad reputation for not supporting upgrades beyond the sell date. I have no idea whether any of the external ones support v6 though.
One thing that might be nice is for routers to internally number using v6 in preference to v4 and NAT that (if needed). Then you can easily tell what is still a laggard. My wifi cams might be poorly supported, but they don't need to interoperate with much on the Internet.
Mike, Google TV has been pretty nice since the Amazon feud finally ended though I hate that the protocol is still pretty proprietary