Increasing problems with geolocation/IPv4 access

Mailman List Mirror (Read Only)
1

I’ve been seeing an increasing problem with IP space not having the ability to be used due to the behaviors of either geolocation or worse, people blocking IP space after it’s been in-use for a period of time.

Before I go back to someone at ARIN and say “your shiny unused 4.10 IP space” is non-functional and am at a place where I need to start/restart/respawn the timer, I have a few questions for people:

1) Do you see 23.138.114.0/24 in any feeds from a security provider that say it can/should be blocked? If so, I’d love to hear from you to track this down. Over the new year we had some local schools start to block this IP space.

2) many companies have geolocation feeds and services that exist and pull in data. The reputable people are easy to find, there are those that are problematic from time-to-time (I had a few customers leave Sling due to the issues with that service).

3) Have you had similar issues? How are you chasing all the issues? We’ve seen things from everything works except uploading check images to banks, to other financial service companies block the space our customers are in. If we move them to another range this solves the problem.

4) We do IPv6, these places aren’t IPv6 modern at all, so that’s no help.

5) IRR+geofeed are published of course. I’m thinking that it might be worthwhile that IP space have published placeholders when it’s well understood, eg: ARIN 4.9 space, I can predict what our next allocation would be, it would be great to have it be pre-warmed.

I’ve only seen a few complaints against all our IP space over time, so I don’t think there’s anything malicious coming from the IP space to justify it, but it’s also possible they didn’t make it through.

If you’re with the FKA Savvis side, can you also ping me, I’d like to see if you can reach out to our most recent complaint source to see if we can find who is publishing this. Same if you’re with Merit or the Michigan Statewide Educational Network - your teachers stopped being able to post to powerschool for their students over the new year break. They’ve fed it up to their tech people towards the ISD. Details available off-list.

Any insights are welcome, and as I said, I’d like to understand where the source list is as it starts out working then gradually breaks, so someone is publishing things and they are going out further.

- Jared

2

I’ve come to the conclusion that the geo-ip feed companies don’t give a damn about the legitimacy of their information and don’t research any of it. They just wait for the end user to complain to make the change.

Had one today, in fact.

They’re lame.

-Mike

3

This is a real and growing problem. I have some networks that have experienced lengthy “no service” issues with streaming services such as Disney+ due to this, and it took many customer generated complaints and “NANOG hallway level” type human back channel escalations to actually get it addressed. And it still took months.

It would be Really Nice if the major streaming and other cloud service companies actually had any sort of NOC that was reachable to open tickets and resolve the issue. But that would require employing people with clues.

It is also sad how many orgs need a NANOG posting prompt to get anyone to look at existing tickets on issues that get ignored for weeks or months.

4

I will repeat what I have been saying since the first discussions of the concept of ip geo-location some decades ago…

An IP address is not tied to any of the following:
  Location
  Person

An IP address may be transiently tied to a host. The definition of transient in this case can vary widely from a few seconds to multiple years.
IP Addresses may be tied to an organization (though this is also usually some level of transient).

Trying to pretend otherwise in any useful way is fraught.

Unfortunately, it is not fraught enough. It works well enough often enough that the times it doesn’t work usually don’t impact the people monetizing it.

Owen

5

Even worse, some don’t even bother taking you off a list or correcting their records. In these cases I’ve had great luck once our lawyers get involved, but that really only works for US-based companies.

Pretty sure the last company who used our IP space was just wrecking the internet for fun, took a while to get off of some large blocklists. At least it was an easy business justification to rapidly deploy IPv6…

6

I think sadly the counterbalance item is that there is some insurance underwriter or similar that wants a checkbox saying “yes there is a firewall” or “you do X,Y,Z”.

Or: Sure, I agree with you, and when I’m in Europe or similar and can’t access my (home) government stuff because they just have off-continent blocked is also an issue.

Also: water wet.

What I’m actually looking for isn’t so much a soapbox but to find where the [bad] data is coming from so it can be updated as appropriate. I’m also fine with telling the customer to phone the service/bank/whatnot (which is what I did in other cases and as much as I also personally dislike the centralization of the internet etc) - my customers do seem to really have good experience with a modern service like YoutubeTV (for example) - oh and it does IPv6 too.

If you see this and go back to the original post, I am interested if you have seen that prefix or any IP space within it, and if it comes from a feed or set of aggregated feeds etc, even the name of the company or source/resources there so I can try knocking on the door.

- Jared

7

What I’m actually looking for isn’t so much a soapbox but to find where the [bad] data is coming from so it can be updated as appropriate. I’m also fine with telling the customer to phone the service/bank/whatnot (which is what I did in other cases and as much as I also personally dislike the centralization of the internet etc) - my customers do seem to really have good experience with a modern service like YoutubeTV (for example) - oh and it does IPv6 too.

Tragically, there’s no license necessary to stand up a geolocation service and the only enforcement of quality standards comes from losing business if enough of their clients complain. Tragically, their clients don’t know that they need to complain because their customers don’t know to blame the appropriate service. All they know is that stuff is broken. (Sure, a few know that broken because bad Geo-IP, but we are in the minority).

Since companies don’t generally disclose their Geo-IP source, there’s no ability to coordinate fixing stuff.

If you see this and go back to the original post, I am interested if you have seen that prefix or any IP space within it, and if it comes from a feed or set of aggregated feeds etc, even the name of the company or source/resources there so I can try knocking on the door.

I don’t see it in any of the few block-list feeds that I subscribe to. Best of luck in your search.

I don’t use IP geo-location (for the very same reasons stated in my previous post), so I can’t help you there.

Owen

8

Are you sure it’s really geolocation blocks? Or is it anonymizer and VPN service detection? The geoIP vendors typically sell both since one of anonymizers’ top applications is to evade geolocation. Have customers using peer-to-peer anonymizers wittingly or unwittingly? Customers with malware or other PUPs hosting anonymizer services?

9

I know in the case of one provider it was a geolocation related issue. I don’t know if they fixed it, as I said the customers left that provider so the complaint went away.

There seem to be a few issues happening. If I’m not getting the bot/threat feeds for those places, I’m happy to follow-up with that customer, but some is just flat out things like “This isn’t IP space in US” or the feedback from the customer says the provider places them in Mexico.

As I said, looking for any place that has 23.138.114.0/24 in a feed to be blocked as some of the ISD (intermediate school district) that aggregates tech for several around the area started blocking over the winter break anyone in that /24, can ping from other subnets but not that one *smh*.

I’m a bit grasping at straws, but also looking for any ideas or information that people may have around it. I get some people may update monthly, or take time to get the changes through their systems, but parts of this have been going on now since mid-late September. If it’s going to take 1.5-2 quarters to have the IP space be viable, this is something I’ll be taking up eventually with folks at ARIN - similar to issues with other things that may not be easily fixed, there’s a level of effort that I’m willing to undertake here, but at some point there is a question about if it’s fit for any purpose.

The reality is I expect if I can find where the feed is that has the space flagged, that will likely address this part of the long tail. I would hate to end up doing more NAT-PT/44 due to one or a few vendors with bad data sources.

- Jared

10

Every block I’ve gotten I just went through TheBrothersWisp geo location page and just had them fix their information. This includes virgin and re-issued blocks from ARIN.

I’ve had a couple of random issues like Hulu thinking I’m a VPN, PSN blocking a /24 because a /32 failed his password too many times, and various streaming issues of which I tell customers to complain to the streaming provider because all of the other ones work.

11

Honestly, the only way I’ve found to fix this is completely fill it with subscribers off a BNG and give support a script about what to tell customers.

I’ve had folks literally get the wrong TV channels because we assign unused blocks in Portland Oregon out of our parent large aggrigates and the geo folks have our whois address in the seattle area so give them seattle channels. God forbid these OTT folks just design the product right and use the verified billing zip code on the account or something else that actually is authoritative.

12

This has a simple solution, Jared. It and telecom workers are incredibly rational people, so simply point out their error, display your credentials, advise them of the path they should take, and soon all will be fixed.

:wink:

-----Mike HammettIntelligent Computing SolutionsMidwest Internet ExchangeThe Brothers WISP

13

One would also think that large OTT content providers which publish Android and IOS apps could use the geolocation-permission data gathered from the device, telemetry reported to their own internal systems to gather their own independent data sets on where customers are geographically located, at least as coarse to a specific metro area… And use that to clean up geolocation features where 3rd party IP geolocation datasets don’t match reality.

At the smallest scale of customer count: For instance if they have many dozens or hundreds of subscribers whose devices often sign in from the same /24 block, and in which that block is not known to be cellular carrier/MNO/MVNO IP space, and the devices’ geolocation API data reports they’re in a certain suburb of Portland. Or even if you have something like a smart TV in a house which has no geolocation ability/API exposed but many of the customers’ other devices which do report geolocation API often sign in to the same account from the same residential-last-mile-provider dhcp pool /32 address.

The amount of telemetry data collected off an android or ios devices these days by most consumer apps is quite comprehensive, and as we all known the average person is extremely likely to click “Yes/accept” on any software/interface modal popups, so the majority of the devices will not have geolocation blocked. They already have whole teams of highly paid software developers working on the DRM-specific code in their video streaming apps, so clearly some use of that data is made already.

14

One would also think that large OTT content providers which publish Android and IOS apps could…

You said the magic word ; could.

It’s the natural extension of MBA Math ; If you can pay for something ‘as a service’ , it’s going to be cheaper than paying people to develop it in house. That ‘service’ is usually a reasonably high percentage of ‘good enough’ so as not to really impact your revenue. For larger ‘chunks’ of problems that could be a notable revenue hit , you’ll allocate some resources to work that out, but the smattering of instances here or there, sorry Charlie.