Geoip lookup

What's the best way to find the networks in a country? I was thinking of
writing some perl with Net::Whois::ARIN or some such module and loop
through the block. But I think I'll have to be smarter than just a simple
loop not to get blocked and I figure I'm not the first to want to do this.

I've noticed some paid databases out there. They don't cost much but are
they even worth what they charge? Ie, countryipblocks.net doesn't list
quite a few addresses from a country I've looked at blocking. Isn't this
information free from the different *NICs anyway?

This is probably two questions: a program that smartly looks for country's
blocks in a block and are GeoIP services worth anything?

If you are looking for registration data, try looking in one or more of

  ftp://ftp.apnic.net/public/apnic/stats/apnic/
  ftp://ftp.ripe.net/ripe/dbase/
  ftp://ftp.lacnic.net/pub/stats/lacnic/
  ftp://ftp.afrinic.net/stats/afrinic/
  ftp://ftp.arin.net/pub/stats/arin/

(poke around and see what you can find; I didn't spend much time trying, but several/all of the RIRs seem to mirror data from all the others)

Note that "networks in a country" is a funny phrase. The sets

- address space assigned to all organisations located in country X
- routes visible in country X (from some viewpoint)
- all addresses assigned to devices physically located within country X
- routes that are considered "in-country" in places where billing is aligned with the necessity to traverse a long bit of wet glass

are frequently incongruent. If this matters, you might want to consider a more detailed specification of "networks in a country".

Joe

What's the best way to find the networks in a country? I was thinking of
writing some perl with Net::Whois::ARIN or some such module and loop
through the block. But I think I'll have to be smarter than just a simple
loop not to get blocked and I figure I'm not the first to want to do this.

If you are looking for registration data, try looking in one or more of

  ftp://ftp.apnic.net/public/apnic/stats/apnic/
  ftp://ftp.ripe.net/ripe/dbase/
  ftp://ftp.lacnic.net/pub/stats/lacnic/
  ftp://ftp.afrinic.net/stats/afrinic/
  ftp://ftp.arin.net/pub/stats/arin/

(poke around and see what you can find; I didn't spend much time trying, but several/all of the RIRs seem to mirror data from all the others)

Thanks

Note that "networks in a country" is a funny phrase. The sets

- address space assigned to all organisations located in country X
- routes visible in country X (from some viewpoint)
- all addresses assigned to devices physically located within country X
- routes that are considered "in-country" in places where billing is aligned with the necessity to traverse a long bit of wet glass

are frequently incongruent. If this matters, you might want to consider a more detailed specification of "networks in a country".

I had somewhat considered the second and the fourth point. I assumed
by using whois data, I am getting the second of those options and that
was good enough. If there's a way to (somewhat easily) implement the
third option, I'm all ears.

It looks you're right and everyone does have the same data in
historical format. Looks like RIPE has everything compiled into what
is current. So if a block hasn't changed for 10 years, it'll be in the
RIPE dataset vs with the others, I'd have to write something to
overlay the data through out time to get current?

I've used the MaxMind Lite geo-ip database plus some perl modules and a BGP
table to get something fairly close. Anything in the BGP table that was
larger than a /20 I split into /20's. For my use case, this was close
enough. I then grabbed 30 or so IP's within the range and geo-ip mapped
them. You can then apply some algebra and get a general idea of where
things are or are not.

Things I used:
http://search.cpan.org/~plonka/Net-Patricia-1.014/Patricia.pm - For
ip/prefix/lat-lon mapping
http://search.cpan.org/~borisz/Geo-IP-1.41/lib/Geo/IP.pm - For Geo-IP
lat/lon data
http://dev.maxmind.com/geoip/legacy/geolite - Maxmind's city database
http://data.caida.org/datasets/routing/routeviews-prefix2as/ - for BGP
prefix/mask + src ASN info

Good luck!

--chip

Could be. You've looked at this more than I have, now -- I was mainly trying to point out that bulk data retrieval is a possible option so you could avoid whois-hammering :slight_smile:

Joe

Actually, I can't find anything better, so I think i'm going to query
the bottom of ranges like so:
% dig +short 0.0.66.77.origin.asn.cymru.com TXT
"16245 | 77.66.0.0/17 | DK | ripencc | 2007-01-24"
% dig +short 0.0.65.77.origin.asn.cymru.com TXT
"13110 | 77.65.0.0/17 | PL | ripencc | 2007-01-17"

According to their web site, they won't block me if I don't do
anything stupid "If you are planning on implementing the use of this
service in any software, application, or device PLEASE let us know in
advance. We would like to adequately plan for capacity and make sure
that we can adequately handle the load. If at all possible, PLEASE use
the DNS based service since it is faster and more efficient,
particularly for larger deployments of individual IP based queries."
http://www.team-cymru.org/Services/ip-to-asn.html#dns
and use this https://metacpan.org/module/MRSAM/Net-CIDR-0.17/CIDR.pm
to find my upper ranges.

So, I'm pretty much thinking about whois-hammering still :frowning:

Also, I just picked those IPs at random (ie, start at one end of the
number, hit twice, dot, next number) nothing particularly interresting
about whoever that is AFAIK.

This may be just a case of getting what you pay for, but Maxmind marks
entire netblocks as proxies, puts 'em in the wrong country, and
ignores repeated efforts by the registrant of the address space to set
the record straight. The problem comes when people actually do stuff
with the information, like block access to legitimate web sites
because the're in "proxy space" and therefore assumed to be bad guys
(believe it or not this practice is widespread by well-intentioned but
clueless folk). Caveat utilitor.

-r

chip <chip.gwyn@gmail.com> writes:

The whole idea of Geoip is flawed. IP dosen't reside in countries, they
are routable adresses that can reside everywhere, I guess soon on mars
even.

Med vänlig hälsning
Andreas Larsen

IP-Only Telecommunication AB| Postadress: 753 81 UPPSALA | Besöksadress:
S:t Persgatan 6, Uppsala |
Telefon: +46 (0)18 843 10 00 | Direkt: +46 (0)18 843 10 56
www.ip-only.se

The whole idea of Geoip is flawed.

Sure, but pragmatically, it's an 80% solution.

IP dosen't reside in countries,

True, according to (at least some of) the RIRs they reside in regions...

Regards,
-drc

If we continue to support and build tools around this geolocation based
ip-dravel, we give people a false notion that this is something we should
do.
Identify users with some other means that Geoip

Couple of things comes to mind.

* normal postage mail that they have to collect at their home and send
back confirming that they are indeed in the country from where their IP is
* Passports scanned.
* Fingerprinting

Or just get rid of the whole idea and realize that the internet is global
and reaches everywhere no matter what your IP currently is.

Med vänlig hälsning
Andreas Larsen

IP-Only Telecommunication AB| Postadress: 753 81 UPPSALA | Besöksadress:
S:t Persgatan 6, Uppsala |
Telefon: +46 (0)18 843 10 00 | Direkt: +46 (0)18 843 10 56
www.ip-only.se

Really? Which ones? I thought they were only issued to organizations that had operations in regions.

Owen

Just because I have operations in one region does not preclude me from having operations
  in other regions. YMMV of course.

/bill

That was exactly my point, Bill... If you have operations in RIPE and ARIN regions, it is entirely possible for you to obtain addresses from RIPE or ARIN and use them in both locations, or, obtain addresses from both RIPE and ARIN and use them in their respective regions, or mix and match in just about any imaginable way. Thus, IP addresses don't reside in regions, either. They are merely issued somewhat regionally.

Owen

>
> Just because I have operations in one region does not preclude me
from having operations
> in other regions. YMMV of course.
>
> /bill

That was exactly my point, Bill... If you have operations in RIPE and ARIN
regions, it is entirely possible for you to obtain addresses from RIPE or ARIN
and use them in both locations, or, obtain addresses from both RIPE and
ARIN and use them in their respective regions, or mix and match in just about
any imaginable way. Thus, IP addresses don't reside in regions, either. They
are merely issued somewhat regionally.

In theory Maxmind is quite accurate. From 1 x /20 that we own we tag different space with the country: flag in the RIPE db. Maxmind picks this up after approx 30 days and says it's in Country X vrs country Y.

e.g.

$ geoiplookup 81.17.247.64
GeoIP Country Edition: US, United States
$ geoiplookup 81.17.247.1
GeoIP Country Edition: IE, Ireland

Obviously the RIPE db structure makes this simple. As for other RIRs it's not as easy. Like someone else said, it's going to be an 80% solution and its really down to good administration from a network operator point of view. i.e. if you route some of your RIPE space in ARIN territory you should specify the country. There's numerous reasons for this but it's just good practice IMO.

Pk

Correct.
But the fact remains that a lot of services assume geolocation works and
do so in terms of restricting access to their content (oftent due to
legacy content rights that require geolocation).

One extreme example. A sports equipment retailer operates under a
different banner in the province of Qu�bec than in the rest of Canada.
They geolocate the user's province and prevent qu�beckers from accessing
the "rest of canada" web site.

So residents of ontario who subscribe to an ISP based in Qu�bec are
blocked from the web site because that web site thinks they are based in
Qu�bec.

The problem is with many web designers and managers who do not
understand geolocation and the ISP business and how they are structured.

In the case of the sports equipment chain. there is no real need to
geoblock. (perhaps to prevent Qu�beckers from seeing the prices in the
rest of canada ?)

But in the case of entertainment, rights to programs are purchased with
strict geolocation requirements. One example are pay TV channels TMN
(Astral) and Movie Central (Shaw). The first has eastern canada, the
later has western Canada.

an IPTV BDU (regulated "cable" carrier (aka: cable competitor) must
therefore ensure that a customer to whom it delivers the IPTV feed for
"TMN" is located in the region for which TMN has rights. Same for all
channels. And there is also pesky channel substitution requirements
rhat are based on your location. In Canada, we are not allowed to watch
a program on a US channel if a local TV channel carries the same program
at same time.

The better solution is to do like satellite BDUs do: billing address.
But some web based systems ignore the unreliable geolocation services
and use them to geolocate their customers.

It is probably the fault of geolocation services which misrepresent the
accuracy of their data. But if you can't beat them, you might as well
join them, and that may mean separate IP blocks for different
provinces/states and separate registrations so geoocation companies can
at least get province/state right.

It will get much worse if governments start to tax purchases/services
based on gelocation.

I knew this would come up. Actually I'm surprised and glad it waited until
I got a solution first.

I'll address a few points:
- this is mainly to stop stupid things from sending packets from countries
we will probably never want to do business with (I'm looking mainly at that
big country under APNIC).
- I'd prefer a solution that blocks all traffic that is routed through
those countries so that they could never see data from us (and when
Jin-rong has a configuration mess up and rerouts ~10% of traffic through
them for a half hour, I don't see any of that traffic). Since I have no
idea how one would go about doing this, just blocking traffic from IP
addresses registered in certain countries is good enough.
- it is well known (I think everyone on this list at least) that you can
evade geographic placement of your origin by tunneling. Given this, I fail
to see the point in bringing up that "GeoIP" doesn't work. Also, if it
doesn't work, why do content providers, CDNs, google, and streaming
services rely on it as part of their business model? The sad truth of the
mater is it does work and surprisingly well. We just don't like it because
it's brittle and a user can fool us (I know Akami and the like look at trip
time and the like because they know there are issues). Given all of this,
how often is looking at the country an IP address originates from via what
is listed for the particular ASN actually fiction?

Again, the input was invaluable for getting me where I wanted to be so
thanks again.

While the Internet is global and reaches everywhere, the same is not
true about most businesses and governments... As a result, there are
many use cases that we may not like, but are seen as basic requirements
by those organizations. Examples include laws and business contracts
that require different behavior depending on the location of the user,
and from the view of these organizations, the Internet almost gives the
impression of shoddy workmanship to omit such an obvious capability.
Luckily, many organizations did come up with workarounds, and the lack
of a 100% reliable solution did not prevent them from distributing
content (software, music, movies, articles, etc.) that they only had
rights to do so in a particular region.

If the approximate geolocation approaches had not been used, we'd
would not have had the region-restricted experimentation in content
distribution that underlies quite a bit of the industry even today.

One can argue that regionally-based business models should be changed,
but the fact is that the not-quite-reliable geolocation services are
actually has been pretty important in enabling traditional content in
making it onto the Internet. (It is left as a exercise for the reader
as to whether more highly reliable geolocation would meaningfully help
the situation, or simply enable its use in non-commercial contexts to
the detriment of the global user community.)

/John

Disclaimer: My views alone (& for folks who wish to filter this email
based on my geolocation, it is presently Northern Virginia USA :wink:

I replied privately to Owen, but might as well share:

That was exactly my point, Bill... If you have operations in RIPE and ARIN regions, it is entirely possible for you to obtain addresses from RIPE or ARIN and use them in both locations, or, obtain addresses from both RIPE and ARIN and use them in their respective regions, or mix and match in just about any imaginable way. Thus, IP addresses don't reside in regions, either. They are merely issued somewhat regionally.

Correct.
But the fact remains that a lot of services assume geolocation works and
do so in terms of restricting access to their content (oftent due to
legacy content rights that require geolocation).

The fact remains that a certain percentage of the population robs banks.

Neither is a particularly good thing, IMHO.

One extreme example. A sports equipment retailer operates under a
different banner in the province of Québec than in the rest of Canada.
They geolocate the user's province and prevent québeckers from accessing
the "rest of canada" web site.

And the quebeckers that care use a tunnel to get an address that doesn't
geolocate to quebec.

So residents of ontario who subscribe to an ISP based in Québec are
blocked from the web site because that web site thinks they are based in
Québec.

Which goes to prove my point wrt. bank robbery.

The problem is with many web designers and managers who do not
understand geolocation and the ISP business and how they are structured.

So called experts who remain rather ignorant in their field of "expertise" are a problem across a wide variety of fields. The internet is not unique in this regard and geolocation is just one aspect of this problem on the internet.

In the case of the sports equipment chain. there is no real need to
geoblock. (perhaps to prevent Québeckers from seeing the prices in the
rest of canada ?)

Yep... And even if there were a reason, geoblocking is a joke anyway because it is trivially subverted by anyone who cares and more of a problem for people who should have access but their IP doesn't match their actual geography.

But in the case of entertainment, rights to programs are purchased with
strict geolocation requirements. One example are pay TV channels TMN
(Astral) and Movie Central (Shaw). The first has eastern canada, the
later has western Canada.

But IP geolocation doesn't help in either of these cases. Those wanting to subvert the programming restrictions simply use a tunnel to do so. On the other hand, a customer who lives near the boundary and gets his internet service from the "wrong side" of the boundary has access to the service from the wrong geography and not the correct geography.

an IPTV BDU (regulated "cable" carrier (aka: cable competitor) must
therefore ensure that a customer to whom it delivers the IPTV feed for
"TMN" is located in the region for which TMN has rights. Same for all
channels. And there is also pesky channel substitution requirements
rhat are based on your location. In Canada, we are not allowed to watch
a program on a US channel if a local TV channel carries the same program
at same time.

And geolocation by IP doesn't actually work to enforce any of these restrictions because tunnels easily circumvent it for those that want to circumvent it. OTOH, it also breaks the process for those who happen to be victims of unfortunate mismatches between topology and geography.

Where the IPTV provider is also the ISP, this isn't really much of a problem, but in that case, geo IP is kind of redundant.

Where the IPTV provider is not the ISP, it gets very strange very quickly.

The better solution is to do like satellite BDUs do: billing address.
But some web based systems ignore the unreliable geolocation services
and use them to geolocate their customers.

Yep... Again, see above comments about ignorance and bank robbery.

It is probably the fault of geolocation services which misrepresent the
accuracy of their data. But if you can't beat them, you might as well
join them, and that may mean separate IP blocks for different
provinces/states and separate registrations so geoocation companies can
at least get province/state right.

Why would I want to help them? I'd much rather give my customers the option of where they want to pretend to be. If I were running a provider that crossed such regional boundaries, I'd likely offer a service (for a fee) where a customer could tunnel through whichever region got them access to the content they wanted at any given time.

It will get much worse if governments start to tax purchases/services
based on gelocation.

ROFLMAO... Indeed... That will likely lead to some very interesting lawsuits and consumer complaints about invalid taxation due to inaccuracies in the geolocation database.

Owen