Hi everyone,
In order to sanitize historical BGP data I would like to use historical Bogon lists. The CIDR report generates those lists on a daily basis (e.g. https://www.cidr-report.org/bogons/freespace-dec.txt for prefixes) but, as far as I know, it does not keep a history of those files - it only holds the most up-to-date file. Does anybody know of a repository that contains such bogon lists for historical data, or, did anybody continiously fecthed and saved CIDR report's bogon lists?
Best regards,
Lars
Lars,
Archive.org has snapshots going back several year. Just feed in the URL you posted, ad you’ll get a history that lets you download each version of the list that archive.org noticed changed. In my experience, that is pretty comprehensive.
-mel beckman
Hi Mel,
I already checked Archive.org - it holds two previous copies.
>> lets you download each version of the list that archive.org noticed changed
According to Archive.org's own Note this seems to be inaccurate:
This calendar view maps the number of times https://www.cidr-report.org/bogons/freespace-dec.txt was crawled by the Wayback Machine, not how many times the site was actually updated.
, or am I missing something?
Best regards,
Lars
My understanding is that the calendar, which in this case showed a dozen or so copies, lists only crawled instances that had changed since The previous crawl. Well it certainly true that time elapses between each crawl, so it's possible that some changes could be missed, my understanding is that the Internet the scroll frequently enough that changes would be detected at least every month or so.
I think it's possible that the bogin list just doesn't change that frequently.
-mel via cell
which in this case showed a dozen or so copies
Yeah sorry, I should have said that I was only looking into the time frame 2017 till now.
>> I think it's possible that the bogin list just doesn't change that frequently.
I think I aggree that it is unlikely to change frequently, however, there is not a single snapshot in archive.org for 2018.
Best regards,
Lars
Hi everyone,
Hi,
In order to sanitize historical BGP data I would like to use historical Bogon lists. The CIDR report generates those lists on a daily basis (e.g. https://www.cidr-report.org/bogons/freespace-dec.txt for prefixes) but, as far as I know, it does not keep a history of those files - it only holds the most up-to-date file. Does anybody know of a repository that contains such bogon lists for historical data, or, did anybody continiously fecthed and saved CIDR report's bogon lists?
Have you tried reaching out to Team Cymru? They provide bogon feeds in a number of different formats. I would be surprised if they don't have the ability to provide historical versions of the bogon list. - It's probably worth asking. I have found them to be very responsive and quite helpful.
Best regards,
Good luck.
There does seem to be some historical data from the CIDR report here:
http://thyme.rand.apnic.net/
Might have a few historical freespace-dec.txt files around from building periodic IP/country maps of all gTLD websites.
Regards...jmcc
Dear team,
Have you tried reaching out to Team Cymru? They provide bogon
feeds in a number of different formats. I would be surprised if
they don't have the ability to provide historical versions of the
bogon list. - It's probably worth asking. I have found them to
be very responsive and quite helpful.
Thank you for the kind words, Grant! Sadly, we don't keep an archive
of prior bogon lists, so we can't help here. Sorry!
Be well,
Rob.
- --
Rabbi Rob Thomas Team Cymru
"It is easy to believe in freedom of speech for those with whom we
agree." - Leo McKern
I wonder if there’s value in having the lists that Team Cymru generates auto pushed to a public Git repo. Covers historical changes for folks who want that, and also provides a more modern ingestion method for automation around that info. (Not that I’m hating on wget / curl … 
)
Dear Tom,
I wonder if there's value in having the lists that Team Cymru
generates auto pushed to a public Git repo. Covers historical
changes for folks who want that, and also provides a more modern
ingestion method for automation around that info. (Not that I'm
hating on wget / curl ... 
)
We'd be happy to make that happen, if folks are keen. We're fine with
Git, as we use it regularly.
Be well!
Rob.
- --
Rabbi Rob Thomas Team Cymru
"It is easy to believe in freedom of speech for those with whom we
agree." - Leo McKern
We'd be happy to make that happen, if folks are keen.
I'm keen.
We're fine with Git, as we use it regularly.
How big is the bogon list? (I can't look at the moment.) Is it something that you can store the entire list periodically, or as it changes?
I'm wondering about long term space consumption. Particularly if I'd like to have a local copy. }
Hello Rob,
IMHO another option is to set up a local BGP daemon (e.g. Quagga, ICE) peering with your bogon route server project and dump bogon information regularly in MRT like route collectors are doing. So folks can select the RIB snapshot they prefer in time and (in case) study the evolution of bogons via update collections in MRT files. The result is something like this http://data.ris.ripe.net/rrc00/ but containing only bogon information. We did it for our studies last months, so I'd be more than happy to help if you decide to follow this path!
Alessandro Improta
IIT-CNR, Isolario project
Hi all,
CAIDA has been collecting Team Cymru’s bogon list from 2013-09-18 to 2018-03-23. Unfortunately we just noticed the script hasn’t been working since then but we just fixed it and restarted it.
We’re happy to share the data as long as Team Cymru’s is ok with it.
Cheers,
Alberto
PS
I think Rabbi+Randy also helped us in the past by recovering diffs for 2010-2013 data. Looking for the files. Stay tuned.
I think Git would be the perfect solution…would definitely appreciate it.
I think Alessandro Isolario Project may be of help completing the missing data where you fell short.
M.
Yes, we got data from Jan 1st, 2018 to Oct 10th, 2018. Happy to share as long as Team Cymru’s is ok with it too!
Ale
Dear team,
The fine folks at CAIDA have maintained a repository of the bogon
lists we've produced, going back to 18 SEP 2013.
http://data.caida.org/datasets/bogon/
They are integrating the 2018 data provided by Alessandro Improta, and
will have that in place after the new year.
Will that work for folks, or would a git repository be useful in
different ways?
My thanks to Paul Hick, CAIDA, and Alessandro Improta for the
assistance here!
Thank you!
Rob.
- --
Rabbi Rob Thomas Team Cymru
"It is easy to believe in freedom of speech for those with whom we
agree." - Leo McKern