Has anybody used (and been successful at) a bit-torrent-like agent for fast
distribution of LEGAL software (install programs of large-DVD size), across
multiple sites, all over the globe, with bad WAN connectivity? I have read a
couple of references online (e.g. http://torrentfreak.com/university-uses-utorrent-080306/) about such, but I
am a little reluctant to do it in a corporate environment, especially in the
light of potential misuse of such ... unless finding a way to install, use
and remove the P2P agent, all in one shot ... catch 22, sort of (distributing
the P2P agent, that is :)) ...
Stefan
P.S. If inappropriate for this mailing list, I apologize - but the "long fat
pipe" thread gave me the idea to ask here, vs. sysadmin-like lists, as the
potential for network impact is my primary concern.
most of the larger free-nix's do BT downloads on release day(s).
Revision3 distributes their content via BT. There were rumors of
Disney and Apple moving to BT models for their content distribution at
one point as well.
If the tracker isn't accessible from untrusted networks, what's the
concern you have?
> Has anybody used (and been successful at) a bit-torrent-like agent for
fast
> distribution of LEGAL software (install programs of large-DVD size),
across
> multiple sites, all over the globe, with bad WAN connectivity?
<snip>
most of the larger free-nix's do BT downloads on release day(s).
Revision3 distributes their content via BT. There were rumors of
Disney and Apple moving to BT models for their content distribution at
one point as well.
<snip>
I believe World of Warcraft uses Bittorrent to push out updates as well
(Steam may, haven't checked, would make sense though). Something we've been
working with for a client is using Amazon's S3 service to host the tracker,
as S3 will natively handle serving it (both the content and the tracker, you
simply need to append ?torrent to the S3 request).
Has anybody used (and been successful at) a bit-torrent-like agent for fast distribution of LEGAL software (install programs of large-DVD size), across multiple sites, all over the globe, with bad WAN connectivity? I have read a couple of references online (e.g. Dutch University Uses BitTorrent to Update Workstations * TorrentFreak) about such, but I am a little reluctant to do it in a corporate environment, especially in the light of potential misuse of such ... unless finding a way to install, use and remove the P2P agent, all in one shot ... catch 22, sort of (distributing the P2P agent, that is :)) ...
well if the connectivity universally sucks no amount of p2p is going to help... We have some experience with large file distribution in this fashion because of the fedora core DVD iso's and we can say generally at this point that the mirror infrastructure serves more iso's on release day than the torrents do...
that said the p2p client does rule out needing to select a mirror that has free slots during a flash crowd.
<random type="idea from tonight">
If only there was a way for a SP to run a BitTorrent type service for
their clients, subscribing the BT server(s) to known-good (ie, not warez-y)
torrents pre-seeded from trusted sources and then leaving it the hell
alone and not having to continuously dump specific torrent files into
it.
</random>
Automatically leeching and then seeding for long periods is trivial to set up if you can get an RSS feed with torrent enclosures. It is my (highly theoretical, naturally) understanding that many BitTorrent trackers make such feeds available.
However just because you have a fast, on-net seed for particular torrents doesn't mean that your on-net leechers will necessarily pick it up. The behaviour I have observed with BitTorrent is that clients are handed a relatively short list of potential peers by the tracker, and it's quite common for sensible, close, local peers not to be included. My assumption has been that the set of potential peers passed to the client is assembled randomly.
If this behaviour is widespread (i.e. if my observations are valid and my interpretation of those observations reasonable) then the more popular the content, the less likely leechers are to see the seed you want them to see. This relegates your local, on-net, fast seed to be a way of distributing unpopular content (that which is not being seeded by many other people).
There has been at least one presentation at NANOG in the past couple of years which describes the benefit to ISPs of p2p, by virtue of keeping traffic for popular content on-net. From memory, however, that presentation was based on a non-deployed p2p protocol which made more of an effort to help peers find local peers than the clients I described above.
most of the larger free-nix's do BT downloads on release day(s).
Revision3 distributes their content via BT. There were rumors of
Disney and Apple moving to BT models for their content distribution at
one point as well.
<random type="idea from tonight">
If only there was a way for a SP to run a BitTorrent type service for
their clients, subscribing the BT server(s) to known-good (ie, not warez-y)
torrents pre-seeded from trusted sources and then leaving it the hell
alone and not having to continuously dump specific torrent files into
it.
</random>
Ah, if only there was a way for my SP to go and look all over the web and figure out what pages are acceptable for me to browse and block out all of the other stuff like porn and warez and phishing --- and other objectionable content like creationism / evolution [delete whichever is appropriate ], those bastard [insert your least favorite ethnic / religious group here ] and any mention of [insert political party]..... Oh, and anything to do with clowns, they freak me out...
Yes, P2P is not the web, but the general principle still applies -- I don't think that handing over the censorship keys to my ISP is a reasonable solution...
W
The behaviour I have observed with BitTorrent is that clients are
handed a relatively short list of potential peers by the tracker,
and it's quite common for sensible, close, local peers not to be
included. My assumption has been that the set of potential peers
passed to the client is assembled randomly.
I did not check seriously so I cannot confirm or deny but do note that
there are several proposals to improve "peer selection" behind random
sorting or crude measurements with ping on a few hosts. A summary of
existing work is on the ALTO Web site
<http://alto.tilab.com/resources.html>\.
ALTO will have a BoF session at the next IETF in Dublin, so we may see
one day a standard protocol for peer selection.
I dunno, an RSS type feed of bittorrent files which can be subscribed
to would be useful. You could then just subscribe to certain content,
implictly trusting that they're publishing sensible content (and filtering
the content you seed your torrent with using tags or some such.)
You could then subscribe to various projects' downloads and mirror
appropriately.
Of course, this could already be being done; I haven't any idea.
There was a product around that would keep track of torrents and fudge the tracker responses to direct you to on-net peers where possible. Not sure what it's called. Inline box thing, much like Sandvine, Allot, etc. I imagine you could either inject the details of a local seed you're running, or keep track of on-net users and inject those.
From a tracker software point of view, it would be fairly trivial to weight peer lists to prefer peers within the same ASN I imagine.
Perhaps that could be turned in to same country, or what not. Better, combine it with some kind of rough AS adjacency graph and <insert algorithm here> and viola.
Is there any data available that would let that happen easily? Obviously routing tables for the ASN/IP mapping, but what about rough ASN adjacency? It doesn't really need to be updated that often - even CAIDA's yearly data that they use to make their pretty pictures could work OK.
Seems like win/win/win - linux distribution vendors can pride themselves on how much faster their torrents run, end users get better speeds for their torrents, networks move less traffic off-net.
.. this is the part where someone bustles off and makes it go.
Has anybody used (and been successful at) a bit-torrent-like agent for fast
distribution of LEGAL software (install programs of large-DVD size), across
multiple sites, all over the globe, with bad WAN connectivity? I have read a
couple of references online (e.g. Dutch University Uses BitTorrent to Update Workstations * TorrentFreak) about such, but I
am a little reluctant to do it in a corporate environment, especially in the
light of potential misuse of such ... unless finding a way to install, use
and remove the P2P agent, all in one shot ... catch 22, sort of (distributing
the P2P agent, that is :)) ...
revision3.com
And we saw how it worked out for Revision3.com. MediaDefender considered them illegal and launched a Denial of Service attack against them over Memorial Day weekend. P2P is considered illegal and wrong by people with lots of money and that makes it hard to use for legitimate services. Because MediaDefender is backed by the RIAA and similar organizations they seem to be immune to prosecution. However, if *I* did the same thing then I know I would be locked up right now.
<random type="idea from tonight">
If only there was a way for a SP to run a BitTorrent type service for
their clients, subscribing the BT server(s) to known-good (ie, not warez-y)
torrents pre-seeded from trusted sources and then leaving it the hell
alone and not having to continuously dump specific torrent files into
it.
</random>
Modifying the P2P protocols might help find good seeds, etc. However,
I always like to take this thought a bit further and combine it
with a particular Network Neutrality "solution."
Imagine a world where "Net Neutral" means that you have a neutral
layer 2 architecture and you're free to choose the layer 3 provider.
(Model it on US West/Qwest's original DSL product.)
Then, sprinkle in a *bunch* of ISPs that must have transparent
layer 3 policies. Let them block/fold/mutilate/spindle/synthesize
packets at their whim -- as long as they *tell* the customer
what they're going to do.
In the end, I can see ISPs that do *nothing* to your traffic, and
charge what we would call "normal" pricing. There would be cut-rate
ISPs that would promise best-effort, but will throttle if they have
congestion issues.
If you're an ISP, you might even try to cut a deal with the RIAA
and/or MPAA so your customers have *fast* access to legitimate
content. As a content provider, I would look seriously into
subsidizing the access costs so that I could capture an
end user...
Guess I picked the wrong week to stop sniffing glue...
There was a product around that would keep track of torrents and fudge the tracker responses to direct you to on-net peers where possible. Not sure what it's called. Inline box thing, much like Sandvine, Allot, etc. I imagine you could either inject the details of a local seed you're running, or keep track of on-net users and inject those.
Out of curiosity, how many SPs out there have local Akamai servers on their network? I inquired about it last Fall and our average bandwidth to Akamai wasn't enough at the time to warrant placing hardware on our site, from their perspective anyway. The bandwidth though accounted for roughly 1/10th of our overall bandwidth. I wonder what it would be today. Our Internet bandwidth is just over 4x what it was last Fall.
