Starting a greenfield(ish) small (10k subs?) multihomed (two ASN) , dual stacked, wireless ISP - i can haz advice?

Mailman List Mirror (Read Only)
1

Hey everybody,

So all this talk about monopolies, small ISPs vs the big bad netflix , muni fiber etc etc has been interesting. Lots and lots of talk, lots of interesting links etc.

I'm an action/results oriented individual, and have been working on actually building out a grassroots ISP, instead of just talking about it. :slight_smile:

Over the past year or so, I've been involved with an effort to launch a community ISP in the Kansas City MO area. It's got several towers up now and a decent amount of users. It's been funded by the community that it serves. Feel free to ask any questions you have about the details. It's an open network in all aspects (design, business model etc). It is intentionally designed/operated in such a way that all aspects can be disclosed.

We are now ready to take the next step and obtain an ASN and v6 space (also looks like we can get a /24 of v4 space as well).

What are the things that we should do before we get those resources? What should we do immediately after? What books/rfc/bcp should we be most familiar with?

As is typical of many small outfits, we have an incredibly high degree of software skill, and a limited budget which goes entirely to hardware.

This is a greenfield network. We've got Ubiquiti gear for the backbone. Running a mix of QMP routers with BMX6 as the IGP linked over AirOS l2 bridge "pseudowires". We'll be homed to two AS upstreams. Using pfSense as the WAN edge routers.

From all my reading of the list, it seems like key things to do in this scenario:

1) Have full flow telemetry at all points to help with (D)DOS mitigation.
2) Do CGN in pools (so perhaps ~500 to 1k users behind each IP)?
3) Provision a /56 of v6 space to each end user. I was thinking of having the CPE with CeroWRT and be multi SSID with a /64 per. I'm interested in folks thoughts on this?
4) Upsell a public v4 address if someone requires it
5) Of course implement bcp38

I'm mostly interested in technical feedback. Business model etc type feedback is welcome as well, but not the primary purpose of this message. :slight_smile:

Thanks!

Charles Wyble
CTO Free Network Foundation

2

Sorry, no feedback from me.. I have couple of questions though, how much licensing do you need to go through, to actually start a WISP?
Also, Kansas.. Are you concerned that you’ll have to compete with Google Fiber at some point?

3

FCC licensing? No licenses as long as you operate in unlicensed bands (ie,
900mhz/2.4ghz/5).

4

I used to correspond with a man out of Hays, Kansas who started a WISP using silos--last I heard he had gotten big enough that somebody bough him out.

Sorry, age-related memory rot denies me his name--there are probably lurkers here who know of him.

5

FCC licensing? No licenses as long as you operate in unlicensed
bands (ie, 900mhz/2.4ghz/5).

Yes. This is correct. Also no licensing needed for 24ghz. We are rolling out a dual uplink 24ghz AirFiber back bone in the next couple of weeks.

The FNF has obtained a 3.65ghz license and that's come in very handy in some of the very noisy parts of our footprint.

Sorry, no feedback from me.. I have couple of questions though, how
much licensing do you need to go through, to actually start a WISP?

Well. I'd recommend being incorporated. Which isn't licensing per se. I'd also recommend being bonded/insured. Just good general business practices.

Also, Kansas.. Are you concerned that you’ll have to compete with
Google Fiber at some point?

Not really. We are serving areas that Google Fiber has decided to not service.

6

charles@thefnf.org wrote the following on 7/23/2014 11:58 AM:

This is a greenfield network. We've got Ubiquiti gear for the backbone. Running a mix of QMP routers with BMX6 as the IGP linked over AirOS l2 bridge "pseudowires". We'll be homed to two AS upstreams. Using pfSense as the WAN edge routers.

From all my reading of the list, it seems like key things to do in this scenario:

1) Have full flow telemetry at all points to help with (D)DOS mitigation.
2) Do CGN in pools (so perhaps ~500 to 1k users behind each IP)?
3) Provision a /56 of v6 space to each end user. I was thinking of having the CPE with CeroWRT and be multi SSID with a /64 per. I'm interested in folks thoughts on this?
4) Upsell a public v4 address if someone requires it
5) Of course implement bcp38

I'm mostly interested in technical feedback. Business model etc type feedback is welcome as well, but not the primary purpose of this message. :slight_smile:

Charles, it sounds like you've got a lot of the technical items on your radar.

I highly recommend pfsense for a firewall (been using pfsense and m0n0wall for years), but do have some concerns about using it at scale for (several) thousands of users. Most of this relates to NAT/State tracking, some of it hardware related, some of it software. If possible, I would suggest you obtain a routable IP address per user and avoid the pitfalls of NAT (I know at some point this may become expensive). If you start with IPv6 from day 1 you are in a lot better place to encourage customers to upgrade to IPv6 capable gear. I would also suggest using stateless firewall rules and routing on your WAN devices. This should simplify the functions performed by these boxes to reduce the need to troubleshoot, apply updates, etc (resulting in better availability). I haven't used pfsense in an ISP WAN router capacity, and personally feel a router from Cisco, MikroTik, or Ubiquiti's EdgeOS devices, etc may be more appropriate in this role. If you've automatically discounted big name gear due to upfront costs, you might consider buying from a used equipment reseller (I can recommend a few, if needed).

If you do need to use NAT, I feel like 500+ users sharing a single NAT IP will result in poor quality of service and more admin overhead. My gut feeling is that <50 may be more appropriate, depending on the quality of service you want to provide. This provides some headroom if one user makes many connections (p2p, virus infection, DoS attack) and also lessens the number of subs you need to look at in cases of abuse that are reported as an IP/port. Individual pfsense servers in a cluster may provide scalable CGN services. I'm not sure how you want to handle logging of all that data, but pfsense should allow you to define rules that allow stateless auditing (ip 1.2.3.4, ports 1000-2000 always NAT to sub A). The XML config file or possibly the shell is probably the easiest way to define such rulesets at scale.

I didn't see it mentioned, where (and to whom) are you multihoming? Do you have a good working relationship with these folks (cell phone, email contacts that reach someone promptly)? Will you be considered a facilities based ISP (and subject to CALEA or other regulation)?

--Blake

7

Hey,

I have started this kind of organization with my friends about 11 years ago
(oh time flies) in Czech Republic in my small hometown. Nowadays it has
around 3000 users. Each user has to pay small membership fee about 8EUR.
Everyone shares 1GBit connectivity to the Internet.

We have started with 4 people on old PCs running Linux (mainly Slackware)
connected via 2.4GHz and backbone running on 2.4 as well with 64Kbit/s
connectivity to the Internet. We went afterwards from 5GHz and 10GHz
backbone, laser backbone and since last year or so ended up with fibres.

In last few years we are forcing users to move towards 5GHz as 2.4GHz is
very noisy in our area and therefore very hard to manage. In part of the
town with apartments we have connected the whole buildings with fibre.

We have seen our organization moving from fully volunteered workforce
towards volunteering organization with two full-timers to give support to
end users as it was unsustainable to support so many end users by
volunteers.

Since we have started as high school students and nowadays half of us has
own families or live/work out of the town it reminds me another thing.
Always share your knowledge with other volunteers and look for other young
and smart people as one day you won't be able to do it (for any reason:)
Our experience is that it is very hard to find other young people to
continue on our work.

Our 'business model' was always volunteering organization which supports
other non profit or non gov organizations. We also supported OSS projects
and help to build hockey pitch in our town et cetera et cetera:)

My last recommendation would be to just have fun. We have seen many points
in our history where we didn't have time or energy to do something but it
has all paid off - it helps us with our careers and most importantly to
other people.

I'm sorry if you were expecting any technical advice;)

Good luck and if you have any other (even technical) questions please let
know:)

Matyas

8

I highly recommend pfsense for a firewall (been using pfsense and
m0n0wall for years), but do have some concerns about using it at scale
for (several) thousands of users.

So far it's gone fairly well for the existing subscriber base. The current service footprint is ~1k homes. I think it's running on a Dell Poweredge ~29xxish , don't know for sure.

  Most of this relates to NAT/State

tracking, some of it hardware related, some of it software.

Right.

  If

possible, I would suggest you obtain a routable IP address per user
and avoid the pitfalls of NAT (I know at some point this may become
expensive).

Exactly.

If you start with IPv6 from day 1 you are in a lot better

place to encourage customers to upgrade to IPv6 capable gear.

Yes. We are doing v6 to every end user CPE. Absolutely. It will be there, be turned on and we hope to send all netflix/facebook/google etc traffic over v6. The v4 will be CGN. (We think we can only get a /24 reasonably).

@Comcast v6 team (and really anyone who has a large dualstack network (*waves* at Owen),

So you guys have v6 turned up. You passed 1tb of traffic. Didn't comcast also write some floss code for CGN? So presumably you'll have to start doing CGN soon.

Thoughts on long tail v4 only internet being seriously degraded by large scale CGN? (Maybe that's a new thread?) If the major properties are v6, shouldn't that be enough to keep the support costs down? (My friends in the MMORPG "cloud gaming" space tell me that my approach could wreak havoc with many game engines).

Thoughts on what happens when you've got v6 at your door and v4 at your CO? Who is running a network like this today (I imagine most small ISPs will be in that boat soon)?

(And also, what's up with people complaining about ARIN fees?). The air fiber radios FNF is installing in KC cost 5k capex. So enough already about a ONE TIME 1k fee and get your v6 space! (I agree with the posters who said if you can't afford the arin fee, GET OUT OF BUSINESS).

  I would

also suggest using stateless firewall rules and routing on your WAN
devices.

That does seem to be the common wisdom. I'm actually not 100% sure what we've got in line. It's OpenWRT based all around, so I'm sure IPTABLES (and maybe even some ebtables).

  This should simplify the functions performed by these boxes

to reduce the need to troubleshoot, apply updates, etc (resulting in
better availability).

Yeah. Of course.

  I haven't used pfsense in an ISP WAN router

capacity, and personally feel a router from Cisco, MikroTik, or
Ubiquiti's EdgeOS devices, etc may be more appropriate in this role.

I've got pretty much every Cisco router/switch in our lab, and an EdgeRouter.

What mikrotik should I evaluate?

Our lab : https://commons.thefnf.org/index.php/FNF_Lab

If you've automatically discounted big name gear due to upfront costs,
you might consider buying from a used equipment reseller (I can
recommend a few, if needed).

No. It's mostly for the customization/scripting etc. "SDN" and all that jazz. :wink:

If you do need to use NAT, I feel like 500+ users sharing a single NAT
IP will result in poor quality of service and more admin overhead.

Quite possibly. However if it's just for long tail v4 only sites, I wonder how much it matters?

  My

gut feeling is that <50 may be more appropriate, depending on the
quality of service you want to provide. This provides some headroom if
one user makes many connections (p2p, virus infection, DoS attack) and
also lessens the number of subs you need to look at in cases of abuse
that are reported as an IP/port. Individual pfsense servers in a
cluster may provide scalable CGN services. I'm not sure how you want
to handle logging of all that data, but pfsense should allow you to
define rules that allow stateless auditing (ip 1.2.3.4, ports
1000-2000 always NAT to sub A). The XML config file or possibly the
shell is probably the easiest way to define such rulesets at scale.

Right right. I'm very familiar with the XML config and CLI. We've gotten to know pfSense well in our AutoTunnel (RADIUS) work. We patched (and released back to upstream) hostapd and other bits to actually correctly implement the RFC :smiley:

So we've got a solution that is multi gateway. So based on the login creds you use, you get dropped into an appropriate vlan / BMX tunnel and get routed out the appropriate gateway.

I didn't see it mentioned, where (and to whom) are you multihoming?

Kansas City Kansas. Joesdatacenter.com is the current tower PoP. We can get transit from him, of course peer with KCIX , and we'll probably get transit from another local ISP in town (CTC). Of course level3/att/vz et al are all in town/on net and just a very short fiber hop away from Joes if we want to go that route.

  Do

you have a good working relationship with these folks (cell phone,
email contacts that reach someone promptly)?

Yes. Very much so.

Will you be considered a

facilities based ISP (and subject to CALEA or other regulation)?

I'm not sure. CALEA compliance is a very big deal for us. Especially in regards to making an open doc about being compliant and any necessary patches to the FLOSS supply chain for compliance.

As far as documentation goes, we're working on a FLOSS book:
https://commons.thefnf.org/index.php/Building_a_local_network_in_your_neighborhood

which will help folks build low cost community based access networks.

We are all about building a (business/technical/operational) model which can be readily and easily replicated by existing community based organizations and not need to wait on muni networks (with all of the complexity/risk/unknown unknowns etc that implies). The current bit about cities having to ASK the federal govt (mother may I build an ISP, even though the bullys have said I can't)? Are you kidding me? What happened to techies banding together, getting some management "bridge" types to organize the community and put up a network!

9

I would

also suggest using stateless firewall rules and routing on your WAN
devices.

That does seem to be the common wisdom. I'm actually not 100% sure what we've got in line. It's OpenWRT based all around, so I'm sure IPTABLES (and maybe even some ebtables).

iptables performs state tracking. So does pf in BSD. Sooner or later you'll run out of room in your state table. This is kernel tunable, and the OpenWRT guys have probably tuned for their needs, but their market is devices serving a few users, not (several) thousands. Even a pfsense box with GB's of RAM caps at 500k simultaneous flows. I would plan on an average of 1000 flows per residential user. Most users will use less, some will use more, and some poor sob will get DOS'd and use 10's or 100's of thousands. If I were to deploy CGN/stateful software I would keep it out of the core and either push it to the edge (user routers) or to a CGN appliance/cluster as a discrete entity in the network; I'd let the routers focus on routing and the switches focus on switching.

I've got pretty much every Cisco router/switch in our lab, and an EdgeRouter.

What mikrotik should I evaluate?

Our lab : https://commons.thefnf.org/index.php/FNF_Lab

If you've automatically discounted big name gear due to upfront costs,
you might consider buying from a used equipment reseller (I can
recommend a few, if needed).

No. It's mostly for the customization/scripting etc. "SDN" and all that jazz. :wink:

OK then. Just wanted to make sure you weren't excluding anything due to perceived budget issues. I'd think of a Cisco/Juniper/Brocade/whatever router as a special purpose server. You can use that Dell and OSS, but you've got a lot of extras in a Dell that can cause it to fail and you can't hot swap line cards, CPU's, etc in a Dell. I haven't used Mikrotik, but several of my clients, especially the ones involved with wireless, have been happy with the support and appliance options. They have the advantage of OSS without the disadvantages of a general purpose Dell/IBM/whatever server.

If you do need to use NAT, I feel like 500+ users sharing a single NAT
IP will result in poor quality of service and more admin overhead.

Quite possibly. However if it's just for long tail v4 only sites, I wonder how much it matters?

Probably depends on the amount of v4 traffic you have on your network. My guess is that v4 flows (not necessarily bits) will be the majority of your traffic for many years. Even services that primarily utilize v6 may still have v4 content. I believe v4 is and will continue to be of vital importance even after all of your users have working v6 connectivity and devices with good v6 support.

I didn't see it mentioned, where (and to whom) are you multihoming?

Kansas City Kansas. Joesdatacenter.com is the current tower PoP. We can get transit from him, of course peer with KCIX , and we'll probably get transit from another local ISP in town (CTC). Of course level3/att/vz et al are all in town/on net and just a very short fiber hop away from Joes if we want to go that route.

Do

you have a good working relationship with these folks (cell phone,
email contacts that reach someone promptly)?

Yes. Very much so.

Sounds like you have that covered.

Will you be considered a

facilities based ISP (and subject to CALEA or other regulation)?

I'm not sure. CALEA compliance is a very big deal for us. Especially in regards to making an open doc about being compliant and any necessary patches to the FLOSS supply chain for compliance.

Looks like something that warrants more investigation.

As far as documentation goes, we're working on a FLOSS book:
https://commons.thefnf.org/index.php/Building_a_local_network_in_your_neighborhood

which will help folks build low cost community based access networks.

We are all about building a (business/technical/operational) model which can be readily and easily replicated by existing community based organizations and not need to wait on muni networks (with all of the complexity/risk/unknown unknowns etc that implies). The current bit about cities having to ASK the federal govt (mother may I build an ISP, even though the bullys have said I can't)? Are you kidding me? What happened to techies banding together, getting some management "bridge" types to organize the community and put up a network!

Let me know how it goes and if you need any help (I'm in Lenexa).

--Blake