DOs and DONTs for small ISP

hi there,

I know there are folks from lots of small ISPs here and I wanted to check-in on asking few advice points as I am involved building an ISP from green-field.

Usually, it’s pretty straight forward to cover high-level important things, filters, routing policies, etc.but we all know the devil is in the details.

I am putting together a public DOs and DONTs blog post and would love to hear from those who have built ISPs and have recommendations from Billing to Interconnection, Routing policy to Out of the band & console setup, Software recommendations, etc. Bottom line is that I would like to publish a checklist with these recommendations which I hope will be useful for all.

thanks in advance for your help and recommendation.

Mehmet

Mehmet, I think this is a cool idea, perhaps a good format for the documentation would be something along the lines of an “awesome list”? (https://github.com/sindresorhus/awesome)

Chris

Here is your checklist in descending order of importance:

  1. market opportunity
  2. finding the right partners (see below)
  3. financial
  4. sales and marketing
  5. organizational capacity and HR
  6. legal, regulatory
  7. capital acquisition
  8. security
  9. technical including equipment selection, routing policy, filtering, etc
    It is a stone cold lock that the success of your new ISP will governed by factors other than technical. Your most important task is to find competent financial and marketing people you can respect and trust. If the market opportunity exists and you find them, you will succeed. If you don’t, all the technical excellence in the world won’t help you. The road is littered with technically excellent companies that failed.

I’m constantly amazed at the number of even medium-sized ISPs that have no network monitoring. An NMS should go in as the first software component — before billing starts and the provider is on the hook to deliver.

The second lacking component is a ticket system, which is silly because turnkey cloud services are not expensive, and open source solutions abound for budget-limited operators.

The third component failure is security, including weak and default (!) passwords, failure to use real certificates, and the complete lack of 2FA or MFA. Security also requires data surveillance, in the form of net flow analysis.

The “two guys and a router” business model must be upgraded with more planning and a cohesive operating plan.

-mel

thank you, great start, just want to jump start to technical part AFTER deciding to start the ISP, and focus on operational, technical, and administrative things though. Question is no longer whether starting ISP is a good idea or not (but i agree this is needed and was already completed) thanks again.

Probably the #1 thing I've seen messed up is BGP config.

1) Nail up your routes using network statements and static null
    routes. Don't rely on redistribute connected to advertise what's
    configured on an ethernet interface. You probably shouldn't be using
    redistribute at all unless you "know what you're doing" with it.

2) Don't advertise your v4 IP space as a collection of /24s if you have a
    larger aggregate block, unless you have good reason to do so...and if
    you do, you should probably still advertise the aggregate unless
    there's a good reason not to.

3) Don't advertise one transit provider's routes to another. Each should
    be filtering your routes, but you never know. Come up with, and use
    BGP communities to control route propagation. As you grow, it sucks
    having to update prefix-list filters in multiple places every time
    something changes...like a new customer with their own IPs.

The "ISP Essentials" publication is still valid in a lot of cases. It might not cover everything but gets a lot of the basics right.

ftp://ftp.securecomp.net/EBook/Cisco�%20ISP%20Essentials.pdf

https://startyourownisp.com/ this is also pretty good, just thought i would share…

I’m constantly amazed at the number of even medium-sized ISPs that have no network monitoring. An NMS should go in as the first software component — before billing starts and the provider is on the hook to deliver.

  often people are using tools like quickbooks to start, these don't support integration with networking tools. You see tools like Sonar or powercode in use. Some of this is changing with newer tools like UNMS and UCRM in some spaces, but often these are vendor locked or don't integrate well.

The second lacking component is a ticket system, which is silly because turnkey cloud services are not expensive, and open source solutions abound for budget-limited operators.

  The number of people who can't do sysadmin functions is high. there's a reason SaaS is a thing, but the costs are often enough to force someone to roll their own. Take something like powercode with a $1/subscriber fee which adds up quickly.

The third component failure is security, including weak and default (!) passwords, failure to use real certificates, and the complete lack of 2FA or MFA. Security also requires data surveillance, in the form of net flow analysis.

  Much of this is because hardware has defaults that aren't sane or lack some ZTP or provisioning that you can do. How do you do this with UBNT, Tik or other cost optimized hardware?

The “two guys and a router” business model must be upgraded with more planning and a cohesive operating plan.

  Most large networks are run with small teams, while usually more than 2 it's often not more than 10 to do the arch + eng work necessary. If you have more, they're often doing installer work not actual eng work.

  - Jared

We have a fledgling web-site with information at http://startawisp.info/

Here is your checklist in descending order of importance:

market opportunity
finding the right partners (see below)
financial
sales and marketing
organizational capacity and HR
legal, regulatory
capital acquisition
security
...
...
...
technical including equipment selection, routing policy, filtering, etc

It is a stone cold lock that the success of your new ISP will governed by factors other than technical. Your most important task is to find competent financial and marketing people you can respect and trust. If the market opportunity exists and you find them, you will succeed. If you don't, all the technical excellence in the world won't help you. The road is littered with technically excellent companies that failed.

Indeed, but you *also* need to have some technical clue. Two or three
years ago a friend and I tried to start a local wireless ISP -- I was
doing this purely as a "My home Internet access sucks, and I'll
happily donate time, equipment, IP space and some startup capital to
fix this" play -- unfortunately it turns out that he and I had very
different ideas on, well, basically everything. I wanted an actual
architecture / design, and diagrams and routin' and such. He was much
more of "We don't need a list of IPs, if I ping it and can't reach it
it must be free" / "routing is too hard, let's just put it all in a
switch and... um... NAT!". I wanted a plan, and was willing to put in
the time and effort to build Ansible / Puppet / an NMS / AAA, etc, he
was more seat-of-the-pants.

But yes, even if we had good technology this would have failed - there
was no real business plan (other than "The current provider is really
bad, if we build something else, people will be breaking down the door
to sign up"), no real marketing plan (see previous), etc.

He was also a bit of a gun nut, and so would arrive at customers with
a (holstered) firearm belted on -- even in Virginia this was not a
winning business move.

Starting a successful ISP is this day and age is hard - make sure
that, if you do it, you and whoever you are doing this with are
compatible, are both committed, and have similar views on things...

W

I would respectfully point out that my point about the importance of finding the right partners. For you, sounds like it was good to have opportunity to get out of this venture.

I would respectfully point out that my point about the importance of finding the right partners. For you, sounds like it was good to have opportunity to get out of this venture.

Oh, goodness yes -- however, I *still* have barely working Internet
access at that location (it's basically a weekend home) -- I've
somewhat started down the Jared Mauch "buy used vibratory plow, trench
(house is off a dirt road, neighbors won't have an issue with right of
way if I provide them free bits!), install fiber" path. My driveway is
@1/4 mile, the dirt road is 0.6 miles and at the end of the road there
is (what used to be) Quest fiber. Unfortunately the fiber was
originally run for Mount Weather
(Mount Weather Emergency Operations Center - Wikipedia)
and no-one I've asked seems to know who controls it now, and or swears
that it doesn't exist (although the local Miss Utility / VA811 will
come mark it if you call).

Some dark nights, when trying to use the network while everyone is
using NetFlix, and the RTT for my SSH sessions starts exceeding
1,500ms I strongly consider walking to the end of the road with a
nice, sharp shovel, and then talking to the splicers when they come
repair the damage :slight_smile:

W

To reiterate all this, FILTER EVERYTHING.

To start with, explicitly specify in a route-map or similar everything you want to advertise. I usually create a separate route-map for each transit/peer and include what I want to advertise via prefix lists (for my IP space) and/or communities (for downstream BGP-speaking customers if anticipated).

When you turn on the session, check what you're squawking AND WHAT YOU'RE FILTERING. You shouldn't be filtering anything you don't expect. Belt + suspenders.

The same goes for anything you accept. Obviously for a blended full transit BGP edge router, you're probably going to accept almost everything. But if you only want default + on-net, try to filter using communities from the peer, etc. Again, right when you turn on the session, "sh ip bgp ... filtered" of whatever's equivalent on your platform. If you're filtering something you don't expect to be receiving at all, figure out where the misunderstanding or misconfiguration lies.

And of course it goes without saying that, if you've got BGP speaking customers, you filter the heck out of them. Use ROAs and/or RPKI if you can to automatically generate filter lists. Encourage your upstreams to do the same if they're filtering you (and they probably are, or at least should be, if you're new). Remember that you are responsible for every route you advertise, at the end of the day, even if you only advertised it because a downstream network made a boo-boo and you didn't filter it.

Filters are useful on your IGP, too, but there's so many ways to set all that up that it's a bit more difficult to come up with nearly universal best practices. Generally speaking, be careful with redistribution, never distribute BGP into IGP or vice versa unless you have a really, really good reason to, and consider filters between both IGP areas/regions or protocols (e.g. RIP coming into OSPF) as well as on redistributions of static/connected to prevent simple typos on a static route or interface configuration from taking down more than just local stuff.

It's way, way easier to remove or relax filters later if they prove more of an operational hazard than asset than it is to add or tighten them if they prove insufficient.

Few tips:

1) Some incumbents don’t know their fiber, so sales reps don’t know how
to quote it but a reseller can. This might be a good route. Then again
I am having trouble with a firm right now so will be poking around at
NANOG to solve my problem.

2) Don’t tell all my secrets yet. I also now own a directional drill
and have a tariff on file with the state. (It wasn’t expensive once you
find the right firm).

3) Fusion splicers and OTDRs are super cheap these days if you’re doing
simple work.

4) Labor is always the expensive part.

5) More to come, maybe by the October NANOG I’ll be ready with my talk.

- Jared

> 3) Don't advertise one transit provider's routes to another. Each should
> be filtering your routes, but you never know. Come up with, and use
> BGP communities to control route propagation. As you grow, it sucks
> having to update prefix-list filters in multiple places every time
> something changes...like a new customer with their own IPs.

To reiterate all this, FILTER EVERYTHING.

To start with, explicitly specify in a route-map or similar everything
you want to advertise. I usually create a separate route-map for each
transit/peer and include what I want to advertise via prefix lists (for
my IP space) and/or communities (for downstream BGP-speaking customers
if anticipated).

I think a related *principle* is: "Build everything as though you are
expecting to scale."

This doesn't mean "spend lots of money to buy huge
[routers|servers|commercial software|<etc>], but rather "when you plan
your addressing structure and routing policies and monitoring and
device config generation and... keep the in mind the question "If this
suddenly takes off, and I hire N more people to run this, can I
explain to them how it works? Do I have documentation I can point them
at or is it stuck in my head / on the devices? If I need to add
another M customers in the next month, can I do that easily?".

This is related to the FILTER EVERYTHING -- when you turn up a new
customer / peer / transit / whatever, you shouldn't be sitting around
trying to figure out how you will write their route-map /
policy-options -- this leads to weird one-offs, and quick hacks.
Instead you should have policies already largely designed and simply
plug in their prefixes (or, better yet, use bgpq3 or similar to build
and populate these). Obviously there will be some cases where a new
connection does require some special handling, but that *should* just
be a plugin/chain in an existing policy-statement. Related to this is
how you end up naming things -- I recently found 9 variants of
firewall-filters which basically do:

filter ACCEPT {
   term ACCEPT {
    then accept;
  }
}
named things like: ACCEPT, ACEPT, Accept, Allow, Permit_all,
AcceptAll, dontdrop [0].

Obviously, there is a tension in the "design for scale" - while it
would be great to design a complete automation system so that
everything from installing a new customer to a new sites is simply
typing 'make <thing>' and having everything pull from a database, at
some point you will need to actually build a network, or you'll never
have customers :slight_smile: Just keep in mind that "Am I building myself into a
corner here?". E.g it only takes 10 or 15 minutes to install something
like NetBox to keep track of addresses (and prefixes and racks and
connections and ...) -- stuffing this in a spreadsheet might save you
a few minutes *now*, but will this scale? Can $new_person easily
figure it out?

W
[0]: My personal favorite is:
filter Accept_All {
    term Accept {
        then {
            count dropped;
            reject;
        }
    }
    term filter_<customer> {
        from {
            prefix-list {
                <customer>;
           }
        }
        then accept;
    }
    term NEXT {
        then log;
    }
}

Presumably this all made sense to <name_removed_to_protect_inoccent>
when they stuck it in at 3AM to deal with some crazy issue, but...

This Gem is fantastic by the way,

https://nsrc.org/workshops/2015/apricot2015/raw-attachment/wiki/Track1Agenda/01-ISP-Network-Design.pdf

  1. Don’t run Mikrotik. I’m kidding… I think :-)… Mark.

14. Go with K56flex, not X2.

triggered :slight_smile: