Evening, Justin. Thanks for the reply.
Justin M. Streiner wrote:
You could certainly store all of the relevant config details in a database of some sort, and it certainly can't hurt to do so. Same goes for backing up your device configurations - always a good idea. As far as storing things like ASNs, allowed prefixes, etc, you may want to look at storing that information in an RPSL format. Many providers require their customers to register route/AS/policy objects either in one of the Internet Routing Registries (RADB, AltDB, etc...), or in a similar system that's operated by the provider. They then use this information for pushing out configuraiton changes such as the list of prefixes that a customer is allowed to announce, etc.
RPSL could definitely be useful when we starting reaching that class of customer. It's probably too grand for our users at this point (especially having them register anything on their own). I'll definitely do more research on this though.
It's good that you're thinking about this now, particularly the "is this customer legitimately allowed to announce prefix ABC, or source their traffic from AS XYZ"? Most larger providers put at least some limitations on what they will allow a customer to announce, though the level of investigation done to attempt to establich legitimacy for those announcements varies. Some providers require customers to provide some sort of Letter of Authorization/Agency for the prefixes they want to announce.
The process hasn't been established yet for validating a request to permit a prefix announcement. I expect it will be a manual verification process involving WHOIS lookups on the prefix, route-view queries to see if it's currently being advertised and probably a historical check on that prefix to see if it was previously advertised and from where. The best way to avoid network abuse issues is to not let them happen to begin with. I think we will also require some sort of written legal agreement stating that the prefix in question belongs to the customer and that we're authorized to permit it's advertisement across our network. If anyone has any sample documents for use in this process I would be interested in seeing them.
Yes. As long as the organizations that are using the private AS aren't 1. trying to advertise the same space, 2. possibly connect directly to each other directly, or 3. expecting to be able to connect to multiple upstream providers, then you should be OK. VZB (former UUNET) did something similar, using AS7046 (not a private AS, but the principle is the same), and I believe other carriers have had similar arrangements for customers.
Ah, yes connecting to each other could be a problem. I would think that it would only be an issue if I carried the private ASN across my iBGP infrastructure, each customer received full routes from me and I let them see the private ASNs as well. I could mitigate that problem with remove-private-as, I believe. I'd need to think on that some more. If the customer wants to be multi-homed and expect reachability then they should get an ASN. Otherwise both SPs are advertising their prefixes and the customer won't have much or possibly any control over which inbound path was preferred.
MD5 is good, but most providers I've seen make this an opt-in feature - they don't force their customers to use it. Setting a reasonable max-prefix limit and adjusting it as the number of prefixes a customer announces is always a good idea, and I'd consider it to be a best practice. Prefix lists and route maps can do different things, or accomplish the same task in different ways. It also depends on what functionality you want to offer your customers. Do you plan to publish and support a consistent set of customer-settable BGP communities for doing things like selective AS prepends? Do you plan to tag incoming advertisements with communities to identify them as customer routes, and pass those communities to your customers and peers? Some providers use AS path ACLs, and others avoid them at all costs.
I think I'll make MD5 part of the default config and let the customer ask for it to be removed if they choose to not have it. Same for GTSM. I'm a fan of max-prefixes. I think double the routes I expect to receive, 75% warning and a restart interval of 5m would be a good place to start. That would let me catch things happening before they got out of hand (in normal circumstances) and give me a fail-safe in case they decide to get crazy.
I do plan on implementing a BGP community solution but for now I'm going to keep it simple. I have bigger fish to fry at the moment but I'll try to get it done before we get asked for it by a customer. I will tag transit and customer routes. The ISP Essentials book had some good insight on that if memory serves me correctly. How fancy it gets will depend on my time and customer demand. I've seen some extremely complex setups that I could not replicate if my life depended on it.
The AS-Path filtering should only come into effect when we get a customer with their own ASN in which cases we'll actually pass on their advertisements (whereas with private ASNs we're only carrying them internally and summarizing on the upstream edges). That way I can ensure 1) that they claim to source the prefixes from their ASN and not someone else's and 2) that they don't insert BS ASNs into the path.
If the customers need to connect to multiple upstream providers, it's much less hassle to have them originate their prefixes from their own AS, and then you just need to propagate them. If they are single-homed, you can announce the prefix(es) for them and then just statically route them to the customer. Other things to consider are sound ingress/egress filtering policies, loose/strict RPF etc. Implementing a Netflow based monitoring solution, and applying flow caching to PE-CE interfaces is also agood idea.
I have a good set of residential and tiny SMB customer ACLs. I'm trying to decide what I should use for these larger SMBs. I'm sure some will be custom but I need to come up with a sane default set of ACLs. Some things I refuse to not block. The customer can find a different provider if they want certain things to not be blocked.
Until we get a multi-homed customer that wants inbound reachability we'll go with strict uRPF; that's already in my interface templates.
I have NFSen set up but all it's really doing right now is filling my hard drives. I haven't had much time to do anything else with it I'm afraid. I'll point NF to MARS box when it gets here though. I forget, does route-cache flow alter the packet switching capabilities of an interface these days or is that just to turn on NF on an interface? Seems like since CEF it only turns on NF.
Again, it's great that you're being proactive about this. Setting up good policies now will make things much easier to maintain and provision in the future as your network grows. Don't forget to document those policies thoroughly, particularly if other people will be tasked with implementing it on a dat yo day basis, i.e. handling new customer turn-ups, making BGP prefix list modifications, etc.
That's my goal. I prefer to be organized from the get go whenever possible. I don't like surprises in my work (unless it's a big, unexpected raise).
It seems like you have a pretty good handle on what you need to do. I don't know that there will be much of the holy war angle to the responses. Different people adopt different customer routing policies for different reasons - some technical, some business, some political. While there are different ways of accomplishing this task, most of the more scalable ways have already been a part of most large providers' policies for awhile.
I have an idea of what all needs to be implemented but I'm short on time to hone my skills by trial and error.
One-offs are bad, in my opinion, so the more you do to avoid them now, the fewer headaches you will have down the road.
Definitely. One-offs are like using duct tape to repair a Porsche. It's just not right.
Thanks for the input. It's much appreciated.
Justin