Greetings,
We have recently added a second ISP (third if you count I2). Our first "ISP" is actually a private state network that peers with two Tier 1 providers. We own an AS number and our IP space but at the last minute learned our state network is advertising our network using two different ASNs (neither ours) so they can load balance their connections. If you hit the right looking glass server you can see our network advertised by three different ASNs. We were told by the new ISP that this is a problem but the state network says it is not.
Looking for opinions and words of wisdom on this split advertising issue.
Thanks
curtis
Curtis Parish
Senior Network Engineer
Middle Tennessee State University
Why aren't you originating your own prefixes and ASN by
yourselves, since you own both?
Mark.
We own an AS number and our IP space but at the last minute
>> learned our state network is advertising our network using two
>> different ASNs (neither ours)
This will work, as in the BGP path selection algorithm will work as
designed in this situation. But it also means that the routing policy
is out of your control which is kind of the point of having an ASN! It
also makes it harder to track down who is operationally responsible
for that address space since it appears to the outside world to be in
two (or three! different places). I'd say don't do this unless you
really have no choice.
> Why aren't you originating your own prefixes and ASN by
> yourselves, since you own both?
Good question.
We (AS60241) almost ended up doing similarly for a while. Because of a
close association with the universities in Scotland, we discussed the
possibility of transit via JANET. This turned out to be difficult
because they run a whole bunch of private ASNs internally -- unlike in
North America where universities typically have their own real one. So
it would have been us -> private stuff -> AS786 and for some reason
that I forget they were unable to remove private ASNs from the
path. The best that might have been possible would be to have had them
announce our networks with synchronisation on, which would have meant
the outside world would have seen them originating in both AS786 and
AS60241. Icky. We (mutually) decided against this.
Just to say that there are strange, but not completely unreasonable
circumstances in which this can happen...
-w
Howdy,
If you drop your connection to the state network, do the routes with their
AS numbers drop out of the looking glasses? If not, then there's a problem.
If you depreference your connection to the state network by prepending your
AS number, do comparable prepends appear at the looking glasses or does the
state network continue to give its advertisement of your address space top
billing? If the state network's behavior strips your ability to load
balance your network then there's a problem.
Conventionally, the state network should be adding its AS number after
yours, not stripping your AS number. More often than not, this convention
is also the technically correct course of action.
Regards,
Bill Herrin
Thanks for all the responses. I will answer a few questions that have come on and off list. (Sorry for length)
We advertise our ASN into the state network with more specific routes that we advertise via ISP2 via our ASN. This is done because the state (vendor managed) network runs stateful firewalls and we have to force other multi-home entities on the state network to use our state connection instead of ISP2. Our network has been removed from the state firewall due to previous problems with asymmetric routing with our I2 circuit. I am told the state network does drop our network from their advertisements when our network is unreachable. That has not been explained or tested.
What we did not realize until about a week before turning up ISP2 was the state was consolidating all state networks to use two of the vendor’s ASNs when it peers with their two ISPs. Our ASN is not part of the path. We had no choice but to turn up ISP2 due to bandwidth reasons. Miraculously we achieved almost a 50/50 balance of traffic. Bandwidth will be increased on ISP2 as demand grows so we will need the ability to prepend on the state network to make ISP2 look more desirable.
I believe the state will modify their advertisements to add our ASN to the path but changes to advertising via the state network has to go through a design and change management process and then be scheduled into maintenance windows. Any attempts to balance the traffic via prepending will take weeks. As long as the traffic stays balanced we are OK. When replaying BGP route changes I normally see our network only advertised out one of state ASNs but occasionally I see it with two so traffic balance may be impacted depending on which ISP the state is egressing.
Here is a question. I know that having one network advertised by multiple ASNs is unconventional and thus it will probably be harder to get help troubleshooting routing problems when they arise. Do you see a situation where our network might be caught in a loop or black hole due to asymmetric routing and conflicting advertisements?
Thanks again. New to the list but have already learned much by reading the archives.
Curtis
Curtis Parish
Senior Network Engineer
Middle Tennessee State University
Howdy,
If you drop your connection to the state network, do the routes with their AS numbers drop out of the looking glasses? If not, then there's a problem.
If you depreference your connection to the state network by prepending your AS number, do comparable prepends appear at the looking glasses or does the state network continue to give its advertisement of your address space top billing? If the state network's behavior strips your ability to load balance your network then there's a problem.
Conventionally, the state network should be adding its AS number after yours, not stripping your AS number. More often than not, this convention is also the technically correct course of action.
We have recently added a second ISP (third if you count
I2). Our first "ISP" is actually a private state
network that peers with two Tier 1 providers. We own an
AS number and our IP space but at the last minute
learned our state network is advertising our network
using two different ASNs (neither ours) so they can load
balance their connections. If you hit the right
looking glass server you can see our network advertised
by three different ASNs. We were told by the new ISP
that this is a problem but the state network says it is
not.
Looking for opinions and words of wisdom on this split
advertising issue.
Why aren't you originating your own prefixes and ASN by
yourselves, since you own both?
The practical problem here is that the control of prefix origination is
distributed. so if there is a need to withdraw it from the state network
or advertise it no export for some reason (e.g. performance problem
maintenance etc) you likely can't. Their grasp of load-balancing seems a
bit shallow also.
We advertise our ASN into the state network with more specific routes
that we advertise via ISP2 via our ASN. This is done because the
state (vendor managed) network runs stateful firewalls and we have
to force other multi-home entities on the state network to use our
state connection instead of ISP2. Our network has been removed
from the state firewall due to previous problems with asymmetric
routing with our I2 circuit.
Hi Curtis,
As you've already noted, the presence of a stateful firewall beyond your
BGP border is inimical to BGP multihoming. Traffic between two multihomed
networks must never cross a stateful firewall that is outside both
networks' borders. Practically speaking, there will asymmetry, path
flapping, per-packet load balancing and other quirks at locations outside
your control. The Internet DFZ is a chaotic system. Over time you won't be
able to make the packets reliably transit the firewall.
It sounds like this is a learning experience for both you and the folks at
the state network. If you have a friendly relationship with them, now would
be a good time to visit and talk about what are likely to be significant
changes to their network architecture to make multihomed users feasible.
Preferably with a the help of a local consultant who has BGP expertise.
If that doesn't sound like it would be a productive conversation then I
suggest you consider three different options:
1. Return to the state network alone,
2. Replace your state network connection with another commercial ISP,
3. Add an additional commercial ISP for the sake of your Internet access
needs, drop the BGP advertisements with the state network and then
implement resources which should only transit the state network using IP
addresses assigned by the state network rather than your BGP addresses.
Here is a question. I know that having one network advertised by
multiple ASNs
is unconventional and thus it will probably be harder to get help
troubleshooting
routing problems when they arise. Do you see a situation where our
network
might be caught in a loop or black hole due to asymmetric routing and
conflicting advertisements?
Yes. And frequently. You have this thing balanced on the head of a pin.
Regards,
Bill Herrin
Agreed. You could still recieve their routes and no/export your as but I wouldn't go beyond the firewall.
Jason Bothe, Manager of Networking
Rice University
o +1 713 348 5500
m +1 713 703 3552
jason@rice.edu
[snip]
In other words, you are in effect not in control of the advertisement
of your prefix,
therefore you practically don't actually have an autonomous system,
you have the number technically, but not the administrative division
that is intended to exist.
An appropriate amount of time to push out any change needed to an
announcement should be no more than 1 business day, but less than 2
hours in an emergency, to add extra impending or pull an announcement.
I would call a change management process that requires any longer
unacceptable, or not reflecting the reality of the importance of
well-maintained optimal properly functioning network connectivity.
You have what seems to be something very fragile, and you have very
low configuration agility, since you cannot change your announcements
as needed out through the state as you need them to.
A stateful firewall, has no correct place outside the border of a
multihomed network; by definition, to have a stateful firewall, there
must be a single point of failure (on the stateful firewall element)
at least for each unique load-balancing tuple.
So I would call (in this case), the origination of your prefix by
multiple ASes a bad thing.
The protocol allows this, but the other constraints related to the
situation are serious impediments that make the solidity multihoming
seem improper or potentially precarious, in terms of the true
originating AS' ability to function as an AS and manage their network
Thanks to everyone for your input on our less than desirable BGP situation.
I do want to make sure I add that the state network we are a part of serves everything from elementary schools, to universities. to the traffic cameras on the interstate. Many of these are in rural locations and in the past each state entity had created their own network including two separate state university networks. The state vendor managed network was created to save money and provide higher level services than just an ISP. Among other things it serves as the private WAN for some state agencies. As our internet redundancy and bandwidth demands have increased we have outgrown the need for the high touch services offered by the state network but we must participate in order to maintain WAN access to other state universities.
Thanks again for the feedback.
Curtis
Curtis Parish
Senior Network Engineer
Middle Tennessee State University
Are there discussion/guidance papers that one can point to, to improve
the depth of understanding, or at least get better configuration
choices? (Those are independent points of improvement...)
d/
Their grasp of load-balancing seems a
bit shallow also.
Are there discussion/guidance papers that one can point to, to improve
the depth of understanding, or at least get better configuration
choices? (Those are independent points of improvement...)
Bassim Halabi's book is getting a bit long in the tooth, but it was my
jumping-off point for my own forays into this space.
http://www.amazon.com/Internet-Routing-Architectures-2nd-Halabi/dp/157870233X/ref=sr_1_1?ie=UTF8&qid=1417390786&sr=8-1&keywords=halabi+routing
The nanog tutorials have been assiduous about updating the bgp materials
So there are several iterations of the practical materials.
joel