I am an Network Operator working in an Enterprise environment with offices all over the country(mostly connected via MPLS). We are currently working towards building a Disaster Recovery Site that will host some of our vendor routers and provide the capability to access these vendors from both our primary and backup data center locations. The routes(as advertised by the vendor's routers) will be the same at both locations. I would like to advertise the routes from multiple locations at the same time, rather than suppress the routes and advertise conditionally.
What is the best method to Instruct the provider's network to prefer the Primary Data Center routes over the DR site? Keep in mind that I am only peering with the provider over BGP and I have no visibility to the underlying MPLS architecture or configuration. Although if you have specific questions about their architecture, I can work to get answers.
Discussing in house, we have gone over a few different options:
-Advertise specific routes from primary site and summary routes from the DR site. Most specific will always be chosen.
-Prepend the routes from the DR site so that they will have a longer AS-path than the Primary location
-Use Community Strings to influence local preference.(Still working to find out if Provider will pass our community strings)
Just looking for some ideas and best practices. Any thoughts or insight would be much welcomed and appreciated. This is my first message on NANOG, so please be gentle. I apologize in advance if I have done something incorrectly.
This assumes however that the failover you require is based on the router at the primary site going down. I understand it's for DR, but if you have your routing equipment on a UPS/Generator at the primary site, it's something to think about. DR may mean more than just your link or site going down, for some people it goes as far as the server farm going down (but the router still up). Something to think about.
For an Internet-based system, here's how you would do it. The private
MPLS-based network you describe won't be quite the same but it'll be
similar.
* Announce with a AS path length from the DR site that has at least 3
prepends. Get your own RIR-assigned AS number for this; you can use
private AS numbers but this will eventually confuse someone debugging
a connectivity problem.
* Local pref the accepted routes to prefer the primary site.
* At least two ISPs at the primary site.
* At the DR site, the usually single ISP should be the same as one of
the ISPs at the primary site. That way when there's trouble talking to
the two sites there's only one vendor to blame and it's the one you
pay directly. It also means the GRE tunnel traffic between sites tends
to stay on a single carrier.
* GRE tunnels between the sites running IBGP. One GRE tunnel for each
pair of Internet connections. Despite your best efforts you'll get a
trickle of traffic into the DR site during normal operation of the
primary. You'll want to send it back to the primary site and that
should all happen outside the firewall.
* In addition to your BGP announced addresses, get a small bank of IP
addresses from each ISP for each Internet connection at each site. I
usually ask for a /28 but a /29 is normally adequate. You'll need
these to anchor your GRE tunnels and management functions.
I am an Network Operator working in an Enterprise environment with offices
all over the country(mostly connected via MPLS). We are currently working
towards building a Disaster Recovery Site that will host some of our vendor
routers and provide the capability to access these vendors from both our
primary and backup data center locations. The routes(as advertised by the
vendor's routers) will be the same at both locations. I would like to
advertise the routes from multiple locations at the same time, rather than
suppress the routes and advertise conditionally.
At work, we have our internal routing protocol running on GRE over
IPSec tunnels & keep the BGP sessions with the MPLS provider limited
to just the MPLS network. And have an ACL on the MPLS network
interface that allows only what's expected in... some providers are
better than others at not having anything hit the 'deny any any log'
line
I think having a GRE tunnel for the internal routing protocol is
unnecessary. Can you explain the reasoning behind this? I understand
the technical issue whereby GRE will allow multicast for EIGRP, OSPF,
etc, but why not just redistribute into BGP?
I work on a lot of MPLS CE routers, and in general you can accomplish
anything you need by redistributing your internal routing protocol into
BGP, and adjusting LP, MED and AS Prepend as needed.
I think having a GRE tunnel for the internal routing protocol is
unnecessary.
It might be, but we have a requirement for multicast over the wan so
the GRE tunnels had to be there.
Can you explain the reasoning behind this? I understand
the technical issue whereby GRE will allow multicast for EIGRP, OSPF,
etc, but why not just redistribute into BGP?
I see no reason to trust the provider that much.
I work on a lot of MPLS CE routers, and in general you can accomplish
anything you need by redistributing your internal routing protocol into
BGP, and adjusting LP, MED and AS Prepend as needed.
Sure.. but how do you *know* you're not getting anything added/removed
by the provider?
1) Ask the provider if they have any traffic engineering communities
available. Many of the large ones offer some.
2) Use BGP MED to influence the output path (works in most cases).
3) If that fails, use as-path pre-pending to influence the output path from
the provider towards you.
GRE tunnels are not necessary for MPLS in most use cases. Additionally,
many SPs support native multicast over their L3VPN services if you need
this -- shop around.
Finally, you mention the vendor can accept traffic on either router.
Consider just announcing the routes equally from both locations into MPLS
and letting the traffic more or less load balance as it sees fit on the way
to your vendor -- this is how the internet generally works.
I think having a GRE tunnel for the internal routing protocol is
unnecessary. Can you explain the reasoning behind this? I
understand the technical issue whereby GRE will allow multicast for
EIGRP, OSPF, etc, but why not just redistribute into BGP?
I work on a lot of MPLS CE routers, and in general you can
accomplish anything you need by redistributing your internal
routing protocol into BGP, and adjusting LP, MED and AS Prepend as
needed.
Thanks,
Bill
- -----------------------
Using bgp communities (MED attribute "inbound") helped influence our
path(s) between our mpls providers.
regards,
/virendra
Sent: Friday, August 31, 2012 11:15 AM To: Tribble, Wesley Cc:
nanog@nanog.org Subject: Re: Redundant Routes, BGP with MPLS
provider
Hello all,
I am an Network Operator working in an Enterprise environment
with offices all over the country(mostly connected via MPLS). We
are currently working towards building a Disaster Recovery Site
that will host some of our vendor routers and provide the
capability to access these vendors from both our primary and
backup data center locations.
The routes(as advertised by the vendor's routers) will be the
same at both locations. I would like to advertise the routes
from multiple locations at the same time, rather than suppress
the routes and
advertise conditionally.
At work, we have our internal routing protocol running on GRE over
IPSec tunnels & keep the BGP sessions with the MPLS provider
limited to just the MPLS network. And have an ACL on the MPLS
network interface that allows only what's expected in... some
providers are better than others at not having anything hit the
'deny any any log' line
Regards, Lee
What is the best method to Instruct the provider's network to
prefer the Primary Data Center routes over the DR site? Keep in
mind that I am only peering with the provider over BGP and I have
no visibility to
the underlying MPLS architecture or configuration. Although if
you have specific questions about their architecture, I can work
to get
answers.
Discussing in house, we have gone over a few different options:
-Advertise specific routes from primary site and summary routes
from the DR site. Most specific will always be chosen. -Prepend
the routes from the DR site so that they will have a longer
AS-path than the Primary location -Use Community Strings to
influence local preference.(Still working to find out if Provider
will pass our community strings)
Just looking for some ideas and best practices. Any thoughts or
insight would be much welcomed and appreciated. This is my
first message on NANOG, so please be gentle. I apologize in
advance if I have done something incorrectly.
I work for an MPLS provider, so I guess I tend to trust them
For certain definitions of "trust" I would also. But.. Monday? I was
told that $AGENCY had just completed an audit of our network and we
had to change the exec timeout from 15 to 10 minutes on all routers
and switches.
Apparently that extra 5 minutes is an unacceptable security risk. But
leaving the network wide-open to all sorts of routing hijinks via
MPLS? (I don't have route filters & acls on all of the mpls
interfaces yet) nada
We can't trust the people in our office area to not to take advantage
of an unattended terminal but we can trust our MPLS providers to not
take advantage of their unrestricted access? Seems backwards to me.
So, rather than run an IGP between siteA and siteZ across
a GRE tunnel, you'd prefer to redistribute your IGP into BGP
at siteA, advertise those routes upstream...and at siteZ, accept
the routes in via BGP, and then redistribute them into the IGP
for the other routers at siteZ, and vice versa?
Or would you have every router at siteA and siteZ participate
in BGP, so that all the routers at siteZ get the routes from
siteA intact?
(choice B tends to have practical implications on what
network gear you can run within the sites; many devices
that will happily speak OSPF or EIGRP won't be quite so
happy participating in an iBGP mesh. And choice A...well,
I think we all know the pitfall with choice A, so enough said
on that score).
Curious to hear the actual mechanism you'd use to make
this work in the real world.
