I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections firewalled by a Juniper SSG of some description.
Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are:
- Should I make sure that the backup Internet connection is from a separate provider?
- How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and external translation?
- In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other?
I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the complexity and spend.
I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned
is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently
4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it
also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and
translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party
support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections
firewalled by a Juniper SSG of some description.
Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make
sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection
and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to
do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and
reliability issues. So my questions regarding this are:
- Should I make sure that the backup Internet connection is from a separate provider?
Yes yes yes yes a thousand times yes. Depending on the criticality of internet connectivity you should also aim to have your redundant
connections coming from a complete separate direction. Example, fiber from Level 3 come from the north in a dedicated conduit and
fiber from Verizon coming in a dedicated conduit from the south of the building. Why? Put simply we had construction ignore the
painted lines and dig up our conduit a few years back. At that point we have 4 bonded T1's from a single carrier. That was a long
couple of days... Carrier diversity is not a bad thing, spend some time shopping an additional provider. Make sure they operate their
own network for last mile, and also make sure they don’t piggyback off the same network your main carrier does anywhere locally.
Comcast Ethernet, Verizon and Cogent make great secondary connections when you need high availability. You don’t need your
secondary to have 99.999% uptime. 97% is usually good enough if it's on a separate network. I wouldn't sway from the big names
for your primary connections either.
- How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its
possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and
external translation?
With recent experience with the Juniper SSG VPN functions put nicely they suck. VPN failover is in there, but we had issues with the
tunnel staying active for extended periods of time. Also depending on if you do a route based or a policy based VPN, it becomes so
much of a headache. We used 2 SSG550 devices as a proof of concept and the one thing which annoyed me to no end was the complete and
total crap options within then VPN configuration. When I typically set up a VPN, I use a SonicWall NSA or E-class device (yes I know
hiss boo) or an ASA. Saying that the Juniper was lacking was a complete understatement. I personally would completely avoid even
attempting VPN failover within a Juniper device. I will say they are rock solid though for generic firewall functionality, just try
to keep the config simple or they turn into giant slow dogs.
- In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other?
Considering you have 4x 100mbit lines, have you looked at BGP? Even if you drop line 2 and its associated backup, you have 2x 100mbit
lines. Or even if you have 3 unique carriers with a 100mbit from each of them it makes BGP very appealing. I think this would be an
ideal situation for a BGP setup using a couple of small routers. You could probably get away with something as small as a Cisco 3825
for each connection (purely redundancy). If the Cisco name scares you Juniper routers are great as well. Don’t forget Vyatta!
If you do BGP, you have 1 VPN to configure, you have 1 tunnel to configure, there is no VPN failover configuration and hopefully you
are not pushing more than 1 subnet across the VPN otherwise you end up doing a route based VPN instead of a policy based VPN and you
will be significantly happier. That’s a Juniper headache for another day however.
I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the
complexity and spend.
What you really need to know to ask this is how much is Mr. Customer going to yell and scream if someone cuts only internet connection
and it's going to be down for about 3 days while they patch the conduit and fiber.
Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are:
I wouldnt jump to any conclusions that everything will work properly if you are terminating multiple connections directly on the SSG, what with egress likely being different than the ingress, even if you are using the same IP range (BGP) on all the links.
You could really be asking for trouble if you are planning on using a different ISP provided IP range on each connection for each purpose.
Front it all with routers that can policy route, whether or not you also use BGP.
I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections firewalled by a Juniper SSG of some description.
Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are:
- Should I make sure that the backup Internet connection is from a separate provider?
- How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and external translation?
- In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other?
Forget all of that and just multihome to two separate providers with
BGP. Also make sure that of the providers you choose that one is not a
customer of the other. Instant, painless redundancy. Having multiple
circuits to one provider *will not* back anything up if that provider
has an outage as they are %99.999 likely to be part of the same larger
circuit and certainly share the same infrastructure at the provider.
I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the complexity and spend.
Only if you calculate the cost (money, time, angry customers, etc.) of
an outage to be greater than the cost of additional connectivity.
Make sure they operate their own network for last mile
[...]
I wouldn't sway from the big names for your primary connections
either.
Because ownership of the provider/subsidiary delivering the last mile
means one hand is talking to the other, and you're going to get good
service and reliability as a result? And "big names" never have any
peering-related spats and always deliver the best possible end-user
experience, right?
(Some good points further on, though important we don't lead the OP
down the wrong path or with a false sense of security there...)
Forget all of that and just multihome to two separate providers with BGP
--Assuming that you're advertising PI space or can work around that appropriately with your providers, I agree, that's the ideal situation.
Having multiple circuits to one provider *will not* back anything up if that provider
has an outage as they are %99.999 likely to be part of the same larger circuit
--True - if you don't specify otherwise when you're ordering, then why would they make the effort? Comments made in some of the other responses in this thread are also valid even with a single service provider - diverse entry points into your facility, diverse upstream circuit routing, and homing to different POPs - which may mean backhauling your secondary circuit away from your local POP and taking a hit for the higher latency on that second link. The moral of this is that whether you're using one provider or more than one, state your diversity requirements clearly up front, and then stay involved and make sure that what's presented to you is _actually_ diverse (oldsflash: even the best intentioned people sometimes make mistakes, especially when there's a handoff to a different last mile provider who may not have been clear on the requirement ). Of course, all of this is potentially wasted effort if the data center you're providing connectivity for does not also maintain the same kind of diversity itself in terms of power, connectivity, architecture, etc.
and certainly share the same infrastructure at the provider.
--If you enter a single provider's network at diverse points, then that local infrastructure isn't the same at least. But by the same measure, if that provider has a major BGP issue for example, then yeah - they're both screwed... in which case we loop back to the dual provider scenario you mentioned in the first place
Ultimately choosing the appropriate solution will boil down to the what level of service unavailability one can tolerate in the first place, and put a business value on that impact. From that one can derive technical options, then go cap in hand with a business case to the poor soul paying the bill
This is, of course, dependent on the OP's location and budget. I know when we
were getting our NLR connection set up, there was a fair amount of "You want
40G worth of DWDM *where*?" involved, and the resulting topology was...
complicated. At least at one time, there were places where our provider was
running our link across lambdas of a subsidiary of ours, which are going across
physical fiber owned by the provider... turtles all the way down.
