I am curious what is the best practice for OOB for a core
infrastructure environment. Obviously, there is
an OOB kit for customer managed devices via POTS, Ethernet, etc ... And
there is OOB for core infrastructure
typically a separate basic network that utilizes diverse carrier and diverse
path when available.
My question is, is it best practice to extend an inband VPN throughout for
device management functions as well?
And are all management services performed OOB, e.g network management, some
monitoring, logging,
authentication, flowdata, etc ..... If a management VPN is used is it also
extended to managed customer devices?
What else is can be done for remote management and troubleshooting
capabilities?
I've generally built an out of band network for the express purpose of saving my behind in the event of an unanticipated traffic problem on the primary network. Secondarily it allows secured access to equipment, and you can monitor (which is often not secure, read snmp) on it as well. However, I've never tried to extend one beyond a facility or campus exactly.
Lots depends on the type of network you're talking about and equipment you're using though.
I am curious what is the best practice for OOB for a core
infrastructure environment. Obviously, there is
an OOB kit for customer managed devices via POTS, Ethernet, etc ... And
there is OOB for core infrastructure
typically a separate basic network that utilizes diverse carrier and
diverse
path when available.
My question is, is it best practice to extend an inband VPN throughout for
device management functions as well?
And are all management services performed OOB, e.g network management,
some
monitoring, logging,
authentication, flowdata, etc ..... If a management VPN is used is it also
extended to managed customer devices?
What else is can be done for remote management and troubleshooting
capabilities?
IMHO, it is always a good idea to have completely different infrastructure
supporting Oob. It is the only way you can recover remotely from many types
of network errors. If the router is hung at rommon, somebody has to get on
console .... or accidentally removes your igp instanance ...
But, the business criticality of the network needs to justify the cost
(dial, DSL, 3rd party L3 vpn ....)
In my experience having your management run over product via VPN is
not a great idea. If possible separate the two.
Having been in Ops for many many years and having worked on both a
well built nationwide network with a dedicated management/oob
infrastructure that is completely separate from the CDN and working on
a not so well built nationwide network that is built as cheap as
possible with VPN's running over the production CDN.. I would highly
recommend separating the two.
No amount of policies or procedures will prevent your management
network from going down during critical times. In my experience both
MTTR and the over all sanity of anyone working on that network starts
to go down the drain as they are always worried about impacting
management and isolating themselves, or during an outage unable to fix
the problems at hand in a reasonable amount of time.
I understand not everyone can spend the money to have a dedicated
management infrastructure, but it's well worth every penny when done
correctly.
to administrate our core backbone routers, management is done inband, the
OOB is only for backup solution when the router is not reachable.
Others things (like our DWDM infrastructure which is RFC1918 addressed), we
use the OOB for the administration.
Our OOB is done this way :
Our principal core infrastructure is in Paris and we have our own dark fiber
backbone there, we decided to have a 'core oob infrastructure' : a layer 2
network dedicated for the OOB is built to cover all our pops (with spanning
tree for path protection) on dedicated dark fibers. On all pops we have
console servers (Opengear) that allow to access our routers console ports
remotely.
We also have 2 smalls Juniper firewalls in cluster to connect the 'outside
Paris' remote sites with VPNs.
On the pops outside Paris we have a basic layer 2 switch, a firewall, a
console server and we take IP connectivity from somebody onsite, the
firewall has a VPN to the 'core oob infranstructure' in Paris which allow us
to access everything.
The IP connectivity on the core oob infrastructure is provided by our
network with a backup IP connectivity from another provider which allow us
to access everything in our backbone in case of a total blackout on our AS.
By VPN I meant a L3VPN for management only functions, and if there is a
L3VPN for management
does anyone extend that to managed CERs? I assumed running and MPLS SP core,
sorry.
I think a remote kit for console, ethernet, power is pretty standard I am
really interested in the other management
data for overall management like monitoring, flowdata, traffic analaysis,
authentication, logging, etc ....
Is this done in band or onthe dedicated OOB network?
Fully acked, but with every service migrating to IP how do you make sure
that the oob infrastructure is completely different from your production
network?
We use a management VRF (which is still technically in-band, but otherwise quite
isolated), plus a few terminal servers for out-of-band console access to the critical
devices.
Honestly - in our core network, this has only happened once in almost 10
years... seriously. Everything in our core networks is redundant ... yes, I
know redundancy breaks of course
When it did happen, we had remote hands reboot the equipment and everything
was restored in approximately 30 minutes.
I'm not saying boldly that we won't get caught with our pants down some day
- just that previous experience has shown us to be prepared for the worst
and the worst hasn't occurred. We have looked at OOB options and it's been
discussed many times - it just slips off the radar constantly. Maybe it's
"once bitten, twice shy" that needs to occur for the priority to change
again.
Honestly - in our core network, this has only happened once in almost 10
years... seriously. Everything in our core networks is redundant ... yes, I
know redundancy breaks of course
I hear you.
When it did happen, we had remote hands reboot the equipment and everything
was restored in approximately 30 minutes.
lucky that the breakage wasn't in east-elbonia...cause that does suck.
"yea, we'll have to get someone on a plane, it'll be up in about 8
hrs..."
I'm not saying boldly that we won't get caught with our pants down some day
- just that previous experience has shown us to be prepared for the worst
and the worst hasn't occurred. We have looked at OOB options and it's been
discussed many times - it just slips off the radar constantly. Maybe it's
"once bitten, twice shy" that needs to occur for the priority to change
again.
perhaps. but given a clean slate, would you:
1) live with more redundancy in the core and hope that you don't lose
access to things downstream from a problem (or the problemchild
itself)
2) think about a solution to provide OOB access via another infrastructure?
Presume you can figure the costs as well so loss of a
node/set-of-nodes SLA-wise is more expensive than 1yr of oob access?
1) live with more redundancy in the core and hope that you don't lose
access to things downstream from a problem (or the problemchild
itself)
2) think about a solution to provide OOB access via another infrastructure?
Presume you can figure the costs as well so loss of a
node/set-of-nodes SLA-wise is more expensive than 1yr of oob access?
-chris
for those w/ OOB built on PSTN or other non-IP based fabrics,
how much would it hurt if the PSTN went away?
We have a management VRF/L3VPN across the network but we also have automatic backup via a OOB network. The OOB network is cheap, based on 3rd party ADSL connectivity that does not touch our network or rely on IXs etc.
We then run IPSec over the ADSL network.
The probability of both our node AND the DSL going down at the same time is minimal, especially as the DSL is copper to a local exchange.
> We do everything in-band with strict monitoring/policies in place.
what do you do if your in-band fails? if a router/switch/ROADM is
isolated from the rest of your network?
(isn't that the core point of the OP?)
Vendor C sells nice small routers with something like CAB-OCTAL-ASYNC
_and_ a 3G modem instead of the BRI port. The 3G modem keeps its
connection up (our telecom provider has true flat rate on domestic 3G,
YMMV) and VPN's to the head office much like any other telecommuter. This
cuts through all telco stupidity with firewalled or NAT'ed 3G phones
etc, especially if one uses the break-out-from-hotel-LAN functions of
the VPN system. The router of course actively keeps the VPN up and
reestablishes it if needed.
how well does that work inside a big metal box like equinix?
You are, of course, just making a singular point: "Find something to
make yourself an OOB network, hey this thing does vpn over 3g, neato!"
I agree, it's neat.. it may not fit all square holes, sometimes you
need a round or triangle shaped plug.
My measured availability for a automatic reverse ssh tunnel connection made through a 4g radio in the field was 52%. this was vs 95% on the lab/office environment with the same equipment. That particular experiment I declared a failure.
how well does that work inside a big metal box like equinix?
Sometimes not so well. OTOH, this is for routers that are installed
in central machine rooms for a radio broadcaster. Faraday issues are
indeed present, but we've managed with a top-of-rack antenna, so far.
(The FM transmitters are off-site and outsourced, anyway)
You are, of course, just making a singular point: "Find something to
make yourself an OOB network, hey this thing does vpn over 3g, neato!"
I agree, it's neat.. it may not fit all square holes, sometimes you
need a round or triangle shaped plug.
The facilities that are somewhat prepared to survive EMP aren't so
forgiving. Modems still have their place, as has the dedicated glass
strand. In some EMP shielded sites I've had predictable trouble getting
copper lines in, even after pointing out the availability of milspec
filtering devices.
