OOB core router connectivity wish list

Mailman List Mirror (Read Only)
21

My main requirements would be:

1. Something that is *not* network (ethernet or otherwise) (isn't that the point of OOB?)

I don't understand this at all. Why can't an OOB network be ethernet based towards the equipment needing management?

2. Something that is standard across everything, and can be aggregated easily onto a "console server" or the like

Yes, ethernet is the proposed management standard interface.

I don't really see what is wrong with with keeping the serial port as the standard.

Because it's slow and can't be multiplexed, and it's expensive, only does short distance (20 meters or so), uses different cabling, requires separate planning etc. There are lot of reasons to drop serial port support.

Things like servers and RAID cards and such are coming with "BIOS"es that are graphical and even require a mouse to use. What use is that when I need to get into the BIOS from a remote site that is completely down?

My email stated https was way down on the priorities, and ssh and telnet was the prime way of managing the OOB part.

It might be nice to have a "management-only" port of some sort to do more advanced things that serial cannot do, but the serial port is ubiquitous already, and I don't see any reason to remove it as the very low-level access method.

An ethernet port is generally a lot cheaper compared to a serial port. Your OOB network would consist of a switch or router with ethernet towards the equipment needing management.

22

> My main requirements would be:
>
> 1. Something that is *not* network (ethernet or otherwise) (isn't
> that the point of OOB?)

I don't understand this at all. Why can't an OOB network be ethernet
based towards the equipment needing management?

How do I connect to it from many miles away when the network is down? I have connected to a misbehaving border device at a remote network via dial-up before, and was able to get it back up and running. I would not have been able to do that if the only options were ethernet or ethernet.

> 2. Something that is standard across everything, and can be
> aggregated
> easily onto a "console server" or the like

Yes, ethernet is the proposed management standard interface.

> I don't really see what is wrong with with keeping the serial port
> as the standard.

Because it's slow and can't be multiplexed,

agreed. Although, I generally don't care if it is slow, as the only need is to get the real network connected features back up and running.

and it's expensive, only
does
short distance (20 meters or so),

I completely disagree. The ability for serial to go over POTS makes it ridiculously cheap compared to building a reliable ethernet connection over hundreds or thousands of miles.

uses different cabling, requires
separate planning etc. There are lot of reasons to drop serial port
support.

The separate part is what makes it useful. The only reason you should need to access the serial port is because the network is not functioning. If you move the last resort access to be network, how do you access it when you have network issues?

An ethernet port is generally a lot cheaper compared to a serial
port.
Your OOB network would consist of a switch or router with ethernet
towards
the equipment needing management.

But having a console->serial is significantly less complex than console->IP_Stack->ethernet. So many more things to go wrong. I've never had a device that had a faulty serial port. I have seen numerous faulty or misbehaving network ports.

I like the current trend of vendors like Juniper. Dedicated management ethernet, *and* serial console port. Best of both worlds.

-Randy

23

That is task for on-band interfaces, which attach to your forwarding-logic.
OOB is separate component, really only relying on same powersupply, it's
key value that it is not fate-sharing any more than absolutely must with
host.
To export flow, you need port to be connected to your forwarding hardware,
not control-plane and certainly not OOB management-plane.

The port sole purpose is disaster recovery, it's like NG RS232 port, it
would not be used as your day to day port to configure the box.

Cheack Cisco's "CMP" this is what we need.

24

1. Something that is *not* network (ethernet or otherwise) (isn't that the point of OOB?)

No. This is not what OOB means. Out-of-band means not fate-sharing your
production network. OOB networks are networks, running ethernet,
frame-relay, ATM or something else. Just that they are broken at different
time to production network, so if you fuck up your production network, you
can still access your device via OOB network.

The serial port you have in your router is not OOB port, it's fate-sharing
the control-plane, if your OS breaks down, your serial port. Think IPMI,
if you break your linux installation, you can't login to linux to reload
the linux, you need IPMI/DRAC/vPro for that, which is normal ethernet.

2. Something that is standard across everything, and can be aggregated easily onto a "console server" or the like

'console server' costs more than ethernet switch. So you get less and you
pay more.
You can't you use RS232 to send images.

I don't really see what is wrong with with keeping the serial port as the standard.

Then you don't have to use one, you're welcome to use existing solutions,
this is not robbing that that on-band RS232 from you. It's adding something
new. Cisco devices which have CMP still have legacy on-band RS232, because
not all people will realise immediately why and even if they do, not all
people can migrate it day1 in non-greenfield.

25

I completely disagree. The ability for serial to go over POTS makes it ridiculously cheap compared to building a reliable ethernet connection over hundreds or thousands of miles.

This is identical to ethernet. You need external device then, dial-up
modem or CPE, no difference.

The separate part is what makes it useful. The only reason you should need to access the serial port is because the network is not functioning. If you move the last resort access to be network, how do you access it when you have network issues?

Or because device is not functioning, and if OS is broken, so is your
'OOB'.
Or maybe you fucked up upgrade and corrupted image? No way to recover over
RS232 (RS232 image upload not supported anymore, even if it would be,
Juniper is hard set to 9600bps, even if it would support 115200 it would
take 12h to upload image, faster to fly with usb key).

But having a console->serial is significantly less complex than console->IP_Stack->ethernet. So many more things to go wrong. I've never had a device that had a faulty serial port. I have seen numerous faulty or misbehaving network ports.

Only thing matters is that it fails at different time to production
network.

I like the current trend of vendors like Juniper. Dedicated management ethernet, *and* serial console port. Best of both worlds.

This is fully on-band port ethernet-port, relies 100% on the host OS being
up and running.
Replace this port with true OOB ethernet, keep RS232 for people who can't
migrate day1.

You never introduce new thing and kill old thing day1, this is not how
things work. You add new thing, then after good amount of time phase-out
old thing.

26

How do I connect to it from many miles away when the network is down? I have connected to a misbehaving border device at a remote network via dial-up before, and was able to get it back up and running. I would not have been able to do that if the only options were ethernet or ethernet.

You have the same kind of "console router", but instead of having lots of serial ports, you have ethernet ports on it. Or you use your DWDM system OOB channel (if available). There are a lot of options.

agreed. Although, I generally don't care if it is slow, as the only need is to get the real network connected features back up and running.

With modern routers the software is a big image so if you need to re-install, serial is dysfunctional (if it even works, on XR devices it doesn't).

I completely disagree. The ability for serial to go over POTS makes it ridiculously cheap compared to building a reliable ethernet connection over hundreds or thousands of miles.

RS232 doesn't go hundreds of thousands of miles. A modem connected to a POTS port does this though.

The separate part is what makes it useful. The only reason you should need to access the serial port is because the network is not functioning. If you move the last resort access to be network, how do you access it when you have network issues?

You seem to be totally misreading everything I'm saying. Did you even read my initial email? This proposed port is OOB, it's completely separate, it's like having a built in serial console router into the chassis with connections to line cards and other devices within the chassis.

I like the current trend of vendors like Juniper. Dedicated management ethernet, *and* serial console port. Best of both worlds.

No it isn't, because if you want to be able to cut power to the device to power cycle it, you need yet another device to do that that sits inline with the power feed and if it's DC then I would imagine there aren't that many options (=expensive).

27

"CMP" this is what we need.

+1000

28

That is task for on-band interfaces, which attach to your forwarding-logic.

No it isn't, any more than SNMP is a task for those interfaces.

To export flow, you need port to be connected to your forwarding hardware, not control-plane and certainly not OOB management-plane.

Again, the analogy is with SNMP. There's no requirement to be part of the data-plane, it's quite possible to get the flow telemetry to the management processor, same as with SNMP.

Cheack Cisco's "CMP" this is what we need.

I'm quite familiar with it, thanks.

29

Well, then what you're looking for is not what we're looking for (?). You seem to want the type of classic mgmt ethernet currently residing on high end router platforms (on the RP) and not a ILO/CMP type interface that we're looking for.

I definitely do not want SNMP and netflow on my disaster recovery OOB network.

30

Of course you do - else you're deaf, dumb, and blind at precisely the time you most need complete network visibility, i.e., during a disruptive event of some sort.

The ability to type commands via ssh and/or console ports isn't very helpful if one lacks enough context to know what to type, heh.

31

Of course you do - else you're deaf, dumb, and blind at precisely the time you most need complete network visibility, i.e., during a disruptive event of some sort.

You and me seem to talk about different types of disasters. In my type of disaster, SNMP and netflow doesn't work because the RP is out of commission or seriously malfunctioning.

The ability to type commands via ssh and/or console ports isn't very helpful if one lacks enough context to know what to type, heh.

I don't know what to respond to this because I don't understand what you're getting at.

32

No it isn't, any more than SNMP is a task for those interfaces.

Sending flowrecords to your slow ppc CPU just to allow export in non-HW
interface is silly, when HW can export it directly, without ever hitting
your control-plane.
Polling SNMP is low volume, so you easily allow RP to poll for them from
HW. But implementing this also in HW would be interesting for low-interval
SNMP polling, then it also would stop working in your non-HW interfaces.

Again, the analogy is with SNMP. There's no requirement to be part of the data-plane, it's quite possible to get the flow telemetry to the management processor, same as with SNMP.

Sure, if performance is not important and if you're too poor to buy
interface in your router.

33

We have encountered cases where a vendor TFTP implementation + latency from the ROMMON can take a few hours to load images. I'm for ditching TFTP and replacing it with HTTP. This forces them to put in a TCP stack, and hopefully something that can window-scale and deal with the latency vs 'wait for block', ok, req next block..

The testers involved in their labs are never loading an image from 1600km away so don't get to enjoy this 'fun'.

- Jared

34

I am very much against USB consoles. there can be a whole plethora of issues involved from OS-level to the device-level. When I'm on the console, things have already gone bad. I don't need to find out if the vendor has the right 'entitlement' established for me to download and load the driver or anything else..

It *needs* to work, I can't wait for the device on the other end to negotiate with the host system, etc..

I understand why people want it, but USB as it exists today isn't the way. (I can screw down a rs232 connector and it can be secure, I can't attach USB with the same certainty).

- Jared

35

I'm certainly not rooting for USB console, I don't want to fix broken
solution with another broken solution.
I'm all for Ethernet OOB (true OOB, not fate-sharing control-plane),
exactly like CMP in Cisco.

36

I absolutely agree that USB is a bad way to go with this, as well as web
management.

I have no interest in trying to use some terrible web app to bring a
network back up when simple 300 baud would suffice. I've got no problem
with telnet/ssh, although I hate the idea of needing to know an ip address
to emergency jack in to a device instead of just a bit rate, but please no
web app.

-Blake

37

Re: other comments:

  - tftp: I've run into enough problems with stupid tftp incompatibilities
that I'd be really happy never having to use it again in my life.

  - netflow: seriously, this is not an appropriate sort of port of exporting
netflow. this is a "your RP is toast" recovery mechanism, at which point
netflow is probably long gone.

   - rs232: please no. it's 2013. I don't want or need a protocol which
was designed for access speeds appropriate to the 1980s.

  - USB: no. can you route USB? No. DNW.

  - original list: sounds great, except that I want ipv4 and ipv6 given
equal priority for mgmt access.

Nick

38

        - netflow: seriously, this is not an appropriate sort of port of exporting
netflow. this is a "your RP is toast" recovery mechanism, at which point
netflow is probably long gone.

it's possible that roland was saying that the oob network should
collect flow records and export them to 'something' so you'd have an
idea about what traffic was on the network... I can see some value in
that.

I don't think roland was really saying that normal netflow from a
device in production pushing a few hundred gbps of traffic would be
appropriate to ship out the OOB network... or I hope that wasn't his
point. I don't think oob networks need to be sized for that.

I do think that having a reliable OOB Ethernet would be nice, having
it not be part of the forwarding plane (and not reachable from the
forwarding plane) of the device in the field would also be nice.
iLO/DRAC are good analogies...

        - rs232: please no. it's 2013. I don't want or need a protocol which
was designed for access speeds appropriate to the 1980s.

I don't think you can get ethernet and transport out-of-the-area in
some places at a reasonable cost, so having serial-console I think is
still a requirement.

-chris

39

I don't think you can get ethernet and transport out-of-the-area in
some places at a reasonable cost, so having serial-console I think is
still a requirement.

TDM is disappearing quickly in at least some parts of the world. We
may not be quite there yet, but I think it's entirely reasonable to
start asking for Ethernet console in procurement documents.

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

40

I don't understand this argument.

Are you connecting your CON directly to something that transports it out-of-the-area? Modem?

If you have a consolerouter there with T1 interface as link to outside world, what's wrong with having ethernet port from that T1 router to the ethernet OOB port on the router needing OOB access, instead of having RS232 port on them. It's cheaper and easier to cable ethernet compared to RS232. RS232 has much shorter cable length compared to ethernet (9600 reaches 20 meters or so).