Curious if anyone has particular hardware they like for OOB / serial management, similar to OpenGear, but preferably with 5G support, maybe even T-Mobile support? It’s becoming increasingly difficult to get static IP 4g machine accounts out of Verizon, and the added speed would be nice too. Or do you separate the serial from the access device (cell+firewall, etc.)?
Thanks!
You could get a 5G Catalyst with an async NIM or SM.
But I think you're setting up yourself for unnecessary costs and
failures by designing your OOB to require static IP. You could design
it so that the OOB spokes dial-in to the central OOB hub, and the OOB
hub doesn't care what IP they come from, using certificates or PSK for
identity, instead of IP.
Yup, I agree — but that simply rewrites the question to be:
“Curious if anyone has particular hardware they like for OOB / serial management, similar to OpenGear, but preferably with 5G support, which can be a spoke that dials in to the central OOB hub, and the OOB hub doesn’t care what IP they come from, using certificates or PSK for identity, instead of IP.”
I’ve been on the same quest, and I have some additional requests / features. Ideally it:
1: would be small - my particular use-case is for a “traveling rack”, and so 0U is preferred.
2: would be fairly cheap.
3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried that for a while, and it was clunky — the SD card died a few times (and jumped out entirely once!), people kept futzing with the OS and fighting over which console software to use, installing other packages, etc.
4: support modern SSH clients (it seems like you shouldn’t have to say this, but… )
5: actually be designed as a termserver - the current thing we are using doesn’t really understand terminals, and so we need to use ‘socat -,raw,echo=0,escape=0x1d TCP::’ to get things like tab-completion and “up-arrow for last command” to work.
6: support logging of serial (e.g crash-messages) to some sort of log / buffer / similar (it’s useful to be able to see what a device barfed all over the console when it crashes.
The Get Console Airconsole TS series meets many of these requirements, but it doesn’t do #6. It also doesn’t really feel like they have been updating / maintaining these.
Yes, I fully acknowledge that #3 falls into the “Doctor, Doctor, it hurts when I do this” camp, but, well…
W
Doesn’t meet #3 but I’m testing Banana Pi BPI-R3 and seems way better than RPI for this purpose.
You need to add the mini-pci modem of your choice but their 2 SIM card slots on board. There are also 5 RJ45 ports if your devices have OOB ethernet ports.
There are 2 onboard storage (NOR and NAND) and you can add a M2 SSD so it is possible to have failover disks.
I also like the fact that there are 2 SFP ports. There are some places in our area where the LTE / 5G network is really awful so we can use a fiber wavelength instead. It depends on the same fiber but at least, doesn’t depend on any active devices on site.
The bad is that you still need a USB to serial ports adapter. Also, you can customise OpenWRT as much as you like.
For me, it’s an advantage but in your case, it seems like an issue. For the OP, having several VPN options like zerotier seems like an advantage.
I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G router. It has a 1x embedded 5G modem (Verizon, AT&T, T-Mobile, and FirstNet). three GigE ports, four antenna connectors, and comes with an stick antenna set and AC PS. It uses a nanoSIM. Yes, it’s a pure IP router with no knowledge of serial protocols. So I would just put an air console behind it to get to my serial ports. I’m still evaluating 5G plans, and Verizon just offered an amazing $15 per month unlimited data deal, but it seems to have a 50 gig limit before you get to throttling. That might not matter at all with serial traffic though.
We’ve been using the Netgear 4G cellular router, but that’s becoming increasingly unreliable. The NG has a nailed up IPsec VPN tunnel, obviating the need for a static IP, and the keepalive traffic is low enough that it doesn’t cost us much on the 4G network. I’m hoping 5G will be even cheaper and faster.
I’d love to see if anybody found anything better before I spring for a Peplink test unit.
-mel
If anyone is interested in https://freetserv.github.io/ but does not want to build one I have sort of documented an alternative at https://lathama.net/Tech/Hardware/USB-32COM-RM so you can use anything to connect the 5G or dialup to
If someone wants to assemble some of the freetserv boxes, I have some of the PCBs and components here if you want them.
- Jared
Does it? To me OP implied they need 5G, because they can get static in
5G product, but not on 4G. So if need for static is solved, they can
keep existing investments.
I've been on the same quest, and I have some additional requests / features. Ideally it:
1: would be small - my particular use-case is for a "traveling rack", and so 0U is preferred.
2: would be fairly cheap.
3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried that for a while, and it was clunky — the SD card died a few times (and jumped out entirely once!), people kept futzing with the OS and fighting over which console software to use, installing other packages, etc.
4: support modern SSH clients (it seems like you shouldn't have to say this, but… )
5: actually be designed as a termserver - the current thing we are using doesn't really understand terminals, and so we need to use 'socat -,raw,echo=0,escape=0x1d TCP:<termserver>:<port>' to get things like tab-completion and "up-arrow for last command" to work.
6: support logging of serial (e.g crash-messages) to some sort of log / buffer / similar (it's useful to be able to see what a device barfed all over the console when it crashes.
Decouple your needs, use whatever hardware to translate RS232 into
SSH, and then use 'conserver' to maintain 24/7 logging and
multiplexing SSH sessions to each console port. Then you have your
logs in your existing NMS box filesystem, and consistent UX
independent of hardware to reach, monitor and multiplex consoles.
For me Cisco is great here, because it's something an organisation
already knows how to source, turn-up, upgrade, troubleshoot, maintain.
And you get a broad set of features you might want, IPSEC, DMVPN, BGP,
ISIS, and so forth.
I keep wondering why everyone is so focused on OOB hardware cost, when
in my experience the ethernet connection is ~200-300USD (150USD can be
just xconn) MRC. So in 10 years, you'll pay 24k to 36k just for the
OOB WAN, masking the hardware price. And 10years, to me, doesn't sound
even particularly long a time for a console setup.
I tend to agree. Cisco do this very well, and if you are really low on cash and okay with acquiring these on the cheap, the open market has tons of deals and options from Cisco that have matured over the decades. Is a 10Mbps DIA link going for US$200 - US$300 MRC nowadays, excluding the x-connect? I’d have though it’s now in US$100 range at the very worst. Or are you looking at an OoB link of more than 10Mbps? Mark.
Quite often I’m looking for OOBM at antenna sites or in remote DCs where there is no Plan B carrier. Cellular has always been the goto choice for this, but we keep getting pushed out of contracts by technology upgrades. 2g, then 3g, and next 4g LTE are being deprecated.
The main reason for network shutdowns is that the carriers have limited spectrum available for expansion. To deliver faster, more cost effective data service to customers, carriers must re-use existing spectrum licenses with newer, more efficient cellular technology. Old 2G/3G infrastructure makes way for new networks, and older cellular devices must be retired. 4g may have a decade left before complete absence, but its footprint is already shrinking where 5G is available.
I’ve seen this first hand with 4g cellular alarm circuits: suddenly they get less reliable or fail completely, and the reason always turns out to be degraded RSSI due to 5G deployment.
So 5G is imperative for cellular OOBM, hence the hunt for COTS drop-in replacements that won’t break the bank. Upgrading, for example, 100 antenna sites is also a major truck roll cost, so we want to get it right the first time. Physical space and power limitations usually rule out 1U rackmount refurb Cisco terminal servers, which is why we need 0U gear. Yes, I can cobble together a raspberry pi and some hats and cables and dingles and dangles and make a science fair solution. But I need something that is commercially supported, won’t have me scratching my head later about what version of the Ubuntu is going to work, and won’t randomly fry its electronics during a power surge.
It’s looking like that solution is firmly priced at ~$500 today.
-mel
Peplink is nice, but there are cheaper options:
[
](MikroTik Routers and Wireless - Products)
Then for cellular service, sign up for an IOT with an IOT MVNO that bills usage based (and can also offer you a static, public, IP address AND will also allow you to build a VPN across all of your devices) such as SimBase:
[
Seamless IoT SIM Card Solutions
simbase.com
](Seamless IoT SIM Card Solutions - Simbase)
Cheers,
Mike
We use MikroTik for this. All manner of interfaces including LTE and 5G are available. I hear you can connect USB serial to them directly, but we also drop a surplus Dell OptiPlex at each location and attach the serial ports to that device. Total cost is <200 USD per site since we already have the older desktops laying around.
Fair enough - if the bulk of your OoB use-case is remote (cell) sites, your typical options won't work or will be limited.
Oddly, in our parts, we find remote, non-city locations, tend to keep their 3G/4G status, or don't even get considered for 5G at all. But I guess this will vary by market the world over, so I could see a remote site in Norway, for example, having 5G vs. a remote site in, say, Egypt, doing the same.
I think what you probably want to consider for the long-term is decoupling the device from the network media. If you can attach a MiFi router via a USB port to a cheap device (like Mikrotik), this would help keep costs down as mobile operators deprecate GSM data technologies in the future. I like Mikrotik because in addition to being cheap and feature-rich for basic network access, the firmware is regularly upgradeable unlike typical consumer-style CPE's.
Mark.
We use GL.inet and set up WireGuard VPNs back to our distributed VPN servers. Our console servers support dual uplink, so we just connect port 1 to the GL.inet LAN and port 2 to our management switch.
Currently, we’re still using their LTE model, and it costs ~100 USD per site, but their 5G models are expensive and cost around $500.
At new job, I am looking at using pfSense-based VPN's to create the DCN. It does consume 1U and a couple of cabinet watts for the server, but it's stable, feature-rich, well-supported, and network media agnostic.
Mark.
Yup, that’s the solution I mentioned above with #5:
“5: actually be designed as a termserver - the current thing we are using doesn’t really understand terminals, and so we need to use ‘socat -,raw,echo=0,escape=0x1d TCP::’ to get things like tab-completion and “up-arrow for last command” to work.”
Most term-server type things allow you to ssh / telnet to a TCP port and the device will expose the serial port wit some useful emulation.
The Mikrotik seems to only expose the serial interface this way using RFC 2217, which means that you need you need to use software that understands virtual Com ports (like ‘Serial’ on OSX, Tactical Software’s “COM Port Redirector”, or PuTTY or, on Unixes ‘socat’). This is far from convenient….
Michel’s Banana Pi BPI-R3 suggestion seems intriguing — yes, it still suffers from the “Now I have another “machine” to manage and patch, and people will try and install iperf / a Quake server / nmap / ruby / 17 different flavors of Emacs on it”, but:
1: Perhaps I can mitigate that by making much of the filesystem read-only and
2: it’s a great excuse to buy another toy!
I also like Jared and Andrew’s freetserv / https://lathama.net/Tech/Hardware/USB-32COM-RM option. I might see about building a bunch of the freetserv boards and connecting them to a Banana Pi…. although, more realistically, I’ll likely buy a few Banana Pi’s, and add them to the ever expanding pile of backlog projects…
I’m not really sure what happened to AirConsole — when they initially launched they were great. They were making new devices with new capabilities, they were updating the software regularly, they had great support, etc. At some point it feels like they changed management and everything sort of stopped…
W
I certainly don’t blame you for your frustrations with abusing MikroTiks as a serial console. The additional computer (Pi or otherwise) is, imo, a must. Unless you are just using the MIkroTik as an ssh jump box into the OOB network, which isn’t so bad.
Michel's Banana Pi BPI-R3 suggestion seems intriguing — yes, it still
suffers from the "Now I have another "machine" to manage and patch, and
people will try and install iperf / a Quake server / nmap / ruby / 17
different flavors of Emacs on it", but:
1: Perhaps I can mitigate that by making much of the filesystem read-only
and
2: it's a great excuse to buy another toy!
Another option would be a RaPi Compute Module 4 with eMMC
onboard. (ie. the non-Lite version does not need a SDCard)
They have breakout boards for the Compute Module that have spots for
5G WAN.
PiKVM does make the file system read-only, that you have to
temporarily disable to do updates, config changes, etc). Potentially, you could
copy what they do there.
I also like Jared and Andrew's freetserv /
Tech/Hardware/USB-32COM-RM - lathama option. I might see about
building a bunch of the freetserv boards and connecting them to a Banana
Pi…. although, more realistically, I'll likely buy a few Banana Pi's, and
add them to the ever expanding pile of backlog projects…
One suggestion for freetserv boards I have would be to get a board printer like https://jlcpcb.com
to also do the assembly of the surface mount components. Last time I checked, they
only did one side, Not both sides.. Have them do the complex side. Pricing was
very reasonable. The back non-complex side wasn't that bad overall.