Open Souce Network Operating Systems

If one were to deploy whitebox switches, X86 servers, low cost ARM and
MIBPS CPE devices, and basically anything that can run linux today, what
network operating system would you recommend? The goal would be to have a
universal network operating system that runs across a variety of devices.

From low cost residential CPE's with wifi to switches to BGP speaking

routers. Is there anything that can do it all today?

I will use something like OpenWRT as an example. I don't consider this
anywhere near carrier grade, but it runs on X86 and low cost routers. I
don't think it will run on whitebox switches though.

Mikrotik RouterOS would be another example as it can run on low cost
Routerboards, and X86 servers. But it is not opensouce.

Is there any up and coming projects to look into?

Hey,

Have a look at a similar thread from recently:
http://seclists.org/nanog/2018/Jan/180

/Ruairi

There's AT&T's dNOS effort[1], though I think that wasn't really targeting CPE so much as DC and carrier type WAN gear. A single platform for DC, aggregation, and other SP roles is already pretty ambitious. Adding CPE into the mix as well is another big stretch even beyond that.

It's also more at the "initiative" stage than anything fully fledged, afaict.

Is there anything that can do it all today?

VyOS, maybe. You'd have a fun time getting it working across the full set of hardware you're thinking of though

And for certain uses, that would be handy.

Of course it would also be handy to an attacker who found or purchased
or was given an exploitable hole in that OS, because then they could pwn
the entire intrastructure simultaneously. One-stop shopping, as it were.

---rsk

Yeah, it'd be silly for organisations to try and standardise their environments for services or infrastructure.

I'm somewhat in two minds there. Options to tackle operational complexity/expense:

Option 1: Require a homogeneous environment or minimize vendors/platforms as much as possible.

Option 2: Accept vendor/platform diversity as inevitable and build systems/abstractions around that.

Is #1 achievable? If you're expending time/effort/resources achieving #1 and fall short, don't you have to do #2 anyway?

Much has also been said on monocultures in infrastructure: having a single bug impact all of your gear sucks. If I can manage a pair of border routers, for instance, from two different vendors in an abstracted/consistent enough manner that I don't deal with their idiosyncrasies on a daily basis, am I not better off than running a single platform / code train in that function?

Yeah, it'd be silly for organisations to try and standardise their environments for services or infrastructure.

Was this spoken tongue-in-check, or in all seriousness?

I would say there is power in standardization. And, yes, there is risk depending upon your attack surface.

When we concern ourselves with the attack service, how much time do we spend on mitigating issues on the edge (the attacked surface), vs internal infrastructure where we can apply labour and effort saving automation and orchestration?

I'm somewhat in two minds there.� Options to tackle operational complexity/expense:

Option 1: Require a homogeneous environment or minimize vendors/platforms as much as possible.

Maybe deal with on a case by case basis? Or implement solutions which are 'easy' to mitigate?

Option 2: Accept vendor/platform diversity as inevitable and build systems/abstractions around that.

And do this in an intelligent manner, depending upon management and risk profiles.

Infrastructure tends to be wide ranging. When one thinks about the big picture, where do you _really_ need the diversity, and where you can you gain the most by standardization? Standard engineering response: it depends.

And I hope that readers are not trying to draw a line in the sand. I'm hoping that we are open to optimization and orchestration based upon the infrastructure at hand.

Is #1 achievable?� If you're expending time/effort/resources achieving #1 and fall short, don't you have to do #2 anyway?

Doesn't this go the other way? ... that if you are spending so much effort that building infrastructure, and you can't get a maintainable/upgradeable/orchestratable solution in place, is #2 relevant?

Much has also been said on monocultures in infrastructure: having a single bug impact all of your gear sucks.� If I can manage a pair of border routers, for instance, from two different vendors in an

but when I think about this, I'm not thinking about just border routers, I'm thinking about core routing, virtualization infrastructure, carrying customer private circuits, delivering traffic to individual customers, gear for telemetry, implementing security, ....

When you think about all the devices involved in various levels and styles of service delivery, there are ways to make that homogenous, and much of it has various attack surfaces, and, well, vendors have their strengths and their weaknesses, so ...

#1 a homogenous network makes it easier to intimately understand the possible weaknesses, and attempt protection mechansims, but for

#2 with multiple vendors, the effort on platform education increases, depending upon the size of your shop, ...

abstracted/consistent enough manner that I don't deal with their idiosyncrasies on a daily basis, am I not better off than running a single platform / code train in that function?

or across many functions?

#Devil'sAdvocate

If one were to deploy whitebox switches, X86 servers, low cost ARM and MIBPS CPE devices, and basically anything that can run linux today, what network operating system would you recommend?

Linux.

I fail to see the need for various packages that act like something else (or their own entity) to translate to underlying Linux configuration.

Why not just put things in the various configuration files that Linux uses?

Why have something that translates?

If you are going to have a translation layer, why not do it with standard automation layers that you're likely already using. I.e. Ansible / Puppet / Chef / Salt et al.

The goal would be to have a universal network operating system that runs across a variety of devices. >From low cost residential CPE's with wifi to switches to BGP speaking routers. Is there anything that can do it all today?

Aside from the concern that others have mentioned about the same vuln on many pieces of equipment, I don't see a problem.

Use a standard (or at least consistent in your org) kernel configuration (as necessary & appropriate for the hardware) and standard configuration of application & configuration.

Compile as necessary for the architecture that you're on.

I will use something like OpenWRT as an example. I don't consider this anywhere near carrier grade, but it runs on X86 and low cost routers. I don't think it will run on whitebox switches though.

What is different about OpenWRT vs VyOS? It's my understanding that they are both running Linux and have some sort of (proprietary) configuration layer abstracting away the kernel.

Mikrotik RouterOS would be another example as it can run on low cost Routerboards, and X86 servers. But it is not opensouce.

Is open source a requirement?

I consider it a strong desire, but not a requirement.

Is there any up and coming projects to look into?

Linux.

I would start with following the Free Range Routing project, and related but independent (and more tangible) projects like pfSense (esp. the upcoming 3.0 release) and Cumulus Linux. Going deeper, perhaps Carrier Grade Linux, DPDK, and ONOS (all Linux Foundation projects). I think scaling vertically from CPEs to core stack is a stretch, especially if you mean a DIY approach, however.

Eron,

Thanks for the advice.

My understanding if Free Range Routing is a package of software that runs
in linux, but not a full and true NOS right?

Is pfSense 3.0 going to be dramatically different that the current version?
I never considered this a NOS but more of a firewall platform with some
routing capabilities.

I looked into Cumulus Linux, but it seems to only run on the supported
hardware which is while box switches. Can you run Cumulus Linux on a X86
server with intel NICs? Can you run Cumulus on a raspberry pi?

Ideally I think I am looking to a Linux operating system that can run on
multiple CPU architectures, has device support for Broadcom and other
Merchant silicon switching and wifi adapters.

My understanding if Free Range Routing is a package of software that runs
in linux, but not a full and true NOS right?

Why not consider Linux a NOS? Installing Free Range Routing adds control
plane protocols: BGP, OSPF, ISIS, etc.

I looked into Cumulus Linux, but it seems to only run on the supported
hardware which is while box switches. Can you run Cumulus Linux on a X86
server with intel NICs? Can you run Cumulus on a raspberry pi?

Cumulus Linux is basically Ubuntu with Free Range Routing pre-installed
along with a daemon that offloads forwarding from the Linux kernel to an
ASIC. CumulusVX is a free Cumulus Linux virtual machine that is useful for
staging / testing configurations since it has the same behavior as the
hardware switch.

On X86 servers with Intel NICs, just run Linux. Cumulus Host Pack can be
installed to add Free Range Routing and other Cumulus tools on the server.
Alternatively, you can choose any Linux control plane, automation, or
monitoring tools and install them on the hosts and Cumulus Linux switches
to unify management and control, e.g. Bird, collectd, telegraf, Puppet,
Chef, Ansible, etc.

Linux distros (including Ubuntu) are available for non-X86 hardware like
Raspberry Pi etc.

Ideally I think I am looking to a Linux operating system that can run on
multiple CPU architectures, has device support for Broadcom and other
Merchant silicon switching and wifi adapters.

If you consider Linux as the NOS then it already meets these requirements.

Peter,

Thanks for the information. Do you have a recommendation of which
distribution of Linux to use for this? Is there one that is more network
centric than another?

I would second Peter's advise. Colton, for you I would recommended you visit Cumulus' web site and follow their tutorials. That should provide you with enough insights for your next step.

Peter,

Thanks for the information. Do you have a recommendation of which
distribution of Linux to use for this? Is there one that is more network
centric than another?

Cumulus Linux, OpenSwitch, and Open Network Linux are all Debian based so
there is probably a greater ecosystem of developers and network centric
tools being built for Debian.

Feedback about Cumulus has been positive :

https://www.mail-archive.com/cisco-nsp@puck.nether.net/msg66192.html

if i am not mistaken, they have added lots of networking enhancements to
the OS, they have videos on youtube that will paint the picture.

Colton,

Maybe it is obvious to some, but I just want to point out that the reason
Cumulus Linux publishes list of supported hardware is kind of two fold:

1st is Linux inherently doesn't program the hardware. So if you install
Ubuntu on some Quanta switch, you still need a way to program the ASIC.
Cumulus Linux is open source with the exception of switchd, which is what
they use to take network state from the kernel and program the silicone
with it. switchd can only program "supported" silicon.

2nd (and I think that applies to any OS, not just a NOS) is that supported
hardware is the one they know how to (or taught linux to) control the LEDs,
the power supplies, the fans, etc. Technically you can run Cumulus Linux on
some unsupported switches, but I would not put that in production.

So if you accept that linux is an OS and it needs drivers to drive the
hardware, you can, hopefully, see that you can't just take any linux distro
and run it on just any switch. You will need drivers that this linux OS can
use to drive the hardware (including the ASIC) that you chose to run this
linux on.

Cumulus Networks does just that. They provide you with the drivers for the
ASIC and environmentals. And since they do it, they list the hardware they
support (aka provided the drivers for).

Hope this helps.
-Andrey.

--Andrey

❦ 3 mai 2018 13:39 -0700, Andrey Khomyakov <khomyakov.andrey@gmail.com> :

1st is Linux inherently doesn't program the hardware. So if you install
Ubuntu on some Quanta switch, you still need a way to program the ASIC.
Cumulus Linux is open source with the exception of switchd, which is what
they use to take network state from the kernel and program the silicone
with it. switchd can only program "supported" silicon.

Since a few years, Linux has an offload framework for L2/L3
(switchdev). There is a toy driver (Rocker, supported by QEMU) and
several silicons supported (at least Mellanox Spectrum, but it seems
there are a few others).