I do not believe that the linux scheduler is run to completion, but to
be honest I'm not 100% certain. I know a big reason for IOS-XE was to
It certainly is not, I'm not proposing it is. I'm saying it is bit of a
stretch to believe that IOSd does not have own legacy scheduler and memory
management as pulling that switch would have been quite major rework.
be able to operate in multicore environments. From a high level you
have IOSd as a process with each traditional process (BGP, OSPF, IP
Input) as a thread within IOSd. Overall IOS-XE is Linux managing a few
processes: IOSd, FMan-RP, CMan-RP (and a few others) FMan deals with
adjacencies and CMan deals with modules/cards and IOSd all the
interesting stuff. Since Linux is the piece actually running the show
IOS-XE gets all the memory management and scheduling benefits that
linux has.
So each IOSd process 'show proc cpu' are separate threads to linux?
So each IOSd process 'show proc cpu' are separate threads to linux?
Yep. The "show platform software..." commands are used to look at things in
software, outside of IOSd. Don't get me started on how absurd the command
lengths are.
To be honest I'm very sceptical about this. I fully accept that IOSd is
multithreaded. But I'm having difficulties accepting that each IOSd process
would be own thread scheduled by Linux and that native/IOS
run-to-completion scheduler isn't used at all.
But we're out of scope for ISSU thread in nanog-ml, I think. I do
appreciate always when vendors pitch in in public forums, so thank you for
that Pete.
Someone who has ASR1004 just ran 'ps auxH' and it's running 3 threads. So
IOS XE control-plane is for most parts run-to-completion and relies on
classic IOS scheduler and memory-management.
I'm not saying this is inherently bad, it is least overhead way to do it,
but it also sets requirement for code quality unrealistically high.
I'm pretty sure it would be easier to start from scratch and loan IOS BGP,
ISIS, OSPF etc code to new project than to try to make IOS be pre-emptive
and SMP safe.
I'm also not saying IOS XE is bad, JunOS is same design and many consider
JunOS good design (I'm not one of them). I think XR and NX-OS are better,
more modern examples how to do things now, when we have some extra CPU time
in control-plane.
It would be interesting ti know what are the roles of these 3 threads. If
I'd have to guess, one is IOS control-plane, one is emulation of IOS
interrupts and one is abstraction for hardware forwarding. But this is
likely far off.
NANOG56? I only found RPKI Propagation by you. Direct URL would be
appreciated.
But I really have 0 doubt that IOSd is run-to-completion, exactly like RPD
is. But IOSd indeed seems to have 3 OS threads, while RPD is single
threaded. As I understand RPD does have green threads though, but that is
not helping at all making things simple for JNPR, more if anything, it's
making things more complex.
If native process separation and even native thread separation cannot be
made scale (I'm highly suspicious why it couldn't). Then I wonder why
vendors don't use some existing VM, instead of inventing their own. Many
existing are free and support green threads and native thread and
many-to-many mapping between, so you could get benefit of minimal overhead
of green thread and you get benefit of OS level threading (SMP, scheduling)
to compartmentalize processes.
NANOG56? I only found RPKI Propagation by you. Direct URL would be
appreciated.
apologies. i slid a second preso into my time allotment, and i
thought the archive maintainer was going to catenate them. here is
the second preso, some measurements of the tcp behavior of bgp.
But I really have 0 doubt that IOSd is run-to-completion
nor i. but, as i said but did not write in my preso, consider the
following:
linux has become a fad in the vendor community. it seems to lend
legitimacy to their products in some way, witness this discussion.
but linux has the gpl poison. so, any code that they wish to keep
proprietary is in userland.
I've sometimes wondered why Linux is so common, and not FreeBSD. Is it
easier to hire people if you use Linux? Or is GPL not really problematic
issue, as you can hide your intellectual property in binary kernel modules?
I've sometimes wondered why Linux is so common, and not FreeBSD.
Historical reasons and good timing.
Is it easier to hire people if you use Linux?
As opposed to... ?
Or is GPL not really problematic issue,
as you can hide your intellectual property in binary kernel modules?
You can't. The GPL has provisions for that. Common mistake, several
lawsuits have shown.
As Randy pointed out, Juniper is FreeBSD inside, and NetApp uses
it as well (+ number of other vendors that don't advertise it because
they don't have to).
In article <xs4all.m2lie9nhko.wl%randy@psg.com> you write:
linux has become a fad in the vendor community. it seems to lend
legitimacy to their products in some way, witness this discussion.
but linux has the gpl poison. so, any code that they wish to keep
proprietary is in userland.
Which isn't really a problem, none of the control plane stuff needs
to run in the kernel. The only thing that needs to run in the
kernel is the device driver(s) to talk to the forwarding plane
hardware, but if you use ethernet or infiniband for that
communication you don't need any proprietary drivers.
Years ago, Linux was relatively immature and FreeBSD wasn't. Vendors
like Juniper, NetApp, Apple, etc., took whatever suited them from
FreeBSD, often ran it on x86, and went on their way. The relatively
low legal bar presented little problem, as was intended.
Over the years, Linux was ported to more platforms, and has matured a
good bit, so now the "obvious" choice when someone was looking for a
cheap platform to build a residential CPE or NAT gateway or whatever
became Linux. At the same time, large companies have poured lots of
dollars and man-hours into Linux, and this has improved code quality
and maturity. However, there are still legal issues relating to the
GPL.
If you're on supported CPU's, the BSD's are likely to be a better
choice if you want to avoid legal entanglements. Otherwise, if you
don't mind code disclosure, Linux supports more platforms. Both
are relatively mature, feature-full operating systems when used for
embedded applications.
GPLv2, which governs the Linux Kernel, does tolorate use of
binary kernel modules under some conditions (the classic
example is the nVidia driver blob which uses a GPL shim).
Regardless, most lawyers would advise a company to avoid
being a test case for some of the poorly defined terms used in
the license, including "derivative work". A recent paper
discussing the issue can be found at:
"LOADED QUESTION: EXAMINING LOADABLE KERNEL
MODULES UNDER THE GENERAL PUBLIC LICENSE V2"
If you're on supported CPU's, the BSD's are likely to be a better
choice if you want to avoid legal entanglements. Otherwise, if you
don't mind code disclosure, Linux supports more platforms. Both
are relatively mature, feature-full operating systems when used for
embedded applications.
If your silicon vendor supports BSD's, of course.
From my (little) experience most vendors SDK will be available to
Linux and vxWorks but not BSD.
This limits companies that are building equipments based on third
parties ASIC to use anything but Linux.
Which isn't really a problem, none of the control plane stuff needs
to run in the kernel. The only thing that needs to run in the
kernel is the device driver(s) to talk to the forwarding plane
Yes. But avoiding kernel mode is a consideration, even before GPL.
Perhaps GPL is just another force to discourage developers from doing what
they shouldn't be doing anyways -- which is to insert complicated code in the
kernel itself to do application-specific things, instead of
providing hardware interfaces
for applications.
You introduce risks if you run control plane things in kernel mode
ring0 and not separate control plane functions into user processes.
Risks that buggy code will be executed with privilege and corrupt
critical data.
Risks that a buffer overflow in the SNMP code will crash the kernel
and cause the entire control unit to reboot.
If instead, each control function is a separate user process, running without
privilege in protected mode, then you have a larger amount of fault isolation
provided by the hardware -- restart the SNMP process automatically,
but leave ISISd/Bgpd alone, and no kernel panic...
If your silicon vendor supports BSD's, of course.
From my (little) experience most vendors SDK will be available to
Linux and vxWorks but not BSD.
This limits companies that are building equipments based on third
parties ASIC to use anything but Linux.
You are right, of course, since the silicon vendors
customers decide what they want the device to support,
and that is (currently) Linux and VxWorks. Some BSD
folk are trying to change that, by investing their time in
the patches/ports needed to support additional
embedded processor types/derivatives and make it a
viable platform. There is even a Raspberry Pi port now
available for FreeBSD as I recall. Ideally those efforts
will produce a viable ecosystem for BSD in this space.
The equipment vendors call upgrades with NSR failover, ISSU; if their
marketing people feel that a 0.5 or 6 second hit is "good enough"..
If you care about the 0.5 seconds, it's important you speak their
language, and require that vague expressions such as "In-Service
Software Update" be clarified.
Personally, I don't trust any of it; routers should have regular
maintenance windows, period, with a minimum duration of 30 minutes.
And software updates to fix known bugs should be done regularly, and
during those windows.
NSR for ISSU, or ISSU with a small hit called ISSU, is likely
inexpensive for the network equipment vendors, because they already
invested hundreds of thousands of developer hours in implementing and
validating NSR functionality to provide redundancy against device
failure.
The process of replacing code on a hot device, and restructuring any
stored data to match expectations of the new code, without suspending
or delaying execution of any code during that process, is possible,
but a non-trivial problem: whose solutions add complexity (and
therefore a higher risk of bugs and unexpected results) to the upgrade
process.
You might reduce the hit from 0.5 seconds to 0.01 seconds by
implementing true in-place upgrade 90% of the time; but 10% of the
time, the online upgrade either fails, because of an issue with the
online patch, or unexpected interactions between partially patched
functional units, result in a period of incorrect device operation
--- until the patching finishes, and continued use of bad data even
after patching finished.
ISSU contains the wording "in service". 6 seconds of outage isn't "in
service". 0.5 seconds of outage isn't "in service". I could accept a few
microseconds of outage as being "ISSU", but tenths of seconds isn't in
service.
What is the maximum percentage more would your organization be able
to justify paying the network equipment vendor for routers/switches,
to reduce the ISSU hit from 0.5 seconds to a few microseconds?
We do it on our Class 5 softswitch ... and it works consistently. There may
be a few seconds, once, where a new call can't be made, but most people will
re-dial. It just works.
It can be done, but the product has to be built with that in mind.
