Vyatta as a BRAS

Mailman List Mirror (Read Only)
21

I think Roland's point was that on "hardware routers", there is a
separation of function between the control and the forwarding planes, and
that the forwarding plane is designed to be able to transmit data in an
efficient parallel manner. I.e. on a well-designed hardware router, if you
trash the data path on the router through ingress A and egress B, the
damage stops there: the control plane is unaffected and ingress C to egress
D is also ok (for arbitrary values of C and D).

Depending on your configuration, this may or may not be important to your
IP connectivity requirements. For many - if not most - companies, it is.

Nick

22

Hi folks,

23

Cisco 7206VXF apparently had some issues dealing with it:

https://puck.nether.net/pipermail/cisco-nsp/2003-September/005578.html

Slammer's use of multicast addresses melted down at least a few large Cisco and
Juniper boxes:

http://paintsquirrel.ucs.indiana.edu/pdf/SLAMMER.pdf

I wasn't aware that the 7206 and M20 classified as software-based.

(cue weasel-words about those routers using ASICs for most forwarding, but
doing multicast forwarding in software in 5.. 4.. 3..)

24

No weasel words necessary.

I won't speak for the M20, but I've always thought of the 7206 as a software-routing platform - it's a pretty good swiss-army-knife software router which supports limited hardware acceleration of specific functions. Is there anyone who considers the 7206 a "hardware" router?

David Barak
Need Geek Rock? Try The Franchise:
http://www.listentothefranchise.com

25

I work for Vyatta. We regularly see 700+kpps per core using a Nehalem
class cpu with higher rates possible in tuned systems. On a multi-core
system this translates to a fairly high level of throughput. To echo an
earlier post, Linux can comfortably handle gigabit.

It wasn't too long ago that this wasn't the case. The growth in the
number of cores available to the end user, the introduction of
multi-queue nics, the move away from the FSB architecture towards QPI,
ever faster PCIe... The technology is directionally trending towards
faster, more consistent network throughputs whether your Linux host is
acting as a router, firewall, web server, or whatever. There are
activities taking place on the software front as well to increase speed
and consistency in the realms of forwarding and firewall, including
technologies that separate the control and forwarding planes. There is
still headroom available in commodity compute to scale further.

I will be the first to admit that Vyatta won't work for everyone. We
still have a lot of work to do for our system to fit seamlessly in some
environments. But, the bet that we have made is that commodity compute
coupled with the amazing OSS dev community can keep pace with a good
portion of the networking worlds needs. So far, that bet looks like a
good one.

To discount all software routing running on general purpose processors
as being antiquated seems to me to be premature, especially given the
various vendors interests as more functionality migrates into the cloud.
As that happens commodity components in the cloud fabric will
necessarily need to behave more like network appliances.

Cheers,
Robert.

26

I think the issue, is that don't expect to build your own router using linux/bsd etc..

There are too many kernel parameters to tweak to make it optimal (unless a suboptimal router is ok with your environment)

You need people that understand network and the appliance they sell you.

Why Cisco is reliable (and expensive), because they give you that experience... Software based router can give you that experience if they are backed by a team that know what they are doing.

27

I missed an 'on' in my sentence; should have read '...software running ON those ASIC's and FPGA's....' My apologies for the error, which completely changed the meaning of my statement.

A perusal of Cisco's own documentation for one of their 'hardware' forwarding engines, the PXF used in the 10k edge services router and others, shows that even with the Toaster ASIC (looking at a pair right now on an older PRE1 for uBR10K) and its associated memory, you have something running its own software doing the work. Cisco's own documentation describes PXF in these words: "Each of the coprocessors in a PXF network processor is an independent, high-performance processor, customized for packet processing. Each processor, called an Express Micro Controller (XMC), provides a sophisticated dual-instruction-issue execution unit, with a variety of special instructions designed to execute packet-processing tasks efficiently."

Instruction issue? Execution unit? Special instructions? Sounds like a software-driven processor to me. Specialized software instruction set, yes. True hardware forwarding, no software involvement? No. More like asymmetrical multiprocessing software routing. Call it hardware accelerated if you like; PXF is to networking as a nVidia GeForce GPU is to graphics.

Now, if we're talking directed attacks at the control plane.... well, COPP exists for a reason in Cisco-land. Tarpits and other techniques (too bad nVidia's ActiveArmor firewall inside their nForce chipset's NIC's is so broken), including transparent layer 2 stateful inspection firewalling (easily doable with Linux iptables and bridging), can do the same for a single-core router.

Now to, as Emeril would say, kick it up a notch, you're going to have a very hard time DoS'ing twenty-four Phenom II cores (four sockets, six cores per socket), though (which will likely set you back less than a midrange Cisco router). I could see Vyatta on 24 Phenom II cores having blistering and nearly DoS-proof performance, for about what accelerated forwarding platforms cost. When the developers of software forwarding engines figure out how to leverage vector processing (SSE and similar, as well as nVidia's CUDA) to do packet forwarding, we're going to see commodity OS network routing performance hit another level.

But specialized network processors don't always guarantee the great scalability that can be obtained with the technique. Catalyst 8540 anyone? (I have several, and use a few in production; great boxes for raw IPv4 routing, but not at the edge, although in theory they should have been DoS-proof, since they're already switching worst-case packet sizes on the shared memory fabric at wire speed; their control plane was their weakest link).

Dedicated network coprocessors can be a good thing, but they're still software-based (even in the Catalyst 8540's case).

28

That's just a completely ignorant statement to make. I notice in
particular how carefully you qualify that with "[w]hen BCPs are
followed"; the fact that hardware router manufacturers have declared
everything and anything that derails their bullet trains as "not a
BCP" is a perfect example of this deceptive sort of misinformation.

There are plenty of FreeBSD based devices out there that are passing
tons of traffic; almost any of them are more competent than any Cisco
router I'm aware of when hitting them directly with traffic, since
the CPU's on your average Cisco are pretty flimsy, the CPU on a
FreeBSD box is going to be fairly current tech, and the code on a
FreeBSD box is going to have been designed to defend against such
attacks because things like IRC server operators often don't have
the luxury of hiding their device management on a protected net.

The fact of the matter is that the way that most "hardware" platforms
try to survive a DoS attack against their control plane is through
hardware filtering; to the extent that that works, it's going to be
pretty effective. However, if we're going to allow for that, then we
have to allow the software platform to defend itself with a firewall
as well, and once you do that, on both platforms, what actually
happens in the end is that both devices can successfully defend at
gigabit speeds, but you start losing traffic because you're filling
the inbound pipe.

Or, put another way:

"When BCP's are followed, software devices don't tend to fall over the
moment someone hits them with a few Mpps of packets either."

... JG

29

Right. And to date, such routers make use of ASICs - i.e., 'hardware-based' routers, in the vernacular.

Routers which use only centralized, general-purpose processors can't handle even a fraction of 'line-rate' without tanking, as innumerable real-world examples of said behavior over the years have repeatedly and conclusively demonstrated.

;>

30

7200 certainly is - I'm not familiar with the minutiae of Juniper boxes, but I believe the M20 is hardware-based. In the classic report you cite, the issue with the M20 occurred due to lack of BCP implementation.

31

The whole point about being DoS resistant is one of horsepower. To do
DoS protection correctly, you need to be able to do packet examination
at line rate.

Right. And to date, such routers make use of ASICs - i.e.,
'hardware-based' routers, in the vernacular.

Routers which use only centralized, general-purpose processors can't
handle even a fraction of 'line-rate' without tanking, as innumerable
real-world examples of said behavior over the years have repeatedly and
conclusively demonstrated.

I'm not really all that opinionated on the subject, other than to say that
the definition of a router, and what qualifies as a sufficient router for
any given administrator's needs, greatly varies.

However, to state something like

as innumerable real-world examples of said behavior over the years have
repeatedly and conclusively demonstrated.

has the appearance of you struggling to hold on to an idea that may have
been more true in the past, and less true today, as is evident based on the
input from other list participants.

32

That's just a completely ignorant statement to make.

It's based on a great deal of real-world experience; I'm sorry you consider that to be 'ignorant'.

I notice in particular how carefully you qualify that with "[w]hen BCPs are
followed"; the fact that hardware router manufacturers have declared
everything and anything that derails their bullet trains as "not a
BCP" is a perfect example of this deceptive sort of misinformation.

Anti-spoofing, iACLs, CoPP (or its equivalent on non-Cisco platforms), et. al. aren't 'misinformation'. They're useful, proven techniques/features which any operator ought to implement.

There are plenty of FreeBSD based devices out there that are passing
tons of traffic; almost any of them are more competent than any Cisco
router I'm aware of when hitting them directly with traffic

Then your experience of Cisco routers (and/or those from other vendors) must be limited to the lower-end platforms; I can assure you that faster Cisco boxes such as ASRs, GSRs, CRSes, and so forth are in another league entirely, and can handle mpps of to-us traffic, when properly configured. Software-based routers simply can't do that; it's not an indictment of them, it's just that they aren't suited to purpose, just as station wagons generally aren't to be found in the Indy 500.

;>

33

has the appearance of you struggling to hold on to an idea that may have been more true in the past,

It's true today, and I'm not 'struggling to hold' onto anything. Take any software-based router from Cisco or Juniper or whomever (if Juniper still make software-based routers, I don't know if they do or not), packet it until it falls over, then repeat the process with a properly-configured hardware-based router from the same manufacturer - you can demonstrate the validity of the proposition for yourself, as the hardware-based router can handle considerably more traffic, whereas the software-based router will pitch over as a result of a surprisingly small amount of traffic.

and less true today, as is evident based on the input from other list participants.

Input based upon experience which is seemingly heavily weighted towards the lower end, rather than the higher end, of network speeds and routing platforms - and which doesn't seem to encompass much examination of the ability of said lower-end devices to maintain availability in the face of direct attack.

It can be quite interesting to take a packet-generator to a software-based router and see just how easy it is to make it fall over, and then repeat the experience with a hardware-based router, and consider the implications thereof, even at relatively low bandwidth/throughput.

34

This is true on a lot of newer Cisco high end platforms. CRS-1 uses multicore processors (hundreds of cores) for forwarding on their linecards, and they achieve 40+ Mpps per linecard.

This is the trend in networking where you need to do intelligent things, it's easier to do multicore parallell processing than doing hugely fast FPGA forwarding (at least it seems that way, and it's faster to upgrade the software on a CPU than on a FPGA).

The lines are blurring between CPU/FPA/ASIC (well, ASIC is really a misnomer as it's just "application specific" which means packaging, not function) and since people want flexibility, general CPUs used for forwarding is the way it's headed, even though the CPUs right now have little to do with the CPUs we see in "normal" PCs.

35

The CRS-1 makes use of the Metro subsystem for forwarding, with multiple Metros per Modular Service Card (MSC). Each Metro complex (there are two per MSC) consists of the Metro chip itself, an NPU which contains 188 embedded RISC cores; two TCAM banks; SRAM; and FCRAM.

There's also a gatekeeper ASIC of some sort on the MSC which handles traffic incoming from the fabric, as well as another interface module ASIC on the Physical Layer Interface Module (PLIM).

I believe the CRS-3-specific MSCs each contain two QFAP complexes, which allow for 140gb/sec per linecard, and that there are various additional supporting ASICs on the MSCs and the PLIMs, as well.

36

But as others have stated, the 7206 has at least some hardware acceleration,
so it's *not* a router that uses *only* centralized general-purpose CPUs. So
at least some hardware-assisted routers tank under loads too.

And even the most heavily hardware-assisted systems have to do call outs from
the FPGA's for *some* stuff, and can be tanked by suitably creative abuse of
their weaknesses. Of course, in general the more hardware assist, the harder
it is to tank (but it's never impossible).

So basically, your definition of "hardware based" router is "one that has enough
FPGAs to not tank under some arbitrary workload". Not very useful,that.

Let's face it Roland - it's a continuum from hardware to software, and in many
places it's downright murky which it is. Is the CRS-1 hardware or software?
Lots of custom hardware in there - but lots of processing cores that look
suspiciously like software engines too.

37

But as others have stated, the 7206 has at least some hardware acceleration,

Unfortunately, said statements are factually incorrect. 7200s have no hardware acceleration of any type whatsoever.

from <http://www.cisco.com/en/US/prod/collateral/routers/ps341/product_data_sheet0900aecd8047177b.html>:

'Processor

1.67-GHz Motorola Freescale 7448 processor'

so it's *not* a router that uses *only* centralized general-purpose CPUs.

Actually, it is. Same with ISRs.

from <http://www.cisco.com/en/US/prod/collateral/routers/ps10538/qa_c67_553891_ps10536_Products_Q_and_A_Item.html>

Note the 'Multicore Processor' line-item - singular.

The SREs for the ISR2s do each contain their own Intel x86 processor - so, the ISR2 models which can take SREs are distributed platforms, but aren't hardware-based in the sense that they contain high-performance forwarding chips. The processors in the SREs are used to run applications on-board the router itself - so, they're kind of like special-purpose servers on a card, rather than high-performance linecards as one finds in higher-end platforms.

So basically, your definition of "hardware based" router is "one that has enough
FPGAs to not tank under some arbitrary workload". Not very useful,that.

It's extremely useful to differentiate routers which have special-purpose forwarding hardware from those which don't, as the latter crumble quite quickly when packeted. If you don't believe me, run some tests of your own with purely software-based routers, such as 7200s, and then with a hardware-based router such as an ASR1K, ASR9K, GSR, CRS-1, N7K, what-have-you.

I've seen this divergent behavior between software-based and hardware-based platforms time and time again in real, live production networks, during real, live attacks. It isn't something which can simply be dismissed by semantic hairsplitting.

And it's not *my* definition - 'hardware-based' vs. 'software-based' are the terms to describe these two fundamental architectural classes of router *within Cisco itself*.

Let's face it Roland - it's a continuum from hardware to software, and in many
places it's downright murky which it is. Is the CRS-1 hardware or software?

Hardware, obviously - it has special-purpose NPUs on the linecards, along with special-purpose ASICs, and TCAMs.

Lots of custom hardware in there - but lots of processing cores that look suspiciously like software engines too.

There's a world of difference in packet-handling mechanisms and sheer performance between a 7200 and a CRS-1, or a GSR, or a CRS-3, or Juniper T-series - and not just one of 'more, faster processors', but of fundamental architecture. This is why 'hardware-based' vs. 'software-based' is a useful distinction; again, note that within Cisco, these are the common terms used to describe these general classes of device, with 7200s and ISRs being termed 'software-based', and the other models mentioned above described as 'hardware-based'.

Anyway, enough on this topic. If folks wish to continue to deploy software-based routers at the edges of their networks, then they oughtn't to be surprised or dismayed when said software-based routers fall over under relatively small amounts of packeting, either deliberate attacks or as the result of misconfiguration, et. al. If, on the other hand, they prize availability, then investing in hardware-based platforms and then configuring said hardware-based routers with the appropriate BCPs greatly reduces the risk of such an occurrence.

38

* Valdis Kletnieks:

(cue weasel-words about those routers using ASICs for most forwarding, but
doing multicast forwarding in software in 5.. 4.. 3..)

There's also the question of IP options (or extension headers). :sunglasses:

39

I know that some modern hardware-based routers have the ability to either ignore options, or to drop option packets altogether.

I believe the same is now true of IPv6 extension-headere, or soon will be. You're absolutely correct that this is a significant possible attack vector, causing the packets in question to be punted, if there isn't a mechanism available to ignore them or to drop said packets.

40

* Roland Dobbins:

There's also the question of IP options (or extension headers). :sunglasses:

I know that some modern hardware-based routers have the ability to
either ignore options, or to drop option packets altogether.

There might be contractual reasons not to enable that feature. 8-/
Some vendors can process options in hardware, though.

I believe the same is now true of IPv6 extension-headere, or soon
will be. You're absolutely correct that this is a significant
possible attack vector, causing the packets in question to be
punted, if there isn't a mechanism available to ignore them or to
drop said packets.

It's probably not a high-priority issue for vendors until there are
network issues (as opposed to potential problems seen in labs), so
it's going to take quite a bit of time. Demand for devices with some
IP-layer inspection capability that can handle (Fast or Gigabit)
Ethernet at line rate, no matter what type of frames come in, is also
a pretty recent thing, and I would be surprised if vendors can provide
such capabilities across their entire relevant product line (where
they advertise line-based forwarding).