Vyatta as a BRAS

There might be contractual reasons not to enable that feature. 8-/

Ignoring is generally pretty harmless; dropping can break traceroute, RSVP, et. al.

Conversely, there are also generally pretty strong contractual reasons not to have one's edge routers go down due to excessive punts.

;>

Some vendors can process options in hardware, though.

True.

It's probably not a high-priority issue for vendors until there are
network issues (as opposed to potential problems seen in labs),

This is always true when it comes to security, and especially to availability. That being said, I know that at least one major vendor is cognizant of the header-extenstion issue, and is taking steps to mitigate the associated risk.

so it's going to take quite a bit of time.

Yes, this is always the case, unfortunately.

Demand for devices with some IP-layer inspection capability that can handle (Fast or Gigabit)
Ethernet at line rate, no matter what type of frames come in, is also
a pretty recent thing, and I would be surprised if vendors can provide
such capabilities across their entire relevant product line (where
they advertise line-based forwarding).

With large vendors, these things are generally accomplished piecemeal, on a BU-by-BY, product-by-product basis. Unfortunate, but true, nonetheless.

> That's just a completely ignorant statement to make.

It's based on a great deal of real-world experience; I'm sorry you consider=
that to be 'ignorant'.

You're speaking to someone who has extensive experience with "software"
based routers, and you're failing to acknowledge the upsides of such an
architecture, when I've already conceded the upsides of a hardware
architecture.

> I notice in particular how carefully you qualify that with "[w]hen BCPs =
are=20
> followed"; the fact that hardware router manufacturers have declared
> everything and anything that derails their bullet trains as "not a
> BCP" is a perfect example of this deceptive sort of misinformation.

Anti-spoofing, iACLs, CoPP (or its equivalent on non-Cisco platforms), et. =
al. aren't 'misinformation'. They're useful, proven techniques/features wh=
ich any operator ought to implement.

The things that any given use scenario ought to implement are highly
dependent on the actual application.

> There are plenty of FreeBSD based devices out there that are passing
> tons of traffic; almost any of them are more competent than any Cisco
> router I'm aware of when hitting them directly with traffic

Then your experience of Cisco routers (and/or those from other vendors) mus=
t be limited to the lower-end platforms; I can assure you that faster Cisco=
boxes such as ASRs, GSRs, CRSes, and so forth are in another league entire=
ly, and can handle mpps of to-us traffic, when properly configured. Softwa=
re-based routers simply can't do that; it's not an indictment of them, it's=
just that they aren't suited to purpose, just as station wagons generally =
aren't to be found in the Indy 500.

So your solution is to keep throwing heavier hardware at the problem until
it works. Okay, I see that. Now, let me quote from a different message:

If maintaining availability is important, then hardware-based (semantic
hairsplitting aside) devices are a requirement.

The truth is that you can keep throwing CPU at a problem as well. I can
size a software based router such that it can remain available.

This is neither new nor exciting technology. Luigi Rizzo was doing
extensive work on this about a decade ago: he took an Athlon 750 platform
with 4 100Mbit ethernet interfaces in it (Athlon 750 = 1999 tech) and was
able to exceed 100Mbps levels without a problem. The UNIX based platforms
have extensive capabilities to defend against attack, even without a
firewall. As with a hardware based platform, there are both good things
and bad things you can do that will impact availability.

Software based platforms have an incredible edge in areas that hardware
based platforms don't, including capex and the ability to find replacement
parts after a disaster. I spent some time after the Haiti quake getting
FreeBSD-based routers up and running, a task made easier because it's a
lot easier to find a working PC and scavenge some network cards than it is
to find a working Cisco router in a city where all inbound and outbound
transportation is paralyzed.

You can continue to defend your position, of course, but it's just looking
a bit silly. A wise engineer knows that there are several ways to tackle
any task, and "one tool for every job" is not a sound policy.

If you'd like to revise your position to "Cisco and Juniper software based
solutions are underpowered PoS", that's probably a defensible position,
and you won't get any argument from me. Please don't generalize such a
position into all software based devices, though. Overall, there are a
lot more software based routers out there than hardware based devices.
Your cablemodem, your ADSL modem, your wifi access point, all these are
probably software based devices. Some of them will melt under a too-great
load. Some won't. This is a function of many different factors. There
is nothing inherent in a software-based device that's going to make it
fail under load - just as there's nothing inherent in a hardware-based
device that's going to make it succeed (which is why you have to qualify
your defense of these with "must follow BCP"). It's the related
engineering that ultimately determines whether or not it all works out.

... JG

The truth is that you can keep throwing CPU at a problem as well. I can size a software based router such that it can remain available.

Not against mpps, or even high kpps, you can't, unfortunately.

Software based platforms have an incredible edge in areas that hardware based platforms don't, including capex and the ability to find replacement parts after a disaster.

I agree 100% with this, and with much of what you say. My point is that at the *edge* - like a BRAS, which is how this thread started - one must have platforms which can be adequately protected against attack/abuse, and hardware-based platforms are the only practical way to do that.

And it's not *my* definition - 'hardware-based' vs. 'software-based' are the terms to describe these two fundamental architectural classes of router *within Cisco itself*.

[snip]

There's a world of difference in packet-handling mechanisms and sheer performance between a 7200 and a CRS-1, or a GSR, or a CRS-3, or Juniper T-series - and not just one of 'more, faster processors', but of fundamental architecture.

CEF is CEF is CEF, whether done on a 2600 or a 7200 or a GSR. Now, don't get me wrong; the engineers who make massively parallel forwarding engines are creative and smart folks, and have come up with very elegant methods of moving the bits ever faster, but the fundamental forwarding architectures, even of these accelerated boxes, can be implemented in pure software, as evidenced by the Cisco Nexus 1000V.

This is why 'hardware-based' vs. 'software-based' is a useful distinction; again, note that within Cisco, these are the common terms used to describe these general classes of device, with 7200s and ISRs being termed 'software-based', and the other models mentioned above described as 'hardware-based'.

Marketingspeak doesn't necessarily reflect reality. The original draft of one of my replies in this thread said this 'Let's run this rabbit, and dispel some marketing hype while we're at it.'

The reality is that 'hardware-based' routers really are AMP (asymmetrical multiprocessing) software-based routers, with specialized processors running specialized software. And when implemented properly they are very good at what they do; I have GSR's, they work great, and regardless of what engine is on the linecard some software at some level running on some processor is making the forwarding decisions at the data plane, and they can even for certain things require a punt to the MIPS processor on the linecard (IPv6 on Engine 1, anyone?).

Knowing the technology and its architecture, without blindly buying into marketingspeak, helps operators make better procurement decisions. And Cisco's website has most of the information you need to make that decision, if you use their hardware, which is very good. Dig deeply enough, and past the hardware versus software pseudodichotomy, and you can make very informed decisions indeed. As a tongue in cheek example, remember the 'switching router' versus 'routing switch' distinction?

If a specialized network processing engine in an AMP router can protect the control plane CPU, then code running on different processors in an SMP system could do the same. Perhaps not as efficiently, but the end result can be the same. I mean, I wonder if Blue Gene or Jaguar could give a CRS series a run for its money in terms of routing power (especially on the control plane), and what the price/performance ratio would be to throwing something like Jaguar (224K Opteron processors, running Linux) at the relatively mundane task of throwing precisely metered bits around the wires. :slight_smile:

Regardless of recommendations, people are using commodity server-grade SMP hardware to run commodity OS's to get the job done, and given the people who have chimed in here, apparently are doing it without lots of problems. The increase on this and other lists of questions about Mikrotik, Vyatta, and other nontraditional routing platforms is an interesting throwback to the days of Proteon routers, the original SUN, and Cisco's multibus roots, and it shows that people are deploying these newer and much faster boxen in the real world, bugs and all.

It's not a 'software-based = bad; hardware-based = good' world, even at the edge anymore; a few years ago, sure, I wouldn't dream of doing such a thing. I for one love what a good parallel state machine in an FPGA can do to your software's performance (we're doing that here, doing interferometric correlation at 96Gb/s on a relatively inexpensive FPGA platform based on IBOB); or what GPU acceleration can do to graphics performance, but it is a help to realize that those activities, accelerated though they may be, are still software-based.

And while it's not a BRAS, one of the most exciting products I've seen in a long while from Cisco is the above-mentioned Nexus 1000V. Pure software virtual switching for VMware.

I don't see why you could call it anything but a software router. That's sort of why things like it and the 7500 before it lasted so long. As the thing ages, cisco comes out with new NPE (or RSP/VIP) processors with faster CPUs / more memory capacity that are able to move more packets. i.e. NPE-100->NPE-150->NPE-200->NPE-225->NPE-300->NPE-400->NPE-G1->NPE-G2

You could start with a VXR with NPE-225 and keep upgrading the CPU and keep the thing in service with the same interface cards 15 years or more.

Regardless of recommendations, people are using commodity server-grade SMP hardware to run commodity OS's to get the job done, and given the people who have chimed in here, apparently are doing it without lots of problems. The increase on this and other lists of questions about Mikrotik, Vyatta, and other nontraditional routing platforms is an interesting throwback to the days of Proteon routers, the original SUN, and Cisco's multibus roots, and it shows that people are deploying these newer and much faster boxen in the real world, bugs and all.

How many of them are deploying server-grade SMP hardware / commodity OS
to handle 10 Gig links and expect to handle DoS attacks using minimum
sized packets? That's 14.88 Mpps with Ethernet encapsulation, for each
10 Gig link.

Anybody?

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

> I wasn't aware that the 7206 and M20 classified as software-based.

I don't see why you could call it anything but a software router.

The 7206 yes. The M20, no.

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

CEF is CEF is CEF, whether done on a 2600 or a 7200 or a GSR. Now, don't get me wrong; the engineers who make massively parallel forwarding engines are creative and smart folks, and have come up with very elegant methods of moving the bits ever faster, but the fundamental forwarding architectures, even of these accelerated boxes, can be implemented in pure software, as evidenced by the Cisco Nexus 1000V.

This is a *vast* oversimplification of the 'life of a packet' in a box like a CRS-1 vs. a 7200, for example. You know this, of course.

Marketingspeak doesn't necessarily reflect reality. The original draft of one of my replies in this thread said this 'Let's run this rabbit, and dispel some marketing hype while we're at it.'

I wasn't a marketing person during my time at Cisco, and I'm not a marketing person now. Technical people within Cisco and outside Cisco routinely make use of the terms 'hardware-based' and 'software-based' to describe these fundamental classes of routers, and have for years.

The reality is that 'hardware-based' routers really are AMP (asymmetrical multiprocessing) software-based routers, with specialized processors running specialized software.

Right - the 'hardware-based routers' idiom comes from the 'specialized processors', known as NPUs, ASICs, TCAMs, and so forth.

And when implemented properly they are very good at what they do; I have GSR's, they work great, and regardless of what engine is on the linecard some software at some level running on some processor is making the forwarding decisions at the data plane,

'Some processor' = ASIC or NPU = 'hardware-baed'.

and they can even for certain things require a punt to the MIPS processor on the linecard (IPv6 on Engine 1, anyone?).

Yes, this is true - or even to the system RP.

Knowing the technology and its architecture, without blindly buying into marketingspeak, helps operators make better procurement decisions.

Nobody in this conversation so far is 'blindly buying into marketingspeak', to my knowledge - certainly not me.

And Cisco's website has most of the information you need to make that decision, if you use their hardware, which is very good.

The content on Cisco's Web site is confusing, redundant, full of *actual* marketing-speak, and highly confusing in many aspects. This isn't a problem unique to Cisco, mind, but afflicts almost all technology companies.

Dig deeply enough, and past the hardware versus software pseudodichotomy, and you can make very informed decisions indeed.

Yes, but you aren't going to do that by looking at product marketing materials on a Web site.

As a tongue in cheek example, remember the 'switching router' versus 'routing switch' distinction?

Yes, which was nonsense. OTOH, 'hardware-based' vs. 'software-based' is a useful distinction commonly employed by technical, non-marketing people within both the vendor and operational communities alike.

If a specialized network processing engine in an AMP router can protect the control plane CPU, then code running on different processors in an SMP system could do the same.

Not on a general-purpose SMP system running commodity processors such as those found in PCs.

Perhaps not as efficiently, but the end result can be the same. I mean, I wonder if Blue Gene or Jaguar could give a CRS series a run for its money in terms of routing power (especially on the control plane),

Possibly - certainly not on the forwarding plane.

and what the price/performance ratio would be to throwing something like Jaguar (224K Opteron processors, running Linux) at the relatively mundane task of throwing precisely metered bits around the wires. :slight_smile:

Fruitless speculation, IMHO.

Regardless of recommendations, people are using commodity server-grade SMP hardware to run commodity OS's to get the job done, and given the people who have chimed in here, apparently are doing it without lots of problems.

My experience is that folks doing this on the edges of their networks eventually end up regretting it, after they get zorched a time or two.

The increase on this and other lists of questions about Mikrotik, Vyatta, and other nontraditional routing platforms is an interesting throwback to the days of Proteon routers, the original SUN, and Cisco's multibus roots, and it shows that people are deploying these newer and much faster boxen in the real world, bugs and all.

It shows me that a lot of folks, because they haven't been in the industry very long and/or don't learn from the mistakes of others in the past, seem determined to repeat those same mistakes, ad nauseam, ad infinitum.

It's not a 'software-based = bad; hardware-based = good' world, even at the edge anymore;

I very strongly disagree with this, based upon my hands-on operational experience in production networks. Running software-based platforms at the edges of one's network is extraordinarily irresponsible, in operational terms, if availability is an important metric for said network.

a few years ago, sure, I wouldn't dream of doing such a thing. I for one love what a good parallel state machine in an FPGA can do to your software's performance (we're doing that here, doing interferometric correlation at 96Gb/s on a relatively inexpensive FPGA platform based on IBOB); or what GPU acceleration can do to graphics performance, but it is a help to realize that those activities, accelerated though they may be, are still software-based.

Again, with the semantics. If you take this hairsplitting to its logical extension, there's no such thing as 'hardware', because it's all 'software' - just some of it is 'hardware' which happens to be instantiated in the form of physical chipsets.

While this may be intellectually appealing to some folks at the hand-waving, 30,000-foot, pseudo-philosophical level, as a matter of practical reality in the real world on real networks using real boxes, the distinction has a great deal of import.

And while it's not a BRAS, one of the most exciting products I've seen in a long while from Cisco is the above-mentioned Nexus 1000V. Pure software virtual switching for VMware.

The N1KV is interesting primarily because it's Cisco's first retail pure-software - i.e., not tied to a box - product intended to move packets around. It also highlights the flexibility and portability of NX-OS, which is Cisco's best OS to date, IMHO.

At the same time, from a performance perspective, it leaves a lot to be desired. I hardly like to think what will happen when a few dozen VMs sending their traffic through an N1KV get botted and start spewing out SYN-floods and so forth.

We all know there's a great deal of difference between a box like a CRS-1 and a box like a 7200, and/or an Intel box running Quagga or what-have-you, and it's absurd to pretend otherwise.

Bottom line - use a software-based box for a layer-3 edge application like a BRAS, and you're asking for trouble. And this time, I'm really done with this thread, as semantic arguments quickly grow tiresome.

Is the CRS-1 hardware or software?
Lots of custom hardware in there - but lots of processing cores that look
suspiciously like software engines too.

It might well be software engines in there, but that's not the point
here. The linecards (MSC/PLIM etc.) in a CRS is designed to handle
wirerate traffic *of any packet length* and simultaneously do ACLs,
QoS or whatever else you throw at them. If that is done using FPGAs,
CPUs or trained monkeys isn't really that interesting, as long as it
works. And it does. But it won't come for free.

A software-based router, i.e something equipped with a central CPU
doing all heavy lifting, of *any kind* isn't designed to do that.

The old corollary 7a in RFC1925 still make sense: "Good, Fast, Cheap:
Pick any two (you can't have all three)."

Some of the arguments expressed also vaguely resembles truth 11 in the
same RFC:
Every old idea will be proposed again with a different name and a
different presentation, regardless of whether it works.

There is a reason internet isn't built on Dell hardware...

Dangerous in places where forwarding table exceeds hardware cache
limits. (See Code Red worm stories)

During the Code Red/Nimda period (2001), and on into the
Slammer/Blaster/Nachi period (2003), all the routers I personally
know of which were adversely affected were software-based, didn't
make use of ASICs for forwarding.

Having msdp turned on was a great way to get nuked by slammer regardless of your choice of forwarding technology.

Which reminds me control plane protection is about more than just acls and rate limiting.

-----------------------------------------------------------------------

Roland Dobbins<rdobbins@arbor.net> //<http://www.arbornetworks.com>

> The truth is that you can keep throwing CPU at a problem as well. I can =
size a software based router such that it can remain available.

Not against mpps, or even high kpps, you can't, unfortunately.

Really? I'm positive that I can, because I *have*, and other people
*have*. The sweet spot for protecting a 100Mbps circuit, in particular,
moved from hardware to software about five years ago. That simply means
it's more cost-effective for a competent admin to spend some time to set
up the box than it is to spend money on dedicated silicon that'll be
obsolete in a few years, a fact that's conveniently ignored by a lot of
the advocates of such solutions. To drive the point home, FreeBSD based
routers that we built in 2004 are able to cope with full routing tables
and IPv6 *today*, at the same traffic levels they were designed for, and
those particular qualities don't seem to be present in many of the
hardware-based offerings of the era. If and when they cease to be useful
in that capacity, they can be trivially repurposed as firewalls or web
servers or other similar tasks, because unlike the pricey purpose-built
router hardware, there are advantages to general purpose hardware.

Quite frankly, this is starting to be a little annoying. Perhaps you
could do some research, or find some competent admins and test a few well
built setups yourself before you make any more disprovable claims. My
claims are not ridiculous and are not a figment of my imagination; I can
point to many-years-old documented examples, such as

http://lists.freebsd.org/pipermail/freebsd-net/2004-September/004840.html

http://info.iet.unipi.it/~luigi/polling/

These are tests of forwarding capabilities, true, but the reality is that
the same sorts of things that make this possible make it relatively easy
to support large numbers of packets directed "at the control plane", since
the concept of the control plane isn't as separated in the FreeBSD software
model as it is in the hardware model. As a result, a FreeBSD box can take
and sink quite a bit of traffic. Doing so does not cripple it.

For giggles, I took two out-of-the-box FreeBSD 8.0 servers, twiddled
*only* device polling to on, and started them running traffic at each
other. Both were sending north of 100Mbps (>>100Kpps) of traffic at
the other, both when listening and when not, no problems, no crashes,
no issues. That doesn't sound too great until I reveal that I was
lazy and it's only some excess capacity on a VMware box that's
available to these two virtual servers.

> Software based platforms have an incredible edge in areas that hardware b=
ased platforms don't, including capex and the ability to find replacement p=
arts after a disaster.

I agree 100% with this, and with much of what you say. My point is that at=
the *edge* - like a BRAS, which is how this thread started - one must have=
platforms which can be adequately protected against attack/abuse, and hard=
ware-based platforms are the only practical way to do that.

In some cases, for some purposes, yes. Otherwise, no.

... JG

I briefly browsed the links and I didn't see any traffic profiles included.

If you are talking about pushing x mbps with no specifics and/or general traffic, I think most of us agree you can do that easily and probably consistently without any issues. And for some icing, you may even do it at <90% average CPU util. Does that mean it should be an edge device at any service provider? No. Some? Sure.

Can you point to any specific tests of attack vectors and/or traffic profiles with: CPU utilization, packet loss levels and pps/mbps/etc data? The reason I ask is that Roland is in a specific business and has a specific point.

As a side, were those 2 VMs on the same box? That traffic out on the wire? What's the traffic profile?

tv

I briefly browsed the links and I didn't see any traffic profiles included.

If you are talking about pushing x mbps with no specifics and/or general
traffic, I think most of us agree you can do that easily and probably
consistently without any issues. And for some icing, you may even do it at
<90% average CPU util. Does that mean it should be an edge device at any
service provider? No. Some? Sure.

Those last two words are the point I've been trying to make. If you'll
recall, Roland said flat out that that wasn't the case.

Can you point to any specific tests of attack vectors and/or traffic
profiles with: CPU utilization, packet loss levels and pps/mbps/etc data?

Not without doing the work; I have no plans to do the work for free just
to prove a point on NANOG. I have Real Work to do.

The reason I ask is that Roland is in a specific business and has a specific
point.

Sure, and I'm making the point that this point isn't universally true in
the way Roland would like to paint it.

As a side, were those 2 VMs on the same box? That traffic out on the wire?
What's the traffic profile?

Yes, no (just between vm's), just sheer UDP blasting of both the vservers
from the other (mutual attack) with ports both closed and opened. Since
Roland's point seems to be that the availability of the platform is
impacted by an attack on the control plane (in this case, for all
reasonable intents and purposes, that would appear to be the host OS's
addresses), I didn't really feel it necessary to get particularly
complicated, and just tested the control plane availability theory.

My point is that a randomly created *virtual* machine can absorb a

100Mbps attack on it at minimum packet size without blinking, while

simultaneously delivering such an attack, in the spare CPU cycles of
a vm host that has dozens of hosts on it. It's meant to suggest that
what Roland is selling includes a healthy dose of FUD; I, on the other
hand, am happy to concede that at a certain point, the hardware stuff
is going to be more effective. It'd be nice if Roland could concede
that software-based routers have some advantages and some reasonable
use profiles.

For example, for a provider whose entire upstream capacity is 1Gbps, I
have a hard time seeing how a Linux- or FreeBSD-based box could credibly
be claimed not to be a suitable edge router.

The problem with Roland's statement is its absoluteness; I have a much
easier side to argue, since I merely need to explain one case where the
use profile does not result in failure, and there are many to choose
from.

... JG

Because it can and will be whacked quite easily by anyone who packets it, either deliberately or inadvertently. I've seen too many software-based routers fall over with far, far less traffic than 1gb/sec to think otherwise.

Since you've seen "many software-based routers fall over", can you
provide details on specific hardware/software/traffic patterns/rates
where you've seen these failures? From what I can tell, software
based routers are almost universally used in SOHO environments; so it
would be nice to know when such solutions are no longer viable in your
experience.

Thanks,
Bill Bogstad

>
>
>> For example, for a provider whose entire upstream capacity is 1Gbps, I have a hard time seeing how a Linux- or FreeBSD-based box could credibly be claimed not to be a suitable edge router.
>
> Because it can and will be whacked quite easily by anyone who packets it, either deliberately or inadvertently. �I've seen too many software-based routers fall over with far, far less traffic than 1gb/sec to think otherwise.

Since you've seen "many software-based routers fall over", can you
provide details on specific hardware/software/traffic patterns/rates
where you've seen these failures? From what I can tell, software
based routers are almost universally used in SOHO environments; so it
would be nice to know when such solutions are no longer viable in your
experience.

SOHO environmnents aren't normally targets of DOS attacks. And if they are,
their pipes are probably small enough to be easily filled with far less
difficulty than making the router fall over.

I'm almost certain they're not the uses that Roland is saying that software
routers are entirely unsuited for.

Correct - I'm talking about SP (and even enterprise) edge routers. I've seen as little as a few hundred kpps totally hose Cisco 7200s, boxes running Zebra/Quagga, and so forth.

You seem to be off in your own little world. Provided with a
counterexample where this isn't true, you simply ignore it. Your
arguments revolve around "My Ford Pinto's gas tank once exploded on me
and it happened to other people too, therefore all inexpensive cars have
unsafe gas tanks."

The sad reality is that any gas tank can be ruptured and can be made to
explode, but concluding that this is limited to inexpensive cars is a
silly conclusion. The fact of the matter is that a /poorly engineered/
gas tank is much more prone to problems, whether it's in an inexpensive
car or a high end one.

You're drawing poor conclusions based on even poorer reasoning. Your
negative experience with some software routers does not mean that they
are all crap, just as my negative experience with some hardware routers
does not mean that they are all crap.

... JG

Provided with a counterexample where this isn't true, you simply ignore it.

I've yet to see a counterexample involving a software-based edge router in a realistic testbed environment being deliberately packeted in order to cause an availability hit - apologies if I've missed one, mind.

Your arguments revolve around "My Ford Pinto's gas tank once exploded on me and it happened to other people too, therefore all inexpensive cars have unsafe gas tanks."

Actually, it's more along the lines of, "I've seen multiple accidents involving multiple brands/models of economy-class automobiles in which the passengers were grievously-injured or worse, while also having observed passengers walking away from similar accidents in similar circumstances involving heavier, more sturdily-built vehicles."