I'd like to get some feedback as to what people's experiences are (if any) with image stream routers.. specifically the industrial ones.
Had a discussion with the manager of a large ISP in Turkey. He's a
transplanted Aussie.. He swears by them.. I believe he is running OC-12
links accross them at near full capacity.
My personal experience has been that they have both the engineering talent
and the experience (7+ years in the business) to pull it off. Their
products are logically built, utilize Linux at the core and they stand
behind their gurantee. If it doesn't work, they'll either fix it, or give
you your money back.
They are now keepers and developers of the VRRP project for Linux, and
have also defined a unified driver architecture called "Inetics" which
makes adding new hardware to Linux trivial.
I'm going to be attending a presentation by one of their core developers
at the Ohio Linuxfest on October 1st (http://www.ohiolinux.org). From the
website, here is the specific talk they will be giving:
"Quality of Service using Open Source Linux Tools
Doug Hass, Imagestream
With increasing penetration of wireless and broadband, service providers
must understand Quality of Service techniques and implement QOS on their
networks. A proper QOS design helps to avoid network bottlenecks caused by
converged voice/video/data services , broadband users, file sharing, and
other bandwidth-intensive applications. Without QOS, service providers are
especially susceptible to bottlenecks and service degradation.
This presentation covers the key concepts of quality of service. The
presentation includes an explanation of standard queuing methods as
defined in the Differentiated Services RFC as well as applications of
these methods through generic case studies.
Doug Hass is the COO of ImageStream, a leading router and WAN product
manufacturer. Prior to joining ImageStream, Mr. Hass was a partner in
Midwest-based Internet provider Skye/net. An Army veteran, certified
personal trainer, avid horseman and outdoorsman, Mr. Hass rode
professional rodeo for five years, and is the founder of Roughstock.com,
an award-winning country music Web site."
I'd be interested to know the relative pros and cons of switching packets in
software (Imagestream) versus handing them off to a dedicated ASIC (Cisco,
Juniper)
Probably a good question for Imagestream to answer, as I can't speak to
it. I'd guess that at some point everything is Software Switched at some
level.. the ASIC has to run some sort of software.. 
Christopher J. Wolff wrote:
I'd be interested to know the relative pros and cons of switching packets in
software (Imagestream) versus handing them off to a dedicated ASIC (Cisco,
Juniper)
[without having looked at Imagestream in any way, shape or form..]
it would be _unlikely_ that any router vendor that wants to support >OC3 could do so with the 'standard' (non-modified) linux IP stack. if they are modifying the 'standard' linux IP stack then its very unlikely that one could do so without having to publish the source-code to it. (i.e. as per GPL).
'standard' linux on standard hardware isn't capable of much more than 100K PPS. sure - some folks have a few hundred packets/sec - but these are minimalist versus the demonstrated performance of ASIC-based forwarding, typically 30M-50M PPS.
one advantage of software is programmability. if there is a bug you can fix it.
if there is a bug in an ASIC, it may or may not be possible to fix it - it depends on awful lot on how the ASIC is built (whether its 100% fixed functionality or supports limited programmability in various stages of the forwarding pipeline).
it may be that its not fixable but that the ASIC allows software-workarounds - in essence, 'fixing' something by diverting it to a (slower) software-path.
note that there is a correction to make here: not all routers _ARE_ ASIC-based for forwarding. in fact, most of the Cisco /router/ product portfolio isn't hardware-forwarding based. generally speaking it isn't necessary - UNTIL you get to the point of having interface speeds & number-of-interfaces which exceed the capabilities of general-purpose processors. that is, typically somewhere between 100K PPS and 1M PPS.
cheers,
lincoln.
Thanks for the thoughtful response.
One of the network architecture issues I'm always trying to gauge and get my
arms around is what I'll call, "Quality of user experience." In other
words, what mix of network hardware, software, customer support, and
management will create a perception that the network is performing at
maximum efficiency.
Although the perception of network performance is entirely subjective there
are some factors that I'm sure we can all agree contribute to overall
satisfaction...i.e.
-WAN link latency.
-Packet Loss.
-Consistency in packet generation/serialization (A packet always enters
interface A and leaves interface B in .5 ms)
So, if all other elements (software, customer support, and management) are
equal, what router hardware architecture will contribute to a positive or
negative user experience? In other words, if the routing device between my
workstation and server is a Juniper M7 instead of Pentium IV running
unix-flavor-of-the-day, how will that affect the quality of user experience?
Thank you,
Christopher
[let me preface this by saying that if you don't know this already, i do happen to work for a router vendor]
from the perspective of ANY router, "quality of end-user experience" is not something which fits into layers 1-7 - its a layer 8-10 thing. however, having said that, certainly routers "doing the wrong thing" can definitely negatively impact end-user experience.
i think its best to answer this by what 'role' various routers have, and what their primary function should be. that ultimately determines what the right 'boxes' are for given 'roles' - and if you put the wrong box in the wrong category, that can negatively impact service in a way that customers think your service sucks.
no doubt there are more roles than this & we can get more & more specific - but its my AU$0.02 worth:
note that i'm deliberately not getting into whether it should be IPv4, IPv6, MPLS, all of the above, none of the above .. thats up to what service you have, how you provision it and how you traffic engineer it.
1. Core router
- generally consist of interface speeds of OC12 upwards.
- move packets from A to B with minimal additional latency
and minimal jitter
- should be capable of implementing ACLs with no
performance degredation but primary role is to push packets
- just about mandatory these days that can handle interfaces
pushing maximum packets/sec with minimum packet size so
as to be able to withstand DDoS attacks - either at it, or
through it
2. Transit or peering-facing router
- interface speeds of >OC3, probably decent GbE density
desirable
- mandatory implementation of ACLs
- mandatory full-feature BGP features & widgets
- mandatory implementation of uRPF or similar
- ideally be capable of traffic 'accounting' mechanisms
(e.g. packet-sampling, netflow etc)
3. customer-facing router (FR/ATM/..)
- decent system-density for customer connections
- GbE uplink interface(s)
- mandatory implementation of ACLs
- mandatory full-feature BGP features & widgets
- mandatory implementation of uRPF or similar
- ideally be capable of traffic 'accounting' mechanisms
(e.g. packet-sampling, netflow, anomoly detection etc)
- ideally be able to implement 'better' queueing mechanisms
than just standard FIFO. e.g. low-latency queueing for
voice traffic, fair-queueing for fairness, deep(er) buffers
to attempt to minimize packet drop due to speed-mismatch
4. broadband aggregation router (e.g. LNS)
- handle large numbers of logical sessions from central
configuration/policy (e.g. tie into RADIUS server(s))
- GbE uplink interface(s)
- mandatory implementation of ACLs
- mandatory implementation of uRPF or similar
- ideally be capable of traffic 'accounting' mechanisms
(e.g. packet-sampling, netflow, anomoly detection etc)
- ideally be able to implement 'better' queueing mechanisms
than just standard FIFO. e.g. low-latency queueing for
voice traffic, fair-queueing for fairness, deep(er) buffers
to attempt to minimize packet drop due to speed-mismatch
- sufficient control-plane CPU to handle large # of connection
establishments/sec (e.g. connection to LAC being lost)
5. customer-premises router (CPE)
- generally low-speed (<30Mbps)
- end-users love ones with built-in NAT, DHCP, firewall,
wireless, probably VoIP, ...
- low-cost
- minimal CPU - no need to handle DoS attack because WAN
bandwidth is exhausted before PPS limit of CPU is hit
going through these, i'd say "ASIC based" or multiple-distributed-CPU is what you want for (1). anything less than that means you're likely to have reduced customer satisfaction.
(2), (3) & (4) generally are a mix of s/w and h/w-based routers - architectures vary quite greatly but with silicon developments in the last few years, most semi-recent products are typically a combination of h/w and s/w with (ideally) the work split 90/10. or 99/1. or 100/0 in an ideal world.
(5) can stay software.
cheers,
lincoln.
Christopher J. Wolff wrote:
It depends. Which part of "gestalt" do you not understand?
Right now, the very first hop off my laptop is a dialup that connected at 44K.
The router architecture is the least of my concerns Better/shorter copper
that let it connect at 56K, or getting DSL/cable connectivity would make more
difference.. (Of course, once I get to my office and connect at 100mbit, it
might start mattering 
In general, whatever router is in use, be it Juniper or Pentium or Gerbil, will
either keep up with the offered packets/sec, or it won't. If it doesn't keep
up, packets get dropped and retransmit timers pop, irritating the user. If it
*does* keep up, then your end-to-end latency is basically switching delays and
speed-of-light delays - so the user cares more about the *number* and
*distance* than the actual architecture. 8 hops that include a trip through a
Juniper in Iceland is worse than 2 hops through PC-class hardware to the next
town over (unless the next town over is, in fact, Reykjavik
And remember - the business goal isn't to maximize the user experience, it's
to to find the cheapest configuration that still lets you avoid having to
pay penalties for not meeting the SLA.
Of course, if your user base is $9.95/mo Joe Sixpacks, you really don't have
the revenue stream to support a user experience any better than "sorta works
most of the time"....
Regarding software based forwarding and pps old docs from the FreeBSD
guys claim that the 1Mpps barrier can be broken on a 2.8GHz XEON, with
todays standards a mediocer pc.
http://people.freebsd.org/~andre/FreeBSD-5.3-Networking.pdf
A collegue smartbits tested a 1GHz pc, with a full feed and 250k
simoultaneons flows it managed around 250kpps. This also with freebsd
and device polling. It sounds to me like a software based machine can
be plenty fast with good code under the hood.
/Tony
A collegue smartbits tested a 1GHz pc, with a full feed and 250k
simoultaneons flows it managed around 250kpps. This also with freebsd
and device polling. It sounds to me like a software based machine can
be plenty fast with good code under the hood.
Sorry, in today's world of high-end routers 250kpps doesn't qualify as
"plenty fast". Can your box do linerate Gigabit Ethernet with minimum
size packets, on several ports simultaneously?
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
In general, whatever router is in use, be it Juniper or Pentium or Gerbil,
will either keep up with the offered packets/sec, or it won't.
and, if you want to know if it does, measure it, don't read marketing
bumph.
randy
I didn't say that a 250kpps box was a high-end box.
One reliable Mpps is not high-end either, but it can carry quite a lot
of Mbps. What is C or M price for a reliable full feed Mpps ?
"My" high-end boxes never manage to impress me with their pps
capability before I'm disapointed in their reliability.
/Tony
> > A collegue smartbits tested a 1GHz pc, with a full feed and 250k
> > simoultaneons flows it managed around 250kpps. This also with freebsd
> > and device polling. It sounds to me like a software based machine can
> > be plenty fast with good code under the hood.
>
> Sorry, in today's world of high-end routers 250kpps doesn't qualify as
> "plenty fast". Can your box do linerate Gigabit Ethernet with minimum
> size packets, on several ports simultaneously?
>I didn't say that a 250kpps box was a high-end box.
One reliable Mpps is not high-end either, but it can carry quite a lot
of Mbps. What is C or M price for a reliable full feed Mpps ?"My" high-end boxes never manage to impress me with their pps
capability before I'm disapointed in their reliability.
I'll reply to myself before Steinar does =)
It sounds to me like a software based machine can
be plenty fast with good code under the hood.
In my experience a datacenter pumping out 1Gbps is usually doing
200-250kpps in that direction. Considering this a box capable of
around 1Mbps is "plenty fast".
pps/$ would be pretty good if I could use those in real life...
/Tony
> Sorry, in today's world of high-end routers 250kpps doesn't qualify as
> "plenty fast". Can your box do linerate Gigabit Ethernet with minimum
> size packets, on several ports simultaneously?
>I didn't say that a 250kpps box was a high-end box.
One reliable Mpps is not high-end either, but it can carry quite a lot
of Mbps. What is C or M price for a reliable full feed Mpps ?"My" high-end boxes never manage to impress me with their pps
capability before I'm disapointed in their reliability.
I usually find that it works better to use routers to forward packets
between interfaces, and Unix servers (in my case mostly FreeBSD, but
that's beside the point) to run applications. Occasionally I configure
a Unix box to actually forward packets.
I found Lincoln Dale's characterization the other day of the "roles"
of various routers,
http://www.merit.edu/mail.archives/nanog/msg11681.html
to be useful. From his list I could imagine using Unix boxes for 3, 4
and 5 - but probably not 1 or 2. Also note that high-end broadband
aggregation may have high enough pps that a box with software based
forwarding doesn't cut it.
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
--- snip ---
It sounds to me like a software based machine can
be plenty fast with good code under the hood.
In my experience a datacenter pumping out 1Gbps is usually doing
200-250kpps in that direction. Considering this a box capable of
around 1Mpps is "plenty fast".
... until you get an inbound ddos over that shiny gige at 1.44 Mpps. in
today's world, planning for normal circumstances is woefully insufficient,
you have to spec based on worst case numbers because you're almost
guaranteed they will hit your network upside the head in the future.
-p
It sounds to me like a software based machine can
be plenty fast with good code under the hood.In my experience a datacenter pumping out 1Gbps is usually doing
200-250kpps in that direction. Considering this a box capable of
around 1Mpps is "plenty fast".... until you get an inbound ddos over that shiny gige at 1.44 Mpps. in
today's world, planning for normal circumstances is woefully insufficient,
you have to spec based on worst case numbers because you're almost
guaranteed they will hit your network upside the head in the future.
Not to belabor the perennial software vs hardware router discussion, these types of platforms can be useful in situations where you have powerful hardware routers upstream of them to protect them. For example if you have access customers terminating on a router like this... if you get a DDOS from them, you simply turn off the port and notify them. If its inbound, your border router takes care of you.
just an idea.
Deepak Jain
AiNET
... until you get an inbound ddos over that shiny gige at 1.44 Mpps. in
today's world, planning for normal circumstances is woefully insufficient,
you have to spec based on worst case numbers because you're almost
guaranteed they will hit your network upside the head in the future.
If I have a GE link and get DDOS'ed at 1.44Mpps I'm on the wrong side
of the bottleneck to do much about it, am I not ?
I don't disagree on that forwarding equipment should be able to handle
worst case situations, but I have never worked on a packet switching
network where that is the case, especially not when counting peers and
transits.
The difference is with a software based router that melts under DDoS traffic, the CLI may become unusable and it may be dropping so many packets, that if you're on the outside, you can't get in to manage it or anything else on the network. With a hardware based router that can handle one or more orders of magnitude more PPS that a DDoS generates, the CLI keeps working as if nothing's wrong, and if you happen to be on the outside trying to get in to manage things, you may suffer a little packet loss if your transit pipes are full, but nothing compared to the first case.
Date: Sat, 17 Sep 2005 19:11:14 +0200 (CEST)
From: sthaug@net...
A collegue smartbits tested a 1GHz pc, with a full feed and 250k
simoultaneons flows it managed around 250kpps. This also with freebsd
and device polling. It sounds to me like a software based machine can
be plenty fast with good code under the hood.
Stock BSD (and Linux) routing code are hardly optimal. With some effort, 210 clk/lookup on a P4 Prescott is doable under a fairly stressful test case.
I'd have to go back and dig up details, but that was FIB in < 512 kB, 135 kroute (full table at the time), ~600 different next-hop possibilities, choosing mostly packets with valid next-hop (which was slower than !N packets). Test was in userland with 4 kB pages, so there was a fair amount of TLB churn; I didn't test with large pages.
Stock *ix, however, is a much different story.
Sorry, in today's world of high-end routers 250kpps doesn't qualify as
"plenty fast". Can your box do linerate Gigabit Ethernet with minimum
size packets, on several ports simultaneously?
Alternatively, one can have a "production" GigE port and a "backplane" GigE port that connects to an ethernet switch, essentially making each router an intelligent single-port GigE blade.
Eddy
Date: Sat, 17 Sep 2005 16:18:28 +1000
From: Lincoln Dale
[without having looked at Imagestream in any way, shape or form..]
it would be _unlikely_ that any router vendor that wants to support >OC3
could do so with the 'standard' (non-modified) linux IP stack. if they are
modifying the 'standard' linux IP stack then its very unlikely that one
could do so without having to publish the source-code to it. (i.e. as per
GPL).
Linux, reportedly with custom RIB/FIB code.
IANAL, but gpl-violations.org strikes me as interesting. A court order
not to ship product is rather serious...
(For those who didn't follow the link, no such injunction has been
granted against IS. However, other embedded-Linux OEMs have not been so
lucky.)
Eddy