Roland,
The only issue I have I with your reply is that is somehow still acceptable to not have these features in a core device.
-jim
Hey,
The only issue I have I with your reply is that is somehow still acceptable to not have these features in a core device.
I'm guessing point Roland was making (which he likely would have not made
couple moons ago:) was related to the lack of IPv6 uRPF, chassis wide uRPF
mode and IPv6 ACL either have /128 look-up and no L4 lookup or L4 lookup
and accordingly reduced lookup, forcing longer prefixes to software
(compression removes bits 24-39 from hardware).
In practice this means, if you enable compressed mode, to allow L4 lookups
in ACL, and you likely will (how else are you going to protect server, if
you can't allow MGMT/ssh and internet/http and drop rest?) you will need to
take care that you never do 'host 2001:db8::1' but stay within the
boundaries. Typically this is non-issue, as you have rather large subnets,
and typically inside this subnet there is same security policy, that is,
all hosts can use same ACL.
It is easy to verify if particular ACE from ACL line is in hardware or
is punted, so it will be easy to fix it, before going live.
This is still definitely something you need to consider. I'd agree that no
IPv6/uRPF is rather show-stopper for longer term edge use, but I don't
think the IPv6/ACL is deal-breaker. In core I personally have no use
for uRPF or ACL, as I'm not facing customers in core.
EARL8 (Nexus7k) fixes the IPv6/uRPF and IPv6/ACL issue.
Someone mentioned the ACL TCAM, planning its usage is also important you
can use 'shot tcam counts' to see the resource usage. Pay particular
attention to 'LOU' usage (which is used for gt/lt/neq/range operators, and
is hence somewhat expensive). But knowing the limitation and how ACL lines
are compiled to ACEs makes it typically easy to scale as far you need to.
I'm guessing point Roland was making (which he likely would have not made couple moons ago:)
I've made this point for years, quite publicly, actually - even when it was unpopular for me to do so in certain quarters.
;>
uRPF for 7600/6500 can only be in one mode for the whole box, all interfaces. This is a major problem in many cases.
The NetFlow issues render flow telemetry unusable in production situations.
The ACLs work very differently on this platform due to LOU issues, as you say. Most folks don't know this, and many end up overflowing their TCAMs and not realizing it until their boxes fall over, heh. If one has fairly complex ACLs covering various ranges of ports, ACLs on 7600/6500 quickly become very difficult to manage.
EARL8 (Nexus7k) fixes the IPv6/uRPF and IPv6/ACL issue.
And the NetFlow issues.
Can someone provide a link, or more detail, on the netflow issues.
Particularly as they relate to 6509's and sup720's.
Thanks!
mls table can hold 256K entries at 93% efficiency, so you end up with about 239K flows total. No packet-sampled control of flow creation, so the table is likely to be overflowed in production edge situations, leading to non-deterministically skewed stats.
No logical OR of TCP flags throughout a TCP flow - can't classify SYN-floods, RST-floods, et. al.
No stats on dropped traffic.
Note that these issues are all fixed on the Nexus 7000, with the EARL8 ASIC.
uRPF for 7600/6500 can only be in one mode for the whole box, all
interfaces. This is a major problem in many cases.
I referred to this as 'chassis wide uRPF'. I'm not sure if that is big
issue in many networks. You run uRPF/strict to single homed customer and
uRPF/loose to transit/peering. Often networks already have dedicated
peering boxes.
I'm not sure, but I believe technically PXF (ES20, SIP600) and EZchip (ES+)
should have no trouble doing uRPF, so I think it's just software
development issue, that even with these cards, you're bound to this
limitation.
From my point of view, as long you can live with LAN cards in 7600 is has
excellent bang for buck, with really no competing products out there. But
the moment you need to terminate multiple customers to single port and
provide QoS (which implies H-QoS) the box has several attractive
competitors, most of them are even newer generation, such as MX.
The long and short of it is the current hardware (EARL7) is incapable of
doing sampling (i.e. looking at 1 out of every Nth packets). It gathers
all of the flow data into tcam and THEN does sampling in software, but
by that point its already too late because the tcam is exhausted.
Turning on sampling actually makes it worse, because it forces a
flowmask which fills the tcam even faster.
In my experience, even with extremely aggressive aging and a dest only
flowmask (discarding all src and L4 port information to make it fit
better), it tops out at around 2Gbps of "generic wholesale IP" traffic
you can sample. Obviously when it runs out of steam is dependent on the
number of flows in your network, you could be much better or much worse
depending on your traffic, but the point is it usually doesn't work for
most people. Adding DFC daughterboards makes this capacity scale
linearly, i.e. you go from 2Gbps system-wide capacity to 2Gbps per slot
capacity, but this typically doesn't make any difference.
The only recent improvement is that in SXH+ and SRB+ software you can
now enable netflow on a per-interface basis rather than a global basis
(before this, all traffic was sampled globally regardless of what you
configured on the interfaces). This can let you exclude interfaces you
don't care about (such as core links) use your limited resources only on
interfaces you do care about (such as edge links).
Until they come out with the EARL8 SUPs (what have they pushed that back
to now, 2011? :P) you are basically SOL in the netflow dept.
There has been a lot of good feedback regarding the deficiencies of the
7600 platform...
So, the new question is: what platforms should a small, start-up ISP
consider when looking to provide Ethernet services to their customers?
- Scalability - 100M, 1G, 10G access speeds (backplane limitations,
number of ports per chassis, etc.)
- MPLS Capabilities
- QoS Features
- Ease of configuration and support, etc. (finding NOC talent, scripting
tools, etc.)
- Software/Hardware "stability" and "longevity" (we don't want something
that is brand-new and therefore "buggy" nor do we want something that is
going EOL next year)
- Bang for the buck (both acquisition and on-going maintenance and
support)
I'm sure I'm missing a lot of things...are there any good presentations
from previous NANOG meetings that one should review?
Thanks in advance,
Ben
What do you consider a "small start-up ISP"? What kind of upstream
connectivity are you considering (or at least falls under the category of
small isp) bandwidht, bgp etc?
Juniper M10i versus Cisco ASR 1000
People use the 6500/7600 platform because it is dirt cheap, it mostly
works especially if you aren't doing anything too interesting or complex
with it (and if you have to ask, you probably aren't), and there is an
unlimited supply of "talent" (if you can call it that) who understands
basic IOS. If you're really a small ISP looking for a safe bet, this is
a fine choice. It's also available in quantity and for cheap on the used
market, which is probably where you want to go as a small ISP.
If on the other hand you're looking for a "good" platform and money is
no object, the Juniper MX is the unquestioned leader in this space.
Unfortunately it costs quite a bit more than a 6500/7600 (around 4x-10x
more depending on how good a deal you get on one, and how bad a deal you
get on the other), but you do get what you pay for. 
The other players in this space are the Foundry MLX/XMR and Force10.
Each has their advantages and disadvantages compared to Cisco, and may
be more appropriate for some people under some circumstances, but at the
end of the day they are both terribly flawed products too just in
different ways. Cisco still comes out with the significant price
advantage though, especially on the used market, which means for "most"
people who have to ask this type of question the 6500/7600 is the way to
go.
What do you consider a "small start-up ISP"? What kind of upstream
connectivity are you considering (or at least falls under the category
of
small isp) bandwidht, bgp etc?
two or three upstreams - OC-12 to 1G to each (BGP full tables)
three "POPs" meshed together
There has been a lot of good feedback regarding the deficiencies of
the
7600 platform...
So, the new question is: what platforms should a small, start-up ISP
consider when looking to provide Ethernet services to their
customers?
- Scalability - 100M, 1G, 10G access speeds (backplane limitations,
number of ports per chassis, etc.)
- MPLS Capabilities
- QoS Features
- Ease of configuration and support, etc. (finding NOC talent,
scripting
tools, etc.)
- Software/Hardware "stability" and "longevity" (we don't want
something
that is brand-new and therefore "buggy" nor do we want something that
is
going EOL next year)
- Bang for the buck (both acquisition and on-going maintenance and
support)I'm sure I'm missing a lot of things...are there any good
presentations
Why are you a "small start-up" and needing 600M-1G of pipe, and from
3x carriers? You can't use 150-200M via GigE ports and scale as needed
(assuming you aren't bound to a SONET loop)?
We started our IP backbone in 2005 with 3x 300M connections on
6509/maxed-Sup2s with 85% BGP tables and 6516-GBIC blades. All of our
drops were GBE or 10/100, no SONET, no fancy stuff. 3x nodes meshed.
Redundancy was our only mandatory requirement. Everything worked well,
until we started getting more sophisticated. This setup now could run
$5-6k/node if you shop around.
Since then, prices have dropped on more powerful stuff and persistent
EOL progression, so if you can pull off funds for Sup720-3BXL engines,
you've got options for 10G, IP6, MPLS, and full tables from day one,
although 10G ports are not cheap (for a shoestring one, at least). I
would consider Sup720-3Bs as a minimum for that platform, considering
EOL and features.
-D