(Jared wrote):
<snip>
Most people I've seen have little data or insight into their
networks, or don't have the level that they would desire as
tools are expensive or impossible to justify due to capital costs.
Tossing in a recurring opex cost of DC XC fee + transport + XC fee +
redundant aggregation often doesn't have the ROI you infer here.
I've put together some models in this area. It seems to me the
DC/real estate companies involved could make a lot (more) money by
offering an OOB service that is 10Mb/s flat-rate for the same as an XC
fee and compete with their customers.
Equinix does have a very aggressively priced 10Mb/s flat-rate OOB (single
IP only but that's not that hard to work around) for essentially XC
pricing. It's been stable but not something you'd rely on for 100%
packet delivery to some other point on the Internet (so more for
reaching a per-pop OOB than for making a coherent OOB network with
a bunch of monitoring running 24x7).
Still, it's a good value for what it is.
<snip>
- Jared
Avi Freedman
CEO, Kentik
avi at kentik dot com
I've not read every reply, but let me add my voice as some who has worked
on and ran SEVERAL large networks, in no case in the last long number of
years have I had access to an OOB network that was sized to carry anything
in large volume, and in fact like many others replied on a robust number of
path at that us many the networks inband. These networks survived many
"large" DDoS attacks and far more fat finger incidents then I like to think
about. I don't think I've even worked with a client network as far as I
can remember that had a nailed up / robust OOB network for data collection
or other purposes.
-jim
What I'm saying is that keeping flow telemetry and other management-plane traffic from mixing with customer data-plane traffic is important in order to ensure visibility and control during a significant network-impacting event.
I've personally been involved in assisting multiple operators in multiple incidents in which either DDoS attack traffic or inadvertent routing redistribution excursions led to loss of visibility and control, resulting in unnecessarily-long times to resolution.
Virtual separation is generally Good Enough, and what we see with customers who run it all in-band is an increasing number who're taking steps to achieve at least virtual separation (~20%, as Avi noted, is about what we see implemented, currently). It isn't nearly as many as we would like to see, and it isn't happening as fast as we'd like to see it, but we encourage it wherever we can.
The OP on this thread was essentially asking about the best approach. OOB, whether virtual or physical, is the best approach. Economic factors may militate against this, at least initially, but a disaster or two can change that economic analysis.
I also suspect that increasing use of 'SDN'-type (apologies for using that overused acronym!) orchestration across the entire network topology (e.g., not just within the IDC) is going to lead to more separation of management-plane traffic from customer data-plane traffic, as the implications sink in.
How many IPv6 addresses do you get?
Frank