Hello,
For those than run Internet connected routers, how do you get your NetFlow data from the routers to your collectors? Do you let the flow export traffic use the same links as your customer traffic to route back to central collectors? Or do you send this traffic over private network management type path? If you send this traffic over the "Internet" (within your AS), are you worried about security?
Thanks,
Serge
This is how to do it.
So in case of a network partition event, you don't end up losing visibility into your network traffic when you need it the most.
You should already have an OOB/DCN network for managing your routers/switches/hosts, no? Use that, after ensuring its scaled adequately.
We forward it in-band.
Have been doing so for years, no major drama.
Mark.
Until there is.
;>
Roland is correct. With the caveat that your Internet customer traffic may flow over the fibers as your separate management circuits. You should aim for end to end physical diversity. This is all common sense, but laziness some times takes precedence.
Roderick Beck
Sales - Europe and the Americas
Hibernia Networks
http://www.hibernianetworks.com
Budapest and New York
36-30-859-5144
rod.beck@hibernianetworks.com
It's usually not laziness, it's most often related to cost.
To answer your first question: i see no issue in transporting flow
export traffic over the same backbone used to serve customer traffic.
Not entirely security related, but a neat trick is to use a tool like
'samplicator' to distribute the UDP packets to all applications of
interest. You'll find that on many router platforms you can only
configure a limited amount of netflow/sflow collectors, often less then
the amount of applications that need the data for dissemination.
Especially if you have multiple independent instances of the application
for redundancy purposes!
And, keep in mind, you can anycast the instances of 'samplicator' in
your network 
https://github.com/sleinen/samplicator
Kind regards,
Job
This is not good advice, for the reasons I stated previously in this thread.
Your advice is not "one size fits all".
I've done netflow over production links for two very large backbone
networks. Over the combined 17(?) years, never saw a problem. If your
network is likely to be partitioned by a small number of failures, that
might be a different case, but you can't make blanket statements.
-Steve
Your advice is not "one size fits all".
Actually, it is.
Large backbone networks have DCNs/OOBs, and that's where they export their NDE.
I've done netflow over production links for two very large backbone
networks.
Did you manage your routers and switches and hosts and so forth in-band, too?
Over the combined 17(?) years, never saw a problem.
Until you do.
Running flow telemetry in-band is penny-wise and pound-foolish, for networks of any size, in any circumstances. All management-plane traffic (and that's what flow telemetry is) should be segregated from the production network data plane.
Roland,
While your way may be best practice, sometimes real life gets in the way of best practice.
Shane
* rdobbins@arbor.net (Roland Dobbins) [Tue 01 Sep 2015, 19:13 CEST]:
Running flow telemetry in-band is penny-wise and pound-foolish, for networks of any size, in any circumstances.
You're just wrong here.
-- Niels.
Sorry, I'm not. I've seen what happens when flow telemetry is 'squeezed out' by pipe-filling DDoS attacks, interrupted by fat-fingers, etc.
It'll happen to you, one day. And then you'll understand.
So in your world, the money always exists for a separate flow telemetry network?
It should've already been spent for an OOB/DCN network, which should've been provisioned with flow telemetry in mind.
I think the key here is that Roland isn't often constrained by
these financial considerations.
I would respectfully disagree with Roland here and agree with
Job, Niels, etc.
A few networks have robust out of band networks, but most
I've seen have an interesting mixture of things and inband is usually
the best method.
Those that do have "seperate" networks may actually be CoC
services from another deparment in the same company riding the same
P/PE devices (sometimes routers).
I've seen oob networks on DSL, datacenter wifi or cable swaps
through the fence to an adjacent rack.
An oob network need not be high bandwidth enough to do netflow
sampling, this is well regarded as a waste of money by many as the costs
for these can often be orders of magnitude more compared to a pure-IP
or internet service.
I'll say this ranks up there with people who think
MPLS VPN == Encryption. It's not unless you think a few byte
label is going to confuse people.
- Jared
* rdobbins@arbor.net (Roland Dobbins) [Tue 01 Sep 2015, 19:30 CEST]:
You're just wrong here.
Sorry, I'm not. I've seen what happens when flow telemetry is 'squeezed out' by pipe-filling DDoS attacks, interrupted by fat-fingers, etc.
This is the dumbest thing I've read on this mailing list in a while.
(On the other hand, I don't read most threads.)
It'll happen to you, one day. And then you'll understand.
Who is saying I haven't done all of the above already?
-- Niels.
On 9/1/15, 1:36 PM, "NANOG on behalf of Roland Dobbins"
It should've already been spent for an OOB/DCN network, which should've
been provisioned with flow telemetry in mind.
I'm going to interpret that "should" in the same way as the MUST in
RFC6919. 
Yes, it's a good practice, but like most other proactive security
measures, is extremely hard to justify spending money on it to avoid the
risk that it breaks fantastically when it is needed most.
Though you could provide a little insurance against the problem you're
highlighting here via a QoS policy that prioritizes flow data over
customer traffic.
Several of the OOB networks/designs I'm familiar with significantly
predate the entire concept of flow telemetry, as well as my own networking
career, and are still rocking the same set of Cisco 2500 routers with
async cards (many with uptimes measured in years) and 64k leased lines or
dialup on demand they've been using for literally almost 2 decades. When
one of those ancient devices dies of old age, you scrounge for the
cheapest equivalent you can find to replace it to maintain your oob access
to the 9600/8/1/none console ports for when things have gone truly
pear-shaped.
Often there is a separate management network that can deal with ethernet
speeds, but it's separate for security reasons and not always as rigidly
independent from the in band network for connectivity, i.e. It might be a
VPN riding over the regular network and thus not completely protected from
the problem you're concerned about.
Thanks,
Wes
Anything below this line has been added by my company’s mail server, I
have no control over it.
hey,
It should've already been spent for an OOB/DCN network, which should've
been provisioned with flow telemetry in mind.
Bad advice. No amount of money will fix major platforms that are not happy to export flow telemetry via router management ports. Sometimes it can be done via nasty vrf leaking hacks, sometimes it cannot be done at all. Management ports are typically directly connected to routing engines while netflow data is generated in hardware in PFE.
In-band netflow works on all platforms without such issues.
And that box ended up in your rack, why exactly?
<insert excuses that boil down to "We were willing to accept gear that didn't
have this functionality because we were OK with sending flow telemetry over
the inband network">