[Paper] B4: Experience with a Globally-Deployed Software Defined

> No, people never use *flow controllers* for anything.

> People have been doing SDN since before Google was around.
> OK, so it was horrible expect scripts but it worked.

Not really.

Note I am talking about flow controllers in my first point.

(And I was trying to be funny to match Todd's tone, though
I guess it's dangerous to try to copy the master)

Re: flow controllers -

The idea of centralized decision makers doing something (typically
per flow) has been proposed, in my experience, by those with little
operational experience or those with extraordinarily constrained
topoligies, types of traffic, and usually external filtering to
constrain the types of traffic one could face.

Because... There have been no proposals that I have seen (or
that those who are at the Major Vendors who follow it more
closely tell me about when I ask a few times/year) to actually
deal with the every-packet-is-a-flow problem we saw first with
7206VXRs and that remain a real possibility for Internet-
connected networks. Distributing flow controllers and making
them hierarchical doesn't seem to help in the architectures
that I've seen proposed.

So it seems to be of use only for very tiny networks or for
very constrained and filtered or non Internet-connected topologies.

I'd be interested to be shown otherwise.

Automatic reconfiguration of routers is not what a software-defined network is.

It is one of the things (but not all of the things) that SDN provides.

A software defined network is one where the forwarding behavior can be
completely defined
in software running outside of the devices that perform the forwarding.

That said, I wince every time someone starts talking about (not suggesting
you are here but many do) making the routing engineer or designer in a box
that sits on the bottom or besides the network.

Those who have experience and/or run larger infrastructure usually say
words like "of course we have to worry about feedback loops" but many don't.

I think innovation is great but I don't think there are that many shops
that are better off writing their own control pane (centralized, distribtued,
whatever) right now.

It's worth remembering that Google is a software company. They are far ahead
in software defined everything.

You can write expect scripts all day; but you cannot turn your basic switch
into a Load balancer or stateful firewall with one.
or decide in real time exactly which destination Ethernet ports a packet
coming in a certain port is going to touch, without having structured
VLANs and static MAC tables on the switches ahead of time.

Changing device configurations with expect scripts is a limited tiny subset
of what SDN is.

True, but the number of production environments that are going to be more
stable or scalable by having people build their own control logic is pretty
small in my experience.

And being able to debug and reach out to a community of operators with
a common set of experience of what to do, not to do, and how to debug
is extraordinarily valuable for production networks.

When I look at most of the non-Google big guys, SDN means pushing the
vendors for better control plane instrumentation and ability to program
(but more on the instrumentation side as where the gaps have been), and
potentially to get some cross-provider way of doing the above.

+ having merchant silicon one can get/use for cheap, typically for more
constrained topologies, doing pretty dumb switching and/or routing stuff.

-JH

Where I see the delta a lot given the customer conversations I have is
in the magic provisioning of cloud network infrastructures.

New school SDN is that everything is a tunnel, magic software maps things,
commercial providers doing this uniformly have to aggressively rate-
limit their clients, and performance for content delivery is limited
because the hypervisors must be briding and can't do PCI passthrough or
SR/IOV.

Old school SDN (not really that old school) is API-based provisioning
of network devices with vendor support (let's say Juniper) to do
filtering, VLANs, and shaping and tunnels where needed.

It'll definitely be interesting to see where things go over the next
few years.

I know tens of companies who have run away from cloud providers with
new(er) school SDN-ish infrastructures for the simplicity of just
having some high performance dedicated machines/hypervisors with
dead simple switching infrastructure.

Anyway, innovation is great but I just see few companies with the
understanding to go build their own control plane software to
connect to the Internet with. And those vendors who do build it
will get borged by one of the routing/switching vendors and things
will become product features, differentiated by providers, most
likely. (Though I hope not)

Avi

> > OK, so it was horrible expect scripts but it worked.
> Not really.

The idea of centralized decision makers doing something (typically

per flow) has been proposed, in my experience, by those with little
operational experience or those with extraordinarily constrained
topoligies, types of traffic, and usually external filtering to
constrain the types of traffic one could face.

Flow controllers are for constrained topologies. You have some
interesting problems, if you decide to try to bring flow controllers into
a carrier network, and have them processing flows coming in from an
unprotected network. How do you feel about a few billion flows per
second, thanks to the attacker's randomized source IPs and port numbers?

You might very well have to "constrain the topology" by having a
capability to provide a flow entry for the destination IP address, and
associating all the traffic with just one "flow".

closely tell me about when I ask a few times/year) to actually
deal with the every-packet-is-a-flow problem we saw first with
7206VXRs and that remain a real possibility for Internet-
connected networks. Distributing flow controllers and making

The centralized controller doesn't work well in topologies where you
cannot take the first packet -- setup the flow tables on the devices,
and keep further control traffic limited in number of messages and limited
in number of bits.

You need a network path for control traffic, you need CPU capacity for
your flow processing -- and there will always be RTT latency and various
other penalties incurred by extra overhead to reach a remote controller
that is geographically farther away from the route processor since the
speed of light is finite, and any centralized controller will have finite
capacity...

Having all the flow processing on central controllers is the opposite of
load-balancing --- it is concentration of a distributed load; which is a
recipe for creating a capacity bottleneck on the controller and on all the
devices that have to have uncongested paths to the controllers...

It's a similar idea to storing part of your router's RIB on a hard disk
or magnetic tape, because RAM is so expensive ---- you're going to
increase the forwarding delays of some traffic, or some connection
initializations by doing so.

Therefore; the only flow controller that really makes sense in all
situations is eventually going to be one that can download its entire
state into every router --- that is: A route controller program built
into every router and switch (even if defined by software).

Such a product for SDN doesn't exist yet? -- other than traditional
networking devices which don't allow you to implement an API and download
arbitrary code to them in a vendor-independent way.

It seems like there is still a lot of work and standardization for vendors
to be doing, before a majority of ISP networks could consider using SDN
anywhere.

So it seems to be of use only for very tiny networks or for
very constrained and filtered or non Internet-connected topologies.

Yes. That seems to be where the greatest benefits of current SDN
products could realistically be experienced at the present time; small
filtered and private topologies, where the controller as a bottleneck risk
is minimal.

> in software running outside of the devices that perform the forwarding.

That said, I wince every time someone starts talking about (not suggesting
you are here but many do) making the routing engineer or designer in a box
that sits on the bottom or besides the network.

I don't know -- if you make a box that can help configure your network;
you'll still need a routing engineer to setup and maintain that box, make
sure it continues to work properly, spend all day on hold waiting for
support to help get your network back online, and diagnose things when
an inevitable bug crops up and starts costing more damage than money saved
by deploying the magic box.

I think innovation is great but I don't think there are that many shops

that are better off writing their own control pane (centralized,
distribtued,
whatever) right now.

Yes. Most would have no business trying to write their own control plane.
Not too long ago someone wrote about "Tempering your MacGuyver" streak.

Which is exactly what should apply here; SDN is something to learn about
developments in, and one of those new flow controller schemes could be
used in a pinch as a way around certain problems ---- new tech doesn't
mean it's somehow better, or somehow magically defeats inherent physical
or computational problems.

Otherwise Rule #1 of building reliable infrastructure -- is use time-tested
technology in well-understood vendor supported configurations in every
instance, except in the situations where there exists compelling
justification to vary.

Flow controllers won't belong on everyone's network just because they're
new tech -- until they have more to offer, it would be the exception to
the rule that you should want one.

It's worth remembering that Google is a software company. They are far
ahead in software defined everything.

It could be interesting if Google released some of their sw and sw/hw
solutions I suppose. Of course there are likely to be people copying
them in such case, because of a perception (true or false), that using
SDN tech early on is part of their success story.

When I look at most of the non-Google big guys, SDN means pushing the
vendors for better control plane instrumentation and ability to program
(but more on the instrumentation side as where the gaps have been), and
potentially to get some cross-provider way of doing the above.

That bit is the most useful. Network devices have notoriously poor
control plane instrumentation and programmability; SNMP and Netflow have
their limitations.

Improvements in these areas are valuable in their own right.