Brainstorming acceptance issues - WAN impediment


I am in the process of testing an 'automation/sdn' kind of controller, it
will be managing configuration on our routers and also deploying some VNFs

Before accepting it, i'd like to perform some testing, to make sure of the
behavior if there are network issues between the controller and the devices
(routers or servers), during creation of services.

From the top of my head, I can think of the basic tests like introducing

jitter and delay but i would appreciate more ideas or even test cases that
i can re-use.



For sure you need to check failover scenarios, reboot the controller or
isolate it during some time.

- Traffic getting blackholed in one direction or the other (e.g. not something obvious like links down or ICMP unreachables)

- "fun" things like MTU issues resulting in some traffic getting through but not all

- Are there redundant controllers? Can they get split-brained?

Introduce packet loss. Trigger timeouts. Arrange to have packets arrive out
of order. Send it fragged packets. Send it fragged packets of incorrect
length. Send it fragged packets, and omit one frag and see if it behaves. Test
its behavior if the frags overlap, or the offset of a packet plus its size put
it over 64K. Send it christmas tree packets. Send it commands that aren't
authenticated by whatever you're using. Find out what the underlying OS
(probably linux or *bsd), and hose it down with nmap to find surprises.
Syn-flood it. Send it a long sequence of commands that change from state 1 to
state 2 and back, see if it gets confused if the commands arrive faster than
the amount of time it takes to process a command.

I could probably think of more, but I'm low on caffeine at the moment...