Hello NANOG,
We have been struggling with firmware bugs from a specific router vendor. I am looking to set up a test lab of our core network and a few remote site routers. Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a little OSPF sprinkled in. I’d be grateful for any tips or resources anyone has that might cover testing strategies and/or best practices.
Thanks,
Ken
In any lab,I find concurrent access to serial ports is still an essential diagnostic tool. In a pinch you can get a used Cisco 2811 for $100, but there are multiport devices from lots of vendors. These let you SSH into the server and then connect to any serial port, giving you separate serial port windows all on the same screen. I’ve become fond of the WiFi-capable multiport modules from get-console.com. The ability to record logs from these serial ports in real-time helps a lot for documenting regression tests.
-mel beckman
If you’re able to virtualize some of these appliances, almost any virtualization platform can be setup to connect nodes together. However I really enjoy using Eve-ng. It’s a network virtualization web app that can be deployed on bare metal or in a VM. It lets you diagram, connect and configure setups and do testing with different network nodes. What is pretty nice with this is you can then just click on any of the connected links and wireshark pops up and you’re viewing packet captures instantly. It kind of reminds me of Cisco’s VIRL, but with maybe less overhead? It has enabled me to run and connect several vendor appliances and build up configuration prior to acquiring the physical hardware for deployment.
Appliance virtualization is perfectly acceptable for a lot of things. But there are large sets of problems that you will never catch that way.
To the OP :
With respect to ‘strategies’ :
Lots of people do #1. Not enough do #2. (Looking at you, equipment vendors. )
I agree with others here...
Physical lab - gotta have console server for the most control - perle console server is good, and also good ole fashion cisco terminal server (2509/2511 or 2600 with asynch module)
Virtual labs are great for testing features and functionality
- Juniper vLabs
- Cisco DevNet sandbox
- Cisco CML (i think fka VIRL)
- EVE-NG
- GNS3
I use these virtual environments a lot and do videos about them on my youtube channel, where I try to cover some SP-related topics. Hope it helps
https://jlabs.juniper.net/vlabs/
-Aaron
++ all that was said thus far. Physical equipment with console access is the best way to test software/firmware issues. As for virtualization, it's great for expanding your topology quickly.
Use a virtual bridge in GNS3 or EVE-NG and you can make your smaller footprint physical lab into a larger topology with ease -- especially around cabling. It also allows you to do packet generation & link simulation (packet loss, jitter) much easier. You can even couple it with T-Rex.
- Mark
if you are just testing control plane features the virtualized stuff works
well. i personally use vrnetlab.
if you are trying to test hw features and use actual hardware:
remote controlled power
optical cross connect boxes(calient/telescent)
if you are testing optical stuff with distance limitation something like a
timbercon spool
ixia/spirent
-b
I love the built-in Wireshark capability in EVE-NG. BTW, EVE-NG Community is free. You just have to get images for anything you want to emulate. Virtual images for various vendor products are sometimes freely available, with trail licenses. For instance Junipers vMX was freely available for a while with a 60 day license. …also vSRX, vQFX, and the new vJunos-switch (I think vEX).
Aaron