Managing IOS Configuration Snippets

Very cool, thanks Erik. I can think of many ways to encode version
metadata. Probably best to be somewhere in between overly verbose (full
version $Id / date / author for every config chunk) and being unreadable
(base64 encoded gzip of unique configlet identifiers and versions).
Updating a banner feels a bit easier when you are pushing a full
device-specific configuration from a templating system. Regardless of where
it is stored, keeping the metadata in one of these fields (banner for
example) means that checking the contents of the banner configlet now
requires slightly more logic - which is fine.

Chuck, interesting use of alias.

Simon, completely agree that the network itself should not be the intent
store. The real focus here is when your intent is in a DB/templating system
thingy, how do you operationally ensure that intent matches reality. Again,
with many devices going through upgrades, disabled/unreachable devices, new
devices, pre-configured devices. The intent pusher is not blindly and
constantly pushing to all devices, and it's likely not safe to do that.

Absolutely. Expect/SNMP/config scraping is a solution here. It's
tedious and boring to write the hooks, but it's not impossible.
A solution like tail-f is a much better solution here. My personal
preference would be to just push/pull JSON off the devices.

I think there are two separate components here (which are often
conflated): the mechanics to push/pull from devices into a data
structure, and the network database to work with those data
structures. There's a place for both device level models and network
level models.

--Simon

This is not any kind of sensible answer to the original question, but the general approach “give ops people a shell on a box with a rancid repository, encourage them to write scripts that do stuff” has the potential to cause all kinds of good things to happen faster than the time taken to organise a conference call to discuss requirements gathering for a “production” system.

Examples, as ever:

  http://www.nanog.org/meetings/nanog26/presentations/stephen.pdf
  ftp://ftp.isc.org/isc/toolmakers/

(It’s not pretty. But it’s not supposed to be pretty.)

Joe

+1000. And that applies equally to the backend. I have yet to meet a fancy, integrated, database-driven configuration management system that can beat a bunch of flat files and a few perl scripts. Hackability of a system can be a definite virtue here.
-e

Definitely. Depends what form the database takes - I don't think SQL
is the right answer here. Sticking with flat files and perl scripts as
much as possible is good guidance.

I'm biased, but I'd go with Python: http://www.youtube.com/watch?v=EGK5jjyUBCQ

--Simon

Putting aside the fact that snippets aren’t a good way to conceptualize deployed router code, my gut still tells me to question the question here. The first is does this stuff change often enough to warrant a fancy versioning solution? I have yet to see NTP deployed in a different way than when I first learned to configure it. Next, when it does change how often is it not rolled out to every router. If NTP or CPP or SNMP or some other administrative option were configured differently across my network I would want to audit it and fix not version control. What if some of the configs don’t match the defined versions? It may be better to create standard templates and version them in SVN or GIT and then use config backups to track which devices have the standard configs. There are some for pay tools that can search for certain statements on various boxes and either alert or remediate when differences are found.

Putting aside the fact that snippets aren't a good way to conceptualize deployed router code, my gut still tells me to question the question here. The first is does this stuff change often enough to warrant a fancy versioning solution? I have yet to see NTP deployed in a different way than when I first learned to configure it. Next, when it does change how often is it not rolled out to every router. If NTP or CPP or SNMP or some other administrative option were configured differently across my

sure, so you're saying that a large bit (maybe) of the router config
is 'one size fits all' and 'never changes' where 'never' is really
'very infrequently'.

sure, agreed... but there are parts of the config that do change more
frequently (depending on the network perhaps)... how do you go about
seeing which version / setup is deployed EXCEPT by building a
home-grown 'config parser' and seeing that 'what is deployed matches
mostly what I have in my config store for this
router/class-of-router/network' ?

It's a shame that vendors of network equipment don't have to manage
large networks of their own equipment under constrained opex
environments (no fair comparing contracted work where you bill for
time + materials, that's the wrong incentive set)... I bet that'd get
them to fix stuff up right quick.

network I would want to audit it and fix not version control. What if
some of the configs don't match the defined versions? It may be
better to create standard templates and version them in SVN or GIT and
then use config backups to track which devices have the standard
configs. There are some for pay tools that can search for certain
statements on various boxes and either alert or remediate when
differences are found.

Keegan, don't get me wrong, I am not suggesting that even if version
numbers were happily encoded in robust comments that this would be the same
as actually digesting the configuration. If the function of checking using
'fancy versioning' is not an operational best practice, what do you suggest
(all-knowing/singing/dancing tool which understands the configuration and
your intent aside)? You say IF NTP or CPP were configured differently -
with a large enough network there will always be configurations which have
differences. With that as an operational constant, how do you determine
which devices have the latest iteration of your line vty configuration.

How often will a change not be rolled out to every router. This is again
related to the size and churn of the network, but my practical estimation
is that once you get into thousands of routers there will almost always be
some that get missed. Comprehensive auditing is very important, and
arguably more useful than version checking - but it requires that you make
knowledgeable and complete assertions. I assert the my snmp config should
look like the snmp snippet version 77 is easier to grok than "make sure our
community string is not set to public" (and repeat hand-crafted audit logic
for every segment of the config).

What if some of the configs don't match the defined versions? This is why
it may make sense to break snippets into functional areas. "Just fix it"
might be sane for a banner, but squashing an interface mtu change that was
put there for a reason could end in tears. I consider this bit out of the
scope of the question, but yes it is another important problem.

Keegan, don't get me wrong, I am not suggesting that even if version numbers were happily encoded in robust comments that this would be the same as actually digesting the configuration. If the function of checking using 'fancy versioning' is not an operational best practice, what do you suggest (all-knowing/singing/dancing tool which understands the configuration and your intent aside)? You say IF NTP or CPP were configured differently - with a large enough network there will always be configurations which have differences. With that as an operational constant, how do you determine which devices have the latest iteration of your line vty configuration.

That’s what I mean. The things that lend well to to version tracking don’t tend to change much. How many ways are there to configure VTY lines, or NTP, or CPP, or even OSPF and if we’re talking about an access ACL why not just audit the configs to make sure that all the entries are there. Am I really going to care that one router has version 1.0 versus another router that has version 2.2.12 build9? It’s not source code..

How often will a change not be rolled out to every router. This is again related to the size and churn of the network, but my practical estimation is that once you get into thousands of routers there will almost always be some that get missed.

Again, a router that was missed is a reason for audit and remediation not versioning. If you find a router with config missing does it really matter what version it’s on and when that version was valid? Not in my experience.

Comprehensive auditing is very important, and arguably more useful than version checking - but it requires that you make knowledgeable and complete assertions. I assert the my snmp config should look like the snmp snippet version 77 is easier to grok than "make sure our community string is not set to public" (and repeat hand-crafted audit logic for every segment of the config).

There may be some differences, but those are normally due to equipment lifecycle, mergers/consolidations and such. It’s easy to refer to something as the config for a particular platform or company than a version number. This can be tracked in GIT or SVN. Even then there will not be constant changes. I’d lean towards standardization. So the equipment that cannot adhere to the defined standards probably won’t evolve much on it’s own.

What if some of the configs don't match the defined versions? This is why it may make sense to break snippets into functional areas. "Just fix it" might be sane for a banner, but squashing an interface mtu change that was put there for a reason could end in tears. I consider this bit out of the scope of the question, but yes it is another important problem.

I wasn’t saying just fix it. I was saying that router configs don’t lend well to versioning. With software for example, if something is different it might be a different version of that application with compatibility issues, dependencies, library issues, etc. When it’s a router config chances are someone fat-fingered something. Most of the time the best thing to do is to fix or at least alert on the error, not to record it as a valid config version. Again, this is for things that lend themselves to snippets. ACL’s, NTP, SNMP, CPP, even Spanning-tree. Not for things like interface IP’s or static routes that may be different across different boxes or location. If you’re referring to the latter I may have misunderstood your question..

What I have always wanted is a way to group configuration, in particular by customer. Ideally with the ability to see it both as a unified view, and also as a per-customer view.

For instance:

customer AAAAA
  interface GigabitEthernet1/2/3.10
    description AAAAA
    ip address 10.0.1.1 255.255.255.0
  router bgp 1
    neighbor 10.0.1.2 prefix-list AAAAA-in in
  ip prefix-list AAAAA-in 10.1.0.0/24
end

customer BBBBB
  interface GigabitEthernet1/2/3.11
    description BBBBB
    ip address 10.0.2.1 255.255.255.0
  router bgp 1
    neighbor 10.0.2.2 prefix-list BBBBB-in in
  ip prefix-list BBBBB-in 10.2.0.0/24
end

Then I should be able to do:

show run - Normal output like we see today, the "device" view.
customer AAAAA show run - Same format as I have above, just config relevant to customer AAAAA.

I can even see extending the tag to work with some other commands:

customer AAAAA show int
customer AAAAA show bgp ipv4 uni sum
customer AAAAA show ip prefix-list

The same functionality would work for snippets:

customer ntp-servers-v1.0
ntp server 1.2.3.4
ntp server 1.2.3.5
ntp server 1.2.3.6
end

Basically this follows the two modes in which engineers look at a device. Most of the time is configuring a specific customer, and wanting to be sure they are configured right; including the hard case of "no customer AAAAA", that is making sure all configuration for a specific customer is removed. The rest of the time is typically troubleshooting a network level problem where you want the device view we have today, I see interface Gig1/2/3 is dropping packets, "show run" to see who's configure on it sort of operations.

I don't know of any platform that has implemented this sort of config framework though.

It sounds like your suggestion is to not check version numbers, write smart
audits and react. Thinking of ACLs for example, version checking from a
remark is a limted shortcut (but still very useful way) to validate
distribution of an ACL. These things may change ~continuously. Grabbing a
config and jamming that into some rancid rcs thing is different than intent
version tracking. Our friends at Juniper (or the IOS-XR folks) seem to
think the version control paradigm lends itself well to router configs,
albeit at the whole-config level.

Oh, and I think we're crossing streams here regarding versioning. I am not
suggesting to take the configured state of the network, consider that
proper intent, record it in a revision and then check for it in the future.
I am assuming that the intent versions represent reviewed and tested
configuration states by ops/neteng. Different across the fleet does not
necessarily mean non-deterministic, so intent versioning can still be
useful. I'm sure (not first-hand sure) that some of the other suggested
templating systems could build parts of the expected config based on
attributes of a device.

Isn't that framework called Juniper Op scripts? Run command which takes xml
output of config (or other RPC) applies xslt, produces the output you
describe.

What I have always wanted is a way to group configuration, in particular by customer. Ideally with the ability to see it both as a unified view, and also as a per-customer view.

For instance:

customer AAAAA
   interface GigabitEthernet1/2/3.10
     description AAAAA
     ip address 10.0.1.1 255.255.255.0
   router bgp 1
     neighbor 10.0.1.2 prefix-list AAAAA-in in
   ip prefix-list AAAAA-in 10.1.0.0/24
end

...

Basically this follows the two modes in which engineers look at a device. Most of the time is configuring a specific customer, and wanting to be sure they are configured right; including the hard case of "no customer AAAAA", that is making sure all configuration for a specific customer is removed. The rest of the time is typically troubleshooting a network level problem where you want the device view we have today, I see interface Gig1/2/3 is dropping packets, "show run" to see who's configure on it sort of operations.

I don't know of any platform that has implemented this sort of config framework though.

It's not the cleanest, but in junos you can pretty much get this by defining a configuration group per customer and applying them all. Your RE may hate you at commit time, but I've seen this approach work quite well.

-e

Thus spake Ryan Shea (ryanshea@google.com) on Thu, Feb 27, 2014 at 09:38:33AM -0500:

Now, I hand you the 'show run' output and ask you if version 77 of the vty
config is on this device. Can you answer the question? Now I hand you the
'show run' from 10,000 more device configs - and 100 more configuration
chunks from revision control. Can you still answer the question? Assume a
magical revision-history-aware configuration cross reference parser (while
a noble and lovely goal) is not available.

Hi Ryan,

If I'm understanding what you're trying to do, you could script around
our rather unsophisticated 'sgrep' (stanza grep) tool combined with
scripting around rancid & rcs to do what I think you are looking for.

http://net.doit.wisc.edu/~dwcarder/scripts/sgrep

sgrep can dump out a "stanza" of ios-like config, then you can rcsdiff
that to your master, per 'chunk' of config.

ex:

sgrep -s "vty" r-cssc-b280c-1-core.conf

Found stanza in r-cssc-b280c-1-core.conf size:9
line vty 0 4
access-class G-A-AdminAccess in
exec-timeout 30 0
ipv6 access-class G-A-v6AdminAccess in
line vty 5 24
access-class G-A-AdminAccess in
exec-timeout 30 0
ipv6 access-class G-A-v6AdminAccess in
!

See the -s and -e options for our sgrep.

Add 'xargs -P' around your glue, and I think you'd be in luck. If you
were building configs around this model, you could use m4.

Dale

Thus spake Keegan Holley (no.spam@comcast.net) on Fri, Feb 28, 2014 at 09:49:19AM -0500:

I wasn’t saying just fix it. I was saying that router configs don’t lend well to versioning.

Um, what?

$> rlog r-cssc-b280c-1-core.conf | grep 'total revision'
total revisions: 2009; selected revisions: 2009

When it’s a router config chances are someone fat-fingered something. Most of the time the best thing to do is to fix or at least alert on the error, not to record it as a valid config version.

We have our operators manually check in revisions (think in rcs terms:
co -l router, go do work, verify it, ci -u router) rather than
unsolicited / cron-triggered checkins. Then the check-in message
contains the operator's description text of the change and often a
ticket number. So there slightly fewer fat-finger configs checked in.

Dale

Thus spake Keegan Holley (no.spam@comcast.net) on Fri, Feb 28, 2014 at 09:49:19AM -0500:

I wasn’t saying just fix it. I was saying that router configs don’t lend well to versioning.

Um, what?

$> rlog r-cssc-b280c-1-core.conf | grep 'total revision'
total revisions: 2009; selected revisions: 2009

I wish you were here to see my eyes rolling.. 2009 versions of something are no more grok-able than one current version. Congrats, you have a config backup system.

When it’s a router config chances are someone fat-fingered something. Most of the time the best thing to do is to fix or at least alert on the error, not to record it as a valid config version.

We have our operators manually check in revisions (think in rcs terms:
co -l router, go do work, verify it, ci -u router) rather than
unsolicited / cron-triggered checkins. Then the check-in message
contains the operator's description text of the change and often a
ticket number. So there slightly fewer fat-finger configs checked in.

That’s not what the OP was looking for AFAIK. This is just change management.

I'm digging the idea of your command. Along the same lines I've got this awk snippet that I made and then forgot about. It functions like the cisco pipe begin/end commands:

#!/bin/sh

if [ "x${2}" = "x" ]; then
     awk "/${1}/{temp=1}; temp==1{print}"
else
     awk "/${1}/{temp=1};/${2}/{temp=0}; temp==1{print}"
fi

Usage:
cat router-confg | begin_regex 'line vty' '!'

If you omit the second argument it just shows you from your match until the end of file.