Hey folks. just curious what people are using for automating updates to
Linux boxes?
Today, we manually do YUM updates to all the CentOS servers . just an
example but a good one. I have heard there are some open source solutions
similar to that of Red Hat Network?
Cheers,
Paul
You can configure yum-updatesd to download and/or apply new updates
automagically.
Whether that's a good idea is a different question.
Hey folks. just curious what people are using for automating updates to
Linux boxes?
yum
Today, we manually do YUM updates to all the CentOS servers . just an
example but a good one. I have heard there are some open source solutions
similar to that of Red Hat Network?
yum install yum-cron
chkconfig yum-cron on
service yum-cron start
It so happens that just yesterday I stumbled across Spacewalk
(http://spacewalk.redhat.com) - which is the open source version of
RHN Satellite.
I ran into a few problems setting the server up - but nothing too
difficult to solve, and client installation is a breeze.
Dan
There's no tool I could recommend that would be very close to RHN.
However, for solving the problem of keeping packages up to date and
systems in a known-state, I would recommend checking out some
configuration management tools.
There are several popular ones nowadays, though I personally prefer
Puppet or Chef.
Both are tools that allow administrators to declare what a system
should look like, and abstract away the hard work of making that
happen on a variety of platforms. In both cases, it's possible to
monitor how well those tools are working and what they're doing in the
background so that you can get an idea of what's up to date and what's
not.
Are you just trying to solve for making sure that packages are up to
date? Making sure that running daemons are also up to date?
Cheers,
jof
We are using Security Blanket. It's a COTs product that works really well....
We use SALT, written in python and setup in 10 minutes. Seriously easy! Wickedly fast!
http://saltstack.org/
-Bret
I run spacewalk (as mentioned above), and have for some time. Once you get
the errata importing set up, it's pretty much full RHN.
-Blake
Awesome! I remember someone telling me about this before and couldn't
remember the name til now...
Cheers,
Paul
At work, we use (and built) a tool called 'tingle'
(https://github.com/anchor/tingle), which handles it all for us across our
internal and managed-for-customers infrastructures.
Personally, I don't run CentOS, but I use unattended-upgrades on my personal
herd of Debian machines, which works well enough.
- Matt
Here at Twitter we make extensive use of Puppet. It's great, but we had a
hard learning curve and much customization to get it to work the way we
wanted to.
I'd also recommend Chef, which is like Puppet but includes more tools (like
a machine database) out of the box.
-j
Something to think about before attempting to centrally manage, your
systems actually have to be centrally manageable -- that doesn't happen
automatically and requires extra work.
The just run yum update strategy is only reliable when all packages on the
system were installed from RPM and all software RPMs installed are
properly maintained by the vendor using Yum. Some packages have updates
that are distributed with Yum, but yum updating "breaks" the application,
until a manual update procedure is completed. Sometimes an updated kernel
won't boot. Sometimes, a third-party driver for RAID card X won't load
in the patched kernel, and after a reboot, the OS never comes back up
because it's sitting at a kernel panic message indicating no hard drive
found.
Cacti/OpenNMS are good examples -- after a yum update to a new version,
you must manually invoke, a potentially dangerous "installer" program or
web page has to be used, after a new update, config files, or database
schema have to be edited or patched by hand; until you manually take some
action to "fix" the config, the application is broken after update.
As soon as you attempt to restart the application it will shutdown OK, but
not come back up.
Occassionally, there is a library update that breaks binary compatibility
with existing applications, for example a certain update to
net-snmp-libs in Centos 5.something.
yum-updatesd surely doesn't know when auto-applying an update will cause an
important service to suddenly break
To centrally manage effectively, you basically need a homogenous
environment with a configuration that is very close to stock config, so
that effective testing is possible; homogenous meaning an identical list
of installed packages and software all installed the same way on every
system centrally managed as a group, identical SKUs for every hardware
component in every installation configured identically, same hw revisions,
etc.
No "extra" applications or files floating around on a one-off server.
So yum-updatesd would be a bad idea for production systems that have any
third-party packages;
even if YUM maintained. And even if YUM maintained, third party YUM
repos may become neglected,
or change into 404 errors, causing yum to break entirely.
Often commercial third-party software used on CentOS systems will be
distributed in another format, such as .tar.gz.
Yum cannot do much with that; the third party package will likely get
neglected and not updated.
Often various applications you require may need versions of libraries or
applications that are not yet
available in RPM format, or they're part of Fedora instead.
In any case, if you wind up rebuilding the RPM for CentOS using rpmbuild or
installing from source, Yum update won't help you with those packages,
and may break their dependencies later.
That might just be a testament to how poor the available packaged software
selections are in CentOS, that commonly needed packages aren't part of the
distribution; and commonly outdated versions of libraries are present.
But YUM-updatesd's usefulness certainly applies to less than 100% of
systems.
this is why i never update. i would rather build a new image and deploy it to the thousands of servers than worry about updates. be it an openssh security notice, or new ntp configuration, for me it is easier to rebuild servers than update config files.
.. you never update? How frequently do you rebuild your entire server stack, weekly?
Paul
For that matter, imaging is a bad way to go about handling this, you'd be better served by setting up something like Puppet or Chef and have them handle configuration management for you centrally, along with necessary software packages.
Paul
I looked into Puppet and though I've got it managing parts of our
infrastructure it seems quite difficult to bolt on to an existing
setup. There are also some things that I can't see how to do easily
with Puppet ("Don't upgrade packages on the live environment until
we've tested them in staging" being a big one.)
I'm starting to look at Blueprint (http://devstructure.com) to help
build the Puppet manifests so that we can deploy Puppet without
breaking any existing machines, Puppet for configuration management
and Spacewalk to audit what is up-to-date and help schedule security
updates.
Dan
Hey folks. just curious what people are using for automating updates to
Linux boxes?Today, we manually do YUM updates to all the CentOS servers . just an
example but a good one. I have heard there are some open source
solutions similar to that of Red Hat Network?
We did create our own solution and are still expanding it. Currently we set
what a server should look like at the servers, we want to change it to the
central system. This would make it easier to deploy extra servers (only
entering a MAC address, selecting software and starting a server should be
enough to auto-deploy it).
Our current solution is designed for Debian/Ubuntu, but should also work on
other Linux distributions.
A working copy might be available; please contact me offlist and I'll look
what I can do.
Kind regards,
Mark
Sounds like a poorly designed package. Wordpress does a good job of allowing back end updates without impacting the services provided, even with database changes.
Part of a well designed and maintained system is the ability to do painless upgrades.
Jared Mauch