Everything is a Freaking DNS problem

A couple of weeks ago I noticed a weird drop in web usage stats on the site you are browsing now. Kinda weird as the drop was right around Fosdem when usually there is a spike in traffic.

So before you start.. no I don't preach on practice on my own blog, it's a blog dammit, so I do the occasional upgrades on the actual platform , with backups available, do some sanity tests and move on, yes I break the theme pretty often but ya'll reading this trough RSS anyhow.

My backups showed me that drush had made a copy of the Piwik module somewhere early february, exactly when this drop started showing. I verified the module , I verified my Piwik , - Oh Piwik you say .. yes Piwik, if you want a free alternative to Google Analytics , Piwik rocks .. - I even checked other sites using the same piwik setup and they were all still functional happily humming and being analyzed.... everything fine ... but traffic stayed low ..

This taught me I actually had to upgrade my Piwik too ...

So that brings me to the point I`m actually wanting to make...
as according to @patrickdebois in his chapter on Monitoring "Quis custodiet ipsos custodes?" who's monitoring the monitoring tools, who's monitoring the analytics tools,

So not only should you monitor the availability of yor monitoring tools, you should also monitor if their api hasn't changed in some way or another.
Just like when you are monitoring an web app you shoulnd't just see if you can connect to the appropriate http port, but you should be checking if you get sensible results back from it , no gibberish.

But then again ... there's no revenue in my blog or its statistics :)

Over the past couple of days I've been fighting with a weird puppet problem , we eventually cracked it , but I promised a bunch of you to fully explain it here ;)

So we were deploying 2 Blade chassis at a pretty remote location with a mix of phyisical and virtual machines, some 48 instances in total. This is a pretty standard rollout, we've got a bunch of similar platforms in our lab , so we knew about a couple of glitches, what to expect etc.

I was just keeping an eye on the deployment, looking at the logs seeing if things were running fine, when suddenly a couple of puppet runs didn't come trough, we had seen such behaviour before, usually it's a matter of running them a gain a couple of times and they will come trough. (Upgrading ruby and putting passenger in front of puppet actually solved those issues,
We'd even had a loop built in the platform that runs puppet a couple of times till it returns with the correct exit code just to make sure. )

We were first scratching the A chain of our setup, so that in the event of failure we could still bring up the B chain of the platform and be up and running again. Actually machines were coming up.. slowly .. some of them took a bit longer . One of the machine's clock was seriously off .. the SSL was barfing on it , so we set the bios clock, and restarted .. it was the machine with 6VM's took a while but everything was back on schedule.. then suddenly things were going down fast more and more puppetruns started failing and .. , at some point in time actually none of our puppet runs were working again .
I'd see the puppetmaster perfectly compile it's catalog

notice: Compiled catalog for ctl-0-a

Then the client .. not wanting to get it ..

Mar  1 11:10:45 ctl-0-a puppet-agent[3674]: Not using expired catalog for ctl-0-a from cache; expired at Tue Mar 01 09:50:06 +0000 2011
Mar  1 11:10:45 ctl-0-a puppet-agent[3674]: Using cached catalog
Mar  1 11:10:45 ctl-0-a puppet-agent[3674]: Could not retrieve catalog; skipping run

We had gone from about 60% of our fresly deployed boxen working fine, to not one
So what do you do .. indeed .. you turn on debugging.
You put both your puppetmaster and client in debug. Nothing, no errors no nothing ..

I asked some collegues, asked on irc .. much ideas but none of them that actually cracked the problem. I did what I knew that solve similar problems before,

I switched our serialization format from yaml back to pson , and back, no luck.
I upgraded ruby to a version from the glei.ch repository. No luck.
I upgraded our Puppet version 2.6 to a version from the TMZ Epel repo , we cleaned out ssl the certificates on all sides multiple times. Cleaned out /var/lib/puppet , We uninstalled puppet and reinstalled it.
It wasn't a DNS Problem

I had started stripping my manifests to empty runs, those worked, then started uncommenting the actual manifests again ... Then in the middle of the debug our VPN connection to the remote location broke down, we'd only be getting it back in the morning ..about 12 hours later not fun. Murphy obviously ..

So the next morning we dived right back in ... making those manifests bigger again, removing all the stages, 1 or 2 successful runs, then with the same config .. back to failure. On and off.. successfull and unscussessful. ... it wasn't in the manifests ..

So we decided to roll the puppetmaster back to it's previous version, that one was known to be stable, there obviously was something really fishy going, so that was the safest bet.

Wrong, the machine came up, but it took longer than expected, and when trying to connect new clients to it .. nothing worked anymore .. same problem as before .. puppetmaster compiles catalog, clients didn't get anything. we started to suspect faulty hardware .. but how could that bee.. the puppetclient looked liked the only malfunctioning thing around .

Then Dim0 suggested me to look at the that one logfile I hadn't looked , /var/log/puppet/masterhttp.log and then we saw it . it was being flooded with ssl errors, ssl errors from clients that shouldn't even be connecting to the puppetmaster at all.

[2011-03-02 13:32:00] ERROR OpenSSL::SSL::SSLError: tlsv1 alert decrypt error
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `accept'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `listen'
        /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
        /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
        /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
        /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
        /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'
        /usr/lib/ruby/1.8/thread.rb:135:in `synchronize'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:127:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:142:in `start'
        /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'
        /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:114:in `main'
        /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:46:in `run_command'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:287:in `run'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:393:in `exit_on_fail'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:287:in `run'
        /usr/sbin/puppetmasterd:4
[2011-03-02 13:32:00] ERROR OpenSSL::SSL::SSLError: tlsv1 alert decrypt error

What happened was that 'we' decided to bring of the one backup machines back online, afterall once the slow starting server came trough, it would be the passive node in the cluster , no worries there, right ? Wrong,
This physical machine had 6 virtual machines with old ssl certificates that got stuck in an loop which was put there to sure their puppetrun came trough correctly at boot time.

Those 7 rogue clients which generated little to no relevant traffic on the network were saturating the default webrick, killing them solved the problem and we were back to regular deployment in no time.

The sad part is that our upcoming release already has passenger , a fresher version of ruby etc .. and that most of the above mentioned errors won't occur anymore there.
But in short .. don't use the default webrick .. it will kill you :)

And no , not everything is a freaking dns problem, ssl is a big pain in the B too .. :)

Has anyone noticed that pretty much every puppet module one finds on the internet by default enables the service they try to configure in the module

When looking at it from a single machine point of view it makes sense to include the module , have it configure your service and directly enable it by default.

So I started wondering .. isn't there anybody out there who is building clusters ? Where services have to configured on multiple nodes but should NOT be running acitvely on all nodes by default because there is an external tool which manages that for you (Pacemaker framework eg.)

Agreed it's a small patch to get the functionality you want , but it brings an extra overhead when one upgrades the modules etc.

So if it doesn't bother you please split your puppet module in 2 parts.. one you call to configure the service, another which you call to enable the service , if you want to.

thnx !

Apparently I don't understand french :(

Copied from my inbox ...

From: 	Info Falstaff <info@lefalstaff.be>
To: 	'Kris Buytaert' <Kris.Buytaert@youalreadyknowmyaddy.be>
Subject: 	RE: Reservation Next Friday 4/2/2011
Date: 	Fri, 28 Jan 2011 16:55:45 +0100
 
je confirme votre  reservation pour le vendredi 04 02 2011
 
vous trouverez ci-joint nos differentes formules
 
bien à vous
 
guy Lommen chef de cuisine

Lenz gave the good example so I`ll follow :)

Next weekend saturday I`ll be giving a talk about devops at StartUp Weekend Brussels, from what I've read so far it promises to be an audience that needs the talk,

The week after I`ll be speaking at the DrupalDevDays, again about devops , however this time with a touch of Drupal , giving a devops talk at Devoxx last year to a Java audience learned me that the devops evangelist need to go outside of their usual conference audiences and als talk to the people that are usually in the other silos.

Next march I`ll be speaking at the UKUUG spring conference in Leeds this time about my experiences on High Availability with Pacemaker

And who knows I might squeeze in a talk at Load this year also ..

If you are around at one of these confs and you want to talk Devops, Clustering, sipx or just have a beer .. don't hesitate ! There's already plenty of people promising me beers , and some even sushi :)

Just last last year we'll have a Devops meetup just before Fosdem,
I've setup a page for registrations , that way we'll know how many people to make reservations for.

If possible we'll go to the same place as last year .. walking distance from the Fosdem Beer event.

Feel free to spread the news !

That's the question Xavier asks in his blog entry titled
Security: DIY or Plug’n'Play

To me the answer is simple, most of the appliances I ran into so far have no way of configuring them apart from the ugly webgui they ship with their device. That means that I can't integrate them with the configuration management framework I have in place for the rest of the infrastructure. There is no way to automatically modify e.g firewall rules together with the relocation of a service which does happen automatically, and there is always some kind of manual interaction required. Applicances tend to sit on a island, either stay un managed ( be honest when's the last time you upgraded the firmware of that terminal server ? ) , or take a lot of additional efort to manage manually. They require yet another set of tools than the set you are already using to manage your network.
They don't integrate with your backup strategy, and don't tell me they all come with perfect MIB's.

There's other arguments one could bring up against appliances, obviously people can spread fud about some organisation alledgedly paying people to put backdoors in certain operation systems.. so why would they not pay people to put backdoors in appliances , they don't even need to hide them in there .. but my main concern is manageability .. and only a web gui to manage the box to me just means that the vendor hates me and dooesn't want my business

A good Appliance (either security or other type) needs to provide me an API that I can use to configure it, in all other cases I prefer a DIY platform, as I can keep it in line with all my other tools, config mgmtn, deployment, upgrade strategies etc.

Mabye a last question for Xavier to finish my reply ... I`m wondering how Xavier thinks he kan achieve High-availability by using a Virtual environment for Virtual Appliances that are not cluster aware using the virtual environment. A fake comfortable feeling of higher availability , maybe.. but High Availability that I'd like to see.

When I first started out giving talks about devops , I realized that I was preaching to the choir, some Barcamps, the Keynote at Loadays , the Dutch Unix Usergroup etc .. lots of people in the audiences knew about the pains we were trying to solve, lots of them already knew some of the tools we use and lots of them already talk a lot with their developers or are part of the deveoplment teams

With our Devoxx talk, Patrick and I started to talk to a different audience , the Java devs , and it was great, we all learned from it. With that experience in mind I submitted a variation of the talk to an audience that is also very important to me ... the Drupal Community .

Devops is gaining importance , while we been practicing devops methodologies since ever, now even the big analyst companies etc are writing and talking about the movement, the drupal community really should also get involved.

So if you care about devops, about devs and ops working together, about continuous integration, continuous deployment, configuration mangement, automation, monitoring and scale, if you've heard about all of the above but have no clue what Puppet, Hudson or Fabric can do for you , vote here for my proposed talk at Drupalcon Chicago,

Apparently December is the month where everybody starts writing guest posts for other blogs.

Earlier this month I wrote an article with the title of this blog for Sysadvent ,

It's a sysadmin relative of the Perl Advent Calendar: One article for each day of December, ending on the 25th article. With the goals of of sharing, openness, and mentoring, we aim to provide great articles about systems administration topics written by fellow sysadmins

My article is here, but there's plenty more other articles written about a variety of topics, such as chef, tcpdump , how ls works, cucumber and Devops.

On the other side, Matthias over at Agile Web Development and Operations is hosting a series on Devops where lots of Devops Advocates and Evangelists are having their say about Devops ...

My entry about the Challenges the Devops Crowd faces was put online yesterday