Everything is a Freaking DNS problem

Couple of Fridays ago we had one of our @Inuits days again. Rather than having some people give talks and presentations about what they have been doing for the past couple of months this time we set out to research, test, and build stuff.

We split up in 3 different groups, one focusing on CI and testing freshly build stuff with cucumber, a second one setup and tested Galera

We setup a 3 node Galera cluster , not really as smooth as we'd like to ..

Our first bump was that the installation of the package on CentOS is hell, it needs manual interaction such as replacing packages. Deploying this from a repository is probably not going to be a straight forward option.

Galera only takes care of replicating data, just as with MySQL MM replication there still is a need for an external tool to define where to access the database, and implement monitoring in such a way that you are connecting to an up to date database.

Karl started wondering about Galera's locking, turns out the locks aren't cluster wide, locks within the same node work fine.. so if galera is solely used for HA with 1 active node and X failover nodes, it will work (so all transactions happening on 1 node).

We also ran into some issues when trying to start a node which couldn't contact the wsrep_cluster_address point (which is a node it will sync from at startup if specified in the wsrep.cnf file) , it just didn't want to start. This means that when the referenced node (configured in wsrep_cluster_address)is down, you will need to comment it out before you are able to start the mysql server.

The fact that Galera replicates everythying brought us to the discussion if we really wanted that , or if we wanted more finegrained control over which databases or even tables we want to replicate and which ones we didn't want to replicate. A minority of people wanted to replicate everything, the majority of our group wanted finere grained control over what is being replicated to another node.

I`m sure Lefred will shortly be writing about the progress his group made on Banquise

The day ended as it should .. with BBQ and plenty of drinks

This morning my HTC Hero told me it had an upgrade available.
It wasn't really the moment to do the upgrade.

So when tonight I wanted to perform the upgrade I couldn't really find out how to initiate it again .

Apparently the trick is to put the date of your phone one month forward and you get the update request again.

So my phone has been updated ... so let's hope this indeed was the preparation for a real upgrade ..

Oh and don't forget to put it back to the original date :)

It's a typical situation, the developers develop on their own boxen, they only start to integrate their code on on the production platform 3 hours before the deadline. And then the problems start, the typical "But it works on my system" , "its your problem now" is something nobody really likes to hear .

So how do you tackle this problem ? As Christian already mentions Talking is the first step of the solution,

But one of the most satisfying approaches to solve this problem is to provide your development teams with a standard platform that you support, and a platform they can play with , if you can't provide them with a fully defined platform, give them a set of guide lines on what they can expect. Things like library versions, database types , memory availability and storage availability are key components of such guidelines.

My platform of choice for this kind of projects today is to for an Enterprise Level distro, a distro that stays stable for a longer period, not one that is bleeding edge and changes every other week. So a CentOS or a Debian based distro is probably going to be the platform of choice. But a stable standard platform also means that all the latest nice features a developer wants to have from the bleeding edge libraries he is using aren't going to be available .

Sometimes your devs really need those features, sometimes its just a nice to have. On the other hand you as an ops guy don't want to be packaging and configurating every single tool they dream off. As usual in a Devops environment the key can be found in communication ... Talking with the devs will teach you what features they really need and how they might solve things in a different, more standardized way

We've learned that by giving them a default platform and keeping an open conversation helps, some developers take longer to understand the process others jump in right away .. but in the long term you really need to talk to your devs as soon as possible when they think of implementing a new project that has to run on your platorms.

Lets you sleep at night ..

Q: Is Dries your Brother ?
A: No he isn't.

Q: Are you related to Dries ?
A: No we aren't (as far as we know)

Q: Do you know Dries ?
A: Yeah .. I've known him since before he started his work on Drupal, now can you stop asking about Dries and refocus on me ?

Q: Are you related to Thomas Buytaert
A: No.

Q: Why are you sdog on irc ?
A: Back in my DemoScene days my nick was Sleeping Dog , whch was too long for a unix account name , sleeping didn't feel like a good choice and neither did dog, so the abbrev became sdog

Q: Are you still involved in openMosix ?
A: It's dead Jim, openMosix never made it to the 2.6 Kernel series, Moshe announced the EOL in July 2007. I've been told the folks over at LinuxPMI are doing good job in trying to revive the project.

Q: Why is Inuits called Inuits ?
A: Try finding a cool name and the matching available .be domain. That and our marketeers didn't know where the Penguins live.

Q: Why is Everything a Freaking DNS Problem
A: You mean you still haven't figured out yourselve ?

Q: I want to connect ?
A: No prob, here's my policy

Q: Do you want to give a talk about Open Source, MySQL HA, Devops, Virtualization, CloudSec, Open Source ....
A: Tell me more about your event and I`ll consider it.

Q: I've got another Question
A: I`m pretty sure you can find my email address and mail me :)

Yesterday @beaker posted his ideas on the #devops movement ...

Apparently we haven't been stressing enough on the fact that it isn't just about Devs and Ops,
So let me repeat it's not just about Devs and Ops, it's about breaking silo's , about being good at our jobs, about getting conversation started, about talking to different stakeholders in the processes . We are absolutely trying to include all groups, not exclude some.

@beaker also seems to have seen many presentations where developers are shown to have evolved in practice and methodology, but operators (of all kinds) are described as being stuck in the dark ages. , is that a different point of view on another continent \, on this side of the Atlantic, it's mostly the Ops people that are already using agile methods spreading the word and it isn't about Devs talking about Deopvs yet. It's actually mostly the ops spreading the word because they feel most of the pain .

Hoff also wonders about routers switches firewall and all the other boxen where we aren't running puppet or chef on , the boxes that are left out of our fully automated environments .
Indeed, Puppetcamp Europe once again woke up the discussion on how to tackle these boxen, the lack of use of existing standards was covered .. and some mentioned that CIM and family are pretty much death or irrelevant for real life usage , both the Puppet and Chef communities are working on manifest, modules and recipes to solve the issues.

But the good thing is that we now have the security people involved too, maybe they'll figure out how to survive longer than 6 months in a CSO position if they talk to the others and come out of their Ivory towers :)

Last week was pretty heavy on conferences for me. On wednesday I had to give my Building Virtual Appliances talk at the at the Sizing Server event on Advanced Virtualization and Hybrid Cloud Computing , but the most important part of the week was the first edition of Puppetcamp Europe.

When the first ideas about PuppetCamp Europe started I asked Luke when and where it'd be held. He replied that I should know as I was supposed to organise it... I thanked for the honour , he went on to ask Patrick , he accepted ... I hope I helped him out enough :) I even handed out a personal invitation to some of the most famous configuration mgmt people on this planet and Inuits sponsored the event too

Luke started with the opening talk, talking about the future and past of puppet , about version numbers, 2.6 does sound familiar and stable doesn't it, about forge.puppetlabs.com
During @puppetmasterd 's talk @kartar played Bugmaster which was great and almost realtime

The real fun started with the Open Spaces ... after everybody presented themselves, a mix of usual suspects, first timers and oldskoolers from irc #puppet that finally got faces, different sessions were proposed, ranging from Puppet 101, Alternative Puppet Architectures, Puppet HA, MultiMaster Puppet to Dating for PuppetMasters

Over the 2 days spread the open space different ideas came up on e.g how to scale puppet. Different people are letting their puppetclients run from cron in batches, but probably the weirdest idea I heard was to run Puppet in Jruby in order to speed it up.

Lots of talk on certificates and how to solve the pains with them .. e.g like in a HA setup .. you need to create an authority chain .. there was also talk about having a
--trust-my-network feature that would disable certificates, Luke was open to accepting such a patch, or a patch that would make the whole certificate setup more pluggable
That would for sure be a feature a lot of people would want to use ..

The thurday evening conference dinner was "Stoofvlees met Frieten" for most of us .. but for me it was a London Devops Curry in Gent, with @unixdaemon @ripienaar and some others ;)

But with lots of interesting chatter, free beer and free icecream there's for sure going to be another similar event in Europe next year ..

For all the security experts : the NLUUG has published it's Call For Abstracts for it's Fall conference.. as you might have guessed the topic is Security, we welcome all abstracts tackling security in a broad sense.

Possible topics include:

* cloud security
* online privacy
* rfid hacking
* secure programming
* programma-analysis-tools
* web services security
* web browser security
* embedded hardware hacking
* incident response and forensics
* malware and rootkits
* responsible disclosure
* legal response
* fighting spam
* patch policies
* identity management
* central point of administration
* DNSsec
* VPN based WANs
* etc.

The NLUUG fall conference is scheduled on 11 November 2010 in De Reehorst in Ede, the Netherlands.

Hint.. maybe a talk on secdevops would be welcomed too :)

Disclaimer : I`m on the program committee

Johan from Sizing Servers asked me if I could talk about my experiences on building (virtual) appliances at their Advanced Virtualization and Hybrid Cloud seminar . Off course I said yes ..

Slides are below ... Enjoy ..

Following up on Wim's example

Last week the NLUUG Spring Conference was held in Ede, this years topic was System Administration in general.. which means there was a pretty wide range of talks ... some of the talks were extremely interesting and gathered a lot of people , others really shouldn't have been put in the main room.. Frequenly organising them myselve it's always a difficult choice for a conference organiser.

Must say this was one of my better talks .. it all went smooth and nicely fit within time. Probably the promise of drinks and food after my talk helped some.

I started it of by showing the audience Patrick's opening Devopsdays'09 Video ...

I slightly modified the the slides for my NLUUG presentation, but they are based on the talks I gave on the same topic before

Please note that the Devops definition I give early in the
slides is there to misguide the audience ... :) Everybody knows drinking beer and eating sushi is just a start in the journey when you want to become a #Devops :)

The fun part about conferences often are the speakers dinners, you get to sit down with interesting people and talk about a variety of topics such as panacotta recipes and configuration management ..

It was fun ... too bad it took so long to drive there.. good thing we got back pretty quick..