Everything is a Freaking DNS problem

Apparently December is the month where everybody starts writing guest posts for other blogs.

Earlier this month I wrote an article with the title of this blog for Sysadvent ,

It's a sysadmin relative of the Perl Advent Calendar: One article for each day of December, ending on the 25th article. With the goals of of sharing, openness, and mentoring, we aim to provide great articles about systems administration topics written by fellow sysadmins

My article is here, but there's plenty more other articles written about a variety of topics, such as chef, tcpdump , how ls works, cucumber and Devops.

On the other side, Matthias over at Agile Web Development and Operations is hosting a series on Devops where lots of Devops Advocates and Evangelists are having their say about Devops ...

My entry about the Challenges the Devops Crowd faces was put online yesterday

When I read on the internetz that Alex Davies was about the publish a Packt book on MySQL HA I pinged my contacts at Packt and suggested that I'd review the book .

I've ran into Alex at some UKUUG conferences before and he's got a solid background on MySQL Cluster and other HA alternatives so I was looking forward to reading the book.

Alex starts of with a couple of indepth chapters on MySQL Cluster, he does mention that it's not a fit for all problems, but I'd hoped he did it a bit more prominently ... an upfront chapter outlining the different approaches and when which approach is a match could have been better. The avid reader now might be 80 pages into MySQL cluster before he realizes it's not going to be a match for his problem.

I really loved the part where Alex correcly mentions that you should probably be using Puppet or so to manage the config files of your environment, rather than scp them around your different boxes ..

Alex then goes on to describe setting up MySQL replication and Multi Master replication with the different approaches one can take here, he gives some nice tips on using LVM to reduce the downtime of your MySQL when having to transfer the dataset of an already existing MySQL setup, good stuff.

He then goes on to describe MySQL with shared storage ... if you only mount your redundant sandisk once on your MySQL nodes my preference would probably be a Pacemaker stack rather than a RedHat Cluster based setup, but his setup seems to work too. Alex quickly touches on using GFS to have your data disk mounted simultaneously on both nodes (keep in mind with only 1 active MySQLd) and then goes on to describe a full DRBD based MySQL HA setup

The last chapter titled Performance tuning gives some very nice tips on both tuning your regular storage, as your
GFS setup but also the tuning parameters for MySQL Cluster

I was also really happy to see the Appendixes on the basic installation where he advocates the use of Cobbler , Kickstart and LVM ..

One of the better books I read the past couple of years .. certainly the best book from Packt so far , I hope there is more quality stuff coming from that direction !

Actually it didn't , but now I got your attention.
We just adopted the use of adding headers to all of our files that are managed by puppet so people will know not to touch it

  file {
                "/etc/xen/scripts/network-custom-vlan-bridges":
                        owner => "root",
                        group => "root",
                        mode => "0755",
                        content => template(
                                "headers/header-hash.erb',
                                "xen/co-mmx-network-custom-vlan-bridges.erb");
        }

All worked nice however upon bootstrapping our Xen host the bridges stopped working .. running the network-custom-vlan-bridges script manually solved everything and created the appropriate bridges. But at boottime it didn't..

I added some debug info to the script and figured it never got executed at boot time.

Turns out that when I removed the headers Xen actually does configure the bridges at boot time, Xen probably checks for a shebang at the beginning of the file.

Putting the header at the end of the file therefore solved the problem. ,

Yesterday @beaker posted his ideas on the #devops movement ...

Apparently we haven't been stressing enough on the fact that it isn't just about Devs and Ops,
So let me repeat it's not just about Devs and Ops, it's about breaking silo's , about being good at our jobs, about getting conversation started, about talking to different stakeholders in the processes . We are absolutely trying to include all groups, not exclude some.

@beaker also seems to have seen many presentations where developers are shown to have evolved in practice and methodology, but operators (of all kinds) are described as being stuck in the dark ages. , is that a different point of view on another continent \, on this side of the Atlantic, it's mostly the Ops people that are already using agile methods spreading the word and it isn't about Devs talking about Deopvs yet. It's actually mostly the ops spreading the word because they feel most of the pain .

Hoff also wonders about routers switches firewall and all the other boxen where we aren't running puppet or chef on , the boxes that are left out of our fully automated environments .
Indeed, Puppetcamp Europe once again woke up the discussion on how to tackle these boxen, the lack of use of existing standards was covered .. and some mentioned that CIM and family are pretty much death or irrelevant for real life usage , both the Puppet and Chef communities are working on manifest, modules and recipes to solve the issues.

But the good thing is that we now have the security people involved too, maybe they'll figure out how to survive longer than 6 months in a CSO position if they talk to the others and come out of their Ivory towers :)

Last week was pretty heavy on conferences for me. On wednesday I had to give my Building Virtual Appliances talk at the at the Sizing Server event on Advanced Virtualization and Hybrid Cloud Computing , but the most important part of the week was the first edition of Puppetcamp Europe.

When the first ideas about PuppetCamp Europe started I asked Luke when and where it'd be held. He replied that I should know as I was supposed to organise it... I thanked for the honour , he went on to ask Patrick , he accepted ... I hope I helped him out enough :) I even handed out a personal invitation to some of the most famous configuration mgmt people on this planet and Inuits sponsored the event too

Luke started with the opening talk, talking about the future and past of puppet , about version numbers, 2.6 does sound familiar and stable doesn't it, about forge.puppetlabs.com
During @puppetmasterd 's talk @kartar played Bugmaster which was great and almost realtime

The real fun started with the Open Spaces ... after everybody presented themselves, a mix of usual suspects, first timers and oldskoolers from irc #puppet that finally got faces, different sessions were proposed, ranging from Puppet 101, Alternative Puppet Architectures, Puppet HA, MultiMaster Puppet to Dating for PuppetMasters

Over the 2 days spread the open space different ideas came up on e.g how to scale puppet. Different people are letting their puppetclients run from cron in batches, but probably the weirdest idea I heard was to run Puppet in Jruby in order to speed it up.

Lots of talk on certificates and how to solve the pains with them .. e.g like in a HA setup .. you need to create an authority chain .. there was also talk about having a
--trust-my-network feature that would disable certificates, Luke was open to accepting such a patch, or a patch that would make the whole certificate setup more pluggable
That would for sure be a feature a lot of people would want to use ..

The thurday evening conference dinner was "Stoofvlees met Frieten" for most of us .. but for me it was a London Devops Curry in Gent, with @unixdaemon @ripienaar and some others ;)

But with lots of interesting chatter, free beer and free icecream there's for sure going to be another similar event in Europe next year ..

So about 4 monts ago there was the crazy idea to start a new FOSS event in Belgium targeted at sysadmins.

What started out as an event for local people to meet local people with some local speakers actually ended up being a small local event with some top international speakers on onfiguration mananagement and system administration mixed with a bunch of good local ones !

I had the honour to open the conference with an extremely short version of the Devops talk I gave earlier last year.. extremely short as I knew that over the course of the weekend the topic would reoccur a lot.

We had the first european talk on Chef, by Joshua Timberman, and we had Puppet talks amongst by Dan Bode from Puppetlabs and CFengine talks , devops was a frequently dropped word,

We had a book raffle where we handed out O'Reilly's .. we had a great free pizza party (got the idea from the saturday pizza event at LCA 2005) , and we had some free beer. Sounds like a good combination for a geeky weekend.

Apart from the regular talks there were plenty of Open Spaces where interesting topics were discussed ... we had spaces on Open Source vs Open Core , strong voices were heard when we discussed what we should do with the Open Core companies that claim to value Open Source , some people think we should actually list the fauxpensource ones somewhere and make sure the world knows about them

We had an awesome configuration management discussion session discussing Chef vs Puppet vs CFengine . And much much more ...

Some people owe me plenty of Sushi as I had to do my MySQL HA talk before their Managing MySQL talk , but other than that .. things just went fine..

Last week I was in Manchester for the 2010 UKUUG Spring Conference, right .. make that 2 weeks ago , :)

The UKUUG usually hosts the more interesting conferences around ... , it's not just the schedule that attrackts me , yes there's the strong focus towards Larger Scale Unix (and mostly Linux) deployments and how to manage them, but there's also the opportunity to chat in real life with the Devops from across the chunnel.

Spending time with R.I.Pienaar, Julian Simpson, Simon Wilkinson , Alex Davies , Simon Riggs , Josette, and many others is always fun .

As I was in town early I went to the preconference beer meetup and met with a lot of people and chatted about config management, virtualization and lots of other stuff ... after the pub the plan was to go for curries nearby .. and while walking to the , ahem Bus stop, I managed to recognise Ben Martin from meeting him back ages ago in Hamburg for LinuxKongress , always fun ..

Apart from having to jump on a bus and our group being split at the curry place , rather than being able to tell the latecomers where to walk to and being seeted upstairs with the whole group , the curries were interesting and fun.

As I had been pushing Simon Wardley on Twitter to submit a talk for the conference it was really great to finally see him present .. His talk was the perfect soft introduction to the conference ...

Simon's talk was followed by a talk on Security for the virtual datacenters, after I questionned the speaker if anyone actualy uses TPM outside an academic lab the talk suddenly changed into a commercial presentation for a Quack, nuff said.

The Ever energetic Matt S Trout talked about 21st century perl before Simon "Life is to short for SELinux" Wilkinson talked about his experiences in getting the openAFS crowd on Git.

Bummer Thierry Carrez didn't show us the real juice of UEC and just the installations of a Cloud Controller and a Node Controller , but he managed to do so in approx 30 minutes as promised .

A talk titled Coherent and Integrated Configuration of Virtual Infrastructures always cathces my eye.. however when that talk turns out to be a Coherent and Integrated configuration only within the Univerity of Edinborough (aka lcfg2) talk I`m dissapointed, specially since it pretty much didn't introduce any new concepts from the ones I introduced back in my Durham UKUUG presentation

Luckily Andrew Stribblehill gave a very interesting talk on MySQL scalability, in which I promised him some answers to his questions for the next day :)

The Conference dinner was without a doubt the best UKUUG dinner so far , no typical english "food", no weird location (Old Trafford, an abandoned warship) , but just a big chinese place and plenty of food !

I started thurday morning in the wrong track, I assumed to be in the Virtualization track, but I ended up in the Sun thinclient and Abusing Linux to serve weird desktops under the Green computing umbrella track, not my favourites ..

When Patrick and Julian started their Hudson hit my Puppet with a Cucumber talk (which featured some aweseom #devops content) I was a afraid that we'd had to look for a replacment PostgreSQL talk as Simon hadn't arrived yet .. Luckily he arrived in time for his presentation and he explained us about the new replication features that are slowly making it into PostgreSQL, one way ... log shipping ... not really up to par with other alternatives yet :(

So with no further ado .. here's the presentation I gave

PS. If at a Ukuug event and not sure about a person's name ... try Simon.. pretty good chance you're correct :)

That's right .. only 11 more ...
The schedule looks promising, there will be some devops juice, some open spaces, some tutorials, som regular talks .. it really looks promising ... the schedule is packed ,

Apart from the talks, tutorials and open spaces there's also the
Pizza party and the Beer event on saturday ...

No need to register .. just show up ..

Devops, Devops, Devops, everybody talks about it but we're still defining it ...

There's so many different interpretations possible for the term Devops , It's automated infrastructure, it's agile infrastucture, it's getting devs and ops closer to eachother, it's briding the gap between devs and ops , it's agile system administration, it's the movement , it's the mindset , it's the spirit.

Lots of people, lots of opinions .. Indeed some people have been doing this kind of work for ages, some claim the cloud is what makes devops become visible (but we've been doing cloud since before the cloud marketeers called it cloud)

Some define the devop as a European based , open source backgrounded , thirtysomething senior sysadmin , or should I say infrastructure architect , originated concept . Others claim it's developers gone sysadmin gone partly developer again ..

But it seems like lots of people claim that Devops is more about the team, not about the unique individual doing a job.

You'll have to agree however that our jobs are significantly different from the system adminstration type jobs you'll find at the average IT department. With that in mind: How shall we call this breed of people cooking up chef stuff, playing the puppeteer or cranking up the CFEngines ?

And no I don't like Devministrator :)

So John wrote down his experiences on deploying Drupal sites with Puppet .

It's not a secret that I've been thinking about similar stuff and how I could get to the best possible setup.

John starts of with using Puppet to download Drush... while I want to use rpm for that ...

I want my core infrastructure to be fully packaged... not downloaded and untarred. I want to be able to reproduce my platform in a couple of months , with the exact same versions I`m using now .. not with the version that happens to be on ftp.drupal.org at that point in time, or with ftp.drupal.org being down.

Now the next question off course is what's the core infrastructure.
Where does the infrastructure end and does the application start. There's little discussion about having a puppet created vhost , an apache conf.d file, a matching .htaccess file if wanted , and the appropriate settings.php for a multisite drupal config.

There's also little doubt to me on using drush to run the updates, manage the drupal site etc . Reading John's article made me think some further about what and when I want things packaged.

John's post lead to a discussion on #infra-talk on getting all drupal modules packaged for Centos with Karan and some others

In a development environment I probably want to have periodic drush updates getting the latest modules from the interwebs and potentially breaking my devs code. But making sure that when you put a site in production it will be on a fairly up to date platform, and not on the platform you started developing on 24 months ago.

In a production environment however you only want tested updates of your modules as indeed they will break code.

It's probably going to be a mix and match setup having a local rpm/deb repo with packaged modules that have been tested and validated in your setup and using drush to enable or configure them for that production setup.

But also having a CI environment wher Drush will get the new modules from the interwebs when needed. and package them for you.

To me that sounds beter than getting all the available Drupal modules and packaging them, even automated, and preparing a repository of those modules of which only a small percentage will actually be used by people.

But I need to think about it some more :)