Everything is a Freaking DNS problem

One of the open sessions last week (corr: last month) at Devopsdays 2010 Hamburg was the one on packaging software. It's always a big question on wether you package the software that runs in your infrastructure or not. And if you package it .. what do you package ..

The general consensus of the open space was pretty much that you always package the software you deploy, unless you have some very good reasons not to. Pretty much the way I've been doing for ages ..

Good reasons that were mentionned were the use of scripting languages that update extremely frequently, but certainly not for compiled code, compiling code on a production machine also is a big nono.

There also was a consensus that you DO NOT PUT CONFIGURATION inside a package. You can put in default templates, but you don't put in config files that should change frequently .. There's plenty of configuration mgmt systems out there do that kind of stuff for you.

The naysayers claimed that packaging brings way to much overhead ... and others claim it takes to much time... however I feel it
should just be a 1 time effort that brings devs and ops closer to eachother and from there on it should automated
New versions of software don't mean that the packaging effort needs to be done again..

Another topic that gathered lots of questions was if you should be capable of installing multiple versions of the same package , lots of people mentionned they didn't like fiddling with symlinks however the best comment in that discussion was that there is already a system out there , the alternatives setup .. provide by most operating systems that allow you to do so in a pretty clean way. I must admit I should look into alternatives more in depth too ..

The ever recurring question is wether one should package war files ? Sure as you then can also use the dependency models a package mgmt system brings to deploy the dependent libraries.

However when people ship products, rather than a live service they seem to package everything , mainly because the code in the product isn't changing as quickly as a live website, or internally used application.

The biggest problem however is the frustration people have with GEM or CPAN packages .. they add yet another layer of management to a system, most lots of CPAN packages are already packaged.. but when it comes to GEM's disaster strikes. There's a lot of work left for distributions to integrate GEM and CPAN style packages.

Actually it didn't , but now I got your attention.
We just adopted the use of adding headers to all of our files that are managed by puppet so people will know not to touch it

  file {
                "/etc/xen/scripts/network-custom-vlan-bridges":
                        owner => "root",
                        group => "root",
                        mode => "0755",
                        content => template(
                                "headers/header-hash.erb',
                                "xen/co-mmx-network-custom-vlan-bridges.erb");
        }

All worked nice however upon bootstrapping our Xen host the bridges stopped working .. running the network-custom-vlan-bridges script manually solved everything and created the appropriate bridges. But at boottime it didn't..

I added some debug info to the script and figured it never got executed at boot time.

Turns out that when I removed the headers Xen actually does configure the bridges at boot time, Xen probably checks for a shebang at the beginning of the file.

Putting the header at the end of the file therefore solved the problem. ,

The next 2 months promise to be the more busier of the year for me .

On october 11th I`ll be giving a presentation ab out Linux and High Availability at the KAHO Sint Lieven in Gent

Just 2 days later I`ll be leaving to Hamburg for the next European version of DevopsDays 2010.

Between Devopsdays Hamburg and Leaving for Malaga I`ll visit CloudCamp Ghent om October 20th.

I'll be in Malaga for the Software Libre Open Source World Conference which will take place on 27 and 28 october

The first weekend of November I`ll be at T-Dose , where I`ll give 2 presentations on Devops.

As I`m part of the program comittee of this years NLUUG Najaars conference on Security , I`ll be spending my official Belgian holliday "working" at a conference in the Netherlands.

And to finish off I`ll be talking Devops again at Devoxx 2010 together with Patrick Debois

Don't hesitate to say hi when you bump in to me :)

It's a typical situation, the developers develop on their own boxen, they only start to integrate their code on on the production platform 3 hours before the deadline. And then the problems start, the typical "But it works on my system" , "its your problem now" is something nobody really likes to hear .

So how do you tackle this problem ? As Christian already mentions Talking is the first step of the solution,

But one of the most satisfying approaches to solve this problem is to provide your development teams with a standard platform that you support, and a platform they can play with , if you can't provide them with a fully defined platform, give them a set of guide lines on what they can expect. Things like library versions, database types , memory availability and storage availability are key components of such guidelines.

My platform of choice for this kind of projects today is to for an Enterprise Level distro, a distro that stays stable for a longer period, not one that is bleeding edge and changes every other week. So a CentOS or a Debian based distro is probably going to be the platform of choice. But a stable standard platform also means that all the latest nice features a developer wants to have from the bleeding edge libraries he is using aren't going to be available .

Sometimes your devs really need those features, sometimes its just a nice to have. On the other hand you as an ops guy don't want to be packaging and configurating every single tool they dream off. As usual in a Devops environment the key can be found in communication ... Talking with the devs will teach you what features they really need and how they might solve things in a different, more standardized way

We've learned that by giving them a default platform and keeping an open conversation helps, some developers take longer to understand the process others jump in right away .. but in the long term you really need to talk to your devs as soon as possible when they think of implementing a new project that has to run on your platorms.

Lets you sleep at night ..

Yesterday @beaker posted his ideas on the #devops movement ...

Apparently we haven't been stressing enough on the fact that it isn't just about Devs and Ops,
So let me repeat it's not just about Devs and Ops, it's about breaking silo's , about being good at our jobs, about getting conversation started, about talking to different stakeholders in the processes . We are absolutely trying to include all groups, not exclude some.

@beaker also seems to have seen many presentations where developers are shown to have evolved in practice and methodology, but operators (of all kinds) are described as being stuck in the dark ages. , is that a different point of view on another continent \, on this side of the Atlantic, it's mostly the Ops people that are already using agile methods spreading the word and it isn't about Devs talking about Deopvs yet. It's actually mostly the ops spreading the word because they feel most of the pain .

Hoff also wonders about routers switches firewall and all the other boxen where we aren't running puppet or chef on , the boxes that are left out of our fully automated environments .
Indeed, Puppetcamp Europe once again woke up the discussion on how to tackle these boxen, the lack of use of existing standards was covered .. and some mentioned that CIM and family are pretty much death or irrelevant for real life usage , both the Puppet and Chef communities are working on manifest, modules and recipes to solve the issues.

But the good thing is that we now have the security people involved too, maybe they'll figure out how to survive longer than 6 months in a CSO position if they talk to the others and come out of their Ivory towers :)

Last week was pretty heavy on conferences for me. On wednesday I had to give my Building Virtual Appliances talk at the at the Sizing Server event on Advanced Virtualization and Hybrid Cloud Computing , but the most important part of the week was the first edition of Puppetcamp Europe.

When the first ideas about PuppetCamp Europe started I asked Luke when and where it'd be held. He replied that I should know as I was supposed to organise it... I thanked for the honour , he went on to ask Patrick , he accepted ... I hope I helped him out enough :) I even handed out a personal invitation to some of the most famous configuration mgmt people on this planet and Inuits sponsored the event too

Luke started with the opening talk, talking about the future and past of puppet , about version numbers, 2.6 does sound familiar and stable doesn't it, about forge.puppetlabs.com
During @puppetmasterd 's talk @kartar played Bugmaster which was great and almost realtime

The real fun started with the Open Spaces ... after everybody presented themselves, a mix of usual suspects, first timers and oldskoolers from irc #puppet that finally got faces, different sessions were proposed, ranging from Puppet 101, Alternative Puppet Architectures, Puppet HA, MultiMaster Puppet to Dating for PuppetMasters

Over the 2 days spread the open space different ideas came up on e.g how to scale puppet. Different people are letting their puppetclients run from cron in batches, but probably the weirdest idea I heard was to run Puppet in Jruby in order to speed it up.

Lots of talk on certificates and how to solve the pains with them .. e.g like in a HA setup .. you need to create an authority chain .. there was also talk about having a
--trust-my-network feature that would disable certificates, Luke was open to accepting such a patch, or a patch that would make the whole certificate setup more pluggable
That would for sure be a feature a lot of people would want to use ..

The thurday evening conference dinner was "Stoofvlees met Frieten" for most of us .. but for me it was a London Devops Curry in Gent, with @unixdaemon @ripienaar and some others ;)

But with lots of interesting chatter, free beer and free icecream there's for sure going to be another similar event in Europe next year ..

For all the security experts : the NLUUG has published it's Call For Abstracts for it's Fall conference.. as you might have guessed the topic is Security, we welcome all abstracts tackling security in a broad sense.

Possible topics include:

* cloud security
* online privacy
* rfid hacking
* secure programming
* programma-analysis-tools
* web services security
* web browser security
* embedded hardware hacking
* incident response and forensics
* malware and rootkits
* responsible disclosure
* legal response
* fighting spam
* patch policies
* identity management
* central point of administration
* DNSsec
* VPN based WANs
* etc.

The NLUUG fall conference is scheduled on 11 November 2010 in De Reehorst in Ede, the Netherlands.

Hint.. maybe a talk on secdevops would be welcomed too :)

Disclaimer : I`m on the program committee

Johan from Sizing Servers asked me if I could talk about my experiences on building (virtual) appliances at their Advanced Virtualization and Hybrid Cloud seminar . Off course I said yes ..

Slides are below ... Enjoy ..

Last week the NLUUG Spring Conference was held in Ede, this years topic was System Administration in general.. which means there was a pretty wide range of talks ... some of the talks were extremely interesting and gathered a lot of people , others really shouldn't have been put in the main room.. Frequenly organising them myselve it's always a difficult choice for a conference organiser.

Must say this was one of my better talks .. it all went smooth and nicely fit within time. Probably the promise of drinks and food after my talk helped some.

I started it of by showing the audience Patrick's opening Devopsdays'09 Video ...

I slightly modified the the slides for my NLUUG presentation, but they are based on the talks I gave on the same topic before

Please note that the Devops definition I give early in the
slides is there to misguide the audience ... :) Everybody knows drinking beer and eating sushi is just a start in the journey when you want to become a #Devops :)

The fun part about conferences often are the speakers dinners, you get to sit down with interesting people and talk about a variety of topics such as panacotta recipes and configuration management ..

It was fun ... too bad it took so long to drive there.. good thing we got back pretty quick..

Whenever I give my Cloud security talk there's a slide in there talking about the most scary idea about Cloud and Security, the fact that Marketing people will build things on their own while IT, or any other departement isn't involved, and as we all know marketing people have no clue about security, it's not on their mind they won't even think about adding some sort of security to their application.

So IT isn't involved, Development isn't involved , and Operations isn't involved ...

Ages ago.. well.. about a decade I was working in those very marketing departments sitting there, writing code, hired by the marketeers, not by IT , the marketing PM did the talking to IT , we still had to go trough their IT department to get stuff deployed.

The marketing people had to deal with their impossible deadlines, a nationwide tv or radio campaign that was going to be launched , with a supporting website which meant that the website functionality needed to go live just before the first airing of the commercial. Obviously the website was lower priority than finding a famous voice or face to record the commercial with, so it became only late in the planning.. even more obvious was the fact that talking to IT about getting these new features deployed was even later on their planning .

Back then, part of my job was to smooth that process, my role was both creating the technical backend for the sites , putting them in production and doing the daily maintenance afterwards ...

Looking back at those days I realize the pains of both deployment and procurement, getting a new machine racked and then installed up to a bare os installations took up to 6 weeks, in a marketing driven world that meant that I'd often had to bypass the whole procurement process of expensive sunboxen and had to quickly deploy a linux box under my desk that could be used to move to production as plan B , and trust me .. we had to use plan B a lot ..

Letting nontechnical people deploy stuff in the cloud will only widen the gap, but getting involved early enough in the concept fase of a project with a good devops methodology/team in place will give the business people the opportunity to learn that things have changed , it doesn't take 6 weeks anymore to get an expensive Sun box racked and a Solaris instance installed after which a team of engineers needs to install an application server, then a different team needs to install the database etc .. these days it's a virtual machine instantiation and a couple of recipes ,in that way we can get manageable, reproducible and scalable deployments in no time.