Sep 13 2012

Gentwerpen Devops Meetup & Conference Season Update

A couple of us have been taking about it a lot already .. we wanted to host a one day #devops event in .be already last year.. then talks about starting a meetup group started again with @wonko_be but it was @fs111 pushing the final button and calling the rest of the .be community to order, we've set a date
and the first session will take place (agenda still needs to be detirmined)

So all you Belgian devops enthousiasts, maark October 11th in your calendar and go register here

We already have 2 other venues (Gent, Boom) lined up .. but let's get this first one started :)

Next to that here's an update for the rest of my upcoming Conference Season :

  • Later this month I`ll be heading to San Francisco for a talk at PuppetConf 2012. I'll probably be around in the valley a bit earlier so if you anyone wants to meet up I`m open for suggestions.
    (Yes I asked Nick Stielau of Pantheon to host a #monitoringsucks / #devops meetup about Sensu but I should have predicted it was about to clash with the PuppetConf speakers dinner :((
  • I was thinking to swing by the MySQL- Connect conference but given the pricetag I don't think I`ll bother ... I am however thinking about crashing the hallway track , or tricking the Foreman Meetup to be colocated with a MySQL event again just like at Fosdem earlier this year
  • I will be attending the Jenkins User conference in San Francisco however before flying back to Europe
  • If you haven't noticed yet , Devopsdays is going to be in Rome this year on october 5 and 6. Registration
    is still open !
  • During the last weekend of october it's time for t-dose.org again. No news on the program yet.
  • And one week later I`ll be heading to Barcelona to speak at LinuxCon Europe I`m really looking forward to that last one again as it looks like the good old LinuxKongresses in Germany .. deep technical topics !
Aug 25 2012

Vagrant and Drupal, a winning team

While heading back home from DrupalCon Munich after 4 days of good interaction with lots of Drupal folks.
I realized to my big suprise that there are a lot of people using Vagrant to make sure that developers are not working on platforms they invented their own. Lots of people have realized that "It works on my computer" is not something they want to hear from a developer and are reaching out to give them viable solutions to work on shared and reproducible solutions.

There were 2 talks proposing solutions to the problem,

the first one was ..Fearless development with Drush, Vagrant and Aegir by Christopher Gervais He talked about Drush VAgrant Integraion and how extentions to Drush allow for easy vagrant integration , bridging this gap allows rupal developers to use a tool they are already familiar with

The second one was Jochen Lillich who explained how he us using Vagrant an Chef for this purpose his talk titled Use datacenter tools to make your dev life easier has been posted already.

During the Vagrant BOF , I briefly ran over @patrickdebois old slides on Vagrant after which people started discussing their use cases.. 2 other projects came up

First is Project Oscar which aims at providing developers with a default Drupal development environment in a Jiffy. they do this by providing a bunch of puppetmanifests that sets up a working environment.

And the second one is Ariadne which is a standardized virtual machine development evironment for easily developing Drupal sites in a local sandbox that is essentially identical to a fully-configured hosted solution. It attempts to emulate a dedicated Acquia/Pantheon server as closely as possible, with added development tools. Project Ariadne is just like the examples from Jochen Lillich based on Chef

With all of these tools and examples around , there should be no excuses anymore for Drupal developers to hack on their own machine and tell the systems people "It works on my machine" (let alone to hack in production).

Aug 25 2012

Open Source Certification , Friend or Foe

With 2 of the bigger Open Source projects I care about talking about certifications programs questions pop up again ...

Should we certify ourselves ?

So let me tell you about my experiences in getting Open Source related Certifications ..

Over a decade ago, (2001) when RedHat was still Redhat and not yet Fedora the company I was working for was about to partner with RedHat and needed to get a number of people certified for that.

So I took the challenge, I bored myselve to death during a 4 day RedHat fast track training and set out to do the exam the next day. Obviosly I scored pretty well given my yearlong experience in the subject. Back then I was told that I scored the one but European Record on the exam which was actually held by another collegue (hey Ico) , our CTO however was not amused when I told that I could have scored better but I didn't bother running a chkconfig smb  on since I didn't see the use in using windows fileshares in a unix environment (Yes I was young , we're all allowd to make stupid mistakes :))

So I was certified, we were expecting the requests to flow in en masse ... nothing happened... not a single customer request... If I recall correctly we got 2 requests for certified engineers over the course of the following years. One was from a customer that wanted to have us do some junior level sysadmin work on their systems which we didn't care about, we proposed a more junior profile, but they insisted on having someone who was certified, The other one was from a Large institution that wanted certified people for their RedHat support, only to quickly learn that the budget they had planned for this project was about half the rate we usually charged ..

When RedHat introduced their certified Architect program my answer was, sure .. if you bring us the customer that will make the investment worthwhile , guess what..

My second experience with Open Source certification came a couple of years later with MySQL, same story partnering etc, . only this time our trainer had put some focus on a couple of slides during the training (Hi Tobias) and during the exam indeed one of those questions popped up, The correct answer to "What are the core values of MySQL AB" was "We reply to email" , I stood up and left the exam ...
I ranted about this to a number of people including Roland Bouman who back then was just starting on the MySQL (NDBD) Cluster certifciation track and I assisted him in making the book to study for that exam better.
Once again .. pretty much no one asked for MySQL certification in Europe back in those days (2007 ?)

I won't go deeper into discussing the Xen certification I got from Citrix, but it involved correcting slides from the presenters at the first European training.

Based on my experience with these certifications in Belgium/Europe you can see that I`m not a big fan of certifications I have not seen a reason for me to certify yet

I actually think that noone within the Open Source community should be looking for certification, we should be looking for people that are active in the community and that are contributing to projects.
Unlike in the proprietary world where you have to cough up tons of money in order to get a license to play with a tool and learn itl In the open source world with projects such as both Drupal and Puppet, there are absolutely no excuses for Junior people not to engage and prove themselves. they have full access to anything they need, the only thing they need to do is want to get involved.

Sadly this world however is still full of incompetent recruiters, middlemarket agencies that will never understand this and will ask for cerftifications of some kind. My fear is indeed that there will be a group of mediocre but certified developers swarming these growing markets at dumping rates and that the people with the real experience that have been involved in the communities for ages already will be the ones pulling the short straw.

Anyhow ... in just a short couple of years everything will be fine again .. as by then my RHCE will be current again and the incompetent recruiters that need people that are RedHat 7 certified will start calling me by the dozen.

Aug 11 2012

Our #monitoringsucks rpm is repository available

Not only our Rubygems Builds have changed, but also my internal #monitoringsucks repository.

You might have noticed a variety of vagrant- projects on my github acount

Being the #monitoringsucks part of them. All of those Vagrant projects are basically my test setups to play with those new tools.

They contain a bunch of puppet modules that install and configure these tools. (Note that they mostly consist of
of git submodules to other puppet module repositories.

Given the fact that I also like to have my software cleanly installed from a package, that means that some of these tools had to be packaged, or I had to create a personal / internal repository which had packages from upstream that were hiding on the internet available.

I've forked of this repository off the internal Inuits epository so you all can also benefit from these efforts.
(You gotta love pulp :))

That means you can now install all of the above mentionned #monitoringsucks tool from our public repo on

  1. yumrepo { 'monitoringsucks':
  2. baseurl => 'http://pulp.inuits.eu/pulp/repos/monitoring',
  3. descr => 'MonitoringSuck at Inuits',
  4. gpgcheck => '0',
  5. }

Patches to both the Vagrant projects and the puppet modules are welcome ...

Aug 06 2012

Breaking the Silence.

3+ months is probably the biggest timeout I've taken from blogging in a while..
Not that I didn't have anything to write ..but more that I was prioritizing writing different content over
over writing blogposts.

Blogging tech snippets and contributing documentation used to be one now all of that has evolved.
Anyhow ..

So to get things going here's my preliminary Conference schedule for the next couple of months.

  • First up, in about one week (august 20-24) I`ll be chairing the #devops track at DrupalCon Munich
    Next to talking there myselve explaining the Drupal Crowd what devops is
    Plenty of interresting content there ranging from culture over to tooling and back. I`m really looking forward to this one.

    There's also talk of the local Devops meetup group hosting an additional meetup !

  • About a month later I`ll be heading to San Francisco for a talk at PuppetConf 2012. I'll probably be around in the valley a bit earlier
    so if you anyone wants to meet up I`m open for suggestions.
  • I've taken over a bit of Patricks workload this year , thus I`m trying to guide the local crew in Rome into organising yet another awesome Devopsdays Europe,
    If you haven't noticed yet , Devopsdays is going to be in Rome this year on october 5 and 6. Both registration and the call for participation are still open !

Next up .. content ... on how monitoring tools still suck .. and I`m still not sure wether a certification program is relevant for open source consultants ..

May 01 2012

Devops in Munich

Devopsdays Mountainview sold out in a short 3 hours .. but there's other events that will breath devops this summer.
DrupalCon in Munich will be one of them ..

Some of you might have noticed that I`m cochairing the devops track for DrupalCon Munich,
The CFP is open till the 11th of this month and we are still actively looking for speakers.

We're trying to bridge the gap between drupal developers and the people that put their code to production, at scale.
But also enhancing the knowledge of infrastructure components Drupal developers depend on.

We're looking for talks both on culture (both success stories and failure) , automation,
specifically looking for people talking about drupal deployments , eg using tools like Capistrano, Chef, Puppet,
We want to hear where Continuous Integration fits in your deployment , do you do Continuous Delivery of a drupal environment.
And how do you test ... yes we like to hear a lot about testing , performance tests, security tests, application tests and so on.
... Or have you solved the content vs code vs config deployment problem yet ?

How are you measuring and monitoring these deployments and adding metrics to them so you can get good visibility on both
system and user actions of your platform. Have you build fancy dashboards showing your whole organisation the current state of your deployment ?

We're also looking for people talking about introducing different data backends, nosql, scaling different search backends , building your own cdn using smart filesystem setups.
Or making smart use of existing backends, such as tuning and scaling MySQL, memcached and others.

So lets make it clear to the community that drupal people do care about their code after they committed it in source control !

Please submit your talks here

Mar 23 2012

FlossUK and Puppetcamp Edinburgh

I've just finished presenting my talk on how I currently work on Puppet modules at Puppetcamp here in Edinburgh where I've been for the week talking on both FlossUK 2012 and Puppetcamp.

Earlier this week I opened FlossUK 2012 with my talk on 7 tools for your devops stack

Sep 24 2011

Fall , Winter and Spring Conference Season 2011 - 2012

Patrick posted his upcoming conference schedule for the next couple of months.
as you can see there are a comple of overlapping conferences :)

Conferences I'm speaking at or likely to attend are:

  • The first week of October I`ll be in the Valley , I`ll be late for Jenkinsconf but I hope to pick up some events while I`m there.. suggestions are welcome , I`m also heading back to Europe earlier than planned so I will miss BadCamp :( ...
  • Devopsdays Goteborg, Sweden : October 14,15 - The yearly Europe devops event is happening in Goteborg this time. It's going to be really exciting this time , as the theme is inclusive. Eploring the boundaries of devops, I`m once again in the organization of this conference.
  • T-Dose 2011, The Technical Dutch Open Source Event, on 5 and 6 november 2011 , I will be talking again about my experiences with complex Puppet setups
  • Citconf , London: November 11-12 - All you ever wanted to know about Continuous Integration. Period, registered, haven't booked flights yet.
  • Cloudcamp Belgium: November 21 - I'm looking forward to this year's event, as there will likely more practioners and less marketing folks.
  • Lisa 2011, Boston, US, I`m giving an Invited talk titled , Devops: The past and futre are here, It's just not evenly distributed (yet), and I`ll be on a panel titled What Will Be HOt Next Year, really looking forward to this one :)
  • Fosdem.org will take place on 4 and 5 February 2012 , and as every year since it inception I'll be there
  • The UKUUG rebranded to FlossUK , they are hosting their Annual Spring conference from 20th to 22nd March in Edinburgh , given their refound focus it will be even more interresting !
  • And as announced earlier this week Loadays.org will take place in Antwerp again this year on 31/3/2012 and 1/4/2012 , as the previous years I`m co organizing this conference

And yes, I do work from time to time. Just that these conferences are a great way to capture and share new ideas. All worth it!

Aug 24 2011

Using Veewee

With @dancarley and @patrickebois just discussing the origin of the name of Veewee I figured I still had that piece of documentation I wrote up for myselve flying around ...

So with no other reason than having my docs mirrored on the internet .

  1. gem install veewee

  1. veewee templates

shows you what templates we have around ..

  1. $veewee init natty ubuntu-11.04-server-amd64
  2. Init a new box natty, starting from template ubuntu-11.04-server-amd64
  3. The basebox 'natty' has been successfully created from the template ''ubuntu-11.04-server-amd64'
  4. You can now edit the definition files stored in definitions/natty
  5. or build the box with:
  6. vagrant basebox build 'natty'

As noted this will generate the definition for your natty box,
It will create a definition.rb file which describes your box.
A preseed (or kickstart or similar file) and a postinstall file

The next step is then to use vagrant to build this basebox

  1. $ vagrant basebox build natty
  3. Verifying the isofile ubuntu-11.04-server-amd64.iso is ok.
  4. Creating vm natty : 384M - 1 CPU - Ubuntu_64
  5. Creating new harddrive of size 10140
  6. VBoxManage createhd --filename '/home/sdog/VirtualBox VMs/natty/natty.vdi' --size '10140' --format vdi > /dev/null
  7. Attaching disk: /home/sdog/VirtualBox VMs/natty/natty.vdi
  8. Mounting cdrom: /home/sdog/iso/ubuntu-11.04-server-amd64.iso
  9. Waiting for the machine to boot
  11. Typing:[1]: <Esc><Esc><Enter>
  12. Typing:[2]: /install/vmlinuz noapic preseed/url=
  13. Typing:[3]: debian-installer=en_US auto locale=en_US kbd-chooser/method=us
  14. Typing:[4]: hostname=natty
  15. Typing:[5]: fb=false debconf/frontend=noninteractive
  16. Typing:[6]: keyboard-configuration/layout=USA keyboard-configuration/variant=USA console-setup/ask_detect=false
  17. Typing:[7]: initrd=/install/initrd.gz -- <Enter>
  18. Done typing.
  20. Starting a webserver on port 7122
  21. Serving file /home/sdog/definitions/natty/preseed.cfg
  23. Waiting for ssh login with user vagrant to sshd on port => 7222 to work
  24. .....................................................................................................................................................Transferring /tmp/vbox.version20110822-6766-1xcca1e-0 to .vbox_version
  25. ..
  28. Step [0] was successfully - saving state
  30. Waiting for ssh login with user vagrant to sshd on port => 7222 to work
  31. .Transferring /home/sdog/definitions/natty/postinstall.sh to postinstall.sh

Plenty more output here !

Be very patient .. you will see VirtualBox launch a VM and start installing it ..

The next steps are clear .. vagrant tells you what you can do next

  1. Now you can:
  2. - verify your box by running : vagrant basebox validate natty
  3. - export your vm to a .box file by running : vagrant basebox export natty

So after validating it , you can now export the basebox and share it with other people.

The next step is to actually use that box in your own Vagrant setup, for that you need to import the box into your box collection

  1. $ vagrant box add 'natty' 'natty.box'
  2. [vagrant] Downloading with Vagrant::Downloaders::File...
  3. [vagrant] Copying box to temporary location...
  4. [vagrant] Extracting box...
  5. [vagrant] Verifying box...
  6. [vagrant] Cleaning up downloaded box...

To verify just run

  1. $ vagrant box list
  2. Centos6
  3. MyCentOS2
  4. debian
  5. natty

your freslhy imported box should be in the list .

You can now use

  1. config.vm.box = "natty"
to refer to the fresly imported box in your Vagrant file, a file that can be created by running vagrant init, or copying around another Vagrant template ..

After that .. regular vagrant fun starts, up, provision, provision, provision, destroy, and so forth ..

Jul 17 2011

Drupal and Configuration Mgmt, we're getting there ...

For those who haven't noticed yet .. I`m into devops .. I`m also a little bit into Drupal, (blame my last name..) , so one of the frustrations I've been having with Drupal (an much other software) is the automation of deployment and upgrades of Drupal sites ...

So for the past couple of days I've been trying to catch up to the ongoing discussion regarding the results of the configuration mgmt sprint , I've been looking at it mainly from a systems point of view , being with the use of Puppet/ Chef or similar tools in mind .. I know I`m late to the discussion but hey , some people take holidays in this season :) So below you can read a bunch of my comments ... and thoughts on the topic ..

First of all , to me JSON looks like a valid option.
Initially there was the plan to wrap the JSON in a PHP header for "security" reasons, but that seems to be gone even while nobody mentioned the problems that would have been caused for external configuration management tools.
When thinking about external tools that should be capable of mangling the file plenty of them support JSON but won't be able to recognize a JSON file with a weird header ( thinking e.g about Augeas (augeas.net) , I`m not talking about IDE's , GUI's etc here, I`m talking about system level tools and libraries that are designed to mangle standard files. For Augeas we could create a separate lens to manage these files , but other tools might have bigger problems with the concept.

As catch suggest a clean .htaccess should be capable of preventing people to access the .json files There's other methods to figure out if files have been tampered with , not sure if this even fits within Drupal (I`m thinking about reusing existing CA setups rather than having yet another security setup to manage) ,

In general to me tools such as puppet should be capable of modifying config files , and then activating that config with no human interaction required , obviously drush is a good candidate here to trigger the system after the config files have been change, but unlike some people think having to browse to a web page to confirm the changes is not an acceptable solution. Just think about having to do this on multiple environments ... manual actions are error prone..

Apart from that I also think the storing of the certificates should not be part of the file. What about a meta file with the appropriate checksums ? (Also if I`m using Puppet or any other tool to manage my config files then the security , preventing to tamper these files, is already covered by the configuration management tools, I do understand that people want to build Drupal in the most secure way possible, but I don't think this belongs in any web application.

When I look at other similar discussions that wanted to provide a similar secure setup they ran into a lot of end user problems with these kind of setups, an alternative approach is to make this configurable and or plugable. The default approach should be to have it enable, but the more experienced users should have the opportunity to disable this, or replace it with another framework. Making it plugable upfront solves a lot of hassle later.

Someone in the discussion noted :
"One simple suggestion for enhancing security might be to make it possible to omit the secret key file and require the user to enter the key into the UI or drush in order to load configuration from disk."

Requiring the user to enter a key in the UI or drush would be counterproductive in the goal one wants to achieve, the last thing you want as a requirement is manual/human interaction when automating setups. therefore a feature like this should never be implemented

Luckily there seems to be new idea around that doesn't plan on using a raped json file
instead of storing the config files in a standard place, we store them in a directory that is named using a hash of your site's private key, like sites/default/config_723fd490de3fb7203c3a408abee8c0bf3c2d302392. The files in this directory would still be protected via .htaccess/web.config, but if that protection failed then the files would still be essentially impossible to find. This means we could store pure, native .json files everywhere instead, to still bring the benefits of JSON (human editable, syntax checkable, interoperability with external configuration management tools, native + speedy encoding/decoding functions), without the confusing and controversial PHP wrapper.

Figuring out the directory name for the configs from a configuration mgmt tool then could be done by something similar to

  1. cd sites/default/conf/$(ls sites/default/conf|head -1)

In general I think the proposed setup looks acceptable , it definitely goes in the right direction of providing systems people with a way to automate the deployment of Drupal sites and applications at scale.

I`ll be keeping a eye on both the direction they are heading into and the evolution of the code !