Everything is a Freaking DNS problem

When I started writing this I wrote "Last week Opscode came" obviously now that is "A couple of months ago Opscode came with a bunch of announcements ... one of them being that they are also going to support the Open Source Chef .. rather than only their own platform.

I'd love to see more companies formally do this .. Over the past couple of years I've had numerous situations where organizations where happy to pay for support to an commercial backer of Open Source software... but they were not interested in , software updates, fancy dashboards , unneeded features.

Let alone being limited by some of the features of the Enterprise product (what do you mean there's no vlan support in Xen ? We've been using that for ages (anno 2008)

Even right now I`m talking with a customer that is interested in getting commercial support for an open source project but he feels that by choosing the Enterprise version of the
software he will be limiting his options...

We've had this kind of situations with MySQL, Xen, Knowledgetree and others ..

The sad story is that with the growth of Open Source adoption, lots of companies are finding their commercial talents in the pool of people that used to work for the proprietary vendors, the kind of sales people that don't get Open Source (aside from some exceptions) and are still trying to hardsell a product based on specsheets and feature roadmaps, where most quality open source software are build by people to solve problems , hence those new sales people keep doing their old job selling products while not listening to their actual customer needs.

I've seen this escalate up to the point where people that are willing to support the Open Source project by paying a vendor for support don't do so because it's not the right form for them eventually leading to even less revenue for the said vendor.

Yes I know that supporting a multitude of distributions , libary combinations and architectures is a complex thing to do, and a lot of the proprietary vendors ruined the market by inventing something like certified platforms on which they
supported their software.

But if you as an open source software company are really interested in improving your product why wouldn't you take money from a customer that wants to pay for bugs to be fixed or features to be implemented in your product.
You've already realized that the software industry is different from 10 years ago and that Open Source is here to stay .. yet you are still thinking in the sales model with products and specsheets of that era.

People often wonder why DBA's used to hate developers, and with DBA's also the System Engineers,
(note that I just expanded devops by adding dba's to the picture..)

So let me tell you a story ..

A couple of weeks ago one of our customers wanted to start experimenting with a new type of CRM. A gamified CRM.
Zurmo ...

So we set this thing up in a dev environment and started playing with it , while at first it looks nice ..
the application actually felt pretty slow.. however given that is a low resource development environment we looked no further.

Yet the next step is that we run into missing features, such as the fact that every contact you create by default is
set to private .. which really isn't productive for a CRM system where you want to be able to follow up on different
customer and share information.

So we tried figuring out what the database changes to do this in bulk would mean, surely it had to be a flag on the contact record .
Wrong, Zurmo uses an ORM for their database connectivity ...so their data model wasn't really trivial.

So we decided to look at the MySQL log file to figure out what db changes happened when updating the record
Yes there's better approaches but this one learned us a lot ..
The procedure I followed was pointing my browser to the page where I wanted to switch the checkbox,
log on to the mysql box, set global logging on . Clicked the checkbox and stopped global logging.

This gave me a log file with all the database actions required to make that one single change.
I had to cross check a number of times ... the file created by this short and small action was.
about 70K

Puzzled you start looking at the queries ...
The query list was full with "SELECT * FROM " stanza's ..
70K whopping K of queries that make your hair turn grey ...

I figured I'd file a bug .. but I couldn't find no bugtracker for Zurmo, only a forum (and forums are the most broken form of communication imvho) , yet the developers responded on Twitter.

The feedback wasn't really satisfying so we quickly decided that supporting this application was not something we would like to do..
and abandonned it..

The real question is who needs a Gamified CRM anyhow...

PS. So while finishing up this article on a late evening this week I might not have put in clear enough that the generated logfile was 70Kb .. I fear some people misunderstood that it generated 70.000 queries. Obviously a huge difference. But still the log file shouldn't have been bigger han 1Kb There should have been 2-3 queries max (https://github.com/KrisBuytaert/snippets/tree/master/zurmo)

But imvho if the size of the queries you are generating is bigger than the page you are generating you are most often doing it wrong.

The age of #monitoringsucks is over, we're now transitioning into a #monitoringlove period.

That however doesn't mean al the work is done, we still need to do a lot of work and a lot of people are working on a lot of stuff.

Therefore like last year we are opening up our offices again right after Fosdem for a #monitoringlove hackfest

That's right on february 4 and 5 a bunch of people interrested to fix the problem will be meeting , discussing and hacking stuff together in Antwerp. In short a #monitoringlove hackathon

Inuits is opening up their offices for everybody who wants to join the effort Please let us (@KrisBuytaert) know if you want to join us in Antwerp. We'll provide caffeine, wireless, chairs and some snacks.

Please register upfront at : http://monitoringlove2013.eventbrite.com/

Obviously if you can't make it to Antwerp you can join the effort on ##monitoringsucks on Freenode or on Twitter.

The location will be Duboistraat 50 , Antwerp
It is about 10 minutes walk from the Antwerp Central Trainstation
Depending on Traffic Antwerp is about half an hour north of Brussels and there are hotels at walking distance from the venue.

Plenty of parking space is available on the other side of the Park

Read last years report http://www.krisbuytaert.be/blog/we-didnt-fix-it to get an idea of what will happen...

PS. Yes I`m trying to get another event of the ground the days before Fosdem but I`m still awaiting confirmation of the venue ..

With 2 of the bigger Open Source projects I care about talking about certifications programs questions pop up again ...

Should we certify ourselves ?

So let me tell you about my experiences in getting Open Source related Certifications ..

Over a decade ago, (2001) when RedHat was still Redhat and not yet Fedora the company I was working for was about to partner with RedHat and needed to get a number of people certified for that.

So I took the challenge, I bored myselve to death during a 4 day RedHat fast track training and set out to do the exam the next day. Obviosly I scored pretty well given my yearlong experience in the subject. Back then I was told that I scored the one but European Record on the exam which was actually held by another collegue (hey Ico) , our CTO however was not amused when I told that I could have scored better but I didn't bother running a chkconfig smb  on since I didn't see the use in using windows fileshares in a unix environment (Yes I was young , we're all allowd to make stupid mistakes :))

So I was certified, we were expecting the requests to flow in en masse ... nothing happened... not a single customer request... If I recall correctly we got 2 requests for certified engineers over the course of the following years. One was from a customer that wanted to have us do some junior level sysadmin work on their systems which we didn't care about, we proposed a more junior profile, but they insisted on having someone who was certified, The other one was from a Large institution that wanted certified people for their RedHat support, only to quickly learn that the budget they had planned for this project was about half the rate we usually charged ..

When RedHat introduced their certified Architect program my answer was, sure .. if you bring us the customer that will make the investment worthwhile , guess what..

My second experience with Open Source certification came a couple of years later with MySQL, same story partnering etc, . only this time our trainer had put some focus on a couple of slides during the training (Hi Tobias) and during the exam indeed one of those questions popped up, The correct answer to "What are the core values of MySQL AB" was "We reply to email" , I stood up and left the exam ...
I ranted about this to a number of people including Roland Bouman who back then was just starting on the MySQL (NDBD) Cluster certifciation track and I assisted him in making the book to study for that exam better.
Once again .. pretty much no one asked for MySQL certification in Europe back in those days (2007 ?)

I won't go deeper into discussing the Xen certification I got from Citrix, but it involved correcting slides from the presenters at the first European training.

Based on my experience with these certifications in Belgium/Europe you can see that I`m not a big fan of certifications I have not seen a reason for me to certify yet

I actually think that noone within the Open Source community should be looking for certification, we should be looking for people that are active in the community and that are contributing to projects.
Unlike in the proprietary world where you have to cough up tons of money in order to get a license to play with a tool and learn itl In the open source world with projects such as both Drupal and Puppet, there are absolutely no excuses for Junior people not to engage and prove themselves. they have full access to anything they need, the only thing they need to do is want to get involved.

Sadly this world however is still full of incompetent recruiters, middlemarket agencies that will never understand this and will ask for cerftifications of some kind. My fear is indeed that there will be a group of mediocre but certified developers swarming these growing markets at dumping rates and that the people with the real experience that have been involved in the communities for ages already will be the ones pulling the short straw.

Anyhow ... in just a short couple of years everything will be fine again .. as by then my RHCE will be current again and the incompetent recruiters that need people that are RedHat 7 certified will start calling me by the dozen.

I`m parsing the responses of the Deploying Drupal survey I started a couple of months ago (more on that later)

One of the questions in the survey is "What is devops" , apparently when you ask a zillion people (ok ok, just a large bunch of Tweeps..), you get a large amount of different answers ranging from totally wrong to spot on.

So let's go over them and see what we can learn from them ..

The most Wrong definition one can give is probably :

• A buzzword

I think we've long passed the buzzword phase, definitely since it's not new, it's a new term we put to an existing practice. A new term that gives a lot of people that were already doing devops , a common word to dicuss about it. Also lots of people still seem to think that devops is a specific role, a job description , that it points to a specific group of people doing a certain job, it's not . Yes you'll see a lot of organisations looing for devops people, and giving them a devops job title. But it's kinda hard to be the only one doing devops in an organisation.

I described one of my current roles as Devops Kickstarter, it pretty much describes what I`m doing and it does contain devops :)

But devops also isn't

• The connection between operations and development.
• people that keep it running
• crazy little fellows who find beauty in black/white letters( aka code) rather than a view like that of Taj in a full moon light.
• the combination of developer and operations into one overall functionality
• The perfect mixture between a developer and a system engineer. Someone who can optimize and simplify certain flows that are required by developers and system engineers, but sometimes are just outside of the scope for both of them.
• Proxy between developer and management
• The people in charge of the build/release cycle and planning.
• A creature, made from 8-bit cells, with the knowledge of a seasoned developer, the skillset of a trained systems engineer and the perseverence of a true hacker.
• The people filling the gap between the developer world and the sysadmin world. They understand dev. issues and system issues as well. They use tools from both world to solve them.

Or

• Developers looking at the operations of the company and how we can save the company time and money

And it's definitely not

• Someone who mixes both a sysop and dev duties
• developers who know how to deploy and manage sites, including content and configuration.
• I believe there's a thin line line between Ops and Devs where we need to do parts of each others jobs (or at least try) to reach our common goal..
• A developer that creates and maintains environments tools to help other developers be more successful in building and releasing new products
• Developers who also do IT operations, or visa versa.
• Software developers that support development teams and assist with infrastructure systems

So no, developers that take on systems roles next to their own role and want to go for NoOps isn't feasable at all ..you really want collaboration, you want people with different skillsets that (try to) understand eachoter and (try to) work together towards a common goal.

Devops is also not just infrastructure as code

• Writing software to manage operations
• system administrators with a development culture.
• Bring code management to operations, automating system admin tasks.
• The melding of the art of Systems Administration and the skill of development with a focus on automation. A side effect of devops is the tearing down of the virtual wall that has existed between SA's and developers.
• Infrastructure as code.
• Applying some of the development worlds techniques (eg source control, builds, testing etc) to the operations world.
• Code for infrastructure

Sure infastructure as code is a big part of the Automation part listed in CAMS, but just because you are doing puppet/chef doesn't mean you are doing devops.
Devops is also not just continous delivery

• A way to let operations deploy sites in regular intervals to enable developers to interact on the systems earlier and make deployments easier.
• Devops is the process of how you go from development to release.

Obviously lots of people doing devops also often try to achieve Continuous delivery, but just like Infrastructure as Code it devops is not limited to that :)

But I guess the truth is somewhere in the definitions below ...

• That sweet spot between "operating system" or platform stack and the application layer. It is wanting sys admins who are willing to go beyond the normal package installers, and developers who know how to make their platform hum with their application.
• Breaking the wall between dev and ops in the same way agile breaks the wall between business and dev e.g. coming to terms with changing requirements, iterative cycles
• Not being an arsehole!
• Sysadmin best-practise, using configuration as code, and facilitating communication between sysadmins and developers, with each understanding and participating in the activities of the other.
• Devops is both the process of developers and system operators working closer together, as well as people who know (or who have worked in) both development and system operations.
• Culture collaboration, tool-chains
• Removing barriers to communication and efficiency through shared vocabulary, ideals, and business objectives to to deliver value.
• A set of principles and good practices to improve the interactions between Operations and Development.
• Collaboration between developers and sysadmins to work towards more reliable platforms
• Building a bridge between development and operations
• The systematic process of building, deploying, managing, and using an application or group of applications such as a drupal site.
• Devops is collaboration and Integration between Software Development and System Administration.
• Devops is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals.[1] It has developed in response to the emerging understanding of the interdependence and importance of both the development and operations disciplines in meeting an organization's goal of rapidly producing software products and services.
• bringing together technology (development) & content (management) closer together
• Making developers and admins understand each other.
• Communication between developers and systems folk.
• a cultural movement to improve agility between dev and ops
• The cultural extension of agile to bring operations into development teams.
• Tight collaboration of developers, operations team (sys admins) and QA-team.

But I can only conclude that there is a huge amount of evangelisation that still needs to be done, Lots of people still don't understand what devops is , or have a totally different view on it.

A number of technology conferences are and have taken up devops as a part of their conference program, inviting experienced people from outside of their focus field to talk about how they improve the quality of life !

There is still a large number of devops related problems to solve, so that's what I`ll be doing in 2012

Patrick posted his upcoming conference schedule for the next couple of months.
as you can see there are a comple of overlapping conferences :)

Conferences I'm speaking at or likely to attend are:

• The first week of October I`ll be in the Valley , I`ll be late for Jenkinsconf but I hope to pick up some events while I`m there.. suggestions are welcome , I`m also heading back to Europe earlier than planned so I will miss BadCamp :( ...
• Devopsdays Goteborg, Sweden : October 14,15 - The yearly Europe devops event is happening in Goteborg this time. It's going to be really exciting this time , as the theme is inclusive. Eploring the boundaries of devops, I`m once again in the organization of this conference.
• T-Dose 2011, The Technical Dutch Open Source Event, on 5 and 6 november 2011 , I will be talking again about my experiences with complex Puppet setups
• Citconf , London: November 11-12 - All you ever wanted to know about Continuous Integration. Period, registered, haven't booked flights yet.
• Cloudcamp Belgium: November 21 - I'm looking forward to this year's event, as there will likely more practioners and less marketing folks.
• Lisa 2011, Boston, US, I`m giving an Invited talk titled , Devops: The past and futre are here, It's just not evenly distributed (yet), and I`ll be on a panel titled What Will Be HOt Next Year, really looking forward to this one :)
• Fosdem.org will take place on 4 and 5 February 2012 , and as every year since it inception I'll be there
• The UKUUG rebranded to FlossUK , they are hosting their Annual Spring conference from 20th to 22nd March in Edinburgh , given their refound focus it will be even more interresting !
• And as announced earlier this week Loadays.org will take place in Antwerp again this year on 31/3/2012 and 1/4/2012 , as the previous years I`m co organizing this conference

And yes, I do work from time to time. Just that these conferences are a great way to capture and share new ideas. All worth it!

Lenz gave the good example so I`ll follow :)

Next weekend saturday I`ll be giving a talk about devops at StartUp Weekend Brussels, from what I've read so far it promises to be an audience that needs the talk,

The week after I`ll be speaking at the DrupalDevDays, again about devops , however this time with a touch of Drupal , giving a devops talk at Devoxx last year to a Java audience learned me that the devops evangelist need to go outside of their usual conference audiences and als talk to the people that are usually in the other silos.

Next march I`ll be speaking at the UKUUG spring conference in Leeds this time about my experiences on High Availability with Pacemaker

And who knows I might squeeze in a talk at Load this year also ..

If you are around at one of these confs and you want to talk Devops, Clustering, sipx or just have a beer .. don't hesitate ! There's already plenty of people promising me beers , and some even sushi :)

That's the question Xavier asks in his blog entry titled
Security: DIY or Plug’n'Play

To me the answer is simple, most of the appliances I ran into so far have no way of configuring them apart from the ugly webgui they ship with their device. That means that I can't integrate them with the configuration management framework I have in place for the rest of the infrastructure. There is no way to automatically modify e.g firewall rules together with the relocation of a service which does happen automatically, and there is always some kind of manual interaction required. Applicances tend to sit on a island, either stay un managed ( be honest when's the last time you upgraded the firmware of that terminal server ? ) , or take a lot of additional efort to manage manually. They require yet another set of tools than the set you are already using to manage your network.
They don't integrate with your backup strategy, and don't tell me they all come with perfect MIB's.

There's other arguments one could bring up against appliances, obviously people can spread fud about some organisation alledgedly paying people to put backdoors in certain operation systems.. so why would they not pay people to put backdoors in appliances , they don't even need to hide them in there .. but my main concern is manageability .. and only a web gui to manage the box to me just means that the vendor hates me and dooesn't want my business

A good Appliance (either security or other type) needs to provide me an API that I can use to configure it, in all other cases I prefer a DIY platform, as I can keep it in line with all my other tools, config mgmtn, deployment, upgrade strategies etc.

Mabye a last question for Xavier to finish my reply ... I`m wondering how Xavier thinks he kan achieve High-availability by using a Virtual environment for Virtual Appliances that are not cluster aware using the virtual environment. A fake comfortable feeling of higher availability , maybe.. but High Availability that I'd like to see.

A couple of weeks ago I ranted about being able to crash about any available music player on Fedora, and the gazillion bugs I filed for that ..

At last the the problem is solved, as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=657971#c17
this is most probably a bug in ffmpeg which is provided by rpmfusion. and which is solved by installing gstreamer-plugins-ugly

At least it works for me :)

Apparently December is the month where everybody starts writing guest posts for other blogs.

Earlier this month I wrote an article with the title of this blog for Sysadvent ,

It's a sysadmin relative of the Perl Advent Calendar: One article for each day of December, ending on the 25th article. With the goals of of sharing, openness, and mentoring, we aim to provide great articles about systems administration topics written by fellow sysadmins

My article is here, but there's plenty more other articles written about a variety of topics, such as chef, tcpdump , how ls works, cucumber and Devops.

On the other side, Matthias over at Agile Web Development and Operations is hosting a series on Devops where lots of Devops Advocates and Evangelists are having their say about Devops ...

My entry about the Challenges the Devops Crowd faces was put online yesterday