Everything is a Freaking DNS problem

You might have noticed that this blog stopped accepting comments about a month ago.. well. stopped accepting is a big word.. I was still accepting comments, only they were never submitted to the database and after entering a comment to my blog people ended up on a white page.

So upon returning from holliday I set out to debug the issue together with one of our Inuits Drupal geeks and quickly ran into the following error.

PHP Fatal error:  Call to a member function has_more_records() on a non-object in /somepath/modules/views/plugins/views_plugin_display.inc on line 1992, referer: http://www.krisbuytaert.be/blog/comment/reply/1014

So apparently my veasion of views 6.x-3.0-alpha3 didn't really like to play with Mollom,
I downgraded views again to 6.x-2.11 and Mollom started showing its Captcha's etc again .

So apart from wondering how I ended up installing that alpha3 version (I`m sure Drush didn't do that), all is back to normal. and you should be able to comment on this blog again

Following up on Wim's example

Whenever I give my Cloud security talk there's a slide in there talking about the most scary idea about Cloud and Security, the fact that Marketing people will build things on their own while IT, or any other departement isn't involved, and as we all know marketing people have no clue about security, it's not on their mind they won't even think about adding some sort of security to their application.

So IT isn't involved, Development isn't involved , and Operations isn't involved ...

Ages ago.. well.. about a decade I was working in those very marketing departments sitting there, writing code, hired by the marketeers, not by IT , the marketing PM did the talking to IT , we still had to go trough their IT department to get stuff deployed.

The marketing people had to deal with their impossible deadlines, a nationwide tv or radio campaign that was going to be launched , with a supporting website which meant that the website functionality needed to go live just before the first airing of the commercial. Obviously the website was lower priority than finding a famous voice or face to record the commercial with, so it became only late in the planning.. even more obvious was the fact that talking to IT about getting these new features deployed was even later on their planning .

Back then, part of my job was to smooth that process, my role was both creating the technical backend for the sites , putting them in production and doing the daily maintenance afterwards ...

Looking back at those days I realize the pains of both deployment and procurement, getting a new machine racked and then installed up to a bare os installations took up to 6 weeks, in a marketing driven world that meant that I'd often had to bypass the whole procurement process of expensive sunboxen and had to quickly deploy a linux box under my desk that could be used to move to production as plan B , and trust me .. we had to use plan B a lot ..

Letting nontechnical people deploy stuff in the cloud will only widen the gap, but getting involved early enough in the concept fase of a project with a good devops methodology/team in place will give the business people the opportunity to learn that things have changed , it doesn't take 6 weeks anymore to get an expensive Sun box racked and a Solaris instance installed after which a team of engineers needs to install an application server, then a different team needs to install the database etc .. these days it's a virtual machine instantiation and a couple of recipes ,in that way we can get manageable, reproducible and scalable deployments in no time.

A couple of weeks a go Brian Profitt pinged me for a chat about Devops , the result of that chat , his article can now be found on the Zenoss blog, it's titled Datacenter Barometer: Better days arrive when dev meets ops

It's a very nice read with some pointers to places regular readers of my blog should already know ;)
So with lots of leading Open Source infrastructure companies on different levels, such as config management (OpsCode and Reductive Labs) , monitoring (Zenoss) , deployment (openQRM, RPath, and obviously Consultancy companies , the upcoming Devops conferences around the planet promise to be a lot of fun ! ;)

Oh, and apparently there is some more on the story on /.

I would like to point the crowd to the Call For Presentaions of Loadays. , the Linux Open Administration Days .

The Linux Open Administration days 2010 will be the first edition of a new conference focusing on Linux and Open Administration, we are trying to fill a gap for System Engineers and Administrators using Open Source technologies"

More details on the Linux Open Administration Days site

I'll probably be there .. given the fact that the event will be 5 minutes from where I live .

I know the biggest part of my fanclub already booked tickets for my upcoming presentations, but the other 2 might want to check their calendars to see if they aren't missing out on the good stuff :)

Next Sunday I`ll giving a shortish overview of MySQL HA alternatives in the MySQL and Friends devroom at Fosdem.

March will bring me to Manchester again for the UKUUG Spring conference where I`ll be giving a longer version of that presentation with a strong focus on integrating with PaceMaker, and automating the whole boostrap procedure of a HA setup.

Early may will bring me to Ede in the Netherlands where I`ll be telling the crowds at the NLUUG spring conference, about their new fancy jobtitles, as all the Systeembeheerders there will have to become Devministrators, or Devops if you prefer ...

Apart from my talks also watch out for LoadAys , PuppetCamp Europe, OpsCamp Europe and maybe a Real CloudCamp in Belgium :)

And I`m not the only Inuit on Tour,

When you pull out a disk from your Raid setup it shows a warning in syslog

Jan 27 10:18:22 EL860 kernel: 3w-9xxx: scsi0: AEN: WARNING (0x04:0x0019): Drive 
removed:port=1.
Jan 27 10:18:22 EL860 kernel: 3w-9xxx: scsi0: AEN: ERROR (0x04:0x0002): Degraded
 unit:unit=0, port=1.

However if no one is looking at syslog that won't really be helpfull.

3Ware provides a tool from their site called tw_cli which can be used to manage
the raid setup from the command line.

[EL860-root@EL860 admin]# tw_cli /c0 show 
 
Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    REBUILDING     41%     -       -       232.82    RiW    ON     
 
VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   232.88 GB SATA  0   -            ST3250310NS         
p1    DEGRADED       u0   232.88 GB SATA  1   -            ST3250310NS         

I'd figure I'd either have to write wrapper script around that or find some other way of integrating it.
Asking the question on ##infra-talk on irc.freenode.net gave me the following link to a check script on github

koollman: sdog: something like http://github.com/stanaka/check_tw should work.

With that in your snmpd.conf you can get the info via snmp

[root snmp]#  snmpwalk  localhost   -v 2c  -c public  .1.3.6.1.4.1.2
021   | grep ext 
UCD-SNMP-MIB::extIndex.1 = INTEGER: 1
UCD-SNMP-MIB::extNames.1 = STRING: TW_RAID
UCD-SNMP-MIB::extCommand.1 = STRING: /usr/local/sbin/check_tw
UCD-SNMP-MIB::extResult.1 = INTEGER: 2
UCD-SNMP-MIB::extOutput.1 = STRING: CRITICAL: Unit: u0, Type: RAID-1, Status: RE
BUILDING
UCD-SNMP-MIB::extErrFix.1 = INTEGER: 0
UCD-SNMP-MIB::extErrFixCmd.1 = STRING: 
UCD-SNMP-MIB::ssSysContext.0 = INTEGER: 2073
UCD-SNMP-MIB::ssRawContexts.0 = Counter32: 11781783
UCD-DLMOD-MIB::dlmodNextIndex.0 = INTEGER: 1

Fosdem is coming up again .. It's going to be the 10th edition already So it's going to be 2 days and nights of fun, tech and geek stuff

Lenz already posted the announcement , but allow me to recapitulate.

The MySQL & Friends meetup is on saturday evening , we'll meet around 1900 in front of the under the big tree in front of the AW building...

As with the Devops Meetup you can once again vote for your preferred food

The crowd voted and Favoured an Italian place , so I've made reservations for a 15+ persons group at Sogno d'Italia which is walking distance from the conference location, so there'wont be any hassle with cars and transport

The schedule for the Devroom is also already available

See u there :)

The Devops crowd does not have it's own devroom at Fosdem but will be meeting on Friday evening for food in a location close to the BeerEvent.

If you want to join us please let us know by voting on your favourite choice of Food. Using this doodle.

There's already about 15 people that are voting for the food selection ....

Plan is to meet outside of the BeerEvent around 2000 and
then move to a preferred food location.

Apparently the crowd wants plain Belgian food, so I've made reservations at Le Falstaff which is walking distance from the BeerEvent.
We should be there around 20:15

Don't hesitate to join us if you like DevOps talk

Here's how it started :

So I used to be a software developer, writing perl for the web, then C, then Java, then PHP, till I realized nobody ever configured my servers correctly and I changed trades becoming a system engineer, while teaching new developers the basics of their trade, whom grew into doing Infrastructure Architecture .. familiar story for much of the crowd at DevopsDays ... a crowd that wants to stopping the war between developers and system engineering , a crowd that wants to automate builds, integrate testing, deploy, deploy on very large scale, deploy in the cloud and much more.

So what do you get when you put together some of the experts on building software, organizing development teams , Agile geeks, Cloud infrastructure projects, and Automating guru's in 1 location for 2 days in Gent ? Exactly .. DevopsDays ..

The format was 2 days .. 3 kickass formal talks in the morning.. Open Space sessions in the afternoon. ... Friday featured talks on Non Functional Requirements, CucumberNagios and Monitoring in the Cloud with FlapJack and Building Agile Infrastructures with Puppet while discussing the James White Manifesto ..

which I had never heard of, but which apparently comes down to this

== Rules ==
 On Infrastructure
 -----------------
 There is one system, not a collection of systems.
 The desired state of the system should be a known quantity.
 The "known quantity" must be machine parseable.
 The actual state of the system must self-correct to the desired state.
 The only authoritative source for the actual state of the system is the system.
 The entire system must be deployable using source media and text files.
 
 On Buying Software
 -------------------
 Keep the components in the infrastructure simple so it will be better understood.
 All products must authenticate and authorize from external, configurable sources.
 Use small tools that interoperate well, not one "do everything poorly" product.
 Do not implement any product that no one in your organization has administered.
 "Administered" does not mean saw it in a rigged demo, online or otherwise.
 If you must deploy the product, hire someone who has implemented it before to do so.
 
 On Automation
 -------------
 Do not author any code you would not buy.
 Do not implement any product that does not provide an API.
 The provided API must have all functionality that the application provides.
 The provided API must be tailored to more than one language and platform.
 Source code counts as an API, and may be restricted to one language or platform.
 The API must include functional examples and not requre someone to be an expert on the product to use.
 Do not use any product with configurations that are not machine parseable and machine writeable.
 All data stored in the product must be machine readable and writeable by applications other than the product itself.
 Writing hacks around the deficiencies in a product should be less work than writing the product's functionality.
 
 In general 
 ----------
 Keep the disparity in your architecture to an absolute minimum.
 Use [http://en.wikipedia.org/wiki/Set_theory Set Theory] to accomplish this.
 Do not improve manual processes if you can automate them instead.
 Do not buy software that requires bare-metal.
 Manual data transfers and datastores maintained manually are to be avoided.

Much unlike the FAIL Manifesto

The openspaces tackled how to migrate from a totally unreproducable environment too a correctly bootstrapped infreaastructure, over the Ubuntu Enterprise Cloud , then dinner and off for beers to the Vooruit . The OpenQRM "crowd" stayed at my place so I didn't stay around too late ..

Saturday morning came early ... sadly I missed the first 10 minutes of a very interresting talk about Kanban in operations ... let's ee if we can convince some more people to try it out ...

The talk on Continuous integration, Build Pipelines and Continuous deployment was also really interresting with lots of stories from the real world.. . after the openqRM talk it was time again for OpenSpaces with e.g discussions on svn vs git and building a feature matrix of Cloud , with @botchagalupe, @openqrm and @maesjoch in the room and @diegomarino online .

Devopsdays ended too soon , with way to much interresting ideas to build on .. Let's hope we can all work them out !