Beyond Configuration Mgmt
(This post has been sitting in the drafts folder for way to long, I decided to push the publish button anyhow .. some people might get ideas from it..)
We've all run in to the problem, you've puppetized, or euh .. cooked , about every part of your infrastructure and then there's this one service which has no config files, a broken api that doesn't allow you to configure antyhing, but a magnificent web gui to configure all aspects of the service. Magnificent for the eye , full of AJAX and other fancy stuff which wget isn't really keen on. Off course before it even starts working you need to set it's password , from that webgui.
Sometimes when you are lucky they store al their config in a database, which you can dump, parse and replace all the host specific parameters for other deployments, but is that an approach you like ? As for each new version you'll need to reanalyze the db layout. But no matter how you look at it ,dumping the DB and restoring it is an ugly hack you don't want.
Other alternatives like sniffing the traffic and replaying the POSTS etc were considered ... but fancy AJAX stuff and SSL make that less trivial than it seems
Wo while discussing with an upstream project they proposed to actually screenscrape their config webgui .
So screenscraping the config gui it is .. but how ... I started looking at tools that are typically used for testing rather than for automation, with the purpose of replaying the scenarios one needs to configure the services.
My first attempt was Selenium, it plugs into a browser , so it's easy to acraully record what it has to do, and it saves it's scenarios in a somewhat readable/ editable format.
Having found the export to perl function it alll looked promising. However the export to perl isn't really an export to perl as I epxected .. I assumed it would just generate the perl code to run the same scneario which would be awesome .. it however generates a perl script that instructs a selenium server to run the script.
One of the annoyancies I ran into with Selenium is that a browser
doesn't accept self signed certificates , and one can't preprovision a browser easyily with those freshly created certificates. (Yes Karl I already read about certutil ... )
I had heard good things about Cucumber so I was pretty eager to start testing it ... In short Cucumber lack documentation ,
I tried a couple of things but I couldn't get beyond testing if a certain string was on a page.. couldn't figure out how to fill in a form etc ...
Maybe if anyone could point me to some great documentation on how you should write recipe's here ... I didn't find the documentation all to easy to find ..
Bummer as it really really looks promisiung .. specially since it is so lightweight ..
IP played with JMeter and Sahi too .. but still
So apart from filing bugs to the upstream project/product and hoping they understand your problem and are willing to oopen up their API , what other options do you folks suggest ?
I gave a short talk about this at Puppetcamp in Amsterdam and the audience came up with a bunch of other potential projects to look at .
The main problem still is that all these are tools to automate testing , they don't provide you with a general purpose approach to solve the configuration mgmt problem, each time the upstream vendor modifies the layout of his page you hav e to do the work again and that .. really doesn't sound promising ..
Comments
#1 hgh : Wondering but how can you
Wondering but how can you create your own self signed certificate as you mentioned? Be it Selenium or anything else, i feel, they msy always have problem with browser compatibility as well.
#2 Fredie G. : I looked into the list with
I looked into the list with potential projects that your audience at the Puppetcamp gave you and find the Watir particularly interesting. Will definitely check it out in more detail.
#3 Yuriy Gerasimov : Very interesting
Great article. I was shocked about Sikuli project.
I believe Hpricot should be removed form the list as it is parsing library but not the framework.
Cucumber is layer above testing framework to have tests written in "human" language. So Cucumber should be applied for already chosen testing framework.
As being fan of Selenium I believe it can handle the task properly. But of course you need to have Selenium RC server running for that somewhere in your infrastructure.
There are plenty of proprietary tools. The best list I have seen is here:
http://en.wikipedia.org/wiki/List_of_GUI_testing_tools
#4 Kris Buytaert : Nice List
Hadn't noticed that list on Wikipedia yet .... nice looking list altough only the upper part interrests me ..
I do work for an Open Source company after all :)
#5 Dick Davies : HTML Unit
We use HTML Unit for functional testing; it's a Java library with the Rhino Javascript engine embedded.
Works great if you use JRuby to drive it; you can query the DOM with CSS selectors to find elements, click them etc.
Even better then you can use IRB to explore the pages and build up a test script.
#6 Sheldon Hearn : You are not alone
Damn. I sit in a small office with 4 other guys. One of them shared this article with the rest of us.
The near-collective groan of disappointment as we all realized that your article didn't present the Final Ultimate Solution was hilarious and tragic.
You're not alone; Zabbix hates us too. :-)