Everything is a Freaking DNS problem

Those who were present at the 1st Belgian MySQL UG might have overheard that I had spend the better part of tuesday , fighting with an unstable MySQL cluster, ndb nodes dying with no reason whatsoever and no traces in the MySQL logfiles.

I found the issue wednesday evening, it wasn't even MySQL related.

 Aug 30 12:30:30 DB-A kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
 Aug 30 12:30:30 DB-A last message repeated 6 times
 Aug 30 12:30:43 DB-A kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
 Aug 30 12:30:44 DB-A kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
 Aug 30 12:30:44 DB-A kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
 Aug 30 12:30:44 DB-A kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
 Aug 30 12:30:44 DB-A kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
 Aug 30 12:30:45 DB-A kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
 Aug 30 12:30:45 DB-A kernel: VM: killing process ndbd

Thing was that I was testing on one of the older platforms which don't have the 2Gb of memory I`m used to on my test platforms, but only have 512.
Addedd some more bars :)

On a related note , someone just posted a comment spam to one of my former MySQL articles pointing to some site trying to sell a product converting a MySQL database to some proprietary file format , so I`m wondering .. which moron would want to do that ? Just as I wonder which morons would actually buy stuff they get offered in spam anyway ..

Kung Fu Apps reports that Amazon Just Out Innovated EMC , it might even be true :)

The concept of serving virtual machines on demand looks really promising,
Kung Fu claims it's all based on Xen, understandable, at first I couldn't find any direct references on the Amazon site but digging deeper showed this thread where one Amazonian mentions their platform is based on an Fedora Core Xen platform.

Really great stuff. Now if only the full code to the API would become available so that we can use it ourselves rather than having to pay Amazon for bandwith and storage which we have plenty locally :) Or did I overlook the download button ?

canvas.be - The IT crowd

Added Keyword search to my Mythtv schedule :)

In Open source location redux Matt wonders if he should add more sales infrastructure in Europe or less.

(Having said that, Alfresco has realized an immediate gain from adding sales/business development people in Germany and France. Suddenly, many downloads that had been inclined toward a free beer discovered the value of paying for freedom. So maybe the real solution to the "problem" is to invest more sales infrastructure in Europe, rather than less? Not sure....)

I think the obvious answer is to in invest smarter

As an open source company , Alfresco luckily was one of the firsts to realise that the only way to conquer the European market is to be present there locally.

RedHat and MySQL learned this slowly, it took years before customers could match a face to RedHat and even longer to to feel local presence. A European customer does not buy easily from some sales office in the UK if it's only a sales office, they want to talk to someone. Timezones and language differences make it even more difficult. Lots of people want to work with someone in their timezone who speaks their language and their culture (Sometimes even their dialect, as from experience I know that a Dutch salesguy isn't likely to sell much in Belgium)

As for Open Source startups it's impossible to do such an investment so they need to find good partners for local support. Being based in the USA is easy.. you have a vast amount of customers living in the same country speaking the same language.

But here's your next question ..
"How should an open source company choose it's partners ? , based on revenue from reselling products or based on finding soulmates with skills to support and contribute to their products ?"

I was looking for an alternative to MySQL Cluster for large tables, less critical tables but still important enough ones ;)

I got MySQL cluster up and running for most of my tables, but we need some tables that are storing log information, it's not really critical, but we don't want to loose it anyway. As we don't want to put that stuf in memory I was looking into replication those tables. Now Linux-HA takes care of which mysqld instance to talk to , but if I fail over the active database IP the applications start writing to the 2nd node which is the replication slave for a couple of tables. That's perfect for the Cluster tables, but it's pain for the replicated InnodB tables as replication breaks and I can't migrate back automagically.

Upon reading the --read-only parameter in the replication documentation I tought I had found the solution, only to realise this actually puts the whole mysql in read-only also the tables being used by MySQL Cluster, not a good solution either therefore.

Fiddling with different users on the different nodes having diffent permissions crossed my mind , but then I started thinking about creating a set of 3 tables. A .master table, a .slave table and on both nodes a .merge table of the previous ones. Read access would be to the .merge table, write access to the .master table, while using --replicate-rewrite-db to replicate to the .slave table. It could have worked (didn't try) but my tables were InnoDB.
I went ahead anyway and setup cross replication for both nodes to another database. Both nodes now have databases with the same tables
blah and blahrepl as this is log information I can always manually merge when I need the data , but at least I don't lose it when a node crashes.

One tricky thing however .. it's documented in the comments of the mysql site

If you're attempting to use both
replicate-do-db=from_name
and
replicate-rewrite-db=from_name->to_name
be aware that you need to actually say
replicate-do-db=to_name
because the rewrite rule apparently happens before
the do-db rule.

And that works.

I should look into upgrading those boxen to MySQL 5.1.11 and continue my tests on Disk Based Cluster , but as they are on an older platform with no prebuild rpm's available that wasn't an option today ..

You know you can't just boot Xen of a PXE server, it's not a normal linuxkernel your booting , so you need
a multibootloader, such as the COM32 one that is packed in recent syslinux versions.

While playing with openQRM I ran into an odd problem. I still haven't figured out the actual problem.

I saw COM32 Multibootloader v0.1 taking a Xen from boot/xen.gz
then the module boot/res-1-kernel and later boot/res-1-initrd.img ...
after loading the last one and about 2.5 lines of dots I got
Booting: MBI=0x000100d8, entry=0x00100000

Then nothing and it seemed like my machine locked.

The fix was to upgrade the multibootloader to the one that is in Syslinux 3.11 from there on.. no more problems.

Life is busy... and then you don't blog that much .

And it doesn't look like it's going to be any less busy in the next couple of weeks.

As usual I`m going to LinuxKongress it's in Nurnberg this year .. and the schedule looks interresting as usual,
well.. maybe not all the talks but at least the people :) Conference is booked, Hotel is booked, Ride back booked , still figuring out how to get there. Why can you get a travel itinary on www.nmbs.be but no pricing, not even an estimate ?

The schedule for LinuxWorldExpo.nl is also online, if you can call it a schedule .. well.. it has Harald, Georg .. some other people .. and me ..
But before that .. Barcamp Brussels. this time I`m gonna try to be on time and try to present something :)

In An Open Letter to Tim Bray about OSCON
documents his experience at OSCON.

I`'ve already commented earlier on different conferences and I feel his pain, some conferences don't fit your expectations because the target audience is different than you expected. Sometimes it's like talking about Single System Image clustering to a bunch of people that have just installed their first Linux box at home, other times it's like talking about the difficulty of configuring someones desktop preferences to a bunch of kernel developers, There's a good chance they don't care since they already know how to do that :)

However for me a talk was worth the effort if there was at least 1 person in the audience who learned something, If I can convince 1 (one) person in the audience to have a look at a project, to think about how he is working today and get feedback from him, it was worth the effort. And it's true that different areas of the opensource world should try to take a closer look at eachother, visit eachothers events and learn. You're never too old :)
So I`m sure that there was at least one guy in the audience who learned about Puppet and will start looking deeper into it.

Actually remembering LISA 05 , I submitted some Xen abstracts and didn't get accepted, only to have lots' of people complain why there was not one single Xen talk, so it seemed there were some issues on that conference program there too :) Actually I still didn't get any feedback on my abstracts on Xen deployments for this years edition.

So before the schedule is out lemme say this . imvho at this year's LISA there should at least be a talk on openQRM and some talks on deploying Virtual Machines. :)

Anyway .. the important part isn't even trying to interrest different kinds of people in your project , it's meeting them , it's hearing different opinions on how to do stuff and their reasons for doing so , It's learing from them , wether it's on kernel level, infrastructure level, or development level :)

So some high level exec at some company who probably never even touched the technology calls Xen unstable and not ready for production , and the whole IT press jumps on it repeating his unfounded statements.

customer@XenBox:~> uname -a
Linux XenBox 2.4.30-xen0 #3 Fri Aug 12 15:01:01 CEST 2005 i686 unknown
unknown GNU/Linux
customer@XenBox:~> w
17:59:44 up 321 days, 18:52,  1 user,  load average: 0.01, 0.00, 0.00

I'd call this stable.

(PS. Yes hostname and username have been stripped on purpose to keep the client info confidential, but this is a production Telco environment, and it's not the only one)

Happy SysAdmin day from the UKUUG