Everything is a Freaking DNS problem

"An expert is a man who has made all the mistakes which can be made, in a narrow field."
Niels Bohr

When I setup Logstash for the very first time I got bitten by an empty search, aparently no logs were indexed. Reading the log files indeed told me about

WARN: org.elasticsearch.discovery.zen.ping.unicast: [Blaire, Allison] failed to send ping to [[#zen_unicast_1#][inet[/127.0.0.1:9300]]]
INFO   | jvm 1    | 2012/02/06 22:45:55 | org.elasticsearch.transport.RemoteTransportException: [Page, Karen][inet[/127 .0.0.1:9300]][discovery/zen/unicast]
INFO   | jvm 1    | 2012/02/06 22:45:55 | Caused by: java.io.EOFException

The above is the typical error when the ElasticSearch version you are using externally is not in sync with the one Logstash is using, yes those versions need to match.

Fast forward a couple of weeks.. and I`m upgrading Logstash and therefore also ElasticSearch .. I have a Vagrant setup to play with so all of the components are running on 1 node.

I kept running into a similar problem, this time however I saw log entries being indexed, I could get data from my ElasticSearch setup using
wget -q -S -O - http://localhost:9200/_status?pretty=true

But the web interface kept showing no results ;(

While nagging about it on irc .. Jordan gave me the insight :

2012-01-31.194347+0100CET.txt:(07:55:36 PM) whack: slight caveat that elasticsearch clients also join the cluster, so if you point everyone at 127.0.0.1:9300, that :9300 could be one of your clients, not the server

Indeed when you by accident start any of the logstash instances (server/shipper/web) before you start your ElasticSearch instance you can be in trouble.
Ordering really matters , you really need to start ElasticSearch before you start the clients.

Obviously is you don't use the unicast setup you don't run into this problem ..

So what other mistakes should I make ?