Jun 01 2016

Linux Troubleshooting 101 , 2016 Edition

Back in 2006 I wrote a blog post about linux troubleshoooting. Bert Van Vreckem pointed out that it might be time for an update ..

There's not that much that has changed .. however :)

Everything is a DNS Problem

Everything is a Fscking DNS Problem
No really, Everything is a Fscking DNS Problem
If it's not a fucking DNS Problem ..
It's a Full Filesystem Problem
If your filesystem isn't full
It is a SELinux problem
If you have SELinux disabled
It might be an ntp problem
If it's not an ntp problem
It's an arp problem
If it's not an arp problem...
It is a Java Garbage Collection problem
If you ain't running Java
It's a natting problem
If you are already on IPv6
It's a Spanning Tree problem
If it's not a spanning Tree problem...
It's a USB problem
If it's not a USB Problem
It's a sharing IRQ Problem
If it's not a sharing IRQ Problem
But most often .. its a Freaking Dns Problem !


May 28 2016

Docker and volumes hell

We're increasingly using Docker to build packages, a fresh chroot in which we prepare a number of packages, builds typically for ruby (rvm) , or python (virtualenv) or node stuf where the language ecosystem fails on us ... and fpm the whole tree as a working artifact.

An example of such a build is my work on packaging Dashing.

Now part of that build is running the actual build script in docker with a local volume mounted inside the container This is your typical -v=/home/src/dashing-docker/package-scripts:/scripts param.

Earlier this week however I was stuck on a box where that combo did not want to work as expected. Docker clearly mounted the local volume, as it could execute the script in the directory, but for some reason it didn't want to write in the mounted volume.

docker run -v=/home/src/dashing-docker/package-scripts:/scripts dashing/rvm /scripts/packagervm
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
corefines: Your Ruby doesn't support refinements, so I'll fake them using plain monkey-patching (not scoped!).
/usr/local/share/gems/gems/corefines-1.9.0/lib/corefines/support/fake_refinements.rb:26: warning: Refinements are experimental, and the behavior may change in future versions of Ruby!
/usr/share/ruby/fileutils.rb:1381:in `initialize': Permission denied - rvm-1.27.0-1.x86_64.rpm (Errno::EACCES)

So what was I doing wrong, did the Docker params change, did I invert the order of the params, did I mistype them ? I added debugging to the script, (ls , chmod, etc..) and I couldn't seem to read or modify the directory. So I asked a coworker to be my wobbling duck.

He did more .. he wondered if this wasn't selinux.

And he was right..

Apr 23 21:47:00 audit[9570]: AVC avc: denied { write } for pid=9570 comm="fpm" name="package-scripts" dev="dm-2" ino=368957 scontext=system_u:system_r:svirt_lxc_net_t:s0:c47,c929 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=dir permissive=0
Apr 23 21:47:02 python3[9597]: SELinux is preventing fpm from write access on the directory /home/src/dashing-docker/package-scripts.

So while I was looking for errors in docker, it was just my selinux set to enforce acting up and me not noticing it.

The quick way to verify obvisously was to setenforce 0 and trigger the build again that however is not a long term fix so I changed the

semanage fcontext -a -t cgroup_t '/home/src/dashing-docker/package-scripts'
restorecon -v '/home/src/dashing-docker/package-scripts'

That solves the problem

Dec 01 2008

Upgrading to Fedora 10

As some of you know I run Fedora on my main Laptop, and Ubuntu on my EEE and my office Desktop. My Ubuntu installs were pretty fresh, but my fedora was long due for an upgrade. As after years of abuse (--force --nodeps) it usually is impossible to do a regular upgrade, so my best way out was a quick and fresh install on an isolated root partition. (my /home/ etc live on separate volumes)

Johan tricked me into installing a FC10 x86-64 setup which required some more work than usual, but I`m back in the same state as I was before I started the upgrade. We'll almost .. as I expect I`ll be reinstalling packages I don't daily use on a frequent basis over the next couple of weeks.

I'll dig a bit deeper in my x86_64 struggles later but first I had to tackle the problems I had with stepping my Evolution to a fresh and hopefully more stable version.

Evolution fails to recognise the fact that I have multiple calendars, or got confused.. anyway .. my calendar was empty and I had to recreate them all and import them again one by one.
Luckily the content was still available in .ics

Also Evolution partly forgot about all my email accounts , it remembered their passwords after recreating them, but not their settings.. well. that is if I managed to recreate them all :)
(If I`m not replying to a mail you sent me sinc last thursday.. I might just have forgotten about that email address ..

On to the x86_64 stuff then. Bort pointed me to the
Adobe Labs site that had a working flash plugin. The right symlink in my .mozilla/plugins and the first problem was solved.

There's 2 apps from Google that I sometimes abuse, I don't know why I still use Picasa, there must be plenty of good open source alternatives out there .. however for Google Earth I don't really have a good alternative. bot work (unlike previous Fedora/Googl Earth combinations)

The only thing that annoyed me was Google Earth and selinux , I had to run some chcon s to solve the problems.

  1. chcon -t textrel_shlib_t '/opt/google-earth/'
  2. chcon -t textrel_shlib_t '/opt/google-earth/'
  3. chcon -t textrel_shlib_t '/opt/google-earth/'
  4. chcon -t textrel_shlib_t '/opt/google-earth/'
  5. chcon -t textrel_shlib_t '/opt/google-earth/lib*'
  6. chcon -t textrel_shlib_t '/opt/google-earth/'
  7. chcon -t textrel_shlib_t '/opt/google-earth/'
  8. chcon -t textrel_shlib_t '/opt/google-earth/'
  9. chcon -t textrel_shlib_t '/opt/google-earth/'
  10. chcon -t textrel_shlib_t '/opt/google-earth/'
  11. chcon -t textrel_shlib_t '/opt/googleearth/'
  12. chcon -t textrel_shlib_t '/opt/googleearth/'
  13. chcon -t textrel_shlib_t '/opt/google-earth/'

A couple of more frustrations later and the SELinux config got changed. Afterall .. Life is to short for SELinux.

A must, if you used too have Livna , or FreshRPMS is RPMFusion a merger of the old repositories

Skype seemed to be a bit more difficult. I tried installing the rpm at first but I failed , then I read some tips which suggested to use yum localinstall skype*.rpm

As I`m more of an apt-get user (yes even on CentOS etc) I don't usually try installing packages that way .. but it worked like a charm. Skype started, I could connect to the servers , however a test call reveals no input sound. Maybe if I try again with a headset later.

Anyway, I`m pretty satisfied so far ...

Next step is to get that skype chat integrated in pidgin.. or hope people will realize they have to drop that proprietary tool one day..

Aug 25 2008

SELinux is Dead !

No really it isn't but so isn't AppArmor, altough Russel claims it is. Weird how he totally rewords the OpenSuse statements

From "While our customer experience shows that AppArmor is the best solution for the vast majority of users, applications, and use cases, we want to give all of our users the ability to choose the security framework that’s appropriate for their respective environments and needs. We continue to enable AppArmor as our default Host Intrusion Prevention System......"

To "The next step will be to make SE Linux the default and AppArmor the one that exists in a repository, and the step after that will be to remove AppArmor."

Given Ubuntu's AppArmor adoption I don't see it die that fast ...

The real problem is who uses AppArmor or SELinux, sadly most of the installations I run into have none of both technologies enabled. Most Admins overrule their favourite distro's default config. The reason is pretty obvious as I've heard a lot of intelligent people say "Life Is Too short For SELinux"

So I wonder, how can you die if you never were really alive ?