Building a Truly Loadbalanced Transparant proxy

with Xen

by Kris Buytaert

Past Present and Future

a peek at 2.6

by Kris Buytaert

Legal openMosix is Copyright (c) 2002, 2003 by Moshe Bar.

Mosix is Copyright (c) 2002 by Amnon Barak. The Mosix is a trademark of Amnon Barak.

Linux is a Registered Trademark of Linus Torvalds.

openMosix is licensed under version 2 of the GNU General Public License as published by the Free Software Foundation.

openMosix package maintainer

Maintainer of the openMosix-HowTo

working as a Linux/open-source consultant

IANAKH :)

Agenda

What is openmosix

Installing and Managing openMosix

openMosix at late 2.4 changes

openMosix at 2.6

openMosix community

Clustering vs Clustering

Redundancy / Failover (LinuxHA, ...)

Loadbalancing (LVS, ...)

High Performance Computing (Beowulf, ...)

openMosix History

1981 MOSIX at the University of Jerusalem

1997 MO6 version 3 on BSD

1999 Mosix for Linux

2002 openMosix fork from Mosix

2005 2.6 release almost ready..

What is openMosix ?

Platform for HPC clustering

Kernel Patch + userland tools

SSI , Single Server Image Cluster

Balancing Load (processes) different machines

(e.g. LVS Balances connections)

What does openMosix ?

Migration of Processes to other nodes (PPM)

Decentralzied loadbalancing algorithm

Autodiscovery of available nodes

Preemptive Process Migration (PPM)

Process is splitted into : user + system context

UHN -> the processes home node

user (working) part migrates

system parts never migrates

used for communication

Loadbalancing without bottlenecks

decentralized algorithm

nodes know statistics of some neighbours

convert the total usage of several

heterogeneous resources into a single

homogeneous "cost"

based on market statistics

openMosix Advantages

no extra libraries required (mpi/pvm)

no changes to code required

no need to actually parrallellize your application

software distribution is easy

zeroconfig (autodiscovery)

Limitations

Threading issues (in Linux)

At one point Neo asks what happens if somebody dies in the Matrix, if he would also die in the real world. Trinity answered to him that a body (a pthread) cannot live without a mind.

(ie memory, ie address space).

Shared Memory

Not all applications benefit from MigSHM

Short programs don't migrate (e.g. Kernel Compiles)

No DFSA ready filesystems (anymore/yet)

Installing openMosix

Apply Kernel patch to clean Linux Kernel Source

Compile userland tools

Reboot all nodes with the same kernel version

Autodiscovery will do it's work now

Add nodes/remove when you need them

Pitfalls

Reboot all nodes with the same kernel version

time synchronisation is important (ntp)

openMosix exhausts hardware / memory

usermanagement (e.g. LDAP)

No Security ! (cfr Community)

Running applications on openMosix

decentralized concept -> no master-node

applications installed only once on one node

applications can be started on any nodes

Running PVM applications on openMosix

master/slave concept of PVM applications

static load/process assignment by PVM

dynamic loadbalancing by openMosix

minimizes the configuration needs for PVM

Running MPI applications on openMosix

MPI standard and implementations

similar master/slave concept as PVM

multiple program single data (MPSD)

openMosix patch to MPICH

openMosix enhances MPI performance !

Sequential applications on openMosix

parallellization of the problem to solve

split into subtasks by splitting data

run as MPMD (multiple program multiple data)

forking using the openMosix API

Administrating openMosix

the openMosix administration utilities :

mosctl	(basic administration)

mosrun	(running applications)

setpe	(configure cluster-map)

migrate + miggroup	(manual process migration)

/proc interface according to openMosix API

GUI for openMosix

Using openMosixView

(by Matt Rechenburger)

QT Based userinterface with :

openMosixView : monitoring

openMosixViewProcs : managing processes

openMosixViewCollector: logging

openMosixViewAnalyser: analyse the logs

openMosixViewHistory : history of the processes

openMosixMigmon : drag and drop migration monitoring tool

the main openMosixview application

the openMosixmigmon

openMosix calibration & tuning

io and/or cpu intensive applications

decay value / load-statistics

virtual node speed

locking processes

openMosix hardware tuning

increasing the network banwith + latency

bonding

firewire

gigabit

myrinet/ dolphinics / infiniband

increasing the processor power

increasing the number of nodes

New features in openMosix in 2.4

LoadLimit

ClusterMask

GOMD

Features Checker

MigSHM

Checkpointing

Recent changes in 2.4.26-om2

add the mighsm compile-time option

add features-checker

eliminate oMFS support

openMosix at 2.6

WE NEED TESTERS !

first beta's available from SVN

moving all possible code from kernel to user-space

zero configuration (just as in 2.4.)

separating arch dependant/independent code

we need testers for openMosix at 2.6

openMosix pid related files have moved from /proc/${pid}/ to /proc/${pid}/om/

node id

openMosix 2.4 identified a node by a nodeid.

2.6 nodes are identified by their network addres

we need testers for openMosix at 2.6

Moving a task

echo "192.168.1.1" > /proc/$pid/om/where will try to move $pid process to 192.168.1.1

echo "home " > /proc/$pid/om/where will bring it back

Ipv6 : echo "fec0:0:b55::1" > /proc/$pid/om/where

Load Balancer

No more auto-loadbalancing in kernelspace

Migration has to be done in userspace now.

we need testers for openMosix at 2.6

This gives more flexibilitiy:

having specific daemons (battery related, random, ...)

more powerful heuristics to choose tasks (arch, type of code, database of programs we know about).

ease the developement of the daemons (no system crash, more languages available : python, C, C++,...)

==> Attract more developers in userspace.

we need testers for openMosix at 2.6

Code cleanups.

no more obscure highly optimized/static code

easy to read, nice algorithms.

More people should be able to understand the code, and thus maintain it.

we need testers for openMosix at 2.6

2.4 userspace programs are no longer useful

2.6 has omd to do

discovery,

information balancing and

task migration election.

omd is written in Python

we need testers for openMosix at 2.6

Initial C userland written by students at the University of Tours (FR)

Matthieu Perotin wants to rewrite them in C++

Last month: new release of the initial C code by Evan Hisey

==> Soon now.

openMosix Community

Active mailing list (sf.net)

Active Wiki with success stories

Most Contributions are still 2.4 based !

openMosix Community Distributions aka

Instant Clusters

ClusterKnoppix

Chaos

PlumpOS

Quantian

Dynebolic

...

openMosix Community: GOMD

General Openmosix Deamon

runs on every node

collects information/statistics

ACL's

ClusterSnapshot , actual cluster status

Secure Cluster-wide (command) eXecution (SCX)

libgomd

openMosix and Security

isolate your cluster , it is insecure by design

mfs gave unauthenticated root filesystem access

no authentication required to join a cluster

processes are migrating unencrypted

firewall your cluster , have the firewall/gateway as a „master node“

What about 5-9 or Campus Clusters ?

openMosix and Security or the lack thereof

Chaos

Bootable CD

Adds IPSec between nodes

New Autodiscovery deamon tyd

openMosix Migration Encryption Patch

add encryption to the openMosix data and information communication channel

Migrating Shared Memory Applications

A process uses shared memory and has to be migrated.

Normally openMosix destroys the memory map and recreates them on the remote node

Other apps are using the same region

2 shared memory processes on the same node,

migrate both processes with their memory to the same remote	node exponential memory use growth

Migrating Shared Memory Applications

MigSHM patch developed by by the Maask group

Maya, Anu, Asmita, Snehal, Krushna, the developers of Migshm, are students of Cummins College of Engineering, University of Pune, India

Initial release for 2.4.21

Sucessfully tested with

Apache 1.3.X

Blast

Compile time option for 2.4.26-om2

YMMV

Checkpointing on openMosix

install chpox and add the chpox module

register process and its libraries

sequential dump process state to file(s)

restore process from dump-file

--> full-migration of processes possible

other openMosix Community Contributions

oMApplet

KludgKollection

wmonload

openMosixLoaf

openMosixBlaster

Gentoo / Debian packages

Who uses openMosix ?

Universities all over the planet

IBM Research

Nasa

Chemical Research

Fluid Dynamics

Bioinformatics

Running Smooth !

blender

grip

lame

freej

lives

kstars

john the ripper

dnetc

seti@home

pxclient

gridengine

wrf

Running Smooth !

Mathlab 5

Blast

MJPEG Tools

bladeenc

povray

mpi

flac

octave

R

maple

etc..

Other SSI Environments

openMosix

openSSI

Kerrighed

B-proc

openMosix near Future:

2.6 release

Integration with Clustering Filesystems

New platforms

omdiscd2

openMosix & Xen integration

Summary and conclusion

linear scalable clustering / decentralized loadbalancing

transparent application clustering

well documented API

huge user base / community

one of the most active linux-clustering project

Question time

?

!

Thank you

Contact:

Kris Buytaert <Kris.Buytaert@x-tend.be>

Further Reading

http://openmosix.sf.net

http://howto.x-tend.be/openMosix-HOWTO/

2004-02-20T13:39:32