HOW TO: Kubernetes Multi-node on Raspberry Pi 2s

Google’s Kubernetes is a powerful orchestration tool for containerised applications across multiple hosts. We achieved the first fully running implementation of Kubernetes on Raspberry Pi 2 today, and thanks to the ease of docker, you can too.

You will need:

At least 2 Raspberry Pi 2s

Two SD cards loaded with Arch Linux | ARM

First, we need to install docker and ntpd on all the machines (the Pis need to have the correct time to download docker images):

pacman -S Docker ntpd

Just hit y to continue. I recommend that you reboot your Pis after this so that both services come up cleanly. Now we need to create a setup implementing this:

Select a Pi to be Pi master, and ssh in. I recommend that you to su root for the following. Then run the this command to bring up docker-bootstrap.

sh -c 'docker -d -H unix:///var/run/docker-bootstrap.sock -p /var/run/docker-bootstrap.pid --iptables=false --ip-masq=false --bridge=none --graph=/var/lib/docker-bootstrap 2> /var/log/docker-bootstrap.log 1> /dev/null &'

Then we need to bring up etcd, the key value store used by Kubernetes. This command and any other docker run command with a new container might take a little while when first running, as docker will need to download the container. I’m working on shrinking the images to make this less of a pain.

docker -H unix:///var/run/docker-bootstrap.sock run --net=host -d andrewpsuedonym/etcd:2.1.1 /bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 –data-dir=/var/etcd/data

Then we should reserve a CIDR range for flannel

docker -H unix:///var/run/docker-bootstrap.sock run --net=host andrewpsuedonym/etcd:2.1.1 etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16" }'

Now we need to stop docker so that we can reconfigure it to use flannel.

systemctl stop docker

Run flannel itself on docker-bootstrap. This command should print a long hash, which is the id of the container

docker -H unix:///var/run/docker-bootstrap.sock run -d --net=host --privileged -v /dev/net:/dev/net andrewpsuedonym/flanneld flanneld

Then we need to get its subnet information.

docker -H unix:///var/run/docker-bootstrap.sock exec <long-hash-from-above-here> cat /run/flannel/subnet.env

This should print out something like this

FLANNEL_SUBNET=10.1.78.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=false

Now we need to configure docker to use this subnet, which is very simple. All we need to do is edit the docker.service file.

nano /usr/lib/systemd/system/docker.service

Then change the line which starts with ExecStart to include the flags –bip and –mtu. It should end up looking something like this.

ExecStart=/usr/bin/docker –bip=FLANNEL_SUBNET –mtu=FLANNEL_MTU -d -H fd://

Now we need to take down the network bridge docker0.

/sbin/ifconfig
docker0 down
brctl delbr docker0

Then we can start Docker up again

systemctl start docker

Now it’s time to launch kubernetes!
This launches the master

docker run --net=host --privileged -d -v /sys:/sys:ro -v /var/run/docker.sock:/var/run/docker.sock  andrewpsuedonym/hyperkube hyperkube kubelet --api-servers=http://localhost:8080 --v=2 --address=0.0.0.0 --enable-server --hostname-override=127.0.0.1
--config=/etc/kubernetes/manifests-multi –pod-infra-container-image=andrewpsuedonym/pause

And then this launches the proxy

docker run -d --net=host --privileged andrewpsuedonym/hyperkube:v1.0.1 /hyperkube proxy --master=http://127.0.0.1:8080 --v=2

You should now have a functioning one node cluster. Download the kubectl binary from here, and then if you run

./kubectl get nodes

You should see your node appear. Now for the first worker node.
These instructions be applied as many times as necessary to gain however many worker nodes you need.
We’ll need a docker-bootstrap again for flannel.

sh -c 'docker -d -H unix:///var/run/docker-bootstrap.sock -p /var/run/docker-bootstrap.pid --iptables=false --ip-masq=false
--bridge=none --graph=/var/lib/docker-bootstrap 2>
/var/log/docker-bootstrap.log 1> /dev/null &'

Then we should stop docker

systemctl stop docker

And add flanneld. This node doesn’t need etcd running on it, because it will use the running etcd from the master node.

docker -H unix:///var/run/docker-bootstrap.sock run -d --net=host --privileged -v /dev/net:/dev/net andrewpsuedonym/flanneld flanneld –etcd-endpoints=http://MASTER_IP:4001

The master IP address is the IP address of the first node we set up. You can
check that you have the right ip by running

curl MASTER_IP:4001

You should get a 404 response.

As before, we need to get the subnet information.

docker -H unix:///var/run/docker-bootstrap.sock exec <long-hash-from-above-here> cat /run/flannel/subnet.env

and edit the /usr/lib/systemd/system/docker.service file to include –bip=FLANNEL_SUBNET –mtu=FLANNEL_MTU when launching docker, just like we did before
Now we bring down docker’s network bridge and reload it.

/sbin/ifconfig docker0 down
brctl delbr docker0
systemctl daemon-reload
systemctl start docker

This Pi is ready for kubernetes now

docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock andrewpsuedonym/hyperkube hyperkube kubelet --api-servers=http://${MASTER_IP}:8080 --v=2 --address=0.0.0.0 --enable-server --hostname-override=$(hostname -i) –pod-infra-container-image=andrewpsuedonym/pause

docker run -d --net=host --privileged andrewpsuedonym/hyperkube hyperkube proxy --master=http://${MASTER_IP}:8080 –v=2

Running kubectl get nodes on the original Pi should now return both nodes.

Graphs and Results!

It’s been a busy fortnight. Much has been done, so I’ll get straight down to business.

Hadoop, being java based, was not overly difficult to package into a container and get running. However, proving that it was capable of networking between containers or even hosts looked like too much of a time sink, and so it has only been tested out as a single node. You can find the dockerfile here, and the container itself here.

It was a great shame that Hadoop was not the success I’d hoped for, because I would have liked to have used it for benchmarking. Instead, after much pontification, I decided to go with Stress, which although it doesn’t claim to be able to set specific CPU loads, it at least fulfils the functions it claims it can do. Lookbusy on the other hand is supposed to be able to set CPU loads as a %, but instead only lets you choose how many cores hit 100% usage. With Stress, and through adjusting the amount of calls to malloc()/free(), I was able to simulate different CPU loads, and here’s the box plot that emerged. Each measurement was taken 10 times, and the command used was:
stress -i 250 –vm-bytes 4 -t 25 -d 2 -m [VARIABLE].

 

 

I also managed to get some interesting read write speed data, which I present here alongside the best raspberry pi 1 readings we achieved

Command used: ./iozone -e -I -a -s 50M -r 4k -r 512k -r 16M -i 0 -i 1 -R

Raspberry Pi 2 w/ 32 GB Sandisk Extreme micro SD

Random Write 4K : 2.1 MB/s
Random Read 4K: 6.7 MB/s
Random Write 512K: 17.7MB/s
Random Read 512K: 19MB/s
Random Write 16MB: 18.388MB/s
Random Read 16MB: 19.2MB/s

Raspberry Pi w/ 16 GB Sandisk Ultra SD card
Random Write 4K : 1.0MB/s
Random Read 4K: 4.2MB/s
Random Write 512K: 19.8MB/s
Random Read 512K: 21MB/s
Random Write 16MB: 21.2MB/s
Random Read 16MB:22MB/s

As you can see, for larger files the Pi 2 is slightly slower, but we’re not sure why yet. It may well be the larger data capacity of the micro SD cards.

It was then time to stress test our defenceless Pi Cloud. I used SaltStack to send a punishing stress command to all 14 Pis, which they had to maintain for ten minutes. Each rack of seven suffered one failure. A single Pi could grind through this command, but the strain of seven was too much for our measly three amp USB hubs, but we have some new ones due for delivery on Monday. This of course means we’ll have to redesign the towers again, but then the struggle towards perfection is eternal, and sometimes fun. A 3d model of the finished design will be made freely available for replication when we have it. The other fruits of my labour can be found in my docker hub repository, here.