Learning Lessons

I’ve learned one of the most valuables lesson of my internship this week, and it’s definitely the most counter-intuitive one so far. It was also the most vexing.

The most annoying lessons that we learn in our life are never new pieces of knowledge, gifted to us by beautiful books or articulate articles. Nor are they the cascading of connections and clicks as assorted bits of information leap together and join up. No, these lessons are ones which take a long time to learn and make us feel very embarrassed for a good while afterwards. They are not the creation of new knowledge, instead they are the destruction of misconceptions. This week my misconception was the necessity of updates.

It doesn’t exist. Updates are a swindle and a lie. If a system is working, don’t change it. Don’t, for example, command all your nodes to update their most vital piece of software when there is absolutely no need for it. I really recommend that if you think that your machines do need some shiny new software, please find a good reason for updating it, and then make sure you test it. Please note that neither shiny or new are good reasons for updating software.

I updated docker to 1:1.7.0-1. This docker version doesn’t allow you to run containers on arm architectures. It’s a pretty big set back. After I realised that the guys at Arch Linux don’t archive previous software versions, and then taking a few moments to deal with this, I spent a decent portion of Monday becoming familiar with the PKGBUILD process and achieving a working docker v1.6.

A fixed docker 1.7.1 was released the day after.

The rest of this week ended up being devoted to creating the first (a scoop! The first!) ARM docker Hadoop image and dockerfile, after I first picked up some knowledge on hadoop, and spent Wednesday at Jeremy’s memory management conference. There were lots of very smart people there, but I missed some of the more interesting talks I’d hoped to make. The speakers will hopefully upload their notes soon enough though. I’ve sometimes noticed that Lecturer’s seem to be disappointed by the depth they have to limit themselves to when teaching, but that upper limit didn’t exist here. I imagine that as a researcher it can sometimes feel like there are very few people who understand or appreciate what you are doing, and the gathering of like minds must be a breath of fresh air.

I’m not sure I’ll have such a luxury when I hope to present the Pi 2 cloud at Glasgow’s Explorathon in September. Nothing has been confirmed yet, but we hope to have a stand there.

I’ll leave you now with this preview of next week’s blog post.

bristol board

A tale of two towers

Behold! Our new testing tower for the Pi Cloud:

pi 2 cloud

I’ve moved the Pis back to a lego rack of my own design. This was sadly necessary, as while Posco‘s 3d printed design looks great, and allows for much more airflow, its compact nature made hot swapping the Pis fiddly. The Lego design also allows easy access to a Pi’s HDMI port, making network trouble shooting just that little bit quicker. I’m sure I used to perform function over form when it came to Lego construction. I would ask myself, “does this spaceship have enough lasers?” The answer was usually no. However, I found the colour mixing on the right tower extremely vexing, and a shopping trip to a Lego shop might be in order. All very professional of course.

Building this Lego tower got me thinking about other designs, and now that the Pi 2 Cloud has its first show booked here at Glasgow University’s SOCS intern-fest, perhaps it’s time to start planning a remodel with something flashier. It’s also worth noting that we’re hoping to take the Pi 2 Cloud on a tour, so if you’ve any expos or shows coming up then please get in touch with us. In general I think we made good progress this week, but I this may only be in comparison to last week.

Kubernetes turned out to be something of a rabbit hole, as I’m not sure anyone has managed to follow the docker multi-node set up through on a Raspberry Pi 2 yet. We’re waiting for Google to get back to us with some help on this, but in the mean time falling back to Saltstack isn’t an awful compromise. I also had some difficulty with the Linux dd utility, which would work, but not quite, creating the correct partition tables on a blank SD card, filling them with the correct files, yet doing it in such a way that prevented booting. I worked around this by copying and pasting working boot files, but am no closer to figuring out what went wrong. Something somewhere corrupted, and as interested as I am in investigating this, I’m starting to gain a greater appreciation for what’s worth my time and what’s not (a dd operation taking 2 hours is not. Always remember to set block size!). Still, we have 14 Raspberry Pis in our cloud now, and next week I’m deploying a very cool distributed chess application to them and doing some benchmarking. I just hope numbering the Pis from 0 to 13 doesn’t prove unlucky!

Names and their meanings

“A rose by any other name would smell as sweet”. This is said by Juliet in Shakespeare’s most famous play, and she is perhaps half right. A rose does not smell like a rose only because it is called a rose. However, if a flower is named as a rose it is expected to behave like a rose, and smell as sweet. When an object is named we bestow upon it certain ideas and expectations, and this is all the more true for things which have been recently named. In this blog post I’ll be investigating why the components of our stack have the names they do.

Raspberry Pi

The reasoning of the naming of the Pi wasn’t very hard to dig up. It seems the dev team were nostalgic about old Home Micro PCs, and indeed wanted to build a successor to these, envisioning a platform which “like those old home computers, could boot into a programming environment”. A lot of these Home Micros had fruit based names, like Apricot, Tangerine and even Apple. The Raspberry part of this name seats the device comfortably into the computing tradition. The Pi part is perhaps less inspired, it comes from the simple fact that Python was the main language that people are expected to use and to learn on the Raspberry Pi. It’s a good name overall though, and makes clear where the Raspberry Pi comes from, and where it’s going.

Docker

To the best of my knowledge, this name comes from extending the metaphor of containers. In a particular “intro to docker” video I once watched, the speaker described the gap which docker fills with a shipping container metaphor. The speaker argues that before shipping containers were made, transporting goods was hard, as there was no standard around which shipping infrastructure could be built. In this context, the goods to be shipped is software, and the shipping containers become Linux Containers. Docker, like a dock, allows locals to easily deal with foreign products, by packaging these foreign products (or code) into a container where it is much easier to deal with.

Kubernetes

Originally a greek word, Kubernetes means ship’s helmsman, the last actor in navigating a ship. This is an interesting name as Plato used a ship of state in the republic, as a metaphor for the governance of a city-state. This classical allusion would make a software company any smaller seem pompous, but as it’s google we’ll have to let it slide. The name Kubernetes reflects the nature of the software as an orchestration tool, although it is much longer and rarer a name than is often encountered in software.

Saltstack

It seems that Salt is so named as an extension of it’s design philosophy, of being highly modular and extremely lightweight. Saltstack continues this theme with grains and pillars, and although it would be very interesting if Saltstack was in some way named for the biblical story of Lot’s wife turning into a pillar of salt, I think it is unlikely.

 

I can’t leave you without a progress update, so here is a picture of a 3d printed tower we’re testing for our friend Posco at the University of Liverpool

 

Posco's Pi tower!

Attrition

This week was a hard one. The cutting edge seems like a much more elegant concept when you’re far away from it, knowing that somewhere smart people are chipping away at the unknown. These wise philosophers use measured accuracy and precision, planning every move and quickly reaping the fruits of their intellectual labours. Now that I’ve joined them on this frontier, I can tell you that this is not how things are at all.

This not to say that all researchers are like myself of course, I’m sure all of them are much more capable, intellectual people than I, with fantastic methodology. Of course, this doesn’t mean that they never run up against problems either, and this week, I’ve had my fair share of them.

It started with Shipyard. Shipyard can be thought of as something of a poster child for docker. Simply download a container and then you instantly have a full docker cluster management suite, without ever having to deal with any of its nasty complexities. However, this is assuming that you are running docker on a 64-bit host. If you are on a raspberry pi, then you have to compile everything yourself from a minimal armv7 Ubuntu image, not just for the application container, but for the separate Rethinkdb container too, Shipyard’s data store. I managed to get shipyard working, but it’s Rethinkdb component is running into compilation errors despite it’s supposed arm support. I struggled against this for a while, but rather than get bogged down and keep hitting my head against a brick wall, I decided to move onto Kubernetes.

Kuberenetes is a hotly anticipated management tool, developed by Google and used as part of their own internal Borg infrastructure. It handles the scheduling of a compute cluster using labels and pods, and is currently in pre-production beta. Unsurprisingly, the developers at Google haven’t yet made single board 32-bit micro computers one of their key targets yet, so compiling Kubernetes is bringing its own host of problems.

Two pieces of advice that I would give to anyone doing something like this is make sure you have at least a GB swapfile, and that you should be more than willing to try to fool the installer. Kubernetes required a golang:1.4 container to install, but as the default one is 64 bit, I instead downloaded and renamed a Raspberry Pi 2 targeted one.

This week has taught me that I have a daunting path ahead of me, one that has been trodden by few others (I found one guy on twitter but he didn’t document ANYTHING). I am not going to give up. Far from it, now I better appreciate the scale of this challenge ahead of me. All these in-alpha orchestration programs have reminded me just how important the education and research that this platform will provide will be. I cannot wait to dive back into the project in a weeks time.

End on a bright note, look at my new mug. See you next time.

It's official now!

It’s official now!

PROGRESS

We have chosen a distribution!

Raspbian was the first to fall, it’s lack of an armv7 targeted derivative leading to concerns about its performance. Next was Debian armhf, for which I’d had big hopes. Unfortunately configuring Debian for Docker was taking too long, and we had to move on. Next down was Ubuntu Mate for the Pi 2, which is honestly an excellent distribution, but just wasn’t right for us. Ubuntu Mate installed docker perfectly well, but came with too many apps we didn’t need and was worryingly specific compared to Arch Linux Arm. Arch suited our requirements perfectly. Docker installed without a hitch, and Arch has a large enough community to ensure that the distribution will continue to be supported for a long time.

With our operating system chosen, it was time to start building up our stack. We needed a tool to administer our small dev group of Pi’s for, which could be easily extended to five times the number of machines. The tool would have to be lightweight, scalable, easy to learn and fully featured. For this reason we went with Saltstack. Saltstack has everything we need. It’s FOSS, and powerful. I can send commands to every machine in just one line, it has a server/agent (salt-master/minion) relationship built in, and it supports docker. It even comes with an API, and after a little bit of fiddling I was able to get SaltPad, a web based GUI for Salt, up and running on pi0, which also hosts the Saltmaster.

Screenshot from 2015-06-12 13:13:13

SaltPad is a promising project, but it’s still in early alpha and as buggy as you might expect. It’s also missing some critical features for us. It’s very worthy regardless, and I might try toying with this alongside shipyard, the free docker management tool, seeing how effective the combination is, and if our Pi master can handle it. If nothing out there ends up working for us, then we’ll have to build our own solution. Just in case that does end up happening, let us know in the comments the features that you think all cloud management software should have.

We’re back!

Yes, we’re doing it. We’re building (what seems to be) the first distributed Raspberry Pi 2 cloud!

A look back and then a look ahead.

Hi, I’m Jim. I’m going to be running this blog for the next 10 weeks, and it’s going to be an exciting time. A lot has changed since we last posted here, in 2013. A wealth of related projects have been started and finished, papers released and cited. Here’s a selection of those which we thought noteworthy.

In the intervening time, the Pi 2 has also been released. The Pi 2 has a much faster armv7 processor and 1GB RAM. While it’s still far from competing with a desktop PC, it’s a massive improvement from the Pi 1. Because of this, we’ve decided to build and benchmark the second iteration of our favourite cloud. Development has just started, and I will be posting weekly updates. We haven’t settled on a stack yet, but we’re definitely going to be using Docker. We’ll be testing the following distributions:

  • Arch Linux | Arm for the Pi 2
  • Ubuntu Mate for the Pi 2
  • Debian armhf
  • Raspbian

If you have any suggestions for other distributions, please let us know in the comments. Any chosen distro will need to take full advantage of the Pi 2’s architecture, and supporting Docker out of the box even better (but I don’t mind compiling a kernel or two). If you check back next week I will hopefully have set up and benchmarked all of these, each on a single Pi. This is a great project and once completed it will be a massively useful tool for research and education.

We’re also resurrecting our twitter account, so follow us on that if you haven’t already: @glasgowpicloud.

 

See you next week.

Raspberry Pi Lego Rack Designs

A few people requested that we describe the design of our racks.

The truth is, each rack is slightly different and the final build is not the one we’d planned. A couple of reasons for this: we didn’t quite receive the Lego pieces we were expecting, and we had to tweak the designs to make things fit better. So this is a somewhat retrospective design document…

We have four racks containing 14 Raspberry Pi’s each, actually composed of two adjacent towers. In between the towers are two USB hubs. The design is such that the front provides access to the SD card slot and micro-USB power supply, so we can easily change SD cards and reset the Pi’s. We can also slide the Pi’s out, which is incredibly useful as we tend to be cannibalising Pi’s quite often, or else swapping them around for testing. The back of the racks has space to reach the ethernet port, and each rack has a dedicated Netgear GS116E switch.

Each rack sits on a green 32 by 32 stud (25cm x 25cm) Lego baseplate.

Ten studs worth of space is left in front of the rack, and eight studs at the back. There’s a gap of two studs either side of the rack.

For the most part, the towers follow four simple layers of Lego, corresponding to a snug fit for a single Raspberry Pi. The only exceptions to these layers are (a) connecting structs between the two towers to hold them together and (b) the top of the racks, which are frankly all of differing design depending on the academic or student that built them! Particularly as the available Lego ran out, the designs became more improvised. So here I’ll just show the three layers that make up the towers and a few examples of the improvisations people chose.

For the first layer, we use 2×4 Lego bricks to create “feet” that protrude into the space for the Pi. The Pi actually sits on these feet, to give it space below for the SD card and to allow airflow underneath the Pi’s (the extra piece in the centre is to keep the USB hub in place):

IMG_2258

Next, we simply add two layers of Lego that do not overlap the feet, just building the perimeter wall with enough room for the Pi:

IMG_2262

Here’s a few horizontal shots to clarify:

IMG_2264 IMG_2263

The fourth layer is more or less the same as the first layer, but we add a long strut at the back instead of the two 2×4 lego pieces we had, to strengthen the structure:

IMG_2265

shots from the side and the back:

IMG_2266

IMG_2270

Now we simply repeat the second, third and fourth layers on top of this, until we eventually have enough room for seven Pi’s in each tower. Just a couple of exceptions:

Struts are added an intervals in the rack to strengthen the towers. For example, see the one halfway up the red tower in the image below:

The tops of the racks are a little improvised. Here’s one example:

IMG_2272 IMG_2271