Participating in Docker global hackday #3

Written on October 29, 2015 by @marcosnils

Some background

It’s been a while since I started playing with docker, I can’t remember the exact date, but my first experiments started a few months before I attended to Oscon and went to Solomon Hykes talk (http://conferences.oreilly.com/oscon/oscon2013/public/schedule/detail/28670)

At the beginning my understanding of docker was really basic and imprecise. I wasn’t sure if it was here to replace VMs or complement them in any way. It was during that Oscon talk that I actually understood how VMs and docker can work together and complement each other and then my mind blew up! Since then and now I had the opportunity to really use it in different scenarios and test it both for development and production. And I still have the same feeling as the first day I came across docker, I am still amused by it’s simplicity and power.

Around September 2014 Jonathan and I joined Mantika (mantika.ca) a machine learning startup that uses the latest in research to solve problems of the industry. It was the first time for both of us that we work for such a different environment with so different requirements and workloads. Out previous experience was mostly working for Software Factories and E-Commerce websites.

But now we had to deal with huge datasets, and processes that run for hours before knowing if there is something that needs to be adjusted. And that represented for us important challenges, specially because we were not in our comfort zone.

One of those challenges is one of our production services, a python bottle web service that ends up executing a neural network trained model to process images in real time and extract some important information from the content of the image. This turns out to be a lot of matrices multiplications, but of very big matrices. To be able to do this efficiently and have a decent response time we use GPUs and special python libraries (cuda and theano) which allow us to get the most of the instances we use to run these processes.

Now this works nicely and response time is acceptable, but it also means that the time it takes for the web service to startup is very long, because all these libraries do a lot of online compilation, and we also need to preload a lot of information to memory, so we avoid hitting disk during the lifetime of the request. So all in all, we need to wait around 5 minutes average for the application to be ready and warmed up. Now that is a long time when you need to scale out fast, and it also make it super expensive when you web app crashes for some reason or you want to just kill the VMs where this is running because you have a better setup on different instances.

Basically we started to feel that we were working like we used to before moving to the cloud, where instances where long living and the need to redeploy because of a dead server was something very rare.

The discovery

We knew we had something that we needed to improve but we couldn’t come up with a solution that would be both simple and efficient until we came across the docker engine 1.7 talk (https://www.youtube.com/watch?v=mL9AFkJJAq0) where Michael and Arnaud showed (without explaining in detail) how they migrated a Quake III server between data centers in “almost” real time.

Being a pro Quake III player in the past and a technology enthusiast we were paralyzed when we saw what was happening in the demo. Our gaming community in Argentina is relatively small and I remember having latency issues constantly when playing Quake 3 online a couple of years ago, so the fact that you could move the server around without disconnecting players was unbelievable. We immediately started to investigate how the dockercon presentation was achieved, we knew it had something to do with a project called CRIU (http://criu.org) but even though the project is well documented, there were still some missing gaps about how to get a complete working usecase.

Luckily for us a a day or two after the dockercon talk, the kubernetes blog posted (http://blog.kubernetes.io/2015/07/how-did-quake-demo-from-dockercon-work.html) a detailed theoretical explanation about how the dockercon demo was achieved.

The trial

After setting up a development box and patching / recompiling our kernels, we managed to get a working version of CRIU where we could checkpoint / restore unix processes without hassle. The next step for us was to be able to set up a demo where we could migrate a container around as it was shown in the dockercon.

At the same time this was happening we found that Medallia Argentina was hosting Docker Meetups and we thought about giving a presentation locally about all the things that we learnt on the way. We got a spot to show our presentation (http://www.meetup.com/es/Docker-Buenos-Aires/events/223513542/) where we prepared some simple demos using runC to show everyone how the dockercon presentation was achieved and which other features Checkpoint / Restore offered to end users.

The Hackday

Even though we learnt a lot about how checkpoint / restore works we realized that the process itself was very error prone and that several pre-conditions and post-checks needed to be made to guarantee a proper migration. In addition, we also found that there was no tool available which helps to perform container migration transparently across multiple hosts. Our initial investigation lead us to very promising results but unfortunately during the upcoming weeks our priorities changed and we couldn’t found the time to improve this project until suddenly we heard about the Docker global hackday #3 (https://blog.docker.com/2015/07/announcing-docker-global-hack-day-3/) and we thought that we could take it as a good excuse to deliver a POC of the container migration tool. We signed into the contest and we set ourselves a realistic goal that we needed to achieve the very first day of the hackday. “We wanted to do something that automatically performs container migration and allows users to understand what’s going on during the whole process and add hooks to perform different actions depending on the ending results”. Jonathan and I discussed about the design and we agreed to make it as simple as possible for the sake of having something finished for the competition. We decided that the most important thing was going to be the CLI and the user API, we weren’t going to use the CRIU rpc protocol (https://criu.org/RPC) and that all the file transfering was going to be made through ssh.

The day of the competition arrived and as we didn’t have too much time in our hands we needed to validate that our design was correct and that what we wanted to achieve was actually doable. Fortunately, the result of the first day was very promising; we got a prototype of our project in form of a shell script (https://github.com/marcosnils/cmt/blob/master/magic_migrator.sh) and we also thought about several other improvements that could even make the project better, like adding support for pre-dumps. From there on it was just a matter of migrating our prototype into something more robust and add some extra logic that we had in mind. We decided to use Golang we have a lot of experience using it and Docker uses it as it’s primary language; we also thought that delivering a single binary for each platform would be convenient.

The script migration was fairly simple, and we came up with nice abstractions like the cmd package (https://github.com/marcosnils/cmt/tree/master/cmd) which would automatically execute commands locally or remotely through ssh depending on the path specified by the user. Some other obstacles that we found in the way was the fact that runC didn’t support pre-dumps and that the CRIU client had a bug when performing cpu compatibility checks. Fortunately for us, it was quite simple to add and fix the missing pieces (https://github.com/marcosnils/cmt#important) and have a working alpha by the end of Sunday.

The submission

As it happens with every software project, we only had one hour remaining to submit and we were still adding features and fixing bugs; and because there was no time for unit testing, we didn’t even know if we were breaking some existing functionality. Twenty minutes before the deadline we just crossed our fingers and opened our PR against the hackday repo (https://github.com/docker/global-hack-day-3/pull/13).

From there on the only thing that we could do was waiting. One thing that wasn’t entirely clear according to the contest rules or site was when the voters were allowed to start voting. Once we submitted our project we tweeted our accomplishment and our relatives started asking how they could vote so we could get better opportunities for the global prices. Unfortunately we had to hold a lot of people and lost some votes due misinformation about how the voting system worked.

The Verdict

For those who are not familiarized to the contest rules, two voting instances were held. In the first one a jury composed of docker experts would choose the best project from each office around the globe and as we were the only project from Buenos Aires, we got the local price. The second round which elected the best global projects in 3 different categories, was decided by social voting.

What we would like to remark about this second round is the fact that we felt that the social voting system wasn’t completely refined by the time the submissions ended. The voting page didn’t allow voters to navigate a specific project when instead they needed to browse the complete list (+60) and search for the project manually. This specifically was annoying because it was really difficult to get votes from friends / family as we needed to send them detailed instructions about how to actually find the project and do the voting. We came across some other minor issues at this point but the GHD team was very supportive and worked hard to get all the necessary stuff in place.

Another thing that would have been amazing is to get some feedback from the juries about the projects. Even for those who were elected, it’s always positive to understand what qualities of the project were the ones who received more points and which ones weren’t. We firmly believe that this kind of feedback can be useful for future projects and will give the competition some transparency about the decisions, specially when the projects were evaluated by too many different aspects like performance, design, applicability, etc.

In the end, we got the surprise that our project was elected first in the plumbing category which brought us undescriptive happiness and since then we’ve been brainstorming different ideas to prepare the best presentation possible for the spot awarded at the dockercon europe by mid november.

Final words

We couldn’t have accomplished this without the support from many people starting from everyone who voted us plus some key players like @boucher who helped to validate some ideas and gave some unvaluable tips about what we could do to make our project better. We’d also like to thank Docker Inc for hosting it’s third hackday and give us this opportunity to put our hands into cool stuff and to continuously contribute to the open source community.

We strongly believe that if the correct amount of effort is put into the project it might become a useful tool which would benefit many use case scenarios like time-lapse debugging, immediate scaling, scientific processes and many others.

Last but not least I’d like to congratulate the CMT members @xetorthio and @geisbruch for the accomplishment and dedicate their free time to get this project going and believing in the idea that we wanted to pursue.