Over the years, I’ve used Jenkins, Concourse and a few other CI software. Recently, when the multistage dockerfile feature was released, it dawned on me that I used CI software for mainly 3 things: watching github repos, having a web UI to monitor builds, and be able to define pipelines.
The last one is the core selling point: defining pipelines allows me to design a build pipe where at the end, only the necessary stuff is included in the final docker container.
With multistage dockerfiles, we can now do that directly inside the dockerfile. We can define build steps, and at each step decide what is sent to the next one.
But then, is it still worth it to master a full fledge CI software just for git watching and web UI?
I finally hacked together a python/celery/flower/rabbitMQ stack to have out of the box: a task queue, a web ui, an api, and a python framework for tasks.
Here is the docker-compose.yml:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| version: '2' | |
| services: | |
| registry: | |
| image: registry:2 | |
| ports: | |
| – "127.0.0.1:5500:5000" | |
| environment: | |
| REGISTRY_STORAGE_DELETE_ENABLED: 'true' | |
| git-watcher: | |
| build: ./watcher | |
| ports: | |
| – "127.0.0.1:6000:3000" | |
| volumes: | |
| – ./logs:/logs | |
| – ./versions:/versions | |
| – ./git_data:/git_data | |
| environment: | |
| NODE_ENV: production | |
| rabbit: | |
| hostname: rabbit | |
| image: rabbitmq:3.7.3 | |
| volumes: | |
| – ./rabbit_data:/var/lib/rabbitmq #for data persistence | |
| environment: | |
| – RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER} | |
| – RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS} | |
| flower: | |
| build: ./worker | |
| command: flower -A worker –port=5555 –persistent=True –db=/db/db –broker=amqp://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbit | |
| environment: | |
| – BROKER=amqp://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbit | |
| volumes: | |
| – ./flower_data:/db #for data persistence | |
| ports: | |
| – "127.0.0.1:5000:5555" | |
| links: | |
| – rabbit | |
| depends_on: | |
| – rabbit | |
| worker-1: | |
| build: ./worker | |
| privileged: true | |
| command: celery -A worker worker –loglevel=info -Ofair –concurrency=1 | |
| environment: | |
| – BROKER=amqp://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbit | |
| – REGISTRY=127.0.0.1:5500 | |
| volumes: | |
| – ./logs:/logs | |
| – ./versions:/versions | |
| – ./git_data:/git_data | |
| – /var/run/docker.sock:/var/run/docker.sock | |
| – /usr/bin/docker:/usr/bin/docker | |
| links: | |
| – rabbit | |
| depends_on: | |
| – rabbit | |
| worker-2: | |
| build: ./worker | |
| privileged: true | |
| command: celery -A worker worker –loglevel=info -Ofair –concurrency=1 | |
| environment: | |
| – BROKER=amqp://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbit | |
| – REGISTRY=127.0.0.1:5500 | |
| volumes: | |
| – ./logs:/logs | |
| – ./versions:/versions | |
| – ./git_data:/git_data | |
| – /var/run/docker.sock:/var/run/docker.sock | |
| – /usr/bin/docker:/usr/bin/docker | |
| links: | |
| – rabbit | |
| depends_on: | |
| – rabbit | |
| volumes: | |
| rabbit-staging: {} |
I then wrote a celery task that runs docker builds in a given folder.
I also set up a simple nodeJS github watcher, and configured github webhooks to ping this watcher.
When a repo changes, the webhook pings the node service, which performs a local git pull.
It then adds a build task in the rabbitMQ queue, using the flower API. The celery worker then executes the task, which consists in copying the freshly pulled repo, running a docker build inside it, and pushing the image to our docker registry.
All this took me a day to setup and debug. We now have a blazingly fast, full docker-based CI system, with a nice web UI and API (thanks to Flower). And it all takes a “docker-compose up -d” to start it.
It is easy to maintain (60 or so lines of js for the nodejs git watcher, and approx. 100 lines of python for the celery task), and scales well (just need to add more celery workers in the docker-compose.yml).
Of course, all this (docker-compose.yml, nodejs service, python celery task) is versionned in github.
So far (a few months), this system has been performing very well, at scale, without any fuss, and is very fast. It is in line with the philosophy I try to apply everywhere: no over engineering, keep things simple, in docker containers.
See the full github repo here: https://github.com/francoisruty/fruty_docker-ci
Tech from the Front Line 
One thought on “Multistage Dockerfiles: do we still need CI Software?”