Automating Linux Packaging for KrakenD API Gateway

Everyone loves reaching the maximum number of available platforms for their software but this usually comes at a cost. In this article we will explain how we generated our packaging for Linux in an automated fashion, being faithful to our DevOps dogma.

Dockerize all the things!

Our dev team uses both Linux and MacOS X in desktop machines and KrakenD has been running and behaving in the same way in all the platforms because from day one we decided to run everything on Docker. We use containers for all the software we produce and this is still true for the rest of our tooling (as what we are going to show today).

When it comes to generating RPM or DEB packages, choosing Docker ensures that anyone building or compiling a package will generate a consistent output.

Having Docker as the platform to work on, let’s see how to build the packages.

Building packages the easy way using FPM

For those who don’t know fpm, this is the key piece for the package generation. According to their own intro:

The goal of fpm is to make it easy and quick to build packages

fpm is a tool written in Ruby that allows you to create packages for multiple platforms in a very easy way. Examples of the packages you can create are deb, rpm, tar and even Mac OS X .pkg, solaris, freebsd or pacman (ArchLinux).

Let’s get started by creating the fpm builder with a Dockerfile to generate the versions deb and rpm, the ones we were most interested in.

The Dockerfile for Debian/Ubuntu:

FROM ubuntu:16.04
LABEL maintainer="[email protected]"

RUN apt-get update && apt-get install -y \
	ruby-dev \
	gcc \
	make \
	ruby \
	&& rm -rf /var/lib/apt/lists/*

RUN gem install fpm -v 1.9.3 --no-ri --no-rdoc

VOLUME [ "/tmp/fpm" ]
WORKDIR /tmp/fpm

ENTRYPOINT [ "/usr/local/bin/fpm" ]
CMD [ "--help" ]

The Dockerfile for CentOS/Rhel:

FROM centos:7
LABEL maintainer="[email protected]"

RUN yum install -y \
	ruby-devel \
	gcc \
	make \
	ruby \
	rpm-build \
	rpm-sign && yum clean all

RUN gem install fpm -v 1.9.3 --no-ri --no-rdoc

VOLUME [ "/tmp/fpm" ]
WORKDIR /tmp/fpm

ENTRYPOINT [ "/usr/local/bin/fpm" ]
CMD [ "--help" ]

Then build it and run it:

$ docker run --rm -it fpm:deb
Intro:

  This is fpm version 1.9.3

Notice that the FROM in each one uses a different OS (because fpm still needs rpmbuild, dpkg-deb and other tools).

Now the container is ready to package anything.

Source code:

• Clone and build docker-fpm repository
• Or run the docker fpm containers from Docker Hub.

Sign packages using PGP

If you want to distribute packages you’ll need to sign them using your PGP key. To do so you need to mount your .pgp directory in the container as a volume, as well as the rpmmacros configuration so the process has everything it needs.

	docker run --rm -it -v "${PWD}/rpmmacros:/root/.rpmmacros" -v $HOME/.gnupg:/root/.gnupg \
		-v "${PWD}:${DOCKER_WDIR}" -w ${DOCKER_WDIR} ${DOCKER_FPM}:rpm -t rpm ${RPM_OPTS} \
		--iteration ${RELEASE}.el7 \
		-C skel/el7 \
		${FPM_OPTS}

The example uses some vars that we haven’t seen so far. Keep reading…

Write a Makefile

Unless your are OK with having an environment that suffers the diogenes syndrome, the next problem you want to face is managing what files go in which package version and leaving the house clean after compiling. How convenient is a Makefile to get that!

Makefile variables, example:

VERSION := 0.3.9
PKGNAME := krakend
LICENSE := Apache 2.0
VENDOR=
URL := https://www.krakend.io
RELEASE := 0
USER := krakend
ARCH := amd64
DESC := High performance API gateway. Aggregate, filter, manipulate and add middlewares
MAINTAINER := Daniel Ortiz <[email protected]>
DOCKER_WDIR := /tmp/fpm
DOCKER_FPM := fpm

Then every specific option for fpm is also added to the Makefile:

FPM_OPTS=-s dir -v $(VERSION) -n $(PKGNAME) \
  --license "$(LICENSE)" \
  --vendor "$(VENDOR)" \
  --maintainer "$(MAINTAINER)" \
  --architecture $(ARCH) \
  --url "$(URL)" \
  --description  "$(DESC)" \
	--config-files etc/ \
  --verbose

DEB_OPTS= -t deb --deb-user $(USER) \
	--before-remove scripts/prerm.deb \
  --after-remove scripts/postrm.deb \
	--before-install scripts/preinst.deb

DEB_INIT=--deb-init krakend.init

RPM_OPTS =--rpm-user $(USER) \
	--before-install scripts/preinst.rpm \
	--before-remove scripts/prerm.rpm \
  --after-remove scripts/postrm.rpm \
  --rpm-sign

Now all the options and variables are controlled inside the Makefile.

Pattern-specific variables

A key of success are the pattern-specific variables present in the Makefile. They are used to define a kind of template that can be reused many times (as a function) but with different declarations.

A single platform would be easy to manage: one config file, a couple of scripts (pre and post install) and the systemd configuration. That’s it.

For us the problem came when we wanted to generate packages for old versions of Ubuntu/Debian and CentOS/RHEL. Some distributions were using upstart while others created scripts in init.d or needed custom scripts for pre and post installations. Differences never ended.

The use of the pattern-specific variables fixes in a simple way the problem (and using something it’s been there for years), this is how we completed the Makefile.

We defined all the files that might be needed during the package creation:

skel/%/etc/init/krakend.conf: krakend.conf
	mkdir -p "$(dir $@)"
	cp krakend.conf "$@"

skel/%/etc/init.d/krakend: krakend.init
	mkdir -p "$(dir $@)"
	cp krakend.init "$@"

When declaring the file using the full path, the directory will be created and the file will be copied inside, generating this way the skeleton for that specific version. In addition, it will save you from having to manually maintain that skel, because it is generated every time is needed, copying the “latest” available versions of them.

.PHONY: ubuntu-trusty
ubuntu-trusty: skel/ubuntu-trusty/usr/bin/krakend
ubuntu-trusty: skel/ubuntu-trusty/etc/krakend/krakend.json
ubuntu-trusty: skel/ubuntu-trusty/etc/krakend/service.yml
ubuntu-trusty: skel/ubuntu-trusty/etc/init.d/krakend
ubuntu-trusty: skel/ubuntu-trusty/etc/init/krakend.conf
	docker run --rm -it -v "${PWD}:${DOCKER_WDIR}" -w ${DOCKER_WDIR} ${DOCKER_FPM}:deb -t deb ${DEB_OPTS} \
		--iteration ${RELEASE}.ubuntu-trusty \
		-C skel/ubuntu-trusty \
		${DEB_INIT} \
		${FPM_OPTS}


.PHONY: ubuntu-xenial
ubuntu-xenial: skel/ubuntu-xenial/usr/bin/krakend
ubuntu-xenial: skel/ubuntu-xenial/etc/krakend/krakend.json
ubuntu-xenial: skel/ubuntu-xenial/etc/krakend/service.yml
	docker run --rm -it -v "${PWD}:${DOCKER_WDIR}" -w ${DOCKER_WDIR} ${DOCKER_FPM}:deb -t deb ${DEB_OPTS} \
		--iteration ${RELEASE}.ubuntu-xenial \
		--deb-systemd krakend.service \
		-C skel/ubuntu-xenial \
		${FPM_OPTS}

Testing your packages

The last step before releasing to the world the packages just created is to test they work properly. There are many ways and strategies to do that and we are not going to show you how, but for us it worked to run a docker container with the OS version we wanted to try and install inside the generated package.

With this done, it is very easy to check that the pre and post installation scripts worked correctly as well as the installation and the service operation.

We did a simple shell script (not being able to invest more time in something elegant and reusable) and a Dockerfile template where the package gets copied and a new container is generated tagged with the version of the package. By specifying the package version as a tag in the docker image, executing docker images will list all the generated packages ready to test!

Dockerfile testing:

FROM ubuntu:16.04
maintainer [email protected]

ARG debfile
ADD $debfile /
RUN dpkg -i /$debfile

EXPOSE 8080

CMD [ "krakend", "run", "-d" ]

Testing script:

#!/bin/sh

DEB=$1
TARGET=$2
VERSION=$3
cp $DEB tests/ubuntu-xenial/
docker build --build-arg debfile=$DEB -t test_${TARGET}_${VERSION} tests/ubuntu-xenial
rm -f tests/ubuntu-xenial/$DEB

Makefile:

.PHONY: ubuntu-xenial
ubuntu-xenial: skel/ubuntu-xenial/usr/bin/krakend
ubuntu-xenial: skel/ubuntu-xenial/etc/krakend/krakend.json
ubuntu-xenial: skel/ubuntu-xenial/etc/krakend/service.yml
	docker run --rm -it -v "${PWD}:${DOCKER_WDIR}" -w ${DOCKER_WDIR} ${DOCKER_FPM}:deb -t deb ${DEB_OPTS} \
		--iteration ${RELEASE}.ubuntu-xenial \
		--deb-systemd krakend.service \
		-C skel/ubuntu-xenial \
		${FPM_OPTS}
	tests/ubuntu-xenial/test.sh ${PKGNAME}_${VERSION}-${RELEASE}.ubuntu-xenial_${ARCH}.deb ubuntu-xenial ${VERSION}

Conclusion

We have seen through several examples how we automated the generation of package files for KrakenD. You can quickly adapt this scripts and containers to your own application and start producing packages in a more automated way. With the combination of the provided Makefile and fpm you will be able to distribute your application to several distributions.

Enjoy!