Maintaining old software on new OS

Like many other companies, we need to maintain our software for a long period of time. Usually, a version lasts several years (we have had some versions in production for more than 10 years now).

While we update our coding environment for new versions, we still need to be able to build and debug those older branches. This can become a problem when our new default system introduces a new tool version (e.g. a new compiler version that will introduce new build errors).

We could of course keep some older systems available or use a virtual machine, and some of us do this. But this is not really convenient as we need to connect to a different computer.

Using a container is a better solution, but it still has some drawbacks as the container needs to be started every time we need it.

QPOD

During the last hackathon, we implemented a new tool to make it easier to maintain those old versions: QPOD. The goal is to work using the same workflow as on new branches (or older systems). The container environment is launched automatically when it is needed.

QPOD intercepts all commands launched in a shell environment and, if needed, redirects them to be launched in a container environment. The redirection is handled by a qpod program.

As we may need different containers depending on the branch, each git repository is linked to a specific container environment. The environment associated to a git repository is saved in a configuration file along with the options to be set when launching the container (mounted filesystem, environment variable, ...).

Command interception

In order to redirect commands to the container, we first need to intercept them. We first tried to write a linux kernel module that would reimplement the exec syscall. But we ran into issues with this approach:

• We had no prior experience with kernel development and it quickly became clear that we did not know enough about it (especially on memory management).
• Documentation on the subject was hard to find and most of what we could find was really old.

As it was clear that we could not really make it work in the allocated time, we decided to use a preload library that would reimplement libc exec functions.

Preload library is a feature of the glibc. Using the `LD_PRELOAD` environment variable, a library can be loaded automatically and replace some functions.

A simple example of a preload library that would replace the exec syscall looks like this:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>

static int (*orig_execve)(const char *file, char *const argv[],
                          char *const envp[]);

int execve(const char *pathname, char *const argv[], char *const envp[])
{
    if (!orig_execve) {
        orig_execve = dlsym(RTLD_NEXT, "execve");
    }

    printf("replaced execve\n");
    return orig_execve(pathname, argv, envp);
}

Then, we just need to compile this as a shared library and we can use it:

$ gcc -shared -ldl -fPIC -o test.so test.c
$ LD_PRELOAD=/home/guillaume/test.so /bin/ls /
bin   dev  home  lib64	     media  opt   root	sbin  sys  usr
boot  etc  lib	 lost+found  mnt    proc  run	srv   tmp  var
$ LD_PRELOAD=/home/guillaume/test.so /bin/sh
$ ls /
replaced execve
bin   dev  home  lib64	     media  opt   root	sbin  sys  usr
boot  etc  lib	 lost+found  mnt    proc  run	srv   tmp  var

As you may have noticed, the LD_PRELOAD is not applied directly. We need to launch a shell that launches the command. This is the main drawback of this method.

The QPOD preload library is a bit more complex as we do not want to redirect all commands to the container. So our library filters out commands if they are not within a list of directories set to be watched.

Launching container

Once the command has been intercepted and redirected to QPOD, this one tries to identify the environment (using the git repository). If no environment can be found, the command is processed normally.

QPOD also checks if the command needs to be launched in a container. All commands whose path is inside an environment is launched in a container. In other cases, QPOD filters most commands and only launches some specific ones. Mainly commands associated with building the product (waf, make, ...) or debugging it (gdb).

Once an environment is found, the associated container is launched with the command to execute.

Environment is defined by a configuration file that specifies the container image, mounted directory, etc.

Usage

We need to log into the image registry:

$ podman login --username $USER registry.intersec.com

If not done previously, we need to inform QPOD of the new git repository (mmsx-2018.3-centos7 is the name of the environment we want to use).

$ qpod init mmsx-2018.3-centos7

Once the initialization is done, we only need to launch the qpod-shell. The qpod-shell is just a wrapper to the user shell that sets up the environment for QPOD. And now the container is used transparently:

$ qpod-shell
launching /bin/bash in qpod mode (using qpod in /home/guillaume/dev/qpod)
$ pwd
/home/guillaume/dev/2018.3
$ make
... start configure
checking build-system ...
make: Entering directory `/qpod/.build-default-iltp016.corp'

The only trace that shows that we are using the container environment, is that the build system is using path starting with `/qpod`.