The WordPress years

This blog was originally hosted using WordPress, the famous CMS. It was quite an active site at the beginning, with a post every 2 months on average.

But then, the contribution rate dropped, and the blog began to be only used to relate the results of our yearly hackathon. As the site was a bit deserted, we forgot to update it, which is quite a fatal mistake when dealing with a WordPress site…

On the 23rd and 24th of January, an internal hackathon took place at Intersec. Our team “Laws of the Universe” took part in this hackathon, with the ambition of “testing” kepler.gl, an open-source solution of geodata viz and analysis.

More precisely, what we meant by “test” was a twofold objective:

• See if we could build nice viz based on the type of data commonly processed by our solutions
• Ideally, integrate them directly in our products, to demonstrate the feasibility of an industrialized solution based on this technology

To be honest, before the hackathon, our knowledge of kepler.gl was no more advanced than “Wow, this looks nice!” when browsing their website. Thankfully, our dream team was composed of two geodatascientists and two full stack developers, so we had all in hand to make it a success!

Objective

In our products, we use a home-made technology called QRRD (for Quick Round Robin Database) to store monitoring metrics (system CPU/memory monitoring, incoming event flows, …).

QRRD (which is written in C) was actively developed between 2009 and 2013, but we have not been investing in it since, so it has not evolved anymore. And even if this is a really great technology (especially in terms of scaling and performances), it has the following drawbacks:

At Intersec, we aim at maintaining a consistent style throughout all the code, depending on the programming language. For example, the C codebase is the most consistent, with our coding rules being enforced on code review. This enforcement, however, can lead to significant time loss and frustrations when patches must be repushed to be adapted to specific rules.

In the past, there has been several attempts at configuring auto-formatting tools. They were never fully satisfactory because several of our coding rules did not fit into the limited configuration options of these tools. This subject was born out of these attempts. However, instead of focusing on adapting the tools to our coding rules, we considered doing the opposite. What about adapting our coding rules (in particular, some of those peculiar rules) so that auto-formatting tools could be applied easily? After all, if some of our rules are too specific to exist in popular tools, maybe those rules cause more harm than improvement to our code.

The goal was to develop benchmarks on a few of our core technologies in lib-common, in order to:

• Be able to compare the performances of our custom implementations with standard implementations.
• Be able to add automated tests on performance (e.g. adding non-regression tests to ensure that changes which seem to be harmless do not worsen performance).

Benchmark library

The first step was to develop a benchmark library; the success criteria we established were the following (compared to the already existing benchmarks in our code base):

At Intersec, we love new technologies that can improve our working tasks, our code, and because it is fun! During Hackathon 0x09, I tested the possibility to use BPF for tracing and debugging our C codebase.

What is BPF?

In the beginning, BPF was a technology used for packet filtering ¹. For example, when using the command tcpdump -i lo arp, BPF is used to filter ARP packets on the loopback interface. BPF has since been enhanced. The BPF instructions set was extended in Linux kernel 3.15 and was called “extended BPF” or eBPF. Today “BPF” could be defined as the technology that uses a set of tools to write and execute eBPF code.

During summer 2014 we organized our first hackathon.
The rules are simple and are still up to date: The subjects are suggested by anyone in the company and there is no defined framework nor limit on what they can be although they often fall in the same categories, I’ll come back to that.

Anyone can show interest in any of the proposed subjects and then work on it. In practice though, participants are mostly people from technical teams. Teams are formed and work on their topic from Thursday morning until Friday 4pm. They can even work late or at night if they wish to.

Introduction

To investigate a bug, people will often resort to two methods once the bug has been reproduced:

• Add traces in the code.
• Use a debugger to instrument the execution of the bug.

The second solution is much more powerful but also quite limited in the types of bug it can handle:

• If the instrumentation of the execution goes too far, it is impossible to go in reverse. A new execution is needed, with new breakpoints or watches, but also new addresses, variable values…
• If the reproduction is random, having to restart the process and reproduce the bug again can be very time consuming.
• Instrumenting the bug with breakpoints can render the bug non-reproducible.

For those complex bugs, the investigation will thus often end as a combination of traces + placed assertion to generate core files and being able to investigate with a debugger the state of the program.

This post is an introduction to a useful tool here at Intersec, a tool that we call IOP: the Intersec Object Packer.

IOP is our take on the IDL approach. It is a method to serialize structured data to use in our communication protocols or data storage technologies. It is used to transmit data over the network in a safe manner, to exchange data between different programming languages or to provide a generic interface to store (and load) C data on disk. IOP provides data integrity checking and backward-compatibility.

About multi-process programming

In modern software engineering, you quickly reach the point where one process cannot handle all the tasks by itself. For performance, maintainability or reliability reasons you do have to write multi-process programs. One can also reach the point where he wants its softwares to speak to each-other. This situation raises the question: how will my processes “talk” to each other?

If you already have written such programs in C, you are probably familiar with the network sockets concept. Those are handy (at least compared to dealing with the TCP/IP layer yourself): it offers an abstraction layer and lets you have endpoints for sending and receiving data from a process to another. But quickly some issues arise: