Posts

Winning a hackathon with kepler.gl

On the 23rd and 24th of January, an internal hackathon took place at Intersec. Our team “Laws of the Universe” took part in this hackathon, with the ambition of “testing” kepler.gl, an open-source solution of geodata viz and analysis. More precisely, what we meant by “test” was a twofold objective: See if we could build nice viz based on the type of data commonly processed by our solutions Ideally, integrate them directly in our products, to demonstrate the feasibility of an industrialized solution based on this technology To be honest, before the hackathon, our knowledge of kepler.

Read more

Hackathon 0x09 – Monitoring with Prometheus/Grafana

Objective In our products, we use a home-made technology called QRRD (for Quick Round Robin Database) to store monitoring metrics (system CPU/memory monitoring, incoming event flows, …). QRRD (which is written in C) was actively developed between 2009 and 2013, but we have not been investing in it since, so it has not evolved anymore. And even if this is a really great technology (especially in terms of scaling and performances), it has the following drawbacks:

Read more

Introducing auto-formatting in an existing codebase

At Intersec, we aim at maintaining a consistent style throughout all the code, depending on the programming language. For example, the C codebase is the most consistent, with our coding rules being enforced on code review. This enforcement, however, can lead to significant time loss and frustrations when patches must be repushed to be adapted to specific rules. In the past, there has been several attempts at configuring auto-formatting tools. They were never fully satisfactory because several of our coding rules did not fit into the limited configuration options of these tools.

Read more

Hackathon 0x09 – lib-common benchmarks

The goal was to develop benchmarks on a few of our core technologies in lib-common, in order to: Be able to compare the performances of our custom implementations with standard implementations. Be able to add automated tests on performance (e.g. adding non-regression tests to ensure that changes which seem to be harmless do not worsen performance). Benchmark library The first step was to develop a benchmark library; the success criteria we established were the following (compared to the already existing benchmarks in our code base):

Read more

Hackathon 0x09 – eBPF

At Intersec, we love new technologies that can improve our working tasks, our code, and because it is fun! During Hackathon 0x09, I tested the possibility to use BPF for tracing and debugging our C codebase. What is BPF? In the beginning, BPF was a technology used for packet filtering 1. For example, when using the command tcpdump -i lo arp, BPF is used to filter ARP packets on the loopback interface.

Read more

Hackathon – 5 years later

During summer 2014 we organized our first hackathon. The rules are simple and are still up to date: The subjects are suggested by anyone in the company and there is no defined framework nor limit on what they can be although they often fall in the same categories, I’ll come back to that. Anyone can show interest in any of the proposed subjects and then work on it. In practice though, participants are mostly people from technical teams.

Read more

Improved debugging with rr

Introduction To investigate a bug, people will often resort to two methods once the bug has been reproduced: Add traces in the code. Use a debugger to instrument the execution of the bug. The second solution is much more powerful but also quite limited in the types of bug it can handle: If the instrumentation of the execution goes too far, it is impossible to go in reverse.

Read more

Intersec Object Packer Part 1 : the basics

This post is an introduction to a useful tool here at Intersec, a tool that we call IOP: the Intersec Object Packer. IOP is our take on the IDL approach. It is a method to serialize structured data to use in our communication protocols or data storage technologies. It is used to transmit data over the network in a safe manner, to exchange data between different programming languages or to provide a generic interface to store (and load) C data on disk.

Read more

Middleware (or how do our processes speak to one-another)

About multi-process programming In modern software engineering, you quickly reach the point where one process cannot handle all the tasks by itself. For performance, maintainability or reliability reasons you do have to write multi-process programs. One can also reach the point where he wants its softwares to speak to each-other. This situation raises the question: how will my processes “talk” to each other? If you already have written such programs in C, you are probably familiar with the network sockets concept.

Read more

C Modules

We do write complex software, and like everyone doing so, we need a way to structure our source code. Our choice was to go modular. A module is a singleton that defines a feature or a set of related feature and exposes some APIs in order for other modules to access these features. Everything else, including the details of the implementation, is private. Examples of modules are the RPC layer, the threading library, the query engine of a database, the authentication engine, … Then, we can compose the various daemons of our application by including the corresponding modules and their dependencies.

Read more

One year at Intersec (actually two)

Jeez! It’s been a year already! A year working at Intersec… Time goes by so quickly! A year ago, I was thinking about giving up the idea of appending “PhD” to my name: I quit the graduate program I was in after only 10 months. Before that I had been working for a year at this other company called… wait! Was it Intersec already? Oh my God! It was! Intersec? Really?

Read more

Blocks rewriting with clang

Introduction Back in 2009, Snow Leopard was quite an exciting OS X release. It didn’t focus on new user-visible features but instead introduced a handful of low level technologies. Two of those technologies Grand Central Dispatch (a.k.a. GCD) and OpenCL were designed to help developers benefit from the new computing power of modern computer architectures: multicore processors for the former and GPUs for the latter. Alongside the GCD engine came a C language extension called blocks.

Read more

Hackathon 0x1 – Pimp my Review or the Epic Birth of a Gerrit Plugin

This series of articles describes some of the best realisations made by Intersec R&D team during the 2-day Hackathon that took place on the 3rd and 4th of July. The goal had been set a day or two prior to the beginning of the hackathon: we were hoping to make Gerrit better at recommending relevant reviewers for a given commit. To those who haven’t heard of it, Gerrit is a web-based code review system.

Read more

Memory – Part 6: Optimizing the FIFO and Stack allocators

Introduction The most used custom allocators at Intersec are the FIFO and the Stack allocators, detailed in a previous article. The stack allocator is extremely convenient, thanks to the t_scope macro, and the FIFO is well fitted to some of our use cases, such as inter-process communication. It is thus important for these allocators to be optimized extensively. We are two interns at Intersec, and our objective for this 6 week internship was to optimize these allocators as far as possible.

Read more

Hackathon 0x1 – Interactive mode in Behave

This series of articles describes some of the best realisations made by Intersec R&D team during the 2-day Hackathon that took place on the 3rd and 4th of July. Presentation of the Project As testers, we spend a lot of time working on behave, our test automation framework1. Our test framework is a great tool, but it takes a lot of time starting and initializing the product, running tests one by one.

Read more

DAZIO: Detecting Activity Zones based on Input/Output sms and calls activity for geomarketing and trade area analysis

Introduction Telecom data is a rich source of information for many purposes, ranging from urban planning (Toole et al., 2012), human mobility patterns (Ficek and Kencl, 2012; Gambs et al., 2011), points of interest detection (Vieira et al., 2010), epidemic spread modeling (Lima et al., 2013), community detection (Morales et al., 2013) disaster planning (Pulse, 2013) and social interactions (Eagle et al., 2013). One common task for these applications is to identify dense areas where many users stay for a significant time (activity zones), the regions relaying theses activity zones (transit zones) as well as the interaction between identified activity zones.

Read more

More about locality

In the third post of the memory series, we briefly explained locality and why it is an important principle to keep in mind while developing a memory-intensive program. This new post is going to be more concrete and explains what actually happens behind the scene in a very simple example. This post is a follow-up to a recent interview with a (brilliant) candidate1. As a subsidiary question, we resented him with the following two structure definitions:

Read more

Memory – Part 5: Debugging Tools

Introduction Here we are! We spent 4 articles explaining what memory is, how to deal with it and what are the kind of problems you can expect from it. Even the best developers write bugs. A commonly accepted estimation seems to be around of few tens of bugs per thousand of lines of code, which is definitely quite huge. As a consequence, even if you proficiently mastered all the concepts covered by our articles, you’ll still probably have a few memory-related bugs.

Read more

Memory - Part 4: Intersec’s custom allocators

malloc() not the one-size-fits-all allocator malloc() is extremely convenient because it is generic. It does not make any assumptions about the context of the allocation and the deallocation. Such allocators may just follow each other, or be separated by a whole job execution. They may take place in the same thread, or not… Since it is generic, each allocation is different from each other, meaning that long term allocations share the same pool as short term ones.

Read more

Memory - Part 3: Managing memory

Developer point of view In the previous articles we dealt with memory classification and analysis from an outer point of view. We saw that memory can be allocated in different ways with various properties. In the remaining articles of the series we will take a developer point of view. At Intersec we write all of our software in C, which means that we are constantly dealing with memory management. We want our developers to have a solid knowledge of the various existing memory pools.

Read more

Memory - Part 2: Understanding Process memory

From Virtual to Physical In the previous article , we introduced a way to classify the memory a process reclaimed. We used 4 quadrants using two axis: private/shared and anonymous/file-backed. We also evoked the complexity of the sharing mechanism and the fact that all memory is basically reclaimed to the kernel. Everything we talked about was virtual. It was all about reservation of memory addresses, but a reserved address is not always immediately mapped to physical memory by the kernel.

Read more

Memory - Part 1: Memory Types

Introduction At Intersec we chose the C programming language because it gives us a full control on what we’re doing, and achieves a high level of performances. For many people, performance is just about using as few CPU instructions as possible. However, on modern hardware it’s much more complicated than just CPU. Algorithms have to deal with memory, CPU, disk and network I/Os… Each of them adds to the cost of the algorithm and each of them must be properly understood in order to guarantee both the performance and the reliability of the algorithm.

Read more

First post

At Intersec, technology matters…Because it’s the core of our business, we aim to provide our clients with the most innovative and disruptive technological solutions. We do not believe in the benefits of reusing and staking external software bricks when developing our products. Our software is built in C language under Linux, with PHP/JavaScript for the web interfaces and it is continuously improved to fit to the simple principle we believe in:

Read more