This is Part 1 of a multi-part series on integrating Rust into a large C codebase.
lib-common is a C library developed by Intersec . It provides core utilities (strings, containers, memory management), networking (HTTP, RPC, event loop), serialization through IOP (Intersec Object Packer, similar to Protocol Buffers), etc. It targets Linux environments and has been in active development for many years.
Read more
Cython is a programming language that makes writing C extensions for Python as easy as Python itself. It lets us write Python code that calls back and forth from and to C/C++ code natively, and compiles to efficient C code usable as a Python extension module.
At Intersec , we use Cython in lib-common , our open-source C library. One of its key components is IOPy, the Python binding for our IOP serialization framework (similar to Protocol Buffers). IOPy dynamically creates Python classes from IOP type definitions at DSO load time, and each class needs to hold a pointer to its C type descriptor for fast serialization.
Read more
In recent years, there has been significant progress in the field of artificial intelligence, especially with the development of lightweight models that can run locally and provide good results. These local AIs are interesting because they ensure data confidentiality, which is a crucial factor in our work environment.
As a software company, we use many digital tools to facilitate our work and collaboration among colleagues. Among these tools, we use Slack for internal communication and Redmine for project management.
Read more
Rive is a brand new animation tool and we tested it to try its innovative features that bring in interactivity while keeping good performance.
Today we’re using Lottie and we’re comparing it to Rive. Lottie is an After Effects plugin. It allows motion designs to be exported out of After Effects and used in-app with their own JS runtime, an NPM Library. While Lottie is built on top of After Effects, Rive is built from scratch with its own animation software allowing developers to add interactivity to their animations. As a result, we have lots of great exclusive features while getting 10 times smaller exported files and better in-app performance.
Read more
Text editors are essential tools for software developers. These editors provide many useful features to make developers’ lives easier and improve their productivity.
There is a wide variety of text editors. Moreover, each language has its own characteristics and each company/project may decide to adopt different coding rules. For all these reasons, it is difficult to offer a configuration suitable for everyone.
In this hackathon, we tried to help developers configure their text editors correctly for optimal performance. For this purpose, using the Language Server Protocol (LSP) seemed like a good option.
Read more
This project was part of the 12th Intersec Hackathon (0x0C). We had 2 days (31 hours) to develop and present our project. Our team consisted of Yohann Balawender and Benoît Mariat.
The purpose of our project was to explore the Web Speech API and to use it in an Intersec product to send verbal commands that would replace the use of a mouse, even for complex actions.
Read more
At Intersec, we have produced and installed multiple applications over the years, and installed them on multiple platforms (various OS) for many customers all over the world.
Today some of these applications are in a “maintenance only” mode (we no longer add features to them). However, we must keep the artifacts (mainly packages for Debian and Red Hat) currently installed on the customer’s premises.
We also regularly build new versions of our “active” products (to add new features and bugfixes), also on multiple OS. And we have continuous delivery pipelines which create “nightly” builds for our internal development and test platforms.
Read more
The previous article on QPOD was mainly about intercepting commands from the shell. The handling of these commands was not really described. This is the purpose of this article.
Before taking a look at how QPOD handles the execution of various programs, we need to introduce the QPOD environment. An environment defines a directory in which QPOD will handle program execution.
As QPOD is designed for building and running programs, it uses the git repository to define an environment location. This makes it easy to find the environment from any sub-directory with some simple git commands.
Read more
At Intersec, we recently launched our new alerting product, Trela. With this product, mayors can alert and communicate local information to their subscribers. And today, in order to easily send send information to people, we use mobile apps. That’s why we decided to enable this communication channel in our product. When we began this project, we had some requirements. The code has to be:
Expertise in location data is at the core of Intersec technologies. Retrieving and exploiting such data from cellular networks helps us address use cases in various fields like public safety, location-based advertising or smart cities and territories.
When dealing with raw locations retrieved at an antenna level within these networks, a central challenge is to infer when a device is staying in the same area or when it is moving. What might seem simple when we think about GNSS trajectories (for example, based on GPS coordinates) is not that easy when we have to handle antenna-based locations.
Read more
This hackathon subject is actually the continuation of work started in a previous hackathon under the name “Help me Please”: a system of interactive tours that guide the user through our products. Our solutions offer a large variety of possibilities, and proposing guidance would unlock their full potential to the end user.
Several conclusions were drawn from the “Help Me Please” experience:
From a user point of view, such tours are typically shown during first launch, which could be, but not always is, the perfect time to bug the user with a long tutorial. Another issue that derives from this is that, once completed or ignored, there is no way to come back to such a guide. “Help me Please” actually solved this problem by presenting a list of scenarios where the user could restart any tutorial from a list. However, we wondered if there were still ways to better integrate the tutorial, our standard documentation and, perhaps, other formats of guidance.
Read more
Our original goal for this hackathon this year was to speed up the web builds.
But, as you will see, this year is special, and we have not tackled one, but two subjects in the same hackathon!
Before the hackathon, our web build system was basically composed of webpack , with ts-loader to transpile TypeScript into JavaScript, fork-ts-checker-webpack-plugin to enable TypeScript type checking, and eslint for additional static analysis.
Read more
At Intersec we love hiring new talents, especially new C developers who can reinforce our teams.
And of course the candidates have to face a technical challenge so that we can assess their technical skills.
This is a typical step in the recruitment process that can become very time consuming for the reviewers and stressful for the candidates. Pimp My Recruitment is an initiative to help automatize this whole process, both for the candidate and the reviewer.
Read more
Currently, when we want to show the upcoming version of a product or new features for a demo, if these features are not already fully available on the code repository, and so existing in a packaged version of the product, it’s a bit painful.
We have to create a dedicated demo platform, by compiling the specific product locally, building packages and installing them manually on a dedicated Virtual Machine.
Read more
Like many other companies, we need to maintain our software for a long period of time. Usually, a version lasts several years (we have had some versions in production for more than 10 years now).
While we update our coding environment for new versions, we still need to be able to build and debug those older branches. This can become a problem when our new default system introduces a new tool version (e.g. a new compiler version that will introduce new build errors).
Read more
In the virtual world, hardware may feel abstract, nonetheless software still depends on physical infrastructure consuming resources where waste can take place.
Let’s look at datacenters, where the physical location is based on the price of electricity, then you have to keep them cool, placing data center underwater is one solution being explored.
There is also the high energy consumption of crypto mining but let’s focus on a simpler example, that app on your phone that consumes a lot of battery or memory with no apparent reason which ends up being removed by the annoyed user.
Read more
At Intersec we aim to provide the best user experience in our geo-oriented applications, where users need to understand a live situation at a glance and make the best decisions, based on accurate map data.
The way to achieve this goal often relies on providing just the right amount of contextual information and actions, keeping everything as simple as possible.
Users of Intersec products need to understand the geographic context:
Read more
This blog was originally hosted using WordPress, the famous CMS. It was quite an active site at the beginning, with a post every 2 months on average.
But then, the contribution rate dropped, and the blog began to be only used to relate the results of our yearly hackathon. As the site was a bit deserted, we forgot to update it, which is quite a fatal mistake when dealing with a WordPress site…
Read more
On the 23rd and 24th of January, an internal hackathon took place at Intersec. Our team “Laws of the Universe” took part in this hackathon, with the ambition of “testing” kepler.gl, an open-source solution of geodata viz and analysis.
More precisely, what we meant by “test” was a twofold objective:
To be honest, before the hackathon, our knowledge of kepler.gl was no more advanced than “Wow, this looks nice!” when browsing their website. Thankfully, our dream team was composed of two geodatascientists and two full stack developers, so we had all in hand to make it a success!
Read more
In our products, we use a home-made technology called QRRD (for Quick Round Robin Database) to store monitoring metrics (system CPU/memory monitoring, incoming event flows, …).
QRRD (which is written in C) was actively developed between 2009 and 2013, but we have not been investing in it since, so it has not evolved anymore. And even if this is a really great technology (especially in terms of scaling and performances), it has the following drawbacks:
Read more
At Intersec, we aim at maintaining a consistent style throughout all the code, depending on the programming language. For example, the C codebase is the most consistent, with our coding rules being enforced on code review. This enforcement, however, can lead to significant time loss and frustrations when patches must be repushed to be adapted to specific rules.
In the past, there has been several attempts at configuring auto-formatting tools. They were never fully satisfactory because several of our coding rules did not fit into the limited configuration options of these tools. This subject was born out of these attempts. However, instead of focusing on adapting the tools to our coding rules, we considered doing the opposite. What about adapting our coding rules (in particular, some of those peculiar rules) so that auto-formatting tools could be applied easily? After all, if some of our rules are too specific to exist in popular tools, maybe those rules cause more harm than improvement to our code.
Read more
The goal was to develop benchmarks on a few of our core technologies in lib-common, in order to:
The first step was to develop a benchmark library; the success criteria we established were the following (compared to the already existing benchmarks in our code base):
Read more
At Intersec, we love new technologies that can improve our working tasks, our code, and because it is fun! During Hackathon 0x09, I tested the possibility to use BPF for tracing and debugging our C codebase.
In the beginning, BPF was a technology used for packet
filtering
1.
For example, when using the command tcpdump -i lo arp, BPF is
used to filter ARP packets on the loopback interface. BPF has since been
enhanced. The BPF instructions set was extended in Linux kernel 3.15 and was
called “extended BPF” or eBPF. Today “BPF” could be defined as the technology
that uses a set of tools to write and execute eBPF code.
During summer 2014 we organized our first hackathon.
The rules are simple and are still up to date: The subjects are suggested by
anyone in the company and there is no defined framework nor limit
on what they can be although they often fall in the same categories,
I’ll come back to that.
Anyone can show interest in any of the proposed subjects and then work on it. In practice though, participants are mostly people from technical teams. Teams are formed and work on their topic from Thursday morning until Friday 4pm. They can even work late or at night if they wish to.
Read more
To investigate a bug, people will often resort to two methods once the bug has been reproduced:
The second solution is much more powerful but also quite limited in the types of bug it can handle:
For those complex bugs, the investigation will thus often end as a combination of traces + placed assertion to generate core files and being able to investigate with a debugger the state of the program.
Read more
This post is an introduction to a useful tool here at Intersec, a tool that we call IOP: the Intersec Object Packer.
IOP is our take on the IDL approach. It is a method to serialize structured data to use in our communication protocols or data storage technologies. It is used to transmit data over the network in a safe manner, to exchange data between different programming languages or to provide a generic interface to store (and load) C data on disk. IOP provides data integrity checking and backward-compatibility.
Read more
In modern software engineering, you quickly reach the point where one process cannot handle all the tasks by itself. For performance, maintainability or reliability reasons you do have to write multi-process programs. One can also reach the point where he wants its softwares to speak to each-other. This situation raises the question: how will my processes “talk” to each other?
If you already have written such programs in C, you are probably familiar with the network sockets concept. Those are handy (at least compared to dealing with the TCP/IP layer yourself): it offers an abstraction layer and lets you have endpoints for sending and receiving data from a process to another. But quickly some issues arise:
Read more
We do write complex software, and like everyone doing so, we need a way to structure our source code. Our choice was to go modular. A module is a singleton that defines a feature or a set of related feature and exposes some APIs in order for other modules to access these features. Everything else, including the details of the implementation, is private. Examples of modules are the RPC layer, the threading library, the query engine of a database, the authentication engine, … Then, we can compose the various daemons of our application by including the corresponding modules and their dependencies.
Read more
Jeez! It’s been a year already! A year working at Intersec… Time goes by so quickly! A year ago, I was thinking about giving up the idea of appending “PhD” to my name: I quit the graduate program I was in after only 10 months. Before that I had been working for a year at this other company called… wait! Was it Intersec already? Oh my God! It was!
Intersec? Really? Is that even a name? Could it be possible that some guy picked such an unsexy name for his company? This is the first train of thought that went through my brain after ending my nth phone conversation with a head hunter. An interview was scheduled a couple of days later.
Read more
Back in 2009, Snow Leopard was quite an exciting OS X release. It
didn’t focus on new user-visible features but instead introduced a
handful of low level technologies. Two of those technologies
Grand Central Dispatch (a.k.a. GCD)
and OpenCL
were designed to help developers benefit from the new computing power
of modern computer architectures: multicore processors for the former
and GPUs for the latter.
Alongside the GCD engine came a C language extension called
blocks.
Blocks are the C-based flavor of what is commonly called
a closure:
a callable object that captures the context in which it was created.
The syntax for blocks is very similar to the one used for functions,
with the exception that the pointer star is * replaced by
a caret ^. This allows inline definition of callbacks which
often can help improving the readability of the code.
This series of articles describes some of the best realisations made by Intersec R&D team during the 2-day Hackathon that took place on the 3rd and 4th of July.
The goal had been set a day or two prior to the beginning of the hackathon: we were hoping to make Gerrit better at recommending relevant reviewers for a given commit. To those who haven’t heard of it, Gerrit is a web-based code review system. It is a nifty Google-backed open-source project evolving amid an active community of users. We have been using this product here at Intersec since 2011 and some famous software projects also rely heavily on it for their development process.
Read more
The most used custom allocators at Intersec are the FIFO and the Stack
allocators, detailed in
a previous article.
The stack allocator is extremely convenient, thanks to the
t_scope macro, and the FIFO is well fitted to some of our use
cases, such as inter-process communication. It is thus important for these
allocators to be optimized extensively.
We are two interns at Intersec, and our objective for this 6 week
internship was to optimize these allocators as far as possible. Optimizing
an allocator can have several meanings: it can be in terms of memory overhead,
resistance to contention, performance… As the FIFO allocator is designed to work
in single threaded environments, and the t_stack is thread local,
we will only cover performance and memory overhead.
This series of articles describes some of the best realisations made by Intersec R&D team during the 2-day Hackathon that took place on the 3rd and 4th of July.
As testers, we spend a lot of time working on
behave, our test automation
framework1.
Our test framework is a great tool, but it takes a lot of time starting
and initializing the product, running tests one by one.
Telecom data is a rich source of information for many purposes, ranging from urban planning (Toole et al., 2012), human mobility patterns (Ficek and Kencl, 2012; Gambs et al., 2011), points of interest detection (Vieira et al., 2010), epidemic spread modeling (Lima et al., 2013), community detection (Morales et al., 2013) disaster planning (Pulse, 2013) and social interactions (Eagle et al., 2013).
One common task for these applications is to identify dense areas where many users stay for a significant time (activity zones), the regions relaying theses activity zones (transit zones) as well as the interaction between identified activity zones. Thus, in the present article we will identify activity and transit zones to monitor and predict the activity levels in the telecom operators network based on the SMS and calls input/output activity levels issued from the Telecom Italia Big Data Challenge. The results of the present study could be directly applied to:
Read more
In the third post of the memory series, we briefly explained locality and why it is an important principle to keep in mind while developing a memory-intensive program. This new post is going to be more concrete and explains what actually happens behind the scene in a very simple example.
This post is a follow-up to a recent interview with a (brilliant) candidate1. As a subsidiary question, we resented him with the following two structure definitions:
Read more
Here we are! We spent 4 articles explaining what memory is, how to deal with it and what are the kind of problems you can expect from it. Even the best developers write bugs. A commonly accepted estimation seems to be around of few tens of bugs per thousand of lines of code, which is definitely quite huge. As a consequence, even if you proficiently mastered all the concepts covered by our articles, you’ll still probably have a few memory-related bugs.
Read more
malloc() not the one-size-fits-all allocatormalloc() is extremely convenient because it is generic.
It does not make any assumptions about the context of the allocation and the
deallocation. Such allocators may just follow each other, or be
separated by a whole job execution. They may take place in the same
thread, or not… Since it is generic, each allocation is different from
each other, meaning that long term allocations share the same pool as
short term ones.
In the previous articles we dealt with memory classification and analysis from an outer point of view. We saw that memory can be allocated in different ways with various properties. In the remaining articles of the series we will take a developer point of view.
At Intersec we write all of our software in C, which means that we are constantly dealing with memory management. We want our developers to have a solid knowledge of the various existing memory pools. In this article we will have an overview of the main sources of memory available to C programmers on Linux. We will also see some rules of memory management that will help you keep your program correct and efficient.
Read more
In the previous article , we introduced a way to classify the memory a process reclaimed. We used 4 quadrants using two axis: private/shared and anonymous/file-backed. We also evoked the complexity of the sharing mechanism and the fact that all memory is basically reclaimed to the kernel.
Everything we talked about was virtual. It was all about reservation of memory addresses, but a reserved address is not always immediately mapped to physical memory by the kernel. Most of the time, the kernel delays the actual allocation of physical memory until the time of the first access (or the time of the first write in some cases)… and even then, this is done with the granularity of a page (commonly 4KiB). Moreover, some pages may be swapped out after being allocated, that means they get written to disk in order to allow other pages to be put in RAM.
Read more
At Intersec we chose the C programming language because it gives us a full control on what we’re doing, and achieves a high level of performances. For many people, performance is just about using as few CPU instructions as possible. However, on modern hardware it’s much more complicated than just CPU. Algorithms have to deal with memory, CPU, disk and network I/Os… Each of them adds to the cost of the algorithm and each of them must be properly understood in order to guarantee both the performance and the reliability of the algorithm.
Read more
At Intersec, technology matters…Because it’s the core of our business, we aim to provide our clients with the most innovative and disruptive technological solutions. We do not believe in the benefits of reusing and staking external software bricks when developing our products. Our software is built in C language under Linux, with PHP/JavaScript for the web interfaces and it is continuously improved to fit to the simple principle we believe in:
Read more