Our original goal for this hackathon this year was to speed up the web builds.
But, as you will see, this year is special, and we have not tackled one, but two subjects in the same hackathon!
Speed up web builds using esbuild
It takes around 1m30s and 1456.5 MiB of memory to compile the web part of one of our products.
To be faster, simply do less!
At Intersec, we don’t always do some web development. In those cases, we don’t need to do the type checking and static analysis.
So a quick win was to add the handling of a new environment variable
NO_WWW_CHECK=1 to our build system to skip
We don’t sacrifice quality for speed here as our CI system still launches ts-checker and eslint on every commit.
It now takes around 40s and 1342.4 MiB of memory to compile while hardly changing anything!
The next step was then to replace ts-loader as our TypeScript transpiler.
There are two candidates that can be used with webpack without changing our entire web build system, swc with swc-loader , and esbuild with esbuild-loader .
Replacing ts-loader by swc-loader was very easy. There were only two minor TypeScript issues and with some webpack configuration, at 11:00 the first day of the hackathon, swc was compiling our code base!
Well, swc is promising, but we don’t want to spend all our hackathon deeply modifying our code base just to use it.
Unexpected success with esbuild
So let’s try the other option, esbuild with esbuild-loader !
The configuration of esbuild-loader is quite similar to swc-loader , so two minutes later, we ran our first build with esbuild …
…and it compiled successfully directly!
Okay, so now, let’s fire the browser and see what amount of work still needs to done…
Let’s start the build again and see what else has to be fixed…
Is the product web page really built with esbuild ? Let’s clean everything and try again…
…well, everything is working perfectly. Neat!
Let’s push our modifications and see what is actually broken…
…all our tests are green in our CI system, no regressions!
Let’s try the compilation time and memory usage now.
Not surprisingly, if we still use ts-checker and eslint , we don’t have much improvements, it takes around 1m23s and 1454.2 MiB of memory to compile.
NO_WWW_CHECK=1, it’s another story! It now takes 18s and 925.3 MiB
of memory to compile the product!
Conclusion and improvements
As you can see on the graph above, the speed-up values are very significant.
In just a bit more than two hours we were done with our hackathon project.
One thing that we could do in the future is completely replace webpack with esbuild . However, this requires some changes in our build system that might need more than a hackathon.
Healthcheck in Grafana
So, it is noon on the first day of the hackathon, and we have already completed our project… what do we do now? Well, we have decided to pick another subject, and start working on it 💪.
More than two years ago, during the Hackathon 0x09, we worked on the introduction of the monitoring of our products using Prometheus and Grafana (see this article ). Since then, Prometheus/Grafana have been properly integrated in our products and their usage is growing, especially with the creation of our new DevOps team. By the way, we have written our own Prometheus client in our open-source C library, the lib-common . Use it, it’s great!
The purpose of using Grafana is to easily find out if a product is working fine, exploiting metrics stored in Prometheus. However, we already have a feature in our products allowing to answer the question “Is everything OK” in real-time that is called the “Healthcheck”. It is a report, computed internally based on a lot of various indicators, and displayed on the web interface of our products, as in this example:
This is not something that can be computed based on Prometheus metrics, so there is no way to display it on Grafana currently. Which is a shame because a platform administrator cannot use Grafana only to get a full status of a platform.
So the purpose of our (second) subject is to be able to display the Healthcheck of our products on Grafana dashboards…
What was done
The first step was to read some documentation about Grafana, to understand how this goal could be achieved. We realized pretty quickly that we would have to write a panel Grafana plugin, following this tutorial .
Bootstraping it was quite easy, and luckily the builtin Grafana plugins are open-sourced and a great source of inspiration. They are coded in typescript, using the React framework. At Intersec, we use typescript as a language for our web interfaces, and the Vue.js framework, which is an equivalent of React, so it was not too difficult for us to adapt.
There were two main things to do in our plugin:
Get the Healthcheck data from our product: we first considered creating a Grafana datasource plugin for that, but we soon realized that it was too much work for a hackathon, and that it was not needed (and probably not even a good idea). So we then decided that the panel plugin itself would fetch the data, just like the builtin news panel does. For that, we first had to expose in HTTP/REST the RPC allowing to get the Healthcheck status from our products, and then call it when the panel loads, is modified, or when the dashboard is refreshed. The URL and credentials to use can be configured in the panel options.
Render the data: rendering the data is “just” a matter of writing a React component to display it. As usual with web development, the main challenge was to make it “pretty”, fighting with CSS rules, especially because the Healthcheck is a tree structure. But nothing is impossible…
Better than a long speech, here is a screen capture showing how to add a Healthcheck panel to a Grafana dashboard:
It is even possible to have several Healthcheck panels on the same dashboard, to have the status of multiple platforms at one glance, in real-time (which is not possible from the web interface of our products):
Nice, isn’t it? 😎