The goal was to develop benchmarks on a few of our core technologies in lib-common, in order to:

• Be able to compare the performances of our custom implementations with standard implementations.
• Be able to add automated tests on performance (e.g. adding non-regression tests to ensure that changes which seem to be harmless do not worsen performance).

Benchmark library

The first step was to develop a benchmark library; the success criteria we established were the following (compared to the already existing benchmarks in our code base):

• Ease writing of benchmarks
• Standardize output format
• Allow factorization of the use of external tools using benchmarks

And the result looks like this, on the user side:

ZBENCH_GROUP_EXPORT(bithacks) {
    const size_t small_res = 71008;

     ZBENCH(membitcount_naive_small) {
        ZBENCH_LOOP() {
            size_t res = 0;

            ZBENCH_MEASURE() {
                res = membitcount_check_small(&membitcount_naive);
            } ZBENCH_MEASURE_END

            if (res != small_res) {
                e_fatal("expected: %zu, got: %zu", small_res, res);
            }
        } ZBENCH_LOOP_END
    } ZBENCH_END
} ZBENCH_GROUP_END

If you are familiar with lib-common, you can see that it looks very similar to the z test library.
The code is more or less translated as follows:

if (BENCHMARK_RUN_GROUP("bithacks") {
    const size_t small_res = 71008;

    if (BENCHMARK_RUN("membitcount_naive_small") {
        for (int i = 0; i < BENCHMARKS_NB_RUNS; i++) {
            size_t res = 0;

            BENCHMARK_START();
            res = membitcount_check_small(&membitcount_naive);
            BENCHMARK_END();

            if (res != small_res) {
                e_fatal("expected: %zu, got: %zu", small_res, res);
            }
        }
    }
}

First benchmark: printf

The first benchmark we did was the benchmark of libc snprintf against our own implementation of snprint (actually called isnprintf internally).

The result was not what we expected (the chart below shows the duration for one million calls to snprintf):

As you can see, our implementation is about two times slower than the standard implementation in libc. So, it might be interesting to use the standard implementation instead of our own.

Unfortunately, we can define and use some custom formatters in our own implementation that are not trivially compatible with the standard implementation.

In conclusion, this is an interesting idea to improve the speed of lib-common, but it needs some rework in order to replace our own implementation.

IOP packing/unpacking

If you have read one of our previous articles, you already know what IOP is.

Long story short, and for what matters here, we use it as a serialization library. Since it is widely used in our products, we also decided to benchmark serialization and deserialization in binary, JSON, and YAML.

The purpose of this benchmark was not really to compare the performances of packing and unpacking with other implementations, but more for non-regression or comparison between JSON and YAML (indeed, as we could have expected, binary packing and unpacking is a lot faster – it is what is used for communication between daemons).

We can see that unpacking costs more than packing, which seems normal, and that YAML unpacking seems particularly costly. This is an interesting point to keep in mind for optimization.

Bithack

The speed of lib-common is partly due to the optimization of low-level functions. One of these functions is membitcount which counts the number of bits set in a buffer.

We have four different implementations:

• membitcount_naive which does the sum of a naive bitcount on each byte of the buffer.
• membitcount_c which takes into account multiple bytes at once when doing the bitcount of the buffer.
• membitcount_ssse3 which uses the SSSE3 processor instruction set.
• membitcount_popcnt which uses the popcnt processor instruction.

The purpose of this benchmark is to check if the optimized implementations have a real impact on performance, or if we can keep a more naive implementation that is more readable and maintainable.

There are some real differences between the four different implementations, so the optimizations are legitimate. membitcount_popcnt is the fastest one, and it is the one actually used when available.

Spinlocks/Mutexes

Recently, there were some discussions about the use of spinlocks in the user space. In lib-common, we use some spinlocks in thread jobs.

The purpose of this benchmark is to check if replacing the spinlocks by mutexes has an impact on the performance of thread jobs.

The results are not conclusive. We see no visible impacts of switching from spinlocks to mutexes in the benchmark.

This might be because:

• There are no real macro differences when switching from spinlocks to mutexes.
• The differences are masked by the background noises of the benchmarks, they are not properly designed to test the modification.

Conclusion

Over this hackathon, we developed a new benchmark library, zbenchmark, that is easy to use and standardizes the output of the different benchmarks.

We also adapted and wrote some benchmarks to find future optimizations and avoid regressions.

Although there is still a lot of things to do (write new benchmarks, compare with standard implementations, find optimizations), the work done during this hackathon is promising.