Objective

Text editors are essential tools for software developers. These editors provide many useful features to make developers’ lives easier and improve their productivity.

There is a wide variety of text editors. Moreover, each language has its own characteristics and each company/project may decide to adopt different coding rules. For all these reasons, it is difficult to offer a configuration suitable for everyone.

In this hackathon, we tried to help developers configure their text editors correctly for optimal performance. For this purpose, using the Language Server Protocol (LSP) seemed like a good option.

LSP defines the protocol used between an editor or IDE and a language server that provides language features like autocomplete, go to definition, find all references etc.

– [Microsoft’s official page for Language Server Protocol(https://microsoft.github.io/language-server-protocol/)

With LSP, text editors communicate with a dedicated server for each specific language and benefit from its advanced features.

During this hackathon, we explored the possibilities of using LSP at Intersec. Our team consisted of Guillaume Chevallereau, Maxime Leblanc, Muyao Chen, and Yannick Li.

LSP for C

Our backend is mostly written in C. There are already many servers dedicated to C so we will reuse one that we can customize. A list of already implemented language servers is available here and here.

For the C language, we decided to use clangd implemented by the LLVM Project. clangd provides the main features: code completion, hover, jump to def, workspace symbols, find references and diagnostics.

clangd is a configurable tool. In order to work correctly, it needs to know the compile command required for a file. These compile commands are passed through a file named compile_commands.json.

At Intersec, we use the Waf framework as build system. Waf provides an extension to generate the compile command file but this extension is only available on recent versions, so features like block would not work correctly. We therefore added a modified version of the waf extension to our build system.

This version mainly handles blocks. To this end, block files (which are translated to C during the build) need specific flags while the generated C files must be discarded in the compilation database. The modified version is available here.

Python

There is not much to do for python besides choosing the python LSP implementation. We decided to use python-lsp-server from Spyder IDE team and community. Adding it to our poetry installation was enough.

IOP

The IOP (Intersec Object Packer) is a method to serialize structured data. Data structures are defined in a custom IOP description language, so we have to create our own dedicated LSP server.

The main features that could be implemented are: code completion, hover, jump to def, workspace symbols, find references and diagnostics.

Diagnostics (linting)

We started by focusing on diagnostics (also known as linting). Diagnostics enable you to see errors, warnings, hints and information right on the editor. This information is provided by the compiler but we ran into a problem with our IOP compiler. Currently, it stops after the first error encountered. So if we want to have a real-time display of all errors, we have to fix the IOP compiler so that it keeps compiling after it encounters the first error.

Allowing a compiler to continue listing correctly all errors is a complicated task as it requires ignoring previous errors while keeping its robustness.

Previously, when the compiler detected an error on an IOP file, it logged it and threw an error which stopped the compilation. (This is the safest and easiest way to implement a compiler).

To improve it without refactoring the whole compiler code, we decided to upgrade each parsing functions. So now, when an error is detected, the compiler still logs it, but sets an error flag and continues the parsing as if there was no error. The error flag is returned to the caller and the function ends. That way, we keep the same behavior as before. (This also allows us to update functions step by step).

However, we cannot handle all kinds of error. Especially syntaxical errors (like missing brackets…) as it can be tricky to detect the real end of the instruction. We thus wanted the compiler to end the parsing for this type of error.

Nevertheless, In this hackathon, we reused the old IOP compiler to show the first bug it found. The result is shown in the following screenshot.

In order to do that, we cloned the Microsoft’s LSP example and changed it to use our own compiler. The changes that we made are as follows:

Activate the LSP

LSP mode is activated on events. The most common event is the file language which is mainly determined by its extension. For example, a file with the extension *.c is considered as a C file. However, we can also choose the language in VS Code if the extension is not recognized automatically by the editor.

Since IOP is a new language that we have created, it is not recognized. We should create a new language package for IOP files to define all the syntax. Because this is not our main concern in the hackathon, we decided to simply declare all plaintext files as IOP files, as the editor will trigger our LSP server when it examines a text file which includes *.iop. If we want to really use the LSP server for IOP files, we have to properly declare a new language by following these steps .

lsp-sample/package.json

       "activationEvents": [
               "onLanguage:plaintext"
       ],

LSP server

On the server side, we first import necessary libraries.

lsp-sample/server/src/server.ts

diff --git a/lsp-sample/server/src/server.ts b/lsp-sample/server/src/server.ts
import {
       Range,
} from 'vscode-languageserver-types';
import { fileURLToPath } from 'url';
import { exec } from 'child_process';
import { promisify } from 'node:util';

Then, we add the compiler to be used:

lsp-sample/server/src/server.ts

const iopcPath = '/home/muyao/dev/worktree1/platform/lib-common/src/iopc/iopc';
const iopcArgs = ' --Wextra --language c ';

(We should later move this part into the config file instead of hard coding it.)

Next, we choose the trigger of the diagnostic:

lsp-sample/server/src/server.ts

documents.onDidSave(change => {
        validateTextDocument(change.document);
});

The check will be triggered when we save the file. After saving, the compiler will be called to examine the validity of the file. In case there is any error, the error message will be collected and parsed as described in the next section of code:

lsp-sample/server/src/server.ts

interface IopError {
       fullFileName: string;
       line: number;
       column: number;
       severity: string;
       message: string;
}

function parseError(errMsg: string): IopError {
       // one line error
       const strings = errMsg.split(':');

       const fullFilename = strings.shift();
       const line = Number(strings.shift());
       const column = Number(strings.shift());
       const severity = strings.shift();
       const message = strings.join("");

       return {
               fullFileName: fullFilename == undefined ? '' : fullFilename,
               line: line,
               column: column,
               severity: severity == undefined ? '' : severity,
               message: message,
       };
}

As the compiler currently stops at the first error, it is the only one that is displayed. Once the compiler is able to return several errors in the file, a loop will allow every one of them to be displayed.

And finally the crucial part is in the function “validateTextDocument()”:

lsp-sample/server/src/server.ts

async function validateTextDocument(textDocument: TextDocument): Promise<void> {
       const filePath = fileURLToPath(textDocument.uri);
       // TODO Based on the file path, find in the config the arguments
       // getClientCapability
       const cmd = iopcPath + iopcArgs + filePath;
       const execPromise = promisify(exec);

       // wait for exec to complete
       try {
               const {stdout, stderr} = await execPromise(cmd);
       } catch (error){
               // console.log(error);
               if(error) {
                       const errStrings = error.toString().split(':');
                       console.log(errStrings);

                       errStrings.shift(); // Shift Error:
                       errStrings.shift(); // Shift Command Failed:
                       const realErrString = errStrings.join(':');
                       const diagnostics: Diagnostic[] = [];
                       const iopError = parseError(realErrString);
                       const range = Range.create(
                                       iopError.line - 1,
                                       iopError.column,
                                       iopError.line,
                                       0,
                       );
                       const diagnostic: Diagnostic = {
                               severity: DiagnosticSeverity.Error,
                               range: range,
                               message: iopError.message,
                               source: "iopLinter"
                       };
                       diagnostics.push(diagnostic);
                       connection.sendDiagnostics({ uri: textDocument.uri, diagnostics });
               }
       }
}

Here, we execute the compiler in a thread and then parse it to create an instance of Diagnostic. By sending it, the client will react according to the range and severity of the error (e.g. the underline color).

LSP client

In the lsp-sample, there is already the client part of code that is compatible with the VS Code editor. Therefore, no action is required for our demo.

Things to be done…

As we discussed before, we didn’t have a LSP server + client that was working. In order to do that, we still have some miles to go:

1. Ideally, we should make our IOP compiler able to take a text stream as input, so that the diagnostic will take place based on live changes.
2. Create a config file that can dynamically set the right arguments for IOP compiler based on the project.
3. Create a language package for the editor to recognize IOP files as a unique language type and enable the code highlight at the same time.
4. Introduce other functionalities than diagnostics: autocomplete, code suggestions, jump to definition, etc.
5. And maybe find a way to link the IOP files with the generated C files, so that we can jump directly to the IOP definition in a C file.

Conclusion

Our goal in this hackathon was to test the use of LSP to make the configuration of text editors easier for everyone. We managed to configure the clangd server well for both C and Python. This already makes it possible to have more features available in all editors.

But there is still a lot of work to do to fully support IOPs, starting with writing a complete LSP server.