In my previous post I explained how ingraind led to the development of RedBPF and in an earlier post I gave an overview of the RedBPF API.
In this post I’m going to show how we compile Rust source code to BPF binary code.
How Rust is compiled to BPF binary code
Rust supports a large number of target platforms, but BPF is not one of them yet. Therefore in order to compile for the BPF virtual machine, cargo-bpf needs to implement some ad-hoc build logic.
The build logic can roughly be summarised with the following sequence of steps:
- Compile Rust source code to LLVM bitcode.
- Post-process the LLVM bitcode to apply BPF specific hacks.
- Optimize the LLVM bitcode with BPF specific optimizations.
- Generate a native object file that can be loaded in the kernel.
Step 1: compile Rust source code to LLVM bitcode
The first step is to compile the source code to LLVM bitcode, which is something rustc supports via the --emit=llvm-bc command-line option:
--emit [asm|llvm-bc|llvm-ir|obj|metadata|link|dep-info|mir]
Comma separated list of types of output for the compiler to emit
cargo bpf build invokes rustc via cargo rustc, and if compilation succeeds it feeds the output to the next step.
Step 2: post-process the LLVM bitcode
In the early days we didn’t do any post-processing of the LLVM IR: we compiled the bitcode with rustc, then generated the BPF native object code with the LLVM Compiler. That worked, however there were some limitations in the kind of code that you could write.
The BPF virtual machine does not allow functions to call other functions, everything has to be inlined into one big entry point. That was was not too bad, it meant we had to make sure to annotate all our functions with #[inline(always)].
But what about dependencies? Most BPF programs don’t depend on external crates so that was not too bad either, however pretty much all programs do depend on the core crate. That meant that whenever we called a core function we had to hope that it got inlined, which mostly did happen. Except when it didn’t, so programs would compile but then would fail to load at runtime. We do run end-to-end tests in the Ingraind CI that check that our BPF code can actually load in a number of supported kernels, but still we could obviously do better.
Another limitation was that panicking was not possible. That per se wasn’t a big deal – all our fallible APIs return Result – so we had no direct use for panic(). The problem was with things that panic internally – like array indexing – which is very common in BPF code where you often inspect memory and network data.
Panicking was not possible for two reasons. The first was that unless the core crate is compiled with the panic_immediate_abort feature (which is disabled by default), the panic functions are annotated with #[inline(never)] to avoid code bloat. That makes perfect sense in general, but not for the BPF case where everything must be inlined by design.
The second reason panic was not supported is that the LLVM BPF target does not support core::intrinsics::abort(). BPF does provide an exit instruction to terminate a program, but since Rust doesn’t officially support BPF, it is not usable from Rust. We could have obviated to this problem by using inline assembly, but since asm! is unstable and RedBPF targets stable, that was not a viable option.
We decided to add a post-processing step to cargo-bpf that uses llvm-sys to transform the LLVM IR produced by rustc. To solve the inlining problem, we iterate over all the functions defined in a module, and modify their attributes. Specifically, we remove the noinline attribute and add alwaysinline instead.
Inlining everything enables us to write programs that call other functions, and solves half of the issue with panics. To make panic!() work we do one more thing: we look for panic handlers and inject an exit instruction with LLVMGetInlineAsm and the instruction builder API. This way the panic code gets inlined and exits properly.
Step 3: optimize the LLVM bitcode
Just changing the inlining attributes in the LLVM IR is not enough, we actually need to run the always-inline LLVM pass using the LLVM optimizer.
The other thing we use the optimizer for is to force loop unrolling, as BPF programs are not allowed to loop. Newer kernels actually do support some bounded loops, however the functionality is currently pretty limited and we need to support older kernels back to version 4.15 anyway.
After learning more than I ever wanted to know about how loop unrolling works in LLVM, adding a bunch of printf()s to the relevant LLVM source file, and finding an obscure way to pass parameters to the unroll pass, we got loop unrolling working too.
Step 4: Generate a native object file
The final step is pretty simple. We take the post-processed, optimized LLVM IR and compile it for the BPF architecture using the LLVM Compiler. The output is an ELF object file with a layout compatible with what Loader::load_file expects.
Conclusion
The Rust compiler does not support compiling code for the BPF platform. Therefore we had to implement some custom compilation logic in cargo-bpf. As part of the compilation process, we use LLVM to tweak the bitcode produced by rustc, applying some hacks that are needed to load the resulting code in the BPF virtual machine. Going forward, we plan on looking into what would be required to add BPF platform support to rustc.
As always for any questions, feel free to ping me at @alessandrod on Twitter.
