Putting Rust in the kernel with eBPF

It was a little more than a year ago that I kicked off development on what became InGRAINd. A lot has happened in that year, so, let’s take a step back before we kick off with the story.

At Red Sift, we’re building our own data analytics platform for cybersecurity. We built our existing products, OnDMARC and OnINBOX, on top of this platform.

During the product development process monitoring our pipelines proved challenging, and we wanted more visibility into our containers. After a short period of exploration, we found that eBPF would address most of the pain points and dark spots we were encountering.

There was one catch: no eBPF tooling would help us deploy and maintain new probes within our small, but focused ops team. BCC, while great for tinkering, requires significant effort to roll out to production. It also makes it difficult to integrate our toolkit into our usual CI/CD deployment models.

Faced with this dilemma, we decided the only option was for us to write our own Rust-based agent that integrated well with our testing and deployment strategies. I had the opportunity to share our experience at the Linux Plumbers Conference in Vancouver last year, and talk about InGRAINd in depth. Doing away with BCC means that we can deploy a 15MB-ish binary, built and tested by our CI, to servers instead of several 100 megabytes of dependencies.

This past year, we spent a lot of effort making eBPF more accessible while we expanded our use of the InGRAINd agent in our fleet. As a culmination of this process, I am extremely happy to announce the early versions of our agent that can run Rust in the kernel, thanks mostly to the work of the amazing Alessandro Decina!

This is how parsing a network buffer works in InGRAINd now. The code below runs on both my NextCloud Raspberry Pi at home and our AMD64 servers in the data center:

#[map("events")]
static mut events: PerfMap<Event> = PerfMap::new();

#[xdp("dns_queries")]
pub extern "C" fn probe(ctx: XdpContext) -> XdpAction {
    let (ip, transport) = match (ctx.ip(), ctx.transport()) {
        (Some(i), Some(t)) => (unsafe { *i }, t),
        _ => return XdpAction::Pass
    };
    let data = match ctx.data() {
        Some(data) => data,
        None => return XdpAction::Pass
    };

    let header = match data.slice(12) {
        Some(s) => s,
        None => return XdpAction::Pass
    };

...

The Journey to Rust

But the process of getting to this stage is just as exciting as the results. We found that mixing C and Rust made it difficult to share data between the kernel and userspace for more complex cases.

Moreover, maintaining our toolchain to compile and work well on newer Linux versions turned into an uphill battle. We needed more flexibility in the toolchain itself, and more useful tools while building probes.

Third, the mix of two languages meant that we experienced a barrier of entry that made it tedious to extract new metrics and tinker with existing ones. You not only need to know how to write code in eBPF, a highly specialized environment, you also had to have a working knowledge of C and Rust, without much documentation or support.

Since it seems like the idea of putting Rust into the kernel is seeing a warm reception, we decided to address these issues by doing just that. There’s definitely a lot more to refine on our interfaces, but we have already made a lot of progress. I hope everyone will agree that having RustDoc eBPF is a lot more approachable than reading the source for Linux.

To get comparisons out of the way, this is what the same probe above looks like in C.

struct bpf_map_def SEC("maps/dns_queries") dns_queries = {
    .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
    .key_size = sizeof(u32),
    .value_size = sizeof(u32),
    .max_entries = 1024,
    .pinning = 0,
    .namespace = "",
};

__inline_fn
s8 parse_dns_packet(struct xdp_md *ctx, void *buffer, void *data_end, struct _data_dns_query *query) {
  struct ethhdr *eth = (struct ethhdr *) buffer;
  struct iphdr *ip;
  struct udphdr *udp;
  void *dns;

  ip = (struct iphdr *) (buffer + sizeof(struct ethhdr));
  udp = (struct udphdr *) (ip + 1);

  /* dns header */
  if (udp + 1 > data_end) {
    return -7;
  }

  if (!(eth->h_proto == bpf_htons(ETH_P_IP)
        && ip->protocol == IPPROTO_UDP
        )) {
    return -6;
  }

  dns = buffer + sizeof(struct ethhdr)
    + sizeof(struct udphdr)
    + (ip->ihl * 4);
  if (dns + 12 > data_end) {
    return -5;
  }

...

Overall, we found that the C compiler is not nearly as friendly as the Rust compiler when it’s reporting errors.

The high-level abstractions for dealing with low-level kernel structures make all the difference! On top of that, the toolchain allows us to target any kernel version with minimal maintenance costs. We can unit test eBPF code with ease!

Next Steps

In the future, we are looking at integrating specific checks into the build process so you can verify your bytecode before it gets rejected by the kernel’s verifier. We are also expanding the number of idiomatic wrappers around eBPF constructs to cover more program types.

All the small improvements do add up. As the mission of Red Sift is to democratize cybersecurity, we are more than happy to contribute to the eBPF ecosystem to make this exciting technology more approachable.

Since the power of eBPF is in how adaptable it is, we are keen on helping ops teams feel empowered to write and deploy their own monitoring modules more easily, and deploy with less risk and more automation than other approaches.

We’re so excited about this work that we’re bringing a few members of our team to the Barcelona RustFest this weekend. If you’re attending and would like to play around with Raspberry Pis and eBPF we have just the workshop for you!

If you can’t make it to RustFest, make sure to check out the InGRAINd repo, and have a play. You can run InGRAINd, as we do it in production, and save the collected data to your favorite StatsD compatible service, or S3 buckets. We would love to hear what you think!

PUBLISHED BY

Peter Parkanyi

7 Nov. 2019

SHARE ARTICLE:

Recent Posts

VIEW ALL
Email

Navigating the “SubdoMailing” attack: How Red Sift proactively identified and remediated a…

Rebecca Warren

In the world of cybersecurity, a new threat has emerged. Known as “SubdoMailing,” this new attack cunningly bypasses some of the safeguards that DMARC sets up to protect email integrity.  In this blog we will focus on how the strategic investments we have made at Red Sift allowed us to discover and protect against…

Read more
Email

Where are we now? One month of Google and Yahoo’s new requirements…

Rebecca Warren

As of March 1, 2024, we are one month into Google and Yahoo’s new requirements for bulk senders. Before these requirements went live, we used Red Sift’s BIMI Radar to understand global readiness, and the picture wasn’t pretty.  At the end of January 2024, one-third of global enterprises were bound to fail the new…

Read more
Cybersecurity

Your guide to the SubdoMailing campaign

Billy McDiarmid

A significant number of well-known organizations have been attacked as part of what’s being called the SubdoMailing (Subdo) campaign that has been going on since at least 2022, research by Guardio Labs has revealed.   The scale of execution of this attack is staggering, and the impact is hugely damaging, but the goal is simple…

Read more
Certificates

A confident deployment guide for TLS and PKI

Ivan Ristic

Our journey to better network transport security has been quite the ride, filled with ups and downs. Back in the ’90s, when SSL and the Netscape browser were just taking off, things were pretty hard. We were dealing with weak encryption, export restrictions on cryptography, and computers that couldn’t keep up. But over the…

Read more