It was a little more than a year ago that I kicked off development on what became InGRAINd. A lot has happened in that year, so, let’s take a step back before we kick off with the story.
At Red Sift, we’re building our own data analytics platform for cybersecurity. We built our existing products, OnDMARC and OnINBOX, on top of this platform.
During the product development process monitoring our pipelines proved challenging, and we wanted more visibility into our containers. After a short period of exploration, we found that eBPF would address most of the pain points and dark spots we were encountering.
There was one catch: no eBPF tooling would help us deploy and maintain new probes within our small, but focused ops team. BCC, while great for tinkering, requires significant effort to roll out to production. It also makes it difficult to integrate our toolkit into our usual CI/CD deployment models.
Faced with this dilemma, we decided the only option was for us to write our own Rust-based agent that integrated well with our testing and deployment strategies. I had the opportunity to share our experience at the Linux Plumbers Conference in Vancouver last year, and talk about InGRAINd in depth. Doing away with BCC means that we can deploy a 15MB-ish binary, built and tested by our CI, to servers instead of several 100 megabytes of dependencies.
This past year, we spent a lot of effort making eBPF more accessible while we expanded our use of the InGRAINd agent in our fleet. As a culmination of this process, I am extremely happy to announce the early versions of our agent that can run Rust in the kernel, thanks mostly to the work of the amazing Alessandro Decina!
This is how parsing a network buffer works in InGRAINd now. The code below runs on both my NextCloud Raspberry Pi at home and our AMD64 servers in the data center:
#[map("events")]
static mut events: PerfMap<Event> = PerfMap::new();
#[xdp("dns_queries")]
pub extern "C" fn probe(ctx: XdpContext) -> XdpAction {
let (ip, transport) = match (ctx.ip(), ctx.transport()) {
(Some(i), Some(t)) => (unsafe { *i }, t),
_ => return XdpAction::Pass
};
let data = match ctx.data() {
Some(data) => data,
None => return XdpAction::Pass
};
let header = match data.slice(12) {
Some(s) => s,
None => return XdpAction::Pass
};
...
The Journey to Rust
But the process of getting to this stage is just as exciting as the results. We found that mixing C and Rust made it difficult to share data between the kernel and userspace for more complex cases.
Moreover, maintaining our toolchain to compile and work well on newer Linux versions turned into an uphill battle. We needed more flexibility in the toolchain itself, and more useful tools while building probes.
Third, the mix of two languages meant that we experienced a barrier of entry that made it tedious to extract new metrics and tinker with existing ones. You not only need to know how to write code in eBPF, a highly specialised environment, you also had to have a working knowledge of C and Rust, without much documentation or support.
Since it seems like the idea of putting Rust into the kernel is seeing a warm reception, we decided to address these issues by doing just that. There’s definitely a lot more to refine on our interfaces, but we have already made a lot of progress. I hope everyone will agree that having RustDoc eBPF a lot more approachable than reading the source for Linux.
To get comparisons out of the way, this is what the same probe above looks like in C.
struct bpf_map_def SEC("maps/dns_queries") dns_queries = {
.type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
.key_size = sizeof(u32),
.value_size = sizeof(u32),
.max_entries = 1024,
.pinning = 0,
.namespace = "",
};
__inline_fn
s8 parse_dns_packet(struct xdp_md *ctx, void *buffer, void *data_end, struct _data_dns_query *query) {
struct ethhdr *eth = (struct ethhdr *) buffer;
struct iphdr *ip;
struct udphdr *udp;
void *dns;
ip = (struct iphdr *) (buffer + sizeof(struct ethhdr));
udp = (struct udphdr *) (ip + 1);
/* dns header */
if (udp + 1 > data_end) {
return -7;
}
if (!(eth->h_proto == bpf_htons(ETH_P_IP)
&& ip->protocol == IPPROTO_UDP
)) {
return -6;
}
dns = buffer + sizeof(struct ethhdr)
+ sizeof(struct udphdr)
+ (ip->ihl * 4);
if (dns + 12 > data_end) {
return -5;
}
...
Overall, we found that the C compiler is not nearly as friendly as the Rust compiler when it’s reporting errors.
The high-level abstractions for dealing with low-level kernel structures make all the difference! On top of that, the toolchain allows us to target any kernel version with minimal maintenance costs. We can unit test eBPF code with ease!
Next Steps
In the future, we are looking at integrating specific checks into the build process so you can verify your bytecode before it gets rejected by the kernel’s verifier. We are also expanding the number of idiomatic wrappers around eBPF constructs to cover more program types.
All the small improvements do add up. As the mission of Red Sift is to democratize cybersecurity, we are more than happy to contribute to the eBPF ecosystem to make this exciting technology more approachable.
Since the power of eBPF is in how adaptable it is, we are keen on helping ops teams feel empowered to write and deploy their own monitoring modules more easily, and deploy with less risk and more automation than other approaches.
We’re so excited about this work that we’re bringing a few members on our team to the Barcelona RustFest this weekend. If you’re attending and would like to play around with Raspberry Pis and eBPF we have just the workshop for you!
If you can’t make it to RustFest, make sure to check out the InGRAINd repo, and have a play. You can run InGRAINd, as we do it in production, and save the collected data to your favourite StatsD compatible service, or S3 buckets. We would love to hear what you think!