Securing BIND 9 with eBPF/XDP Workshop

The virtual machines have a domain name in the form ebpfNNN.dnslab.org.

Please login to the machines with a modern web browser under the URL https://ebpfNNN.dnslab.org:9090 with the username user and the password DNSandBIND. You can also login with SSH and the same username and password.

The virtual machines run the cockpit tool (https://cockpit-project.org) to provide a terminal in the web browser.

Then select the terminal (last menu option on the left) and start the tutorial.

% apt install bind9

options {
  directory "/var/cache/bind";
  allow-recursion { localhost };
};

% named-checkconf
% rndc reconfig

% hostname -I

; <<>> DiG 9.16.22 <<>> @138.197.66.242 ch txt version.bind
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14922
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 86bfaf26127e2c3c01000000619356412aabbab08bd3eabf (good)
;; QUESTION SECTION:
;version.bind.			CH	TXT

;; ANSWER SECTION:
version.bind.		0	CH	TXT	"9.16.22-Debian"

;; Query time: 39 msec
;; SERVER: 138.197.66.242#53(138.197.66.242)
;; WHEN: Tue Nov 16 00:57:05 CST 2021
;; MSG SIZE  rcvd: 96

• Install the BCC Tools and the Linux header for the current kernel

% apt install bpfcc-introspection  bpfcc-tools python3-bpfcc
% apt install linux-headers-$(uname -r)

• Save the following C source code into the file drop-non-dns-udp.c., This eBPF program will
  • be executed on every network packet
  • print the text got a packet for every network packet received
  • extract the IP and UDP header from the packet
  • if neither the UDP source nor the UDP destination port is 53 (DNS), the return code XDP_DROP is returned, dropping the packet. A message is reported when dropping the packet
  • all other packets are given to the Linux TCP/IP stack with the return code XDP_PASS
  • production XDP code should not use bpf_trace_printk to communicate information, but should write the data into a map structure that will then be read from the user-space loader

#define KBUILD_MODNAME "filter"
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/in.h>
#include <linux/udp.h>

int udpfilter(struct xdp_md *ctx) {
  bpf_trace_printk("got a packet\n");
  void *data = (void *)(long)ctx->data;
  void *data_end = (void *)(long)ctx->data_end;
  struct ethhdr *eth = data;
  if ((void*)eth + sizeof(*eth) <= data_end) {
    struct iphdr *ip = data + sizeof(*eth);
    if ((void*)ip + sizeof(*ip) <= data_end) {
      if (ip->protocol == IPPROTO_UDP) {
        struct udphdr *udp = (void*)ip + sizeof(*ip);
        if ((void*)udp + sizeof(*udp) <= data_end) {
          if ((udp->dest != ntohs(53)) && (udp->source != ntohs(53))) {
            if (udp->dest != udp->source) {
              bpf_trace_printk("drop udp src/dest port %d/%d\n", ntohs(udp->source), ntohs(udp->dest));
              return XDP_DROP;
            }
          }
        }
      }
    }
  }
  return XDP_PASS;
}

• Save the eBPF loader (written in Python) into the file drop-non-dns-udp.py.
  • This loader will compile the eBPF program above, load it into the Linux kernel and will attach it to the loopback network interface (lo)
  • There will be warnings during compilation that can be ignored (for this test)
  • The virtio driver used in the lab machines do not allow XDP programs on the eth0 and eth1 interfaces, so we're testing with the loopback lo interface

#!/usr/bin/env python3

from bcc import BPF
import time

device = "lo"
b = BPF(src_file="drop-non-dns-udp.c")
fn = b.load_func("udpfilter", BPF.XDP)
b.attach_xdp(device, fn, 0)

try:
  b.trace_print()
except KeyboardInterrupt:
  pass

b.remove_xdp(device, 0)

• Make the python program executable

% chmod +x drop-non-dns-udp.py

• Execute the Python loader program (there will be warnings that we can ignore)

% ./drop-non-dns-udp.py

• Login to the lab machine with a new connection (or use tmux)
• Execute DNS queries towards the BIND 9 DNS server via the loopback interface. This query should not trigger the XDP program to drop packets

% dig @localhost isc.org

• If we send the DNS query to a port other than port 53, the packet get's dropped

% dig -p 5353 @localhost isc.org

• Configure BIND 9 to forward all request for the domain isc.org to Quad9 (9.9.9.9). In the file /etc/bind/named.conf add

zone "isc.org" {
  type forward;
  forwarders { 9.9.9.9; };
};

• Check the configuration with named-checkconf and reload the BIND 9 DNS server with rndc reconfig
• Install bpftrace

apt install bpftrace

• Save the following source into the file forward-trace.bt

#!/usr/bin/bpftrace

struct dns_name {
        unsigned int   magic;
        unsigned char *ndata;
        unsigned int   length;
        unsigned int   labels;
        unsigned int   attributes;
        unsigned char *offsets;
//      isc_buffer_t  *buffer;
//      ISC_LINK(dns_name_t) link;
//      ISC_LIST(dns_rdataset_t) list;
};

BEGIN
{
  print("Waiting for forward decision...\n");
}
uprobe:/lib/x86_64-linux-gnu/libdns-9.16.22-Debian.so:dns_fwdtable_find
{
  @dns_name[tid] = ((struct dns_name *)arg1)->ndata
}

uretprobe:/lib/x86_64-linux-gnu/libdns-9.16.22-Debian.so:dns_fwdtable_find
{
 if (retval == 0) {
    printf("Forwarded domain name: %s\n", str(@dns_name[tid]));
 }
 delete(@dns_name[tid]);
}

• Make the file executeable

% chmod +x forward-trace.bt

• Execute the bpftrace script

% ./forward-trace.bt

• Login to the lab machine from a different terminal (or use tmux). Flush the cache of the BIND 9 resolver with rndc flush, then execute a DNS name resolution for isc.org and for other domains. See that the bpftrace script reports the forward decisions in BIND 9

% dig @localhost isc.org

• Switch the operating system to use the BIND 9 DNS resolver

% echo "nameserver 127.0.0.1" > /etc/resolv.conf

• Start gethostlatency.bt in one terminal, execute some other network tools that require DNS name resolution (ping, apt, traceroute etc) to see the DNS name resolution latency
• Count UDP send/receives by on-CPU PID and process name (Excerpt From "BPF Performance Tools" by Brendan Gregg)
  • Execute some DNS queries against the BIND 9 resolver
  • Exec the bpftrace script with CTRL+C to see the report

% bpftrace -e "k:udp*_sendmsg,k:udp*_recvmsg { @[func, pid, comm] = count(); }"

• Show packets received over UDP by the named process as a histogram (execute some queries, then terminate the script with CTRL+C)

% bpftrace -e "kr:udp_recvmsg /pid == $(pgrep named)/ { @recv_bytes = hist(retval); }"

• Histogram of UDP packets send by the BIND 9 process (execute some queries, then terminate the script with CTRL+C)

% bpftrace -e "kprobe:udp_sendmsg /pid == $(pgrep named)/ { @size = hist(arg2); }"

• Example of BIND 9 tracing: print whenever rndc status is executed

% bpftrace -e 'uprobe:named:named_server_status { print("rndc status executed") }'

• How long does it take (in nanoseconds) to flush the cache with rndc flush? Do some queries against the BIND 9 DNS resolver to fill the cache. Then call rndc flush to flush the cache. The printed value should get higher the more cache entries need to be cleaned.

% bpftrace -e "uprobe:/usr/sbin/named:named_server_flushcache / pid == $(pgrep named) /{ @start[tid] = nsecs; }
    uretprobe:/usr/sbin/named:named_server_flushcache /@start[tid]/ { print(nsecs - @start[tid]); delete(@start[tid]); }"

• Trace the cache related functions BIND 9 executes. Send some queries to the BIND 9 DNS resolver, then use rndc flush, rndc flushname <domain-name> and rndc flushtree <name> and see which functions are executed inside BIND 9

% bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libdns-9.16.22-Debian.so:dns_cache* { print(func) }'