Compiling OpenWrt to support DPDK

Community Builds, Projects & Packages
1

I've started to toy around with DPDK and running it on OpenWRT for fun, and I've decided to start with building it
for x86_64 to test it on QEMU - before trying to cross compile it for any real hardware.
It took me some time to get things working and I tried various guides online - but each had its own quirks, so I decided to share my process in case anyone else runs into the same issues.

Note: I tested this guide on my local machine running Ubuntu 22.04 with:

  • OpenWRT == 22.03.3
  • DPDK == 22.11.1

I used qemu to test the results using the official guide DPDK On OpenWRT.

Prerequisites

sudo apt install gcc g++ libncurses-dev gawk autoconf libtool python3 python3-distutils

Building OpenWRT with DPDK support

I encountered several errors during the compilation when I run the build as root - so don't run this guide as a root user.
First, clone OpenWRT (v22.03.3):

git clone https://github.com/openwrt/openwrt.git -b v22.03.3
cd openwrt

Next, copy the sources from feeds.conf.default:

cp feeds.conf.default feeds.conf

And update the feeds:

./scripts/feeds update
./scripts/feeds install -a -p

At this point, we need to configure the OpenWRT build:

  1. Run make menuconfig and enable the following (Set to *):

    1. Target System = x86
    2. Subtarget = x86_64
    3. Global build settings ā†’ Kernel build options ā†’ Compile the kernel with HugeTLB support
    4. Global build settings ā†’ Kernel build options ā†’ Enable /proc page monitoring
    5. Advanced configurations options (for developers)
    6. Advanced configurations options (for developers) ā†’ Toolchain Options
    7. Advanced configurations options (for developers) ā†’ Toolchain Options ā†’ C Library implementation = use glibc
    8. Build the openwrt SDK
    9. Languages ā†’ Python ā†’ python3 (python and several additional packages are required for building numactl)
    10. Utilities ā†’ pciutils

  2. Edit target/linux/generic/config-5.10 - this is required in order to enable certain kernel features that DPDK depends on that do not appear in the main OpenWRT menuconfig screen

  3. Edit the following lines:

    1. # CONFIG_VFIO is not set ā†’ CONFIG_VFIO=y
    2. # CONFIG_NUMA is not set ā†’ CONFIG_NUMA=y

  4. Append these lines to the file:

    CONFIG_VFIO_NOIOMMU=y
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO_VIRQFD=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_MMAP=y

Now, download the build dependencies:

make download -j $(nproc)

And build OpenWRT:

make -j $(nproc)

If the kernel compilation stage fails - you may need to fill some defaults in the kernel configuration - retry with make -j $(nproc) V=sc - and press enter when prompted to enter the default value.

Building with libnuma

DPDK depends on NUMA for memory layout optimizations. To build libnuma, we can follow the tutorial at The DPDK Site. I've attached it here and filled some blanks:

Run this from the root directory of your OpenWRT clone.

git clone https://github.com/numactl/numactl.git -b v2.0.13
cd numactl
./autogen.sh
autoconf -i
export PATH=$(realpath ../staging_dir/toolchain-x86_64_gcc-11.2.0_glibc/bin/):$PATH
export STAGING_DIR=$(realpath ../staging_dir)

#Build into the rootfs
./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=$(realpath ../files)
make install

#Build into the openwrt compiler
./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=$(realpath ../staging_dir/toolchain-x86_64_gcc-11.2.0_glibc/x86_64-openwrt-linux/)

make install
cd ..

Finally, rebuild the entire OpenWRT image with libnuma:

make -j $(nproc)

Usage with DPDK

To include your binaries in the image, copy them to files/ and they will appear in the relative path to your project's rootfs. To make sure your build works with DPDK, you can download DPDK v22.11.1 and test it with dpdk-devbind.py:

# In your local openwrt repo
# Download DPDK sources
wget http://static.dpdk.org/rel/dpdk-22.11.1.tar.xz
tar -xf dpdk-22.11.1.tar.xz
# Create the target directory in the openwrt rootfs
mkdir -p files/usr/bin
cp dpdk-stable-22.11.1/usertools/dpdk-devbind.py files/usr/bin

# Rebuild the image with your new files
make -j $(nproc)

The target image are built in bin/targets/x86/64-glibc. We can then test the target image in QEMU using the tutorial in the DPDK site or with ./scripts/qemustart x86 64-glibc :

/usr/bin/dpdk-devbind.py --status
# Needed in order to prevent `vfio-pci: probe of 0000:00:03.0 failed with error -22`
echo Y > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
/usr/bin/dpdk-devbind.py --force --bind vfio-pci 0000:00:03.0

screenshot
screenshot732Ɨ149 6.99 KB

And now you should be able to smoothly run your DPDK application on an OpenWRT system :slight_smile:

Note: dpdk-devbind.py may write lspci: Unable to load libkmod resources: error -12 - but it seems that you can safely ignore this message as it doesn't really affect anything.

I've tried to make this guide as thorough as possible, please let me know if you had issues with guide or if this worked for you on other versions or architectures :slight_smile:

2 Likes
2

Hi you can use this package for openwrt: https://github.com/k13132/openwrt-dpdk

3

Hi, if we implement NAT DPDK package, is there any NAT performance gain compared with current linux kernel flow table/fast path?

4

Bumping this as Iā€™m sure others are curious as well. :nerd_face: