Full-featured custom build for Dynalink DL-WRX36 (AX3600)

I compiled from source a custom firmware image for the WRX36. I'm pretty happy with how it is performing, and figured Id share it.

LINK to firmware (hosted on github)

The firmware images are in the targets/qualcommax/ipq807x folder. There are also a whole bunch of "extra" .ipk packages that I built but didnt install (including most of the possible kernel modules).

If you have any comments / suggestions / bug-reports let me know. I hope some of you will find this useful.



It is worth mentioning that this firmware is a bit different from a "standard" build in several ways. Refer to the extras folder for the exact .config files and custom configuration files that went into the build, but I'll highlight a few key differences here:

1. DNS/DHCP

It uses unbound (with DNSSEC) + odhcpd for DNS/DHCP (note: dnsmasq isnt installed). Adblock is also included, which integrates automatically with unbound. This should all be set up and ready to go on the first bootup.

Note: unbound has been compiled with everything optional (except python support) compiled in. It is set up to default to be a validating recursive resolver, with settings more aggressive than the "max" openwrt preset.

2. Compiler optimizations

It was compiled with -03 -march=cortex-a53+crc+crypto optimization flags with link time optimization (LTO) + dead code elimination using mold as the linker

3. Full software suite

It probably has more than you'll really need/want (the firmware image is ~50 mb), but i figure that Im better off compiling everything I might want to use at some point into the squashfs image (where it takes up a quarter of the space) than installing it on the overlay later on.

It includes (but isnt limited to):

  • full versions of several network utilities (wpan, ip, iw, ethtool),
  • ksmbd support + code to easily run an on-router plex media server + drivers/kmods for usb-attached storage
  • bash + vim + other miscellanious useful cli programs
  • support for various network protocols and ffile transfer protocols: SNMP, TFTP, full-featured wget and curl

4. Custom kernel

Kernel settings were tweaked in a number of ways: enabling only cortex-a53 erratum, enabling better multithreading support, enabling memory compaction, enabling crypto functions that use armv8 crypto extensions, fixing a bug where the assembler 's -march and the -DASM_ARM_ARCH flag were being incorrectly set to armv8.5-a`, etc. (note: i didnt fool with the "network" or "drivers" sections of the kernel config)

One last noteworthy kernel tweak is that I made the kernel fully preemptable and set it to use a 1000 Hz timer. This may have dropped the total throughput of the kernel slightly, but it can still handle my 1000/500 fiber connection and has done wonders for latency jitter. a few speedtests are provided in the repo's "extras" folder, but to give an example: running

 ping -c 100 -U -A gstatic.com

the overall statistics for the set of all 100 pings was

--- gstatic.com ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 2295ms
rtt min/avg/max/mdev = 21.851/21.974/22.120/0.060 ms, ipg/ewma 23.177/21.955 ms

The overall spread for all 100 pings was about 1/4 of a second, and the std deviation was only 60 ms.

speedtest-petperf.sh was giving me results like

 Download: 900.25 Mbps
  Latency: [in msec, 31 pings, 0.00% packet loss]
      Min:  21.900
    10pct:  22.000
   Median:  22.700
      Avg:  24.352
    90pct:  28.900
      Max:  31.000

Indicating at 90% saturation of my connection my pings were on average about 2.5 ms and at worse about 9ms. Not bad.
The

3 Likes

Can I still use opkg to install whatever I need, like dnsmasq-full, wireguard*, openvpn*, etc?

Yeah. The standard

opkg update
opkg install $PKG

works, and will point you to the aarch64_cortex-a53 openwret repos.

one warning: I didnt realize this until just now, but it also points you to the targets/qualcommax/ipq807x repo that has kmods. This could be dangerous, in the sense that it might allow you to install kmod packages that arent actually comparable with the firmware's custom kernel.

I have most of the kmod packages built for the custom kernel. You can find them here. Should you need a package requiring a kmod Id try and make sure you install the one from my github repo and not the main openwrt one.

But that is just for kmods...for normal packages the standard openwrt repos should work just fine.

Hi!
I have an issue while trying to install "kmod-ipt-ipopt" from your kmods folder on github. Any help?

Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.1.65-1-1763aacc7248702f887609f8c80e68c4) for kmod-ipt-ipopt
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-ipt-ipopt found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-ipt-ipopt.

How are you installing it? I tried (with the opkg --noaction flag) and it seemed to work. However, i did get that message when i tried to install it from the openwrt repos.

Try running

cd /tmp
wget https://github.com/jkool702/openwrt-custom-builds/raw/main/WRX36/bin/targets/qualcommax/ipq807x/packages/kmods/kmod-ipt-ipopt_6.1.65-1_aarch64_cortex-a53.ipk
opkg install /tmp/kmod-ipt-ipopt_6.1.65-1_aarch64_cortex-a53.ipk
rm /tmp/kmod-ipt-ipopt_6.1.65-1_aarch64_cortex-a53.ipk

Side note: on my WRX36 running this firmware, I've discovered that the ntpd and sysntpd service appear broken, and were causing some issues. running

service ntpd disable
service sysntpd disable
reboot

Fixed these issues. If you're not sure if they are working or not, run logread. If you see any entries starting with

daemon.err ntpd

then your ntpd is broken too.

If you see a bunch of

user.err : jail: failed to load dependencies

Then your sysntpd is broken too.

I was installing it through luci (upload package), but your method seems to be working without errors. Looking at ntpds and sysntpd in logread; it doesn't output any errors. But for now I'm struggling with ksmbd, I've changed the name there and for some reason i still can't connect.

daemon.err ksmbd: modprobe of ksmbd module failed, can\'t start ksmbd!
  • I have some errors from irqbalance
daemon.warn /usr/sbin/irqbalance: Cannot change IRQ 34 affinity: Invalid argument
daemon.warn /usr/sbin/irqbalance: Cannot change IRQ 63 affinity: Invalid argument

I got these too. I think it is an error with how it parses luci. In /etc/config/irqbalance comment out everything exceppt the "enabled" option, so that the only un-commented lines are

config irqbalance 'irqbalance'
        option enabled '1'

This fixed it for me - it now runs without complaining about any IRQs.

Thats....weird.

ksmbd is running and working on my wrx36 (accessible from both windows and linux machines).

I wrote you a little script that (if it works like it is supposed to) should hopefully fix your issue (by reinstalling ksmbd) and will set you up with the same working config I have for ksmbd.

before running it, set the 3 paramaters at the top of the script that determine password, share name, and ksmbd root directory.

Note: this sets up password protected smb access. To access the share, use

user=SMBUSER
password=<WHATEVER YOU USED AS THE PASSWORD>
(if needed) domain/workgroup=WORKGROUP  

SCRIPT

# if not already using a bash shell run this line first
exec /bin/bash

# replace ______ with desired password for SMB
# id recommend using all caps here...IIRC it works better 
SMBPASSWORD=______

# replace ______ with share name
share_name=______

# replace ______ with root path for share
# if empty itll assume it is at /mnt/${share_name}
share_mnt=______


#####################################################################


{ [[ -z ${share_mnt} ]] || [[ "${share_mnt}" == '______' ]]; } && share_mnt="/mnt/${share_name}"

host_name="$(uci get system.@system[0].hostname)"
host_name="${host_name^^}"

mkdir -p /etc/ksmbd

# resinstall ksmbd kmod
cd /tmp
wget https://github.com/jkool702/openwrt-custom-builds/raw/main/WRX36/bin/targets/qualcommax/ipq807x/packages/kmods/kmod-fs-ksmbd_6.1.65-1_aarch64_cortex-a53.ipk
opkg update
opkg install --force-reinstall /tmp/kmod-fs-ksmbd_6.1.65-1_aarch64_cortex-a53.ipk
depmod -a

# ensure SMBUSER exists as a valid user
grep SMBUSER </etc/passwd || echo 'SMBUSER:x:2468:2468:SMBUSER:/mnt:/bin/false' >> /etc/passwd
grep SMBUSER </etc/group || echo 'SMBUSER:x:2468:SMBUSER' >>/etc/group
grep SMBUSER </etc/shadow || echo 'SMBUSER:x:0:0:99999:7:::' >>/etc/shadow

# give SMBUSER a system password
passwd -u SMBUSER
echo "$SMBPASSWORD" | passwd SMBUSER

# give SMBUSER the same password as its ksmbd password
ksmbd.adduser -d SMBUSER
ksmbd.adduser -a SMBUSER -p "$SMBPASSWORD"

# setup config for ksmbd (1/2)
cat<<EOF>/etc/ksmbd/ksmbd.conf.template
[global]
        netbios name = ${host_name}
        server string = ${host_name}
        workgroup = |WORKGROUP|
        interfaces = |INTERFACES|
        bind interfaces only = yes
        ipc timeout = 20
        deadtime = 15
        map to guest = never
        cache read buffers = no
        cache trans buffers = no
        server min protocol = SMB3_11
        server multi channel support = yes
EOF

# setup config for ksmbd (2/2)
cat<<EOF>/etc/config/ksmbd
config globals
        option workgroup 'WORKGROUP'
        option interface 'lan'
        option description 'OpenWrt_WRX36'

config share
        option name '${share_name}'
        option path '${share_mnt}'
        option read_only 'no'
        option users 'SMBUSER'
        option guest_ok 'no'
        option create_mask '0666'
        option dir_mask '0777'
EOF

# restart services
service wsdd2 enable
service ksmbd enable
service wsdd2 stop
service ksmbd stop
service wsdd2 start
service ksmbd start
1 Like

Hey, I really appreciate that you made some effort to make this custom firmware! Did you also enable the hardware offloading features. For example NSS Offloading? Is hardware encryption possible, too? And can you add snort into the build (if it is security-wise suggested). My goal is to buy this router and get an all-in-one secured system.

Very nice work, I don't have this device but looking through your kernel tweaks finding them interesting.
Btw, I added a link to your build on the DL-WRX36 wiki page.

Hi! Long time lurker here.

Installed this build dated Dec 8 2023. I don't know how you plan to release upgrades, unless we're bound to burn them ourselves.

I have some minor concerns on this build:

  1. No luck with InitRAMFS

I had to download the InitRAMFS from the Installation Instructions instead of the one supplied in the repository. The latter just hang up on boot with the purple light and couldn't connect to any address, no matter what port.

  1. Consider using UBI block devices

I get a lot of logs with this line:

Fri Dec  8 13:37:28 2023 kern.warn kernel: [   10.105851] mtdblock: MTD device 'rootfs' is NAND, please consider using UBI block devices instead.

Dunno what means.

  1. IRQ

Also, more lines with this:

Fri Dec  8 13:38:40 2023 daemon.warn /usr/sbin/irqbalance: Cannot change IRQ 34 affinity: Invalid argument
Fri Dec  8 13:38:40 2023 daemon.warn /usr/sbin/irqbalance: Cannot change IRQ 63 affinity: Invalid argument

Fixed by these instructions:

What does this fix, anyway?

  1. Low space

Also, it fathoms me how the build only sets around 35~MB of free space for packages given the 256MB NAND onboard, unless I'm missing several things about the device storage.

Dunno what can be made to have more space if I want to install some software later. Last time I tried to install SMB4 on my WNDR3800 it used 9MB.

  1. KMods

I have most of the kmod packages built for the custom kernel. You can find them here 4. Should you need a package requiring a kmod Id try and make sure you install the one from my github repo and not the main openwrt one.

How do I point that on OPKG sources? Should I install them manually by using their URL?

  1. IPs

I think it should be set to the usual Class C space for the sake of standards. I'm a fan of 10.0.0.0 but the Dynalink Firmware doesn't like Class C, dunno about the other. So that's my take, it should use 192.168.1.1 by default.

  1. No DNS and DHCP tab

That. It seems that's because dnsmasq is not installed, hence Luci things there is no DNS and DHCP to manage, which is wrong. An upstream bug, or can be fixed by installing a Luci package for odhcpd? By the way, it should appear given this piece of code that checks if dnsmasq or odhcpd is present as a feature to show the aforementioned tab.

  1. AdGuard DNS is better?

I don't know if using the public DNS from AdGuard is better than running adblock on your router. Theoretically, it should be faster for the devices, in exchange of holding the services at router-level, plus blacklist/whitelist option.

I would rather use something smaller/simplier for blocking via DNS.

image

That AdGuard DNS can be any DNS setup, like Google, Cloudflare, etc.

Dunno if that is something feasible with adblock, unbound and odhcpd.


Apart from that, overall, the build is extremely feature-complete, even the Plex Server is a great touch. The only thing I missed was Transmission (I'm open to hear any alternatives), thus I had to install it separately.

This device should be awesome given the CPU power and 1GB of RAM. I don't know if that tempfs with half the ram is going to be useful. I would rather have 768MB of RAM and leave 256MB (or less) for temp files, like... what could fit there, except for cache files?

It would be great to have a source code with your full config to create exact the same firmware but with baked own packages. Or just instructions how to do that. For now I’m experimenting with nss builds, but this one is working great too.

1 Like

Also, found a huge security flaw as ksmbd or something decided to share the root filesystem:

image

Old stuff, normal.

Current irqbalance is more verbose about certain internal errors than earlier versions. 1.9.3-2 fixed most of the spam.

Hi hnyman!

Off topic, I used your WNDR3800 build for ten years. Finally, I decided to upgrade it to a DL-WRX36. Wonderful job, but I'm like more using stable releases rather than bleeding edge since I don't upgrade devices unless for bugs or performance.

Back to this, I currently trying to get a grasp of a custom build for this with Transmission, Samba4. While this works wonders, there are some problems I wrote earlier, excepting MTD and IRQ things.

Also, are there more kernel flags to squeeze CPU performance? How we can know them?

I thought we could not use irqbalance for the DL-WRX36 but have to set it manually?

Is this custom build with the latest version of OpenWRT 23.05.2

irqbalance seems to work for me so long as its UCI config only contains

config irqbalance 'irqbalance'
    option enabled '1'

and nothing else. That said, when it starts up, it logs that all the irq's it handles (9, 13, 16-21, and 23-70) it guesses as class 0. Im not sure if this is expected and correct behavior or if it means something isnt working right.

Its based on a snapshot from a few months ago. I think that it is from about the same time as 23.05.2 (last november-ish iirc) but uses snapshot (main github branch) not 23.05 (meaning target is qualcommax, not ipq807x). It is a few months behind current "bleeding edge" snapshots though.