Full-featured custom build for Dynalink DL-WRX36 (AX3600)

Community Builds, Projects & Packages
1

I compiled from source a custom firmware image for the WRX36. I'm pretty happy with how it is performing, and figured Id share it.

LINK to firmware (hosted on github)

The firmware images are in the targets/qualcommax/ipq807x folder. There are also a whole bunch of "extra" .ipk packages that I built but didnt install (including most of the possible kernel modules).

If you have any comments / suggestions / bug-reports let me know. I hope some of you will find this useful.

It is worth mentioning that this firmware is a bit different from a "standard" build in several ways. Refer to the extras folder for the exact .config files and custom configuration files that went into the build, but I'll highlight a few key differences here:

1. DNS/DHCP

It uses unbound (with DNSSEC) + odhcpd for DNS/DHCP (note: dnsmasq isnt installed). Adblock is also included, which integrates automatically with unbound. This should all be set up and ready to go on the first bootup.

Note: unbound has been compiled with everything optional (except python support) compiled in. It is set up to default to be a validating recursive resolver, with settings more aggressive than the "max" openwrt preset.

2. Compiler optimizations

It was compiled with -03 -march=cortex-a53+crc+crypto optimization flags with link time optimization (LTO) + dead code elimination using mold as the linker

3. Full software suite

It probably has more than you'll really need/want (the firmware image is ~50 mb), but i figure that Im better off compiling everything I might want to use at some point into the squashfs image (where it takes up a quarter of the space) than installing it on the overlay later on.

It includes (but isnt limited to):

  • full versions of several network utilities (wpan, ip, iw, ethtool),
  • ksmbd support + code to easily run an on-router plex media server + drivers/kmods for usb-attached storage
  • bash + vim + other miscellanious useful cli programs
  • support for various network protocols and ffile transfer protocols: SNMP, TFTP, full-featured wget and curl

4. Custom kernel

Kernel settings were tweaked in a number of ways: enabling only cortex-a53 erratum, enabling better multithreading support, enabling memory compaction, enabling crypto functions that use armv8 crypto extensions, fixing a bug where the assembler 's -march and the -DASM_ARM_ARCH flag were being incorrectly set to armv8.5-a`, etc. (note: i didnt fool with the "network" or "drivers" sections of the kernel config)

One last noteworthy kernel tweak is that I made the kernel fully preemptable and set it to use a 1000 Hz timer. This may have dropped the total throughput of the kernel slightly, but it can still handle my 1000/500 fiber connection and has done wonders for latency jitter. a few speedtests are provided in the repo's "extras" folder, but to give an example: running

 ping -c 100 -U -A gstatic.com

the overall statistics for the set of all 100 pings was

--- gstatic.com ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 2295ms
rtt min/avg/max/mdev = 21.851/21.974/22.120/0.060 ms, ipg/ewma 23.177/21.955 ms

The overall spread for all 100 pings was about 1/4 of a second, and the std deviation was only 60 ms.

speedtest-petperf.sh was giving me results like

 Download: 900.25 Mbps
  Latency: [in msec, 31 pings, 0.00% packet loss]
      Min:  21.900
    10pct:  22.000
   Median:  22.700
      Avg:  24.352
    90pct:  28.900
      Max:  31.000

Indicating at 90% saturation of my connection my pings were on average about 2.5 ms and at worse about 9ms. Not bad.
The

5 Likes
2

Can I still use opkg to install whatever I need, like dnsmasq-full, wireguard*, openvpn*, etc?

3

Yeah. The standard

opkg update
opkg install $PKG

works, and will point you to the aarch64_cortex-a53 openwret repos.

one warning: I didnt realize this until just now, but it also points you to the targets/qualcommax/ipq807x repo that has kmods. This could be dangerous, in the sense that it might allow you to install kmod packages that arent actually comparable with the firmware's custom kernel.

I have most of the kmod packages built for the custom kernel. You can find them here. Should you need a package requiring a kmod Id try and make sure you install the one from my github repo and not the main openwrt one.

But that is just for kmods...for normal packages the standard openwrt repos should work just fine.

4

Hi!
I have an issue while trying to install "kmod-ipt-ipopt" from your kmods folder on github. Any help?

Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.1.65-1-1763aacc7248702f887609f8c80e68c4) for kmod-ipt-ipopt
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-ipt-ipopt found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-ipt-ipopt.
5

How are you installing it? I tried (with the opkg --noaction flag) and it seemed to work. However, i did get that message when i tried to install it from the openwrt repos.

Try running

cd /tmp
wget https://github.com/jkool702/openwrt-custom-builds/raw/main/WRX36/bin/targets/qualcommax/ipq807x/packages/kmods/kmod-ipt-ipopt_6.1.65-1_aarch64_cortex-a53.ipk
opkg install /tmp/kmod-ipt-ipopt_6.1.65-1_aarch64_cortex-a53.ipk
rm /tmp/kmod-ipt-ipopt_6.1.65-1_aarch64_cortex-a53.ipk

Side note: on my WRX36 running this firmware, I've discovered that the ntpd and sysntpd service appear broken, and were causing some issues. running

service ntpd disable
service sysntpd disable
reboot

Fixed these issues. If you're not sure if they are working or not, run logread. If you see any entries starting with

daemon.err ntpd

then your ntpd is broken too.

If you see a bunch of

user.err : jail: failed to load dependencies

Then your sysntpd is broken too.

6

I was installing it through luci (upload package), but your method seems to be working without errors. Looking at ntpds and sysntpd in logread; it doesn't output any errors. But for now I'm struggling with ksmbd, I've changed the name there and for some reason i still can't connect.

daemon.err ksmbd: modprobe of ksmbd module failed, can\'t start ksmbd!
  • I have some errors from irqbalance
daemon.warn /usr/sbin/irqbalance: Cannot change IRQ 34 affinity: Invalid argument
daemon.warn /usr/sbin/irqbalance: Cannot change IRQ 63 affinity: Invalid argument
7

I got these too. I think it is an error with how it parses luci. In /etc/config/irqbalance comment out everything exceppt the "enabled" option, so that the only un-commented lines are

config irqbalance 'irqbalance'
        option enabled '1'

This fixed it for me - it now runs without complaining about any IRQs.

Thats....weird.

ksmbd is running and working on my wrx36 (accessible from both windows and linux machines).

I wrote you a little script that (if it works like it is supposed to) should hopefully fix your issue (by reinstalling ksmbd) and will set you up with the same working config I have for ksmbd.

before running it, set the 3 paramaters at the top of the script that determine password, share name, and ksmbd root directory.

Note: this sets up password protected smb access. To access the share, use

user=SMBUSER
password=<WHATEVER YOU USED AS THE PASSWORD>
(if needed) domain/workgroup=WORKGROUP

SCRIPT

# if not already using a bash shell run this line first
exec /bin/bash

# replace ______ with desired password for SMB
# id recommend using all caps here...IIRC it works better 
SMBPASSWORD=______

# replace ______ with share name
share_name=______

# replace ______ with root path for share
# if empty itll assume it is at /mnt/${share_name}
share_mnt=______


#####################################################################


{ [[ -z ${share_mnt} ]] || [[ "${share_mnt}" == '______' ]]; } && share_mnt="/mnt/${share_name}"

host_name="$(uci get system.@system[0].hostname)"
host_name="${host_name^^}"

mkdir -p /etc/ksmbd

# resinstall ksmbd kmod
cd /tmp
wget https://github.com/jkool702/openwrt-custom-builds/raw/main/WRX36/bin/targets/qualcommax/ipq807x/packages/kmods/kmod-fs-ksmbd_6.1.65-1_aarch64_cortex-a53.ipk
opkg update
opkg install --force-reinstall /tmp/kmod-fs-ksmbd_6.1.65-1_aarch64_cortex-a53.ipk
depmod -a

# ensure SMBUSER exists as a valid user
grep SMBUSER </etc/passwd || echo 'SMBUSER:x:2468:2468:SMBUSER:/mnt:/bin/false' >> /etc/passwd
grep SMBUSER </etc/group || echo 'SMBUSER:x:2468:SMBUSER' >>/etc/group
grep SMBUSER </etc/shadow || echo 'SMBUSER:x:0:0:99999:7:::' >>/etc/shadow

# give SMBUSER a system password
passwd -u SMBUSER
echo "$SMBPASSWORD" | passwd SMBUSER

# give SMBUSER the same password as its ksmbd password
ksmbd.adduser -d SMBUSER
ksmbd.adduser -a SMBUSER -p "$SMBPASSWORD"

# setup config for ksmbd (1/2)
cat<<EOF>/etc/ksmbd/ksmbd.conf.template
[global]
        netbios name = ${host_name}
        server string = ${host_name}
        workgroup = |WORKGROUP|
        interfaces = |INTERFACES|
        bind interfaces only = yes
        ipc timeout = 20
        deadtime = 15
        map to guest = never
        cache read buffers = no
        cache trans buffers = no
        server min protocol = SMB3_11
        server multi channel support = yes
EOF

# setup config for ksmbd (2/2)
cat<<EOF>/etc/config/ksmbd
config globals
        option workgroup 'WORKGROUP'
        option interface 'lan'
        option description 'OpenWrt_WRX36'

config share
        option name '${share_name}'
        option path '${share_mnt}'
        option read_only 'no'
        option users 'SMBUSER'
        option guest_ok 'no'
        option create_mask '0666'
        option dir_mask '0777'
EOF

# restart services
service wsdd2 enable
service ksmbd enable
service wsdd2 stop
service ksmbd stop
service wsdd2 start
service ksmbd start
1 Like
8

Hey, I really appreciate that you made some effort to make this custom firmware! Did you also enable the hardware offloading features. For example NSS Offloading? Is hardware encryption possible, too? And can you add snort into the build (if it is security-wise suggested). My goal is to buy this router and get an all-in-one secured system.

9

Very nice work, I don't have this device but looking through your kernel tweaks finding them interesting.
Btw, I added a link to your build on the DL-WRX36 wiki page.

10

Hi! Long time lurker here.

Installed this build dated Dec 8 2023. I don't know how you plan to release upgrades, unless we're bound to burn them ourselves.

I have some minor concerns on this build:

  1. No luck with InitRAMFS

I had to download the InitRAMFS from the Installation Instructions instead of the one supplied in the repository. The latter just hang up on boot with the purple light and couldn't connect to any address, no matter what port.

  1. Consider using UBI block devices

I get a lot of logs with this line:

Fri Dec  8 13:37:28 2023 kern.warn kernel: [   10.105851] mtdblock: MTD device 'rootfs' is NAND, please consider using UBI block devices instead.

Dunno what means.

  1. IRQ

Also, more lines with this:

Fri Dec  8 13:38:40 2023 daemon.warn /usr/sbin/irqbalance: Cannot change IRQ 34 affinity: Invalid argument
Fri Dec  8 13:38:40 2023 daemon.warn /usr/sbin/irqbalance: Cannot change IRQ 63 affinity: Invalid argument

Fixed by these instructions:

What does this fix, anyway?

  1. Low space

Also, it fathoms me how the build only sets around 35~MB of free space for packages given the 256MB NAND onboard, unless I'm missing several things about the device storage.

Dunno what can be made to have more space if I want to install some software later. Last time I tried to install SMB4 on my WNDR3800 it used 9MB.

  1. KMods

I have most of the kmod packages built for the custom kernel. You can find them here 4. Should you need a package requiring a kmod Id try and make sure you install the one from my github repo and not the main openwrt one.

How do I point that on OPKG sources? Should I install them manually by using their URL?

  1. IPs

I think it should be set to the usual Class C space for the sake of standards. I'm a fan of 10.0.0.0 but the Dynalink Firmware doesn't like Class C, dunno about the other. So that's my take, it should use 192.168.1.1 by default.

  1. No DNS and DHCP tab

That. It seems that's because dnsmasq is not installed, hence Luci things there is no DNS and DHCP to manage, which is wrong. An upstream bug, or can be fixed by installing a Luci package for odhcpd? By the way, it should appear given this piece of code that checks if dnsmasq or odhcpd is present as a feature to show the aforementioned tab.

  1. AdGuard DNS is better?

I don't know if using the public DNS from AdGuard is better than running adblock on your router. Theoretically, it should be faster for the devices, in exchange of holding the services at router-level, plus blacklist/whitelist option.

I would rather use something smaller/simplier for blocking via DNS.

image

That AdGuard DNS can be any DNS setup, like Google, Cloudflare, etc.

Dunno if that is something feasible with adblock, unbound and odhcpd.

Apart from that, overall, the build is extremely feature-complete, even the Plex Server is a great touch. The only thing I missed was Transmission (I'm open to hear any alternatives), thus I had to install it separately.

This device should be awesome given the CPU power and 1GB of RAM. I don't know if that tempfs with half the ram is going to be useful. I would rather have 768MB of RAM and leave 256MB (or less) for temp files, like... what could fit there, except for cache files?

11

It would be great to have a source code with your full config to create exact the same firmware but with baked own packages. Or just instructions how to do that. For now I’m experimenting with nss builds, but this one is working great too.

1 Like
12

Also, found a huge security flaw as ksmbd or something decided to share the root filesystem:

image

13

Old stuff, normal.

Current irqbalance is more verbose about certain internal errors than earlier versions. 1.9.3-2 fixed most of the spam.

14

Hi hnyman!

Off topic, I used your WNDR3800 build for ten years. Finally, I decided to upgrade it to a DL-WRX36. Wonderful job, but I'm like more using stable releases rather than bleeding edge since I don't upgrade devices unless for bugs or performance.

Back to this, I currently trying to get a grasp of a custom build for this with Transmission, Samba4. While this works wonders, there are some problems I wrote earlier, excepting MTD and IRQ things.

Also, are there more kernel flags to squeeze CPU performance? How we can know them?

15

I thought we could not use irqbalance for the DL-WRX36 but have to set it manually?

16

Is this custom build with the latest version of OpenWRT 23.05.2

17

irqbalance seems to work for me so long as its UCI config only contains

config irqbalance 'irqbalance'
    option enabled '1'

and nothing else. That said, when it starts up, it logs that all the irq's it handles (9, 13, 16-21, and 23-70) it guesses as class 0. Im not sure if this is expected and correct behavior or if it means something isnt working right.

18

Its based on a snapshot from a few months ago. I think that it is from about the same time as 23.05.2 (last november-ish iirc) but uses snapshot (main github branch) not 23.05 (meaning target is qualcommax, not ipq807x). It is a few months behind current "bleeding edge" snapshots though.

19

UPDATE - NEW CUSTOM NSS-ENABLED BUILD IS NOW AVAILABLE

*I have released an update to my custom openwrt build for the dynalink wrx36 that is now NSS-enabled! The original link now leads to this new build, but here is another link to it. It took a few tries to get a stable NSS-enabled build, but this build has (so far on my router) been working fantastic and is (seemingly) 100% stable with everything working as it should.

Note: If you havent already, be sure to set up usb recovery before flashing the NSS-enabled firmware. You shouldn't need it, but in case something goes wrong that makes recovery a piece of cake.

The README in the GitHub repo has been updated as well. This contains additional info, including some recommended first post-install steps to take. Ill give some of the highlights about the build here though.

Some info regarding the build

This build has been fairly heavily customized and has many features compiled in.

  1. has full wpad and all NSS packages, with 1 exception: no mesh support (even without nss). iwinfo + the full versions of iw, ethtool are also compiled in. NOTE: mesh is NOT supported with this build.
  2. usteer + 80211r/w/v are available and pre-setup and running out of the box. usteer is configured for band steering between the 2g and 5g wifi frequencies, but can easily be modified to steer to other openwrt ap's.
  3. it uses unbound+odhcpd for DNS+DHCP. dnsmasq is not installed. unbound has all the extra goodies (libpthreads, libevent....everything except python support) compiled in, and uses a customized startup script. Adblock is also pre-setup and running. adblock seamlessly integrates with unbound - it sets up an unbound blacklist and then unbound does the ad blocking).
  4. There are 2 LAN-side interfaces set up. LAN1-3 and WiFi are on interface "lan" / bridge "br-lan". LAN4 is on interface IoT. The idea is that you plug a 2nd router into LAN 4 and have IoT devices connect to that where they a) wont slow down your main network and b) will be on different wifi channels. NOTE: the firewall is setup so that devices on "IoT" can not access the router nor can they access devices on "lan". lan IP is 10.0.0.1/24. IoT IP is 192.168.0.1/24.
  5. WiFi is on the lan interface with default name "OpenWrt_WiFi" and default password "password"
  6. ksmbd is compiled in and mostly pre-setup for password-protected smb3 file sharing. user/pass/workgroup is SMBUSER / SMBPASSWORD / WORKGROUP.
  7. LUCI is compiled in, and has some extra nice status monitoring packages installed (from the "fantastic packages" github repo), plus control for nbound, ksmbd, adblock, etc.
  8. A bunch of shell tools are compiled in (bash, many coreutils, htop, atop, various compression + checksum tools, etc.). Several miscellaneous network tools / protocols (e.g., gre, rd6, ipip6 tunnels) are also compiled in.
  9. Numerous kernel-level tweaks have been made. These include (but are not limited to): making the kernel fully preemptable, increasing the interrupt timer to 1000 hz, automatic cgroup scheduling (that favors dynamic latency reduction), zswap is enabled (zsmalloc+zstd), all the crypto (except those requiring armv8.2 crypto acceleration) are either compiled in or available as a kmod, etc.
  10. chrony provides NTP service (getting the time. It defaultstop using a stratum 1 time server with NTS enabled
  11. My plex media server init script is compiled in and ready to go. It assumes that you will mount the external hdd at /mnt/PLEX, but this can be changed in the plexmediaserver UCI config.

Should you wish to try and build this yourself, the github repo includes a document that outlines my build process / instructions.

6 Likes
20

Nicely detailed changelog very interesting. Could you explain the move to unbound? Dnsmasq seems like a major component of OpenWrt, why move to unbound? Think the project as a whole should go that route (hypothetically speaking)?