Rpi4 < $(community_build)

Hello there, been using your build and enjoying it a lot!

Just wanted to say thanks for all the effort in getting it to work, and also a few questions.

Up until SNAPSHOT r17189-1ed9fc663e, I was able to use dockerd from Entware and host several containers from within the router, but after upgrading yesterday it seems that the kernel hash started mismatching between the installed version and the GitHub/OpenWRT opkg repositories,as trying to install the dependencies for dockerd now (I have no idea why them and several other packages uinstalled themselves) reports the following error log:

[root@RPi4-OpenWRT /]$ opkg install kmod-lib-lzo kmod-lib-zlib-inflate kmod-lib-zlib-inflate kmod-lib-zlib-deflate kmod-lib-raid6 kmod-lib-xor kmod-lib-zstd kmod-fs-btrfs  kmod-dax kmod-random-core kmod-tpm kmod-keys-trusted kmod-keys-encrypted kmod-dm kmod-br-netfilter kmod-nf-ipvs
Unknown package 'kmod-lib-lzo'.
Unknown package 'kmod-lib-zlib-inflate'.
Installing kmod-lib-zlib-inflate (5.4.132-1) to root...
Downloading https://downloads.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-8c34d2a86d75b793acaec716ad3d8596/kmod-lib-zlib-inflate_5.4.132-1_aarch64_cortex-a72.ipk
Unknown package 'kmod-lib-zlib-deflate'.
Unknown package 'kmod-lib-raid6'.
Unknown package 'kmod-lib-xor'.
Unknown package 'kmod-lib-zstd'.
Unknown package 'kmod-fs-btrfs'.
Unknown package 'kmod-dax'.
Unknown package 'kmod-random-core'.
Unknown package 'kmod-tpm'.
Unknown package 'kmod-keys-trusted'.
Unknown package 'kmod-keys-encrypted'.
Unknown package 'kmod-dm'.
Unknown package 'kmod-br-netfilter'.
Unknown package 'kmod-nf-ipvs'.
Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-lzo
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-lzo found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-lib-lzo.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-zlib-inflate
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-zlib-inflate found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-lib-zlib-inflate.
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-lib-zlib-inflate:
 *      kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596)
 * opkg_install_cmd: Cannot install package kmod-lib-zlib-inflate.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-zlib-deflate
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-zlib-deflate found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-lib-zlib-deflate.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-raid6
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-raid6 found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-lib-raid6.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-xor
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-xor found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-lib-xor.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-zstd
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-zstd found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-lib-zstd.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-fs-btrfs
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-fs-btrfs found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-fs-btrfs.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-dax
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-dax found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-dax.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-random-core
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-random-core found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-random-core.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-tpm
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-tpm found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-tpm.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-keys-trusted
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-keys-trusted found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-keys-trusted.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-keys-encrypted
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-keys-encrypted found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-keys-encrypted.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-dm
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-dm found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-dm.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-br-netfilter
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-br-netfilter found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-br-netfilter.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-nf-ipvs
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-nf-ipvs found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-nf-ipvs.

Given that seemingly nothing else changed, I'm not sure how I should go about debugging this particular problem.

If there's anything I forgot to add, please let me know.

Kudos!

1 Like

glad I was able to help..

1 Like

on openwrt an upgrade is a "flash"... so it's likely nothing was uninstalled... just not re-installed automatically for you...

as for the other stuff... its possible there is some anomaly with the repo versioning or something... but as you mention entware... i'm wondering if I understand whats going on properly...

i've never had issue with opkg install before... so i'll test your install command and double check the repo consistency... after that we may look further into the use case and what else may be going on...

edit: works fine for me

kmods-install-ok
opkg install kmod-lib-lzo kmod-lib-zlib-inflate kmod-lib-zlib-inflate kmod-lib-zlib-deflate kmod-lib-raid6 kmod-lib-xor kmod-lib-zstd kmod-fs-btrfs  kmod-dax kmod-random-core kmod-tpm kmod-keys-trusted kmod-keys-encrypted kmod-dm kmod-br-netfilter kmod-nf-ipvs
Package opkg (2021-06-13-1bf042dd-3) installed in root is up to date.
Installing kmod-lib-lzo (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-lzo_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-lib-zlib-inflate (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-zlib-inflate_5.4.132-1_aarch64_cortex-a72.ipk
Package kmod-lib-zlib-inflate (5.4.132-1) installed in root is up to date.
Installing kmod-lib-zlib-deflate (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-zlib-deflate_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-lib-raid6 (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-raid6_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-lib-xor (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-xor_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-lib-zstd (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-zstd_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-fs-btrfs (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-fs-btrfs_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-dax (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-dax_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-random-core (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-random-core_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-tpm (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-tpm_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-keys-trusted (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-keys-trusted_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-keys-encrypted (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-keys-encrypted_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-dm (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-dm_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-br-netfilter (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-br-netfilter_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-nf-ipvs (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-nf-ipvs_5.4.132-1_aarch64_cortex-a72.ipk
Configuring kmod-lib-zlib-deflate.
Configuring kmod-br-netfilter.
Configuring kmod-lib-xor.
Configuring kmod-dax.
Configuring kmod-random-core.
Configuring kmod-tpm.
Configuring kmod-keys-trusted.
Configuring kmod-keys-encrypted.
Configuring kmod-dm.
Configuring kmod-lib-zlib-inflate.
Configuring kmod-lib-raid6.
Configuring kmod-lib-lzo.
Configuring kmod-lib-zstd.
Configuring kmod-fs-btrfs.
Configuring kmod-nf-ipvs.

based on the above... i'd say you possibly have an entware related dependency issue... ( i.e. 'kernel dependant package on extroot syndrome' )... typically... entware should not be needed... and if I were doing the same... i'd just be using regular opkg for anything docker related...

just out of curiosity... how many times in total have you upgraded so far? what version did you upgrade from?

rpi4-installhistory.sh
rpi-support.sh | grep -A30 'restore-actions'
dmesg | grep -C2 -i expand

Hey, thanks for the quick reply!

To the current installation, I have upgraded just once coming from Snapshot-26567-3.2.61-70-r17073, the only unusual change I made to the image was before the ROOTFSEXPAND variable was supported was to manually expand the filesystem using the following commands:

mount -o remount,ro / #Remount root as ReadOnly
tune2fs -O^resize_inode /dev/mmcblk0p2 #Remove reserved GDT blocks
fsck.ext4 /dev/mmcblk0p2 #Fix part, answer yes to remove GDT blocks remnants
fdisk /dev/mmcblk0
fdisk > p #list partitions
fdisk > d #delete the second partition, keeping track of sector start
fdisk > n 
fdisk > p
#create new primary partition from previous sector start and end at disk end
fdisk > w #write new partition table to disk
reboot
resize2fs /dev/mmcblk0p2

Also, I added the variable to rc.local before the upgrade just to be sure, and took a backup for safekeeping.

The mSD card I'm using as a boot device is the following one:
SanDisk Extreme microSDXC UHS-I SDSQXA2-064G-GN6MA

Thanks for the help and Kudos!

1 Like

they go in /root/wrt.ini

rc.local calls most custom build logic... alterations from the build default could render most build related stuff non-functional... (including the firstboot opkg feed file creation logic)

the stuff above should not be related to EXPAND...

suggest you;

cp /etc/rc.local.copy /etc/rc.local

and upgrade again... to ensure all build related stuff operates properly...

1 Like

Alright, I see...

In cases like the IRQAFFINITY, ENABLEDSERVICES or PERFTWEAKS variables, should I move my changes as well to a different file?

Mostly out of concern for Cake_QoS SQM and packet affinity tweaks. At the moment the modifications I've done are the following:

POWERPROFILE="quickest"
#!20210330 new users get this (for now) #@!> needs bin/x.sh stop
#equivalent to FWCUSTOM aka comment out to disable all ( for now ) later will require the subactionVARS also
PERFTWEAKS=1
#PERFTWEAKS="default"
PFAFFINITY="32:1 33:2 26:f"
#PFRENICE=1
#PFSERVICECPU=1 #taskset some procs to cpu's
1 Like

should already be set for you, commenting it out will likely mean you dont get any perftweaks at all... the affinity stuff was disabled (technically hardcoded) due to little or inadequate feedback as mentioned in the top post/s ( but no harm in leaving it )

only POWERPROFILE is currently honoured... and possibly PACKETSTEERING=1 i'd need to double check those...

Perfect, let me give it a go and I'll report back.

Thanks for the help!

Edit: Looks like it worked correctly! Thanks for the help. Now to continue struggling with NGINX :woozy_face:

1 Like

no problem at all... thankyou for reporting back and the detailed report... it's very useful to me to get detailed bug/problem reports and I learn alot from them...

in this case you've stumbled/wrestled with two of the 'build specific' 'must-haves'... they can be a little bit of a pain at first especially for advanced users...

fwiw for rc.local type stuff... you can... do something like...;

cp /etc/custom/dca632563177.sh /etc/custom/YOURMAC.sh
echo '/etc/custom/YOURMAC.sh' >> /etc/sysupgrade.conf

this will give you your own per device script that gets stored in upgrades...

the next time you upgrade... your docker packages and kmods should re-install automatically for you... if they do not... feel free to post here and we can take a look...

cheers... :+1:

1 Like

In terms of efficiency I've found that if you're not pushing the bounds with speed that it's been more beneficial to leave everything pinned to the first CPU. My own speed is 80/25 Mbit/s so I've also disabled all offloading for all devices/interfaces with an iface hotplug script. I was having a lot of problems with strange fluctuations where ping would skyrocket and throughput would drop dramatically relatively frequently, some odd consequence of having SQM and a VPN running in tandem; as with one or the other it wouldn't crop up.

Also, as an aside wulfy, which files would I plumb from the install image to get a handle on your custom general tweaks (e.g: processes pinned to CPU, etc.)?

1 Like

the current stripped down tweaks are in the same spot... although as you mention are stripped down to some bare minimals...

/bin/rpi-perftweaks.sh

it used to print out everthing it does properly but when I had to hack it back to something somple I've hadnt time to do yet and as it's a 'DUMMY' version haven't been bothered

one eth interrupt was moved for the last two builds due to @sammutd88 s suggestion but its been moved back for the next as he did not provide the proof of benefit from this change...

the only change since the real scripts were pulled is to lookup interript numbers dynamically...

as CM4 / enabling dwc2 etc. etc. alters numbering...


one thing i've noted recently that should possibly in an ideal world be addressed ( via some sysctls ) is dd'ing 1G zeros hangs luci... but this is mostly only beneficial for people who use alot of samba or something that heavily writes to disks and does not have a huge impact on routing as a whole...


speaking of this... some 'perftweaks' are also derived from '/etc/custom/firstboot/01-sysctl' also...

1 Like

Thanks. Alright, it's good to know that I already had a grasp of things. Just to clarify, the /boot/config.txt addition for DWC is for CM4 only? Are there any dtb additions suggested for the classic RPi4 config?

My previous anecdote, I'll add a bit more clarity as I realise that I worded it somewhat poorly. As an aside, though, I'll note that my experiences are from images built by myself, fairly light on packages as I prefer things to be as lean as possible and I prefer to go as long as possible without opening the terminal or LuCi. As mentioned, with SQM and VPN in tandem I was seeing bufferbloat spikes and bizarre throughput drops. This held true across a number of VPN servers and even when dropping SQM limitations far below actual line speed, despite CPU having a good deal of overhead. The main factors that I found to resolve this were to leave all interrupts as default on CPU0 and to disable offloading everywhere. There were a few other minor tweaks but I don't recall in my testing that they provided much benefit to resolving this particular problem. The only other benefit of note that comes to mind is banIP, I didn't realise that port-scanning and random pings were so prevalent and can only presume that the little bit of time to process these instead of immediately dropping them caused the issues I saw. My main 'real-world' performance metric was to gauge consistency in Qbittorrent seeding (upload) performance which seems heavily dependent on latency and whatever else due to it being wholly TCP. The before and after of these changes is remarkable.

1 Like

I had a failed attempt at adding it for the rpi4... and for the CM4... right now based on CM4 users input... i'm merely suggesting it...

however... for the last three builds i've injected pci-bus perf onto the command line... ( there is a similar config.txt parameter but i'm yet to determine if its reqired or one overrides the other etc. )

dtoverlay=disable-wifi

no real performance related dtbo mods are applied... from the top of my head... although for most users they would benefit from dtoverlay=disable-wifi as a means of lowering temp and interrupts...

yes... i also agree/have witnessed poorer latency when shifting interrupts from cpu0


for banip(or any other application related perf hits) i'll have a closer look at in my travels... it's possible my environment doesn't trigger the right levels to adequately reveal alot of things...


[quote="Mint, post:1110, topic:69998"] prefer things to be as lean as possible [/quote]

for this philosophy although the benefits may be negligable... I provide;

RMMOD=""

in /root/wrt.ini to trash several unused loaded kmods during runtime... or /etc/packagesremove.txt could be used to the same effect...

1 Like

Nice. I'm unfamiliar with the concept of shifting pci-bus to the command line, could you please expand upon that? Yeah, WiFi seems a real problem. I use it as a minor sort of thing novelly due to the RPi's positioning, but performance under load is atrocious. It might be a product of RC3 support (or lack thereof), but taxing throughput via WiFi on a Macbook caused all kinds of weird errors and CPU load to skyrocket to a degree that seems outside of norms.

1 Like

i'd elaborate if I really knew what I was talking about...

but as I don't... these are the options for looking up

for cmdline.txt

 pci=pcie_bus_perf 

for config.txt

dtparam=axiperf=on

only stumbled upon these after seeing the CM4 dtb is slightly different in that it beefs up pci-e bandwidth or something like that...

yeah... i've also seen OS-level performance GAINS and DROPS based on whats going on with wifi on this board... ( i.e. actually enabling it even if not using it seemed to increase overall OS function rather than leaving it in a disabled(uci) state )

1 Like

Interesting, I'll certainly see if integrating one or both of those pci configs provides noticeable benefit when I've some spare time. That's damned peculiar with the WiFi, but such is the way of much of this as I've found through trial and error. How does it fare with respect to br-lan vs. not (eth0 instead)? Perhaps the software bridge provides some measure of benefit? Just spitballing as I've never disabled wlan, hence I've always had the br-lan interface.

1 Like

funny you mention that... until yesterday I had no idea this could lead to possible perf gains...

but added it to my 'to-do' (move wan to br-wan) list after someone on another thread(board) mentioned this exact phenomenon... ( in their case... the software bridges performed better i believe )

1 Like

Interesting. I've never thought about the prospect of adding WAN to a bridged interface, definitely would be keen to see your results as I've no idea on how to properly go about doing so myself as yet. Also, on the note of banIP, I've integrated the following default lists: firehol1, firehol2, firehol3, iblockads, iblockspy, threat, whitelist, yoyo for both WAN and Wireguard Client interface. At two weeks uptime I'd say I'm at least 100,000 hits from the IPSet report, just for a frame of reference.

1 Like

my environment/load is probably not enough... but if there is any perf hit from banip...

it's typically as a result of the report feature (which I think you already mentioned need time to re-read and let it sink in)

so definately test with that off for a while... i'm not sure what/if any sysctls might assist if this is the case... maybe file:max_nr(probably not) or min_rsv(mem) aka settings for process memory allocation and pruning (otoh) might do something...?

generally tho... i'm running similar lists but less overall load on the router and I don't see noticable issues ( but I haven't really looked properly )...

(udp/conntrack sysctls may also be of gain for torrent style tests and uses)


there is some gremlin thats been bugging me though...

  1. in that periodically... web pages may fail to load/router function hangs for a few seconds... it's infrequent but something i'm tracking (could be related to me giving masq more memory or cache-ttl)

  2. have not really noticed this recently (last 5 or so builds) but after a reboot or new flash... responsiveness seems to magically speed up... so over time... something hits performance... likely candidate as above is the masq-cache settings or independant of masq... the ipset stack itself...


i'm not exactly 'taking-it-easy' with ipsets on this :scream: thing

Summary
for iSET in $(ipset -L -n); do echo "$iSET $(ipset -L $iSET | wc -l)"; done
BE4 8
BE6 8
Bulk4 8
Bulk6 8
Vid4 8
Vid6 8
Voice4 8
Voice6 8
Zoom4 8
Zoom6 8
bulk 95
bulk6 76
chunky4hit 8
chunky6hit 8
chunkyDOWN4_stk 8
chunkyDOWN6_stk 8
chunkyUP4_stk 8
chunkyUP6_stk 8
cloudflare-v4 22
cloudflare-v6 15
dshield_4 71
gamecache4 60
gamecache6 31
largehttp 8
largehttp2 8
latsens 9
latsens6 8
limiter4_conn 8
limiter6_conn 8
mwan3_connected_v4 11
mwan3_connected_v6 11
mwan3_custom_v4 8
mwan3_custom_v6 8
mwan3_dynamic_v4 8
mwan3_dynamic_v6 8
mwan3_sticky_v4_https 8
mwan3_sticky_v6_https 8
steam4 61
streaming 1136
streaming6 1670
usrcdn 508
usrcdn6 394
whitelist_4 10
whitelist_6 8
gamingdevice4 8
gamingdevice6 8
bogon_4 1386
bogon_6 123977
drop_6 46
drop_4 1038
threat_4 1319
darklist_4 9157
debl_6 83
yoyo_4 10188
debl_4 25667
firehol3_4 19186
voip_4 14997
cloudflare 10
latsensitive 10
mwan3_connected 14
mwan3_sticky_https 10
gamingdevice 10
### roughly 215302 entries
### dca632 /usbstick 48°# iptables-save -c | grep match-set | grep -v '0:0'
[6134:451012] -A QOS_MARK_F_eth1 -m set --match-set usrcdn dst -m comment --comment CDN4dstB-AF21 -j DSCP --set-dscp 0x12
[44:3200] -A QOS_MARK_F_eth1 -m set --match-set cloudflare-v4 dst -m comment --comment CLOUDFL4dst-CS3 -j DSCP --set-dscp 0x18
[104:6572] -A QOS_MARK_F_eth1 -m set --match-set gamecache4 dst -m comment --comment GAMECACHE4dst-CS6 -j DSCP --set-dscp 0x30
[1186:113640] -A QOS_MARK_F_eth1 -m set --match-set streaming dst -m comment --comment STREAMING4-CS3 -j DSCP --set-dscp 0x18
[773:68508] -A QOS_MARK_F_eth1 -m set --match-set latsens src -m comment --comment LATSENS4-SRC-CS4 -j DSCP --set-dscp 0x20
[237:14146] -A dscptagstatic -p tcp -m set --match-set bulk dst -m comment --comment STATIX-BKdsttcp4-CS1 -j DSCP --set-dscp 0x08
[116:22554] -A banIP -i eth1 -m set --match-set whitelist_4 src -j RETURN
[170:13792] -A banIP -o eth1 -m set --match-set bogon_4 dst -j REJECT --reject-with icmp-port-unreachable
[34:2040] -A banIP -o eth1 -m set --match-set darklist_4 dst -j REJECT --reject-with icmp-port-unreachable
[6:384] -A banIP -o eth1 -m set --match-set yoyo_4 dst -j REJECT --reject-with icmp-port-unreachable
1 Like

Ah, sorry, I may have conveyed the wrong notion with how I worded it once again. I meant, instead, that banIP afforded a benefit rather than a detriment. I can only guess on the why and how with my current knowledge base, but my current leading presumption is that the explicit firewall rules (pertaining to their source) that immediately result in the drop rather than allowing for further probes is a net benefit as from my own usecase testing it has shown that the inclusion of banIP seems to mitigate the bizarre bufferbloat/throughput drop issue when applied in conjunction with the other changes I've mentioned.

And on your issue of DNSMasq issue I've never encountered such problems. However I instead use Adguard Home as my default DNS, binding to port 53 with DNSMasq on port 1053. The luci integration provides an interesting addition, https://github.com/kongfl888/luci-app-adguardhome/releases . It's also not immediately obvious, and it doesn't seem to jump out on the github, but it's definitely recommended to add something like "[/lan/arpa/]127.0.0.1:1053" to your upstream DNS. Where inside the brackets are the domains/TLDs to which the upstream server is applied, and of course the IP:Port is what your DNSMasq responds to. The general behaviour to be expected is that AGH automatically does so for PTR and local domain requests, but I found that it doesn't always follow so an explicit rule helps in this way.

1 Like