Thank you so much, 
I just upgraded to latest build without any issues and I have added services to
SYSTEM > STARTUP > LOCALSTARTUP
ENABLEDSERVICES="sqm simple-adblock https-dns-proxy"
it just upgraded without any issues and these services are working without making any changes or starting them manually after every update
I think you've upgraded twice... since ROOTFSEXPAND was enabled...
did your disk get expanded?
df -h | grep root; dmesg | grep -i expand
( thanks to @oskari.rauta for adding parted to OpenWrt packages repo... made this task alot easier )
in brief steps;
other than that... you'll have to poke around or ask on the general forum...
collectd-mod-dns has been in my packagesremove.txt for the last few months as I found it would put collectd into a crash loop every few days.
I’d vote it to be removed unless there are active users 
o cool... i didn't think it would startup if you just remove the package edit: spoke too soon ( it does )... so I wont have to push a script to [ conditionally 
] remove the config lines...
Sun Jul 25 05:11:25 2021 daemon.err collectd[6902]: There is configuration for the `dns' plugin, but the plugin isn't loaded. Please check your configuration.
my guess is it's got something to do with some patches that made busybox more ipv6 aware a few weeks ago... as i'm pretty sure it was working for 3-5months...
I also vote to remove it... it as added when dnsmasq was getting all those fixes to keep an eye on it...
it's easy enough for someone to add to /etc/packagesinstall.txt if they want it as it's not really required for firstboot... ( I have not had issues with it though... so my suspicion is that it's failing for those that don't have ipv6 )
thanks for the info
when masq is chugging along ok... you do get a teency amount of query distribution info from it... but once you've seen it... for the resources it uses... it's not really worth running 24/7 anyways imho...
edit: removed and reported refs:
Iirc this was the reason why when we dug into it. My isp doesn’t hand out IPV6 addresses yet.
Almost all isp in my country not using ipv6. Maybe jn the future.
Hello there, been using your build and enjoying it a lot!
Just wanted to say thanks for all the effort in getting it to work, and also a few questions.
Up until SNAPSHOT r17189-1ed9fc663e, I was able to use dockerd from Entware and host several containers from within the router, but after upgrading yesterday it seems that the kernel hash started mismatching between the installed version and the GitHub/OpenWRT opkg repositories,as trying to install the dependencies for dockerd now (I have no idea why them and several other packages uinstalled themselves) reports the following error log:
[root@RPi4-OpenWRT /]$ opkg install kmod-lib-lzo kmod-lib-zlib-inflate kmod-lib-zlib-inflate kmod-lib-zlib-deflate kmod-lib-raid6 kmod-lib-xor kmod-lib-zstd kmod-fs-btrfs kmod-dax kmod-random-core kmod-tpm kmod-keys-trusted kmod-keys-encrypted kmod-dm kmod-br-netfilter kmod-nf-ipvs
Unknown package 'kmod-lib-lzo'.
Unknown package 'kmod-lib-zlib-inflate'.
Installing kmod-lib-zlib-inflate (5.4.132-1) to root...
Downloading https://downloads.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-8c34d2a86d75b793acaec716ad3d8596/kmod-lib-zlib-inflate_5.4.132-1_aarch64_cortex-a72.ipk
Unknown package 'kmod-lib-zlib-deflate'.
Unknown package 'kmod-lib-raid6'.
Unknown package 'kmod-lib-xor'.
Unknown package 'kmod-lib-zstd'.
Unknown package 'kmod-fs-btrfs'.
Unknown package 'kmod-dax'.
Unknown package 'kmod-random-core'.
Unknown package 'kmod-tpm'.
Unknown package 'kmod-keys-trusted'.
Unknown package 'kmod-keys-encrypted'.
Unknown package 'kmod-dm'.
Unknown package 'kmod-br-netfilter'.
Unknown package 'kmod-nf-ipvs'.
Collected errors:
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-lzo
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-lzo found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-lib-lzo.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-zlib-inflate
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-zlib-inflate found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-lib-zlib-inflate.
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-lib-zlib-inflate:
* kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596)
* opkg_install_cmd: Cannot install package kmod-lib-zlib-inflate.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-zlib-deflate
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-zlib-deflate found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-lib-zlib-deflate.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-raid6
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-raid6 found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-lib-raid6.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-xor
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-xor found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-lib-xor.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-lib-zstd
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-lib-zstd found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-lib-zstd.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-fs-btrfs
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-fs-btrfs found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-fs-btrfs.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-dax
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-dax found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-dax.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-random-core
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-random-core found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-random-core.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-tpm
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-tpm found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-tpm.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-keys-trusted
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-keys-trusted found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-keys-trusted.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-keys-encrypted
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-keys-encrypted found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-keys-encrypted.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-dm
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-dm found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-dm.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-br-netfilter
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-br-netfilter found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-br-netfilter.
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.132-1-8c34d2a86d75b793acaec716ad3d8596) for kmod-nf-ipvs
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-nf-ipvs found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-nf-ipvs.
Given that seemingly nothing else changed, I'm not sure how I should go about debugging this particular problem.
If there's anything I forgot to add, please let me know.
Kudos!
glad I was able to help..
on openwrt an upgrade is a "flash"... so it's likely nothing was uninstalled... just not re-installed automatically for you...
as for the other stuff... its possible there is some anomaly with the repo versioning or something... but as you mention entware... i'm wondering if I understand whats going on properly...
i've never had issue with opkg install before... so i'll test your install command and double check the repo consistency... after that we may look further into the use case and what else may be going on...
edit: works fine for me
opkg install kmod-lib-lzo kmod-lib-zlib-inflate kmod-lib-zlib-inflate kmod-lib-zlib-deflate kmod-lib-raid6 kmod-lib-xor kmod-lib-zstd kmod-fs-btrfs kmod-dax kmod-random-core kmod-tpm kmod-keys-trusted kmod-keys-encrypted kmod-dm kmod-br-netfilter kmod-nf-ipvs
Package opkg (2021-06-13-1bf042dd-3) installed in root is up to date.
Installing kmod-lib-lzo (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-lzo_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-lib-zlib-inflate (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-zlib-inflate_5.4.132-1_aarch64_cortex-a72.ipk
Package kmod-lib-zlib-inflate (5.4.132-1) installed in root is up to date.
Installing kmod-lib-zlib-deflate (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-zlib-deflate_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-lib-raid6 (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-raid6_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-lib-xor (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-xor_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-lib-zstd (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-lib-zstd_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-fs-btrfs (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-fs-btrfs_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-dax (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-dax_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-random-core (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-random-core_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-tpm (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-tpm_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-keys-trusted (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-keys-trusted_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-keys-encrypted (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-keys-encrypted_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-dm (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-dm_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-br-netfilter (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-br-netfilter_5.4.132-1_aarch64_cortex-a72.ipk
Installing kmod-nf-ipvs (5.4.132-1) to root...
Downloading http://downloads.cdn.openwrt.org/snapshots/targets/bcm27xx/bcm2711/kmods/5.4.132-1-272d8d859b9c04efc790e7b6c91cc80f/kmod-nf-ipvs_5.4.132-1_aarch64_cortex-a72.ipk
Configuring kmod-lib-zlib-deflate.
Configuring kmod-br-netfilter.
Configuring kmod-lib-xor.
Configuring kmod-dax.
Configuring kmod-random-core.
Configuring kmod-tpm.
Configuring kmod-keys-trusted.
Configuring kmod-keys-encrypted.
Configuring kmod-dm.
Configuring kmod-lib-zlib-inflate.
Configuring kmod-lib-raid6.
Configuring kmod-lib-lzo.
Configuring kmod-lib-zstd.
Configuring kmod-fs-btrfs.
Configuring kmod-nf-ipvs.
based on the above... i'd say you possibly have an entware related dependency issue... ( i.e. 'kernel dependant package on extroot syndrome' )... typically... entware should not be needed... and if I were doing the same... i'd just be using regular opkg for anything docker related...
just out of curiosity... how many times in total have you upgraded so far? what version did you upgrade from?
rpi4-installhistory.sh
rpi-support.sh | grep -A30 'restore-actions'
dmesg | grep -C2 -i expand
Hey, thanks for the quick reply!
To the current installation, I have upgraded just once coming from Snapshot-26567-3.2.61-70-r17073, the only unusual change I made to the image was before the ROOTFSEXPAND variable was supported was to manually expand the filesystem using the following commands:
mount -o remount,ro / #Remount root as ReadOnly
tune2fs -O^resize_inode /dev/mmcblk0p2 #Remove reserved GDT blocks
fsck.ext4 /dev/mmcblk0p2 #Fix part, answer yes to remove GDT blocks remnants
fdisk /dev/mmcblk0
fdisk > p #list partitions
fdisk > d #delete the second partition, keeping track of sector start
fdisk > n
fdisk > p
#create new primary partition from previous sector start and end at disk end
fdisk > w #write new partition table to disk
reboot
resize2fs /dev/mmcblk0p2
Also, I added the variable to rc.local before the upgrade just to be sure, and took a backup for safekeeping.
The mSD card I'm using as a boot device is the following one:
SanDisk Extreme microSDXC UHS-I SDSQXA2-064G-GN6MA
Thanks for the help and Kudos!
they go in /root/wrt.ini
rc.local calls most custom build logic... alterations from the build default could render most build related stuff non-functional... (including the firstboot opkg feed file creation logic)
the stuff above should not be related to EXPAND...
suggest you;
cp /etc/rc.local.copy /etc/rc.local
and upgrade again... to ensure all build related stuff operates properly...
Alright, I see...
In cases like the IRQAFFINITY, ENABLEDSERVICES or PERFTWEAKS variables, should I move my changes as well to a different file?
Mostly out of concern for Cake_QoS SQM and packet affinity tweaks. At the moment the modifications I've done are the following:
POWERPROFILE="quickest"
#!20210330 new users get this (for now) #@!> needs bin/x.sh stop
#equivalent to FWCUSTOM aka comment out to disable all ( for now ) later will require the subactionVARS also
PERFTWEAKS=1
#PERFTWEAKS="default"
PFAFFINITY="32:1 33:2 26:f"
#PFRENICE=1
#PFSERVICECPU=1 #taskset some procs to cpu's
should already be set for you, commenting it out will likely mean you dont get any perftweaks at all... the affinity stuff was disabled (technically hardcoded) due to little or inadequate feedback as mentioned in the top post/s ( but no harm in leaving it )
only POWERPROFILE is currently honoured... and possibly PACKETSTEERING=1 i'd need to double check those...
Perfect, let me give it a go and I'll report back.
Thanks for the help!
Edit: Looks like it worked correctly! Thanks for the help. Now to continue struggling with NGINX 
no problem at all... thankyou for reporting back and the detailed report... it's very useful to me to get detailed bug/problem reports and I learn alot from them...
in this case you've stumbled/wrestled with two of the 'build specific' 'must-haves'... they can be a little bit of a pain at first especially for advanced users...
fwiw for rc.local type stuff... you can... do something like...;
cp /etc/custom/dca632563177.sh /etc/custom/YOURMAC.sh
echo '/etc/custom/YOURMAC.sh' >> /etc/sysupgrade.conf
this will give you your own per device script that gets stored in upgrades...
the next time you upgrade... your docker packages and kmods should re-install automatically for you... if they do not... feel free to post here and we can take a look...
cheers... 
In terms of efficiency I've found that if you're not pushing the bounds with speed that it's been more beneficial to leave everything pinned to the first CPU. My own speed is 80/25 Mbit/s so I've also disabled all offloading for all devices/interfaces with an iface hotplug script. I was having a lot of problems with strange fluctuations where ping would skyrocket and throughput would drop dramatically relatively frequently, some odd consequence of having SQM and a VPN running in tandem; as with one or the other it wouldn't crop up.
Also, as an aside wulfy, which files would I plumb from the install image to get a handle on your custom general tweaks (e.g: processes pinned to CPU, etc.)?
the current stripped down tweaks are in the same spot... although as you mention are stripped down to some bare minimals...
/bin/rpi-perftweaks.sh
it used to print out everthing it does properly but when I had to hack it back to something somple I've hadnt time to do yet and as it's a 'DUMMY' version haven't been bothered
one eth interrupt was moved for the last two builds due to @sammutd88 s suggestion but its been moved back for the next as he did not provide the proof of benefit from this change...
the only change since the real scripts were pulled is to lookup interript numbers dynamically...
as CM4 / enabling dwc2 etc. etc. alters numbering...
one thing i've noted recently that should possibly in an ideal world be addressed ( via some sysctls ) is dd'ing 1G zeros hangs luci... but this is mostly only beneficial for people who use alot of samba or something that heavily writes to disks and does not have a huge impact on routing as a whole...
speaking of this... some 'perftweaks' are also derived from '/etc/custom/firstboot/01-sysctl' also...
Thanks. Alright, it's good to know that I already had a grasp of things. Just to clarify, the /boot/config.txt addition for DWC is for CM4 only? Are there any dtb additions suggested for the classic RPi4 config?
My previous anecdote, I'll add a bit more clarity as I realise that I worded it somewhat poorly. As an aside, though, I'll note that my experiences are from images built by myself, fairly light on packages as I prefer things to be as lean as possible and I prefer to go as long as possible without opening the terminal or LuCi. As mentioned, with SQM and VPN in tandem I was seeing bufferbloat spikes and bizarre throughput drops. This held true across a number of VPN servers and even when dropping SQM limitations far below actual line speed, despite CPU having a good deal of overhead. The main factors that I found to resolve this were to leave all interrupts as default on CPU0 and to disable offloading everywhere. There were a few other minor tweaks but I don't recall in my testing that they provided much benefit to resolving this particular problem. The only other benefit of note that comes to mind is banIP, I didn't realise that port-scanning and random pings were so prevalent and can only presume that the little bit of time to process these instead of immediately dropping them caused the issues I saw. My main 'real-world' performance metric was to gauge consistency in Qbittorrent seeding (upload) performance which seems heavily dependent on latency and whatever else due to it being wholly TCP. The before and after of these changes is remarkable.