Custom Build Bootloop after reboot (4/32 device)

[shaumux]

Hi
I'm compiled a build for WR841N V8, with luci, relayd, and stubby(removed everything related to firewall, iptables, netfilter, ppp etc.)
I intend to use it as a wifi extender and a DNS over TLS server using stubby.

The thing with this build is that after flashing the sysupgrade it boots fine, once I reboot even if I don't make any changes to the config it goes into a bootloop and I need to do a factory reset, and the same thing happens, boots fine the first time and then bootloops on reboot.

Any help?

The confif file I used is

CONFIG_TARGET_ar71xx=y
CONFIG_TARGET_ar71xx_tiny=y
CONFIG_TARGET_ar71xx_tiny_DEVICE_tl-wr841-v8=y
CONFIG_DEVEL=y
CONFIG_TOOLCHAINOPTS=y
CONFIG_FSTOOLS_OVL_MOUNT_COMPRESS_ZLIB=y
# CONFIG_GDB is not set
CONFIG_GETDNS_ENABLE_STUB_ONLY=y
# CONFIG_KERNEL_CRASHLOG is not set
# CONFIG_KERNEL_DEBUG_FS is not set
# CONFIG_KERNEL_DEBUG_INFO is not set
# CONFIG_KERNEL_DEBUG_KERNEL is not set
# CONFIG_KERNEL_MAGIC_SYSRQ is not set
# CONFIG_KERNEL_PRINTK is not set
# CONFIG_KERNEL_PRINTK_TIME is not set
# CONFIG_KERNEL_SWAP is not set
CONFIG_LUCI_SRCDIET=y
CONFIG_OPENSSL_WITH_DEPRECATED=y
CONFIG_OPENSSL_WITH_EC=y
CONFIG_OPENSSL_WITH_NPN=y
CONFIG_OPENSSL_WITH_PSK=y
CONFIG_OPENSSL_WITH_SRP=y
# CONFIG_PACKAGE_MAC80211_DEBUGFS is not set
# CONFIG_PACKAGE_dnsmasq is not set
# CONFIG_PACKAGE_firewall is not set
CONFIG_PACKAGE_getdns=y
# CONFIG_PACKAGE_ip6tables is not set
# CONFIG_PACKAGE_iptables is not set
# CONFIG_PACKAGE_kmod-ip6tables is not set
# CONFIG_PACKAGE_kmod-ipt-conntrack is not set
# CONFIG_PACKAGE_kmod-ipt-core is not set
# CONFIG_PACKAGE_kmod-ipt-nat is not set
# CONFIG_PACKAGE_kmod-nf-conntrack is not set
# CONFIG_PACKAGE_kmod-nf-conntrack6 is not set
# CONFIG_PACKAGE_kmod-nf-ipt is not set
# CONFIG_PACKAGE_kmod-nf-ipt6 is not set
# CONFIG_PACKAGE_kmod-nf-nat is not set
# CONFIG_PACKAGE_kmod-nf-reject is not set
# CONFIG_PACKAGE_kmod-nf-reject6 is not set
CONFIG_PACKAGE_kmod-nls-base=y
# CONFIG_PACKAGE_kmod-ppp is not set
# CONFIG_PACKAGE_libip4tc is not set
# CONFIG_PACKAGE_libip6tc is not set
CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_liblucihttp=y
CONFIG_PACKAGE_liblucihttp-lua=y
CONFIG_PACKAGE_libopenssl=y
CONFIG_PACKAGE_libubus-lua=y
# CONFIG_PACKAGE_libxtables is not set
CONFIG_PACKAGE_libyaml=y
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_luci-base=y
CONFIG_PACKAGE_luci-lib-ip=y
CONFIG_PACKAGE_luci-lib-jsonc=y
CONFIG_PACKAGE_luci-lib-nixio=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-proto-relay=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
# CONFIG_PACKAGE_odhcpd-ipv6only is not set
# CONFIG_PACKAGE_ppp is not set
CONFIG_PACKAGE_relayd=y
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-rrdns=y
CONFIG_PACKAGE_stubby=y
CONFIG_PACKAGE_uhttpd=y
CONFIG_PACKAGE_uhttpd-mod-ubus=y
# CONFIG_PKG_RELRO_FULL is not set
CONFIG_PKG_RELRO_PARTIAL=y
CONFIG_PROCD_ZRAM_TMPFS=y
CONFIG_STRIP_KERNEL_EXPORTS=y
CONFIG_TARGET_ROOTFS_TARGZ=y
CONFIG_TARGET_SQUASHFS_BLOCK_SIZE=1024
CONFIG_USE_MKLIBS=y
CONFIG_PACKAGE_kmod-lib-crc-ccitt=y

This is the generated manifest

base-files - 191-r7404-7ec931b7f0
busybox - 1.28.4-2
dropbear - 2017.75-6
fstools - 2018-05-23-dd02dad3-1
fwtool - 1
getdns - 1.4.2-1
hostapd-common - 2018-05-21-62566bc2-4
iw - 4.14-1
iwinfo - 2018-05-18-e59f9253-1
jshn - 2018-04-12-6eff829d-2
jsonfilter - 2018-02-04-c7e938d6-1
kernel - 4.9.111-1-5ce88375c3c13e298e6d1a4f1618219a
kmod-ath - 4.9.111+2017-11-01-9
kmod-ath9k - 4.9.111+2017-11-01-9
kmod-ath9k-common - 4.9.111+2017-11-01-9
kmod-cfg80211 - 4.9.111+2017-11-01-9
kmod-gpio-button-hotplug - 4.9.111-2
kmod-lib-crc-ccitt - 4.9.111-1
kmod-mac80211 - 4.9.111+2017-11-01-9
kmod-nls-base - 4.9.111-1
libblobmsg-json - 2018-04-12-6eff829d-2
libc - 1.1.19-1
libgcc - 7.3.0-1
libiwinfo - 2018-05-18-e59f9253-1
libiwinfo-lua - 2018-05-18-e59f9253-1
libjson-c - 0.12.1-1
libjson-script - 2018-04-12-6eff829d-2
liblua - 5.1.5-1
liblucihttp - 2018-05-18-cb119ded-1
liblucihttp-lua - 2018-05-18-cb119ded-1
libnl-tiny - 0.1-5
libopenssl - 1.0.2o-1
libpthread - 1.1.19-1
libubox - 2018-04-12-6eff829d-2
libubus - 2018-01-16-5bae22eb-1
libubus-lua - 2018-01-16-5bae22eb-1
libuci - 2018-03-24-5d2bf09e-1
libuclient - 2017-11-02-4b87d831-1
libyaml - 0.1.7-1
logd - 2018-02-14-128bc35f-2
lua - 5.1.5-1
luci-base - git-18.188.35117-711d103-1
luci-lib-ip - git-18.188.35117-711d103-1
luci-lib-jsonc - git-18.188.35117-711d103-1
luci-lib-nixio - git-18.188.35117-711d103-1
luci-mod-admin-full - git-18.188.35117-711d103-1
luci-proto-relay - git-18.188.35117-711d103-1
luci-theme-bootstrap - git-18.188.35117-711d103-1
mtd - 21
netifd - 2018-05-30-a580028d-1
odhcp6c - 2018-06-20-b99c1f63-13
openwrt-keyring - 2018-05-18-103a32e9-1
opkg - 2017-12-07-3b417b9f-2
procd - 2018-03-28-dfb68f85-1
relayd - 2016-02-07-ad0b25ad-2
rpcd - 2018-05-13-82062195-1
rpcd-mod-rrdns - 20170710
stubby - 0.2.3-1
swconfig - 11
uboot-envtools - 2018.03-1
ubox - 2018-02-14-128bc35f-2
ubus - 2018-01-16-5bae22eb-1
ubusd - 2018-01-16-5bae22eb-1
uci - 2018-03-24-5d2bf09e-1
uclient-fetch - 2017-11-02-4b87d831-1
uhttpd - 2018-06-26-796d42bc-1
uhttpd-mod-ubus - 2018-06-26-796d42bc-1
usign - 2015-07-04-ef641914-1
wireless-regdb - 2017-10-20-4343d359
wpad-mini - 2018-05-21-62566bc2-4

[hnyman]

@shaumux

after flashing the sysupgrade it boots fine, once I reboot even if I don't make any changes to the config it goes into a bootloop and I need to do a factory reset, and the same thing happens, boots fine the first time and then bootloops on reboot.

Two comments:

1. using a 4/32 device with current Openwrt is asking for trouble. You asked for it, you got it
   The 4/32 warning is there for a reason.
   https://openwrt.org/supported_devices/432_warning

2. Based on your description, I guess that either

   • on the first boot the router creates or at least tries to create r/w jffs2 overlay partition. That succeeds(?), but due to small flash space there is not enough free working space on the next boot. Even if you do not change any config, some small things do get written into overlay (effects of uci-defaults scripts, /etc/urandom.seed etc.)
   • alternatively, the creation of the overlay partition fails half-way on the first boot, and on the second boot there is half-created jffs2 partition that causes error.

You might try achieving 64 kB more free space.

What does "df -h" show? How much free space? is there overlay on flash (or on tmp)?

Those are pure guesses, but read carefully the kernel & system logs after the first boot in order to see what jffs2 related messages there are.

With a successful boot and jffs2 creation there should be something like

# logread | grep jffs2
Sat Jul  7 17:20:12 2018 kern.notice kernel: [    0.000000] Kernel command line:  board=WNDR3700 mtdparts=spi0.0:320k(u-boot)ro,128k(u-boot-env)ro,15872k(firmware),64k(art)ro console=ttyS0,115200 rootfstype=squashfs,jffs2 noinitrd
Sat Jul  7 17:20:12 2018 kern.info kernel: [    2.908243] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
Sat Jul  7 17:20:12 2018 kern.warn kernel: [   13.830258] jffs2_scan_eraseblock(): End of filesystem marker found at 0x10000
Sat Jul  7 17:20:12 2018 kern.warn kernel: [   13.837515] jffs2_build_filesystem(): unlocking the mtd device... done.
Sat Jul  7 17:20:12 2018 kern.warn kernel: [   13.844117] jffs2_build_filesystem(): erasing all blocks after the end marker... done.
Sat Jul  7 17:20:12 2018 kern.notice kernel: [   73.591620] jffs2: notice: (420) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
Sat Jul  7 17:20:12 2018 kern.notice kernel: [   73.766340] jffs2: notice: (417) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
Sat Jul  7 17:20:12 2018 user.info kernel: [   74.008447] mount_root: switching to jffs2 overlay

and

# logread | grep -i overlay
Sat Jul  7 17:20:12 2018 user.info kernel: [   13.525330] mount_root: loading kmods from internal overlay
Sat Jul  7 17:20:12 2018 user.info kernel: [   73.999619] mount_root: overlay filesystem has not been fully initialized yet
Sat Jul  7 17:20:12 2018 user.info kernel: [   74.008447] mount_root: switching to jffs2 overlay
Sat Jul  7 17:20:17 2018 daemon.err block: /dev/mtdblock5 is already mounted on /overlay

and normal df with /overlay on mtdblockX

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 5.0M      5.0M         0 100% /rom
tmpfs                    29.5M    912.0K     28.6M   3% /tmp
/dev/mtdblock5            9.2M    456.0K      8.7M   5% /overlay
overlayfs:/overlay        9.2M    456.0K      8.7M   5% /
tmpfs                   512.0K         0    512.0K   0% /dev

[shaumux]

Thanks for the help.
I understand the 4/32 problem, this is not a device I purchased recently, I had this lying around and decided to re-purpose it for a specific task.
Anyways, I managed to figure out the problem, though I am not sure of the root cause, so it turns out

FSTOOLS_OVL_MOUNT_COMPRESS_ZLIB=y

was the problem.

Whenever I enable it the same thing happens, I disabled it and my build seems to be working fine, even compiled a build with the openssl-1.1 patch, which also seems to working great.
Thanks for your help