19.07.xx: Strange kernel jffs2 bug

Hi,

I'm currently writing a script that allows for offline upgrading OpenWrt through a debian remote management server. The scenario is as follows:

MANUAL ONE-TIME PREPARATION:

  • I've put a line in /etc/rc.local that triggers "/bin/sh /root/post-upgrade.sh &" on every boot. post-upgrade.sh looks for a marker that gets removed by OpenWRT on sysupgrade (/etc/_OPKG_INSTALL_DONE). If the marker is missing, it one-time shots /root/opkg_reinstall.sh (which is capable to do offline reinstallation of packages uploaded to /root/packages/ before the sysupgrade commenced). After successful opkg reinstallation, it creates the marker (/etc/_OPKG_INSTALL_DONE).

SCRIPT RUN:

  • Management server downloads openwrt_release and board.json via ssh from the router we'd like to upgrade with a new (official stable) firmware image
  • It reaches out to downloads.openwrt.org to fetch the specified firmware version and the opkg lists consisting of multiple "packages.gz" files.
  • It extracts the package lists and looks through them for mission-critical packages that are required for a mesh AP to come online again via wireless (batman-adv, wpad-mesh-openssl, kmod-ath10k, non-ct firmware) because the stations I need to upgrade don't have LAN access.
  • It downloads the mission-critical IPK files
  • It uploads the new firmware image to /tmp/sysupgrade.bin of the AP and then uploads the IPK package files to /root/packages/ of the AP.
  • It remotely triggers "sysupgrade -k -v /tmp/sysupgrade.bin" to commence the firmware upgrade.
  • The router itself then does a reboot after flash.
  • The router automatically fires /root/post-upgrade.sh (and subsequently /root/opkg_reinstall.sh) after 30 secs time of sleep.
  • I'm logging what happens during the opkg_reinstall.sh to /root/opkg_reinstall.log so I can later have a look that everything went fine.
  • The scripts trigger "reboot" after they complete. Because batman-adv, kmod-ath10k, wpad-mesh-openssl and the ath10k-firmware need a reboot to bring up the wifi and get the device online via mesh again.

MY PROBLEM:

  • If I initiate the sysupgrade via the script on the management server, sysupgrade is carried out successfully and the OpenWRT router comes back to life short after, it preserved ALL settings and its IP address. So after the flash (first) reboot everything is fine.
  • The scripts complete and reboot a second time. According to the logs, every IPK was installed successfully. After the second reboot, my /etc/config has been reverted to stock defaults and I don't know why that (reproducibly) happens. The device is then reachable through 192.168.1.1/24 instead of the normal IP I expect it to have. The contents of /etc/config/ are defaults now, other files that are directly in /etc/ are still there (no revert occured) and /root/ is fully there (no revert occured). I can also see my logs on /root/.

WHAT I TRIED TO REMEDIATE THE PROBLEM:

  1. I've tried to remove the "reboot" command at the end of /root/post-upgrade.sh. When I log on via SSH, all IPK had been installed successfully and there is only one single indication of error in the dmesg.

[after 74s kernel boot time] jffs2: Newly-erased block contained word xxxxxxx at offset xxxxxxx

I think that's the reason why I've got the "/etc/config revert phenomenon on next reboot". I've checked, it does not matter if the script issues "reboot" or if I do that manually from SSH. On next reboot, my /etc/config is gone and the dmesg says "jffs2 whiteout" because of errors detected.

  1. I've verified flash is fine. I think it's not flash failure. I can manually put /etc/config back from my documentation, power cycle or reboot the router, and it will start operating correctly with all settings in place.

  2. I've verified my overlay doesn't run out of space. I've got around 9 MByte overlay free. With my 11 IPK files (temporarily) stores onto /root/packages, I'm around 47% space used. So no trouble expected from this.

  3. I've removed the "/bin/sh /root/post-upgrade.sh &" line from "/etc/rc.local". Started all over, let the sysupgrade flash go through (again triggered by the management server). Then I logged on via SSH and manually issued "/bin/sh /root/post-upgrade.sh" from command line. SUPRISINGLY - this always works (tried multiple times). The jffs2 doesn't "corrupt", a second reboot follows and the router has all settings (preserved and effective). Normal operation.

This somehow led me here to report it to you. Please let me know if I should file the bug elsewhere in another place if it fits better there. I suspect it could be a kernel/jffs2 bug.

Especially when changing my "opkg_reinstall.sh" file and removing the parts "opkg install|remove .... 2>&1 | tee -a /root/opkg_reinstall.log". If I remove them, I can also run from /etc/rc.local and the corruption does NOT occur.

Observing the dmesg/logread -f while the opkg_reinstall.sh takes place, I see kmodloader lines loading the new kernel modules (e.g. BATMAN-ADV).

May it be that the kernel doesn't allow writing to the overlay (for the log) for just some microseconds? It's really reproducible. Adding the "tee -a" log lines and running from "/etc/rc.local" kills my partition. Even if I wait for minutes (sleep command) before the second reboot. Removing the tee log to /root/... or running from SSH manually, no corruption occurs.

Thanks for your support on this.

Kind regards,
Catfriend1

Here is a list of files I'm using for reference, viewed on the management server:

For reference:

  • The management server runs this script to prepare offline sysupgrade and then upgrade remotely

/root/openwrt/remote_update.sh

#!/bin/sh
#
# Command line
# 	sh "/root/openwrt/remote_update.sh" "IP" "19.07.4"
# 	ssh -oStrictHostKeyChecking=no -i "/etc/ssh/ssh_host_rsa_key" "root@IP" "/bin/sh /root/opkg_reinstall.sh; reboot"
#
# Script configuration.
DRY_RUN=0
SSH_PRIVATE_KEY="/etc/ssh/ssh_host_rsa_key"
#
#########################
# FUNCTIONS BLOCK START	#
#########################
fetchPackageList () {
	# Syntax:
	#	fetchPackageList "[WEBSERVER_DIRECTORY]"
	#
	# Global vars.
	# 	[IN] FP_PACKAGES
	#
	# Called By:
	# 	fetchPackageLists
	#
	# Variables.
	TMP_FPL_URL="${1}"
	#
	echo "[INFO] Downloading ${TMP_FPL_URL}/Packages.gz"
	TMP_FPL_RESULT="$(wget -q -O - -c "${TMP_FPL_URL}/Packages.gz" | gzip -d 2> /dev/null)"
	if [ -z "${TMP_FPL_RESULT}" ]; then
		echo "[ERROR] fetchPackageList: Failed to download and unpack ${TMP_FPL_URL##*/}/Packages.gz"
		return 1
	fi
	echo "${TMP_FPL_RESULT}" | egrep "^Package:|^Filename:" | sed -E "s^\^Filename:\s+^URL:"${TMP_FPL_URL}"/^" >> "${FP_PACKAGES}"
	return 0
}

fetchPackageLists () {
	# Syntax:
	# 	fetchPackageLists
	#
	# Global vars.
	# 	[IN] ARCH
	# 	[IN] FP_PACKAGES
	# 	[IN] TARGET
	# 	[IN] UPDATE_TO_VERSION
	#
	# Called By:
	# 	fetchPackages
	#
	# Check prerequisites.
	if [ -z "${FP_PACKAGES}" ]; then
		return 1
	fi
	#
	echo "[INFO] fetchPackageLists"
	#
	# Get dir listing and extract kmods/[KMOD_DIR_NAME], e.g. "4.14.195-1-b84a5a29b1d5ae1dc33ccf9ba292ca1d"
	# This information is also available from "/etc/opkg/distfeeds.conf" if we would unpack the new firmware image. 
	KMOD_DIR_NAME="$(wget -q -O - "http://downloads.openwrt.org/releases/${UPDATE_TO_VERSION}/targets/${TARGET}/generic/kmods" | grep -o '<a .*href=.*>' | sed -e 's/<a /\n<a /g' | sed -e 's/<a .*href=['"'"'"]//' -e 's/["'"'"'].*$//' -e '/^$/ d' | grep -v "^/" | sed -e "s/\/$//")"
	if [ -z "${KMOD_DIR_NAME}" ]; then
		echo "[ERROR] fetchPackageLists: Failed to determine kmods dir name on server."
		return 1
	fi
	#
	rm -f "${FP_PACKAGES}"
	#
	fetchPackageList "http://downloads.openwrt.org/releases/${UPDATE_TO_VERSION}/targets/${TARGET}/generic/packages"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackageList "http://downloads.openwrt.org/releases/${UPDATE_TO_VERSION}/targets/${TARGET}/generic/kmods/${KMOD_DIR_NAME}"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackageList "http://downloads.openwrt.org/releases/${UPDATE_TO_VERSION}/packages/${ARCH}/base"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackageList "http://downloads.openwrt.org/releases/${UPDATE_TO_VERSION}/packages/${ARCH}/luci"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackageList "http://downloads.openwrt.org/releases/${UPDATE_TO_VERSION}/packages/${ARCH}/packages"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackageList "http://downloads.openwrt.org/releases/${UPDATE_TO_VERSION}/packages/${ARCH}/routing"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackageList "http://downloads.openwrt.org/releases/${UPDATE_TO_VERSION}/packages/${ARCH}/telephony"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	return 0
}

fetchPackage () {
	# Syntax:
	# 	fetchPackage "[PACKAGE_NAME]"
	#
	# Global vars.
	#	[IN] DEVICE_IP
	# 	[IN] DOWNLOAD_DIR
	# 	[IN] FP_PACKAGES
	#
	# Called By:
	# 	fetchPackages
	#
	# Variables.
	TMP_FP_PACKAGE_NAME="${1}"
	#
	# Check prerequisites.
	if [ -z "${FP_PACKAGES}" ] || [ -z "${TMP_FP_PACKAGE_NAME}" ] ; then
		return 1
	fi
	#
	TMP_FP_URL="$(cat "${FP_PACKAGES}" 2> /dev/null | grep "^URL:" | grep "/"${TMP_FP_PACKAGE_NAME}"_.*\.ipk$" | sed -e "s/^URL://" | head -n 1)"
	if [ -z "${TMP_FP_URL}" ]; then
		echo "[ERROR] fetchPackage: Package [${TMP_FP_PACKAGE_NAME}] not found."
		return 1
	fi
	#
	echo "[INFO] Downloading ${TMP_FP_URL}"
	wget -q -O "${DOWNLOAD_DIR}/${TMP_FP_URL##*/}" "${TMP_FP_URL}"
	if [ ! "$?" = "0" ]; then
		echo "[ERROR] fetchPackage: Failed to download ${TMP_FP_URL##*/}"
		return 1
	fi
	#
	return 0
}

fetchPackages () {
	# Syntax:
	# 	fetchPackages
	#
	# Global vars.
	# 	[IN] ARCH
	# 	[IN] DOWNLOAD_DIR
	# 	[IN] TARGET
	# 	[IN] UPDATE_TO_VERSION
	#
	# Called By:
	# 	MAIN
	#
	# Variables.
	FP_PACKAGES="${DOWNLOAD_DIR}/packages.txt"
	#
	echo "[INFO] fetchPackages"
	fetchPackageLists
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	echo "[INFO] fetchPackages: Downloading IPK files for offline update"
	#
	fetchPackage "ath10k-firmware-qca988x"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackage "batctl-full"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackage "kmod-ath10k"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackage "kmod-batman-adv"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackage "kmod-crypto-crc32c"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackage "kmod-crypto-hash"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackage "kmod-lib-crc16"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackage "kmod-lib-crc32c"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackage "libopenssl1.1"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackage "librt"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	fetchPackage "wpad-mesh-openssl"
	if [ ! "$?" = 0 ]; then
		return 1
	fi
	#
	return 0
}

scpDownloadFile () {
	# Syntax:
	# 	scpDownloadFile "[DEVICE_IP]" "[SSH_PRIVATE_KEY]" "[REMOTE_FILE]" "[LOCAL_FILE]"
	#
	# Called By:
	# 	MAIN
	#
	# Variables.
	TMP_SDF_DEVICE_IP="${1}"
	TMP_SDF_SSH_PRIVATE_KEY="${2}"
	TMP_SDF_REMOTE_FILE="${3}"
	TMP_SDF_LOCAL_FILE="${4}"
	#
	# Create local parent directory.
	mkdir -p "${TMP_SDF_LOCAL_FILE%/*}"
	#
	# Download file.
	scp -oStrictHostKeyChecking=no -i "${TMP_SDF_SSH_PRIVATE_KEY}" "root@${TMP_SDF_DEVICE_IP}:${TMP_SDF_REMOTE_FILE}" "${TMP_SDF_LOCAL_FILE}"
	#
	return $?
}

scpUploadFile () {
	# Syntax:
	# 	scpDownloadFile "[DEVICE_IP]" "[SSH_PRIVATE_KEY]" "[LOCAL_FILE]" "[REMOTE_FILE]"
	#
	# Called By:
	# 	MAIN
	#
	# Variables.
	TMP_SDF_DEVICE_IP="${1}"
	TMP_SDF_SSH_PRIVATE_KEY="${2}"
	TMP_SDF_LOCAL_FILE="${3}"
	TMP_SDF_REMOTE_FILE="${4}"
	#
	# Download file.
	scp -oStrictHostKeyChecking=no -i "${TMP_SDF_SSH_PRIVATE_KEY}" "${TMP_SDF_LOCAL_FILE}" "root@${TMP_SDF_DEVICE_IP}:${TMP_SDF_REMOTE_FILE}"
	#
	return $?
}
#########################
# FUNCTIONS BLOCK END	#
#########################
#
# Verify SSH prerequisites.
if [ ! -f "${SSH_PRIVATE_KEY}" ]; then
	echo "[ERROR] SSH_PRIVATE_KEY=[${SSH_PRIVATE_KEY}] is missing."
	exit 99
fi
chmod 0600 "${SSH_PRIVATE_KEY}"
#
# Get command line params.
DEVICE_IP="${1}"
UPDATE_TO_VERSION="${2}"
#
# Verify command line params.
if [ -z "${DEVICE_IP}" ]; then
	echo "[ERROR] Param #1 DEVICE_IP missing."
	exit 99
fi
## echo "[INFO] DEVICE_IP=[${DEVICE_IP}]"
#
if [ -z "${UPDATE_TO_VERSION}" ]; then
	echo "[ERROR] Param #1 UPDATE_TO_VERSION missing."
	exit 99
fi
## echo "[INFO] UPDATE_TO_VERSION=[${UPDATE_TO_VERSION}]"
#
# Firmware download dir
DOWNLOAD_DIR="/tmp/device_${DEVICE_IP}_${UPDATE_TO_VERSION}"
#
echo "[INFO] Getting device info from [${DEVICE_IP}] ..."
BOARD_JSON="${DOWNLOAD_DIR}/board.json"
scpDownloadFile "${DEVICE_IP}" "${SSH_PRIVATE_KEY}" "/etc/board.json" "${BOARD_JSON}"
#
OPENWRT_RELEASE="${DOWNLOAD_DIR}/openwrt_release"
scpDownloadFile "${DEVICE_IP}" "${SSH_PRIVATE_KEY}" "/etc/openwrt_release" "${OPENWRT_RELEASE}"
#
# Generate firmware download URL.
#
## Example: MODEL="archer-c7-v5"
MANUFACTURER="$(cat "${BOARD_JSON}" | grep "\"id\"" | cut -d '"' -f 4 | cut -d "," -f 1)"
echo "[INFO] MANUFACTURER=[${MANUFACTURER}]"
#
## Example: MODEL="archer-c7-v5"
MODEL="$(cat "${BOARD_JSON}" | grep "\"id\"" | cut -d '"' -f 4 | cut -d "," -f 2)"
echo "[INFO] MODEL=[${MODEL}]"
#
## Example: ARCH="mips_24kc"
ARCH="$(cat "${OPENWRT_RELEASE}" | grep "^DISTRIB_ARCH=" | cut -d "'" -f 2 | cut -d "/" -f 1)"
echo "[INFO] ARCH=[${ARCH}]"
#
## Example: TARGET="ath79"
TARGET="$(cat "${OPENWRT_RELEASE}" | grep "^DISTRIB_TARGET=" | cut -d "'" -f 2 | cut -d "/" -f 1)"
echo "[INFO] TARGET=[${TARGET}]"
#
## Example: http://downloads.openwrt.org/releases/19.07.3/targets/ath79/generic/openwrt-19.07.3-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin
FIRMWARE_DOWNLOAD_URL="http://downloads.openwrt.org/releases/${UPDATE_TO_VERSION}/targets/${TARGET}/generic/openwrt-${UPDATE_TO_VERSION}-${TARGET}-generic-${MANUFACTURER}_${MODEL}-squashfs-sysupgrade.bin"
SYSUPGRADE_IMAGE="${DOWNLOAD_DIR}/openwrt-${UPDATE_TO_VERSION}-${TARGET}-generic-${MANUFACTURER}_${MODEL}-squashfs-sysupgrade.bin"
#
SHASUMS_DOWNLOAD_URL="http://downloads.openwrt.org/releases/${UPDATE_TO_VERSION}/targets/${TARGET}/generic/sha256sums"
SHASUMS="${DOWNLOAD_DIR}/sha256sums"
if [ ! -f "${SHASUMS}" ]; then
	echo "[INFO] Downloading [${SHASUMS_DOWNLOAD_URL}] to [${SHASUMS}]..."
	wget -qq -O "${SHASUMS}" "${SHASUMS_DOWNLOAD_URL}"
fi
#
echo "[INFO] Checking for existing firmware image ..."
CHECKSUM_RESULT="$(cd "${SYSUPGRADE_IMAGE%/*}"; cat "${SHASUMS}" | grep "${SYSUPGRADE_IMAGE##*/}" | sha256sum -c 2> /dev/null)"
if ( ! echo "${CHECKSUM_RESULT}" | grep -q "OK" ); then
	if [ -f "${SYSUPGRADE_IMAGE}" ]; then
		echo "[WARN] Firmware image FAILED checksum test."
	fi
	echo "[INFO] Downloading firmware image from [${FIRMWARE_DOWNLOAD_URL}] to [${SYSUPGRADE_IMAGE}] ..."
	wget -qq -O "${SYSUPGRADE_IMAGE}" "${FIRMWARE_DOWNLOAD_URL}"
	CHECKSUM_RESULT="$(cd "${SYSUPGRADE_IMAGE%/*}"; cat "${SHASUMS}" | grep "${SYSUPGRADE_IMAGE##*/}" | sha256sum -c 2> /dev/null)"
fi
#
if ( ! echo "${CHECKSUM_RESULT}" | grep -q "OK" ); then
	echo "[ERROR] Firmware image FAILED checksum test."
	exit 99
fi
echo "[INFO] Firmware image PASSED checksum test."
#
rm -f ${DOWNLOAD_DIR}/*.ipk
fetchPackages
if [ ! "$?" = 0 ]; then
	echo "[ERROR] fetchPackages FAILED."
	exit 99
fi
#
ssh -oStrictHostKeyChecking=no -i "${SSH_PRIVATE_KEY}" "root@${DEVICE_IP}" "rm -rf /root/packages/; mkdir -p /root/packages"
for ipk in ${DOWNLOAD_DIR}/*.ipk; do
	echo "[INFO] Uploading ${ipk##*/}"
	scpUploadFile "${DEVICE_IP}" "${SSH_PRIVATE_KEY}" "${ipk}" "/root/packages/${ipk##*/}"
	if [ ! "$?" = "0" ]; then
		logAdd "[ERROR] Upload FAILED."
		exit 99
	fi
done
#
if [ "${DRY_RUN}" = "0" ]; then
	echo "[INFO] Uploading firmware image to device ..."
	scpUploadFile "${DEVICE_IP}" "${SSH_PRIVATE_KEY}" "${SYSUPGRADE_IMAGE}" "/tmp/sysupgrade.bin"
	if [ ! "$?" = "0" ]; then
		logAdd "[ERROR] Upload FAILED."
		exit 99
	fi
fi
#
if [ "${DRY_RUN}" = "0" ]; then
	echo "[INFO] Flashing image ..."
	ssh -oStrictHostKeyChecking=no -i "${SSH_PRIVATE_KEY}" "root@${DEVICE_IP}" "sync; echo 3 > /proc/sys/vm/drop_caches; sysupgrade -k -v /tmp/sysupgrade.bin"
else
	echo "[INFO] DRY_RUN flag set. Skipping flash."
fi 
#
echo "[INFO] Done."
#
exit 0

For reference:

  • The OpenWRT device (TP Link Archer C7v5) runs the following:

/etc/rc.local

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

/bin/sh /root/post-upgrade.sh &

exit 0

/root/post-upgrade.sh

#!/bin/sh
trap "" SIGHUP
#
if [ -f "/etc/_OPKG_INSTALL_DONE" ]; then
	exit 99
fi
#
if [ ! -d "/root/packages" ]; then
	exit 99
fi
#
sleep 30
/bin/sh /root/opkg_reinstall.sh
if [ "$?" = "0" ]; then
	rm -rf "/root/packages/"
	touch "/etc/_OPKG_INSTALL_DONE"
	reboot
fi
#
exit 0
/root/opkg_reinstall.sh

#!/bin/sh
#
# Notes
## opkg remove ath10k-firmware-qca988x kmod-ath10k kmod-batman-adv wpad-mesh-openssl; opkg remove batctl-full;
#
# Consts
PATH=/usr/bin:/usr/sbin:/sbin:/bin
LOGFILE="/root/opkg_reinstall.log"
LOG_MAX_LINES="1000"
#
LOG_COLLECTOR_HOSTNAME="WifiAP-01"
INSTALL_BATMANADV="1"
INSTALL_OPENVPN="1"
INSTALL_RELAYD="0"
INSTALL_WIFI_NON_CT_DRIVERS="1"
PACKAGE_CACHE="/root/packages"
#
# Runtime vars
INSTALL_ONLINE="1"
#
#
# -----------------------------------------------------
# -------------- START OF FUNCTION BLOCK --------------
# -----------------------------------------------------
logAdd ()
{
	TMP_DATETIME="$(date '+%Y-%m-%d [%H-%M-%S]')"
	TMP_LOGSTREAM="$(tail -n ${LOG_MAX_LINES} ${LOGFILE} 2>/dev/null)"
	echo "${TMP_LOGSTREAM}" > "$LOGFILE"
	echo "${TMP_DATETIME} $*" | tee -a "${LOGFILE}"
	return
}


opkgInstall () {
	# Syntax:
	# 	opkgInstall "[PACKAGE_NAME]"
	#
	# Online mode.
	if [ "${INSTALL_ONLINE}" = "1" ]; then
		opkg install "${@}"
		return
	fi
	#
	# Offline mode
	IPKG_FULLFN=""
	for package_name in $@; do
		if ( ! ls -1 ${PACKAGE_CACHE} | grep -q "^${package_name}_" ); then
			logAdd "[ERROR] opkgInstall: Package missing in cache - [${package_name}]"
			continue
		fi
		IPKG_FULLFN="${IPKG_FULLFN} ${PACKAGE_CACHE}/$(ls -1 ${PACKAGE_CACHE} | grep "^${package_name}_")"
	done
	if [ -z "${IPKG_FULLFN}" ]; then
		return 0
	fi
	eval "opkg --cache ${PACKAGE_CACHE} install ${IPKG_FULLFN}" 2>&1
	return 0
}


opkgRemove () {
	# Syntax:
	# 	opkgRemove "[PACKAGE_NAME]"
	#
	for package_name in $@; do
		logAdd "[INFO] opkgRemove: Removing ${package_name}"
		opkg remove "${package_name}" 2>&1
		if ( echo "${package_name}" | grep -q "^kmod-" ); then
			logAdd "[INFO] opkgRemove: Kernel module removed. Will sleep a bit to avoid crash."
			sleep 10
		fi
	done
	return 0
}
# ---------------------------------------------------
# -------------- END OF FUNCTION BLOCK --------------
# ---------------------------------------------------
#
# 
logAdd "[INFO] Checking internet connection ..."
if ( ! echo -e "GET / HTTP/1.1\r\nHost: downloads.openwrt.org\r\n" | nc downloads.openwrt.org 80 | grep -q "https://downloads.openwrt.org/" ); then
	logAdd "[INFO] No internet connection. Switching to offline mode."
	INSTALL_ONLINE="0"
else
	logAdd "[INFO] Downloading package information ..."
	opkg update | grep "http://" 2>&1 | tee -a "$LOGFILE"
fi
#
logAdd "[INFO] Removing packages ..."
opkgRemove wpad-basic
opkgRemove wpad
#
logAdd "[INFO] Installing base packages ..."
opkgInstall bash curl htop nano wget
opkgInstall terminfo tcpdump
opkgInstall libopenssl1.1 wpad-mesh-openssl
# 
if [ "${HOSTNAME}" == "${LOG_COLLECTOR_HOSTNAME}" ]; then
	logAdd "[INFO] Installing syslog-ng ..."
	opkgRemove logd
	opkgInstall syslog-ng
else
	logAdd "[INFO] Removing syslog-ng ..."
	opkgRemove syslog-ng
	opkgInstall logd
fi
#
if [ "${INSTALL_BATMANADV}" = "1" ]; then
	logAdd "[INFO] Installing BATMAN-ADV ..."
	opkgInstall batctl-full kmod-batman-adv kmod-crypto-crc32c kmod-lib-crc16 kmod-lib-crc32c kmod-crypto-hash librt
fi
#
# logAdd "[INFO] Installing igmpproxy ..."
## opkgInstall igmpproxy
#
# logAdd "[INFO] Installing kernel mod br-netfilter ..."
## opkgInstall kmod-br-netfilter
# 
if [ "${INSTALL_OPENVPN}" = "1" ]; then
	logAdd "[INFO] Installing OpenVPN ..."
	opkgInstall luci-app-openvpn
	opkgInstall openvpn-easy-rsa
	opkgInstall openvpn-openssl
fi
#
if [ "${INSTALL_RELAYD}" = "1" ]; then
	logAdd "[INFO] Installing relayd ..."
	opkgInstall luci-proto-relay
	opkgInstall relayd
fi
#
# logAdd "[INFO] Installing ubus rpc mod interface ..."
## opkgInstall luci-mod-rpc
## opkgInstall uhttpd-mod-ubus
#
if [ "${INSTALL_WIFI_NON_CT_DRIVERS}" = "1" ]; then
	logAdd "[INFO] Switching WiFi firmware from ct to non-ct ..."
	opkgRemove ath10k-firmware-qca988x-ct kmod-ath10k-ct
	opkgInstall ath10k-firmware-qca988x kmod-ath10k
fi
#
logAdd "[INFO] Installing usb storage drivers, luafilesystem, mailsend, vsftpd"
opkgInstall block-mount e2fsprogs kmod-fs-ext4 kmod-fs-msdos kmod-scsi-core kmod-usb-storage libncurses libpcre lua luafilesystem mailsend vsftpd
# 
logAdd "[INFO] Done. Reboot required."
exit 0

Newly erased block contains word... is likely a hardware issue-- the block was erased but then found not to be blank. Testing on different hardware should be considered.

Another possibility is the jffs is being overfilled, which causes a lot of strange things to happen. Don't let the jffs get overfilled.

2 Likes

@mk24 It's 47% filled at max. Really, I just thought the same like you at first but I have 4 to 5 MBytes free there. I can also reproduce it on another Archer C7v2. One hardware is months old, the other was bought back in 2016.

I thought again why it works executing the script via SSH and why the autorun way via /etc/rc.local brings up that kernel error. Maybe it plays a role if I have this " | tee -a ...log" after opkg in place that SSH will slow down the commands a little bit by waiting for the "screen output pipe". SSH isn't that fast than local execution without monitoring the terminal. I've had a similar issue on a third TPLink Archer c7 last year when I made the watchdog script which should monitor ath10k_pci failing via logread and then remove the kmod and readd it back without reboot to get WiFi working again. The watchdog does its job pretty well until today, but at my first try I also had a " > /root/mylogfile.log" in place behind the "rmmod" and "insmod" commands and I suffered kernel crashes then.

It all looks like the kernel is microseconds unavailable to serve the jffs2 filesystem and using exact that timeframe to write to it (because of the log pipe to /root/...) causes the above failure to occur. I'm sorry I cannot do any kernel debugging but closely observing and telling you there might be a bug.

So to sum up: Avoid to pipe "rmmod", "insmod", "opkg" output to a log placed on /root/... and the jffs2 "corruption" does not take place. I can read/write heavily on the whole partition if I don't do these things.

1 Like

So after doing more tests, I think this a multicausal problem. Because I did revert /etc/rc.local to NOT contain anything, the router flashes, comes up with the settings preserved, I'm looking at dmesg (in this case tested on 19.07.2) and always find the message

jffs2: Newly-erased block contained word 0x19852003 at offset 0x00000000

I bet this is NOT hardware failure. This is also exactly the same message (0x19852003) as in:

To get this nailed down to root cause and be able to report back to you, I need an exact info in which OpenWrt stable 19.07.xx release the fix mentioned in the other thread ( ref: https://github.com/gl-inet/openwrt/commit/4b1f073f843dc4be655a868f6a6e31f74baa727c ) is contained, if yet released. @mk24 Can you please shed some light?

Did you notice that that code is not even from an OpenWrt repo?
It is from gl-inet's repo of their derivative firmware for their devices based on an old kernel version...
(Both the ar71xx target and also the kernel 4.9 have already been deprecated in OpenWrt)

@hnyman I'm just a user, so just deploying and upgrading devices when needed. I'm not so into it what which commit or repo means. Can you offer a solution because it seems to be exactly the problem I'm suffering.

  • I've tested on 19.07.2 upgrading to 19.07.2 -> jffs error occurs.
  • I've tested on 19.07.2 upgrading to 19.07.3 -> jffs error occurs.
  • I've tested on 19.07.3 upgrading to 19.07.3 -> jffs error occurs.
  • I've tested on 19.07.3 upgrading to 19.07.4 -> jffs error occurs.
  • I've tested on 19.07.4 upgrading to 19.07.4 -> jffs error does NOT occur.

So it seems the SPI write flash issue is indeed a software issue. Testing around, I assume that during a sysupgrade the old kernel still boots one time, flashes the new image and then directly fires up everything without rebooting. So that might be why I'm experiencing the config loss on next reboot (no matter if my scripts do it or if I do it manually). If this is true, that explains why going from an unaffected firmware to another one works without the config loss, e.g. 19.07.4 to 19.07.4. That's why I asked for confirmation if OpenWrt did solve this issue in the meantime. Can you answer it? If OpenWrt didn't do anything about it, maybe it will return some day and all of us should be aware of this.

Just another thesis: It maybe that no one experiences the issue if only a "small" amount of overlay data is persisted during sysupgrade. I've got 9 MB overlay capacity and using only 3 MB of it to persist, so that definitely is well calculated and is expected to work. If not, OpenWrt (or kernel) has a bug and it should be investigated and fixed.

This is one of the drawbacks of MIPS memory-mapping. If you use memory-mapping to read data smaller than 32bytes from flash, then erase flash, and then use memory-mapping to read data smaller than 32bytes from the same location, the data will wrong.
As far as I know, the latest OpenWRT does not fix this problem.

1 Like

@luochongjun Can I avoid this issue somehow? I'm asking because ocassionally logging important things to /root/ from my 24/7 running bash scripts didn't cause any "config loss on reboot" issues over years. What's also unclear to me is why only /etc/config/* is lost while the rest, for example, /etc/passwd, /root/* and soon is still there. Before the second reboot after flash, everything - including /etc/config/ - is correctly in place. What could OpenWrt cause to just drop a valid /etc/config ? If I connect to 192.168.1.1 after the config loss and reupload it from documentation, reboot, the config is well running (so no errors in it) and peristed across future reboots.

I think I'm having the same problem like described in [ SOLVED ] Getting error failed to sync jffs2 overlay where jeff posted that it's "bad timing handling the erasure mark on the tplink archer c7v5". What is different in my case is that the only jffs2 error in dmesg is the above mentioned "word at offset" thing. I do not get the "failed to sync overlay" message.

It can be reproduced scp uploading ca. 4 mbytes to the tp link archer c7v5's /root . Set sysupgrade.conf to persist /root . Upload openwrt-sysupgrade.bin image to /tmp and run sysupgrade -k via ssh. When the router reboots after flash (like @jeff described it as first reboot after flash in the topic) I had 28 failures and 3 successes trying the same procedure over and again. (This can be reproduced without using the offline installer scripts from my first post.) Failure means the jffs2 erase error message was there.

Don't you think if someone took time to mention it, that it may be important?

  • Commit (or 4b1f073f843dc4be655a868f6a6e31f74baa727c) is the serial number the versioning system gave to the specific change/addition/removal of code to a repository (or code repo)
  • Repo (or code repository) is where you're downloading software from (it's kind of a security issue that you don't know where the code is from) - It means your script is not from OpenWrt, just look at the link. Have you asked GL-Inet?

The I'd suggest not developing a script to upgrade it. Simply use the normal sysupgrade method.

Sure, I even loaded the files manually to /root I need persisted and no matter what they are, I go to web Ui, flash upgrade, and the jffs2 error sometimes comes up and some times not . It has nothing to do with the scripts. If it would work fine the manual way in all cases the script would also be okay.

Having kilobytes to persist -- always works
Having megabytes to persist (part free over 70% still) -- often fails.

@lleachii please help correcting the bug

is this not a sysupgrade data size thing?

2 Likes

I'm not sure if you're purposely ignoring me; or didn't understand again...

...or perhaps the information you offered prior to tagging me wasn't in response to my post. I asked why haven't you inquired with the developers of these scripts (it isn't OpenWrt):

screen115

Are you sure you're calculating this correctly...because the sysupgrade image itself is bigger than the 3 MB you just quoted:

I'm estimating you have less than a few hundred KBs...before the sysupgrade bundles all persistent data.

I'm quite confused on how your logic always comes to a bug. Also, please be mindful, even if you use the approach to "scream" there's a bug to get assistance, you still have to clearly identity it. Lastly, why would OpenWrt investigate problem in code written by another firmware manufacturer - for their firmware? :confused:

Without your cooperation (i.e. mentioning things should work manually, but not informing us if things do work manually now) and gleaning information from the OpenWrt Wiki and this post, I'm simply starting to agree with @mk24 that you may have overfilled your device.

Can you actually provide the real free and file totals; and other information previously requested?

I think you'll find it's over 9 MB.

@lleachii I'm NOT using this gl-inet stuff , I use official openwrt stable builds, 19.07.4 for tpl archer c7v5 . So why getting a foreign image,discuss about foreign code or sth? Just the problem seems to be the same looking at glinet.

If I upgrade via web ui - problem. Via ssh sysupgrade, too.

Ssh, running df -h , find /overlay , du -sh /overlay made sure its not over filled. Having ca. 2.2 MByte filled before running sysupgrade, while it executes it goes up to 4.4, i guess because it backups things for later.

Sysupgrade image is bigger, but not relevant for the used space in the overlay. Web ui and myself if doing via scp places the image in /tmp/firmware.bin . There are over 50 mbytes free on it.

@lleachii do you need screenshots? Or is it enough if some experienced it expert says he's tested 2 days and is sure there is sth wrong and he has observed it correctly.

@Catfriend1, I'm lost at how you think someone has tested two days.

And if you're referring to yourself, please provide the solution in a code report, developer list, etc.. To reiterate, just saying "there's a bug" doesn't identify where it is; nor get it fixed. OK, [hypothetically], there's a bug...now where is it and what's the fix?

This is community-based support and you provide little and conflicting information. I've had the following draft post saved for those 2 days:


:man_facepalming:

The script if from GL-INET! Look at your link:

:arrow_down:

GL-Inet does-not-equal OpenWrt. Are you identifying that code as the isue?

If so, locate the code in OpenWrt and note the bug there.

:open_mouth: /tmp and overlay are not the same...I'm not even sure how you honestly presented a number greater than 3 MB; but greater than your installed flash chip...hummm.

Yea, I think you may not understand.

@lleachii yeah your comments come up to me like side tracking my facts by interpreting my posts misunderstandingly and not willing to help. I'm no coder but admin and user. I know a lot from those perspectives about openwrt and I do know how to make a good bug report, investigate and clearly sum up what has proved to be wrong. I need help from a developer willing to take the report serious and looking at code with his expertise. I don't need your comments far away from doing so and accusing me why I can't communicate like a dev and do the bug hunting in code myself. I have done hours of flashing, resetting, starting afresh to prove there is a problem when openwrt handles the spi reads concurrently while restoring a 2 mbyte jffs overlay after sysupgrade. Period. Don't troll me. Let the others read , understand and help. You just did jump roughly over what I wrote in an offensive sounding way. I did not use glinet, so it's okay not to search if it is their bug too. But that is written there is a good pointer what could be the cause of this verified symptom in openwrt. If it's not useful, okay, then an expert needs to make theories, but the problem is real and will stay.

Please explain the Gl-INET link then (because I know I'm not the only once confused)...then you said:

So please explain what the link means???

Maybe this will help.

:laughing:

(A developer, or at least a Core Team Member, responded to you already! - See: Post No. 7 - You refused to explain why there's a GL-INET link.)