Thank you for your patience. Finally, it's done. Here I share my config files in case other WRT1900ACS users want to do the same thing.
IT'S ONLY TESTED ON OPENWRT 19.07.3 .
I saw snapshot changed directories and files. Need furthur customization I guess.
The fstab config file to be preinstalled here: 'target/linux/mvebu/base-files/etc/config/fstab'.
[root@WRT1900ACS:~]# cat /etc/config/fstab
config global
option anon_swap '0'
option anon_mount '0'
option auto_swap '1'
option auto_mount '1'
option delay_root '5'
option check_fs '1'
config mount 'overlay'
option enabled '1'
option enabled_fsck '1'
option fstype 'ubifs'
option options 'bulk_read,compr=zlib'
option target '/overlay'
option device '/dev/ubi8_0'
Find syscfg device by running command below. In my case for WRT1900ACS, it's mtd8.
[root@WRT1900ACS:~]# cat /proc/mtd
dev: size erasesize name
mtd0: 00200000 00020000 "u-boot"
mtd1: 00040000 00020000 "u_env"
mtd2: 00040000 00020000 "s_env"
mtd3: 00100000 00020000 "devinfo"
mtd4: 02800000 00020000 "kernel1"
mtd5: 02200000 00020000 "ubi"
mtd6: 02800000 00020000 "kernel2"
mtd7: 02200000 00020000 "rootfs2"
mtd8: 02600000 00020000 "syscfg"
mtd9: 00680000 00020000 "unused_area"
Remove Preinit file 'target/linux/mvebu/base-files/lib/preinint/81_linksys_syscfg'. Split it into 2 files as below and put in the same folder.
File 1: 78_linksys_syscfg_overlay
#
# Copyright (C) 2014-2016 OpenWrt.org
# Copyright (C) 2016 LEDE-Project.org
#
preinit_mount_syscfg_overlay() {
. /lib/functions.sh
case $(board_name) in
linksys,caiman|linksys,cobra|linksys,mamba|linksys,rango|linksys,shelby|linksys,venom)
needs_recovery=0
syscfg_part=$(grep syscfg /proc/mtd |cut -c4)
ubiattach -m $syscfg_part -d $syscfg_part || needs_recovery=1
if [ $needs_recovery -eq 1 ]
then
echo "ubifs syscfg partition is damaged, reformatting"
ubidetach -m $syscfg_part
ubiformat -y -O 2048 -q /dev/mtd$syscfg_part
ubiattach -m $syscfg_part -d $syscfg_part
ubimkvol /dev/ubi1 -n 0 -N syscfg -t dynamic --maxavsize
echo "reformatting syscfg..." >> /tmp/syscfg_overlay.log
else
node=$(awk '/'"ubi$syscfg_part"'/{print $1}' /proc/devices)
if [ "x$node" != "x" ]; then
echo "creating syscfg node..." >> /tmp/syscfg_overlay.log
[ "$(ls /dev | grep ubi$syscfg_part)" == "" ] && mknod -m 600 /dev/ubi${syscfg_part} c $node 0 &> /dev/null && echo "creating /dev/ubi$syscfg_part ..." >> /tmp/syscfg_overlay.log
[ "$(ls /dev | grep ubi${syscfg_part}_0)" == "" ] && mknod -m 600 /dev/ubi${syscfg_part}_0 c $node 1 &> /dev/null && echo "creating /dev/ubi${syscfg_part}_0 ..." >> /tmp/syscfg_overlay.log
fi
fi
mkdir /tmp/olaychk
mount -t ubifs /dev/ubi${syscfg_part}_0 /tmp/olaychk && {
[ -f "/tmp/olaychk/.syscfg_overlay" ] && echo "test mount ok..." >> /tmp/syscfg_overlay.log
if [ -f "/tmp/olaychk/.firstboot" ]; then
[ "$(ls -a /tmp/olaychk | grep .fs_state)" != "" ] && rm -f /tmp/olaychk/.fs_state && echo "delete old .fs_state..." >> /tmp/syscfg_overlay.log
[ -d "/tmp/olaychk/upper" ] && rm -rf /tmp/olaychk/upper && echo "delete old upper..." >> /tmp/syscfg_overlay.log
[ -d "/tmp/olaychk/work" ] && rm -rf /tmp/olaychk/work && echo "delete old work..." >> /tmp/syscfg_overlay.log
[ -d "/tmp/olaychk/etc" ] && rm -rf /tmp/olaychk/etc && echo "delete old etc..." >> /tmp/syscfg_overlay.log
rm -f /tmp/olaychk/.firstboot
fi
umount /tmp/olaychk
rm -r /tmp/olaychk
}
;;
esac
}
boot_hook_add preinit_main preinit_mount_syscfg_overlay
File 2: 81_linksys_syscfg
#
# Copyright (C) 2014-2016 OpenWrt.org
# Copyright (C) 2016 LEDE-Project.org
#
preinit_mount_syscfg() {
. /lib/functions.sh
. /lib/upgrade/common.sh
case $(board_name) in
linksys,caiman|linksys,cobra|linksys,mamba|linksys,rango|linksys,shelby|linksys,venom)
syscfg_part=$(grep syscfg /proc/mtd |cut -c4)
if [ "$(mount | grep overlay | grep ubi${syscfg_part}_0)" != "" ]; then
# if mount syscfg as overlay successfully, then link /tmp/syscfg to /overlay.
ln -s /overlay /tmp/syscfg && echo "ceate sombolic link for /tmp/syscfg ..." >> /tmp/syscfg_overlay.log
else
# if mount syscfg as overlay failed, then try to mount syscfg itself.
mkdir /tmp/syscfg
mount -t ubifs ubi1:syscfg /tmp/syscfg && echo "mount syscfg itself ..." >> /tmp/syscfg_overlay.log
fi
[ -f "/tmp/syscfg/$BACKUP_FILE" ] && {
echo "restoring config ..." >> /tmp/syscfg_overlay.log
echo "- config restore -"
cd /
mv "/tmp/syscfg/$BACKUP_FILE" /tmp
tar xzf "/tmp/$BACKUP_FILE"
rm -f "/tmp/$BACKUP_FILE"
sync
}
;;
esac
}
boot_hook_add preinit_main preinit_mount_syscfg
And append code in function platform_do_upgrade_linksys in file 'target/linux/mvebu/base-files/lib/upgrade/linksys.sh'.
platform_do_upgrade_linksys() {
...
# if using syscfg as extroot, then create .firstboot file to indicate it's newly flashed rom.
# when 78_syscfg_ubi_attach detects this file, it will delete old overlay files.
syscfg_part=$(grep syscfg /proc/mtd |cut -c4)
[ "$(mount | grep overlay | grep ubi${syscfg_part}_0)" != "" ] && touch /overlay/.firstboot
}
That's all the modifications needed. I didn't find a good solution to detect if it's first time boot. If you have better solution, please tell me.
Here's the larger size overlay. Take advantage of syscfg partition.
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 19.07.3, r11063-85e04e9f46
-----------------------------------------------------
[root@WRT1900ACS:~]# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 24.8M 24.8M 0 100% /rom
tmpfs 249.5M 2.7M 246.8M 1% /tmp
/dev/ubi8_0 29.6M 700.0K 27.3M 2% /overlay
overlayfs:/overlay 29.6M 700.0K 27.3M 2% /
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 915.9G 607.1G 263.2G 70% /mnt/nas
After all, it took a lot of time to try and debug. Flash again and again. But finally, it's done. Thank you again for all the help! Cheers! 
