Getting error failed to sync jffs2 overlay

I'm getting the error bellow at boot using a custom build from 18.06.04 (same happens with 18.06.02)

daemon.err mount_root: failed to sync jffs2 overlay

From docs jffs2 is a compressed filesystem and overlay is the layer that provides the access to this filesystem merged with the read-only filesystem . Device is functional even with this error but I not sure what this error can cause and what's causing it. Any help is appreciated.

I'm going to guess that you've got a unit with 4 MB of flash or less, or have installed several packages or files on your overlay.

JFFS2 requires at least three erase blocks (typically 192 kB) free to be "safely" functional and at least one erase block (64 kB) free at all times to be marginally functional (this is after including the size of files to be written).

Hi Jeff, thanks for answering. The device is supposed to have a 16MB flash memory. Runing df -f it returns me the following:

root@98-DA-C4-30-8F-26:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 3.8M      3.8M         0 100% /rom
tmpfs                    61.2M    444.0K     60.7M   1% /tmp
/dev/mtdblock6            9.8M    540.0K      9.3M   5% /overlay
overlayfs:/overlay        9.8M    540.0K      9.3M   5% /
tmpfs                   512.0K         0    512.0K   0% /dev

I do add some extra packages but I removed many of them to see if it would solve the problem.

It only happens after flashing the firmware by the way.

It takes 30 seconds or more before the JFFS2 file system is initialized and usable after a flash. Does this occur after that file system is acknowledged as ready?

Apparently it is after.

Mon Nov  4 13:02:49 2019 user.notice kernel: [    8.373889] mount_root: jffs2 not ready yet, using temporary tmpfs overlay
Mon Nov  4 13:03:00 2019 kern.warn kernel: [   30.732739] jffs2_scan_eraseblock(): End of filesystem marker found at 0x0
Mon Nov  4 13:03:00 2019 kern.warn kernel: [   30.766785] jffs2_build_filesystem(): unlocking the mtd device...
Mon Nov  4 13:03:00 2019 kern.warn kernel: [   30.786760] done.
Mon Nov  4 13:03:01 2019 kern.warn kernel: [   30.788765] jffs2_build_filesystem(): erasing all blocks after the end marker...
Mon Nov  4 13:03:37 2019 kern.notice kernel: [   68.570928] jffs2: notice: (1546) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
Mon Nov  4 13:03:38 2019 daemon.info mount_root: performing overlay whiteout
Mon Nov  4 13:03:38 2019 daemon.info mount_root: syncronizing overlay
Mon Nov  4 13:03:38 2019 daemon.err mount_root: failed to sync jffs2 overlay


Just curious: What device exactly are we talking about? We have roughly 400 devices with 16MB in our list :slight_smile:

Hi. That's the TP-Link Archer C7 V5!

I am getting the same error after an upgrade. No packages installed outside of those pre-built into the firmware file. Plenty of space left on the device itself (GL-B1300 running 19.07-SNAPSHOT). I did notice, that mount_boot takes several minutes minutes to finish running.

egrep "jffs|overlay" /var/log/system.log 
kern.warn kernel: [   64.577915] jffs2_scan_eraseblock(): End of filesystem marker found at 0x0
kern.warn kernel: [   64.577981] jffs2_build_filesystem(): unlocking the mtd device...
kern.warn kernel: [   64.589844] jffs2_build_filesystem(): erasing all blocks after the end marker...
kern.notice kernel: [  122.249514] jffs2: notice: (3665) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
daemon.info mount_root: performing overlay whiteout
kern.warn kernel: [  124.642967] overlayfs: upper fs does not support tmpfile.
daemon.info mount_root: syncronizing overlay
daemon.err mount_root: failed to sync jffs2 overlay
df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                16.3M     16.3M         0 100% /rom
tmpfs                   122.5M    228.0K    122.3M   0% /tmp
tmpfs                   122.5M    520.0K    122.0M   0% /tmp/root
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock11          11.6M      2.2M      9.3M  19% /overlay
overlayfs:/overlay       11.6M      2.2M      9.3M  19% /

Out of the two routers I have, this only happens on a GL-B1300 and not on R7800: is this a sign of a HW issue?

I am getting the same error in the master build as well and it is seen in both GL-B1300 that I have, so not a HW issues. The routers are otherwise functional. I wish someone could shed some light on this error message.

Not much light, but...

libfstools/overlay.c

        case FS_DEADCODE:
                if (switch2jffs(v))
                        return -1;

                ULOG_INFO("performing overlay whiteout\n");
                umount2("/tmp/root", MNT_DETACH);
                foreachdir("/overlay/", handle_whiteout);

                /* try hard to be in sync */
                ULOG_INFO("syncronizing overlay\n");
                if (system("cp -a /tmp/root/upper/* / 2>/dev/null"))
                        ULOG_ERR("failed to sync jffs2 overlay\n");
                break;

busybox's libbb/copy_file.c unfortunately doesn't seem to return distinct error codes, though it does look to give some kind of error messages in some cases with bb_perror_msg()

Perhaps

if (system("cp -av /tmp/root/upper/* / 2>/dev/null"))

could provide some insight?

Thank you; for whatever reason I did not consider looking at the source code. It is too simple now...

  1. OpenWrt router does not have a syslog configured by default, so the errors are not visible usually.
  2. The line you pointed out is copying a /tmp/root/upper/etc/uci-defaults/luci-sqm: character special (0/0) into /etc/uci-defaults/, BUT those are not regular files any more
  3. I guess the filesystem does not allow copying those files.
  4. I tried cp -a /overlay/upper/etc/uci-defaults/bcp38 /root/test/ on R7800 & GL-B1300 and it was unsuccessful both times, so I am not sure why I am only seeing these on the GL router.
  5. These routers do have /overlay mounted differently, though:
  • R7800 : /dev/ubi0_1 on /overlay type ubifs (rw,noatime,ubi=0,vol=1)
  • GL-B1330: /dev/mtdblock11 on /overlay type jffs2 (rw,noatime)
cp -a /tmp/root/upper/* /root/test/
cp: can't create '/root/test/etc/uci-defaults/luci-sqm': Operation not permitted
cp: can't create '/root/test/etc/uci-defaults/ddns': Operation not permitted
cp: can't create '/root/test/etc/uci-defaults/bcp38': Operation not permitted
ls -la /tmp/root/upper/etc/uci-defaults/
c---------    1 root     root        0,   0 Nov 26 16:48 bcp38
c---------    1 root     root        0,   0 Nov 26 16:48 ddns
c---------    1 root     root        0,   0 Nov 26 16:48 luci-sqm

UPDATE: The output above is truncated; there are more character files there than three.

And this could be why the issue is not happening on r7800 (ubifs): a different approach is used that might not be copying the files...

320         case FS_UBIFS:
321                 if (overlay_mount(v, fs_name))
322                         return -1;
323                 if (mount_move("/tmp", "", "/overlay") || fopivot("/overlay", "/rom")) {
324                         ULOG_ERR("switching to %s failed\n", fs_name);
325                         return -1;
326                 }
327                 break;
328         }
1 Like

I’ll bet that’s the “erasure mark” of the successful run of the script (I haven’t checked how overlayfs marks). Maybe it should be handled differently. If so, maybe it’s completing after the erasure marks get handled - “bad timing”.

It is the erasure mark for sure. I will try to figure out what package it belongs to and raise a bug in a proper place.
Looks like there is no side effect, though: all other files are still copied over and there is no action taken based on the error code.