Mvebu wrt3200acm file rename problem - kernel 4.9 and ubifs? (FS#579)

@cybrnook @stangri & other wrt1900/1200/3200ac users

I have noticed a strange rename problem with the newest r3623 master builds, both my own and the buildbot snapshot.

In nutshell, in the current r3623 version renaming (or moving) a file fails if that file exists both in /rom (=the flashed firmware) and in /overlay (=the later changes to the file). So, for example the settings files related to the packages included in the build are touched by the bug. This may cause subtle errors in scripts and packages.

Rename fails.
Deleting the same file succeeds.
Copying is also ok.

The problem does not exist in 17.01.0 that branched of in January, so this is something new. I am trying to identify if this has surfaced with the kernel version bump to 4.9 two weeks ago (Feb 16th). Or did it break at the bump to 4.9.13 ? Or earlier?

Details and a longer story are in the bug report:

In the current r3623 with kernel 4.9.13 this fails with "Invalid argument":

root@LEDE:~# mv /etc/dropbear/dropbear_rsa_host_key /etc/dropbear/dropbear_rsa_host_key-x
mv: can't rename '/etc/dropbear/dropbear_rsa_host_key': Invalid argument

In 17.01.0 it works ok:

root@LEDE:/etc# mv /etc/dropbear/dropbear_rsa_host_key  /etc/dropbear/dropbear_rsa_host_key-x

I am using /etc/dropbear/dropbear_rsa_host_key as the test case here as it should exist in each build as long you have the default SSH server (dropbear).

I will try and test for you this evening.

I compiled myself the last commit where mvebu is with kernel 4.4 and rename works normally. the next commit bumping mvebu to 4.9 makes rename to fail. So it is pretty evident that this has came with kernel 4.9.

supposedly kernel 4.9 supports ubifs rename "built-in", so the patches for that were dropped from LEDE.

r3509-2374549916 kernel 4.4 works ok.
r3510-2bf9ea6a31 kernel 4.9 fails to rename

Linux kernel's ubifs rename support commits have been committed on 2016-10-02:

The interesting question is: does this affect the other target platforms with kernel 4.9 and ubifs...

1 Like

weird bug ... anyway, renaming in mc (midnight commander) still works as expected ...:slight_smile:

Probably depends on how mc performs the rename. Using copy+delete action or using the rename system function. You are sure that you tested with a suitable file?

And yes, it is weird. I spent quite a long time patching debug statements to the new opkg after I noticed that opkg did not add a copy of adblock default settings as adblock-opkg config file when I reinstalled the new adblock 2.4.0-2 in mvebu, so you are partially culprit for me to spend time with this bug. I first thought that this is related to the new opkg-lede, but finally I realised that this is much more deeply in the filesystem :frowning:

Likely it is about overlayfs not playing well with the native ubifs rename/move in 4.9, as otherwise identical non-ubi ar71xx 4.4 build works as expected.

Personally I find this bug very nasty, as it goes to the core of the filesystem. You simply expect a "legal and correct"" rename to work. Now it fails without a clear explanation.

Confirmed that mv doesn't work and that rename with mc does on r3627.

I've re-tested this on an apu2 device (with a "normal" ext4 filesystem) - mv, cp, rm ... no issues so far.

Reboot (SNAPSHOT, r3617-f9da6238e4)
root@blackhole:~# uname -a
Linux blackhole 4.9.13 #0 SMP Tue Feb 28 19:22:10 2017 x86_64 GNU/Linux
root@blackhole:~# mv /etc/adblock/adblock.whitelist /etc/adblock/adblock.whitelist-1

I think that I have identified the culprit commit causing the rename bug for ubifs with overlayfs, but I am not sure about the fix. Deleting the offending two lines?

I compared the respective files in 4.4 and 4.9. I took the files that the kernel 4.4 patches 052-... handled, meaning fs/ubifs/ubifs.h, journal.c and dir.c.

  • 4.4 patched journal.c equals 4.9 journal.c
  • 4.4 patched ubifs.h has only trivial differences to 4.9
  • dir.c has otherwise only trivial changes (e.g. mutex lock changes), but there is one significant change in 4.9, where "return -EINVAL" has been inserted.

	struct timespec time;
	unsigned int uninitialized_var(saved_nlink);

	if (flags & ~RENAME_NOREPLACE)
		return -EINVAL;


The change is here:

Full commit:

fs: support RENAME_NOREPLACE for local filesystems

This is trivial to do:

 - add flags argument to foo_rename()
 - check if flags doesn't have any other than RENAME_NOREPLACE
 - assign foo_rename() to .rename2 instead of .rename

Filesystems converted:

affs, bfs, exofs, ext2, hfs, hfsplus, jffs2, jfs, logfs, minix, msdos,
nilfs2, omfs, reiserfs, sysvfs, ubifs, udf, ufs, vfat.

(overlayfs is not mentioned in the list.)

I patched my system with this:


--- a/fs/ubifs/dir.c
+++ b/fs/ubifs/dir.c
@@ -1088,8 +1088,10 @@
 	struct timespec time;
 	unsigned int uninitialized_var(saved_nlink);
+	/*
	if (flags & ~RENAME_NOREPLACE)
		return -EINVAL;
+	*/
 	 * Budget request settings: deletion direntry, new direntry, removing

At the first glance it looks ok:
Renaming a problematic file works again, and both the deletion record and the newly named file are visible in /overlay. And /etc looks normal:

root@LEDE:~# mv /etc/dropbear/dropbear_rsa_host_key  /etc/dropbear/dropbear_rsa_host_key-x
root@LEDE:~# ls -l /rom/etc/dropbear/
-rw-r--r--    1 root     root             0 Mar  2 10:09 dropbear_rsa_host_key
root@LEDE:~# ls -l /overlay/upper/etc/dropbear/
-rwxr--r--    1 root     root           959 Feb 18 18:09 authorized_keys
c---------    1 root     root        0,   0 Mar  2 10:16 dropbear_rsa_host_key
-rw-------    1 root     root           804 Feb 20 17:13 dropbear_rsa_host_key-x
root@LEDE:~# ls -l /etc/dropbear/
-rwxr--r--    1 root     root           959 Feb 18 18:09 authorized_keys
-rw-------    1 root     root           804 Feb 20 17:13 dropbear_rsa_host_key-x