Paragon NTFS3 driver backported to OpenWrt 5.10 kernel - how to test

Summary (tl;dr)
It is eminently doable to get Paragon's ntfs read/write (ntfs3) driver into OpenWrt today. Further work and testing are, of course, required, but the following proof of concept works and the performance gains are not insignificant, even over the in-kernel antfs driver.

EDIT: Reference 4 below now contains pre-built binaries for a few platforms. See the later post in this thread for a list of platforms which are included. It also contains a README with a mostly copy-and-paste set of instructions for building.

References:

  1. Paragon's initial announcement/release of their NTFS driver
  2. Complete Linux Kernel 5.15 with Paragon's in-tree NTFS read-write driver (NTFS3)
  3. ARCH Linux backport patches of the in-tree NTFS3 driver to kernel 5.12
  4. My 5.12 -> 5.10 backport patch (plus a complete unified diff with driver and backports applied)

Background:
In August last year Paragon released (1) an open source rewrite of their decade (at least) old Linux NTFS driver. It is available in two flavours: a module designed to be compiled outside the Linux kernel tree, and an in-tree version (2) designed to be made by the standard Linux kernel config and build system. The Arch Linux project then created (3) a set of patches to backport this driver back to kernel 5.12. I did a little work to bring this backport back to kernel 5.10 (4) and have successfully tested it in OpenWrt.

Procedure:
This kernel module is not (yet) in a state to be made into an OpenWrt package. To use it you must, in broad strokes:

  1. Set up and build any version of OpenWrt that uses kernel 5.10
  2. Patch the in-OpenWrt-tree kernel with the unified diff I made (4)
  3. Perform a $ make kernel_menuconfig, navigate to File systems -> DOS/FAT/EXFAT/NT Filesystems and turn on "NTFS Read-Write file system support" as a module
  4. Optionally you may turn on the option "activate support of external compressions lzx/xpress", but not (yet) "NTFS POSIX Access Control Lists"
  5. $ make the kernel
  6. Copy out the ntfs3.ko created in the kernel treet, and move it to a device running the same snapshot you downloaded and built above.

To do the above, I have created a unified patch (4) that combines all the individual Arch backport patches, my own modest patch, and which also patches it into fs/Makefile and fs/Kconfig. I created this patch by:

  1. Downloading the Linux tree with the in-tree driver from Paragon's git repository (2)
  2. Downloading the Arch patches from their git repository (3)
  3. Pulling fs/ntfs3/ from Paragon's 5.15 kernel tree and putting it in OpenWrt's kernel 5.10 tree
  4. Applying, in order, Arch's 5.15, 5.14, and 5.12 backport patches
  5. Editing xattr.c to fix minor issues and creating my own 5.10 backport patch with these changes
  6. Editing fs/Makefile and fs/Kconfig to add ntfs3 to both
3 Likes

Seems that part would negate "today"...

So that's a big reach.

I didn't say wide deployment. I set getting it into OpenWrt. Since it's sitting on my R6700v2, today as I type this, I assess it is not as much of a reach as you maintain.

1 Like

Maintenance doesn't stop immediately after throwing a patchset over the fence, it needs to remain supported for OpenWrt's predictable 2-3 year release cycle. It doesn't exactly instil confidence either that Paragon has gone silent pretty much immediately after ntfs3 has been merged mainline - and their userspace support tools (mkfs/ fsck) are also still 'pending'.

3 Likes

Initial post was long, so I left out the preliminary testing. Initial performance testing is encouragingly positive:
mt7621-based Netgear R6700v2 test platform with OpenWrt snapshot with a Seagate 8TB USB3 external archival drive (~150MB/s sustained write speed when connected to my laptop)

antfs:

root@shadow:/mnt/Trunk# pv -S -s 1G /dev/zero > zero
1.00GiB 0:00:21 [46.7MiB/s] [===================================>] 100%
root@shadow:/mnt/Trunk# pv zero > /dev/null 
1.00GiB 0:01:28 [11.6MiB/s] [===================================>] 100%

CPU 4-logical-core average, 54.5%.
Breakdown: 80%/51%/50%/47%

ntfs3:

root@shadow:/mnt/Trunk# pv -S -s 1G /dev/zero > zero
1.00GiB 0:00:12 [79.5MiB/s][===================================>] 100%
root@shadow:/mnt/Trunk# pv zero > /dev/null 
1.00GiB 0:00:11 [85.4MiB/s] [==================================>] 100% 

CPU 4-logical-core average, 38.75%
Breakdown: 87%/24%/25%/19%

The CPU load was measured with htop on a 10-second average.

This isn't an extensive test by any means. Sustained writes appear to be about 70% faster with ntfs3, and when you factor out the CPU core with pv itself, the CPU load for ntfs3 is about half that of antfs. I'm not quite sure what to make of antfs's terrible sustained read speed, but I ran the test several times on files located all over that drive. Can someone independently verify?

Perhaps. To that I'll only point out that antfs is itself a homebrew kitbash of the user-space ntfs-3g library into kernel space and it hasn't seen an official commit in the four years since it was first made.

<soapbox>Point is, I'm not slagging antfs. It was a great idea and it filled (fills) a need. The need was the fact that the FUSE ntfs-3g was the rate limiting step in the network file access chain for NTFS. Right now, for even lower end consumer-grade networking devices, antfs is (by a large factor) the rate limiting step in that same chain. If it was the network, I'd say screw it, we can afford to make people wait a year and a half. But I maintain the same need exists to bring in ntfs3 today that existed when the same decision was made for antfs.</soapbox>

3 Likes

I have made pre-built ntfs3 kernel module binaries for a few platforms to make it easier for people to play with. They are included for the following platforms:
RAMIPS/mt7621
MEDIATEK/mt7222
MEDIATEK/mt7623
IPQ806x/generic

These are located in the patch download at reference 4 above. Anyone without the resources to build the module for themselves and who wants to test it, feel free to let me know and I may be willing to add more platforms to the above.

That being said, to make building it a little easier I have also included a README in the tarball with a mostly copy-and-paste set of commands.

1 Like

I came here due to some searching regarding OpenWRT and antfs, because the last time I rebuilt OpenWRT I switched. (I've been using ntfs-3g for quite a while). It turns out antfs was a gigantic mistake. (It took a while to notice because I haven't used DLNA from the router for a while...)

On an Archer C2600, a 50MB read takes more than 50 seconds with antfs. Yes, that's right, I'm seeing sustained reads of <1 MB/second, and it's CPU-bound. ntfs-3g can do 38+ and I'm pretty sure it's bound by the limits of a 5400RPM rustspinner. (Will check CPU utilization tonight.)