WRT32X USB-NAS R/W throughput

If "block info" does not show your HDD, then it's either disconnected from the router, or the router was reinstalled and you forgot to add USB drivers for kernel. Try this:

opkg update && opkg install kmod-usb2 kmod-usb-storage kmod-usb-storage-extras usbutils
lsusb # Does it show your usb-hdd?
block info # Does it show partitions of usb-hdd now?
opkg update && opkg install kmod-usb2 kmod-usb-storage kmod-usb-storage-extras usbutils
...
Signature check passed.
Package kmod-usb2 (4.14.180-1) installed in root is up to date.
Package kmod-usb-storage (4.14.180-1) installed in root is up to date.
Installing kmod-usb-storage-extras (4.14.180-1) to root...
Downloading http://downloads.openwrt.org/releases/19.07.3/targets/mvebu/cortexa9/kmods/4.14.180-1-a92a3f5c5bed2671533484c7ace9d5b5/kmod-usb-storage-extras_4.14.180-1_arm_cortex-a9_vfpv3-d16.ipk
Package usbutils (007-10) installed in root is up to date.
Configuring kmod-usb-storage-extras.

~# lsusb
Bus 003 Device 003: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

block info
/dev/mtdblock6: UUID="1431149825" VERSION="1" TYPE="ubi"
/dev/mtdblock8: UUID="469652220" VERSION="1" TYPE="ubi"
/dev/ubiblock0_0: UUID="b28d4e9e-5451dc49-133c1a49-dd884e53" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/ubi0_1: UUID="dadc7e3a-f378-4c5a-85c2-edece6f7195a" VERSION="w4r0" MOUNT="/overlay" TYPE="ubifs"
/dev/sda1: UUID="4861a676-43c2-4d71-a0e5-e4ae6d394ff1" LABEL="HD5" VERSION="1.0" MOUNT="/mnt/hd5" TYPE="ext4"

From client

sudo showmount -e 192.168.1.1
Export list for 192.168.1.1:
/mnt/hd5 192.168.1.0/255.255.255.0

sudo mount /mnt/HD5
[sudo] password for root: 
mount.nfs: Stale file handle

Doesn't mount

As you can see, /dev/sda1 is now properly mounted on the router. Restart NFS server (since it needs to realize that the HD5 drive reappeared) and try to mount it on the client again.

Back to good, mounted and I can see the file again.

 ls -la /mnt/HD5
total 107248
drwxrwxrwx 5 nobody nogroup      4096 Jul  6 19:31 .
drwxr-xr-x 5 root   root         4096 Jun 27 23:53 ..
-rwxrwxrwx 1 nobody nogroup         0 Jul  3 18:25 Hard_Drive_Mounted
drwxrwxrwx 2 nobody nogroup     16384 Jul  3 13:24 lost+found
drwxr-xr-x 2 nobody nogroup      4096 Jul  6 19:31 My new dir
-rw-r--r-- 1 nobody nogroup         0 Jul  6 19:31 My new file
-rw-r--r-- 1 nobody nogroup 109789184 Jul  5 16:40 tmpfile

Now I want "My new dir" and "My new file" with a+rw (777) when I create them.
AFAIK I should add umask=000 in the mount line. But what mount?
If client, only this client will have the good parameter.

The option is only supported for filesystems that do not natively support Unux-style permissions (.e.g FAT, NTFS). So, don't use it.

Since you have your system up and running again, think about the options that I told you earlier - rather then dealing with automatic ownership/permissions squashing by NFSD, just use explicit "chmod go+rw ..." on the client after you copy files onto the nfs share.

Seems to be working fine now

I'll stick at that
Now I am back to performance.
The real life is to copy dir and files in the share. Now I test with a dir with
Number of files: 9,994 (reg: 9,925, dir: 69)
mount with sync option

rsync -r --progress -h --stats /home/jp/Videos/tmpdir8 /mnt/HD5/

with rsync write is between 4.05 and 4.13 MB/s !!!

time (sync; scp -r /home/jp/Videos/tmpdir8 /mnt/HD5/; sync)

with scp write is around 6 MB/s

Change to async option in the 3 mount points, same command
with rsync write is around 48, 49 MB/s
with scp write is between 59 and 64 MB/s

I read that rsize and wsize should change rw performance. I notice in the PC

mount
...
192.168.1.1:/mnt/hd5 on /mnt/HD5 type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.1,mountvers=3,mountport=32780,mountproto=udp,local_lock=none,addr=192.168.1.1)

rsize and wsize = 65636
but I mounted a nfs with a laptop

192.168.1.115:/store on /mnt/Store type nfs4 (rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.110,local_lock=none,addr=192.168.1.115)

rsize wsize is 524288. a lot bigger
Should I try it with the router? Should I change as well server side (2 mounts)?
Edit: What is the nfs version in Openwrt, should it be the same in server and client?

What stops you from trying? :wink:
You have a unique hardware setup that is different from others, so try running tests and see what parameters are going to be optimal for your environment.

Also, when comparing test results - pay attention to details. For example - in the configurations that you mentioned above, your PC mounts NFS with nfs3 type and laptop with nfs4 type. So, the difference is not only in rsize/wsize parameters.

Question; What is the difference between block detect and cat /etc/config/fstab

~# block detect 
config 'global'
	option	anon_swap	'0'
	option	anon_mount	'0'
	option	auto_swap	'1'
	option	auto_mount	'1'
	option	delay_root	'5'
	option	check_fs	'0'

config 'mount'
	option	target	'/mnt/hd5'
	option	uuid	'4861a676-43c2-4d71-a0e5-e4ae6d394ff1'
	option	enabled	'0'

~# cat /etc/config/fstab

config global
	option anon_swap '0'
	option anon_mount '0'
	option auto_swap '1'
	option auto_mount '1'
	option delay_root '5'
	option check_fs '0'

config mount
	option target '/mnt/hd5'
	option uuid '4861a676-43c2-4d71-a0e5-e4ae6d394ff1'
	option enabled '1'
	option fstype 'ext4'
	option options 'rw,async'

enable 0 with block detect, 1 in fstab
options only in fstab
I make the change with vi /etc/config/fstab, I hope this is the right way

Edit:
One more question:
In PC client I get

mount
...
192.168.1.1:/mnt/hd5 on /mnt/HD5 type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.1,mountvers=3,mountport=32780,mountproto=udp,local_lock=none,addr=192.168.1.1)

cat /etc/fstab
...
192.168.1.1:/mnt/hd5                       /mnt/HD5    nfs    rw,async,wsize=524288              0  0

Which one has the values (wsize) used by nfs?

When setting up a new drive or system, "block detect" gives you a sample configuration for the currently attached block devices, so that you can persist it in "/etc/config/fstab" and fine-tune it further. I completely forgot about that option, since you rarely change "fstab" file after setting up the system. When the router starts up, it uses "fstab" file to mount drives. Additional help can be found here: https://openwrt.org/docs/techref/block_mount

Similar case with "mount" vs. "/etc/fstab" - "fstab" is the configuration used by OS during startup and manual mount/umount manipulations. But running "mount" produces the list of current storage devices mounted on the machine and their configuration options.

In server

~# cat /proc/fs/nfsd/max_block_size
65536

I want to change block value in the server

Check out /proc/fs/nfsd/max_block_size if client reports other block size then what you have specified. It can be changed if the server is not running, but nfsd mounted. Now you can echo the right value to it. - Updated by fathom

I stopped nsfd and rpcd but I get the error

 echo 131072 > /proc/fs/nfsd/max_block_size
-ash: can't create /proc/fs/nfsd/max_block_size: nonexistent directory

What does mean "but nfsd mounted" ?

I'm not sure. :man_shrugging: Probably a typo.
Everyone is saying that the value in "/proc/fs/nfsd/max_block_size" needs to be set before starting NFSD, but OpenWRT's "procd" does not allow creating a file in that folder. Only changing an existing file, which will unlikely change NFSD after it started.

OK, it is not possible to change block_size.

So this is almost the end of the tests. I found the versions available from the client

~> sudo rpcinfo -p 192.168.1.1
[sudo] password for root: 
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   udp  32777  nlockmgr
    100021    3   udp  32777  nlockmgr
    100021    4   udp  32777  nlockmgr
    100021    1   tcp  32777  nlockmgr
    100021    3   tcp  32777  nlockmgr
    100021    4   tcp  32777  nlockmgr
    100024    1   udp  32778  status
    100024    1   tcp  32778  status
    100005    1   udp  32780  mountd
    100005    1   tcp  32780  mountd
    100005    2   udp  32780  mountd
    100005    2   tcp  32780  mountd
    100005    3   udp  32780  mountd
    100005    3   tcp  32780  mountd

I added vers=4 in fstab. Now I have in client /etc/fstab

192.168.1.1:/mnt/hd5                       /mnt/HD5    nfs    rw,async,wsize=65536,rsize=65536,vers=4              0  0

I tried to copy a directory with 69 sub-dir and 6925 files with client options:
sync/async
wsize, rsize 1024, 65536
All write tests was between
Protocol scp 63 and 65 MB/s
Protocol rsync 49 and 50 MB/s
Only async improved the speed from
rsync 4 (four) MB/s to 49MB/s
scp 6 MB/s to 63 MB/s
Other options did NOT change write speed.

Read test with the last options
Protocol scp and rsync around 49 MB/s

Do you have any other idea, option to get it better?

This morning I boot my PC/client. I try to run again one test with scp. After a while, error: /mnt/HD% is read-only.
I reboot the router/server, nothing better, reboot the PC, nothing ... reboot again the router and back working, rw.
I test again with scp, get the result, OK. I run again the same test and after a while command freeze, no return after more than 5mn.
It looks like this system is not reliable.
I am wondering if it could be better to buy a real NAS with Ethernet, more expensive but (maybe) no problem!?

Edit: Still no return and now LuCi freeze!!!
Edit2: router log

[    7.700536] EXT4-fs warning (device sda1): ext4_clear_journal_err:4982: Filesystem error recorded from previous mount: IO failure
[    7.712260] EXT4-fs warning (device sda1): ext4_clear_journal_err:4983: Marking fs in need of filesystem check.
[    7.803868] EXT4-fs (sda1): warning: mounting fs with errors, running e2fsck is recommended

Edit3: "The superblock could not be read or..."
I'll have to format the HDD as I never succeeded to fix superblock corruption.

Yesterday, before posting a suggestion to try Samba, I wanted to check the throughput on my NFS setup (even though it uses encrypted drive) ... and after several tests, all ssh sessions to the router froze and when I opened a new one, the USB-HDD was no longer mounted. WTF!? Rebooted the router and everything was back to normal, but it was not a fun experience!

Here are the NFS test results on a 3GB file before the crash:

  • reading from the server with "rsync" - 118MB/s on average
  • writing with "rsync" - 65MB/s on average, but the speed fluctuates a lot during the test due to large RAM that allows it to fill up buffers quickly before flushing them to the disk
  • the final blow to the router was done by a test with "pv" (essentially "cp"), but the range of speeds reported at runtime was crazy - up to 250MB/s (while loading PC's buffers) and down to 1MB/s (while waiting for router to flush buffers and start accepting new data)

If I find time, I'll run some more tests on a spare router with OpenWRT 18 and 19 just to check the version which is stable.

Meanwhile, I'd suggest @Toutatis to try configuring Samba and testing it out. I've used Samba a lot and had never experienced such problems as you reported in this thread. So, maybe it can bring the stability? Good luck!

@wind
Could you recall me what is your config, drive and router, please?

Two times I got the same issue. From a brand new formatted partition, no issue in log, I run scp test. First was good. After ~30s I run the same again and LED stop binding after few seconds and LuCi freeze. I waited around half hour and got

~> time (sync; scp -r /home/jp/Videos/tmpfile8 /mnt/HD5/media; sync)
cp: error writing '/mnt/HD5/media/tmpfile8': Read-only file system

real	26m46.227s
user	0m0.003s
sys	0m2.911s

umount/mount from client didn't change anything
server

~# ls -la /mnt/hd5
drwxrwxrwx    4 nobody   nogroup       4096 Jul 10 12:51 .
drwxr-xr-x    1 root     root           224 Jul 10 10:42 ..
drwxrw-rw-    2 nobody   nogroup      16384 Jul 10 10:47 lost+found
drwxrwxrwx    2 nobody   nogroup       4096 Jul 10 12:51 media

~# touch /mnt/hd5/media/testfile
touch: /mnt/hd5/media/testfile: Read-only file system

~# umount /mnt/hd5
umount: can't unmount /mnt/hd5: Resource busy

media dir is read-only and I can't umount!!!!
I give up for now, I'll be back later
Many thanks

My setup: WRT3200ACM with WD MyBook USB-HDD. The file "/etc/exports" is:

/share  192.168.1.0/24(fsid=0,rw,wdelay,async,all_squash,no_subtree_check,insecure,crossmnt)

Now that you're mentioning your router crashing to the point that even LED lights went out ... I'm not really sure what happened to mine. I saw all ssh sessions freeze, but was able to open a new one after a minute of waiting for them to unfreeze. So, maybe it rebooted too, but I did not check the uptime.

Sorry, I didn't say it right; The HDD LED stopped blinding, become steady, the router still working for routing but LuCi freeze on opening Mount Point window. I had to switch the router off.
I'll will check what are wdelay, insecure from your export maybe good for me as well.

New issue while transferring data to HDD.
See this thread.

I'll be back here for more speed tests

You could have a look at the thread in my previous post. Maybe a fix for your issue.

Hello, did issue with max speed ~11MB/s over lan cable fixed?
I have samba and copying file just using Explorer.

mount: /dev/sda1 on /mnt/sda1 type ext3 (rw,relatime)

root@Matilda:~# dmesg | grep usb
[    0.018772] usbcore: registered new interface driver usbfs
[    0.018792] usbcore: registered new interface driver hub
[    0.018814] usbcore: registered new device driver usb
[    1.588486] orion-ehci f1058000.usb: EHCI Host Controller
[    1.593921] orion-ehci f1058000.usb: new USB bus registered, assigned bus number 1
[    1.601567] orion-ehci f1058000.usb: irq 46, io mem 0xf1058000
[    1.650024] orion-ehci f1058000.usb: USB 2.0 started, EHCI 1.00
[    1.664274] usbcore: registered new interface driver usb-storage
[    1.753825] usb_phy_generic usb3_1-phy: dummy supplies not allowed for exclusive requests
[    2.256903] xhci-hcd f10f8000.usb3: xHCI Host Controller
[    2.262257] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 2
[    2.269824] xhci-hcd f10f8000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x0000000000010010
[    2.279108] xhci-hcd f10f8000.usb3: irq 52, io mem 0xf10f8000
[    2.292986] xhci-hcd f10f8000.usb3: xHCI Host Controller
[    2.298325] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 3
[    2.305860] xhci-hcd f10f8000.usb3: Host supports USB 3.0 SuperSpeed
[    2.320233] usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
[    3.126595] usb 3-1: new SuperSpeed Gen 1 USB device number 2 using xhci-hcd
[    3.166825] usb-storage 3-1:1.0: USB Mass Storage device detected
[    3.197723] scsi host2: usb-storage 3-1:1.0
[    8.666803] usbcore: registered new interface driver uas
[   16.997716] usbcore: registered new interface driver btusb