WRT32X USB-NAS R/W throughput

Toutatis · June 24, 2020, 8:18pm

Hi there,
I have read that modern routers can be used as USB-NAS controler.
I own a Linksys WRT32X with OpenWRT 19.07.3. and a HDD-USB3 enclosure.
I found some tests/measurements that say I could reach 60+ Mbytes/s write/read with WRT32X. See here.
I tried from my PC Opensuse 15.1 to router, 1Gbits/s wired to confirm.
Write

dd if=/dev/zero bs=1M count=1024 | sshpass -p mypass ssh root@192.168.1.1 dd of=/mnt/HD5/tmpfile conv=fsync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 76.9132 s, 14.0 MB/s

scp tmpfile1 root@192.168.1.1:/mnt/HD5
tmpfile1                                                                               100% 1024MB  12.3MB/s   01:23

Very slow I think. I tried to investigate. Router as server, PC as client:

 iperf3 -c 192.168.1.1 -O 2 -t 25
...
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-25.00  sec  2.73 GBytes   937 Mbits/sec  1233             sender
[  5]   0.00-25.04  sec  2.73 GBytes   937 Mbits/sec                  receiver

Can't be better
Then
Enclosure connected to PC motherboard USB3

Write
 dd if=/dev/zero of=/run/media/jp/HD5/tmpfile bs=1M count=1024 conv=fdatasync
Clean cache
sudo /sbin/sysctl -w vm.drop_caches=3
vm.drop_caches = 3
Read
sync; dd if=/run/media/jp/HD5/tmpfile of=/dev/null bs=1M count=1024; sync

Write 1073741824 bytes = 130 to 134 MB/s => 1040 to 1072 Mbits/s
Read 1073741824 bytes = 155 to 157 MB/s => 1240 to 1256 Mbits/s
Excellent!
Then
Enclusure-HDD connected to router
SSH router (wired) then

Write
 time dd if=/dev/zero of=/mnt/HD5/tmpfile bs=1M count=1024 conv=fsync
Clean cache
sudo /sbin/sysctl -w vm.drop_caches=3
vm.drop_caches = 3
Read
time dd if=/mnt/HD5/tmpfile of=/dev/null bs=1M count=1024

Write = 145 to 147 MB/s
Read = 156 MB/s
Good!

I understand that the bottleneck is in the router between Ethernet and USB.
Questions:
Do you agree with the measurement?
If yes then What I could do to go further and fix?
If not then Let me know your code to find the issue.
Thank you for your help

anon50098793 · June 24, 2020, 8:34pm

you have an usb3 ssd?
your linked benchmarks copied with what protocol? what filesystem?

Toutatis · June 24, 2020, 10:04pm

you have an usb3 ssd?

Western Digital WDC WD40 EFRX-68WT0N0
Enclosure asmedia Product: 1153 USB3
Filesystem: ext4

your linked benchmarks copied with what protocol?

I don't understand the question. Do you mean network protocol? When HDD connected to the router, dd+ssh, scp or ssh, see in the post.

anon50098793 · June 24, 2020, 10:08pm

???

yes. what protocol / filesystem did your claimed benchmark use?

no

journal? atime?

Toutatis · June 24, 2020, 10:29pm

When HDD is connected to router and I ssh router, I run

and get

I can say the HDD is able to write 145 MB/s even it is not a SDD. This command is from the router (if=/dev/zero) to the router USB (/mnt/HD5/tmpfile).
This is not the bottleneck.
In the other side, when I iperf the router (server) from the PC (client) I get

This is not the bottleneck.
Why when I dd from the PC to the router USB-HDD I get only

BTW, I get around 13MB/s with scp from PC to router.

anon50098793 · June 25, 2020, 1:16am

309Mb/s = 37MB/s + rotational + crypto + journal + atime + core competition = ?

where does it say 60MB/s?

wind · June 25, 2020, 3:18am

Your measurements are correct. The dropbear (the ssh-server on the router) is the bottleneck, because you're piping the data through it. The strangest part is that it shows low CPU usage during transfers, yet its performance is terrible. I've been surprised by this behavior as well, because normally, a process would struggle on CPU resources or on network capacity, but dropbear is just one lazy program that does not care how much free resources the machine can allocate for it.

I have not checked how to replace dropbear with openssh-server, since I use SMB and NFS servers on the router to share the USB drive connected to it. Give that a try.

To prove that dropbear is the problem - run the following test:

from the router, scp something to/from your PC. You will notice small CPU usage and the average throughput of 14MB/s (as you listed above). Make sure that "ls -l /usr/bin/scp" points to /usr/sbin/dropbear
now, install OpenSSH client through opkg install openssh-client and run the same scp test on the router. The CPU will now be the bottleneck for the transfer. Let us know how fast the throughput is.

Toutatis · June 25, 2020, 1:07pm

Thanks a lot wind, you made my day

I checked where scp point in the router. You are right:
"ls -l /usr/bin/scp" points to /usr/sbin/dropbear

From PC, ssh router. From this connection scp PC with (write in USB-HDD command)

time sshpass -p mypass scp root@loja:/home/jp/Videos/tmpfile1 /mnt/HD5/tmpfile1; sync

TOP information CPU %:
......Mini ...Maxi
usr 36......50
sys 4........7
nic..0
idle 44......46
io ...0........8
/usr/bin/dbclient -lroot ....45......50
scp root@ ....1.....2
Time: around 72 s => 14-15 MB/s

Install openssh-client same write command
"ls -l /usr/bin/scp" points to /usr/bin/openssh-scp
TOP information CPU %:
......Mini....Maxi
usr...31....37
sys..10....16
nic...0
idle..38.... 48
io......1..... 8
/usr/bin/ssh -x -oForwardAgent=no -oPermitLocalCommand=no 50
scp root@...... 7

Time (9 try):
Mini 21.37 s => 50.2 MB/s
Maxi 22.71 s => 47.3 MB/s
Median 22.35 s => 48 MB/s

Read command and clean cache before each

sudo /sbin/sysctl -w vm.drop_caches=3

time sshpass -p mypass scp /mnt/HD5/tmpfile1 root@loja:/home/jp/Videos/tmpfileout; sync

Time (5 try):
Mini 22.39 s => 47.9 MB/s
Maxi 23.26 s => 46.1 MB/s
Median 22.77 s => 47.1 MB/s

After that I installed openssh-server. Unfortunately when I run the command from my PC to write the USB-HDD on the router it is still slow and I see dropbear running in top.
Questions:
How to replace permanently dropbear by openssh-client and server?
Can I just uninstall dropbear?
What speed do you get with SMB and NFS. Did you try SFTP (I use it in caja/nautilus)?

wind · June 25, 2020, 1:55pm

It should be possible, but not as simple as just installing openssh-server. I have not researched the topic. Try to find the way it can be done and let me know, since I'm wondering that myself.
Word of caution: be very careful when removing dropbear, since incorrect changes can leave you without ssh access to the router. You maybe able to restore it through Luci, but will most likely require you to re-flash the router through tftp. Take backups!

SMB throughout is 85MB/s on average, but only because the storage is encrypted with Crypto, thus it uses a lot of CPU. Without encryption I'd expect the throughout to be bound by the speed of the network, which will be in the 110MB/s range.

I have not tried SFTP.

Toutatis · June 25, 2020, 2:02pm

Unfortunately I am not enough skilled to investigate and change the router packages. I'll try to find documents and testimonies.

I forgot to ask, what router and disk you use, you didn't mention it.

wind · June 25, 2020, 2:40pm

A quick search found this page on the old wiki : https://oldwiki.archive.openwrt.org/inbox/replacingdropbearbyopensshserver
It's from 2016, but it looks fine and the only thing that is missing is opening port 2222 on the firewall temporarily to preserve access. Give it a try.

I'm using WRT3200ACM with WD My Book USB-HDD attached to it. Just looked at the benchmarks and it seems pretty slow in comparison to others (https://usb.userbenchmark.com/SpeedTest/2929/WD-My-Book-1230). So, it's probably not the cryptography layer that's slowing it down.

Toutatis · June 25, 2020, 5:53pm

I started to replace Openssh as the link provided.
Now I am stuck with error when I try to connect with ssh at step

Reconnect to your router via SSH on port 22.

I don't have the right password, however I can connect with port 2222 (dropbear)
I still have access to route via LuCi/html.
Looks like the password is not the same at port 22 vs port 2222
Any idea?

Toutatis · June 25, 2020, 6:04pm

In the following I always enter the same working password

ssh -p 2222 root@192.168.1.1
root@192.168.1.1's password:

Success to enter in router then

passwod
-ash: passwod: not found
~# passwd
Changing password for root
New password: 
Retype password: 
passwd: password for root changed by root
exit

Connection again to check the new password = OK
Then

ssh -p 22 root@192.168.1.1
root@192.168.1.1's password: 
Permission denied, please try again.
root@192.168.1.1's password:

I enter the same password but refused. Any idea?

Edit:
Maybe there is an issue with the sshd_config file

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

As the line is

#PermitRootLogin prohibit-password

instead of

#PermitRootLogin without-password

in the howto, is the line

PermitRootLogin yes

right?

wind · June 25, 2020, 6:56pm

Seems like you're not alone who's getting login denied. Check this post: https://www.reddit.com/r/openwrt/comments/daescl/openssh_does_not_work/

Edit: The PermitRootLogin=yes must be set in sshd_config.
I was referencing old man page. The current one says that "prohibit-password" is the default value (https://man.openbsd.org/sshd_config).

If that does not help, post the output of the following command to show your configuration:
grep -v '^#' /etc/ssh/sshd_config | grep -v '^$'

wind · June 25, 2020, 6:59pm

Whenever you change sshd_config file - don't forget to restart the SSHD with command /etc/init.d/sshd restart

Toutatis · June 25, 2020, 10:13pm

Good catch
And now from PC to router (write) I get

 dd if=/dev/zero bs=1M count=1024 | sshpass -p Shogue2019 ssh root@192.168.1.1 dd of=/mnt/HD5/tmpfile conv=fsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 21.7422 s, 49.4 MB/s

Min: 48.2 MB/s
Max: 49.4 MB/s
A lot better
WRT3200ACM and WRT32X have the same hardware, your WD USB-HDD is similar to my WD40.
I'll investigate a bit further to see why speed is not close to your 85 MB/s

eduperez · June 25, 2020, 11:09pm

Aren't we focusing too much on SSH? If the final goal is to use the router as a NAS, shouldn't these test be made with SAMBA, NFS, or whatever NAS software is going to be used?

The tests used here are good as tests, but I feel there is too much effort being put into optimizing one specific protocol that is not going to be used later.

wind · June 26, 2020, 2:18am

That's great @Toutatis !

That's because SMB and NFS are not using any cryptographic protection by default. Even authentication protection needs to be explicitly configured, which is the only thing that I did locally. So, the 85MB/s speed I get is through unencrypted traffic on my lan.

Now, since you had the experience of replacing dropbear with openssh-server on an OpenWRT 19.07, would you mind helping to port and correct the old wiki-page to the new wiki, please? It would be a great help to anyone who wants to use OpenSSH and I'm sure many people would appreciate your contribution. I can help too, if you need anything.

I'm sure you will keep experimenting and eventually install SMB/NFS to test the limits of your hardware. When you get those numbers - don't be shy and post back your numbers. I'd like to see what you can acomplish.

@eduperez, the point of this thread was to figure out where the performance bottleneck was for a specific test that OP was running. I'm sure s/he will eventually switch to the "proper" tools, since right now the throughput with OpenSSH is less than half of what gigabit link allows to push through.

Toutatis · June 27, 2020, 5:25pm

I'd not say that as I could confirm what wind said about sluggish Dropbear.

As I have only Linux based boxes I'll not try SAMBA and let one try what is suitable for his box unless another protocol is faster than NFS or SFTP.

I never did that. In fact the howto is good, the only issue I had was when I tried to connect to OpenSSH. I don't remember exactly but I has issue as well with the know_hosts file. Moreover, as I didn't remember to restart sshd, it failed. This is only because I am inexperienced.

Now I'll try NFS. This is the first time I use it and just finish to test with another Opensuse box. Now I try on OpenWrt.
Thanks. See you soon

Toutatis · June 27, 2020, 6:23pm

I started the install and have already some questions:
I found the howto here

Firewall; Should I change something as I only use NFS in the LAN, 1 network?
LuCi install nfs-kernel-server return error

exportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/mnt".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: /mnt does not support NFS export

Is it important?

I modified /etc/exports as written, copy/past

. /etc/init.d/portmap start
. /etc/init.d/portmap enable
. /etc/init.d/nfsd start
. /etc/init.d/nfsd enable

I hope this is right

netstat -a return

...
tcp        0      0 0.0.0.0:32777           0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:32778           0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:32780           0.0.0.0:*               LISTEN      
...
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:nfs             0.0.0.0:*               LISTEN      
...
tcp        0      0 :::32777                :::*                    LISTEN      
tcp        0      0 :::32778                :::*                    LISTEN      
tcp        0      0 :::32780                :::*                    LISTEN      
...
tcp        0      0 :::ssh                  :::*                    LISTEN      
tcp        0      0 :::nfs                  :::*                    LISTEN      
...
udp        0      0 :::32777                :::*                                
udp        0      0 :::32778                :::*                                
udp        0      0 :::32780                :::*

Should I change something? How?
Any details on install welcome

Edit:
My /etc/exports

cat /etc/exports
/mnt	*(ro,all_squash,insecure,sync)
. /etc/init.d/portmap start
. /etc/init.d/portmap enable
. /etc/init.d/nfsd start
. /etc/init.d/nfsd enable
/mnt/HD5	192.168.1.0/255.255.255.0(rw,sync,no_subtree_check)

I run exportfs -ar

exportfs -ar
exportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/mnt".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: No options for . /etc/init.d/portmap: suggest /etc/init.d/portmap(sync) to avoid warning
exportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export "/etc/init.d/portmap:.".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: Invalid IP address /etc/init.d/portmap
exportfs: Invalid IP address /etc/init.d/portmap
exportfs: No options for . start: suggest start(sync) to avoid warning
exportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export "start:.".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: Failed to resolve start
exportfs: Failed to resolve start
exportfs: No options for . /etc/init.d/portmap: suggest /etc/init.d/portmap(sync) to avoid warning
exportfs: /etc/exports [3]: Neither 'subtree_check' or 'no_subtree_check' specified for export "/etc/init.d/portmap:.".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: Invalid IP address /etc/init.d/portmap
exportfs: Invalid IP address /etc/init.d/portmap
exportfs: No options for . enable: suggest enable(sync) to avoid warning
exportfs: /etc/exports [3]: Neither 'subtree_check' or 'no_subtree_check' specified for export "enable:.".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: Failed to resolve enable
exportfs: Failed to resolve enable
exportfs: No options for . /etc/init.d/nfsd: suggest /etc/init.d/nfsd(sync) to avoid warning
exportfs: /etc/exports [4]: Neither 'subtree_check' or 'no_subtree_check' specified for export "/etc/init.d/nfsd:.".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: Invalid IP address /etc/init.d/nfsd
exportfs: Invalid IP address /etc/init.d/nfsd
exportfs: No options for . start: suggest start(sync) to avoid warning
exportfs: /etc/exports [4]: Neither 'subtree_check' or 'no_subtree_check' specified for export "start:.".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: Failed to resolve start
exportfs: Failed to resolve start
exportfs: No options for . /etc/init.d/nfsd: suggest /etc/init.d/nfsd(sync) to avoid warning
exportfs: /etc/exports [5]: Neither 'subtree_check' or 'no_subtree_check' specified for export "/etc/init.d/nfsd:.".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: Invalid IP address /etc/init.d/nfsd
exportfs: Invalid IP address /etc/init.d/nfsd
exportfs: No options for . enable: suggest enable(sync) to avoid warning
exportfs: /etc/exports [5]: Neither 'subtree_check' or 'no_subtree_check' specified for export "enable:.".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: Failed to resolve enable
exportfs: Failed to resolve enable
exportfs: /mnt does not support NFS export

Lot of errors. How to fix?