1 (edited by XChesser 2010-12-16 22:19:43)

Topic: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

[size=15]Testing methodology.[/size]
The objective was to measure the speed of filesystems on the router with OpenWrt excluding the impact of other factors as much as possible. My TP-Link WR1043ND runs OpenWrt Backfire 10.03.1-RC4. Kernel version: 2.6.32.25-1.
As an external USB-drive I used Samsung HD204UI connected to the router via SATA-USB controller based on Prolific Technology PL3507 chip. About 15 partitions with different FS and 1GB in size were created with the default settings. For comparison I also made a read/write test with unformatted partition - this rate can be considered as the limit of the hardware, because not any FS-driver is involved in this case.

As benchmark tool full version of dd was used. Thus the resources of the router were not spent on data transmission over the network and the corresponding server. To minimize source file read/write expenses on the router side, /dev/zero and /dev/null weer used, respectively. Multiple tests with a file of 800 MB on Ext2 gave 3% spread only so this file size was used in all tests subsequently. All filesystems were tested with 2 values of the read/write block size: 4 KB and 2 MB.
For the purity of the experiment unnecessary services and servers were unloaded for benchmark time; no router accesses except for SSH took place. In this state, the average CPU Idle was 99.7%.

[size=15]Results.[/size]
The results are shown in the graphs. In some cases, the mount options mentioned.
http://img831.imageshack.us/img831/9743/32801851.png
In all tests the CPU Idle was 0% (measurements were performed on another iteration, as it might influence the results).

Reiser4, UFS, JFS, ZFS are not supported by OpenWrt, so not tested.

Hope this test will be useful for someone.

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

Well, comments are appreciated. smile

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

thanks for you effort and it confirms again that ntfs should be avoided at any costs smile.

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

Thanks for the effort! About time we take performance more seriously! Can you please post the mount options?

Next thing would be network transfer over ftp, nfs, samba and.... scp.

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

Orca,
Ext3, ordered, barrier: -t ext3 -o data=ordered,barrier=1
Ext4, ordered, barrier: -t ext4 -o data=ordered,barrier=1
Ext4, writeback, barrier: -t ext4 -o data=writeback,barrier=1
Ext4, ordered, no barrier: -t ext4 -o data=ordered,barrier=0
Ext4, journal, barrier: -t ext4 -o data=journal,barrier=1
In other cases default options were used (just 'mount device mountpoint').

Yeah, I really was going to test network file servers and even made some tests for performance with NFS-kernel, unfs3 and Samba3 with different options but I faced the problem that I get much worse speed than some other TP-Link 1043 owners. I have described it with them, tried to fix the problem with replacing a patch cord, testing with the other client hardware and with different OS (Linux/Windows), tried to tweak Ethernet card with ethtool, tested network performance with iperf and so on - no solution yet. Via NFS I see real speed at ~7MB/s that is too low - this router is capable to give twice more...

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

These systems are quite strange.  My buffalo g300nh dd tests (ext2,ext3,ext4 same mount settings) are around 7-8 MB sec write and 25 MB read with dd, yet in samba3 7 MB/sec write and 11 MB/sec read despite any changes to settings.  Same benchmarks with ftp too! 

My load according to htop during the write tests was 35% with ext2, up to 40% with ext4.  With samba or ftp, it pegs the cpu.

In the past, I've tried an Asus RT-N16 with more ram and faster CPU, using tomato and stock firmware and got slower!  So, something else is causing these limits!

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

XChesser,

Thanks a lot for your tests!

My understanding is that ReiserFS is a journaling file system, so it should be benchmarked against ext4, right?

I am surprised though about relatively low performance of XFS file system.

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

robnitro wrote:

These systems are quite strange.  My buffalo g300nh dd tests (ext2,ext3,ext4 same mount settings) are around 7-8 MB sec write and 25 MB read with dd, yet in samba3 7 MB/sec write and 11 MB/sec read despite any changes to settings.  Same benchmarks with ftp too! 

My load according to htop during the write tests was 35% with ext2, up to 40% with ext4.  With samba or ftp, it pegs the cpu.

Maximum performance for reading which I could achieve with USB flash drive connected to USB port on my WR1043ND is 12.5MB/sec.
It was with vsftpd as FTP server, and lukemftp client (OpenSUSE 11.4 Factory), and gFTP

Connection to router was 1Gigabit Ethernet cable.

CPU utilization was close to 100% during those tests.

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

vplessky, yes, Reiser is a journaling file system but you must distinguish all data journaling and metadata only journaling. Both Ext3 and Ext4 support any of these kinds of journaling (see 'data' mount option).
Reiser also supports both types of journaling so you should compare its results properly.

I am surprised though about relatively low performance of XFS file system.

XFS is known for its high CPU utilization so results are predictable for the router I think.

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

HFS+ seems to be the best.But does it run stably in Openwrt?

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

axishero wrote:

HFS+ seems to be the best.But does it run stably in Openwrt?

It should be as stable on OpenWrt as with other Linux systems.

I have never used it - recent Fedora and OpenSUSE default to ext4, you can also use xfs (in Fedora) and ReiserFS in OpenSUSE.
Give it a try, would be interesting to know results for HFS+ with FTP, NFS, Samba servers.

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

I found out why I was running a bit slow even in DD write tests.
cat /sys/block/sda/device/max_sectors
240

echo 1024 > /sys/block/sda/device/max_sectors  OR add /sys/block/sda/device/max_sectors=1024 to /etc/sysctl.conf file and then run sysctl -p or reboot.
Then running the DD tests I get 19 MB a second, similar to your results!

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

robnitro wrote:

These systems are quite strange.  My buffalo g300nh dd tests (ext2,ext3,ext4 same mount settings) are around 7-8 MB sec write and 25 MB read with dd, yet in samba3 7 MB/sec write and 11 MB/sec read despite any changes to settings.  Same benchmarks with ftp too! 

My load according to htop during the write tests was 35% with ext2, up to 40% with ext4.  With samba or ftp, it pegs the cpu.

In the past, I've tried an Asus RT-N16 with more ram and faster CPU, using tomato and stock firmware and got slower!  So, something else is causing these limits!

Why samba slow is because of this
https://forum.openwrt.org/viewtopic.php?id=28578

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

axishero wrote:

Why samba slow is because of this
https://forum.openwrt.org/viewtopic.php?id=28578

I doubt it, when CPU is idle, it is still slow.  With htop installed I can change smbd to nice 0 or even negative values and it is still slow.  Same goes for ftp transfers.  Straight DD gets 19 MB/sec writes, but ftp or samba, no matter what nice/prio seem to be limited to max 10 MB/sec on this box.  I even changed many settings on smb.conf!

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

robnitro wrote:

I found out why I was running a bit slow even in DD write tests.

I haven't done such optimizations. Probably it will increase the performance in my case too, I'll make a try. Thanks!

Straight DD gets 19 MB/sec writes

What block size do you use for write tests? By default dd sets bs=512 that is not optimal for writes. If it's your case you must have a luck with larger blocks, for example bs=4k.

but ftp or samba, no matter what nice/prio seem to be limited to max 10 MB/sec on this box.  I even changed many settings on smb.conf!

I get ~14MB/s read and 10.7MB/s write speed with FTP. It is average speed not peak. I tested 16 FTP-clients in Linux with vsftpd on my router. Speed is much dependent on client! For me FOFF appeared to be the fastest client. Filezilla, ncftp make good work too. I also tested OS influence: 4 cross-platform clients were benchmarked both in Linux and Windows 7 on the same computer. Windows gave much worse performance. Such strange results were also confirmed by two other users (on non-English forum). Active/passive FTP-mode doesn't make any difference.
With Samba I get opposite results: Windows client is faster.

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

It would be interesting to lock the transfer rate to a common level and see what the system loads are for each of the different systems.  Is it possible to limit the transfer rate??

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

This whole thread gets more and more tumultuous (UNübersichtlich). We should continue discussing here, but relocate the documentation where it belongs:
http://wiki.openwrt.org/doc/howto/performance

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

Writing a file to /dev/null is hardly a good filesystem benchmark, more like a sequential write hard disk benchmark. As the results show there is very little difference between filesystems in this type of test.

Here's an example how you benchmark filesystems:

http://www.phoronix.com/scan.php?page=article&item=ext4_btrfs_nilfs2&num=1

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

Yes, I tested just sequential read/write. I think it's the most common behavior for a HDD connected to router. Network storage is usual model of HDD use with router and in this case it's just seq operations that we see.
Another reason is lack of benchmark tools for OpenWrt. There are no packages for iozone, bonnie, dbench in repo. So dd is the only tool for OpenWrt that I see.

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

As I already stated in the wiki, the remote filesystems could be benchmarked on the client pc. If you have the time ;-)

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

What about remote filesystems I mount them and benchmark with iozone. I tried many other tools but my choice is iozone. 'bonnie++' is inconvenient as it has no autoremount option that must be used not to measure cache performance. Manual scripting is a solution of course but iozone does it for me.

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

Hi guys,

I'm having very bad results in terms of disk speed with my TP-WDR3600 with OpenWrt Attitude Adjustment 12.09-rc1:

time sh -c "dd if=/dev/zero of=ddfile bs=8k count=20000 && sync"
20000+0 records in
20000+0 records out
real    1m 0.80s
user    0m 0.44s
sys    0m 7.82s

This gives ~2.6Mb/s which is really bad! I tested the same disk on a desktop computer and I got 32.2Mb/s. It is a 300Gb sata disk which I divided in two partitions, both on ext4. The first partition (512Mb) I use with extroot to increase the system disk space. The second partition I use for downloads.

Kind regards,
Américo

Re: Filesystem performance benchmark on OpenWrt / TP-Link WR1043ND

I was using fstab to mount the disk. I discovered that if I mount manually (using the mount command) I was able to reach 30Mb/s.

Thanks anyway.

Regards,
Américo