Feedbacks on Gnubee Personal Cloud device performances

Hey here,

I'm a moderately happy owner of GnuBee PC1. Moderately happy because of the overall performance of this device. I run some tests, but I would be curious if you have the same performances on your device. If not, I would be curious how performance could be increased, especially on the disk IO performances. I updated the wiki page with all the result and the test protocols.

Do you have better setup ? My whole point would be to have data redundancy (mdadm or RAID1) on my disks, but I'm almost close to give up.

Cheers

mt7621a is not a fast SOC, while it has very well optimized networking drivers, it isn't fast enough for satisfactory NAS usage (even less with multiple disks, SATA multiplexers and all those shenanigans) - and the overclocking attempts by its manufacturer weren't very fruitful either.

1 Like

What are you intending to do with it?

This device is intended to get a massive amount of slow storage on the network using the least amount of power in a free (fully open) device, and that is its only specialty. The only task suitable for this workload is an always-online backup storage service. Though it surely can do other tasks, any other task assigned to this device will be comparatively poorer than other systems.

For this purpose, these are excellent if you have a lot of old hard drives and want to make them useful on the network without using much power.

EDIT:
I use a GnuBee 2 with 6x 2TB drives (older drives from my primary NAS) with a 60W PSU and configured all drives for staggered spinup. This uses max ~57W while booting, but ~17W idle and ~35W rebuilding a RAID5 (all disks in use - 5 read, 1 write). A 2TB disk rebuild takes more than 25hrs, which is ~22MB/s and terribly slow even for these HDs. In my primary server this 2TB disk rebuild took less than 4hrs, but at an operating cost of ~400W.

1 Like

Thanks for your answers,

Well, I was considering it as slow storage as well, but I hoped to consider it reliable as well. Without raid or anything like this I would say I'm around 40mb/s, which is slow but acceptable. With raid, I was around 11~12/mb/s. 22Mb/s is still twice faster from what I got tho, either with mdam or LVM mirroring, but going below those speed does not allows you to browse correctly the hosted files (i mean pictures at least) via nfs/smb, so I had to giveup.

I guess by considering this device as slow backup solution is fine, but not really more, as you stated. Thank you very much both of you for your feedbacks, they are useful, and confirm me what I though on this device :slight_smile:

1 Like

mt7621a is still a rather amazing (entry level) SOC for a router or AP, and it's heavily optimized for these tasks, but it's simply the wrong SOC for a NAS/ file server or anything that needs computing power.

2 Likes

As this is the OpenWRT forum, I'm assuming you're using OpenWRT on it? I never got RAID5 to work quite right with OpenWRT, so my GnuBee 2 is running debian from a 4GB SD card, where it runs the 6x 2TB drives in a RAID6 using mdadm. So my configuration might not be a direct comparison and your speed might be affected by the different OS or raid config?

2024-01-13 18:33:13 root@gnubee:~# lsblk -o +label,fstype,uuid
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS LABEL         FSTYPE            UUID
sda           8:0    0   1.8T  0 disk              gnubee:gnubee linux_raid_member 40db63a3-59b9-e5ba-ea18-2d772a8ca66b
└─md127       9:127  0   7.3T  0 raid6 /home       GNUBEE-DATA   btrfs             9cadcce7-973a-4453-a1d1-190e5702dd6f
sdb           8:16   0   1.8T  0 disk              gnubee:gnubee linux_raid_member 40db63a3-59b9-e5ba-ea18-2d772a8ca66b
└─md127       9:127  0   7.3T  0 raid6 /home       GNUBEE-DATA   btrfs             9cadcce7-973a-4453-a1d1-190e5702dd6f
sdc           8:32   0   1.8T  0 disk              gnubee:gnubee linux_raid_member 40db63a3-59b9-e5ba-ea18-2d772a8ca66b
└─md127       9:127  0   7.3T  0 raid6 /home       GNUBEE-DATA   btrfs             9cadcce7-973a-4453-a1d1-190e5702dd6f
sdd           8:48   0   1.8T  0 disk              gnubee:gnubee linux_raid_member 40db63a3-59b9-e5ba-ea18-2d772a8ca66b
└─md127       9:127  0   7.3T  0 raid6 /home       GNUBEE-DATA   btrfs             9cadcce7-973a-4453-a1d1-190e5702dd6f
sde           8:64   0   1.8T  0 disk              gnubee:gnubee linux_raid_member 40db63a3-59b9-e5ba-ea18-2d772a8ca66b
└─md127       9:127  0   7.3T  0 raid6 /home       GNUBEE-DATA   btrfs             9cadcce7-973a-4453-a1d1-190e5702dd6f
sdf           8:80   0   1.8T  0 disk              gnubee:gnubee linux_raid_member 40db63a3-59b9-e5ba-ea18-2d772a8ca66b
└─md127       9:127  0   7.3T  0 raid6 /home       GNUBEE-DATA   btrfs             9cadcce7-973a-4453-a1d1-190e5702dd6f
mtdblock0    31:0    0   192K  1 disk                                              
mtdblock1    31:1    0    64K  1 disk                                              
mtdblock2    31:2    0    64K  1 disk                                              
mtdblock3    31:3    0  31.7M  0 disk                                              
mmcblk0     179:0    0   3.7G  0 disk                                              
└─mmcblk0p1 179:1    0   3.7G  0 part  /           GNUBEE-ROOT   ext4              56c3f4a0-9914-4a9c-a961-1ea373efcfdb
zram0       251:0    0 992.1M  0 disk  [SWAP] 
2024-01-13 18:33:17 root@gnubee:~# cat /proc/mdstat 
Personalities : [raid1] [raid6] [raid5] [raid4] 
md127 : active raid6 sde[4] sdd[3] sdc[2] sdf[5] sdb[1] sda[0]
      7813529600 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU]
      bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>
2024-01-13 18:33:24 root@gnubee:~# 
1 Like

I did a few benchmarks after installing OpenWrt:
SNAPSHOT r27119-c609c3bd6e

Raw networking performance: ~750 Mbps
iperf3 from x86_64 desktop to server on gnubee

SATA disk benchmarks (a single device at a time)

TEST_DIR=/mnt/tmp
SIZE=1G
IODEPTH=16
RUNTIME=30s

PARAMS="--directory=$TEST_DIR --size=$SIZE --iodepth=$IODEPTH --time_based --runtime=$RUNTIME --ramp_time=2s --direct=1 --verify=0 --ioengine=libaio --group_reporting=1"

{
  sync ; sleep 10 ; fio $PARAMS --name=write_bw --bs=1M --rw=write     --numjobs=1 ; echo ;
  sync ; sleep 10 ; fio $PARAMS --name=write_io --bs=4K --rw=randwrite             ; echo ;
  sync ; sleep 10 ; fio $PARAMS --name=read_bw  --bs=1M --rw=read      --numjobs=1 ; echo ;
  sync ; sleep 10 ; fio $PARAMS --name=read_io  --bs=4K --rw=randread              ; echo ;
} > bench-fio.txt

Results:

Seagate green 1TB (spinning) - EXT4

  write_bw: IOPS=75, BW=76.3MiB/s
    cpu: usr=4.53%, sys=4.99%, ctx=2224
    drive util=99.05%
  
  write_io: IOPS=833, BW=3337KiB/s
    cpu: usr=8.71%, sys=26.15%, ctx=3982
    drive util=89.63%
  
  read_bw: IOPS=79, BW=79.7MiB/s
    cpu: usr=0.45%, sys=6.48%, ctx=2404
    drive util=99.96%
  
  read_io: IOPS=148, BW=597KiB/s
    cpu: usr=1.45%, sys=1.81%, ctx=4512
    drive util=99.94%

Crucial MX500 (SSD) - EXT4

  write_bw: IOPS=158, BW=159MiB/s
    cpu: usr=9.33%, sys=10.60%, ctx=4688
    drive util=83.72%
  
  write_io: IOPS=2105, BW=8422KiB/s
    cpu: usr=26.14%, sys=61.66%, ctx=59665
    drive util=23.39%
  
  read_bw: IOPS=153, BW=154MiB/s
    cpu: usr=1.53%, sys=11.78%, ctx=4634
    drive util=94.28%
  
  read_io: IOPS=6550, BW=25.6MiB/s
    cpu: usr=44.41%, sys=55.54%, ctx=311
    drive util=90.73%

Crucial BX120 (SSD) - XFS

  write_bw: IOPS=157, BW=158MiB/s
    cpu: usr=10.13%, sys=10.60%, ctx=4754
    drive util=83.87%
  
  write_io: IOPS=2184, BW=8742KiB/s
    cpu: usr=32.56%, sys=29.77%, ctx=65382
    drive util=24.53%
  
  read_bw: IOPS=153, BW=154MiB/s
    cpu: usr=1.69%, sys=12.35%, ctx=4645
    drive util=93.98%
  
  read_io: IOPS=6921, BW=27.0MiB/s
    cpu: usr=44.86%, sys=55.12%, ctx=295
    drive util=87.53%
1 Like

@mrjk I'm not seeing as bad performance as you. Maybe you can retry with a recent snapshot. And if it's better, update the wiki page to reflect that.

EDIT: I'll try to do some multiple disk parallel benchs too.

Were you able to get RAID5 working on OpenWrt? Though flashing OpenWrt isn't too tedious, it will still take some time to test out setting up RAID5 with OpenWrt.

No, not yet.

I'm in the process of trying to build an openwrt custom image first (to get back 10% of that scarce 512MB RAM, see here).

I'm willing to test RAID 0 & 1, maybe RAID-5 afterwards, for curiosity's sake.

I remember having done some benchmarking in the past, on the previous OS, and the outcome was that the unit will start saturating something (bus, RAM BW, ...) before reaching the 3 disks RAID-5 lower bound. So that was not really practical to use this level, especially if you have to add networking to the stack...

But this was on really outdated kernel & userspace, so we'll see...

I'm already grateful for having a supported OS.
Thanks (again), OpenWRT team !

1 Like

Did you find a way to get CONFIG_HIGHMEM=y into the Gnubee default builds without affecting all the other boards?

I was originally using Neil Brown's 6.1.1 build for debian but I really want zram swap, so I've been using Brett Neumeier's 6.5.5 build as I've been unable to successfully build the kernel for my debian install.

No, see this reply in the other thread.

I don't think it's possible without creating an entirely separate target just for the gnubees.

And this probably won't be accepted upstream.

I've given up on the 64MB RAM salvage, as it does not look like a good
idea. Details in the above linked thread.

But I've done some MD benchs. All MD tests are done with 3 almost
identical Crucial BX500 SSDs, XFS on MD RAID. Same bench as above.

Baseline, single drive, no MD RAID, XFS:

    xfs_write_bw: IOPS=156, BW=157MiB/s
    xfs_write_io: IOPS=2170, BW=8683KiB/s
    xfs_read_bw:  IOPS=153, BW=154MiB/s
    xfs_read_io:  IOPS=6593, BW=25.8MiB/s

RAID 0 with 2 devices

    xfs_write_bw: IOPS=253, BW=254MiB/s
    xfs_write_io: IOPS=1986, BW=7948KiB/s
    xfs_read_bw:  IOPS=305, BW=306MiB/s
    xfs_read_io:  IOPS=6284, BW=24.5MiB/s

RAID 0 with 3 devices

    xfs_write_bw: IOPS=415, BW=416MiB/s
    xfs_write_io: IOPS=1949, BW=7801KiB/s
    xfs_read_bw:  IOPS=461, BW=462MiB/s
    xfs_read_io:  IOPS=6189, BW=24.2MiB/s

RAID 5 with 3 devices

    xfs_write_bw: IOPS=49, BW=50.1MiB/s
    xfs_write_io: IOPS=1910, BW=7643KiB/s
    xfs_read_bw:  IOPS=447, BW=448MiB/s
    xfs_read_io:  IOPS=5978, BW=23.4MiB/s

The RAID 5 synchro started at 26MB/s and went to 44MB/s near the end.

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid5 sdd1[3] sdc1[1] sdb1[0]
      20953088 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
      [>....................]  recovery =  2.8% (296892/10476544) finish=6.2min speed=26990K/sec
      [====>................]  recovery = 21.7% (2276092/10476544) finish=3.6min speed=37375K/sec
      [==========>..........]  recovery = 50.7% (5317372/10476544) finish=2.0min speed=41103K/sec
      [===============>.....]  recovery = 78.0% (8181724/10476544) finish=0.9min speed=42301K/sec
      [===================>.]  recovery = 98.2% (10298184/10476544) finish=0.0min speed=44049K/sec
      20953088 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
1 Like

Whow, from your benchmarks, your worst score was my best score! On my side, I bought a real NAS instead, but still considering this hardware as slow backup. When I'll be more available, I'll rerun some tests to see if I see improvement.

@vincele Thanks for your contribution on personal_cloud_one. Super cool the TFTP way :stuck_out_tongue:

Yeah, the benchmarks I did in the past were not so great, and a bit disappointing. But that was with an antiquated kernel.

These look good, if you can avoid RAID5, I think I'll try RAID1, which should look similar to RAID0.

And yes TFTP indeed is very useful to avoid flashing...

I'd like to hack on u-boot, but am afraid of bricking the board.

RAID 1 with 3 devices

ext4_write_bw: IOPS=151, BW=152MiB/s
ext4_write_io: IOPS=1510, BW=6046KiB/s
ext4_read_bw:  IOPS=176, BW=177MiB/s
ext4_read_io:  IOPS=5997, BW=23.4MiB/s

RAID 1 resync speed was that of one disk worth of MiB/s...

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10]
md0 : active raid1 sdd1[2] sdc1[1] sdb1[0]
      10476544 blocks super 1.2 [3/3] [UUU]
      [==>..................]  resync = 10.8% (1133184/10476544) finish=0.9min speed=161883K/sec
      [====>................]  resync = 20.3% (2135424/10476544) finish=0.8min speed=164263K/sec
      [============>........]  resync = 60.2% (6315264/10476544) finish=0.4min speed=159316K/sec
      [================>....]  resync = 82.6% (8662656/10476544) finish=0.1min speed=159644K/sec
      [===================>.]  resync = 97.6% (10230720/10476544) finish=0.0min speed=159788K/sec

I'm a bit surprised by read_bw, I thought it'd be more like RAID 0, maybe I should go read something about that, my expectations may be off...

So now I'm forced to test RAID10...

EDIT:

Here are the RAID10,far2 results for 2x Crucial BX500 SSDs, array creation:

# mdadm --create /dev/md0 --level=10 --layout=f2 --raid-devices=2 /dev/sd[bd]1

RAID resync performance between 60MB/s and 90MB/s:

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10]
md0 : active raid10 sdd1[1] sdb1[0]
      10476544 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
      [=>...................]  resync =  7.2% (757504/10476544) finish=2.5min speed=63125K/sec
      [====>................]  resync = 20.6% (2166464/10476544) finish=2.0min speed=67702K/sec
      [=======>.............]  resync = 37.5% (3929088/10476544) finish=1.2min speed=85824K/sec
      [===========>.........]  resync = 55.4% (5805056/10476544) finish=0.7min speed=96196K/sec
      [==============>......]  resync = 70.6% (7404544/10476544) finish=0.5min speed=87862K/sec
      [===================>.]  resync = 96.5% (10114048/10476544) finish=0.0min speed=90960K/sec

And the benchmarking:

ext4_write_bw: IOPS=152, BW=153MiB/s
ext4_write_io: IOPS=1610, BW=6443KiB/s
ext4_read_bw:  IOPS=306, BW=307MiB/s
ext4_read_io:  IOPS=5535, BW=21.6MiB/s

@reukiodo : BTW, zram is activated:

root@OpenWrt:/# dmesg | grep zram
[   24.896637] zram: Added device: zram0
[   30.980444] zram0: detected capacity change from 0 to 442368
[   31.023738] Adding 221180k swap on /dev/zram0.  Priority:100 extents:1 across:221180k SSDsc

EDIT, to overcome the 3-post-limit:

Do you mean "SATA port multipliers" ?

I don't think there are SATA port multipl{i,ex}ers involved on this board.

There are 3x separate ASMedia 1061 chips, and those are 1 lane PCIe 2.0, dual port SATA controllers. So I think all the 6 SATA ports on the board each have their own SATA port.

They will all share the PCIe bandwidth in the end, so there still will be bottlenecks. But the board should not suffer from the usual port multipliers problems.

@slh : WDYT ?