I have not used F2FS before, and AFAIK some FS kmod's need direct access to drives for some low-level commands like trim/etc, and I do not know how mdadm will work in this scenario. I would like suggestions on config, and setup step ordering if there is a difference (ie create array before/after mkfs.f2fs).
I'm setting up a Netgear router (RAX120v2) for a few different roles, including as a small NAS. For various reasons, I'm using 2x USB thumbdrives for storage in a RAID1 config. The router will end up at a non-technical friend's house a few hours drive away, as an end point of a multi site homelab playground. Thus trying to do things "the right way". but on the cheap end of implementation.
AFAIK:
F2FS is theoretically the best choice for USB thumbdrives. please correct if wrong
mdadm RAID1 is the only real option given that thumbdrives WILL eventually fail. Other backups will be done, but I need drives/array to continue to function as long as possible even with one dead drive (refer to location being a few hours drive away)
My main concerns are:
Worry that mdadm might interfere with f2fs wear leveling/minimizing mechanisms (such as might be implemented via direct commands to device, I have no idea on this...)
Wear leveling, and squeezing out the maximum longevity out of the thumbdrives
Am I proceeding down a path toward Stupid-ville?
Any feedback and suggestions are appreciated, and feel free to correct any misconceptions I may have. Thanks
Hmm. I haven't used F2FS lots either. You're not going to get TRIM on a thumbdrive anyway. You're at the behest of whatever wear levelling algorithm they have on there. (F2FS assumes the FTL does wear levelling, F2FS doesn't have wear levelling built in). My understanding is that it's just coalescing writes.
I'd be worried about USB flash drives as compared to something like an nvme/sata ssd on a USB adapter, that is if it's under a heavy write load.
You mention thumbdrives. Do you know that that flash drive does wear levelling?What sort of write load are we talking about here? Do you have a known endurance?
Where I absolutely needed lots of write and "known" write endurance in a small package I went for USB->sd card adapters with dashcam high endurance sd cards. That is reasonably comparable to larger capacity usb flash drives without going "usb ssd".
Do you need error correction between the drives? I'd be worried about silent data corruption. What is being stored on the NAS?
Only writeup I've found on F2FS is more than a decade old. Does F2FS currently have usage quotas, NFS export, xattrs? I've read things about fsck behaviour being long and causing issues on boot with F2FS.
On my bigger servers when I needed to run a USB drive with high endurance I used USB->m.2 adapters with 16-32gb m.2 drives on the used market. These can end up quite expensive compared to SD cards / usb flash drives. Still cheaper than commercial USB SSD.
Personally i've tried mdadm and btrfs raid with usb flash drives. Haven't had good experiences given the failure modes of cheap USB flash drives. I don't know enough about how mdadm and btrfs figure out a bad drive if it decides to start writing/reading really slowly, or goes in read only mode? Only ever really used for long periods in bigger drives / SAS controllers etc.
Normally flash sticks do not have discard/trim support unless they contain some nvme/sata controller (read - very expensive)
xattr is supported in snapshots, but is missing for f2fs and ext4 in stable branch.
Rule of a thumb - readyboost from recognizable manufacturer means wear levelling.
You have to check with smartctl if any flash wear is reported, without that you are in the blind. With only possible test being destructive.
I,d suggest xfs (like fedora desktop) - fixed memory usage, no fsck needed, still you need to be able to predict failures and drop half raid1 when those approach.
You have a ~400 buck router, adding 5+15 bucks for M.2/NVMe<-->USB adapter and a 256 GB NVMe SSD sounds like a negligible tally in comparison.
Disclaimer: I am not a fan of overloading your border gateway with server tasks to begin with and would recommend against doing so, nor do I think that more moving parts (2 USB sticks instead of 1) are actually beneficial to a non-technical friend (more things to plug in wrong); one 'reliable' (USB-) SSD certainly trumps two crappy USB sticks (RAID or not). Keep in mind that accidental file deletion is going to be more likely than hardware failure (especially if you aim a tad higher than USB sticks or sdhc cards); offline backups will be required either way. My suggestion would be to make failure (hardware/ software) easier to recover (keep a fully configured cold spare, an old/ cheap/ slow one suffices, just enough to get back onto the web, to get the VoIP phones working again, the TV, access to search engines to research your situation - not a full-featured long term replacement, but enough to buy & setup a replacement, should the dark matter hit the fan), than wasting too much thought on preventing it (you will miss some tiny detail orthogonal to your failsafe bastion anyways).
I think I'm going to go with mdadm RAID1 with probably XFS. With 2x thumb drives at least for the "zoo" remote site. The other site I might go ahead with USB->NVMe. Don't know how these will fair, but should be 5-10x larger than expected drive use, so wear leveling can work it's best: SDCZ430-512G-GAM46
If interested in the seemingly bizarre use of thumb drives in this application, read on. Feedback welcome.
By "non-technical friend's house", I am really referring to a couple relative's households with varying degrees of technical ability, from basic TV operation skill level up to one adult male who truly believes he is an expert at everything but is far more likely to permanently brick something claiming it was a manufacturer defect than anything else. Yes, he is the most dangerous of them all, just tech-ish enough to do real damage. One household has an overpopulation of indoor animals that outnumber people and it is an open debate on who is actually in charge there. Case in point, a dog destroyed the screen of my nephews laptop, and while helping him repair what could be salvaged, we had to use 90 PSI compressed air from my shop to dislodge significant quantities of cat hair that was amassed inside his GPU heat sink, the canned air and shop vac were not sufficient for the job.
ANYway quick rundown of the "why's" of this project:
I'm trying to get niece and nephews interested in technical things that are not solely for gaming, and I've been pushing for ESP32 kinda projects. By building a site-2-site plus Avahi, (plus local webserver and whatever else I install on remote OpenWRT routers), I can do a hellova lot more remotely than otherwise. And yea, probably some LAN games.
Every household is behind CGNAT including myself, so cloud router is needed
My main project is spinning up a pfSense instance on my cheapo cloud VPS so I can remote into my home lab, but it will also be the hub for all site-2-site, but that's out of scope for this thread.
I am a big fan of Sanoid and I use ZFS for all my home lab servers, workstations, and laptop, with the exception of workstation VMs that have dedicated GPU/usb keyboard/mouse. Workstation VMs are stored on a mdadm RAID0 nvme array with 64k stripe, with ext4. I did perf comparisons a few years back and that config was the winner for qcow.
Part of the whole reason for doing this is to build my own real-world site-2-site extension of my homelab, as I have not done one before.
I have Starlink plus TMobile 5G multi-WAN, which has decent download speed, but my upload speed is just crap, so i want to make sure remote sites have some local storage for essentially file caching from my home server plus whatever else I decide to push remotely.
Clarification:
these routers will not be edge gateways, just routers (and webserver/mini NAS/etc) for split tunnel site-2-site VPN, shimmed into other relative's existing household networks.
I'm not buying a full-blown NAS for somebody else's household.
I am happy to do tech support for kids, but the adults are on their own, thus I'm not touching their network configs just shimming in from the side.
Rationale for F2FS:
I really do not have any other than it's advertized as good for flash memory, but apparently it doesn't really matter in this case. So... bye bye F2FS, and hello.... EXT4 or XFS?
Why I never use thumbdrives for anything important:
The previous motherboard I had for my promox workstation did not have power headers for SATADOM, so I "temporarily" used USB thumb drives in mirrored ZFS for hypervisor boot (along side normal disk arrays for VMs) while I ordered PSU adapters for the tiny 5V SATADOM power cables, which I forgot to order, and then forgot about the whole issue. The first thumb drive lasted almost a year, but both died in quick succesion with essentially zero notice. With backups it wasn't a big deal, got up and running again with a few hours work, but yea, will never make that mistake again.
Rationale for USB itty bitty thumb drives that I would NEVER use for NAS, EXCEPT:
With one (NOT mine) household that is more zoo than home, and another (NOT mine) household with non-technical adults thus kids are "in charge" of technical things...
I think a physiclally tiny USB thumb drive that is almost entirely housed within the USB port itself would physically outlast pretty much any other storage medium that connects via USB. Typical "wear leveling" algorithms do NOT take into consideration dogs, cats, kids, and adults of questionable judgement. Long memory sticks can easily get snapped off, and any other drive type dangling by a cable from the router is a ticking timebomb.
I have seriously considered mounting 1 or 2 NVMe 2230's inside the router's plastic enclosure and just run the cable through a hole to the jack, but current OpenWRT snapshots do not turn on the case fan and the router can get pretty hot, bad for nvme.
If my confidence changes, perhaps I might swap out the thumb drives for something more appropriate at a later time.
It is trivial to "tape" or 3DP a mini bracket to strongly discourage the removal of thumb drives from the port, but cabled or other drives are non-trivial to physically secure.
Rationale for 2x drives and RAID1:
Simply trying to get the thing to last as long as possible without having to drive several hours to fix something.
But you're looking at ~30USD for a 32G unit. CCTV/Industrial SD cards can be had cheaper but then you factor in USB adapter prices.
Having a look at the sandisk write intensive USB/SD card flash memory brochure. They do sell USB model for 16GB. SDUFDEC-016G and say up to 160TBW. That's 34USD. Brochure says smart features support but it's not written as the acronym so I don't know whether it actually means SMART data.
One other thing I just thought about cheap USB storage and USB adapters is the serial numbers can be blank and/or all the same. Which can create issues depending on how you're trying to initialise the devices in a repeatable manner.
Mm. Thanks for making it clear regarding the cost comparison. I only hinted at that in my original reply.
Not saying it's a good choice in terms of cost benefit. More regarding the OP's statement on form factor, that we do have USB flash drives that have the features one would want. At least according to the datasheet.
Still yeah I'm cautious regarding the cheapest m.2 adapters (and the USB->SD card adapters) as they can be set to identical serial numbers.
