Does OpenWrt support ZFS filesystem?

[bobafetthotmail]

@pwned

I never touched my setup in terms of reformatting with BTRFS to compare. So this is very interesting for me. Beside that mine is not so huge as yours and older.

I didn't have much choice, I started with btrfs and it was running way too bad for VMs.

But as I said already it depends all on usage. Most ppl. just want a fileserver with (maybe) a bit ISCSI. Most do not run a DB or VMWare ESXi server.

You somehow implied ZFS has worse performance than btrfs, which is not really the case.

Btrfs performance is good enough for NAS and PC use, but it's still too bad for anything requiring high performance, like VMs or databases.

As I was searching on a comparision BTRFS vs. ZFS I found only this:

https://www.diva-portal.org/smash/get/diva2:822493/FULLTEXT01.pdf

This document is very spread in WWW and has (IMO) certain issues in terms of usage scenarios. But it is telling that BTRFS is better. There is sadly nothing else available at this level.

That document is ancient, both ZFS on Linux and Btrfs were immature back then.
I suggest looking at Phoronix benchmarks https://www.phoronix.com/scan.php?page=search&q=ZFS and you will see a different picture.

the feature where you can replace your drives simply with bigger ones step by step

Meh, that's one of the things btrfs does better. You can add, remove and do whatever without any limitation, you can RAID1 mismatched drives and it will balance stuff around anyway (i.e. 2 drives 1TB and 1 drive 2TB, you have 2TB of space on btrfs)

But if you need performance there is no alternative yet. As I said I could set up a bcache on top of the btrfs array but it's annoying

[bobafetthotmail]

@dlakelan

For this kind of usage if you are running vms off a disk image you need that disk image to be set nodatacow

Nodatacow disables checksums and compression (check the manual), and since this 20-drive array is dedicated to VM virtual drives I don't see the point of using btrfs to then disable COW, checksums and compression. Why use btrfs if I disable all features I needed.

The "nodatacow" option is designed to let you use btrfs on root, so you can disable cow on some folders and not the rest of the root filesystem, so some small database used by an application does not spam the fragmentation.

[dlakelan]

@bobafetthotmail

bad for anything requiring high performance, like VMs or databases.

Both of these usages require nodatacow. Basically they are running filesystems on top of btrfs and the copy on write (COW) makes no sense for that.

[dlakelan]

@bobafetthotmail

I don't see the point of using btrfs to then disable COW, checksums and compression.

Does zfs provide checksums and compression on VM disk images? It's hard for me to imagine either of those could be performant with VM disk images. Every time your VM writes a block the entire disk image has to be rechecksummed? And recompressed?

[bobafetthotmail]

@dlakelan

Does zfs provide checksums and compression on VM disk images?

Yes

It's hard for me to imagine either of those could be performant with VM disk images. Every time your VM writes a block the entire disk image has to be rechecksummed? And recompressed?

No, both btrfs and ZFS calculate checksum and do compression at block level, not at file level.

So they will write or compress in 4kb or 64kb or 128kb chunks, and that's close enough to working only on the part that was changed.

[dlakelan]

In any case the btrfs wiki specifically mentions both databases and VM images as being appropriate for the nodatacow https://btrfs.wiki.kernel.org/index.php/SysadminGuide

Yes you will lose checksumming and compression, but you still retain the RAID redundancy and snapshotting (technically once you do a snapshot it starts to COW again)

I'm having a hard time seeing how zfs could make snapshots and still do in place checksum and compression... But whatever it's good that ZFS works for you, if people want to use btrfs with databases or VM images the recommended way is with nodatacow and to limit snapshotting because it will force COW

[pwned]

@bobafetthotmail

You somehow implied ZFS has worse performance than btrfs, which is not really the case.

Btrfs performance is good enough for NAS and PC use, but it's still too bad for anything requiring high performance, like VMs or databases.

That was not my intention. My intention was that what you have written. I see BTRFS as the main usage for an avarage user (as I wrote NAS, fileserver, whatever). ZFS is (IMO) nothing for an average user running ZoL. For ZoL you want to have a rescue image with ZoL integrated. But most distributions lack native support of ZFS. Even Ubuntu removed it afaik. You have to make your own images for emergency. Otherwise you have to use Solaris and its derivates or sth. like napp-it for your NAS.

@bobafetthotmail

That document is ancient

Y, I saw and know some phronix benchmarks. But it is to much work to aggregate them to make my own conclusions (I don't need them atm). I'm not keeping up hard with the progress on this sector.

@bobafetthotmail

Meh, that's one of the things btrfs does better.

I never have used BTRFS for this case. I rely on ZFS mainly. Thats because it was first on market long before BTRFS. But its good to know.

[bobafetthotmail]

@dlakelan

In any case the btrfs wiki specifically mentions both databases and VM images as being appropriate for the nodatacow https://btrfs.wiki.kernel.org/index.php/SysadminGuide

And I specifically mentioned that as it is, with the mentioned limitations, it is an option mostly useful for people using btrfs on a single device as root filesystem as it is more convenient to tag a subvolume (a folder) with "nodatacow" so you can keep your VMs and stuff in there while the rest of the drive is still using btrfs. This saves the major headache of repartitioning the drive to store a few basic stuff or a few test VMs and things that aren't critical.

While for large and/or dedicated arrays where you are doing ONLY ONE THING it makes little sense to not choose the best setup for it.

Yes you will lose checksumming and compression, but you still retain the RAID redundancy and snapshotting

mdadm RAID provides the same redundancy at a much better speed and snapshotting is not useful in my case.

All half-decent virtualization software (KVM aka the default Linux hypervisor, VMWare ESXi, Citrix XenServer or its free counterpart XCP-ng+XenOrchestra) are able to snapshot the VMs on their own in a very much more convenient way since ages ago.

(technically once you do a snapshot it starts to COW again)

Yes and no. It will COW only the first time you modify the data (again at the block level), then if you are modifying data that is already modified (i.e. out of the snapshot) it will not COW on that.
Again, this happens at block level, so at the 4kbyte chunks of the file, not the whole file.

I'm having a hard time seeing how zfs could make snapshots and still do in place checksum and compression...

btrfs can make snapshots and still do checksum and compression too, the general mechanism is the same, as it's supposed to be an evolution of ZFS general concept. It's just slower because it does not use GBs of RAM as cache.

if people want to use btrfs with databases or VM images the recommended way is with nodatacow and to limit snapshotting because it will force COW

People that want to use btrfs with "nodatacow" should also be informed that it disables checksumming and compression, because they should be able to make their own decision, based on their needs in the particular system.

I'm not a btrfs hater, I use it for my "cold storage drive pool" (a massive RAID1 pool of 20-ish random mismatched sata drives that btrfs can actually use fully as mentioned above) and I'm using OpenSUSE on my PC and home server/NAS, that uses btrfs by default.

I just can't use it for VM storage.

[dlakelan]

I should have mentioned that I specifically meant nodatacow on the individual files ie. The +C xattr rather than the whole volume, I basically agree with what you're saying here but wanted to mention that was available... http://infotinks.com/btrfs-disabling-cow-file-directory-nodatacow/

If someone creates a big NAS type array, and then wants to run a few VMs from one directory or something I do think this is a viable option in some use cases. If you are doing nothing but VMs it probably makes sense to use something like ext4 or xfs or ZFS as you did