tldr: I’m going to set up raid z2 with 4x8TB hard drives. I’ll have photos, documents (text, pdf, etc.), movies/tv shows, and music on the pool. Are the below commands good enough? Anything extra you think I should add?

sudo zpool create mypool raidz2 -o ashift=12 /dev/disk/by-id/12345 ...

zfs set compression=lz4 mypool #maybe zstd?
zpool set autoexpand=on mypool
zpool set autoreplace=on mypool #I might keep this off. I can see myself forgetting in the future
zpool set listsnapshots=on mypool

With ai raising hard drive prices, I over spent on 3x10TB drives in order to reorganize my current pool and have 3 hard drives sitting on a shelf in the event of a failure. My current pool was built over time but it currently consists of 4x8TB drives. They are a mirrored stripe so a usable 16TB. If I understand it correctly, I can lose 1 drive for sure without losing data and maybe a second drive depending on which drive fails. Because of that, I want to move to raid z2 to ensure I can lose 2 drives without data loss. I’m going to move data from my 4x8TB drives, to the 3x10TB, reconfigure the 4x8TB, and move everything back. I run Immich, plex/jellyfin, and navidrome off the pool. All other documents are basically there for long term storage just in case. What options should I use for raid z2 when setting it up?

I know I can look this stuff up. I have been and continue to do so, I was just hoping for some advise from people that are more knowledgeable about this than me. The move from the 4x8TB drives to the 3x10TB is going to take ~3 days so I really don’t want to mess this up and have to start over 😅

Edit:

After looking up each property, this is the command I will probably end up using to create the raid z2 pool, thanks Avid Amoeba:

sudo zpool create
-o ashift=12 -o acltype=posixacl -o xattr=sa
-o compression=lz4 -o dnodesize=auto -o relatime=on
-o normalization=formD
raidz2
mypool
/dev/disk/by-id/12345 …

Edit2:

Above command didn’t work on my machine. The order and uppercase “O” matters. Had to do this:

sudo zpool create \
  mypool \
  raidz2 \
  -o ashift=12 -O compression=lz4 \
  -O normalization=formD -O acltype=posixacl \
  -O xattr=sa -O dnodesize=auto \
  -O relatime=on \
  /dev/disk/by-id/12345 ...

Edit3:

And finally, after all this, I set up my tmp pool of 3x10TB disks as a raid z2 instead of raid z1. Spent a day and a half transferring before I finally saw my mistake after running out of space 🫠

  • greyfox@lemmy.worldEnglish
    1·
    7 hours ago

    Well I am not claiming to be a ZFS expert, I have been using it since 2008ish both personally and professionally. So I am fairly certain what I have said here is correct semantics aside.

    Both lz4 and zstd have almost no performance impact on modern hardware.

    So “almost” is zero in your mind? Why waste the CPU cycles on compressing data that is already compressed? I recognize that you might not care, but I sure do. And I wouldn’t say it would be wrong to think that way.

    compression acts on blocks in ZFS, therefore it is enabled at the pool level

    This is incorrect. You can zfs set compression=lz4 dataset (or off) on a per dataset basis. You can see your compression efficiency per dataset by running zfs get compressratio dataset, if your blocks were written to a dataset with compress=off you will see no compression for that dataset. You can absolutely mix compressed and uncompressed datasets in the same pool.

    OP added a -O option to set compression when he created the pool but that is not a pool level setting. If you look at the documentation for zpool-create you will see that -O are just properties passed to the root dataset verses -o options which are actual pool level parameters.

    You might be confusing compression and deduplication. Deduplication is more pool wide.

    ZFS does indeed need to allocate some space at the front and end of a pool for slop, metaslab, and metadata. I think you are confusing filesystem and datasets.

    Well yes and no here. You are right I should have been calling them datasets. Datasets are a generic term, and there are different dataset types, like file systems, volumes and snapshots.

    So yeah I maybe should have been more generic and called them datasets but unless OP is using block volumes we are probably talking about ZFS file systems here. Go to say the zfsprops man page and you will see file system mentioned about 60 times when discussing properties that can be set for file type datasets.

    I’m not sure what you’re trying to say about NFS and ZFS, here but this is completely false, even if you mean datasets.

    It sounds like you are unaware of the native NFS/SMB integrations that ZFS has.

    It is totally optional but instead of using your normal /etc/exports to set NFS settings ZFS can dynamically load export settings when your dataset is mounted.

    This is done with the sharenfs parameter zfs set sharenfs=<export options> dataset. Doing this means you keep your export settings with your pool instead of the system it is mounted on, that way if you say replicate your pool to another system those export settings automatically come with it.

    There are also sharesmb options for samba.

    My point was then that you should lay out your dataset hierarchy based on your expected permissions for NFS/SMB. You could certainly skip all of this and handle these exports manually yourself in which case you wouldn’t have to worry about separate filesystems and this point is moot.

    My post was less about compression and more about saying that you should consider splitting your datasets based on what is in them because the more separate they are the more control you have. You gain a lot of control, and lose very little since it all comes from the same pool.

    Some of the reasons I have these options are less important than they were a decade ago. i.e. doing a 20tb ZFS send before resuming a send was possible sucked. Any little problem and you have to start over. Having more smaller filesystems meant smaller sends. And yeah I was using ZFS before lz4 was even an option and CPU was more precious back then, but I don’t see any reason to waste CPU cycles when you can create a separate file system for your media and set compression to off on it.

    And most importantly I would want different snapshot policies for different data types. I don’t need years worth of retention for a movie collection, but I would like to have years worth of retention on my documents filesystem because it is relatively small so the storage consumed is minimal to protect against accidental deletion.