tldr: I’m going to set up raid z2 with 4x8TB hard drives. I’ll have photos, documents (text, pdf, etc.), movies/tv shows, and music on the pool. Are the below commands good enough? Anything extra you think I should add?
sudo zpool create mypool raidz2 -o ashift=12 /dev/disk/by-id/12345 ...
zfs set compression=lz4 mypool #maybe zstd?
zpool set autoexpand=on mypool
zpool set autoreplace=on mypool #I might keep this off. I can see myself forgetting in the future
zpool set listsnapshots=on mypool
With ai raising hard drive prices, I over spent on 3x10TB drives in order to reorganize my current pool and have 3 hard drives sitting on a shelf in the event of a failure. My current pool was built over time but it currently consists of 4x8TB drives. They are a mirrored stripe so a usable 16TB. If I understand it correctly, I can lose 1 drive for sure without losing data and maybe a second drive depending on which drive fails. Because of that, I want to move to raid z2 to ensure I can lose 2 drives without data loss. I’m going to move data from my 4x8TB drives, to the 3x10TB, reconfigure the 4x8TB, and move everything back. I run Immich, plex/jellyfin, and navidrome off the pool. All other documents are basically there for long term storage just in case. What options should I use for raid z2 when setting it up?
I know I can look this stuff up. I have been and continue to do so, I was just hoping for some advise from people that are more knowledgeable about this than me. The move from the 4x8TB drives to the 3x10TB is going to take ~3 days so I really don’t want to mess this up and have to start over 😅
Edit:
After looking up each property, this is the command I will probably end up using to create the raid z2 pool, thanks Avid Amoeba:
sudo zpool create
-o ashift=12 -o acltype=posixacl -o xattr=sa
-o compression=lz4 -o dnodesize=auto -o relatime=on
-o normalization=formD
raidz2
mypool
/dev/disk/by-id/12345 …
Edit2:
Above command didn’t work on my machine. The order and uppercase “O” matters. Had to do this:
sudo zpool create \
mypool \
raidz2 \
-o ashift=12 -O compression=lz4 \
-O normalization=formD -O acltype=posixacl \
-O xattr=sa -O dnodesize=auto \
-O relatime=on \
/dev/disk/by-id/12345 ...
Edit3:
And finally, after all this, I set up my tmp pool of 3x10TB disks as a raid z2 instead of raid z1. Spent a day and a half transferring before I finally saw my mistake after running out of space 🫠
If I recall correctly compression, especially lz4, has been shown to impact performance negligably.
Yeah it won’t make much difference these days.
I suppose my point was more so that because ZFS is a pool that can be split up with filesystems new users should be thinking a little differently than they would have been used to with traditional raid volumes/partitions.
With a normal filesystem partitions are extremely limiting, requiring you to know how much space you need for each partition. ZFS filesystems just being part of the pool means that you can get logical separation between data types without needing that kind of pre-planning.
So many settings with ZFS that you may want to set differently between data types. Compression, export settings, snapshot schedules, replicating particular data sets to other systems, quotas, etc.
So I was mostly just saying “you should consider splitting those up so that you can adjust settings per filesystem that make sense”.
There is also a bit of danger with a single ZFS filesystems if you have no snapshots. ZFS being a copy on write filesystem means that even deleting something actually needs space. A bit counter intuitive but deleting something means writing a new block first then updating the FS to point at the new block. If you fill the pool to 100% you can’t delete anything to free up space. Your only option is to delete a snapshot or delete entire filesystems to free up a single block so that you can cleanup. If you don’t have a snapshot to delete you have to delete the entire filesystem and if you only have one filesystem you need to backup+delete everything… ask me how I know this ;)
If you have several filesystems you only need to backup and destroy the smallest one to get things moving again. Or better yet have some snapshots you can roll off to free up space or have quotas in place so that you don’t fill the pool entirely.