I recently decided to rebuild my homelab after a nasty double hard drive failure (no important files were lost, thanks to ddrescue). The new setup uses one SSD as the PVE root drive, and two Ironwolf HDDs in a RAID 1 MD array (which I’ll probably expand to RAID 5 in the near future).
Previously the storage array had a simple ext4 filesystem mounted to /mnt/storage, which was then bind-mounted to LXC containers running my services. It worked well enough, but figuring out permissions between the host, the container, and potentially nested containers was a bit of a challenge. Now I’m using brand new hard drives and I want to do the first steps right.
The host is an old PC living a new life: i3-4160 with 8 GB DDR3 non-ECC memory.
-
Option 1 would be to do what I did before: format the array as an ext4 volume, mount on the host, and bind mount to the containers. I don’t use VMs much because the system is memory constrained, but if I did, I’d probably have to use NFS or something similar to give the VMs access to the disk.
-
Option 2 is to create an LVM volume group on the RAID array, then use Proxmox to manage LVs. This would be my preferred option from an administration perspective since privileges would become a non-issue and I could mount the LVs directly to VMs, but I have some concerns:
- If the host were to break irrecoverably, is it possible to open LVs created by Proxmox on a different system? If I need to back up some LVM config files to make that happen, which files are those? I’ve tried following several guides to mount the LVs, but never been successful.
- I’m planning to put things on the server that will grow over time, like game installers, media files, and Git LFS storage. Is it better to use thinpools or should I just allocate some appropriately huge LVs to those services?
-
Option 3 is to forget
mdadmand use Proxmox’s ZFS to set up redundancy. My main concern here, in addition to everything in option 2, is that ZFS needs a lot of memory for caching. Right now I can dedicate 4 GB to it, which is less than the recommendation – is it responsible to run a ZFS pool with that?
My primary objective is data resilience above all. Obviously nothing can replace a good backup solution, but that’s not something I can afford at the moment. I want to be able to reassemble and mount the array on a different system if the server falls to pieces. Option 1 seems the most conducive for that (I’ve had to do it once), but if LVM on RAID or ZFS can offer the same resilience without any major drawbacks (like difficulty mounting LVs or other issues I might encounter)… I’d like to know what others use or recommend.
You must log in or # to comment.
I don’t use it so I can’t recommend it, but if you’re interested in other options to research there’s a mergerfs+snapraid combo.
I currently pass through my disks to an unraid VM and then mount them through nfs which works (but from the sounds of it probably not a valid option for you, not would I recommend it), but I want to try replacing it with mergerfs at some point.
The thing that has mainly turned me off of zfs is (from what I understand) that you kinda need to plan how you’re going to expand when you set it up. Which really doesn’t work for me with a random collection of disks of varying sizes.
Another note for option 1, since proxmox 8.4 there is virtiofs which would allow you to mount a folder to a VM without having to go through nfs. You may have to mess with selinux in the VM depending on what you do in there, but just fyi it’s a thing.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters LVM (Linux) Logical Volume Manager for filesystem mapping SBC Single-Board Computer ZFS Solaris/Linux filesystem focusing on data integrity
3 acronyms in this thread; the most compressed thread commented on today has 8 acronyms.
[Thread #234 for this comm, first seen 12th Apr 2026, 07:10] [FAQ] [Full list] [Contact] [Source code]
Zfs can become painfully slow if you don’t have RAM for it. I tried to run ZFS on my old setup with 64GB RAM and with moderate amount of virtual hosts and it was nearly useless with heavier io-loads. I didn’t try to tweak settings for it, so there might be some workarounds to make it work better, I just repartitioned all the storage drives with mdadm raid5 array and lvm-thin on top of that. Zfs will work with limited memory in a sense that you don’t risk losing data because of it, but as mentioned, performance might drop significantly. Now that I have a system which has memory to run raidz2 it’s pretty damn good, but with limited hardware I would not recommend it.
LVM itself is pretty trivial to move on a another system, most modern kernels just autodetect volume groups and you can use them as any normal filesystem. If you move full, intact, mdadm array to a new system (and have necessary utils installed) it should be autodetected too, but specially with degraded array manual reassembly might be needed. I don’t know what kind of issues you’ve been getting, but in general moving both lvm and mdadm drives between systems is pretty painless. Instead of mdadm you could also run lvm-mirroring on the drives so it’ll drop one layer off from your setup and it potentially makes rebuilding the array a bit simpler on another system, but neither approach should prevent moving drives to another host.
Lvm-thin is more flexible and while it might be a slightly slower on some scenarios I’d still recommend using that. Maybe the biggest benefit you’ll get from it is an option to take snapshots from VMs. Mounting plain directories will work too, but if your storage is only used by proxmox I don’t see any point in that over LVM setup.
That whole “1 GB per TB of capacity” is some generic rule someone made up once that doesn’t really have anything backing it up. It depends completely on your use case. If it’s mostly media storage that is rarely accessed, I’m sure that 4 GB is plenty.
I run a beefy TrueNAS server for a friends video production company with a 170 GB ZFS array, right now ARC is using 40 GB of RAM with 34 GB free that it’s not even bothering to touch, I’m sure most of the ARC space is just wasted as well. That’s just one example of how 1 TB = 1 GB makes no sense.
Does it have reduplication enabled? Afaik that’s the functionality that the high memory footprint is usually quoted.
(Also, was that meant to be 170TB?)
Zfs for some reasons is always loved by the self hosting experts.
Personally I don’t like it because it’s over complicated and not officially part of the Linux kernel.
20y of self hosting for me means Linux software raid (raid1 then moved to raid5) and mostly ext4, recently (=last 2 years) upgraded to btrfs on my data raid. Btrfs at least is integrated in the Linux kernel and while has some drawbacks (don’t do raid with btrfs, put btrfs on MD raid instead) its been super rock solid.
I would have kept ext4, but thought why not try btrfs, and it’s been so smooth that I had no reasons to go back.
If you plan to expand further, or add another node, I would recommend starting with zfs and see how it performs, because it’s integrated with proxmox and is required for native replication. But you can’t safely convert a zfs mirror to raidz1, you have to take one of the mirrors offline, create a degraded raidz1 with only two disks, copy the data, then wipe the third disk and add it. It’s sketchy and should not be performed with data you can’t afford to lose. But it works, I’ve done it.
Second to that, I’d do traditional software raid, though you’d probably end up doing the same process.
I personally don’t like lvm. I find it frustrating to work with. I’m not sure if you could do the same operations as above.
I’m not sure if you can do data recovery on proxmox virtual disks, but even if the host is completely dead, you can reinstall it and import the VMs and disks. I’ve done that too. But now I run pbs and rclone the backup files to the cloud.
ZFS is slightly more portable than md and comes with some additional benefits and performance tunables you don’t get with md, so that’s my 0.02.
Note that ZFS can use memory for arc, but it is happy to run on much less than what you have. My nas is a rock 5 SBC with 4GB memory and it’s fine, performance-wise, but I’ve run ZFS on much less and it adapts well to more or less ram.
If you end up with a second proxmox server with its own storage for redundancy in a cluster then ZFS is great. You can mirror each VM. The only issue is the ZFS storage needs to be named the same. But it can be a different size or media.
My main concern here, in addition to everything in option 2, is that ZFS needs a lot of memory for caching. Right now I can dedicate 4 GB to it, which is less than the recommendation – is it responsible to run a ZFS pool with that?
Yes, you’re going to be OK.
In your typical “self hosting” scenarios you can safely stay well below the RAM recommendation - assuming you don’t have many simultaneous users and tasks.
ZFS does not actually need that much RAM for basic operation, but it can make very good use of it in medium/heavy load scenarios.
Gotta preach for the cult of ZFS. It’s check summing, copy on write, and zraid features are all exactly what you want for data resilience. Plus you get transparent compression, and snapshots that can provide a bit of a stop-gap for your lack of backups.
It will normally soak up any and all memory for buffers and caches, but is meant to quickly free up when it’s needed by an app. Linux already does this on any filesystem with its page cache.
Oh and mounting a ZFS dataset on a new machine is super quick and easy, it stores it’s config on the drives themselves, so you can plug them into a new box and
zpool import -afand boom it’s mounted and ready to go.Just curious – why is the 4GB cache needed?
ZFS uses the RAM intensively for caching operations. Way more than traditional filesystems. The recommended cache size is 2 GB plus 1 GB per terabyte of capacity. For my server, that would be three quarters of the RAM dedicated entirely to the filesystem.
