tl;dr: Proxmox or bare metal? Containers yes/no? What is your use case?
I used a Dell R710 when I first started self-hosting, it ran ESXi with one VM for each service I wanted. Restarting the server required me to first shutdown each VM in order and then the whole host. When setting up a new service to host I had to create a new VM (allocate RAM, disk etc), install the OS (I ran Debian) and then follow instructions for how to setup the service. This mostly was not a problem but one software I never managed to get working was Apache Guacamole.
Nowadays I have a Dell Optiplex I salvaged for parts, got a new case and all my HDDs from my R710. Because it has much less RAM and an old Intel i5 (6th or 7th generation), I decided to get into Docker. With Docker containers you write your compose file and it will just work. No more need to dig through documentation for which version of a dependency to use, how to handle if two services on the same host need different versions (this was part of the reason for one VM/service). With Docker, I can try out a software in seconds and have it configured to my liking in minutes.
Today I have NixOS (bare metal) and it comes with Podman which uses systemd. Hence restarting my OS (albeit not that often, almost never unless I mess up my config) is a no-brainer because Podman via systemd will manage everything. Adding, stopping or removing containers in general is easy. I have a script running as a service which will stop a container, create a BTRFS subvolume snapshot, start the service again and start borg backup to backup from the snapshot.
For me using Proxmox would just an extra layer of complexity I don’t need. I only have one server and I am the only user.
Questions:
- Do you use Proxmox instead of a bare metal installation?
- Do you use containers or do you install manually?
- What is you use case that requires your setup the way it is?
You must log in or # to comment.
I run podman in proxmox vms
Install incus on your OS of choice to manage LXCs and VMs, it’s ideal!
No need to chain yourself to an OS that is rolling on the free branch, get stability and control!
As for LXCs vs Podman containers, seems it is preference of control. LXCs are little OSes you need to keep up to date, containers need to be rebuilt to keep up to date. (I think only Linuxserver images actually rebuild just for base OS updates, hopefully the reverse proxy and authentication images too)
Podman brings some nice networking, read-only features, and user abstraction with it, I think that helps it push ahead.
That said, LXCs are little OSes and that flexibility can be very useful. For instance, incus is able to make an LXC with a unique Mac from an Ethernet adapter - I haven’t cooked how to do that with Podman yet. So I run my DNS from there so it doesn’t mess with my server’s DNS port.
I don’t care much for proxmox but that’s because I live in the terminal anyway and have a lot of experience administering Linux systems. So for me it’s just an extra layer of complexity, the benefits don’t really apply when my setup has IaC, most of my containers are in Podman rather than LXCs, my backups are automated with my own scripts and cronjobs, and I only run a couple of VMs. Also it makes encrypting the base system more difficult than it is with a standard distro
I like the flexibility that proxmox provides me. I do this as a hobby and I’m self taught. I can try a bunch of things and if I mess up I can start over without much hassle.
Proxmox running LXC’s and VM’s, all of them also have docker installed
I used to be bare metal Debian, but I moved to ProxMox for a few reasons.
- Backup and restore is a breeze, and the UI makes things human friendly.
- It makes it easier to separate my wiki notes in an LXC so that I can still see them if I’m doing maintenance on my main server VM that requires restarts.
- There is essentially no noticeable performance reduction.
It works, it’s easy, and the simplicity has saved my butt a few times. I don’t think I’ll be switching, and I’d recommend it to anyone running a homelab. I still use docker to manage most of my services inside a VM.
I’ll die on this hill.
Containers run on “bare metal” in the same way that other processes on your system run.
Who was saying it was otherwise? Containers are not virtualization and never have been.
Maybe I misunderstood - The “NixOS (bare metal)” seemed to imply to me that the containers were not bare-metal by omission. If not then my complaint is retracted.
Generally speaking:
- just want it to work with minimal effort: containers
- need very custom things, or very high efficiency: bare metal
- need high security, emulating bare metal, or like to do extra IT work: virtualization
Generally, these days, containers win 99% of the time as the best home lab backbone.
You should be using them depending on your needs. There’s a difference between app containers (single app per container), system containers (multiple apps in the same container) and VMs (OS + whatever, virtualized rather than containerized).
You probably need app containers most of the time so docker or podman is a good fit. But sometimes you might feel more confortable with another level of abstraction. Tools like Proxmox or Incus make it easy to manage “system”-level abstractions like system containers (with LXC) or VMs (with KVM) and give you a unified management approach.
You don’t have to give up app containers either. You can run docker or podman inside an LXC system container and have the best of both worlds.
Deciding when to take advantage of the system abstraction is the hard part. A simple rule of thumb is to do it when you’d like to manage the “machine” that holds the stack in a way that’s different from the host. Maybe you want to run a different Linux distro; maybe it’s the same distro as the host but you want to organize it differently; maybe you need to run a non-Linux OS.
I’m running Docker inside a Debian VM on Proxmox and I forgot why. Next time I’d just run Docker on Debian.
When I got started I saw a lot of suggestions for proxmox. For me, it was an extra layer I didn’t feel like I needed. I installed Open Media Vault (just Debian with a web interface) and I have my docker containers for running the apps and eventually setup VMs for various other things. Staring out, Docker was a bit of a headache but now that I’ve got practice with it docker compose is fairly simple.
I appreciate that these days there are a bunch of really good tools out there and options.
I use proxmox for basically anything in my homelab, but there’s no particular reason. I use it because it does what I need. It supports VMs in case I need a bit more control over what runs in the system (like a Windows VM) and it supports LXCs that fundamentally work like docker with less configuration steps. It provides a nice webUI if I need to check up on some things and an API so I can orchestrate all my systems with terraform.
It does the job and I never saw a reason to switch away from it.
I moved away from proxmox about 3 years ago to incus, primarily because Proxmox has a lot going on that I don’t need, but also incus is much more amenable to configuration with ansible. Also, and maybe this has changed recently, but passing through PCI devices to incus is much easier.
I use Proxmox as a VDI server. I understand that this isn’t homelab territory, but selfhosted VDIs have proven exponentially more reliable and easier to manage than cloud based ones (after the initial PitA setup). Flawless copy/paste, screen resize, and most importantly, file transfers. When you connect to 12 different clients, each with a different set of security requirements, system hardening, monitoring, and VPN access, being able to console amd fully interact with a sandboxed desktop becomes priceless.
Just about everything is Proxmox for me, though a few things are just Debian.
I use LXCs, in general I don’t like Docker so I avoid it in most cases. I have a couple VMs for it where I may spin something up to test, but if its around long-term it gets moved to an LXC. Usually because I prefer a bit more control on setup, and find it more manageable than having a bunch of mounts for everything. It also keeps things arranged in a way I prefer, gives me a single interface to access what I want, and makes backup more straightforward.
Whether I want a whole VM or just to run a single app as a service, I only have to go to one spot to do it.
