I’m in the process of wiring a home before moving in and getting excited about running 10g from my server to the computer. Then I see 25g gear isn’t that much more expensive so I might was well run at least one fiber line. But what kind of three node ceph monster will it take to make use of any of this bandwidth (plus run all my Proxmox VMs and LXCs in HA) and how much heat will I have to deal with. What’s your experience with high speed homelab NAS builds and the electric bill shock that comes later? Epyc 7002 series looks perfect but seems to idle high.
You most likely won’t utilize these speeds in a home lab, but I understand why you want them. I do too. I settled for 2.5GBit because that was a sweet spot in terms of speed, cost and power draw. In total, I idle at about 60W for following systems:
- Lenovo M90q (i7 10700, 32GB, 3 x 1 TB SSD) running Proxmox, 15W idle
- Custom NAS (Ryzen 2400G, 16GB, 4x12TB HDD)v running Truenas (30W idle)
- Firewall (N5105, 8GB) running OPNsense (8W idle)
- FritzBox 6660 Cable, which functions as a glorified access point, 10W idle
With 25 GbE, even 10, I’d be tempted to PXE boot client systems. Maybe still have a local PCIe SSD for windows game files.
Dunno how that would actually work with Windows, but it was fun when I did it for beowulf nodes. Setting RPis to netboot is a little involved, but you can create an OSMC image and give all your TVs a consistent ‘smart’ interface. You don’t even need 10GbE to be pretty functional for the Pi, but my experience is that WiFi is not fast enough.