Seeking input for deploying 3-node cluster to a data center - Learning Together Forum
discuss.livingcartoon.org
Part of Living Cartoon Company

cross-posted from: https://discuss.online/post/39090314

Full layout on my forum here. Basic idea is 3 thin clients that fit within:

  • 1u space requirement
  • Possibly adding 2.5gb nic to the pci-e low profile slots of each hp t740 for Ceph shared storage.
  • Kubernetes and/or Proxmox experiments
    • Adding some sort of remote management via jetkvm or pikvm or similar.
    • Additional expansion no doubt needed for connecting to the three nodes.
+----------------------------------------------------------+
|                    CLUSTER TOTALS                        |
+----------------------+-----------------------------------+
| TOTAL CORES          | 12 physical / 24 threads          |
| TOTAL RAM            | 192 GB DDR4                       |
| TOTAL STORAGE        | 1.5 TB NVMe (local only)          |
| TOTAL NODES          | 3                                 |
| TOTAL NICs           | 3–7 (depending on 4-port NIC)     |
| GPU                  | None                              |
| KVM/IPMI             | None                              |
+----------------------------------------------------------+

All suggestions greatly appreciated! Never done this before, but hoping to save myself from the future hassle of having to drive to the data center… would much rather log in on Wireguard and address things by remotely rebooting, etc.

  • kiol@discuss.onlineOPEnglish
    2·
    9 hours ago

    Seems Mellanox ConnectX-3 Pro Dual Port 10G SFP+ Low Profile MCX312C-XCCT is decent choice, which I can use for 10gb triangle between the current 3 nodes. I was thinking of using jetkvm + rs-232 expansion serial cable with 4-port hdmi/usb switcher for controlling the nodes. Financially, a future expansion would be moving from the triangle to a 10gb switch, allowing for NAS or other node additions. Also, each node has an empty Sata SSD port currently. Updated the forum thread.

    • fruitycoder@sh.itjust.worksEnglish
      2·
      6 hours ago

      I really enjoy the pikvm and the switcher for my home lab. Redfish support gets fishy with a switcher if that is a concern though.

      I do love a good mesh for a cluster block though. My next next next project is using KubeOVN to turn my cluster block into a switch with “out” connections to connection other devices (wifi, laptop, cameras, etc) to it as my network router and of course upstream from the modem and hotspot for Internet connection.