Probably best to look at it as a competitor to a Xeon D system, rather than any full-size server.
We use a few of the Dell XR4000 at work (https://www.dell.com/en-us/shop/ipovw/poweredge-xr4510c), as they’re small, low power, and able to be mounted in a 2-post comms rack.
Our CPU of choice there is the Xeon D-2776NT (https://www.intel.com/content/www/us/en/products/sku/226239/intel-xeon-d2776nt-processor-25m-cache-up-to-3-20-ghz/specifications.html), which features 16 cores @ 2.1GHz, 32 PCIe 4.0 lanes, and is rated 117W.
The ostensibly top of this range 4584PX, also with 16 cores but at double the clock speed, 28 PCIe 5.0 lanes, and 120W seems like it would be a perfectly fine drop-in replacement for that.
(I will note there is one significant difference that the Xeon does come with a built-in NIC; in this case the 4-port 25Gb “E823-C”, saving you space and PCIe lanes in your system)
As more PCIe 5.0 expansion options land, I’d expect the need for large quantities of PCIe to diminish somewhat. A 100Gb NIC would only require a x4 port, and even a x8 HBA could push more than 15GB/s. Indeed, if you compare the total possible PCIe throughput of those CPUs, 32x 4.0 is ~63GB/s, while 28x 5.0 gets you ~110GB/s.
Unfortunately, we’re now at the mercy of what server designs these wind up in. I have to say though, I fully expect it is going to be smaller designs marketed as “edge” compute, like that Dell system.
We’ll see if they even make them. I can’t imagine there’s a huge customer base who really needs to cram all that I/o through only two or 4 lanes. Why make these ubiquitous cards more expensive if most of the customers buying them are not short PCI-E lanes? So far most making use of 5.0 are graphics and storage devices. I’ve not seen any hint of someone making a sas or 10 gbe card that uses 5.0 and fewer lanes. Most cards for sale today still use 3.0 let alone 4.0.
I might as well just drop the cash on a real EPYC CPU with 128 lanes if I’m only going to be able to buy cutting edge expansion cards that companies may or may not be motivated to make.
