It’s been a while! Let us know what you’re currently working on, what exploded, what cool stuff you found, or just brag that everything is running fine unattended.
I haven’t been doing a lot, renovate auto upgrades for patches and sometimes minor updates work great so far. I got involved in my union to spin up a mastodon server soon, so there’s a new project, but I already run mastodon so it shouldn’t be much of a deal, I just need to get comfortable with the infrastructure. I also set up ntfy but only sent some test notifications and didn’t configure any app yet - planned for Element Chat.
[Meta] I’m not sure how to apply the new tags. In the recent versions of this thread, people sometimes posted about their ollama setup or the like, but I’m not mentioning AI in the post body (except this meta thingy) and it shouldn’t be much of the discussion, but it’s an open post. Maybe curbstickle can chime in to let me know if I should apply one of them.
LOL Not quite, however, I am choking on the 5090 but as I understand, that’s the better option, and 32 GB seems to be the sweetspot for AI. I actually picked up the tower and original RAM for less than $500 USD.
First of all, I mean zero offense with any purchase decision. A 5090 is very good.
…But if I were paying that kind of money, I’d probably get a 4090 and a new motherboard/CPU instead. Maybe a used DRR4 threadripper system.
Hybrid (CPU + GPU) inference is where it’s at these days. It opens up a whole world of huge MoE models, whereas on an 5090 you are stuck with Qwen 27B.
Having a fast CPU, with lots of RAM channels, with full PCIe bandwidth is much more important for that than having a 5090, where a 3090 or 4090 will get the job done.
Even if pure speed is your primary concern, you can tune a sparse 120B model (like Laguna) to run almost as fast as Qwen 27B on a 5090, and get at-least-good results.
It’s more finicky and involved, though. For sure.
Running an LLM on a 5090 is a task. Hybrid CPU + GPU inference is a hobby.