I am in the middle of considering an upgrade to my Home Server. I want to get a decent GPU for LocalAI. I mean, I have an RTX 3060 TI 16GB, I know that is much more than most people have, but even though it has a lot of cram the bus width is really slowing it down - I get ~23 tok/s on Gemma 4 26b A4
