I've tried renting some cloud instances to get an idea of the speed of various GPUs. I'm using a recent version llama.cpp with CUDA 12.8 support. I've tried running a 31B dense model, Q6, on an RTX 5090 and an H100, and the results surprised me. The 5090 generates at about 57 tok/sec, while the H100
