PrismML Bonsai 27B is surprisingly usable on the Jetson Orin Nano 8GB

Ctx Size: 48k Prompt processing: 27t/s Token/s: 4.31 t/s RAM Usage (on cold start): 6.2GB with model loaded (no-mmap) Yes it is not very fast but it is a good AI setup consuming roughly 25W and it can do complex tasks pretty well now.