I’m having a hellluva time trying to fine-tune an FP8 LLM model (both FFT and LORA). I’ve been using VLLM and they sure don’t make it easy on SM120 for FP8. So far everything I’ve tried even semi-successfully requires upscaling (either layers or LORAs) to BF16 which kind defeats the purpose.
