At the moment MLX (and Llama.cpp for Macs) run 16bit activations everywhere. Despite this, the M5 generation silicon actually does support INT8 activations - it actually allows w4a8 d_type. It's just that no inference backends are using them yet I built some w8a8 kernels and have managed to get 1.4x
