I converted Mistral Nemo from a dense model into a sixteen expert MoE model: https://huggingface.co/blascotobasco/Mistral-NeMoE-12B-16E The core problem is that I am a student with budget constraints and can’t afford full parameter or extended fine tuning. I did my best to restore coherence, and it
