I've been wondering about models that are trained with MTP (Multi-Token Prediction) and whether the intermediate prediction heads can effectively serve as standalone smaller models. For example, DeepSeek has released DS4 Flash and DS4 Pro, but let's say a future model only releases a large flagship
