Back with an update. Some of you saw v10 FSP, the one where I found that Future Sentence Prediction gave a 2.5x PPL improvement. Well, I've run 20+ more experiments since then trying to get the model to actually understand what it's saying. Spoiler: lower PPL does not mean more coherent. Quick clari
