Current neural networks have a fundamental geometry problem: If you feed them garbage data, they won't admit that they have no clue. They will confidently hallucinate. This happens because the standard Cross-Entropy loss requires models to push their features "infinitely" far away from the origin to
