Profiling PyTorch training has an interesting measurement problem: the more you measure, the more you can change the behavior of the run itself. A simple example is torch.cuda.synchronize(). It gives cleaner timing boundaries, but it also inserts synchronization points into an otherwise asynchronous
