[BUG] Crash on attempted checkpoint save when training loop is too fast

[sdatkinson]

Need to wrap the checkpointing callback with a retrying

[kf-rahman]

Hey! New to the project could I take a crack at this??

[sdatkinson]

@kf-rahman go for it! I'd prefer if you do it by extending the checkpoint class and wrap the save action with some try/except and retry a few times. Sound good? Or let me know if you have another opinion.

Also, it'd be good to write a test that's got a sufficiently-tight train loop that this Issue can cause the test to fail until your change is implemented.

[kf-rahman]

Hey, here is my PR for the solution: #600. Followed your approach and added a little time delay for extra precaution.