TU/e ARIA lab
Batch size #48
-
|
So i was trying some implementation of my model and it is currently getting really big, so to train it properly I have to reduce the batch size to 4. Is there a way to increase the batch size while maintaining model complexity? My training is currently quite unstable compared to previous models although performance isn't too bad yet. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
You could use automatic mixed precision (AMP) to reduce memory usage allowing for a larger actual batchsize on the gpu. If that's still not enough consider implementing gradient accumulation to increase your effective batchsize (accumulating gradients over multiple batches and updating weights after a specific number of batches). |
Beta Was this translation helpful? Give feedback.
You could use automatic mixed precision (AMP) to reduce memory usage allowing for a larger actual batchsize on the gpu. If that's still not enough consider implementing gradient accumulation to increase your effective batchsize (accumulating gradients over multiple batches and updating weights after a specific number of batches).