Training args with COCO dataset

[hwan6615]

I'm currently trying to reproduce the results using the COCO dataset, but I'm unable to achieve the performance reported in the paper.

Would it be possible to know the arguments used for training?

[DavidHuji]

Thank you for your interest in this repository! I have set the best hyperparameters as the default values in argparse. Which arguments are missing that you need assistance with? What performance did you achieve, and how does it compare to the results reported in the paper?

[hwan6615]

Thank you for your good paper and response.

The experiments were conducted using the COCO dataset.

First, the arguments and performance before normalization are as follows.

When normalization was not performed, the performance was very low. Performance is best at 1 epoch, and as training progresses, performance declines.

In contrast, when normalization was performed, the performance improved, but compared to the performance in the paper, it is considerably low, and BLEU-4 is particularly low.

Do you think there might be any issues, and is there any additional information you would like?

[DavidHuji]

Looks like the batch size is a bit bigger but i do not think this is the issue. Did you evaluate on the last epoch? Maybe try earlier ones where the loss is already low as much as in the last epoch (i.e. the 7th rather than the 10th).

[hwan6615]

I conducted evaluations for all 10 epochs, and this represents the best performance among them. As mentioned above, without normalization, performance decreases as epochs progress. When normalization is applied, performance improves with each epoch, but falls significantly short of the performance reported in the original paper.

[DavidHuji]

Oh I see, I missed that earlier. I think then trying a smaller bs as in the default parameters might be helpful (or at least increase the lr instead or the amount of epochs) because it sound like the training was just not long enough (when the bs is x4 bigger, there are x4 less weights updates).

[hwan6615]

Thank you for your response. I will proceed with an experiment.