Issues with Fine-Tuning the AMT Model

[Xiugapurin]

Hi,

Thank you for sharing this great work! I have been following the provided instructions to fine-tune the AMT model, but I encountered issues when performing inference with the trained model. Specifically, the model does not generate a MIDI file correctly. Below are the details of my setup and observations:

My Setup:

• GPU: 1 × NVIDIA A100 (40GB) (instead of 2 × A100 as in the paper)
• Batch size: 2 (instead of 8 due to memory constraints)
• Followed all preprocessing steps as described in the README
• Used the default config.json without modifications

Issues Observed:

After fine-tuning the model, I attempted inference using my trained parameters. However, the model consistently fails to generate any notes in the MIDI output. Below are some key observations:

• Feature Extraction (wav2feature): The extracted feature tensors appear reasonable and match the paper’s expected shapes.
• Onset, Offset, Frame, and Velocity Predictions (transcript): The maximum values of the predicted matrices are significantly lower than those reported in the paper.
  • My model’s output:
    • onset max: 0.046
    • offset max: 0.0079
    • frame max: 0.318
    • velocity max: 0 (no velocity information detected)
  • Paper’s reported output:
    • onset max: 0.766
    • offset max: 0.704
    • frame max: 0.998
    • velocity max: 105
• MIDI Output (note2midi): Since all activation values are low, the model detects 0 notes and does not generate a MIDI file.

Possible Causes & Questions:

1. Could the batch size reduction (from 8 to 2) be the reason for this degradation? If so, do you recommend adjusting the learning rate, batch normalization, or any other hyperparameters to compensate?
2. Does training on a single GPU affect the results significantly due to different gradient accumulation behavior?
3. Are there any additional hyperparameter adjustments that might help when training with a smaller batch size?

I’d really appreciate any insights or recommendations on how to improve the fine-tuning performance and MIDI output quality. Thanks in advance for your help!

[310hz]

Thank you for verifying our model.

Proposed Solution

First, please modify the $\theta$ parameters in config.json as follows:

{
  "train": {
    ...
    "theta_onset": 0.03,
    "theta_frame": 0.1,
    "theta_velocity": 0.0
  }
}

These values control the extent to which gradients are computed from positions where no sound is present. If set too high, the model incorporates excessive gradient information from silent regions, leading to an overall suppression of note activation.

Our model is highly sensitive to these $\theta$ parameters.

• If they are too high, too many notes may be generated.
• If they are too low, no notes may be detected.

Finding the optimal values can be challenging, and we also struggled to determine the best parameters. This constraint seems to have affected reproducibility.

Possible Causes

As you pointed out, a major difference between our setup and yours is batch size, which can affect the training results. Another critical factor is training data quality.

After collecting the data, we manually filtered out inappropriate segments, such as non-music sections included at the beginning of promotional videos. This preprocessing step may have contributed to performance differences when using the default settings.

Next Steps

To address this issue, please try the following steps:

1. Train the model again with the updated $\theta$ values and check the generated MIDI output as well as the maximum values of each prediction matrix.
2. Adjust the $\theta$ values if necessary:
   • If no notes are detected, reduce the values further.
   • If too many notes are generated, increase them slightly.
3. If the issue persists, please share the details of the problem along with the log.txt file recorded during training.

We recognize that these adjustments add complexity to the training process and are actively working on a more robust version of the model.

Thank you for your patience and for bringing this to our attention. We appreciate your collaboration in improving the model.

[Xiugapurin]

Hi,

Thank you for your response! After implementing your suggestions and conducting further experiments, I have successfully reproduced your AMT model and, unexpectedly, achieved a significant performance improvement. Below are the details of my training process, which may serve as a reference for future model development:

1. Batch size remained at 2, with a total of 5 epochs, resulting in a final loss of 0.458 and an F1 average of 0.318.
2. Hyperparameter adjustments: theta_onset & theta_velocity set to 0.02, and theta_frame set to 0.1
3. Using custom dataset:
   • Consists of 2858 paired songs (JPOP & KPOP).
   • Aligned using a weakly-alignment approach (as described in the PiCoGen2 paper).
   • A total of 45,335 samples were used during training.

I firmly believe that these improvements were made possible thanks to the well-designed architecture of your model, which provided a solid foundation for further enhancements. The method you proposed in your paper is highly promising and efficient.

If you are interested in these results, I have uploaded the trained model checkpoint to the Releases section of my personal GitHub repository. You can find it here.

I would be happy to discuss the details further if you’d like—please feel free to reach out to me privately!

[310hz]

Glad to hear that things went well!

The fact that you conducted further validation with a larger dataset and achieved even better results is a significant contribution to future research on automatic piano cover generation. Thank you for your efforts.

I have checked the shared model and noticed that the parameters for handling the style vector (e.g., fc_sv.weight) were not included. As a result, when generating MIDI using infer/__main__.py, no notes were produced. However, by writing a separate script and running inference without providing a style vector, I was able to generate a normal MIDI file.

Upon investigation, I found that there was an issue in the program shared in this repository, which caused training to start without incorporating the style vector. I apologize for this oversight.

Although training and inference are still possible without the style vector, it prevents control over the note density. In the shared model, the generated MIDI also seemed to contain a slightly higher number of notes than expected.

The program has already been fixed. Incorporating the style vector is expected to further improve accuracy, so if you have the time, please consider retraining the model with this update. I apologize for the inconvenience.

[lokman2k5]

Hi @Xiugapurin,
When do you plan on releasing the code/checkpoints for Etude? I liked the results it got in the demo.

[Xiugapurin]

Hi @lokman2k5,

Thank you for your interest in Etude. The project is currently private, but I’m finalizing the code and checkpoints and plan to make it public by next weekend (around September 7). I’ll share the link once it’s available.

[Xiugapurin]

Hi @misya11p ,

I’m sorry for the late reply, and I truly appreciate your detailed and thoughtful message regarding the issue — thank you for carefully pointing this out and sharing the fix. Your clear explanation and well-organized code made it easy for me to understand the problem, and I’m very grateful for the effort you put into maintaining this repository!

As a follow-up, I also want to acknowledge that the AMT-APC project has been a major inspiration for my own research. It not only guided me technically but also directly shaped the direction of my current work and upcoming paper. Building on this foundation, I started a related project of my own, called Etude. I’m currently preparing the code and checkpoints, and plan to make them public by next weekend (around September 7). It would be an honor if you could take a look once it’s released :)

[310hz]

Hi @Xiugapurin

I'm so glad my comments were helpful! It's also great to hear that my research was useful for your work. I'm really looking forward to seeing the Etude project when it's public!

[Xiugapurin]

Hi @misya11p & @lokman2k5,

The code and checkpoints for Etude are now available.
Feel free to take a look!

[310hz]

Hi @Xiugapurin

I checked out the demo and was really impressed. The performance is amazing. Your model seems to reproduce the original song more accurately than traditional models, and the output is much cleaner and better structured as sheet music than our own model's.

Is the paper available yet? Please let me know if it's published.

I'm truly honored that our research was helpful to you. I'm sure your work will be of great help in our future research as well!

[Xiugapurin]

Hi @misya11p

I’m sorry for the late reply, and thank you very much for your generous feedback on the demo — it really means a lot to me.

The paper will be made available this weekend (September 20), and I’ll be glad to share it with you once it’s released. Thanks again for your support and encouragement!

[Xiugapurin]

Hi @misya11p

I’m glad to let you know that the paper is now ready — you can find it on our project page. The arXiv submission is still under review, so for now I’m only providing the PDF version. Once the arXiv version is accepted, I’ll update the project page and the repository with the official link.