Piano Dynamic Estimation

This repo corresponding to our submitted paper to ICASSP2026.

• Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-scale Network [PDF]

Our proposed multitask model can estimate piano dynamics, change points, beats, and downbeats from audio. We are polishing our inference stage, about how to integrate with "High-resolution Piano Transcription (TASLP2021)" system, more AMT systems in progress.

Figure 1: User case demonstration.

To use our model,

1. Setup python environment
2. Inference with our pretrain model

We made a PyTorch implementation of Pampalk’s PsychoFeatureExtractor, to provide Bark-scale specific loudness (BSSL) senation based on "Zwicker 1999 method" (yielding a up to 24-dim feature). We found the BSSL outperformed the logMel in our tasks, with a 5x smaller feature size. Find more in

• PsychoFeatureExtractor

If want to retrain the model or reproduce our paper results, continue the follow sections.

3. Download MazurkaBL Dataset
4. Perform Data Preprocessing
5. Training and WandB Report

You can check our training history in WandB Report and Evaluation Results on

• Evaluation (Ours and Baselines)

Environment Setup

Create and activate Conda env with CPU-compatible PyTorch 2.2, then Add the pip-only packages.

conda create -n dynest python=3.10 numpy pandas h5py "pytorch=2.2.*" "torchaudio=2.2.*" -c pytorch -c conda-forge
conda activate dynest
pip install hydra-core omegaconf wandb einops librosa mido tqdm cython
pip install "numpy<2"
pip install seaborn
pip install madmom

This environment is for Ubuntu 22.04 + CUDA 12.2 + RTX3090 24GiB. Basically it can support upper version, we also tested in Macbook Pro M4 with the same environment, and Ubuntu 22.04 + CUDA 12.8 + RTX5090 32GiB using PyTorch 2.6.

Inference & Checkpoints

Add predicted dynamic markings to an existing or AMT-transcribed score. Start with Inference.ipynb.

• We provide a pretrained multi-task, multi-scale checkpoint at workspaces/checkpoint/ folder. This checkpoint is our best pre-trained model under the 5-fold protocol, selected for real-world inference.
• More checkpoints (single-task OR ablation variants) are available upon request, or obtained by the training instruction we provided.

PsychoFeatureExtractor

We implemented this PsychoFeatureExtractor in PyTorch framwork, according to the Pampalk’s paper (2002) and MATLAB Music Analysis Toolbox (2007). This providing the BSSL in sones, and we compare our PsychoFeatureExtractor results with MoSQITo and MATLAB original, used their total loudness to visualize.

Figure 2: Bark-scale total loudness from different implementation.

Our PyTorch implemented BSSL (max 24-dim) has the similar total loudness as in MATLAB 2007 version (max 24-dim), while the high-res variant BSSL (max 240-dim) used in paper 2024 provided slightly different total loudness. We use the original BSSL as the foundation of this project. Explore the Bark-scale feature (incl. high-res and different psychoacoustic masking functions) is our future work.

NOTE: The following section is for training & reproduce

MazurkaBL Dataset

To download the dataset, visit the github repo MazurkaBL-master to download the annotation files. You should request the author for the audio files. Place these dataset at our workspaces/Dataset/ folder, and then doing the data cleanup we mentioned in Data Preprocessing.

• OR, you can download our cleanup data here [Link], and then pack_h5 with any sample_rate.

Data Preprocessing

There are some small problems we found in MazurkaBL dataset. We solved them with the following command. You can follow our Data_Preprocess.ipynb to easily handle these, as well as packaged the audio into h5 files for a quick loading.

# start here if you download raw mazurkaBL
python pytorch/data_preprocess.py --mode fix_problem  # <can skip>
python pytorch/data_preprocess.py --mode cleanup_meta # <can skip>
# start here if you download our cleanup data
python pytorch/data_preprocess.py --mode pack_h5 --sample_rate 22050

Here is the problem we found:

1. MazurkaBL-master M41-1 has an error. Its performance ID (pid) 9070b-01[wrong] should be corrected to 9070b-09[correct]. Affected files:

   • MazurkaBL-master/beat_time/M41-1beat_time.csv
   • MazurkaBL-master/beat_dyn/M41-1beat_dynNORM.csv

2. MazurkaBL M06-4 and M63-2 does not contain regular dynamic markings. We skip these two opus by setting the dataset.exclude_opus in the pytorch/config.yaml file.

   dataset.exclude_opus:
   - "M06-4"
   - "M63-2"

3. Due to the annotation quality, some pids performance will raise up the gradient dramatically during the training. NOTE: this is train-stage ONLY. You can skip this with null because it is a stable training trick. If set a enough epochs (e.g., 120 epochs), the training will converge, so you won't need this.

   dataset.exclude_pids: # null # Optional: for a stable training
   - "mazurka17-4/pid9058-13.h5"
   - "mazurka33-4/pid9080-08.h5"
   - ...
   
   dataset.exclude_pids: null # Evaluation

Training and WandB Report

Our training histroy and hyperparameters can be found in this WandB Report [Link].

Figure 3: Compare our multitask model (green) vs. each single-task model. We also compare the feature used "BSSL: solid line, logMel: dashed line"

Once the dataset is downloaded, and the data pre-processing done, you can follow our Train_Model.ipynb to start the model training. We are doing the 5-fold cross-validation, the data is split on the Mazurka opus to becomes 5 folds, these five split.csv files can be found on workspaces/ folder. To train with 5-fold:

python pytorch/train.py wandb.note='formal latent8 mmoe8 5x5' exp.model_name=MultiTaskCNN dataset.run_all_folds=True

Or simple train in the first fold:

python pytorch/train.py wandb.note='formal latent8 mmoe8 5x5' exp.model_name=MultiTaskCNN dataset.run_all_folds=False dataset.fold_index=0

You can modify other settings in the pytorch/config.yaml.

Evaluation (Ours and Baselines)

Our paper results can be found on the eval_and_benchmarks/ folder. You can find our ”SingleTask“ and “Multitask” models‘ evaluation results from:

• Eval_Singletask.ipynb
• Eval_Multitask.ipynb

Beat and downbeat tracking baselines results are obtained from their original implementation in GitHub: TCN + DBN by Bock et al.(2020) and Beat This by F. Foscarin et al. (2024). We retrain their model using their implementation. You can find the training history with evaluation results from:

• beat_tcn/Eval_BeatTCN_MazurkaBL.ipynb
• beat_this/Eval_BeatThis_MazurkaBL.ipynb

Dynamic and change point baseline is machine learning/algorithmic mehods, thereby we not reproducing their results, reported the literature instead. Download K. Kosta PhD Thesis (2017, PDF) to find these results.