BR Restore incremental data encountered Error: [kv:1062]Duplicate entry

[yilongrong]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.

Cluster TiDB1

1. create table test( id int,name varchar(20));
2. insert into test values(1,'a'),(3,'b'));
3. tiup br:v5.4.0 backup table --pd "172.xx.x.138:16279" --db test --table test --storage "local:///tmp/backup" --log-file backup.log
4. alter table test add age int default 1;
5. update test set age=2 where id=1;
6. update test set age=3 where id=3;
7. alter table test add unique index indx(age);
8. tiup br:v5.4.0 backup table --pd "172.xx.xx.138:16279" --db test --table test --storage "local:///tmp/backupincr" --log-file backup.log --lastbackupts 434262749356228611

Cluster TiDB2

1. tiup br:v5.4.0 restore table --pd "172.xx.xx.136:16279" --db "test" --table "test" --storage "local:///tmp/backup"

2. tiup br:v5.4.0 restore table --pd "172.xx.xx.136:16279" --db "test" --table "test" --storage "local:///tmp/backupincr"

3. What did you expect to see?

restore incremental data success.

3. What did you see instead?

alter table test add unique index indx(age);
[2022/06/30 17:35:16.975 +08:00] [INFO] [collector.go:67] ["Table restore failed summary"] [total-ranges=0] [ranges-succeed=0] [ranges-failed=0]
Error: [kv:1062]Duplicate entry '1' for key 'indx'
Error: run /home/tidbryl/.tiup/components/br/v5.4.0/br (wd:/home/tidbryl/.tiup/data/TAES1t4) failed: exit status 1

4. What version of BR and TiDB/TiKV/PD are you using?

BR version:
Release Version: v5.4.0
Git Commit Hash: 55f3b24c1c9f506bd652ef1d162283541e428872
Git Branch: heads/refs/tags/v5.4.0
Go Version: go1.16.4
UTC Build Time: 2022-01-25 08:36:34

cluster version: v5.4.0

5. Operation logs

   • Please upload br.log for BR if possible
   • Please upload tidb-lightning.log for TiDB-Lightning if possible
   • Please upload tikv-importer.log from TiKV-Importer if possible
   • Other interesting logs

6. Configuration of the cluster and the task

   • tidb-lightning.toml for TiDB-Lightning if possible
   • tikv-importer.toml for TiKV-Importer if possible
   • topology.yml if deployed by TiUP

7. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus if possible

[WangLe1321]

Cloud you please add the sql statement of adding column age?

[yilongrong]

Done，Please check.

[yuqi1129]

This problem seems to affect all 4.x and 5.x and I'am not sure whether it has been fixed in 6.x

[yuqi1129]

The problem lies the logic: When restore, we always restore ddl first and then the table data, maybe we should rethink this logic again.

[WangLe1321]

/assign @WangLe1321