Skip to content
This repository was archived by the owner on Jul 24, 2024. It is now read-only.

Commit 194a4ff

Browse files
3pointerovervenus5kbperskennytmdisksing
authored
Merge master to Release 4.0 (#206)
* restore: merge tidb-tools/pkg/restore-util (#146) * restore-util: Implement split/scatter (#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (#299) * restore-util: add method to set placement rules and store labels (#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> * Fixed handling for a dbName that do not exist in the backup being restored (#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored * validate: fix debug meta test ci (#153) * validate: fix debug meta test ci * *: extracts runBackup/runRestore in cmd into pkg/task (#156) * *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta * restore: fix restore summary log (#150) Co-authored-by: kennytm <kennytm@gmail.com> * restore: enhance error handling (#152) * restore: enhance error handling Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix region epoch error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove `Restore*` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add debug log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix retry error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * handle RegionNotFound error Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Incremental BR: support DDL (#155) * support backup&restore ddl Signed-off-by: 5kbpers <tangminghua@pingcap.com> * integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update kvproto Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable fast checksum in incremental br Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix no valid key error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Reduce TiDB dependencies (#158) * utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <qdlc2010@gmail.com> * go.mod: update tidb (#168) Signed-off-by: Neil Shen <overvenus@gmail.com> * BR support TLS (#161) * *: support tls * move tikv.driver to glue * fix comments * upgrade golangci and prepare for go 1.14 (#171) Signed-off-by: Neil Shen <overvenus@gmail.com> * backup: add raw backup command (#101) * backup: add raw backup command * restore: speed up retry on not leader (#179) * tests: stable cluster start up Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: fix unbound var Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: speed retry on not leader Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: add --cacert flag Signed-off-by: Neil Shen <overvenus@gmail.com> * make codecov green Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * conn, restore: paginate scan regions (#165) * conn, restore: paginate scan regions Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: large timeout Signed-off-by: Neil Shen <overvenus@gmail.com> * Batch restore (#167) * *: unify Range and RangeTree Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: split restore files into small batch Signed-off-by: Neil Shen <overvenus@gmail.com> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: unused table worker pool Signed-off-by: Neil Shen <overvenus@gmail.com> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <overvenus@gmail.com> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <overvenus@gmail.com> * README, docker: add quick start (#181) * README, docker: add quick start Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: disable some TiDB log Signed-off-by: Neil Shen <overvenus@gmail.com> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <overvenus@gmail.com> * *: update tidb dependency build with go1.14 (#176) * *: add license header (#182) * rtree: move checkFile into backup Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add license header Signed-off-by: Neil Shen <overvenus@gmail.com> * Update LICENSE.md Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: support not shutting down the storage when closing the connection (#185) Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: use GetDomain to avoid some TiDB breaking changes (#186) * conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <overvenus@gmail.com> * minor usability improvement Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * fix check safepoint & unhide experimental features (#175) * backup: check safepoint for last backup ts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check lastbackupts > 0 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unhide experimental features Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * support backupts (#172) * support backupts * address comment * address comment * fix space * *: update pd deps to v4 (#184) Co-authored-by: 3pointer <luancheng@pingcap.com> * restore: support online restore (#114) Signed-off-by: disksing <i@disksing.com> * metrics: add grafana scripts (#140) * add grafana scripts * fix Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: kennytm <kennytm@gmail.com> * filter out all TiFlash nodes when retrieving lists of stores from PD (#187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup * Create integration test for S3 storage (#174) * Fix summary log (#191) * *: fix restore summary log after restore logic changed to files * fix * fix * fix Co-authored-by: kennytm <kennytm@gmail.com> * Implement Raw Restore (#104) * Update kvproto * Implement raw restore * fix build * Set range for file importer Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Remove unnecessary comments Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * check cf and support multi ranges in BackupMeta Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Check files' cf; address comments * adjust structure to keep consistent with master * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build and make check, avoid accessing TiDB in rawkv mode * Fix test Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix tests Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix broken logic after merging master * Update pkg/task/restore_raw.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Address comments * Address comments * Mark raw restore as experimental * Fix build * Address comments * test: Add check for deleting data and partial backup * Fix build * Add license header * fix ci * fix ci Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> * restore: remove tiflash replica before restore (#194) * restore: remove tiflash replica before restore Signed-off-by: 5kbpers <tangminghua@pingcap.com> * rename errSplit variable Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check replica count by region info Signed-off-by: 5kbpers <tangminghua@pingcap.com> * cleanup Signed-off-by: 5kbpers <tangminghua@pingcap.com> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix save crcxor Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix decode the key of placement rule Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/task/restore.go Co-Authored-By: 3pointer <qdlc2010@gmail.com> Co-authored-by: 3pointer <qdlc2010@gmail.com> * summary: put summary log at last (#197) * summary: put summary log at last * fix switch sql * *: abstract the progress channel (updateCh) into the glue package (#196) * *: abstract the progress channel (updateCh) into the glue package * restore: fix crash in truncateTS() when the bound is unlimited * task: fix comment Co-authored-by: Ian <ArGregoryIan@gmail.com> * *: refline logs (#189) * tests: disable TLS test (#204) Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add S3 quick start and few enhancement of log (#202) * README, docker: add quick start about S3 storage Signed-off-by: Neil Shen <overvenus@gmail.com> * pkg/summary: make sure to output correct summary Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd, tests: log to terminal if BR_LOG_TO_TERM is set Signed-off-by: Neil Shen <overvenus@gmail.com> * Update pkg/task/common.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: cat log if br fails Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * restore: add error field to `DownloadResponse` (#195) * restore: add error field to `DownloadResponse` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore: populate restore cancel error (#207) Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * enhance usability of br (#208) * silenceUsage only when parse cmd flags failed * udpate tidb Co-authored-by: kennytm <kennytm@gmail.com> * task: do not run checksum if restore failed (#209) * fix incremental bug in llroad test (#199) * restore: filter same table ddl * *: do not return error when backup/restore data is empty * fix create database double during incremental restore * add tests * fix ci * address comment * add skip create sqls (#211) * Revert "tests: disable TLS test (#204)" (#218) This reverts commit e168a60. * doc: add `minio` to dependence list. (#221) The README of test omitted `minio` in the dependence list, which is needed for run the integration test. Co-authored-by: Neil Shen <overvenus@gmail.com> * move waiting reject stores in import file (#222) * move wait rejectstores into import files * restore: use new table id to search placementRules * Update pkg/restore/import.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix ci Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Max index length (#220) * restore: set max-index-length to max * restore:add max-index-length params * address comment * address comment * glue: create schema/table directly with info (#216) * glue: create schema/table directly with info * go.mod: change to use the master version * gluetidb: fix failure to create schema * gluetidb: exclude non-public indices when restoring * go.mod: removed unused replace Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Kolbe Kegel <kolbe@pingcap.com> Co-authored-by: WangXiangUSTC <wx347249478@gmail.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> Co-authored-by: 庄天翼 <zty0826@gmail.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
1 parent 3239e7d commit 194a4ff

46 files changed

Lines changed: 836 additions & 331 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ go-ycsb load mysql -p workload=core \
5656
mysql -uroot -htidb -P4000 -E -e "SELECT COUNT(*) FROM test.usertable"
5757

5858
# Build BR and backup!
59-
make release && \
59+
make build && \
6060
bin/br backup full --pd pd0:2379 --storage "local:///data/backup/full" \
6161
--log-file "/logs/br_backup.log"
6262

@@ -69,6 +69,20 @@ bin/br restore full --pd pd0:2379 --storage "local:///data/backup/full" \
6969

7070
# How many rows do we get again? Expected to be 100000 rows.
7171
mysql -uroot -htidb -P4000 -E -e "SELECT COUNT(*) FROM test.usertable"
72+
73+
# Test S3 compatible storage (MinIO).
74+
# Create a bucket to save backup by mc (a MinIO Client).
75+
mc config host add minio $S3_ENDPOINT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY && \
76+
mc mb minio/mybucket
77+
78+
# Backup to S3 compatible storage.
79+
bin/br backup full --pd pd0:2379 --storage "s3://mybucket/full" \
80+
--s3.endpoint="$S3_ENDPOINT"
81+
82+
# Drop database and restore!
83+
mysql -uroot -htidb -P4000 -E -e "DROP DATABASE test; SHOW DATABASES;" && \
84+
bin/br restore full --pd pd0:2379 --storage "s3://mybucket/full" \
85+
--s3.endpoint="$S3_ENDPOINT"
7286
```
7387

7488
## Contributing

cmd/backup.go

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import (
1616
func runBackupCommand(command *cobra.Command, cmdName string) error {
1717
cfg := task.BackupConfig{Config: task.Config{LogProgress: HasLogFile()}}
1818
if err := cfg.ParseFromFlags(command.Flags()); err != nil {
19+
command.SilenceUsage = false
1920
return err
2021
}
2122
return task.RunBackup(GetDefaultContext(), tidbGlue, cmdName, &cfg)
@@ -24,6 +25,7 @@ func runBackupCommand(command *cobra.Command, cmdName string) error {
2425
func runBackupRawCommand(command *cobra.Command, cmdName string) error {
2526
cfg := task.RawKvConfig{Config: task.Config{LogProgress: HasLogFile()}}
2627
if err := cfg.ParseFromFlags(command.Flags()); err != nil {
28+
command.SilenceUsage = false
2729
return err
2830
}
2931
return task.RunBackupRaw(GetDefaultContext(), gluetikv.Glue{}, cmdName, &cfg)
@@ -34,7 +36,7 @@ func NewBackupCommand() *cobra.Command {
3436
command := &cobra.Command{
3537
Use: "backup",
3638
Short: "backup a TiDB/TiKV cluster",
37-
SilenceUsage: false,
39+
SilenceUsage: true,
3840
PersistentPreRunE: func(c *cobra.Command, args []string) error {
3941
if err := Init(c); err != nil {
4042
return err

cmd/cmd.go

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,11 @@ import (
66
"context"
77
"net/http"
88
"net/http/pprof"
9+
"os"
10+
"path/filepath"
911
"sync"
1012
"sync/atomic"
13+
"time"
1114

1215
"github.com/pingcap/log"
1316
"github.com/pingcap/tidb/util/logutil"
@@ -16,15 +19,17 @@ import (
1619
"go.uber.org/zap"
1720

1821
"github.com/pingcap/br/pkg/gluetidb"
22+
"github.com/pingcap/br/pkg/summary"
1923
"github.com/pingcap/br/pkg/task"
2024
"github.com/pingcap/br/pkg/utils"
2125
)
2226

2327
var (
24-
initOnce = sync.Once{}
25-
defaultContext context.Context
26-
hasLogFile uint64
27-
tidbGlue = gluetidb.Glue{}
28+
initOnce = sync.Once{}
29+
defaultContext context.Context
30+
hasLogFile uint64
31+
tidbGlue = gluetidb.Glue{}
32+
envLogToTermKey = "BR_LOG_TO_TERM"
2833
)
2934

3035
const (
@@ -41,6 +46,10 @@ const (
4146
flagVersionShort = "V"
4247
)
4348

49+
func timestampLogFileName() string {
50+
return filepath.Join(os.TempDir(), "br.log."+time.Now().Format(time.RFC3339))
51+
}
52+
4453
// AddFlags adds flags to the given cmd.
4554
func AddFlags(cmd *cobra.Command) {
4655
cmd.Version = utils.BRInfo()
@@ -49,8 +58,8 @@ func AddFlags(cmd *cobra.Command) {
4958

5059
cmd.PersistentFlags().StringP(FlagLogLevel, "L", "info",
5160
"Set the log level")
52-
cmd.PersistentFlags().String(FlagLogFile, "",
53-
"Set the log file path. If not set, logs will output to stdout")
61+
cmd.PersistentFlags().String(FlagLogFile, timestampLogFileName(),
62+
"Set the log file path. If not set, logs will output to temp file")
5463
cmd.PersistentFlags().String(FlagStatusAddr, "",
5564
"Set the HTTP listening address for the status report service. Set to empty string to disable")
5665
task.DefineCommonFlags(cmd.PersistentFlags())
@@ -73,8 +82,15 @@ func Init(cmd *cobra.Command) (err error) {
7382
if err != nil {
7483
return
7584
}
85+
_, outputLogToTerm := os.LookupEnv(envLogToTermKey)
86+
if outputLogToTerm {
87+
// Log to term if env `BR_LOG_TO_TERM` is set.
88+
conf.File.Filename = ""
89+
}
7690
if len(conf.File.Filename) != 0 {
7791
atomic.StoreUint64(&hasLogFile, 1)
92+
summary.InitCollector(true)
93+
cmd.Printf("Detial BR log in %s\n", conf.File.Filename)
7894
}
7995
lg, p, e := log.InitLogger(conf)
8096
if e != nil {

cmd/restore.go

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import (
1515
func runRestoreCommand(command *cobra.Command, cmdName string) error {
1616
cfg := task.RestoreConfig{Config: task.Config{LogProgress: HasLogFile()}}
1717
if err := cfg.ParseFromFlags(command.Flags()); err != nil {
18+
command.SilenceUsage = false
1819
return err
1920
}
2021
return task.RunRestore(GetDefaultContext(), tidbGlue, cmdName, &cfg)
@@ -25,6 +26,7 @@ func runRestoreRawCommand(command *cobra.Command, cmdName string) error {
2526
RawKvConfig: task.RawKvConfig{Config: task.Config{LogProgress: HasLogFile()}},
2627
}
2728
if err := cfg.ParseFromFlags(command.Flags()); err != nil {
29+
command.SilenceUsage = false
2830
return err
2931
}
3032
return task.RunRestoreRaw(GetDefaultContext(), gluetikv.Glue{}, cmdName, &cfg)
@@ -33,6 +35,7 @@ func runRestoreRawCommand(command *cobra.Command, cmdName string) error {
3335
func runRestoreTiflashReplicaCommand(command *cobra.Command, cmdName string) error {
3436
cfg := task.RestoreConfig{Config: task.Config{LogProgress: HasLogFile()}}
3537
if err := cfg.ParseFromFlags(command.Flags()); err != nil {
38+
command.SilenceUsage = false
3639
return err
3740
}
3841

@@ -44,7 +47,7 @@ func NewRestoreCommand() *cobra.Command {
4447
command := &cobra.Command{
4548
Use: "restore",
4649
Short: "restore a TiDB/TiKV cluster",
47-
SilenceUsage: false,
50+
SilenceUsage: true,
4851
PersistentPreRunE: func(c *cobra.Command, args []string) error {
4952
if err := Init(c); err != nil {
5053
return err

docker-compose.yaml

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
# Source: tidb-docker-compose/templates/docker-compose.yml
3-
version: '2.1'
3+
version: '3.2'
44

55
services:
66
control:
@@ -10,11 +10,13 @@ services:
1010
dockerfile: ./docker/Dockerfile
1111
volumes:
1212
- ./docker/data:/data
13-
- ./docker/logs:/logs
13+
- ./docker/logs:/tmp
1414
command: -c "/usr/bin/tail -f /dev/null"
1515
depends_on:
1616
- "tidb"
1717
restart: on-failure
18+
env_file:
19+
- ./docker/minio.env
1820

1921
pd0:
2022
image: pingcap/pd:latest
@@ -64,6 +66,8 @@ services:
6466
# soft: 1000000
6567
# hard: 1000000
6668
restart: on-failure
69+
env_file:
70+
- ./docker/minio.env
6771

6872
tikv1:
6973
image: pingcap/tikv:latest
@@ -87,6 +91,8 @@ services:
8791
# soft: 1000000
8892
# hard: 1000000
8993
restart: on-failure
94+
env_file:
95+
- ./docker/minio.env
9096

9197
tikv2:
9298
image: pingcap/tikv:latest
@@ -110,6 +116,8 @@ services:
110116
# soft: 1000000
111117
# hard: 1000000
112118
restart: on-failure
119+
env_file:
120+
- ./docker/minio.env
113121

114122
tikv3:
115123
image: pingcap/tikv:latest
@@ -133,6 +141,8 @@ services:
133141
# soft: 1000000
134142
# hard: 1000000
135143
restart: on-failure
144+
env_file:
145+
- ./docker/minio.env
136146

137147
tikv4:
138148
image: pingcap/tikv:latest
@@ -156,6 +166,8 @@ services:
156166
# soft: 1000000
157167
# hard: 1000000
158168
restart: on-failure
169+
env_file:
170+
- ./docker/minio.env
159171

160172
tidb:
161173
image: pingcap/tidb:latest
@@ -185,6 +197,16 @@ services:
185197
# hard: 1000000
186198
restart: on-failure
187199

200+
minio:
201+
image: minio/minio
202+
ports:
203+
- 24927:24927
204+
volumes:
205+
- ./docker/data/s3:/data/s3
206+
command: server --address=:24927 /data/s3
207+
env_file:
208+
- ./docker/minio.env
209+
188210
tidb-vision:
189211
image: pingcap/tidb-vision:latest
190212
environment:

docker/Dockerfile

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
1-
FROM golang:1.13.8-buster as builder
2-
31
# For loading data to TiDB
2+
FROM golang:1.13.8-buster as go-ycsb-builder
43
WORKDIR /go/src/github.com/pingcap/
54
RUN git clone https://github.com/pingcap/go-ycsb.git && \
65
cd go-ycsb && \
76
make
87

8+
# For operating minio S3 compatible storage
9+
FROM minio/mc as mc-builder
10+
911
FROM golang:1.13.8-buster
1012

1113
RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -19,6 +21,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
1921
WORKDIR /go/src/github.com/pingcap/br
2022
COPY . .
2123

22-
COPY --from=builder /go/src/github.com/pingcap/go-ycsb/bin/go-ycsb /go/bin/go-ycsb
24+
COPY --from=go-ycsb-builder /go/src/github.com/pingcap/go-ycsb/bin/go-ycsb /go/bin/go-ycsb
25+
COPY --from=mc-builder /usr/bin/mc /usr/bin/mc
2326

2427
ENTRYPOINT ["/bin/bash"]

docker/minio.env

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
MINIO_ACCESS_KEY=brs3accesskey
2+
MINIO_SECRET_KEY=brs3secretkey
3+
MINIO_BROWSER=off
4+
AWS_ACCESS_KEY_ID=brs3accesskey
5+
AWS_SECRET_ACCESS_KEY=brs3secretkey
6+
S3_ENDPOINT=http://minio:24927

go.mod

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,11 @@ require (
2121
github.com/onsi/gomega v1.8.1 // indirect
2222
github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712
2323
github.com/pingcap/errors v0.11.5-0.20190809092503-95897b64e011
24-
github.com/pingcap/kvproto v0.0.0-20200317112120-78042b285b75
24+
github.com/pingcap/kvproto v0.0.0-20200330093347-98f910b71904
2525
github.com/pingcap/log v0.0.0-20200117041106-d28c14d3b1cd
26-
github.com/pingcap/parser v0.0.0-20200305120128-bde9faa0df84
26+
github.com/pingcap/parser v0.0.0-20200326020624-68d423641be5
2727
github.com/pingcap/pd/v4 v4.0.0-beta.1.0.20200305072537-61d9f9cc35d3
28-
github.com/pingcap/tidb v1.1.0-beta.0.20200310133602-7c39e5e5e0bc
28+
github.com/pingcap/tidb v0.0.0-20200401141416-959eca8f3a39
2929
github.com/pingcap/tidb-tools v4.0.0-beta.1.0.20200306084441-875bd09aa3d5+incompatible
3030
github.com/pingcap/tipb v0.0.0-20200212061130-c4d518eb1d60
3131
github.com/prometheus/client_golang v1.0.0
@@ -37,7 +37,7 @@ require (
3737
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5 // indirect
3838
go.etcd.io/etcd v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738
3939
go.opencensus.io v0.22.2 // indirect
40-
go.uber.org/zap v1.14.0
40+
go.uber.org/zap v1.14.1
4141
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
4242
google.golang.org/api v0.14.0
4343
google.golang.org/grpc v1.25.1

go.sum

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -363,25 +363,23 @@ github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989/go.mod h1:O17Xtb
363363
github.com/pingcap/kvproto v0.0.0-20191211054548-3c6b38ea5107/go.mod h1:WWLmULLO7l8IOcQG+t+ItJ3fEcrL5FxF0Wu+HrMy26w=
364364
github.com/pingcap/kvproto v0.0.0-20200214064158-62d31900d88e/go.mod h1:IOdRDPLyda8GX2hE/jO7gqaCV/PNFh8BZQCQZXfIOqI=
365365
github.com/pingcap/kvproto v0.0.0-20200221034943-a2aa1d1e20a8/go.mod h1:IOdRDPLyda8GX2hE/jO7gqaCV/PNFh8BZQCQZXfIOqI=
366-
github.com/pingcap/kvproto v0.0.0-20200228095611-2cf9a243b8d5 h1:knEvP4R5v5b2T107/Q6VzB0C8/6T7NXB/V7Vl1FtQsg=
367-
github.com/pingcap/kvproto v0.0.0-20200228095611-2cf9a243b8d5/go.mod h1:IOdRDPLyda8GX2hE/jO7gqaCV/PNFh8BZQCQZXfIOqI=
368-
github.com/pingcap/kvproto v0.0.0-20200317112120-78042b285b75 h1:DB3NTM0ilba/6sW+vccdEnP10bVvrVunDwWvRa0hSKc=
369-
github.com/pingcap/kvproto v0.0.0-20200317112120-78042b285b75/go.mod h1:IOdRDPLyda8GX2hE/jO7gqaCV/PNFh8BZQCQZXfIOqI=
366+
github.com/pingcap/kvproto v0.0.0-20200330093347-98f910b71904 h1:pMFUXvhJ62hX8m0Q4RsL7L+hSW1mAMG26So5eFMoAtI=
367+
github.com/pingcap/kvproto v0.0.0-20200330093347-98f910b71904/go.mod h1:IOdRDPLyda8GX2hE/jO7gqaCV/PNFh8BZQCQZXfIOqI=
370368
github.com/pingcap/log v0.0.0-20191012051959-b742a5d432e9 h1:AJD9pZYm72vMgPcQDww9rkZ1DnWfl0pXV3BOWlkYIjA=
371369
github.com/pingcap/log v0.0.0-20191012051959-b742a5d432e9/go.mod h1:4rbK1p9ILyIfb6hU7OG2CiWSqMXnp3JMbiaVJ6mvoY8=
372370
github.com/pingcap/log v0.0.0-20200117041106-d28c14d3b1cd h1:CV3VsP3Z02MVtdpTMfEgRJ4T9NGgGTxdHpJerent7rM=
373371
github.com/pingcap/log v0.0.0-20200117041106-d28c14d3b1cd/go.mod h1:4rbK1p9ILyIfb6hU7OG2CiWSqMXnp3JMbiaVJ6mvoY8=
374-
github.com/pingcap/parser v0.0.0-20200305120128-bde9faa0df84 h1:u5FOwUw9muF8mBTZVV1dQhoAKiEo2Ci54CxN9XchEEY=
375-
github.com/pingcap/parser v0.0.0-20200305120128-bde9faa0df84/go.mod h1:9v0Edh8IbgjGYW2ArJr19E+bvL8zKahsFp+ixWeId+4=
372+
github.com/pingcap/parser v0.0.0-20200326020624-68d423641be5 h1:fXVqoeYfV+xI8K2he5NNv00c6YksrjeM6+vkNo1ZK2Q=
373+
github.com/pingcap/parser v0.0.0-20200326020624-68d423641be5/go.mod h1:9v0Edh8IbgjGYW2ArJr19E+bvL8zKahsFp+ixWeId+4=
376374
github.com/pingcap/pd/v4 v4.0.0-beta.1.0.20200305072537-61d9f9cc35d3 h1:Yrp99FnjHAEuDrSBql2l0IqCtJX7KwJbTsD5hIArkvk=
377375
github.com/pingcap/pd/v4 v4.0.0-beta.1.0.20200305072537-61d9f9cc35d3/go.mod h1:25GfNw6+Jcr9kca5rtmTb4gKCJ4jOpow2zV2S9Dgafs=
378376
github.com/pingcap/sysutil v0.0.0-20200206130906-2bfa6dc40bcd/go.mod h1:EB/852NMQ+aRKioCpToQ94Wl7fktV+FNnxf3CX/TTXI=
379377
github.com/pingcap/sysutil v0.0.0-20200302022240-21c8c70d0ab1 h1:YUnUZ914SHFMsOSe/xgH5DKK/thtRma8X8hcszRo3CA=
380378
github.com/pingcap/sysutil v0.0.0-20200302022240-21c8c70d0ab1/go.mod h1:EB/852NMQ+aRKioCpToQ94Wl7fktV+FNnxf3CX/TTXI=
381379
github.com/pingcap/sysutil v0.0.0-20200309085538-962fd285f3bb h1:bDbgLaNTRNK6Qw7KjvEqqfCQstY8WMEcXyXTU7yzYKg=
382380
github.com/pingcap/sysutil v0.0.0-20200309085538-962fd285f3bb/go.mod h1:EB/852NMQ+aRKioCpToQ94Wl7fktV+FNnxf3CX/TTXI=
383-
github.com/pingcap/tidb v1.1.0-beta.0.20200310133602-7c39e5e5e0bc h1:1aW3qTRJZjnosvXt1b75KL73b28XRJWBx6jtTtHsybg=
384-
github.com/pingcap/tidb v1.1.0-beta.0.20200310133602-7c39e5e5e0bc/go.mod h1:WTmfs5zrUGMPw3Enn5FI3buzkU8BDuJ6BhsO/JC239U=
381+
github.com/pingcap/tidb v0.0.0-20200401141416-959eca8f3a39 h1:nYRL69Qc4kuvp+tlDNB5wXjvDetX0J7g0DsW4RQxfXM=
382+
github.com/pingcap/tidb v0.0.0-20200401141416-959eca8f3a39/go.mod h1:btnHsqUQvJnY18+OP2Z6MCRq1tX4B8JUCrmqctSKxOg=
385383
github.com/pingcap/tidb-tools v4.0.0-beta.1.0.20200306084441-875bd09aa3d5+incompatible h1:84F7MFMfdAYObrznvRslmVu43aoihrlL+7mMyMlOi0o=
386384
github.com/pingcap/tidb-tools v4.0.0-beta.1.0.20200306084441-875bd09aa3d5+incompatible/go.mod h1:XGdcy9+yqlDSEMTpOXnwf3hiTeqrV6MN/u1se9N8yIM=
387385
github.com/pingcap/tipb v0.0.0-20190428032612-535e1abaa330/go.mod h1:RtkHW8WbcNxj8lsbzjaILci01CtYnYbIkQhjyZWrWVI=
@@ -501,6 +499,7 @@ github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q
501499
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
502500
github.com/yookoala/realpath v1.0.0 h1:7OA9pj4FZd+oZDsyvXWQvjn5oBdcHRTV44PpdMSuImQ=
503501
github.com/yookoala/realpath v1.0.0/go.mod h1:gJJMA9wuX7AcqLy1+ffPatSCySA1FQ2S8Ya9AIoYBpE=
502+
github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
504503
go.etcd.io/bbolt v1.3.3 h1:MUGmc65QhB3pIlaQ5bB4LwqSj6GIonVJXpZiaKNyaKk=
505504
go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
506505
go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738/go.mod h1:dnLIgRNXwCJa5e+c6mIZCrds/GIG4ncV9HhK5PX7jPg=
@@ -535,8 +534,8 @@ go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
535534
go.uber.org/zap v1.12.0 h1:dySoUQPFBGj6xwjmBzageVL8jGi8uxc6bEmJQjA06bw=
536535
go.uber.org/zap v1.12.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM=
537536
go.uber.org/zap v1.13.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM=
538-
go.uber.org/zap v1.14.0 h1:/pduUoebOeeJzTDFuoMgC6nRkiasr1sBCIEorly7m4o=
539-
go.uber.org/zap v1.14.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM=
537+
go.uber.org/zap v1.14.1 h1:nYDKopTbvAPq/NrUVZwT15y2lpROBiLLyoRTbXOYWOo=
538+
go.uber.org/zap v1.14.1/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc=
540539
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
541540
golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
542541
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
@@ -676,8 +675,8 @@ golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtn
676675
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
677676
golang.org/x/tools v0.0.0-20200301222351-066e0c02454c h1:FD7jysxM+EJqg5UYYy3XYDsAiUickFsn4UiaanJkf8c=
678677
golang.org/x/tools v0.0.0-20200301222351-066e0c02454c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
679-
golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb h1:iKlO7ROJc6SttHKlxzwGytRtBUqX4VARrNTgP2YLX5M=
680-
golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw=
678+
golang.org/x/tools v0.0.0-20200325203130-f53864d0dba1 h1:odiryKYJy7CjdrZxhrcE1Z8L9+kGyGZOnfpuauvdCeU=
679+
golang.org/x/tools v0.0.0-20200325203130-f53864d0dba1/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8=
681680
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
682681
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
683682
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=

main.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ func main() {
4141
Use: "br",
4242
Short: "br is a TiDB/TiKV cluster backup restore tool.",
4343
TraverseChildren: true,
44-
SilenceUsage: false,
44+
SilenceUsage: true,
4545
}
4646
cmd.AddFlags(rootCmd)
4747
cmd.SetDefaultContext(ctx)

0 commit comments

Comments
 (0)