Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion docs/decisions/r276-lock-root-cause-fix.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# #276 근본 처방 — 재시도가 아니라 «중복 하나가 파티션 끝을 잠그는» 자리를 없앤다

작성: 2026-09-24
상태: **ㄴ(RC) 채택 · 적용 (2026-09-24 사용자 confirm)** — ㄷ(자연키 PK) 비용 측정 **착수·설계 확정(2026-11-08)**, rig 미작성 — §5
상태: **ㄴ(RC) 채택 · 적용 (2026-09-24 사용자 confirm)** — ㄷ(자연키 PK) 비용 **측정 완료(2026-11-09, EC2)** — §5-6. ㄴ 을 다시 열지는 미결
근거: [`loadtest/results/r276-lock-trace-2026-09-24/`](../../loadtest/results/r276-lock-trace-2026-09-24/README.md)
관련: [#276](https://github.com/Shadowfit/init/issues/276) · [`r276-retry-followup.md`](./r276-retry-followup.md) · [`pose-batch-idempotency-vs-partition.md`](./pose-batch-idempotency-vs-partition.md) · [`online-ddl-vs-blocking-alter.md`](./online-ddl-vs-blocking-alter.md)

Expand Down Expand Up @@ -129,6 +129,22 @@ RR 에서 **중복 키 한 건**이 `PRIMARY` 의 파티션 끝(supremum)에 `X`
3. **③ 을 이번에 같이 할지** — 추천: 같이(② 의 무대를 그대로 쓰면 추가 시딩이 없다)
4. **B′ 를 넣을지** — 추천: 넣는다(중복 인덱스 질문이 §2 ㄷ 표에 «검토 안 함» 으로 남아 있다)

### 5-6. 결과 (2026-11-09, EC2) — [`r276-natpk-aws-2026-11-08/`](../../loadtest/results/r276-natpk-aws-2026-11-08/README.md)

**상시 비용(삽입·읽기)은 거의 없거나 이득이고, 비용은 한 번 치르는 PK 재구성에 몰려 있다.**

| 단계 | 결과 |
|---|---|
| ① 삽입 | 4조건 중 3조건에서 B 가 **+15~22% 빠름**. 버퍼풀 초과 · 활성 675 에서만 B **−10%**(디스크 읽기 3.2배), B′ ±0. «행 전체가 흩어지면 크게 느려진다» 는 §5-2 가설은 120만~140만 행 무대에서 안 맞았다 |
| ② 재구성 (1,000만 행 · 약 21GB) | 서버가 INPLACE·LOCK=NONE 을 **받았다**. pt-osc **약 16분**(binlog +17GB) · 네이티브 INPLACE **약 100분**(binlog +0.85GB). 둘 다 쓰기 실패 0 · 최대 공백 1.2~1.35초. **네이티브는 기본 `innodb_online_alter_log_max_size`(128MiB)면 약 2시간 돌고 1799 로 매번 실패** — 2GiB 로 올려 통과(최소 필요값은 안 잼) |
| ③ 읽기 | 세션 단위 차이 없음(데움 p50 4.8~5.2ms). id 단건은 B 가 데움 +0.02~0.07ms |

§5 에 적은 것 중 틀린 것 둘:
- §5-4 «08-12 와 같은 시더·규모라 직접 비교된다» — **행 크기가 약 9배 다르다**(08-12 약 240B/행, 이번 2,119B/행). 시간을 나란히 놓으면 안 된다
- §5-3·§5-4 «A 는 `idx_session_timestamp` → PK 룩업» — 실제 계획은 **`uk_pose_event`** 를 탄다. `idx_session_timestamp` 는 A 에서도 이 조회에 안 쓰인다(B′ 에서 빼면 보조 인덱스 681→363MB)

§5-1 의 조건(«ㄷ 이 싸면 ㄴ 을 다시 연다»)이 걸린다. **다시 열지는 미결 — 사용자 결정.**

## 결정 로그

- 2026-09-24 — 문서 작성. 결정 없음
Expand All @@ -143,3 +159,4 @@ RR 에서 **중복 키 한 건**이 `PRIMARY` 의 파티션 끝(supremum)에 `X`
③ 같이 · B′ 넣음. 다음은 rig 작성 → 로컬 축소 스모크 → EC2 무인 라운드
- 2026-11-08 — **ㄴ 의 «모르는 것» 두 줄 닫음.** `findById` 등 같은 트랜잭션의 다른 문장은 코드 대조로 «보장 불변» 확인.
binlog 형식은 **기동 시 검사로 막기로 함 (사용자 confirm)** — `BinlogFormatStartupCheck` · 테스트 2벌(판정 · 실 MySQL 전제 1665)
- 2026-11-09 — **ㄷ 비용 측정 완료(EC2 3라운드).** 결과 §5-6. 결정 없음 — ㄴ 을 다시 열지는 사용자 confirm 대기
5 changes: 5 additions & 0 deletions loadtest/r276-natpk/_smoke/phase1/env.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# start 2026-11-08 04:38:23
host_cpu= Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz nproc=4
REGIMES=mem:256 ACTIVES=10 ARMS=A B Bp BLOCKS=2 TRIAL_ROWS=2000 WORKERS=15 SEED_SESSIONS=20
git=
mem mysql=8.0.46 pool=268435456
7 changes: 7 additions & 0 deletions loadtest/r276-natpk/_smoke/phase1/raw.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
regime pool_mb active block arm rows_before rows sec rows_per_sec batch_p50_ms batch_p99_ms errors bp_reads bp_read_req page_splits data_len index_len bytes_per_row
mem 256 10 0 A 12000 2000 2.294 871.9 194.43 392.22 0 75 21347 277 27918336 2228224 2153.3
mem 256 10 0 B 12000 2000 2.506 798.2 230.85 343.5 0 19 19330 247 26918912 2228224 2081.9
mem 256 10 0 Bp 12000 2000 3.563 561.3 318.45 537.99 0 0 14944 239 26902528 1654784 2039.8
mem 256 10 1 B 14000 2000 3.935 508.3 297.54 1113.05 0 8 19553 240 31113216 3309568 2151.4
mem 256 10 1 Bp 14000 2000 3.245 616.4 283.91 577.93 0 2 14971 233 32145408 1654784 2112.5
mem 256 10 1 A 14000 2000 1.912 1046.0 154.73 378.14 0 29 20977 267 33161216 3309568 2279.4
11 changes: 11 additions & 0 deletions loadtest/r276-natpk/_smoke/phase1/run.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[phase1 04:37:59] 클라이언트 기동
[phase1 04:38:23] MySQL 기동 — 버퍼풀 256MB
[phase1 04:40:12] == 체제 mem(풀 256MB) · 활성 세션 10 — 팔 표 새로 깔기(세션 20 개)
[phase1 04:40:56] MySQL 기동 — 버퍼풀 256MB
[phase1 04:41:55] 블록 0(버림) 팔 A — 871.9 행/초 · p99 392.22ms · 디스크읽기 75 · 분할 277 · 오류 0
[phase1 04:42:06] 블록 0(버림) 팔 B — 798.2 행/초 · p99 343.5ms · 디스크읽기 19 · 분할 247 · 오류 0
[phase1 04:42:23] 블록 0(버림) 팔 Bp — 561.3 행/초 · p99 537.99ms · 디스크읽기 0 · 분할 239 · 오류 0
[phase1 04:42:39] 블록 1 팔 B — 508.3 행/초 · p99 1113.05ms · 디스크읽기 8 · 분할 240 · 오류 0
[phase1 04:42:55] 블록 1 팔 Bp — 616.4 행/초 · p99 577.93ms · 디스크읽기 2 · 분할 233 · 오류 0
[phase1 04:43:03] 블록 1 팔 A — 1046.0 행/초 · p99 378.14ms · 디스크읽기 29 · 분할 267 · 오류 0
[phase1 04:43:03] 완료 → /e/init-wt-natpk/loadtest/r276-natpk/_smoke/phase1/raw.tsv
3 changes: 3 additions & 0 deletions loadtest/r276-natpk/_smoke/phase1/seed.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"table": "t_A", "sessions": 20, "rows": 12000, "sec": 10.9}
{"table": "t_B", "sessions": 20, "rows": 12000, "sec": 8.7}
{"table": "t_Bp", "sessions": 20, "rows": 12000, "sec": 9.3}
6 changes: 6 additions & 0 deletions loadtest/r276-natpk/_smoke/phase1/trials.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{"regime":"mem","active":10,"block":0,"arm":"A","insert":{"table": "t_A", "active": 10, "rows": 2000, "workers": 15, "sec": 2.294, "rows_per_sec": 871.9, "batch_p50_ms": 194.43, "batch_p99_ms": 392.22, "batches": 100, "errors": 0, "error_sample": [], "first_sid": 21, "last_sid": 30},"stats":{"table": "t_A", "rows": 14000, "data_length": 27918336, "index_length": 2228224, "bytes_per_row": 2153.3}}
{"regime":"mem","active":10,"block":0,"arm":"B","insert":{"table": "t_B", "active": 10, "rows": 2000, "workers": 15, "sec": 2.506, "rows_per_sec": 798.2, "batch_p50_ms": 230.85, "batch_p99_ms": 343.5, "batches": 100, "errors": 0, "error_sample": [], "first_sid": 21, "last_sid": 30},"stats":{"table": "t_B", "rows": 14000, "data_length": 26918912, "index_length": 2228224, "bytes_per_row": 2081.9}}
{"regime":"mem","active":10,"block":0,"arm":"Bp","insert":{"table": "t_Bp", "active": 10, "rows": 2000, "workers": 15, "sec": 3.563, "rows_per_sec": 561.3, "batch_p50_ms": 318.45, "batch_p99_ms": 537.99, "batches": 100, "errors": 0, "error_sample": [], "first_sid": 21, "last_sid": 30},"stats":{"table": "t_Bp", "rows": 14000, "data_length": 26902528, "index_length": 1654784, "bytes_per_row": 2039.8}}
{"regime":"mem","active":10,"block":1,"arm":"B","insert":{"table": "t_B", "active": 10, "rows": 2000, "workers": 15, "sec": 3.935, "rows_per_sec": 508.3, "batch_p50_ms": 297.54, "batch_p99_ms": 1113.05, "batches": 100, "errors": 0, "error_sample": [], "first_sid": 31, "last_sid": 40},"stats":{"table": "t_B", "rows": 16000, "data_length": 31113216, "index_length": 3309568, "bytes_per_row": 2151.4}}
{"regime":"mem","active":10,"block":1,"arm":"Bp","insert":{"table": "t_Bp", "active": 10, "rows": 2000, "workers": 15, "sec": 3.245, "rows_per_sec": 616.4, "batch_p50_ms": 283.91, "batch_p99_ms": 577.93, "batches": 100, "errors": 0, "error_sample": [], "first_sid": 31, "last_sid": 40},"stats":{"table": "t_Bp", "rows": 16000, "data_length": 32145408, "index_length": 1654784, "bytes_per_row": 2112.5}}
{"regime":"mem","active":10,"block":1,"arm":"A","insert":{"table": "t_A", "active": 10, "rows": 2000, "workers": 15, "sec": 1.912, "rows_per_sec": 1046.0, "batch_p50_ms": 154.73, "batch_p99_ms": 378.14, "batches": 100, "errors": 0, "error_sample": [], "first_sid": 31, "last_sid": 40},"stats":{"table": "t_A", "rows": 16000, "data_length": 33161216, "index_length": 3309568, "bytes_per_row": 2279.4}}
10 changes: 10 additions & 0 deletions loadtest/r276-natpk/_smoke/phase2.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[phase2 12:19:17] 기동 — 버퍼풀 512MB
[phase2 12:22:12] [0] tpl_A 시딩 — 세션 200 × 600 = 120000 행
{"table": "tpl_A", "sessions": 200, "rows": 120000, "sec": 127.3}
[phase2 12:24:27] [0] tpl_A 120000 행 · PK=id,created_at
[phase2 12:25:03] [1] ✅ 서버가 INPLACE·LOCK=NONE 을 받았다 — 팔 N 은 온라인 DDL 이다
[phase2 12:25:09] [2] 판 discard 팔 N — tpl_A 복사
[phase2 12:32:58] 팔 N — DDL 235.2초 · writer {"attempts": 633, "ok": 633, "errors": 0, "err_kinds": [], "max_attempt_ms": 21492, "p50_attempt_ms": 104.0, "max_gap_ms": 21700} · binlog +21MB · 디스크 최대 1622MB
[phase2 12:32:58] [2] 판 discard 팔 P — tpl_A 복사
[phase2 12:39:47] 팔 P — DDL 288.8초 · writer {"attempts": 440, "ok": 440, "errors": 0, "err_kinds": [], "max_attempt_ms": 4480, "p50_attempt_ms": 309.5, "max_gap_ms": 4684} · binlog +232MB · 디스크 최대 1604MB
[phase2 12:39:47] [2] 판 r1 팔 N — tpl_A 복사
1 change: 1 addition & 0 deletions loadtest/r276-natpk/_smoke/phase2/.disk_peak
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1626
32 changes: 32 additions & 0 deletions loadtest/r276-natpk/_smoke/phase2/discard_ddl.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
No slaves found. See --recursion-method if host 2d1042f78d17 has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
analyze_table, 10, 1
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `natpk`.`t_rb`...
Creating new table...
Created new table natpk._t_rb_new OK.
Altering new table...
Altered `natpk`.`_t_rb_new` OK.
2026-11-08T03:35:01 Creating triggers...
2026-11-08T03:35:02 Created triggers OK.
2026-11-08T03:35:02 Copying approximately 102475 rows...
Copying `natpk`.`t_rb`: 30% 01:08 remain
Copying `natpk`.`t_rb`: 46% 01:09 remain
Copying `natpk`.`t_rb`: 57% 01:06 remain
Copying `natpk`.`t_rb`: 70% 00:50 remain
Copying `natpk`.`t_rb`: 80% 00:35 remain
Copying `natpk`.`t_rb`: 89% 00:20 remain
2026-11-08T03:39:27 Copied rows OK.
2026-11-08T03:39:27 Analyzing new table...
2026-11-08T03:39:28 Swapping tables...
2026-11-08T03:39:32 Swapped original and new tables OK.
2026-11-08T03:39:32 Dropping old table...
2026-11-08T03:39:34 Dropped old table `natpk`.`_t_rb_old` OK.
2026-11-08T03:39:34 Dropping triggers...
2026-11-08T03:39:34 Dropped triggers OK.
Successfully altered `natpk`.`t_rb`.
5 changes: 5 additions & 0 deletions loadtest/r276-natpk/_smoke/phase2/env.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# start 2026-11-08 12:22:09
host_cpu= Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz nproc=4
mysql=8.0.46 pool=536870912 ptosc=percona/percona-toolkit:3.6.0
SESSIONS=200 SEQ=discard:N discard:P r1:N r2:P GAP_MS=200 git=
native=ALGORITHM=INPLACE, LOCK=NONE probe_pk=session_id,rep_number,timestamp_sec,created_at
65 changes: 65 additions & 0 deletions loadtest/r276-natpk/_smoke/phase2/mysql_stderr.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Empty file.
3 changes: 3 additions & 0 deletions loadtest/r276-natpk/_smoke/phase2/raw.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
tag arm ddl_sec attempts ok errors max_attempt_ms max_gap_ms binlog_mb disk_peak_mb pk_after rows_after
discard N 235.2 633 633 0 21492 21700 22 1622 session_id,rep_number,timestamp_sec,created_at 132660
discard P 288.8 440 440 0 4480 4684 233 1604 session_id,rep_number,timestamp_sec,created_at 128800
9 changes: 9 additions & 0 deletions loadtest/r276-natpk/_smoke/phase2/run.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[phase2 12:19:17] 기동 — 버퍼풀 512MB
[phase2 12:22:12] [0] tpl_A 시딩 — 세션 200 × 600 = 120000 행
[phase2 12:24:27] [0] tpl_A 120000 행 · PK=id,created_at
[phase2 12:25:03] [1] ✅ 서버가 INPLACE·LOCK=NONE 을 받았다 — 팔 N 은 온라인 DDL 이다
[phase2 12:25:09] [2] 판 discard 팔 N — tpl_A 복사
[phase2 12:32:58] 팔 N — DDL 235.2초 · writer {"attempts": 633, "ok": 633, "errors": 0, "err_kinds": [], "max_attempt_ms": 21492, "p50_attempt_ms": 104.0, "max_gap_ms": 21700} · binlog +21MB · 디스크 최대 1622MB
[phase2 12:32:58] [2] 판 discard 팔 P — tpl_A 복사
[phase2 12:39:47] 팔 P — DDL 288.8초 · writer {"attempts": 440, "ok": 440, "errors": 0, "err_kinds": [], "max_attempt_ms": 4480, "p50_attempt_ms": 309.5, "max_gap_ms": 4684} · binlog +232MB · 디스크 최대 1604MB
[phase2 12:39:47] [2] 판 r1 팔 N — tpl_A 복사
1 change: 1 addition & 0 deletions loadtest/r276-natpk/_smoke/phase2/seed.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"table": "tpl_A", "sessions": 200, "rows": 120000, "sec": 127.3}
1 change: 1 addition & 0 deletions loadtest/r276-natpk/_smoke/phase2/stats.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"table": "tpl_A", "rows": 120000, "data_length": 246333440, "index_length": 9601024, "bytes_per_row": 2132.8}
Loading
Loading