Skip to content

Commit 51159ba

Browse files
committed
feat(v2.3): security hardening, multi-algorithm watermark, architecture upgrade
Security: - Fix admin sub-role enforcement (admin1/admin2/admin3 role-specific decorators) - Fix user impersonation vulnerability (JWT identity instead of request body) - Fix frontend admin_sub_role bug in login.vue and router guard - Fix QR_SECRET_KEY crash guard (None.encode() when env var unset) - Activate per-user rate limiting on 7+ key endpoints Features: - Application withdrawal endpoint (PUT /api/applications/{id}/withdraw) - 3 raster watermark algorithms: LSB, DWT, Histogram Shifting - GeoTIFF CRS/transform preservation via rasterio - Algorithm quality metrics (PSNR, SSIM, NC) - Grafana alerting rules (error rate, latency, DB, cache) - Loki + Promtail log aggregation - Socket.IO real-time chat with HTTP polling fallback - Centralized API service layer (10 modules, all 49 views migrated) Cleanup: - Fix destructive is_multiple.py (now saves to _single.shp, never overwrites) - Remove hardcoded paths in embed.py and is_multiple.py - Remove commented-out print() debug statements - Delete 13 unused stub/model files - Add dev key warning in DevelopmentConfig Tests: 140 tests (138 passed, 2 skipped) Docs: Updated README.md and CHANGELOG.md with full v2.3 documentation
1 parent dbb4f18 commit 51159ba

111 files changed

Lines changed: 6379 additions & 1654 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CHANGELOG.md

Lines changed: 187 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,188 @@ All notable changes to this project are documented in this file.
44

55
---
66

7+
## [v2.3.0] — 2026-05-09 (Security Hardening, Multi-Algorithm Watermark & Monitoring)
8+
9+
### Summary
10+
Fixed critical security vulnerabilities (user impersonation, missing role enforcement), added admin sub-role differentiation (admin1/admin2/admin3), implemented 3 raster watermark algorithms (LSB, DWT, Histogram Shifting) with GeoTIFF CRS preservation, added application withdrawal lifecycle, centralized the frontend API service layer, upgraded chat to Socket.IO, and added Grafana alerting + Loki log aggregation.
11+
12+
**Total changes:** 40+ files modified/added
13+
14+
---
15+
16+
### Backend — Security Fixes
17+
18+
#### 1. Admin Sub-role Enforcement (`utils/required.py`, resource files)
19+
- **What:** `admin_role_required(*roles)` decorator enforces specific admin sub-roles (admin1/admin2/admin3)
20+
- **Applied to:**
21+
- `Adm1PassResource` / `Adm1FailResource``@admin_role_required('admin1')`
22+
- `Adm2PassResource` / `Adm2FailResource``@admin_role_required('admin2')`
23+
- `Adm3AdditionalReviewResource``@admin_role_required('admin3')`
24+
- **Helper:** `is_admin_role(role)` replaces all hardcoded `== 'admin'` / `!= 'admin'` checks across 6 resource files
25+
- **Files changed:** `application_resource.py`, `admin_application_resource.py`, `dashboard_resource.py`, `collaboration_resource.py`, `recall_resource.py`
26+
27+
#### 2. User Impersonation Fix (`resource/application_resource.py`, `download_file_resource.py`)
28+
- **What:** `user_name` and `user_number` now derived from `get_jwt_identity()` instead of request body
29+
- **Endpoints fixed:** `SubmitApplicationResource`, `get_applications`, `get_approved_applications`, `RecordDownloadResource`
30+
31+
#### 3. Input Validation
32+
- Registration: password complexity (8+ chars, letters + digits)
33+
- Profile update: email format validation
34+
- Password change: old password verification required
35+
- Chat messages: 2000-character limit
36+
37+
---
38+
39+
### Backend — New Features
40+
41+
#### 4. Application Withdrawal (`resource/application_resource.py`)
42+
- **Endpoint:** `PUT /api/applications/<id>/withdraw`
43+
- **Rules:** Only own applications, only before adm1 review starts
44+
- **Side effects:** Audit log entry + dashboard cache invalidation
45+
- **Registered in:** `app.py`
46+
47+
#### 5. Multi-Algorithm Raster Watermark (`algorithm/`, `resource/watermark_resource.py`, `resource/raster_resource.py`)
48+
- **LSB** (`raster_reversible_watermark.py`) — Least Significant Bit embedding
49+
- **DWT** (`raster_dwt_watermark.py`) — Discrete Wavelet Transform embedding
50+
- **Histogram Shifting** (`raster_histogram_watermark.py`) — Histogram-based reversible watermarking
51+
- **GeoTIFF preservation** (`raster_geotiff_utils.py`) — CRS and affine transform preserved through embed/extract
52+
- **Dispatch:** Both `watermark_resource.py` and `raster_resource.py` route by `algorithm` parameter
53+
54+
#### 6. Algorithm Quality Metrics (`algorithm/quality_metrics.py`)
55+
- PSNR, SSIM, NC (Normalized Correlation) computation for watermark quality assessment
56+
57+
---
58+
59+
### Backend — Test Suite Expansion
60+
61+
#### 7. Robustness Tests (`tests/test_robustness.py`) — 6 new tests
62+
| Test | Description |
63+
|------|-------------|
64+
| `test_lsb_baseline_roundtrip` | Embed → extract → NC ≥ 0.99 |
65+
| `test_jpeg_compression_robustness` | NC after JPEG quality 50 |
66+
| `test_gaussian_noise_robustness` | NC after σ=10 noise |
67+
| `test_crop_robustness` | NC after 10% crop |
68+
| `test_watermark_recovery` | Recover original from watermarked |
69+
| `test_vector_roundtrip` | Full vector embed → extract pipeline |
70+
71+
#### 8. Admin Workflow Tests Updated (`tests/test_approval_workflow.py`, `tests/test_integration.py`)
72+
- Admin2 tests use dedicated `adm2_headers` fixture (role='admin2')
73+
- `conftest.py` seeds admin1 (role='admin1') + admin2 (role='admin2')
74+
75+
**Test count:** 140 tests (up from 115), 138 passed, 2 skipped
76+
77+
---
78+
79+
### Backend — Monitoring & Logging
80+
81+
#### 9. Grafana Alerting Rules (`grafana/provisioning/alerting/alerts.yml`)
82+
- Error rate > 5% for 5min → alert
83+
- p95 latency > 2s for 5min → alert
84+
- Database connection failure → immediate alert
85+
- Cache hit rate < 50% → alert
86+
87+
#### 10. Loki Log Aggregation
88+
- **Loki config:** `loki-config.yml` — local filesystem storage
89+
- **Promtail config:** `promtail-config.yml` — scrapes `testrealend/logs/`
90+
- **Grafana dashboard:** `grafana/provisioning/dashboards/json/logs.json` — log volume + error panels
91+
- **Docker Compose:** Added `loki` and `promtail` services (total: 9 services)
92+
- **Datasource:** `grafana/provisioning/datasources/datasource.yml` — added Loki
93+
94+
---
95+
96+
### Frontend — Architecture
97+
98+
#### 11. Centralized API Service Layer (`src/api/`)
99+
| Module | File | Endpoints |
100+
|--------|------|-----------|
101+
| Auth | `auth.js` | login, register, refresh, logout |
102+
| Admin | `admin.js` | dashboard, employees, approval, batch review, logs |
103+
| Employee | `employee.js` | applications, profile, notifications, downloads |
104+
| Watermark | `watermark.js` | generate, embed, extract |
105+
| Chat | `chat.js` | conversations, messages, search, friend requests |
106+
| Recall | `recall.js` | list, create, vote, detail |
107+
| Upload | `upload.js` | SHP upload, raster upload |
108+
| Data | `data.js` | vector data, raster data, tiles |
109+
| Data Viewing | `data_viewing_api.js` | viewing, search |
110+
| Navigation | `NaviApi.js` | nav tree |
111+
112+
- All 49 Vue components migrated from inline `axios` calls to centralized API modules
113+
114+
#### 12. Socket.IO Real-time Chat (`src/utils/socket.js`)
115+
- **What:** Event-driven messaging replaces 30-second polling
116+
- **Fallback:** Automatic HTTP polling when WebSocket unavailable
117+
- **Events:** `new_message`, `typing`, `mark_read`, `friend_request`
118+
- **Files changed:** `employee_chat.vue`, `AdminChat.vue`
119+
120+
#### 13. LogViewer Filter Fix (`views/admin/Log Management/LogViewer.vue`)
121+
- Filter values changed from UI labels (用户登录) to backend action types (登录, 申请提交, 一审通过, etc.)
122+
123+
#### 14. Application Withdrawal UI (`views/employee/Data Application/data_application.vue`)
124+
- "撤回" button with confirmation dialog for pending applications
125+
- "已撤回" badge for recalled applications
126+
127+
---
128+
129+
### Files Changed Summary
130+
131+
| Category | Files | Description |
132+
|----------|-------|-------------|
133+
| Security fixes | 8 | required.py, application_resource.py, admin_application_resource.py, dashboard_resource.py, collaboration_resource.py, recall_resource.py, download_file_resource.py, common_resource.py |
134+
| New algorithms | 4 | raster_dwt_watermark.py, raster_histogram_watermark.py, raster_geotiff_utils.py, quality_metrics.py |
135+
| New tests | 1 | test_robustness.py (6 tests) |
136+
| Modified tests | 3 | conftest.py, test_approval_workflow.py, test_integration.py |
137+
| New Grafana/Loki | 4 | alerts.yml, loki-config.yml, promtail-config.yml, logs.json |
138+
| New frontend utils | 1 | socket.js |
139+
| New API modules | 6 | auth.js, recall.js, upload.js, data.js, watermark.js, chat.js |
140+
| Modified frontend | 15+ | Vue components migrated to API modules |
141+
| Modified infra | 2 | docker-compose.yml, datasource.yml |
142+
| Documentation | 2 | README.md, CHANGELOG.md |
143+
144+
---
145+
146+
### Architecture Upgrade (Deep Audit Fixes)
147+
148+
#### 15. Frontend Admin Sub-role Bug Fix (`login.vue`, `router/index.js`)
149+
- **Problem:** `login.vue` did not pass `admin_sub_role` to `setUserInfo()`, causing all admins to default to `adm1` on page refresh. Router guard used exact match `=== 'admin'` but backend returns `'admin1'`/`'admin2'`/`'admin3'`.
150+
- **Fix:** `login.vue` now passes `admin_sub_role` from login response. Router guard normalizes role with `startsWith('admin')` check.
151+
152+
#### 16. QR_SECRET_KEY Crash Guard (`resource/watermark_resource.py`)
153+
- **Problem:** If `QR_SECRET_KEY` env var is unset, `None.encode()` would crash watermark generation at runtime.
154+
- **Fix:** Added dev-only fallback key with warning. Watermark generation no longer crashes; extraction already handled this gracefully.
155+
156+
#### 17. Per-user Rate Limiting Activated (`utils/user_limiter.py` → resource files)
157+
- **Problem:** `strict_limit`/`normal_limit`/`relaxed_limit` decorators were defined but never applied to any endpoint.
158+
- **Fix:** Applied to key endpoints:
159+
- `SubmitApplicationResource.post` — already had `@limiter.limit`
160+
- `WithdrawApplicationResource.put``@normal_limit`
161+
- `GetApplicationsResource.get``@relaxed_limit`
162+
- `Adm1PassResource.post` / `Adm2PassResource.post``@normal_limit`
163+
- `BatchReviewResource.post``@normal_limit`
164+
- `VectorExtractResource.post``@normal_limit`
165+
- `AdminDashboardResource.get``@relaxed_limit`
166+
167+
#### 18. Destructive `is_multiple.py` Fixed (`algorithm/is_multiple.py`)
168+
- **Problem:** `is_multiple()` overwrote original shapefiles in place, permanently destroying all but the first sub-geometry of MultiPolygon/MultiLineString features.
169+
- **Fix:** Now saves to `{filename}_single.shp` by default. Original file is never overwritten. Accepts optional `output_path` parameter. Returns output path for chaining.
170+
171+
#### 19. Algorithm Code Cleanup (`algorithm/embed.py`, `algorithm/is_multiple.py`)
172+
- Removed 6 commented-out `print()` debug statements from `embed.py`
173+
- Replaced hardcoded `E:\矢量数据\...` path in `embed.py` `__main__` with argparse CLI
174+
- Replaced hardcoded `D:\Desktop\Projects\yingbianma` path in `is_multiple.py` `__main__` with env var
175+
176+
#### 20. Dev Key Warning (`config.py`)
177+
- **Problem:** `DevelopmentConfig` silently used insecure default keys with no warning.
178+
- **Fix:** Added logging warning when dev keys are detected in development mode.
179+
180+
---
181+
182+
### Known Limitations (v2.3)
183+
1. **DWT/Histogram algorithms** — Only LSB has end-to-end integration tests; DWT and Histogram are tested via unit tests only
184+
2. **`datetime.utcnow()`** — Used in several places; should migrate to `datetime.now(datetime.UTC)` for Python 3.12+
185+
3. **Frontend E2E tests** — No Cypress/Playwright tests yet; all frontend testing is manual
186+
187+
---
188+
7189
## [v2.2.0] — 2026-05-07 (Full i18n Translation & UI Polish)
8190

9191
### Summary
@@ -358,12 +540,15 @@ Transformed from a student-level project into a production-grade, resume-worthy
358540
- [x] ~~Per-user rate limiting with Redis backend~~ — Done in v2.1 (Redis sorted sets)
359541
- [x] ~~WebSocket authentication (JWT handshake)~~ — Done in v2.1
360542
- [x] ~~Translate remaining 38 Vue view templates (~1800 strings)~~ — Done in v2.2 (all 49 views, 1200+ keys)
543+
- [x] ~~Add alerting rules to Grafana~~ — Done in v2.3 (error rate, latency, DB, cache alerts)
544+
- [x] ~~Add Loki for log aggregation in Grafana~~ — Done in v2.3 (Loki + Promtail + logs dashboard)
545+
- [x] ~~Admin sub-role differentiation~~ — Done in v2.3 (admin1/admin2/admin3 enforcement)
546+
- [x] ~~Application withdrawal~~ — Done in v2.3 (PUT /api/applications/{id}/withdraw)
547+
- [x] ~~Multi-algorithm raster watermark~~ — Done in v2.3 (LSB, DWT, Histogram Shifting)
361548
- [ ] Increase test coverage to 80%+ (currently ~50% of endpoints covered)
362549
- [ ] Add integration tests with real MySQL/PostgreSQL
363550
- [ ] Add E2E tests with Playwright or Cypress
364551
- [ ] Redis session store for multi-instance deployments
365-
- [ ] Add alerting rules to Grafana (error rate thresholds, latency alerts)
366-
- [ ] Add Loki for log aggregation in Grafana
367552

368553
---
369554

0 commit comments

Comments
 (0)