Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.ko.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!-- translated from README.md @ 1273153 -->
<!-- translated from README.md @ bb9fecf -->
# vmsctl

[![CI](https://github.com/with-developer/vmsctl/actions/workflows/ci.yml/badge.svg)](https://github.com/with-developer/vmsctl/actions/workflows/ci.yml)
Expand Down Expand Up @@ -70,7 +70,8 @@ vmsctl server demo # 가상의 호스트 12대로 둘러보기. 실제로
```

- **통과율이 시간에 따라 어떻게 달라졌는지** 7일·30일·90일 차트로 보고, 호스트마다 최근 추이도 봅니다.
- **가장 많은 호스트에서 실패하는 점검 항목과 방금 나빠진 점검 항목**을 바로 확인합니다.
- **먼저 고칠 것:** 실패한 점검 항목을 심각도와 실패한 호스트 수로 정렬하고, 고치면 통과율이 얼마나 오르는지 보여 줍니다.
최근 악화되거나 해결된 점검 항목과 보고가 끊긴 호스트도 한 목록으로 봅니다.
- **모든 결과에 근거**(파일과 줄)가 붙고, 실패한 항목 옆에 **조치 방법**이 나옵니다.
- **모든 호스트 × 모든 점검 항목 매트릭스.** 칸을 누르면 그 호스트로 이동합니다.
- 프로파일(KISA, essentials, 직접 만든 정책 팩)을 **한 번에 하나씩** 골라 봅니다. 목록에는 호스트의 OS와 라벨이 보입니다.
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ vmsctl server demo # a made-up fleet of twelve hosts, nothing scanned, noth
```

- **Pass rate over time** for the whole fleet (7, 30 or 90 days), and a trend line on every host.
- **What fails most, and what just got worse:** the controls that fail on most hosts, and the regressions of the last scans.
- **What to fix first:** the failing controls ranked by severity and the hosts that fail them, each with what fixing it adds to the
pass rate; and what just changed (controls that got worse or were fixed, hosts that stopped reporting).
- **The evidence behind every result,** the file and the line, with **how to fix it** next to each failure.
- **A matrix of every control on every host,** one click from a cell to the host.
- **One profile at a time** (KISA, essentials, your own policy pack), with host OS and labels in the list.
Expand Down
Binary file modified docs/demo/dashboard-overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions docs/fleet.ko.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!-- translated from docs/fleet.md @ bd440c1 -->
<!-- translated from docs/fleet.md @ bb9fecf -->
# 서버로 모아 보기

> 설계는 [ADR 0010](adr/0010-fleet-server-and-dashboard.md)에 있습니다. 호스트가 결과를 보내고, 서버가 저장해 두었다가 조회
Expand Down Expand Up @@ -297,7 +297,7 @@ curl --cacert tls.crt -H "Authorization: Bearer hlt_api_..." https://vmsctl.exam
|---|---|
| `GET /api/v1/profiles` | 최신 스캔에 쓰인 프로파일과 프로파일별 호스트 수, 가장 많은 호스트가 쓰는 프로파일(`default`), 각 프로파일의 제목 |
| `GET /api/v1/trend` | 전체 호스트의 최근 추이를 하루에 한 점씩, 오래된 날부터: 호스트 수, 실패한 점검 항목이 있는 호스트 수, 그날 기준의 상태별 개수와 통과율(`?days=30`, 1~90). 저장된 스캔으로 계산하므로 `server prune`을 하면 볼 수 있는 날짜가 달라지고, 같은 답을 30초 동안 재사용합니다 |
| `GET /api/v1/overview` | 호스트 수, 실패한 점검 항목이 있는 호스트 수, 오래 보고하지 않은 호스트 수, 상태별 개수, 가장 많은 호스트에서 실패하는 점검 항목, 최근 악화(`?staleDays=7`) |
| `GET /api/v1/overview` | 호스트 수, 실패한 점검 항목이 있는 호스트 수, 오래 보고하지 않은 호스트 수, 상태별 개수, 심각도와 호스트 수로 정렬한 실패 점검 항목(각각 고치면 오르는 통과율 `gain`, %p 단위), 최근 악화와 최근 해결(`?staleDays=7`) |
| `GET /api/v1/hosts` | 호스트 목록과 각 호스트의 최신 스캔 개수·통과율(`?q=`, `?status=failing\|stale`, `?sort=name\|score\|last`, `?limit=`, `?offset=`, `?format=csv`, 이미지를 보려면 `?kind=image`, 레이블이 있는 호스트만 보려면 `?label=env%3Dprod`. 대시보드에서는 레이블을 눌러 거릅니다). 호스트마다 `os`, `labels`(최신 결과가 알려 준 값이고 개수에 상한이 있습니다), `agentOnline`, 최근 스캔의 통과율 `trend`도 들어 있습니다 |
| `GET /api/v1/controls`, `GET /api/v1/controls/PROFILE/ID` | 점검 항목 전체와 최신 스캔의 상태별 개수. 점검 항목 하나의 경우 해당 호스트, 무엇을 점검하는지, 어떻게 고치는지 |
| `GET /api/v1/matrix` | 모든 호스트의 모든 점검 항목의 최신 상태. 가장 많이 실패한 호스트가 먼저 나옵니다. `?format=csv`를 붙이면 스프레드시트용 표(호스트마다 한 행, 점검 항목마다 한 열, `?lang=ko`로 열 제목을 한국어로)로 받으며, 대시보드의 매트릭스 화면에서 내려받을 수 있습니다 |
Expand Down
2 changes: 1 addition & 1 deletion docs/fleet.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ curl --cacert tls.crt -H "Authorization: Bearer hlt_api_..." https://vmsctl.exam
|---|---|
| `GET /api/v1/profiles` | The profiles of the latest scans with the number of hosts of each, the one most hosts use (`default`), and their titles |
| `GET /api/v1/trend` | The fleet over the last days, one point a day, oldest first: hosts, hosts with a failing control, counts and pass rate as of that day (`?days=30`, 1 to 90). It is computed from the scans that are stored, so `server prune` changes the days it covers, and is served again for 30 seconds |
| `GET /api/v1/overview` | Hosts, hosts with a failing control, stale hosts, counts, the controls that fail on most hosts, recent regressions (`?staleDays=7`) |
| `GET /api/v1/overview` | Hosts, hosts with a failing control, stale hosts, counts, the failing controls ranked by severity times hosts with the `gain` in pass rate (percentage points) of fixing each, recent regressions and recent fixes (`?staleDays=7`) |
| `GET /api/v1/hosts` | The hosts with the counts and score of their latest scan (`?q=`, `?status=failing\|stale`, `?sort=name\|score\|last`, `?limit=`, `?offset=`, `?format=csv`, `?kind=image` for the images, `?label=env%3Dprod` for the hosts with a label; the dashboard filters by a clicked label); each host also has its `os`, `labels` (what its latest result says, bounded), `agentOnline` and a `trend` of the pass rate of its latest scans |
| `GET /api/v1/controls`, `GET /api/v1/controls/PROFILE/ID` | Every control with the status counts of the latest scans; one control with its hosts, and what it checks and how to fix it |
| `GET /api/v1/matrix` | The latest status of every control on every host, hosts that fail most first; with `?format=csv`, a sheet for a spreadsheet (a row per host, a column per control, `?lang=ko` for Korean column titles; the dashboard offers it on the matrix page) |
Expand Down
3 changes: 3 additions & 0 deletions internal/server/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,9 @@ func (s *Server) handleOverview(w http.ResponseWriter, r *http.Request) {
for i := range o.RecentRegression {
o.RecentRegression[i].TitleKo = s.titleKo(o.RecentRegression[i].Profile, o.RecentRegression[i].ControlID)
}
for i := range o.RecentFixes {
o.RecentFixes[i].TitleKo = s.titleKo(o.RecentFixes[i].Profile, o.RecentFixes[i].ControlID)
}
writeJSON(w, http.StatusOK, o)
}

Expand Down
94 changes: 94 additions & 0 deletions internal/server/api_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"encoding/csv"
"encoding/json"
"fmt"
"math"
"net/http"
"net/http/httptest"
"path/filepath"
Expand Down Expand Up @@ -948,3 +949,96 @@ func TestMatrixCSV(t *testing.T) {
t.Errorf("csv without a token: %d", code)
}
}

// The overview says what fixing a failing control would add to the pass rate, and lists a control that a host fixed since its
// previous scan next to the regressions.
// rescan posts a fixture again as a new scan an hour later, with its controls changed by edit.
func (f *apiFixture) rescan(t *testing.T, name, id string, edit func(controls []any) []any) {
t.Helper()
var doc map[string]any
if err := json.Unmarshal(fixture(t, name), &doc); err != nil {
t.Fatal(err)
}
doc["scan"].(map[string]any)["id"] = id
doc["controls"] = edit(doc["controls"].([]any))
raw, err := json.Marshal(doc)
if err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(time.Hour)
if code, out := post(t, f.srv, token(t, f.store, ScopeIngest, ""), raw); code != http.StatusCreated {
t.Fatalf("%s: the scan was not stored: %d %v", id, code, out)
}
}

func setStatus(controls []any, status map[string]string) []any {
for _, c := range controls {
if st, ok := status[c.(map[string]any)["id"].(string)]; ok {
c.(map[string]any)["status"] = st
}
}
return controls
}

func TestOverviewGainAndFixes(t *testing.T) {
f := newAPIFixture(t)
var o Overview
f.json(t, "/api/v1/overview", &o)
decided := o.Counts["PASS"] + o.Counts["FAIL"] + o.Counts["ERROR"] + o.Counts["MANUAL"]
for _, c := range o.TopFailing {
want := 100 * float64(c.Hosts) / float64(decided)
if math.Abs(c.Gain-want) > 0.001 || c.Gain <= 0 {
t.Errorf("%s: gain %v, want %v", c.ID, c.Gain, want)
}
}
if len(o.RecentFixes) != 0 {
t.Errorf("fixes before any = %+v", o.RecentFixes)
}
// web-01 fixes U-01 (FAIL), and U-04 (ERROR) passes again: a check that ran, not a fix.
f.rescan(t, "web-01.json", "web-01-fixed", func(cs []any) []any { return setStatus(cs, map[string]string{"U-01": "PASS", "U-04": "PASS"}) })
f.json(t, "/api/v1/overview", &o)
if len(o.RecentFixes) != 1 || o.RecentFixes[0].Host != "web-01" || o.RecentFixes[0].ControlID != "U-01" || o.RecentFixes[0].Status != "PASS" {
t.Errorf("fixes = %+v", o.RecentFixes)
}
for _, r := range o.RecentRegression {
if r.Host == "web-01" && r.ControlID == "U-01" {
t.Errorf("a fixed control is still a regression: %+v", r)
}
}
}

// The fix-first order is severity (high 3, medium 2, low 1) times the failing hosts, and a control that hosts report with
// different severities counts with the highest one.
func TestOverviewFixFirstOrder(t *testing.T) {
f := newAPIFixture(t)
add := func(sev map[string]string) func([]any) []any {
return func(cs []any) []any {
for _, id := range []string{"X-1", "X-2", "X-3"} {
if sev[id] != "" {
cs = append(cs, map[string]any{"profile": "kisa-2026", "id": id, "title": id, "severity": sev[id], "status": "FAIL", "checks": []string{}})
}
}
return cs
}
}
f.rescan(t, "web-01.json", "web-01-more", add(map[string]string{"X-1": "low", "X-2": "medium", "X-3": "high"}))
f.rescan(t, "db-01.json", "db-01-more", add(map[string]string{"X-1": "low", "X-3": "medium"}))
var o Overview
f.json(t, "/api/v1/overview", &o)
at := map[string]int{}
for i, c := range o.TopFailing {
at[c.ID] = i
}
for _, id := range []string{"U-08", "X-1", "X-2"} {
if _, ok := at[id]; !ok {
t.Fatalf("%s is not in %+v", id, o.TopFailing)
}
}
// X-3: 2 hosts × high = 6; U-08: 1 host × high = 3; X-1: 2 hosts × low = 2; X-2: 1 host × medium = 2, fewer hosts.
if len(o.TopFailing) == 0 || o.TopFailing[0].ID != "X-3" || o.TopFailing[0].Severity != "high" {
t.Fatalf("first = %+v", o.TopFailing)
}
if at["U-08"] >= at["X-1"] || at["X-1"] >= at["X-2"] {
t.Errorf("order = %+v", o.TopFailing)
}
}
80 changes: 58 additions & 22 deletions internal/server/query.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"errors"
"fmt"
"io"
"math"
"sort"
"strconv"
"strings"
Expand Down Expand Up @@ -486,6 +487,9 @@ type FailingControl struct {
TitleKo string `json:"titleKo,omitempty"`
Severity string `json:"severity"`
Hosts int `json:"hosts"`
// Gain is what fixing the control on every host that fails it would add to the pass rate, in percentage points: each host
// moves one result from FAIL to PASS, over the same total.
Gain float64 `json:"gain"`
}

// Overview is the numbers of the dashboard's first page.
Expand All @@ -500,7 +504,10 @@ type Overview struct {
Score *float64 `json:"score"`
TopFailing []FailingControl `json:"topFailing"`
RecentRegression []RegressedControl `json:"recentRegressions"`
StaleAfterDays int `json:"staleAfterDays"`
// RecentFixes are controls that were FAIL in a host's previous scan and pass now (or are not applicable),
// in the same time as the regressions.
RecentFixes []RegressedControl `json:"recentFixes"`
StaleAfterDays int `json:"staleAfterDays"`
}

// Overview computes the first page of the dashboard. A host is stale when it has not reported
Expand All @@ -513,7 +520,7 @@ func (s *Store) Overview(ctx context.Context, staleAfter time.Duration, profile
if err != nil {
return nil, err
}
o := &Overview{Counts: Counts{}, TopFailing: []FailingControl{}, RecentRegression: []RegressedControl{}, StaleAfterDays: int(staleAfter.Hours() / 24), Profile: profile}
o := &Overview{Counts: Counts{}, TopFailing: []FailingControl{}, RecentRegression: []RegressedControl{}, RecentFixes: []RegressedControl{}, StaleAfterDays: int(staleAfter.Hours() / 24), Profile: profile}
for _, h := range hosts {
if h.Stale {
o.HostsStale++ // that a host is silent does not depend on the profile it scanned with
Expand All @@ -533,10 +540,13 @@ func (s *Store) Overview(ctx context.Context, staleAfter time.Duration, profile
o.Score = o.Counts.Score()

rs, err := s.db.QueryContext(ctx, `
SELECT c.profile, c.control_id, MAX(c.title), MAX(c.severity), COUNT(*)
FROM scan_controls c JOIN (`+latestScans+`) l ON l.id = c.scan_row
WHERE c.status = 'FAIL' AND (? = '' OR c.profile = ?) GROUP BY c.profile, c.control_id
ORDER BY COUNT(*) DESC, c.control_id LIMIT 10`, profile, profile)
SELECT profile, control_id, title, CASE w WHEN 3 THEN 'high' WHEN 2 THEN 'medium' ELSE sev END, n FROM (
SELECT c.profile AS profile, c.control_id AS control_id, MAX(c.title) AS title, MAX(c.severity) AS sev, COUNT(*) AS n,
MAX(CASE c.severity WHEN 'high' THEN 3 WHEN 'medium' THEN 2 ELSE 1 END) AS w
FROM scan_controls c JOIN (`+latestScans+`) l ON l.id = c.scan_row
WHERE c.status = 'FAIL' AND (? = '' OR c.profile = ?) GROUP BY c.profile, c.control_id
)
ORDER BY n * w DESC, n DESC, control_id LIMIT 10`, profile, profile)
if err != nil {
return nil, err
}
Expand All @@ -546,6 +556,9 @@ func (s *Store) Overview(ctx context.Context, staleAfter time.Duration, profile
_ = rs.Close()
return nil, err
}
if decided := o.Counts["PASS"] + o.Counts["FAIL"] + o.Counts["ERROR"] + o.Counts["MANUAL"]; decided > 0 {
f.Gain = math.Round(1e5*float64(f.Hosts)/float64(decided)) / 1e3 // three decimals: the dashboard rounds for display
}
o.TopFailing = append(o.TopFailing, f)
}
if err := rs.Err(); err != nil {
Expand All @@ -554,31 +567,54 @@ func (s *Store) Overview(ctx context.Context, staleAfter time.Duration, profile
_ = rs.Close()

cutoff := s.now().UTC().Add(-staleAfter).Format(stampLayout)
rr, err := s.db.QueryContext(ctx, `
if o.RecentRegression, o.RecentFixes, err = s.recentChanges(ctx, cutoff, profile); err != nil {
return nil, err
}
return o, nil
}

// recentChanges compares each host's latest scan, received since cutoff, with its previous one. A regression is a control that is
// FAIL or ERROR now and was not before; a fix is a control that was FAIL and is PASS or NOT_APPLICABLE now (ERROR to PASS is a
// check that ran again, not a fix, and a waived control is not a fix). At most 20 of each, newest first.
func (s *Store) recentChanges(ctx context.Context, cutoff, profile string) (regressions, fixes []RegressedControl, err error) {
rows, err := s.db.QueryContext(ctx, `
WITH ranked AS (
SELECT s.id AS id, s.host_row AS host_row, s.received_at AS received_at, ROW_NUMBER() OVER (PARTITION BY s.host_row ORDER BY s.received_at DESC, s.id DESC) AS rn
FROM scans s JOIN hosts k ON k.id = s.host_row AND k.kind = 'host'
), changed AS (
SELECT h.id AS host_id, h.name AS host, c.profile AS profile, c.control_id AS control_id, c.title AS title, c.severity AS severity,
c.status AS status, l.received_at AS received_at, c.status IN ('FAIL', 'ERROR') AS regressed
FROM ranked l
JOIN ranked pv ON pv.host_row = l.host_row AND pv.rn = 2
JOIN hosts h ON h.id = l.host_row
JOIN scan_controls c ON c.scan_row = l.id
LEFT JOIN scan_controls p ON p.scan_row = pv.id AND p.profile = c.profile AND p.control_id = c.control_id
WHERE l.rn = 1 AND l.received_at >= ? AND (? = '' OR c.profile = ?) AND (
(c.status IN ('FAIL', 'ERROR') AND (p.status IS NULL OR p.status NOT IN ('FAIL', 'ERROR')))
OR (c.status IN ('PASS', 'NOT_APPLICABLE') AND p.status = 'FAIL'))
), numbered AS (
SELECT *, ROW_NUMBER() OVER (PARTITION BY regressed ORDER BY received_at DESC, host, control_id) AS k FROM changed
)
SELECT h.id, h.name, c.profile, c.control_id, c.title, c.severity, c.status, l.received_at
FROM ranked l
JOIN ranked pv ON pv.host_row = l.host_row AND pv.rn = 2
JOIN hosts h ON h.id = l.host_row
JOIN scan_controls c ON c.scan_row = l.id AND c.status IN ('FAIL', 'ERROR')
LEFT JOIN scan_controls p ON p.scan_row = pv.id AND p.profile = c.profile AND p.control_id = c.control_id
WHERE l.rn = 1 AND (p.status IS NULL OR p.status NOT IN ('FAIL', 'ERROR')) AND l.received_at >= ? AND (? = '' OR c.profile = ?)
ORDER BY l.received_at DESC, h.name, c.control_id LIMIT 20`, cutoff, profile, profile)
SELECT host_id, host, profile, control_id, title, severity, status, received_at, regressed
FROM numbered WHERE k <= 20 ORDER BY received_at DESC, host, control_id`, cutoff, profile, profile)
if err != nil {
return nil, err
return nil, nil, err
}
defer rr.Close()
for rr.Next() {
defer rows.Close()
regressions, fixes = []RegressedControl{}, []RegressedControl{}
for rows.Next() {
var r RegressedControl
if err := rr.Scan(&r.HostID, &r.Host, &r.Profile, &r.ControlID, &r.Title, &r.Severity, &r.Status, &r.ReceivedAt); err != nil {
return nil, err
var regressed bool
if err := rows.Scan(&r.HostID, &r.Host, &r.Profile, &r.ControlID, &r.Title, &r.Severity, &r.Status, &r.ReceivedAt, &regressed); err != nil {
return nil, nil, err
}
if regressed {
regressions = append(regressions, r)
} else {
fixes = append(fixes, r)
}
o.RecentRegression = append(o.RecentRegression, r)
}
return o, rr.Err()
return regressions, fixes, rows.Err()
}

// MatrixControl is a column of the matrix.
Expand Down
Binary file modified site/img/dashboard-overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified site/img/ko/dashboard-overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading