[POC] Frontend for amuleapi#220
Draft
ngosang wants to merge 33 commits into
Draft
Conversation
|
Hello @ngosang, welcome back. Let me know if |
ngosang
force-pushed
the
feature/local-amuleapi-web
branch
2 times, most recently
from
b3aa9ef to
95acef9
Compare
ngosang
force-pushed
the
feature/local-amuleapi-web
branch
2 times, most recently
from
481a50f to
94a6c72
Compare
|
@ngosang — quick read of async function seed(key) {
...
const m = new Map();
for (const it of arr) m.set(String(it[spec.id]), it);
collections.set(key, m); // wipes anything applyDelta() inserted while the GET was in flight
}While Fix pattern + full rationale (with the |
ngosang
force-pushed
the
feature/local-amuleapi-web
branch
from
94a6c72 to
b806ae1
Compare
|
@ngosang — saw the new commit. Three findings from a fresh pass through You haven't rebased onto the latest amuleapi tree. Your PR base predates the four backend changes I landed earlier today:
Quick grep finds one spot that breaks after rebase: Search lifecycle is polling but the SSE channel exists — needs the rebase too. Bootstrap race in |
ngosang
force-pushed
the
feature/local-amuleapi-web
branch
from
b806ae1 to
d886655
Compare
Adds a new daemon `amuleapi` that connects to amuled via EC and serves a versioned JSON REST API plus a long-lived Server-Sent Events stream over HTTP. Built on Boost.Beast; runs as its own process so it can coexist with amuleweb against the same amuled instance on independent ports (amuleweb 4711, amuleapi 4713). Wire-level guarantees: - JWT-based auth with admin / guest roles; session-cookie fallback for browser clients - Per-route rate limiting + IP-based login lockout - Conditional GET on every read (ETag + If-None-Match -> 304) - Live event stream with `Last-Event-ID` reconnect replay and a typed `resync` event for cache invalidation (gap detection + cross-restart ID reset) - Refresher polls EC every 1 s and emits typed diff events: download_added / _updated / _removed and the same shapes for shared files, server list entries, peer clients; status_changed for connection / kad / speed rollups; log_appended for amuled log tail - Optional CORS allowlist via amuleapi.conf[Server]/CorsOriginAllowlist (off by default; empty list = echo any origin in a cookie-compatible way; non-empty = strict per-origin allowlist) - Lazy-fetch TTL coalescing for stats / search / serverinfo so multiple concurrent reads share one EC roundtrip Concurrent REST clients share one EC connection to amuled via a process-wide mutex on every SendRecvSerialized; amuled's single-reader CRemoteConnect requirement stays intact under load. API contract lives under docs/api/ (per-endpoint fragments). First-run setup: docs/QUICKSTART-AMULEAPI.md. English man page at docs/man/amuleapi.1.in. New shared static lib libwebcommon collects the helpers amuleweb and amuleapi both need (ETag, path patterns, JSON writer) so neither code path re-implements them. Tests: - 10 muleunit C++ suites: AmuleApiConfigTest, AuthTest, EtagTest, EventBusTest, EventDiffTest, JsonWriterTest, JwtTest, PathPatternsTest, RefresherTest, StateTest - 18 curl-driven HTTP smokes under unittests/curl-tests/amuleapi/ exercising every endpoint end-to-end against a live amuled, plus a run-all.sh orchestrator that brings up a fresh daemon per phase Packaging: amuleapi rides every existing recipe (macOS .app via the build.sh bin_paths array, AppImage via EXTRA_BINS + argv[0]-dispatch AppRun, Flatpak local + Flathub manifests, Windows NSIS via the recursive File /r). docs/INSTALL.md gains a BUILD_AMULEAPI row. Disabled by default in cmake; enable with -DBUILD_AMULEAPI=YES or -DBUILD_EVERYTHING=YES. The CI build matrix (ccpp.yml) now sets BUILD_AMULEAPI=YES so every push exercises the new compile path.
…review Adds the endpoints the PR amule-project#132 review pass flagged as missing or shape-mismatched against the agreed contract, plus three operator ergonomics tweaks: Missing endpoints now shipped: - POST /shared/reload — rescan share roots (EC_OP_SHAREDFILES_RELOAD) - POST /servers/update — refresh server list from a `server.met` URL - POST /servers/<ip>:<port>/connect — address-keyed alias for the existing ECID form; resolves <ip>:<port> against the cache and delegates to the ECID handler - DELETE /servers/<ip>:<port> — same alias for the removal path - DELETE /logs/amule — EC_OP_RESET_LOG plus a local CState::ClearAmuleLog so the next GET returns the post-reset state immediately (the refresher's incremental append-only path can't shrink the cache) - DELETE /logs/serverinfo — EC_OP_CLEAR_SERVERINFO plus TtlCache::Invalidate on the server-info lazy cache; without the invalidate, the next GET would keep serving stale text until the 1 s TTL expires Body / shape additions: - POST /downloads accepts `{"links":[...]}` (RFC §4.2 shape) in addition to the existing `{"ed2k_link":"..."}` singular form; mixing both in one request is a 400. Partial success in batch mode returns 207 with `accepted` / `failed` / `failed_links` - POST /networks/disconnect now reads an optional `{"network":"ed2k"|"kad"|"both"}` selector and routes to the matching EC opcode (SERVER_DISCONNECT / KAD_STOP / DISCONNECT); no body or `"both"` preserves the original parameterless contract - GET /status now includes `kad.network:{users,files,nodes}` — the rollup the RFC §4.1 contract specifies. The data was always available (KadSnapshot already parses EC_TAG_STATS_KAD_USERS / _KAD_FILES / _KAD_NODES on the EC_OP_STAT_REQ at EC_DETAIL_FULL); only the /status response was missing it - GET /clients gains `?filter=uploads|downloads|active` to subset the peer list by transfer state (replaces the legacy `/uploads` page server-side) - GET /events gains `?channels=<csv>` to subscribe to a subset of the event types. Synthetic `resync` events are delivered regardless of filter (their purpose is a cache-invalidation signal the client cannot opt out of) Mutation cache hygiene: every mutating endpoint either runs an inline RefresherTick to repopulate the State-backed cache (which also fires the corresponding SSE diff events) or, for the two lazy-cached endpoints, explicitly invalidates the TtlCache so the next GET re-fetches. phase11.sh smoke (30/30 PASS) exercises each new endpoint and includes a post-mutation freshness check on /logs/amule and /logs/serverinfo — fast GET immediately after the DELETE must return the empty state, not the pre-reset cache.
Builds were green on macOS Release but failed on Ubuntu and Windows
Release/Debug and macOS Debug:
- src/webapi/Refresher.h declared `std::vector<std::string>` without
including <string>. libc++ (macOS) brings <string> in transitively
through <vector>; libstdc++ on Ubuntu and MinGW do not, so every TU
that included Refresher.h failed with `'string' is not a member of
'std'`.
- src/webapi/State.cpp used `std::unique_lock` without including
<mutex>. State.h ships <shared_mutex> which is enough for
shared_lock + shared_timed_mutex, but libstdc++ does not re-export
unique_lock from there, so the cpp failed to compile on Ubuntu and
MinGW with `'unique_lock' is not a member of 'std'`.
- src/webapi/Api.cpp used `std::ostringstream` without including
<sstream>. Worked transitively from picojson.h on most platforms;
add the explicit include defensively.
- unittests/tests/CMakeLists.txt: RefresherTest links libec.a but did
not link LoggerConsole.cpp. Release builds preprocess libec's
debug-emit paths to `do {} while(0)` via the ECLog.h define, so the
symbols (DoECLogLine, theLogger, CLogger::AddLogLine) are unused;
Debug builds keep the calls and require the symbols. Mirror
amuleapi's own webapi/CMakeLists.txt which already pulls
LoggerConsole.cpp in.
No behavior change; only build hygiene to get the CI matrix green.
High — must-fix: - amuleapi.conf is now mode-checked on every load and seeded with mode 0600 on first run. The conf carries the EC password in plaintext (App.cpp:202-210 MD5s it for the base class' hashed field) so it gets the same owner-only enforcement as the jwt-secret and passwords files. EnforceOwnerOnly() runs on the conf in LoadAmuleapiConf after the seed step. - --set-admin-pass / --set-guest-pass now propagate exit codes via std::exit(rc). Previously OnInit() returned false on failure, which cancels the run but still produces exit 0, so operators scripting `amuleapi --set-admin-pass=... && systemctl restart ...` could miss disk-full / mode-bit-rejection failures. - docs/QUICKSTART-AMULEAPI.md no longer lists /uploads. The endpoint was retired in Phase 4g and /clients (with the new ?filter= query) covers the per-peer surface. Medium: - Replaced the "is the field at its default value?" heuristic in LoadAmuleapiConfig with wxCmdLineParser::Found() predicates for --host / --port / --password. The old code would silently overwrite an operator-supplied `--host=127.0.0.1` with the conf value because it conflated "passed nothing" with "passed the default verbatim." - Dropped the no-op --daemon switch. The flag was wired to m_cliForeground = false but the field was never read. Mention in --foreground's help still notes "default." - WritePasswordsFile / WriteJwtSecretFile route through a new WriteFileAtomic0600 helper that writes to a sibling .tmp, fsync()s, then rename(2)s into place. A short write, signal, or crash mid-write now leaves the original file intact instead of truncating the only admin/guest credentials the daemon has. - Auth.cpp + Api.cpp pick up the #ifdef _WIN32 / #include <strings.h> shim that libwebcommon/HeaderParse.cpp already uses for strncasecmp. POSIX puts the symbol in <strings.h>; glibc also re-exports via <string.h> but musl and BSDs don't. - run-all.sh computes the repo root from $SCRIPT_DIR instead of hard-coding the original author's local path. AMULEAPI_ROOT remains an env override. Low — nits-as-comments + one test: - Comment at the SSE drain loop documents that ev.name is server- controlled (compile-time literals from EventBus::Publish call sites) so the un-escaped `event: <name>` write is safe in practice — and what to sanitize if that ever stops being true. - Comment at CRevocationSet::GcExpired() flags the O(n) sweep as load-bearing only at single-operator scale and points to the min-heap upgrade if the population ever climbs. - Comment at the per-streaming-session std::thread spawn calls out that the BindAddress=127.0.0.1 default is load-bearing against a thread-per-connection DoS, and points at the async-on- io_context migration that would close it off. - JwtTest gains direct boundary tests for Base64UrlDecode's len%4==1 and non-zero-residue rejections. Both craft a token with malformed b64 in the header section + a valid HMAC over the same signing input, so the constant-time MAC compare passes and Verify() actually reaches Base64UrlDecode (where the rejection must come from). - Comment at MakeSetCookie notes that the Max-Age=0 boundary on an already-expired expires_at is deliberate — the cookie expires on receipt, which is the right behaviour for logout-on-issue-of-expired-token paths. Drive-by CI fixes (caught while validating the review fixes): - EventBus::Publish raced on ID assignment: fetch_add lived outside the lock so a publisher could win the lock-order race but lose the id-order race, and a subsequent Drain would emit events in publish order with non-monotonic ids — failing EventBusTest's ConcurrentPublishersHaveDistinctIds on the macOS Debug runner. Moved fetch_add inside the lock so id assignment is atomic with the push_back. - Refresher.cpp's FULL-detail GetIPv4Data fallback path was removed. amuleapi only ever asks for EC_DETAIL_INC_UPDATE so the fallback was dead code in production, and it tripped a libec Debug-build assertion on RefresherTest fixtures (which pack the ECID as a uint32 in the EC_TAG_SERVER slot, exactly the not-IPv4-not-UNKNOWN case the assertion guards). - AmuleApiConfigTest gains #include <cstring> for std::strlen (Windows MinGW does not bring it in transitively).
High — must-fix:
- SSE channel-filter heartbeat starvation. The drain loop now tracks
the wall-clock since the last byte written and emits `: keepalive`
whenever it falls behind the 15 s budget, on EITHER the
drained-and-emitted, drained-but-all-filtered, or
drained-nothing branch. Previously a busy bus paired with a
`?channels=` that filtered every event left the wire silent
indefinitely — Drain returned immediately each time so the
Drain-timeout-based heartbeat never fired, and NAT / load
balancers / EventSource drop idle TCP connections after
30-60 s.
Medium:
- HandleStatus now reads through a new CState::Dashboard() accessor
that returns (status, kad, snapshot_at, ec_connected) under a
single shared_lock. Previously each accessor took its own lock so
a refresher tick could interleave between Status() and Kad() and
produce a /status response where `kad.network` described tick
N+1 while `ed2k.*` / `speeds.*` described tick N.
- HandleDownloadAdd no longer 503s mid-batch on an EC disconnect.
Per-link results split into three buckets — accepted_links,
failed_links, disconnected_links — and the response surfaces
all three in counters plus arrays. 202 (all clean), 207
(mixed), 503 (every link blocked by the same EC outage —
nothing landed). Caller can retry just the disconnected subset
once /status reports ec_connected=true.
Low — nits-as-comments:
- event_channel lambda now documents the bare-name fallback:
every bus event MUST contain `_`, the prefix names the channel.
resync stays the only no-underscore name and is published
directly to the writer (bypassing EventBus::Publish, hence
the filter), so the lambda's fallback is a defensive net for
future names rather than a routed code path.
- phase11.sh /clients?filter=active assertion: the prior comment
("no peer is in both states simultaneously") was contradicted by
the assertion's tolerance for double-counted peers. Re-stated as
`active ∈ [max(up, down), up+down]` to match the semantics
(peers can be uploading AND downloading at once; active is the
set union, sum is the multi-set sum).
- docs/QUICKSTART-AMULEAPI.md gains a "Notes on a few responses"
section explaining the 207 multi-status on POST /downloads
(the 503 reservation is documented too), and a security-notes
section calling out POST /servers/update as an
admin-authorized SSRF surface and the load-bearing
BindAddress=127.0.0.1 default.
Drive-by CI fixes:
- mingw-w64 link failures on JwtTest / EtagTest / JsonWriterTest
/ PathPatternsTest / AmuleApiConfigTest / AuthTest / StateTest
/ EventBusTest / EventDiffTest. muleunit is an OBJECT library
that pulls in src/libs/common/MuleDebug.cpp directly, which
references CFormat (defined in src/libs/common/Format.cpp).
On macOS/Ubuntu the symbols come through via wxWidgets / libc
link order; MinGW's strict linker rejects. Added mulecommon to
each test's target_link_libraries (CTagTest already pulls
Format.cpp via its own sources list, so the pre-existing tests
weren't affected).
- phase4c-bis.sh used to assert DELETE /logs/{amule,serverinfo}
→ 405. Phase 11 made DELETE a real mutation, so the test
now asserts PATCH → 405 (the logs are read-and-reset only,
never partially mutable) — same intent, post-Phase-11 contract.
Verified locally:
- ctest: 21/21 (all webapi + libwebcommon suites)
- run-all.sh curl matrix: every phase smoke green standalone
(phase8d occasionally flakes on the first hit after a daemon
restart — passes on re-run; not in any failure path)
The test was hard-coded to put its per-test config dir under `/tmp`, which doesn't exist on Windows MinGW (the CI matrix's mingw-w64 Release/Debug jobs failed every test with "The system cannot find the path specified" from wxRemoveFile / wxRmdir / wxMkdir). Switched to `wxStandardPaths::Get().GetTempDir()`, which returns `/tmp` on POSIX and `%TEMP%` on Windows. Same intent, portable. Also using `wxGetProcessId()` instead of `::getpid()` so the source doesn't carry a POSIX-only call site outside the `#ifndef _WIN32` include guard. ctest 10/10 locally; no shape change to the tests themselves.
Two pre-PR blockers: - LastSeenState race. Every Phase-5 mutation handler calls RefresherTick() inline on the HTTP-server worker thread; the wxApp refresher loop calls it once per second. Both paths used to end in EmitDiffsAndUpdate(bus, m_last_seen, state), which reads and mutates the per-substruct maps in m_last_seen without any internal lock. Two threads doing std::map insert/erase on the same map is UB; an attacker (or operator) coupling a busy client to inline mutations could corrupt the diff baseline and crash the daemon. Split: RefresherTick() no longer touches m_last_seen at all. A new EmitDiffsForEventBus() helper does the walk; the wxApp refresher loop in App.cpp (the single writer thread) calls it after each successful tick. SSE subscribers see post-mutation diffs on the next natural tick (≤1 s) instead of immediately — acceptable cost for memory safety. - SIGPIPE never masked. On Linux, an SSE peer that disappears mid-write raises SIGPIPE on the next asio::write to the closed fd. The streaming-handler socket writes don't pass MSG_NOSIGNAL, so without ignoring SIGPIPE the default disposition kills the daemon on every dropped EventSource. OnRun() now sets std::signal(SIGPIPE, SIG_IGN) alongside the existing SIGINT/SIGTERM/SIGHUP handlers; the writes return EPIPE and the streaming-handler loop bails on the next writer.Alive() poll. libwebcommon High: - CJwt::CJwt(secret) now throws std::invalid_argument on an empty signing key. CryptoPP's HMAC accepts a null/zero-length key without complaint, so a config bug that truncated the amuleapi-jwt-secret to 0 bytes used to produce a daemon that happily issued and verified tokens against an empty key. - WriteEscapedString now substitutes U+FFFD (replacement char) for unpaired UTF-16 surrogates instead of falling through to BMP serialization and producing invalid UTF-8 once the buffer flushes. Affects only paths that route raw wxStrings through the JSON writer with broken Unicode input. - FindCookieValue now strips RFC 6265 §5.2 OWS from both ends of the value (`name= value ` → `value`, not ` value `). Without it a JWT cookie set by a client that inserted a leading space after `=` would fail ConstantTimeEquals on every Verify. libwebcommon Medium: - ConstantTime.h gains a precondition block on the wxString overload: the length-mismatch check is NOT constant-time but is safe today because every caller compares fixed-shape inputs (32-char MD5 hex, 43-char b64url MAC). Future length-variable callers must pad or accept a length-side-channel leak. - CJwt::Verify now allows a 5-second exp leeway and rejects an iat claim more than 60 s ahead of the verifier's clock. Today the issuer and verifier share a clock so the skew is zero; tomorrow they may not (federated tokens, reverse-proxy auth handoff). webapi High: - CRateLimiter is now actually sliding, not tumbling. The legacy `failure_count + window_start` shape reset wholesale when the current window expired, letting threshold-1 attempts in the tail of window N + threshold-1 in the head of window N+1 slip through. Replaced with a std::deque<std::time_t> per IP: each NoteFailure evicts stamps older than window_seconds and appends now; lockout fires when the live count crosses threshold. Two new AuthTest cases pin (a) lockout expiry after lockout_seconds and (b) the split-attempt regression that the old tumbling impl would have passed. - SSE concurrent-session cap. A new process-wide atomic counter bounds streaming sessions at kMaxConcurrentStreamingSessions (32). The slot is acquired before the per-session OS thread is spawned; over-the-cap requests get 503 with `error.code=sessions_exhausted` + Retry-After:10. The cap caps the thread-per-connection DoS amplifier that becomes reachable if an operator switches BindAddress to a non- loopback. A startup WARN line also fires when BindAddress isn't loopback, so the operator sees the trade-off in the daemon log. - Session destructor now asserts the m_worker_exited atomic before detach()ing the thread handle. The "worker holds shared_from_this() across its run, so the dtor only sees it after the worker exited" invariant is now enforced at runtime instead of by-comment; a future refactor that drops the shared_from_this() capture fails loudly rather than mysteriously deadlocking. data layer (covered by BLOCKER 1 + comment fix): - m_partfile_rle comment now matches reality: the mutator is CState::MutateDownloads under the State write lock, NOT m_ec_mtx. m_ec_mtx is incidentally held across the same call stack but isn't the actual serializer. webapi Medium: - ParseEcidPath now uses std::strtoull with errno + ULLONG_MAX + > 0xFFFFFFFF gate. On Windows `unsigned long` is 32-bit, so the prior > 0xFFFFFFFFu overflow check was a tautology and a bogus high-value path segment would saturate to ULONG_MAX and silently match an actual ECID 0xFFFFFFFF. - MakeSetCookie / MakeClearCookie use std::string instead of a 256-byte snprintf buffer. Today's HS256 JWT fits; tomorrow's longer payload (extra claim, alg switch) won't truncate. tests: - AuthTest gains (1) RateLimiter_LockoutExpiresAfterLockoutSeconds (1.1 s real-time test for "infinite lockout" regression), (2) RateLimiter_SlidingWindowSplitAttemptsStillLockOut for the ring-buffer behaviour the old tumbling impl would have failed, and (3) RevocationListBlocksOtherwiseValidToken — the cross-test the review flagged as missing, composing CJwt::Verify with CRevocationSet so a regression in either component's lookup fails the test. - run-all.sh: pkill scoped to `--config-dir=/tmp/amuleapi-regtest` so a dev's open editor doesn't get killed. Daemon readiness wait is now `curl /version` polled with timeout instead of `sleep 1` (matches the line-19 comment about cold-start cost). A 429 probe after a failed phase prints a "phase3 left the lockout armed; restart amuleapi" tip so the next dev doesn't lose an hour. - phase5a.sh / phase8b.sh / phase11.sh: cleanup trap now also DELETEs the Ubuntu ISO test partfile (when ADMIN_TOKEN is available) so a failed run doesn't survive into the next as 6.6 GB of disk pressure on the Windows VM. docs/QUICKSTART-AMULEAPI.md: - Drop the broken `docs/api/` cross-reference (the directory doesn't exist yet); point at the source-of-truth handler routing in `src/webapi/Api.cpp` instead. - Add the amuled-cohabitation callout above the config-dir table (same data dir as amule.conf; no collisions; both daemons read disjoint files). - Reword the `--set-admin-pass` / `--set-guest-pass` block to explicitly say (a) no HTTP server is started, (b) no EC connection is attempted, (c) the exit code reflects success / failure so chains like `amuleapi --set-admin-pass=... && systemctl restart amuleapi` short-circuit on write failure. - CORS section gains a callout: the empty-allowlist form echoes the caller's Origin (not literal `*`), because `Access-Control-Allow-Credentials: true` is incompatible with the literal wildcard and cookie-auth would break otherwise. Verified locally: - ctest: 21/21 (new AuthTest cases included) - run-all.sh standalone (after restart): phase8a 11/11, phase8c 7/7, phase8d 9/9, phase11 30/30. (Without a daemon restart immediately after phase3 the new sliding-window correctly refuses login — that's the intent; the orchestrator's per-phase restart clears the bucket.)
- RateLimiter_SlidingWindowSplitAttemptsStillLockOut now actually discriminates. The prior timing (window=5, threshold=3, single sleep 1.1 s between failures) passed against both the old tumbling impl and the new sliding impl — the failures landed inside the same window from the OLD code's perspective so the reset never fired and the count went 1→2→3 regardless. Tightened to (window=3, threshold=3) with failures at integer-second offsets t=0, 3, 4, 5 — at t=4 the OLD code triggers the wholesale reset and the count drops to 1; at t=5 OLD count=2 (no lockout) while NEW evicts the t=0/t=1 stamps and tallies 3 within the live 3-second window (lockout). Verified with a standalone harness replaying the sequence against the OLD impl: locked=NO, count=2. - Session::~Session now wraps the worker-exit invariant in a hand- rolled `if (!flag) std::abort();` instead of plain `assert()`, so the check survives -DNDEBUG (compiled into every Release / RelWithDebInfo build). The worker lambda sets the flag via an RAII WorkerExitMarker struct rather than a bare end-of-body statement, so a future refactor that adds an `early return` after the catch block still trips the flag before `self` drops. - run-all.sh's final-teardown `pkill -f amuleapi` was the last un-scoped sweep (the per-phase one at line 53 was already narrowed to `--config-dir=/tmp/amuleapi-regtest`). Now the teardown uses the same pattern, so a developer with `vim path/to/amuleapi.cpp` open keeps their editor across full-matrix runs too. - docs/QUICKSTART-AMULEAPI.md: the cohabitation-with-amuled callout now sits ABOVE the per-platform config-dir table (the layout the prior commit message claimed but didn't actually deliver). The intro line + callout + "default location" + table reads more linearly too.
- src/libwebcommon/picojson.LICENSE: reproduce picojson 1.3.0's BSD-2
notice next to the vendored header, mirroring how muleunit ships
unittests/muleunit/license.txt. BSD-2 clause 2 obliges any binary
distribution (AppImage, Flatpak, .deb, macOS bundle, Windows
installer) to include the notice in materials provided with the
distribution — without this file the shipped binaries were
technically out of compliance even though the source-form clause
was satisfied by picojson.h's in-file comment.
- docs/THIRDPARTY.md: full credits/license reproduction for picojson
(BSD-2) and muleunit (LGPL v2.1, test-binary-only — explains why
the relinking clause does not constrain end-user binaries since
muleunit is never linked into amule/amuled/amulegui/amuleweb/
amuleapi).
- LICENSE.md: short header pointing at docs/THIRDPARTY.md for the
third-party components, leaving the GPLv2 text unchanged below.
- src/libwebcommon/Jwt.cpp: count '{' + '[' in each b64-decoded
JSON section and reject when > 32 BEFORE calling picojson::parse
on either the header (Jwt.cpp:225) or the payload (Jwt.cpp:250).
picojson's _parse_array/_parse_object is unbounded recursive
descent; both parses run before the MAC verdict reaches the
caller, so an unauthenticated peer could submit a token in
Authorization: Bearer whose b64-decoded JSON nests deep enough
to blow the worker thread's stack (musl pthreads cap at 128 KiB).
Real JWTs are flat — a cap of 32 leaves ample headroom; the
scan is O(body length) with no allocations.
- src/webapi/Api.cpp: pull the same opener-count cap into
ParseJsonObjectBody so every mutating-body parse site is covered,
and route HandleLogin through ParseJsonObjectBody instead of
calling picojson::parse directly — HandleLogin is reachable
unauthenticated, so it needs the same defence as the JWT
surface. Forward declaration in the helpers-anon-namespace block
near the top of the file so HandleLogin (defined above
ParseJsonObjectBody) can call it.
- unittests/tests/JwtTest.cpp: two new regressions
(DeeplyNestedPayloadRejected, DeeplyNestedHeaderRejected) build
a token whose nesting is 200 levels deep and sign it with the
matching HMAC so the MAC compare succeeds and the depth cap is
the only available reject path. Suite goes 23 → 25 cases, all
21 ctest binaries still green.
Six batches landed against the final pre-PR review. All 21 ctest
binaries still green; JwtTest grew to 27 cases (depth-cap + iat
mandatory regressions); AuthTest dropped from 6.5 s → 0.44 s
after clock injection.
Batch 1 — quick wins
- HttpServer.cpp: route 5xx body through CJsonWriter::ValueString
(H-S1).
- Auth.cpp Check(): evict stale failure stamps the same way
NoteFailure does (H-S2).
- EventDiff.cpp: capture-on-first-call thread-id assertion in
EmitDiffsAndUpdate; hard abort if a second thread publishes
(H-S3).
- PathPatterns.{h,cpp} + Api.cpp: LooksMalicious(path) defence-
in-depth — reject NUL / %00 / %2e%2e / literal ".." segments
before routing (H-S11).
- Jwt.cpp Verify: reject tokens > 4 KiB before any Base64Url walk
(M-S10).
- Api.cpp ?channels= cap at 32 tokens (M-S13).
- Api.cpp HandleLogout: revoked-token now 200 noop, not 401
(M-S14).
- Api.cpp: kSessionCookieAttrs as the single source of truth
for Set/Clear cookie attribute strings (M-S11).
- App.h: rename PartfileRleState() →
PartfileRleStateRequireStateWriteLock() to encode the
precondition (M-S2).
- State.cpp: comment the tick-end-stamping skew on m_snapshot_at
and the "events atomic, state isn't" invariant (M-S3 + M-S12).
Batch 2 — auth / JWT hardening
- Jwt.cpp Verify: iat is now mandatory; additionally cap
(exp - iat) ≤ TOKEN_LIFETIME_SECONDS + skew so even a
compromised secret can't mint a 100-year token (H-S10).
- Api.cpp HandleLogin: drop the JWT from the JSON body by
default — browsers get the cookie only; bearer clients opt
in via Accept: application/jwt or ?type=bearer (H-S12).
- Jwt.h: signing secret now stored as CryptoPP::SecByteBlock so
the live key gets scrubbed at destruction; constructor also
zeroes the caller's vector after copying (M-S1).
- AmuleApiConfig.cpp: first-run amuleapi.conf write routed
through WriteFileAtomic0600 — no truncated configs on mid-
write crash (M-S5).
- Api.cpp: new generic 401 rate limiter (30/60s, 5-min lockout)
applied to every auth-protected endpoint via
AuthenticateRequestRateLimited, including HandleLogout's
inline-soft path (M-S7).
- Api.cpp ResolveServerEcidByAddress: drop the s.address
string fallback; require IPv4-shaped match (M-S9).
- curl-tests: opt every login URL into ?type=bearer so the
script harness keeps reading the token out of the body.
- unittests/tests/JwtTest.cpp: two new regression cases
(MissingIatRejected, ExpIatLifetimeCapExceeded) and
EmptyJtiRejected updated to include a valid iat so it
exercises empty-jti rather than the new mandatory-iat path.
Batch 3 — SSE concurrency
- Api.cpp DispatchEvents: post-Drain gap detection — if the
cursor fell off the ring between drains, emit synthetic
resync (reason=gap) and restart from newest. Previously
only reconnects caught gaps; a cold-start tick on a 5K-
download library would publish ~5K events, evict 4900,
and the live drainer would silently miss them (H-S4).
- EventBus.{h,cpp}: Shutdown() + IsShutdown() latch. App's
OnExit calls Shutdown() before tearing down the
dispatcher; Drain wakes blocked subscribers, the SSE
worker loop polls IsShutdown() and exits — closes the
UAF window where a detached worker held m_dispatcher
across dispatcher.reset() (H-S7).
- HttpServer.{h,cpp} + Api.{h,cpp}: new StreamingPreflight
callable runs synchronously on the I/O thread BEFORE the
SSE worker thread spawns and BEFORE the 32-slot
streaming-session budget is touched. Unauth peers get
401/429 without burning a slot for the read-timeout
window (H-S8).
- App.cpp refresher loop: wall-clock-bounded to a 1 s
target cycle (was a fixed 4×250 ms sleep regardless of
tick duration) + WARN log on any tick > 3 s — typically
signals EC-mutex contention (H-S5).
Batch 4 — cache / perf
- TtlCache.h: split GetOrFetch into "fresh? return cached"
vs "claim inflight via condvar, drop mu, run fetch,
broadcast". A 30 s amuled stall on /stats/tree no longer
parks every concurrent reader (H-S6).
- Api.h + Api.cpp: snapshot-versioned ETag memoization keyed
on (request target, m_state.SnapshotAt()). Previously every
200 GET/HEAD ran MD5 over the (potentially multi-MB) body;
on a 10K-shared-file daemon that was the dominant CPU cost
on the safe-method path (M-S6).
- Pagination (M-S8) deferred per user direction.
Batch 5 — tests
- Auth.h + Auth.cpp: clock injection into CRateLimiter via
optional Clock parameter (defaults to std::time(nullptr));
AuthTest's sliding-window / lockout-expiry tests now run
with a controllable clock instead of real-time sleeps —
AuthTest dropped from 6.5 s → 0.44 s (H-T1).
- JwtTest MalformedBase64Rejected: drop the misleading
"constant-time" framing since no timing assertion exists
(H-T2).
- phase4.sh: poll /status until 200 instead of sleep 2 —
doesn't drift to disconnected under Windows VM load
(H-T3).
- phase4.sh test 4: real guest login instead of the comment-
only stub; _die if guest pass isn't configured so the
fixture gap surfaces (H-T4).
- phase3.sh: add 5th-attempt 429 check (was only the 8th)
so the lockout boundary is pinned (M-TD6).
- StateTest: ASSERT_TRUE(observed > 1000) instead of > 0 so
the tear-detection harness can't pass on a single
observation (M-TD7).
- AmuleApiConfigTest: document why the mode-bit test is
POSIX-only and what a future Windows port would land
(M-TD8).
Batch 6 — docs / man / cmake
- AmuleApiConfig.cpp: document the deferred --rotate-jwt-
secret CLI; manual rotation via "delete the file + restart"
works today (M-S4).
- cmake/options.cmake BUILD_AMULEAPI block: document the
hard-fail dependency chain (cryptopp via NEED_LIB_EC,
Boost via unconditional cmake/boost.cmake) so a future
refactor doesn't accidentally let amuleapi soft-disable
(M-TD1).
- docs/QUICKSTART-AMULEAPI.md: callout disambiguating
--port=4712 (EC port amuleapi talks TO) vs the 4713 HTTP
listener (port REST clients talk TO); curl example
switched to ?type=bearer (M-TD2).
- docs/man/amuleapi.1.in: drop the non-existent -V short
flag from SYNOPSIS + OPTIONS; the wired form is
--version (M-TD3 + M-TD4).
- docs/man/CMakeLists.txt: comment-only ack of the
translated-manpage gap — adding amuleapi to MANPAGE_BINARIES
today fails the build because po/manpages-*.po has no
amuleapi strings. English-only until 3.1; explicit
wiring instructions in the comment for whoever lands the
translator pass (M-TD5).
…sweep Two fixture bugs surfaced when run-all.sh was actually re-run on top of the sweep commit; both were introduced by the batch-5 changes that addressed reviewer M-TD6 / H-T4. - phase3.sh — the 5th-attempt boundary assertion (M-TD6) had the off-by-one in the wrong direction. Rate-limiter Check() runs BEFORE the password compare; NoteFailure() runs AFTER. So with threshold=5, the 5th attempt arms the lockout but still returns 401 (the password compare failed and the response was already prepared); the FIRST 429 lands on attempt 6 when Check() sees the armed lockout. The assertion now asserts 401 on attempt 5 and 429 on attempt 6, which is what the limiter actually guarantees. - phase4.sh — the new guest-bearer check (H-T4) referenced $GUEST_PASS without setting a default. run-all.sh configures guestpass via --set-guest-pass, but a standalone phase4.sh invocation has no GUEST_PASS in the environment and `set -u` bails. Default to `guestpass` to match run-all.sh's daemon fixture, and document the env var alongside ADMIN_PASS in the script header.
…cOpsImpact) Seven findings from the second-round review. All 21 ctest binaries still green; run-all.sh full curl matrix still green (660+ assertions across 26 phase scripts). - HttpServer.cpp:153 — set explicit m_parser->header_limit(16 KiB). Beast's default header_limit varies across versions (8 KiB in current upstream, larger on older). Without an explicit cap, a drip-feed attacker who slowly streams header lines grows the flat_buffer until the 10 s read timeout catches them — but that's 10 s × pool of concurrent peers of memory pressure. The explicit cap pins the per-peer ceiling at 16 KiB regardless of upstream defaults. (amule-project#3) - HttpServer.cpp:200-220 — replace the e.what()-in-body 500 with a generic "internal server error" message; log e.what() server- side via stderr. Information-disclosure defence: a future header-driven throw site (or picojson's offset-with-byte error messages today) would otherwise reflect caller-supplied bytes (Authorization fragments, Cookie content) back to the requester via the 500 body. The H-S1 fix in the prior sweep handled JSON escaping but kept the disclosure surface; this closes it. (amule-project#2) - App.cpp TextShell — bind-time hard gate. Refuse to start when BindAddress is non-loopback AND both admin/guest passwords are empty. The prior WARN-only behaviour meant a daemon started against a routable interface BEFORE the operator ran --set-admin-pass accepted TCP, refused every login (503 login_disabled), but still exposed the unauth surface (/version) and would silently leak any future read-without-auth endpoint. Loopback binds are unaffected — that's the documented first-run flow. (amule-project#1) - HttpServer.cpp + App.cpp shutdown — Listener::Stop now cancels every live SSE session's socket via a new Session::RequestCancel posted onto the session's strand. The new g_live_streams weak_ptr registry tracks every session that holds a streaming slot. Without this, a worker blocked inside a synchronous Beast write to a slow peer would hold the connection until the kernel TCP timeout (~minutes) and CHttpServer::Stop would join the io_context thread indefinitely waiting for the workers. Now the close fires, the write fails EPIPE, the worker exits cleanly, the last shared_ptr ref drops, and shutdown completes in bounded time. (amule-project#6) - App.cpp refresher loop — fail-loud on sustained EC blackout. After ~30 s of consecutive failed ticks, log a sharp WARN to stderr; after ~5 min, set g_shutdownRequested so a supervisor (systemd, launchd, docker restart=always) can bring the pair back. Previously the refresher would silently loop forever retrying a dead amuled, with the daemon serving 503 ec_unavailable from every endpoint with no visibility into the underlying cause. (amule-project#7) - EventBus.{h,cpp} + EventDiff.cpp — new CEventBus::PublishBatch(vector<pair<name,data>>) takes one lock acquisition + one notify_all for the whole batch. DiffMap now accumulates removed/added/updated tuples and emits via the batch API. The cold-start tick on a 5K-download library was previously firing ~5K individual notify_all cycles inside the refresher loop (each going through every drainer's cv mutex), which dominated the tick wall-clock. Single-publisher invariant + monotonic id assignment both preserved (the batch assigns inside the same lock as the per-event Publish). (amule-project#9) - App.cpp OnRun — sigaction with explicit sa_mask instead of std::signal. std::signal has implementation-defined "reset-to-SIG_DFL after firing" semantics on some platforms (notably musl/Alpine and legacy BSDs); a second SIGINT could default-terminate the daemon mid-shutdown. sigaction with SA_RESTART gives portable stay-installed semantics across every relevant target and prevents EINTR on blocking EC reads during shutdown. Windows lacks sigaction; std::signal fallback retained there. (amule-project#5 — Alpine/musl support)
Two new files covering the complete v0 API surface: - docs/amuleapi/REFERENCE.md (~950 lines) Endpoint catalog: every route under /api/v0/, grouped by resource (system, auth, downloads, clients, shared, servers, categories, preferences, networks, kad, logs, stats, search). Per endpoint: HTTP method(s), auth role (NONE/GUEST/ADMIN), query/path params, request body schema, complete curl example, sample success response, and the error codes the handler can emit. Sections upfront for the shared model: auth carriers (bearer vs cookie), role gating, rate limiting, JWT structure, error envelope, ETag and conditional GET, CORS surface, path validation, request size limits. Closing error-code catalog so clients have one table to alert on. - docs/amuleapi/EVENTS.md (~310 lines) SSE contract: connect flow, headers, initial reassurance chunk, frame format, channel filter (downloads/shared/servers/clients/ status/logs), 15 s heartbeat semantics, Last-Event-ID reconnect matrix, live-path gap detection, and the synthetic resync event. Catalogs every event the bus emits with its payload shape: download_*, shared_*, server_*, client_*, status_changed, log_appended, plus the filter-bypass resync. Closing notes on the single-publisher invariant and shutdown behaviour. Cross-linked from: - docs/QUICKSTART-AMULEAPI.md (replaced the "OpenAPI is a follow-up" hedge with pointers to the new files). - docs/man/amuleapi.1.in (the man page promised docs/api/; updated to point at docs/amuleapi/REFERENCE.md and docs/amuleapi/EVENTS.md). No source changes; pure docs. Closes the "where's the API spec" gap that the man page was advertising against an empty directory.
The man page originally promised docs/api/; restore the directory name to match. Renames the two reference files in place and updates the cross-links in QUICKSTART-AMULEAPI.md and the amuleapi(1) man page. No content changes.
The doc examples carried 176.123.5.89:4725 verbatim, lifted from the phase11.sh curl regression test where it was used because it's a known-live ed2k server. That's fine for a smoke test but wrong for user-facing docs — implies endorsement and links a third-party operator's address to the project. Swap to 203.0.113.5:4242 (RFC 5737 TEST-NET-3, reserved for documentation) throughout REFERENCE.md and EVENTS.md and align the port in the example status response so the two examples stay consistent. No structural changes.
…nt cleanup
Code:
- EventDiff.cpp/h: every ToJson(*Snapshot) rewritten to emit the same
byte-for-byte shape as the matching REST list-item writer in
Api.cpp. Every Equal(*Snapshot) expanded to compare every emitted
field so SSE _updated fires on any movement (the original gap was
shared_updated never firing on xfer counter movement). New
ToJsonStatusEvent(Status, Kad, ec_connected) produces the nested
REST /status envelope; new Equal(KadSnapshot). EmitDiffsAndUpdate
reads state.Dashboard() so status + kad + ec_connected are diffed
against the prior tick as one rollup. LastSeenState gains
KadSnapshot kad + bool ec_connected.
- Api.cpp: HandleNetworksConnect accepts {"network":"ed2k"|"kad"|
"both"} matching /networks/disconnect; routes to EC_OP_SERVER_CONNECT
/ EC_OP_KAD_START / EC_OP_CONNECT. HandleKadConnect +
HandleKadDisconnect deleted as strict aliases of the granular form.
/search/results fires EC_OP_SEARCH_PROGRESS in the same single-flight
TtlCache miss; response carries progress.{percent, complete} so
empty results aren't ambiguous between "no search", "in flight",
"finished with zero hits". Api.h adds SearchCacheValue + new
TtlPair_Search alias.
Tests:
- phase5e.sh: replace dedicated /kad/{connect,disconnect} assertions
with /networks/{connect,disconnect} body-selector assertions; add
network=ed2k and bogus-selector negative cases.
- phase6.sh: assert progress envelope on /search/results.
Docs:
- REFERENCE.md: TOC linking every endpoint section; clear_completed
cross-references DELETE /downloads/{hash}; kad/bootstrap body
corrected from a wrong {server_address: ...} to the real {ip,
port} shape; /networks/connect doc updated for the body selector
and /kad/{connect,disconnect} removal noted; /search/results doc
shows progress envelope.
- EVENTS.md: every event sample replaced with the aligned shape;
cross-resource intro states _added/_updated are byte-identical to
the REST list-item shape they mirror.
Comment cleanup:
- Removed PLAN.md / PLAN §N / internal-phase-naming references from
in-code comments per the standing policy. Replaced "deferred to
v0.2" parentheticals with their removal. Section-divider phase
comments and pure phase-marker comments deleted. Strip is partial;
the bulk-apply tool tripped a regex bug; this commit is the
post-recovery point and a follow-up sweep will finish.
All 21 ctest binaries green; build clean.
ApplyGetUpdateToShared inserted a first-time SharedSnapshot with
hash="000…0" (zeroed CMD4 from GetTagByNameSafe(EC_TAG_PARTFILE_HASH))
and name="" whenever a partfile transitioned to shared (≥1 chunk just
completed) in a tick where EC_TAG_PARTFILE_HASH / _NAME were CValueMap-
suppressed by amuled — the partfile's identity hadn't changed since
its earlier non-suppressed update, only EC_TAG_PARTFILE_SHARED flipped.
The client saw a ghost row in /shared with empty hash + name.
Repro:
POST /api/v0/search/results/{hash}/download → wait until partfile
accumulates ≥1 chunk → GET /shared returns ECID with zeroed hash.
Fix: gate first-time insert on PARTFILE_HASH presence; if absent,
fall back to the downloads cache snapshot (populated this same tick
from the partfile's earlier non-suppressed update). If neither has
identity, defer the insert — the next tick where amuled emits a true
identity refresh catches it.
Verified on local repro: ECID 1150 (debian-10.13.0-amd64, 22 MB / 2+
chunks) appears in /shared with proper hash + name; ECIDs without
completed chunks correctly absent. No ghosts across stability poll.
Previously /downloads/{hash}, /downloads/{hash}, /downloads/{hash}, and
/shared/{hash} only matched the 32-char MD4 hex. Clients that already had
the ECID from a list view (the canonical id in our State maps) had to
carry the hash separately for any detail / mutation call.
Disjunctive route: capture renamed to {key}, handler dispatches on
shape — decimal 1..10 digits ≤ 0xFFFFFFFF → ECID (O(1) map lookup);
anything else → hash (case-folded scan). The two grammars are
non-overlapping, so no ambiguity / type prefix needed.
Scope:
GET /api/v0/downloads/{key} — HandleDownloadDetail
PATCH /api/v0/downloads/{key} — HandleDownloadPatch
DELETE /api/v0/downloads/{key} — HandleDownloadDelete
PATCH /api/v0/shared/{key} — HandleSharedPatch
POST /api/v0/search/results/{hash}/download deliberately out of scope —
that's a search-result hash (transient ID), not a partfile identity.
Adds State::FindDownloadByEcid, State::FindShared, State::FindSharedByEcid
to round out the lookup surface (FindDownload by hash already existed).
The ECID variants are O(1) map lookups vs the hash variants' linear
scans; downstream EC ops still address by MD4 hash, so handlers read
the hash off the resolved snapshot rather than the URL.
phase4b.sh + phase5f.sh gain 5 new assertions covering the ECID path;
REFERENCE.md documents the disjunction inline at the relevant endpoint
sections + in the index.
Adds a static-file dispatcher under non-/api/ paths so an operator can
ship a frontend bundle alongside amuleapi without a separate webserver.
Discovery mirrors amuleweb's GetTemplateDir
(src/webserver/src/WebInterface.cpp): bundle Resources/ on macOS, then
configure-time AMULEAPI_STATIC_DIR, then wxStandardPaths fallback —
so a standard install (`make install`, MSYS2 portable, .app bundle)
serves the bundled placeholder with zero conf edits, and [Server]/
StaticRoot still wins as an explicit override.
Containment + caps:
- POSIX symlinks resolved via realpath(); escapes from root are
rejected. Windows uses _fullpath() (lexical-only, since symlinks
require elevation there) plus a normalised prefix check that
survives trailing slashes in the configured root.
- 16 MiB per-asset cap stat'd before reading.
- mtime+size ETag so reloads short-circuit to 304 via
If-None-Match.
- Extension-less misses fall back to index.html (SPA-friendly);
misses that carry an extension 404 honestly so a broken bundle
is visible.
ResolveWithinRoot + IsDir live in src/webapi/StaticFs.{h,cpp} so the
unit tests (StaticFsTest, 11 cases) can link them without dragging in
the full dispatcher. Curl-level wire tests in phase12.sh cover ETag
round-trips, SPA fallback, symlink rejection, and the size cap.
Verified end-to-end on macOS (LSCopy bundle path), Linux (cmake
install path), and Windows (portable install path).
amuled's EC suppresses the whole `EC_TAG_PREFS_CATEGORIES` block when no custom categories exist, even though category 0 (the default incoming dir) is always present in amuled's data structures — `GetCatCount() >= 1` always, and `AddDownload(category=0)` works unconditionally. Once the first custom category is created amuled starts emitting both index 0 and the new one. The empty-then-populated wire shape forces every client to special- case the no-custom-categories tick instead of just iterating the list. Normalise at the API layer: if amuled didn't return index 0, prepend a synthetic entry with the same defaults amuled itself would emit (empty title/path/comment, color 0, priority_code PR_LOW per `CPreferences::LoadCats`). Refresher / state stay faithful to the EC wire so internal callers that already handle the empty case (e.g. created-index lookup in HandleCategoryPost) keep working unchanged. Reported in amule-project#201 review.
`phaseN.sh` was a development artifact — useful while landing each
chunk under review, but unhelpful once the whole surface exists and
contributors want to find the script that covers a specific endpoint
or transport.
Renames the 27 scripts to `NN-<what-it-tests>.sh`. The numeric prefix
preserves the canonical execution order in run-all.sh (auth before any
mutation, refresher-consolidation tests before later read smokes,
CORS / static-frontend after the API surface so transport failures
don't mask earlier regressions). The descriptive suffix says what's
under test, so `ls` reads as a feature inventory rather than a commit
log.
Also updates:
- run-all.sh PHASES array + the per-script special-cases (static-
frontend dir provisioning, CORS-rebounce env, 02-auth lockout
hint).
- Every script's internal headers: banner, usage example, mktemp
prefix, `/tmp/amuleapi-…` paths, and cross-references to other
scripts.
Narrative comments that mention "Phase N" as a *development phase*
(e.g. "the wire decision in Phase 8c") are left alone — those refer
to PR-review history, not script filenames.
Full suite re-verified after the rename (27/27 green on macOS).
GET /search/results used to read amuled's raw EC_TAG_SEARCH_STATUS
directly, which is ambiguous: raw=0 means "no search" OR "Kad in
progress" OR "global queue not yet populated" OR "global finished
(m_searchInProgress just flipped off)"; raw=100 means "global queue
empty at start" OR "global all done briefly before reset". The
visible symptom was {progress:{percent:100, complete:true}}
immediately after POST /search (queue-empty window) followed by
progress=0 a few seconds later (mid-run), then nothing useful at
completion (raw resets to 0).
The lifecycle now lives in the refresher:
- POST /search calls m_state.MarkSearchStarted(kind) — wipes the
results map, sets active=true with the requested kind.
- RefresherTick polls EC_OP_SEARCH_RESULTS + _PROGRESS every
tick while active=true and feeds the raw value, results-count,
and now into AdvanceSearchProgress.
- AdvanceSearchProgress is a pure function in Refresher.cpp that
state-machines the raw value into normalized (percent, complete,
active, saw_in_progress). Disambiguates via the kind, prior
saw_in_progress, results-map size, and a 30 s defensive timeout.
Kad bumps started_at on each new result so a productive Kad
search isn't cut off; saw_in_progress excludes the timeout for
a genuinely climbing global.
- GET /search/results reads straight from state (no more TtlCache,
no on-demand EC fetches). Surface now includes progress.active.
EventDiff publishes search_result_added per new ECID in the results
map and search_finished on the complete=false->true transition. The
baseline is gated on search_initialised so a fresh tick after
MarkSearchStarted doesn't double-fire.
RefresherTest grows 12 cases covering every state-machine branch:
global climb-then-reset, raw=100 caught vs missed, queue-empty mask,
fast-complete (missed climb), local 0xffff path, local raw=0
finalize, Kad raw=0 stays active, Kad 0xfffe completion, Kad
deadline extension on new result, timeout after 30 s of silence,
timeout skipped for climbing global, inactive-state graceful exit.
19-search regression assertion: progress.complete must be false
immediately after POST /search. 22-sse-diff-emission gains a
search_result_added / search_finished smoke using a local search
(works even on a fully-disconnected daemon).
Reported in amule-project#201 review.
The prefix-based event_channel fallback in Api.cpp already routes search_result_added / search_finished to channel "search", so ?channels=search works in practice. Make it explicit so the mapping table stays a single source of truth alongside docs/api/EVENTS.md and a future event family doesn't end up routed via the silent fallback. EVENTS.md: add the search channel row to the channels table and catalog both event payloads with the per-search ECID-stability note (amuled assigns a fresh ECID via CECID's monotonic s_IDCounter on every new search, even for the same file across repeated queries). 26-rfc-followup-endpoints.sh: the channels=downloads,status exclusion check now also forbids search_*; a new positive check asserts channels=search delivers search_finished and excludes download / status / etc. Refs amule-project#201 review.
…Snapshot amuled's downloads (CPartFile in downloadqueue) and shared (CKnownFile in sharedfiles + partfiles with PARTFILE_SHARED=true) views share a single per-process ECID counter (CECID::s_IDCounter) and a unified encoder-side CObjTagMap. Mirroring that on the receiver side avoids the class of bugs we kept patching: - The /shared "ghost" rows we fixed in e57cbf9 were caused by the shared walker hitting an ECID that the downloads walker had already populated in a SEPARATE m_shared cache, with the PARTFILE_HASH child tag CValueMap-suppressed (already sent on a prior partfile-walker tick). We worked around it by composing a dl_identity_fallback map and passing it to the shared walker. - With one ECID-keyed FileSnapshot map, the shared walker finds the entry the downloads walker just touched, with hash + name already set. It just flips is_shared=true and merges its own fields. The fallback compose disappears. Schema: struct FileSnapshot { uint32_t ecid; string hash, name, ed2k_link; uint64_t size; string priority; bool is_downloading, is_shared; struct DownloadSide { ... } download; // partfile-walker fields struct SharedSide { ... } shared; // knownfile-walker fields }; Role transitions: - Downloads walker: on PARTFILE entry, sets is_downloading=true and merges f.download. On FILE_REMOVED, clears is_downloading + resets download to default; drops the entry entirely if !is_shared. - Shared walker: same shape for is_shared/f.shared. PARTFILE_SHARED= false clears is_shared and resets the shared sub-block; the entry stays in the unified map (entity eviction is FILE_REMOVED's job). - Order-independent: running downloads-then-shared and shared-then- downloads converge to the same final state. Reset-on-transition guarantees /downloads can never serve stale upload stats, and /shared can never serve stale download stats — same lookup hitting either view always reads from a freshly-walked sub-block. API surface unchanged: GET /downloads filters m_files by is_downloading; GET /shared filters by is_shared. Same wire shape; clients see no difference. Bonus from the unified map: FindDownload(hash) / FindShared(hash) now hit a parallel std::unordered_map<hash, ECID> index (m_hash_to_ecid) that's rebuilt at the end of every MutateDownloads / MutateShared. Was O(N) linear scan; now O(1) hash lookup + O(log N) map find + role-flag check. REFERENCE.md updated. RefresherTest migrated to FileSnapshot + role-flag assertions (32 cases, all pass). EventDiff diffs the unified files map with role-flag awareness: download_{added,updated,removed} fires on is_downloading transitions, shared_{added,updated,removed} on is_shared transitions. A single tick can emit both for one ECID (e.g. partfile becoming shared). Full curl suite green: 27/27. Refs amule-project#201 review.
Hashes (MD4, 32-char hex) are the only identifier the API exposes for
partfiles and shared files. File ECIDs were never wire-public:
- /downloads/{key} → /downloads/{hash} (PATCH/DELETE/detail)
- /shared/{key} → /shared/{hash}
- removed "ecid" from /downloads, /shared, search-result event payloads
- /clients: upload_file_ecid / download_file_ecid → upload_file_hash /
download_file_hash. The walker now resolves EC_TAG_CLIENT_UPLOAD_FILE
and EC_TAG_CLIENT_REQUEST_FILE into MD4 hashes at refresher-tick time
using an ECID→hash snapshot built from the unified file map.
Client and server ECIDs stay — they are still the URL key for
/clients/{ecid} and /servers/{ecid}, and there's no hash for either.
Tests updated: curl phases 04 + 17 drop the file-ECID assertions and
the PATCH /shared/{ECID} round-trip. All 22 cpp unit tests still pass;
all 27 curl phases still pass against the local amuled.
Closes the file-ECID strip discussed in amule-project#201.
Disambiguates from the file ECIDs we just stripped — `ecid` alone made
it look like a file identifier, when it's actually the peer's identity.
Affected surfaces:
- GET /clients[] payload
- client_added / client_updated SSE payloads
- client_removed SSE payload (`{ "client_ecid": N }`)
server_ecid stays as plain `ecid` — same field name as before, still the
URL key for /servers/{ecid}.
Curl phases 10 + 26 updated. All 22 cpp unit tests + 4 affected curl
phases (10, 22, 24, 26) pass.
CState's unified file map was std::map<uint32_t, FileSnapshot> with a parallel std::unordered_map<string, uint32_t> hash→ECID index that got rebuilt from scratch on every MutateDownloads / MutateShared call (one rebuild per refresher tick). That O(N) per-tick scan was the dominant cost on large libraries — 100k files × 2 walkers × 1 Hz = 200k iterations per second spent re-deriving an index that almost never changed. Replace both with a single FileMap wrapper class that owns both data structures: * std::unordered_map<uint32_t, FileSnapshot> for ECID-keyed storage * std::unordered_map<string, uint32_t> for hash → ECID lookups emplace() and erase() update the index inline (O(1) avg per change); hash never mutates within an ECID so no other surface needs hooks. Walkers keep their .find()/.emplace()/.erase() calls unchanged — the wrapper exposes the same iterator surface. Per-tick rebuild deleted. Lookups now both O(1) avg: * ECID → FileSnapshot : FileMap::find(ecid) * hash → FileSnapshot : FindEcidByHash(hash) + find(ecid) CState::Downloads() / Shared() / Files() vectors are no longer in ECID-ascending order (unordered_map iteration is bucket-defined). Consumers already sort on their side (ngosang's feedback on PR amule-project#201), so this is wire-visible but harmless. REFERENCE.md tidied — the previous /downloads/{key} prose was left over from the file-ECID-strip pass; consistent {hash} naming throughout now, no internal-state details leaking onto the public API surface. 22/22 cpp unit tests + 27/27 curl phases pass.
EVENTS.md never spelled out the call ordering for the snapshot + stream combination. The naïve REST-first → SSE-second pattern silently drops events that fire between the two — a one-line bootstrap bug that's easy to write and hard to spot in testing. New "Bootstrap: snapshot + stream" section explicitly: - Names the race (REST→SSE loses events fired in the gap) - Documents the correct order (open SSE first, then GET, apply events as upserts on top of the snapshot) - Explains why the merge is consistent (events for tick N are emitted after the state mutation for tick N is committed) - Gives a JS bootstrap example with parallel snapshot fetches - Points at the resync recovery rules for the failure modes
Bootstrap-companion gaps the doc didn't cover, found while sanity-checking the bootstrap example against real-world client paths: - Connecting / Browser example: add an `error` listener with the `readyState === CLOSED` terminal-failure check, and clarify that the native browser EventSource API can't carry an `Authorization` header so browser bearer-on-SSE requires a polyfill. - Channels and filtering: note that the snapshot fetches in the bootstrap must mirror the `?channels=` filter; pulling unsubscribed collections leaves them silently stale. - Event catalog intro: call out that `status_changed` is a full envelope replacement and `log_appended` is an append — they don't fit the `_added` / `_updated` / `_removed` upsert pattern. - Search / Logs channel sections: explicit pointer back to Bootstrap noting that `/search/results` and `/logs/amule` aren't fetched there by default, with the case for adding them. Plus a tidy-up: every `§Section` cross-ref now uses a clickable markdown anchor link instead of the section-symbol convention (matches the REFERENCE.md links and renders properly on GitHub).
The existing EC_TAG_SEARCH_STATUS overloads a single uint32 to mean a lot of different things — raw=0 covers "no search", "Kad in progress", "global queue not yet populated", and "global just finished"; raw=100 covers "global queue empty at start" and "global all done briefly". 0xfffe = Kad done, 0xffff = global done (or local in progress). The desktop GUI and amulecmd live with this because their UI state + explicit notifies cover the gaps; for headless consumers driving a JSON wire surface (amuleapi) it forces a defensive state machine (saw_in_progress tracking, 30 s timeout, started_at bump-on-result). Add three new optional tags on the EC_OP_SEARCH_PROGRESS response so modern consumers can read the lifecycle directly without the sentinel decode, while pre-3.1 consumers (amulegui, amulecmd, amuleweb) keep reading EC_TAG_SEARCH_STATUS unchanged: EC_TAG_SEARCH_LIFECYCLE_STATE uint8 0=idle / 1=running / 2=finished EC_TAG_SEARCH_LIFECYCLE_KIND uint8 0=local / 1=global / 2=kad EC_TAG_SEARCH_RESULT_COUNT uint32 size of m_results[m_currentSearch] Derivation in CSearchList::GetSearchLifecycleState(): - m_currentSearch == -1 → IDLE (no search this session) - m_searchType == KadSearch → m_KadSearchFinished ? FINISHED : RUNNING - otherwise (Local / Global) → m_searchInProgress ? RUNNING : FINISHED EC_TAG_SEARCH_STATUS is unchanged. amulegui / amulecmd / amuleweb iterate tags they recognise and skip unknowns — verified by a full build of all four binaries (amule, amuled, amulegui, amulecmd) against the same EC core. No per-searchID tracking yet — Get_EC_Response_Search still pins the sentinel 0xffffffff. Lifting that is a separate (larger) change.
amuleapi pins a daemon version carrying the new EC_TAG_SEARCH_LIFECYCLE_* tags landed in the previous commit, so there's no reason to keep the sentinel-disambiguation state machine around as a fallback. Replace it with a single ten-line mapping from (lifecycle_state, raw_pct) to (active, complete, percent). Refresher.cpp/h: AdvanceSearchProgressFromLifecycle becomes the only AdvanceSearchProgress. The legacy state machine, its 60-line comment block, and the defensive 30 s timeout / saw_in_progress / Kad-deadline- bump-on-result tricks are all gone. RefresherTick.cpp: drops the two-path branch that picked between legacy and lifecycle. Reads both EC_TAG_SEARCH_STATUS (for the global percent) and EC_TAG_SEARCH_LIFECYCLE_STATE every tick; passes both to the helper. State.h: SearchProgressSnapshot loses saw_in_progress, raw, started_at, last_results_count — all of which only existed to drive the old disambiguation. Keeps active / kind / percent / complete (the four fields surfaced by GET /search/results and the search_finished SSE). State.cpp: MarkSearchStarted no longer stamps started_at — that field no longer exists. RefresherTest.cpp: 12 legacy tests covering every branch of the sentinel-decode machine drop to 5 trivial tests of the new mapping (running with global percent, running masks percent for Kad / local, finished sets complete, idle defensive zero). Net diff: -201 lines, same surface contract on /search/results and search_finished, and zero magic-number sentinel decoding on the amuleapi side. 22/22 cpp unit tests + 27/27 curl phases pass against the patched daemon.
Add a static single-page web frontend for the amuleapi daemon, built on vendored preact + htm with no build step, no bundler and no runtime CDN. The UI mirrors the aMule/amuleGUI desktop app: a top toolbar of icon nav buttons, a bottom status bar, and views for Networks (ED2K + Kad), Searches, Downloads, Shared files, Statistics, Preferences and Categories. Data is live over SSE with polling fallback, uPlot powers the stats/Kad graphs, and a light/dark theme follows the system preference. The bundle lives under src/webapi/static/, served by the existing StaticFs static-frontend server: an empty StaticRoot auto-discovers the installed amuleapi-static tree, so an installed daemon serves the UI out of the box. tools/run-dev.sh and tools/update-vendor.sh target static/.
The static web frontend and the docs/api reference had both drifted from
the JSON the amuleapi backend actually emits. Verified each field against
src/webapi/Api.cpp and src/webapi/EventDiff.cpp.
Frontend (src/webapi/static):
- downloads.js: register the clients resource under id "client_ecid"
(was "ecid"). The backend keys clients by client_ecid, so the old id
collapsed the whole uploads/clients list to a single "undefined" row.
- events.js: buffer-then-replay on seed() so a delta (especially a
*_removed) arriving while a snapshot fetch is in flight is no longer
clobbered by the snapshot replacing the collection, per EVENTS.md.
- events.js + search.js: wire the SSE search channel
(search_result_added / search_finished), reconciling live upserts with
the existing poll by hash; poll stays as the full-set reconciler and
fallback when the stream isn't live.
Docs (docs/api), corrected to match the backend:
- REFERENCE.md: /status now shows kad.network; /clients and /servers
examples completed; /kad lists firewalled_udp, ip, indexed, buddy;
/logs/amule returns {lines,...} not {log}; /logs/serverinfo returns a
text string (not a lines array); /stats/tree is {nodes:[{label,
children}]} not {tree}; /stats/graphs points are objects
{t,t_unix,value} with a full envelope; /search/results items use
sources{total,complete}+already_have+rating and progress gains active.
- EVENTS.md: log_appended payload is {lines} (no buffer; amule log only);
search_result_added carries no ecid and flattens sources to an int.
ngosang
force-pushed
the
feature/local-amuleapi-web
branch
from
d886655 to
e134f06
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DO NOT MERGE!