Skip to content

Redact the Scraper API's x-debug header before logging it - #37

Merged
jehrr merged 1 commit into
mainfrom
xdebug-redaction
Sep 21, 2026
Merged

jehrr merged 1 commit into
mainfrom
xdebug-redaction

Conversation

@jehrr

@jehrr jehrr commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

SECURITY.md in this repo names the Scraper API's x-debug response
header as one of three places credentials reach a log unmasked. The client
then logged it verbatim:

debug = resp.headers.get("x-debug")
if debug:
    logger.info("x-debug: %s", debug)

The API echoes back the task it ran, so a run driven through a credentialed
CDP endpoint put that endpoint's username and password into the log, and a
key passed as a query parameter would go the same way. A rule stated in one
file and broken in another is invisible to both.

Measured across the family on 2026-09-21: the header was logged raw in 27
repos
. This is that fix, applied here.

What it does

_redact_debug_header() masks two shapes, globally — a masker that
handles the first occurrence prints the password the other four times and
looks like it is working:

  • scheme://user:pass@host anywhere in the text, not only in a string that
    is entirely a URL;
  • key= / clientKey= / token= / api_key= values.

Redaction rather than an allowlist of fields, deliberately: x-debug is the
remote API's own metadata and its shape is not ours to pin, so an allowlist
would silently drop the cost and timing figures the line is logged for
the first time the API adds a field.

The guard

The fixtures are the SHAPES a credential takes — a Scraping Browser
endpoint, an authenticated proxy, a key as a query parameter — not the
literals this repo happens to contain today, and they assert both halves:
the secret is gone and the cost/host/status worth logging survives.

They are assembled from pieces rather than written out whole, because this
file is scanned by the repo's own credential check and a fixture that LOOKS
like a live key fails it. An allowlist entry would have been a hole a real
credential could later hide in.

Controlled. With the fix in, the suite is green; with the log line
pointed back at the raw value, the suite goes RED and names this check.
That control is what this change was verified by, rather than by "the suite
is still green" — a guard that is defined and never called satisfies that
perfectly, and one repo's guard turned out to be vacuous for exactly that
reason until the control found it.

🤖 Generated with Claude Code

SECURITY.md names this header as one of three places credentials reach a log
unmasked, and the client then logged it verbatim. The API echoes back the
task it ran, so a run driven through a credentialed CDP endpoint put that
endpoint's username and password into the log, and a key passed as a query
parameter would go the same way. A rule stated in one file and broken in
another is invisible to both.

Both patterns are global: a masker that handles the first occurrence prints
the password the other four times and looks like it is working. Redaction
rather than an allowlist of fields, because x-debug is the remote API's own
metadata and its shape is not ours to pin — an allowlist would silently drop
the cost and timing figures the line is logged FOR the first time the API
adds a field.

The guard uses the SHAPES a credential takes, not the literals this repo
contains today, and asserts both halves: the secret is gone and the
cost/host/status survive. Its fixtures are assembled from pieces rather than
written out whole, because this file is scanned by the repo's own credential
check; an allowlist entry would be a hole a real credential could hide in.

Controlled by pointing the log line back at the raw value: the suite goes
red and names this check. Verified that way rather than by "the suite is
still green", which a guard that never executes also satisfies.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Sep 21, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@jehrr
jehrr merged commit 83445a7 into main Sep 21, 2026
8 checks passed
@jehrr
jehrr deleted the xdebug-redaction branch September 21, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants