Add bounded structured operational logging and local log inspection

[Haz3-jolt]

Outcome

Add bounded, structured operational logging for Axl daemon processes and an axl logs command for trusted local inspection.

Operational logs must help diagnose startup failures, daemon lifecycle problems, runtime assembly failures, provider errors, sandbox availability, RPC failures, queue failures, and shutdown behavior without becoming canonical session state.

Axl already has authoritative per-session JSONL events, a durable mutation journal, safe provider diagnostics, and SDK projections. This issue adds a separate disposable diagnostic channel. It must never replace, repair, or influence canonical behavior.

Current state

Axl currently has:

• authoritative per-session JSONL through JsonlEventLog
• canonical events for session behavior, model requests, tools, interactions, compaction, configuration, and errors
• a durable command mutation journal
• bounded and redacted provider diagnostics
• client-visible error projection
• isolated direct console.error usage for daemon background failure reporting

Axl does not currently have:

• a shared operational logger
• structured severity levels
• subsystem and request correlation fields
• bounded daemon log files
• log rotation and retention
• deterministic shutdown flushing
• discoverable daemon log locations
• a uniform redaction boundary for non-canonical diagnostics

Architectural boundary

Canonical events record behavior that affects model-visible context, replay, tools, interactions, configuration, operation ownership, client projections, and durable user-visible outcomes. They remain authoritative.

Operational logs record implementation diagnostics such as daemon startup and shutdown, runtime assembly timing, socket lifecycle, provider catalog failures, sandbox detection failures, rejected RPC metadata, background queue failures, internal exceptions, rotation, and sink failure.

Deleting operational logs must not affect session recovery, replay, SDK projection, or behavior. Logging success must never be treated as evidence that an operation succeeded.

Ownership

• packages/runtime owns local logger construction, filesystem destinations, rotation, and process-level configuration.
• packages/daemon accepts a narrow injected operational log sink and emits daemon lifecycle diagnostics.
• packages/cli owns axl logs argument parsing and human-readable formatting.
• packages/ai continues producing bounded safe diagnostics. Runtime adapters may forward those diagnostics to the operational sink.
• packages/sandbox continues returning bounded typed results. Runtime adapters may log those results.
• packages/kernel must not depend on the operational logger.
• packages/protocol must not define operational records because they are not canonical events or wire contracts.
• Clients must not reconstruct canonical state from operational logs.

Do not add a generic logging package until another process host needs the exact same implementation.

Log record format

Use newline-delimited JSON with one object per record.

interface OperationalLogRecord {
  timestamp: string;
  level: "error" | "warn" | "info" | "debug";
  subsystem: string;
  message: string;

  daemonInstanceId?: string;
  sessionId?: string;
  operationId?: string;
  requestId?: string;

  code?: string;
  durationMs?: number;
  fields?: Record<string, SafeOperationalValue>;
}

SafeOperationalValue must be a bounded JSON-safe scalar or bounded collection. Arbitrary class instances, request bodies, headers, environment objects, recursive values, and unknown objects are rejected.

Each serialized line must have a fixed UTF-8 size limit. Oversized fields must be rejected or replaced with explicit bounded truncation metadata before writing.

Default storage

Store logs inside the selected daemon placement state directory:

<state-directory>/logs/daemon.jsonl

Examples:

~/.axl/logs/daemon.jsonl
~/.axl/unsafe/logs/daemon.jsonl
~/.axl/oci-<placement>/logs/daemon.jsonl

Requirements:

• create the log directory with mode 0700
• create log files with mode 0600
• reject symlinked log directories and files
• canonicalize the destination before opening
• never follow a log path outside the selected state directory
• never expose log contents through daemon session RPC

Rotation and retention

Use bounded local retention:

• active log maximum: 10 MiB
• retained files: 5, including the active file
• maximum placement log storage: approximately 50 MiB
• rotate before writing a record that would exceed the active-file limit
• rename retained generations deterministically
• delete the oldest generation during rotation
• keep every individual JSONL line complete
• never truncate a line in place
• serialize writes and rotation through one queue

Suggested filenames:

daemon.jsonl
daemon.1.jsonl
daemon.2.jsonl
daemon.3.jsonl
daemon.4.jsonl

Rotation failure must be reported loudly to stderr. It must not corrupt canonical history or terminate an otherwise healthy daemon.

Severity configuration

Support:

error
warn
info
debug

Configuration precedence:

1. explicit process option
2. --log-level
3. AXL_LOG
4. default info

Defaults:

• daemon file logging: info
• foreground stderr logging: warn
• tests: explicitly configured

Invalid values must fail startup with a clear configuration error.

Do not add per-subsystem filters in the first implementation.

Privacy and redaction

Operational logs must be safe by construction.

Never log:

• credential values
• authorization headers
• cookies
• complete environment maps
• provider request headers
• raw provider response bodies
• user prompt bodies
• assistant response bodies
• tool input bodies
• tool output bodies
• uploaded file contents or blob bytes
• complete AGENTS.md or SKILL.md contents
• arbitrary MCP payloads
• canonical event payloads as generic objects

Safe metadata may include:

• provider and model IDs
• tool names
• capability identities
• event types
• error and HTTP status codes
• retry attempts
• byte and token counts
• durations
• bounded canonical paths
• sandbox provider
• operation and request identifiers

Reuse existing resolved-secret redaction values where available. Apply redaction before serialization and before either file or stderr output.

Unknown Error objects must become a bounded safe shape. Stack traces may be included only at debug level after redaction and path normalization.

Correlation

Generate one daemon instance ID at process startup.

Include available identifiers on relevant records:

• daemon instance ID
• session ID
• operation ID
• RPC request ID
• provider ID
• model ID
• capability identity

Correlation fields are diagnostic metadata only. Canonical operation and event IDs remain authoritative.

Required initial instrumentation

Daemon lifecycle

Log:

• startup requested
• state directory accepted
• socket bind succeeded
• daemon ready
• shutdown requested and completed
• forced termination
• stale socket rejection
• incompatible daemon or client version

Runtime assembly

Log:

• runtime assembly started and completed
• provider registry restoration summary
• sandbox selection and enforced status
• runtime rebuild reason
• /reload completion or failure
• model or tool configuration boundary
• extension-host startup or disposal failure

Do not log prompt content, loaded resource content, or tool schemas.

RPC boundary

Log:

• rejected initialization
• unauthorized capability request
• invalid request shape
• unknown method
• operation conflict
• internal RPC failure
• request cancellation
• request duration at debug

Do not log RPC parameter objects.

Providers

Log bounded metadata for:

• authentication unavailable
• catalog refresh failure
• model unavailable
• request dispatch failure
• retry scheduling
• context overflow classification
• provider response termination failure

Provider diagnostics must pass through the existing safe diagnostic and credential redaction boundary.

Sandbox and process execution

Log:

• selected sandbox provider
• unavailable required isolation
• sandbox startup failure
• process spawn failure
• timeout or cancellation
• cleanup failure

Do not log full commands at default levels. Tool commands are already represented canonically where appropriate.

Background operations

Replace direct daemon console.error paths with the operational sink.

Log:

• queue drain failure
• subscription delivery failure
• cursor cleanup failure
• background disposal failure
• log writer or rotation failure

axl logs command

Support:

axl logs
axl logs --follow
axl logs --lines <count>
axl logs --json

Default behavior

axl logs must:

• resolve the currently selected daemon placement
• print the active log path
• print the latest 200 records
• format timestamp, level, subsystem, correlation IDs, and message for humans
• report clearly when logging is disabled or no log has been created

Follow mode

axl logs --follow must:

• print the existing bounded tail
• wait for appended records
• detect active-file rotation and continue with the replacement file
• exit cleanly on Ctrl+C
• use filesystem watching instead of polling when it provides equivalent behavior
• never start, stop, or restart a daemon

Line limit

axl logs --lines <count> must:

• default to 200
• accept values from 1 through 10,000
• read retained generations when the active file contains fewer requested records
• emit records in chronological order
• reject invalid and ambiguous values

JSON output

axl logs --json must:

• emit original validated JSONL records
• write no headings or formatting to stdout
• support use with jq and scripts
• fail loudly on malformed complete records
• ignore only an incomplete final line that may still be in flight

--json and --follow may be combined.

Placement behavior

Use the same placement selection as daemon lifecycle commands:

axl logs
axl logs --unsafe
axl logs --sandbox podman --image <digest>

Never merge logs from multiple placements without a future explicit option.

Inspection safety

• canonicalize state and log directories
• reject symlink escapes
• refuse files outside the selected state directory
• open files read-only
• never change permissions while reading
• never delete, truncate, clear, or rotate logs from axl logs
• bound line length and total output
• sanitize terminal control characters in human-readable output
• preserve exact validated records only in --json mode
• never expose log content over daemon RPC
• never automatically upload or share logs

Failure behavior

• Reject malformed records before they enter the writer queue.
• Report serialization failure to stderr.
• On file sink failure, disable that sink and report the failure once to stderr.
• Never recursively log sink failure.
• Preserve existing canonical append failure behavior.
• Wait for a bounded logger flush during shutdown.
• Report flush timeout loudly without rewriting canonical state.
• Never silently fall back to another persistent location.

Discoverability

Extend axl daemon status and axl doctor to report:

• operational logging enabled or disabled
• configured level
• canonical log directory
• active log file
• rotation limits
• last sink failure, if any

Do not expose log contents or credential-bearing paths through browser session RPC.

Non-goals

This issue does not add:

• remote telemetry
• hosted log collection
• automatic log upload
• OpenTelemetry export
• metrics aggregation
• distributed tracing
• browser or TUI log viewers
• a model-visible logging tool
• automatic bug reports
• full provider request or response capture
• prompt or tool-content logging
• canonical event replacement
• SQLite log storage
• extension access to the logger
• arbitrary third-party sinks
• log deletion or clearing commands
• full-text log indexing

Acceptance criteria

• Add a narrow structured operational log interface without changing canonical event semantics.
• Implement a stdlib-only JSONL file sink owned by the local runtime.
• Store logs under the selected daemon state directory.
• Enforce private directory and file permissions.
• Reject symlink escapes and destinations outside the state directory.
• Support error, warn, info, and debug levels.
• Support explicit option, --log-level, and AXL_LOG precedence.
• Default file logging to info.
• Bound every serialized record.
• Rotate at 10 MiB and retain at most five files.
• Serialize concurrent writes and rotation deterministically.
• Redact resolved credential values before every sink.
• Prohibit prompts, responses, tool bodies, headers, cookies, environment maps, and file contents from structured fields.
• Include daemon, session, operation, and request correlation when available.
• Instrument daemon startup, shutdown, runtime rebuild, RPC rejection, provider failure, sandbox failure, and background operation failure.
• Replace direct daemon console.error usage with the operational sink.
• Report sink failure once to stderr without recursion.
• Flush on shutdown with a fixed timeout.
• Report logging state and paths through local daemon status and doctor output.
• Add axl logs with a default 200-record tail.
• Add bounded --lines <count> support.
• Add rotation-aware --follow.
• Add raw validated --json output.
• Preserve chronological order across retained generations.
• Keep axl logs local and independent from daemon startup and RPC.
• Keep operational logs disposable and independent from session replay.
• Keep disabled logging free of file creation and background work.
• Add no production dependency.

Tests

Add deterministic tests for:

• level filtering and configuration precedence
• invalid level rejection
• JSONL record validity and UTF-8 byte bounds
• structured field validation
• credential redaction and prohibited field rejection
• private directory and file modes
• symlinked directory and file rejection
• state-directory containment
• concurrent write ordering
• rotation before overflow and retention deletion
• complete final JSONL lines
• file sink failure and one-time stderr reporting
• recursion prevention
• bounded shutdown flush
• daemon instance, session, operation, and request correlation
• disabled zero file creation
• daemon restart with a fresh instance ID
• canonical replay after operational logs are deleted
• canonical append failure remaining visible when logging succeeds
• default 200-record tail
• custom line bounds
• chronological reads across rotated generations
• active-file rotation during follow mode
• Ctrl+C during follow mode
• malformed complete record rejection
• incomplete live tail handling
• raw JSON output
• human-readable formatting and terminal control sanitization
• placement-specific log selection
• missing and disabled logs
• no daemon startup or RPC connection from axl logs
• no mutation of files while reading

Verification

Run:

pnpm check
uvx reuse lint
git diff --check

Also run a local daemon smoke test:

1. Start a daemon with info logging.
2. Create and use a session.
3. Run /reload.
4. Trigger one bounded invalid RPC.
5. Stop the daemon normally.
6. Verify complete JSONL records and shutdown flush.
7. Inspect them through axl logs.
8. Restart with debug and verify a new daemon instance ID.
9. Follow the active log through one forced rotation.
10. Confirm that no prompt, response, tool body, credential, header, or environment value appears in retained logs.

Sequencing

This work is independent from canonical capability logging in #16.

Recommended sequence:

1. Continue Build BM25 capability index and capability_search #16 using canonical events.
2. Implement operational logging as a separate focused branch.
3. Instrument capability discovery failures only after this logger lands.
4. Keep capability searches, activations, denials, and exact model-visible additions canonical regardless of logger availability.

References

• Codex local diagnostics and RUST_LOG: https://github.com/openai/codex/blob/main/docs/install.md
• OpenCode troubleshooting logs: https://opencode.ai/docs/troubleshooting/
• Goose logging system: https://goose-docs.ai/docs/guides/logs/
• Continue troubleshooting logs: https://docs.continue.dev/troubleshooting
• OpenHands observability: https://docs.openhands.dev/sdk/guides/observability

[iamrishu11]

if no one is working on it , i would be happy to work on this