Tracking:replay-guided Experience evolution #1986

Background

[RFC #1750](#1750) proposes transferring the core idea of replay-guided exploration to PowerContext Experience and Skill generation.

This proposal only borrows the underlying idea and mechanism. It does not depend on, integrate, fork, vendor, or execute any code, package, protocol, schema, data format, or script from Dream-RSI.

The first target is Experience generation rather than the prepare_context hot path:

• no change to online context preparation;
• no default increase in Dream generation attempts;
• no change to the existing Candidate, Review, Artifact Revision, or authorization boundaries;
• no persistence of experimental proposal attempts in Runtime tables or the public API.

This issue tracks the near-term implementation scope of RFC #1750: P0, P1, and a small Experience-only P2 feasibility spike.

Phase Overview

P0, P1, and P2 are the implementation scope of this tracking issue. P3a, P3b, and P4 are recorded only as follow-up directions and are not short-term delivery commitments.

Current Objects and Boundaries

PowerContext needs to distinguish the following objects:

Experience incubation and explicit DreamRun are different implementation objects. Their cursors, storage, APIs, and Candidate-count semantics must not be merged.

The current DreamBudget.max_model_calls primarily provides retry capacity for recoverable inference failures. It does not imply that multiple independent proposal branches already exist. P0, P1, and P2 must not increase this limit or change the existing retry semantics.

Non-goals

This issue does not include:

• optimizing prepare_context;
• changing default Dream behavior;
• converting all Dream runs to multi-attempt execution;
• increasing DreamBudget.max_model_calls;
• allowing a shadow proposal to become a Candidate;
• persisting ProposalAttempt in Runtime storage;
• adding a public HTTP API;
• automatic usage-driven Skill replacement;
• cross-policy replay or automatic promotion;
• introducing an LLM evaluator in the first phase;
• depending on or reusing the Dream-RSI implementation.

P0: Baseline Boundaries and Behavioral Equivalence

P0 does not change production behavior. It documents and protects the current behavior of:

• Experience incubation, including its Source window, generation, cursor, and Candidate behavior;
• explicit refine_experience Dream;
• explicit derive_skill Dream;
• the distinction between retries and independent proposal attempts;
• exact evidence, lineage, target, and expected-head handling;
• Review and authorization boundaries.

P0 acceptance criteria:

• the same resolved inputs and deterministic collaborator results produce the same observable outcomes;
• Candidate count, family, operation, target, and pending status remain unchanged;
• Experience incubation cursor behavior remains unchanged;
• retry and model-call accounting remain unchanged;
• no migration, public API, default configuration, additional Candidate, or additional model call is introduced.

P1: Evaluation Recording Contract and Faithful Replay

P1 defines the evaluation contract before implementing replay.

P2 depends on the P1 recording and replay contract. The Dream Evolution Evaluation Harness ownership, bundle schema, evidence-retention rules, cost accounting, and exact-replay coverage requirements must be fixed before the P2 spike begins.

Evaluation ownership

P1 and P2 are owned by a Dream Evolution Evaluation Harness. The harness may reuse existing evaluation components for Scope isolation, workload selection, retention, and report rendering through an adapter. Dream bundles must not be treated as Bub/Harbor workloads or SWE-Pro runs.

The only new versioned artifact introduced by this issue is:

powercontext.artifact-evolution/v1

Its execution.type is dream. The workload section records at least:

• the refine_experience operation;
• exact Memory, Experience, and Source references;
• target and expected head;
• input fixture digest;
• treatment;
• proposal and evaluation budgets.

When upstream task provenance exists, the bundle may reference powercontext.e2e-task/v1. Dream inputs do not need to be forced into the Bub workload catalog. Typed evidence produced by an underlying run may reference powercontext.e2e-evidence/v1, while the adapter adds Dream-specific references, action signatures, validator results, and nomination state.

P1 MVP recording requirements

The P1 MVP records at least:

• bundle schema version and run identity;
• exact input manifest, snapshot digest, operation, target, and expected head;
• policy, generator, model, schema, and runtime identities;
• observations revealed before the decision;
• canonical action signature;
• proposal output, no_change, needs_evidence, or error result;
• deterministic validator/scorer identity, version, and results;
• proposal-attempt, retry, model-call, token, wall-time, and validation costs;
• exact DreamRun, input, and Candidate references when available;
• nomination status and reason;
• redaction, retention, and Scope-isolation information.

The schema must remain extensible. The following are not P1 MVP blockers:

• the complete action set;
• logging-policy version;
• selection probability;
• deterministic selection reason and tie-break rule;
• split/holdout identity;
• cross-policy support and coverage;
• sealed post-decision outcome links.

These fields belong to a later cross-policy research extension.

Faithful replay semantics

Faithful replay is a read-only lookup over realized proposal attempts:

• for the same revealed observations and canonical action signature, return the recorded result, validator results, and cost;
• do not call a generator, model, or evaluator;
• do not create a Candidate;
• do not mutate Review, Artifact, or Runtime state;
• do not use future Review, Revision, or downstream outcomes;
• do not use the “most similar” historical attempt to fill a missing action;
• do not fabricate rewards, costs, or evidence.

When no matching action signature exists, replay must return:

out_of_support

with an explicit reason.

P1 must report basic exact-replay coverage:

• supported requested decisions / all requested decisions;
• complete supported trajectories / all requested trajectories.

Unsupported decisions and trajectories must remain in the denominator and must not be silently dropped.

Ordinary production capture is disabled by default. Complete prompts, task bodies, and evidence bodies may only be retained when a controlled evaluation explicitly provides reviewed, synthetic, or redacted material.

P2: Experience-only Feasibility Spike

P2 is an asynchronous, explicitly enabled, time-boxed feasibility experiment. It is not a production default change.

The experiment is limited to:

explicit refine_experience Dream

It does not change:

• scheduled Experience incubation;
• explicit derive_skill;
• the existing Dream public API;
• the existing Candidate/Review path.

Baseline and shadow proposal attempt

For each pre-registered matched workload:

1. Run the current Dream behavior using immutable inputs as the baseline.
2. When permitted by the pre-registered decision rule, run one additional shadow proposal attempt.
3. Record the output, validation, quality result, and cost of each attempt separately.

The issue intentionally uses the full terms baseline and shadow proposal attempt rather than abbreviations.

The comparison must follow these rules:

• the baseline is the only result eligible for the existing Candidate/Review path;
• the shadow proposal attempt is recorded only in the evaluation bundle;
• the shadow proposal attempt does not create a Candidate;
• the two results are not collapsed into a best-of-two production result;
• baseline, shadow, retry, validation, token, wall-time, failed, and discarded attempt costs are all reported;
• baseline execution failure followed by shadow success is reported separately as rescue-after-failure;
• a workload with zero eligible proposals receives no synthetic quality score;
• equal baseline and shadow results are recorded as zero improvement.

P2 must not increase DreamBudget.max_model_calls, repurpose the existing retry budget, or place two independent branches into one existing DreamRun. If multiple branches are later allowed to produce a Candidate, that requires a separate Dream budget/API contract and migration discussion.

Initial validators

The first phase uses deterministic validators only:

• output schema;
• exact evidence and citation resolution;
• Scope consistency;
• target and expected-head correctness;
• lineage completeness;
• duplicate detection;
• deterministically detectable conflicts;
• budget and cost accounting;
• checks against fabricated evidence, approval, publication, installation, or execution claims.

These validators establish eligibility. They do not prove semantic superiority over the baseline.

Semantic quality is evaluated through blinded human Review on a fixed workload. No LLM evaluator is included in the first spike.

P2 exit gate

Before execution, the experiment must pre-register:

• the minimum paired quality improvement; or
• the acceptable cost reduction with no quality or guardrail regression.

The experiment can proceed only if:

• grounded reusable-proposal quality improves or evolution cost decreases;
• Scope, lineage, target, fabrication, and Review-bypass violations do not increase;
• the result is validated by fresh live Dream runs;
• all proposal attempts and costs are completely recorded.

If the exit gate is not met, multi-attempt exploration stops. P0 and P1 may still proceed independently.

Evaluation and Cost Requirements

Evaluation is required at every stage, with proportional scope:

Candidate count, approval rate, or a single blended score must not be the sole objective.

Reports must separate:

• proposal validity;
• semantic quality;
• Review workload;
• evolution cost;
• failed and discarded attempts;
• downstream outcomes when exact attribution is available.

Runtime, token, turn, and tool-call metrics are meaningful only after an approved Revision has been exercised on comparable tasks. Installing this mechanism does not itself imply a runtime improvement.

Separation from the Recurrence Ledger

Proposal attempt/replay records and the recurring-failure repair ledger answer different questions:

Proposal attempt / replay record
= what the Dream generation strategy explored

Recurrence ledger
= what happened to an approved Experience Revision in later real tasks

They remain separate semantic stores and may only be joined through exact Candidate, Artifact Revision, DreamRun, and Source references.

An unapproved proposal attempt must not receive a recurrence outcome. Missing selected, avoided, or downstream outcome evidence must remain unknown rather than being interpreted as a negative result.

Deferred Follow-up Work

The following items belong to RFC #1750 but are not part of the current implementation scope:

• P3a: Experience-to-Skill
  Evaluate applying the same bounded exploration mechanism to derive_skill after the Experience-only spike satisfies its quality, cost, and safety gates.

• P3b: Usage-driven Skill replacement
  Design an automatic replacement producer only after usage coverage, outcome quality, and environment stratification are sufficiently reliable.

• P4: Cross-policy replay research
  Study alternative attempt allocation, pruning, stopping, and nomination policies once action support, logging metadata, workload/holdout isolation, and fresh-live promotion gates are available.

Each of these follow-up items should be tracked and reviewed separately.

Open Questions

1. What is the smallest useful canonical action vocabulary for the refine_experience spike?
2. Which existing evaluation isolation and reporting components should the Dream Evolution Evaluation Harness reuse?
3. What conditions should trigger an additional shadow proposal attempt?
4. What workload, blinded Review protocol, and minimum sample size are appropriate?
5. Should a controlled replay bundle store encrypted payloads, or only manifests/digests with separately managed fixtures?

References

• Issue Tracking issue:replay-guided self-evolution for Experience / Skill generation #1635
• [RFC #1750](docs(rfc): refine artifact evolution proposal #1750)
• [Dream-RSI paper](https://arxiv.org/abs/2609.14858)
• [Chinese overview](https://mp.weixin.qq.com/s/VRjsoQLqHx80NZpeS5aqkA?scene=1)