Skip to content

Skill review experience: act on proposals, see what changed, and track improvement over time #389

Description

@Sheldenshi

Context

The skill-learning-from-outcomes feature has landed (ADR docs/adr/skill-learning-from-outcomes.md, docs/skill-learning.md). The backend loop works end-to-end (verified live), but the user-facing surface is thin: today a user only sees a daily digest land in a "Skill review" channel. They cannot act on proposals in-app, cannot see what changed in a skill, and there is no over-time view. This issue tracks the full Skill review experience (web first; mobile follow-up).

What a user sees today (the gap)

  • Digest only. The daily digest renders as a chat block in the auto-provisioned "Skill review" channel (feature: "skill-review") and flags unread like any channel. That's the only proactive surface.
  • No in-app actions. Approve/Reject a proposal is CLI-only (gini improvement). useImprovements() exists in web/src/lib/queries.ts but no component renders it.
  • Skill changes are invisible / misleading. Outcome-driven edits change the SKILL.md body, but SkillVersion/previousVersions doesn't carry body, so the Skills page shows only a version bump, not a diff — and the existing Rollback button is wrong for body edits (see the linked correctness bug).
  • No "improvements over time." skillOutcomes / learningFindings are API-only; nothing renders them. There's no trend, no per-skill reliability, no changelog. (successCount/failureCount are validation counts, not the learning signal — don't surface them as quality.)

Scope

1. Actionable proposals

  • List pending ImprovementProposals (kind:"skill", payload.mode:"edit"): show the bounded diff (payload.edits / candidateBody vs baseBody) + rationale + evidence (sourceTaskId).
  • Approve / Reject in-app (POST /api/improvements/:id/approve|reject).
  • Revert an applied edit in-app — needs a new POST /api/improvements/:id/revert route wrapping the existing revertImprovement (re-installs baseBody). Without it there is no correct in-app undo for a body edit.

2. Findings

  • Open LearningFindings (environment / credential / model_ignored / bundled_skill) with a dismiss action.

3. Improvements over time

  • Per-skill reliability / failure trend from SkillOutcome rows over a recent window, plus a recent-outcomes feed and a record of accepted edits (a real changelog).
  • This needs a defined skill score — see the scoring note below; the honest metric is reliability + adjudication-confidence, not a fake accuracy number.

4. Skill change history (pairs with the linked bug)

  • Show the body diff for an applied edit and a correct revert. Requires capturing the prior body (see linked bug) — previousVersions currently can't represent body edits.

Backend available today

  • GET /api/learning/outcomes, GET /api/learning/findings
  • GET /api/improvements + POST /api/improvements/:id/approve|reject
  • Digest renders in the feature:"skill-review" channel.

Still needed (backend)

  • POST /api/improvements/:id/revert (wrap revertImprovement).
  • Body-level skill history (linked bug) so the panel can show a diff and revert correctly.
  • (Optional) a per-skill score/aggregation endpoint, or compute reliability client-side from /api/learning/outcomes.

Notes

  • Dual client: keep web + mobile renderers in sync if chat-block content is surfaced (ADR chat-block-protocol.md). Web-first is fine.
  • Consistent with the existing Jobs/Agents panels.

Related correctness bug: skill body edits are not represented in version history and the Rollback button doesn't undo them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions