Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/workflows/model-catalog-audit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,14 @@ jobs:
--input "$RUNNER_TEMP/ai-chip-intelligence-claims.json" \
--output "$RUNNER_TEMP/ai-chip-intelligence-evidence-graph.json" \
--summary-path "$RUNNER_TEMP/ai-chip-intelligence-evidence-review.md"
- name: Audit evidence provenance and quote traceability
run: |
python3 scripts/audit_intelligence_evidence_quality.py \
--discovery "$RUNNER_TEMP/ai-chip-intelligence-discovery.json" \
--claims "$RUNNER_TEMP/ai-chip-intelligence-claims.json" \
--graph "$RUNNER_TEMP/ai-chip-intelligence-evidence-graph.json" \
--output "$RUNNER_TEMP/ai-chip-intelligence-evidence-quality.json" \
--markdown "$RUNNER_TEMP/ai-chip-intelligence-evidence-quality.md"
- name: Prioritize unverified evidence review queue
run: |
python3 scripts/triage_intelligence_claims.py \
Expand Down Expand Up @@ -226,6 +234,9 @@ jobs:
if [ -f "$RUNNER_TEMP/ai-chip-intelligence-evidence-review.md" ]; then
cat "$RUNNER_TEMP/ai-chip-intelligence-evidence-review.md" >> "$GITHUB_STEP_SUMMARY"
fi
if [ -f "$RUNNER_TEMP/ai-chip-intelligence-evidence-quality.md" ]; then
cat "$RUNNER_TEMP/ai-chip-intelligence-evidence-quality.md" >> "$GITHUB_STEP_SUMMARY"
fi
- name: Upload discovery report
if: always()
uses: actions/upload-artifact@v4
Expand All @@ -250,6 +261,8 @@ jobs:
${{ runner.temp }}/ai-chip-intelligence-retrieval-review-template.md
${{ runner.temp }}/ai-chip-intelligence-evidence-graph.json
${{ runner.temp }}/ai-chip-intelligence-evidence-review.md
${{ runner.temp }}/ai-chip-intelligence-evidence-quality.json
${{ runner.temp }}/ai-chip-intelligence-evidence-quality.md
${{ runner.temp }}/ai-chip-intelligence-triage.json
${{ runner.temp }}/ai-chip-intelligence-triage.md
${{ runner.temp }}/ai-chip-intelligence-review-state.json
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/validate-compatibility.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ jobs:
python3 -m unittest discover -s tests -p "test_intelligence_retrieval_review.py"
python3 scripts/evaluate_intelligence_quality.py --golden data/intelligence-quality-golden.json
python3 -m unittest discover -s tests -p "test_intelligence_evidence_graph.py"
python3 -m unittest discover -s tests -p "test_intelligence_evidence_quality.py"
python3 -m unittest discover -s tests -p "test_intelligence_triage.py"
python3 -m unittest discover -s tests -p "test_intelligence_review_state.py"
python3 -m unittest discover -s tests -p "test_intelligence_backlog_hydration.py"
Expand Down
46 changes: 46 additions & 0 deletions docs/evidence-graph-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,3 +75,49 @@ candidate files.

The workflow never edits published model, accelerator, company or benchmark
registries and never opens or merges data update PRs.


## Evidence provenance and quote traceability audit (PR #129)

The weekly discovery workflow now runs a separate read-only audit after Claim
extraction and evidence-graph construction:

python3 scripts/audit_intelligence_evidence_quality.py \
--discovery /tmp/ai-chip-intelligence-discovery.json \
--claims /tmp/ai-chip-intelligence-claims.json \
--graph /tmp/ai-chip-intelligence-evidence-graph.json \
--output /tmp/ai-chip-intelligence-evidence-quality.json \
--markdown /tmp/ai-chip-intelligence-evidence-quality.md

The report measures operational evidence hygiene, not factual correctness:

- **Quote traceability:** whether the extracted evidence quote can be found in
the fetched candidate excerpt after the same text normalization used by
extraction. This does not establish that the quote logically supports the
interpretation.
- **Provenance completeness:** presence of a source URL, publisher, source
type, declared independence label, evidence scope and parseable publication
timestamp.
- **Metadata consistency:** whether source metadata copied onto a Claim still
matches its originating discovery candidate.
- **Source diversity:** counts of distinct source documents and publishers.
These are descriptive counts, not independent corroboration.
- **Evidence/assertion classes:** distributions of preprint, vendor, project,
repository and reporting classes, plus assertion categories such as
roadmap, marketing, research-result candidate or specification.
- **Reviewer-assigned origins:** counts only explicit origin annotations
already present in the evidence graph. The audit never infers origin from
publisher names or matching text.

The audit is deliberately not a single weighted quality score. It does not
rank a research preprint above a vendor primary source in all contexts,
equate media coverage with replication, infer peer-review status, or convert
the declared `independent` label into a verified relationship. Syndicated
articles and documents from the same underlying source may still appear as
different URLs or publishers. Exact-quote presence is necessary traceability,
not semantic entailment.

Every Claim remains `unverified_candidate`. No review state, company profile,
model support record, benchmark result or hardware registry is changed by this
audit. Human source inspection and the separate Evidence-to-PR approval gate
remain mandatory.
Loading
Loading