Fixes 30158: Improve AI context rendering for MCP consumers

[harshach]

Describe your changes:

Fixes #30158

This fixes AI context consumption for MCP clients by distinguishing absent schema constraints, preserving bounded column-level lineage, and replacing REST follow-up pointers with registered MCP tool calls.

Type of change:

• Bug fix

High-level design:

• Schema: additive upstreamEdges and downstreamEdges contexts while retaining the existing FQN arrays
• Builder: resolve neighboring assets once and retain up to 25 recorded column mappings per edge
• Markdown: nested mappings only when present, byte-compatible flat fallback otherwise, and explicit -- for absent constraints
• Persona context: copy heavy lineage edge data and emit get_asset_context(...) pointers
• Compatibility: optional fields only; no database migration required; cached persona documents refresh through the existing TTL/fingerprint flow
• Alternative rejected: replacing the existing FQN arrays, which would break REST consumers

Tests:

Use cases covered

• Missing, NULL, and PRIMARY_KEY constraint rendering
• Upstream/downstream column mappings, functions, prefix handling, malformed values, flat fallback, and cap notes
• Persona heavy-section copying and MCP pointers in profile, compact, and manifest tiers
• Schema generation and generated TypeScript synchronization

Unit tests

• I added unit tests for the new/changed logic
• Files updated: AIContextBuilderTest, AIContextMarkdownTest, PersonaContextBuilderTest, PersonaContextMarkdownTest
• Result: 57 targeted tests passed
• Coverage was not collected

Backend integration tests

• I added integration tests in openmetadata-integration-tests/
• Existing AIContextRestIT, AIContextMcpIT, and PersonaAIContextIT were not run locally

Ingestion integration tests

• Not applicable (no ingestion changes)

Playwright (UI) tests

• Not applicable (generated type only; no UI behavior changes)

Manual testing performed

No manual testing was performed beyond the automated checks above.

UI screen recording / screenshots:

Not applicable — no UI behavior changes.

Checklist:

• I have read the CONTRIBUTING document
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #<issue-number> above
• I have commented on my code, particularly in hard-to-understand areas
• For JSON Schema changes: no migration is needed because the new fields are additive and optional
• For UI changes: not applicable; generated types only
• I have added tests and listed them above
• I have added tests covering the exact bug scenarios

Greptile Summary

This PR improves AI context output for MCP consumers. The main changes are:

• Adds bounded column-level mappings to upstream and downstream lineage edges.
• Preserves existing flat lineage fields for compatibility.
• Distinguishes absent schema constraints in Markdown output.
• Copies detailed lineage into persona contexts.
• Replaces REST follow-up links with registered MCP tool calls.
• Updates generated schemas, TypeScript models, and targeted tests.

Confidence Score: 5/5

The latest changes look safe to merge.

• The reported Markdown delimiter cases are handled across lineage values and persona tool pointers.
• The schema additions remain optional and preserve existing flat lineage fields.
• No new distinct blocking issue qualifies for this follow-up round.

Important Files Changed

Filename	Overview
openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextBuilder.java	Builds optional lineage-edge contexts while preserving the existing flat FQN lists.
openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextMarkdown.java	Renders nested column lineage, explicit absent constraints, and sanitized inline values.
openmetadata-service/src/main/java/org/openmetadata/service/aicontext/PersonaContextBuilder.java	Copies detailed lineage edges into persona contexts when lineage is requested.
openmetadata-service/src/main/java/org/openmetadata/service/aicontext/PersonaContextMarkdown.java	Emits get_asset_context pointers for profile, compact, and manifest output.
openmetadata-spec/src/main/resources/json/schema/type/aiContext.json	Adds optional lineage-edge fields without replacing existing compatibility fields.

Comments Outside Diff (1)

1. openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextMarkdown.java, line 580-589 (link)

   Exact Limit Reports Truncation

   An edge with exactly 25 mappings satisfies this condition even though edgeColumns() only truncates lists larger than 25. The Markdown therefore tells consumers to fetch omitted lineage when the complete mapping list was already rendered.

_{Reviews (5): Last reviewed commit: "Merge branch 'main' into harshach/ai-con..." | Re-trigger Greptile}

Context used (3)

• Context used - CLAUDE.md (source)
• Context used - openmetadata-ui-core-components/CLAUDE.md (source)
• Context used - AGENTS.md (source)

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Issue Fix AI context constraints, column lineage, and MCP pointers #30158 is missing Shipping → Priority.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[github-actions]

✅ TypeScript Types Auto-Updated

The generated TypeScript types have been automatically updated based on JSON schema changes in this PR.

[github-actions]

Jest test Coverage

UI tests summary

Lines	Statements	Branches	Functions
	65% (75506/116149)	48.8% (44927/92060)	49.61% (13613/27437)

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gitar-bot]

Code Review ✅ Approved 3 resolved / 3 findings

Improves AI context rendering for MCP consumers by adding bounded column-level lineage and replacing REST pointers with tool-call references. Resolved issues related to redundant truncation notices and Markdown corruption.

✅ 3 resolved

✅ Edge Case: Cap note shown even when column mappings not truncated

📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextMarkdown.java:578-586 📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextBuilder.java:467-474
hasCappedMappings() flags an edge with >= MAX_COLUMN_MAPPINGS_PER_EDGE (25) columns, but edgeColumns() also truncates to exactly 25. An edge that genuinely has exactly 25 mappings (none dropped) still emits the misleading 'Column mappings are capped at 25 per edge…' note, telling the agent to fetch a fuller graph that doesn't exist. Since truncation is lossy, the count alone can't distinguish 'exactly 25' from 'truncated to 25'; consider tracking a truncated flag on LineageEdgeContext (e.g. set only when the source list exceeded the cap) and keying the note off that.

✅ Edge Case: Cap note emitted for edges with exactly 25 mappings

📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextMarkdown.java:578-586 📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextBuilder.java:467-473
edgeColumns() truncates only when the mapping list size is strictly greater than MAX_COLUMN_MAPPINGS_PER_EDGE (capList uses size() > max), so an edge with exactly 25 mappings is rendered in full. But hasCappedMappings() uses >= MAX_COLUMN_MAPPINGS_PER_EDGE, so it wrongly appends the 'capped, fetch full lineage graph' note even though nothing was omitted. Because the builder already discards overflow, the count cannot distinguish 'exactly 25' from 'truncated from 30' at render time; to avoid a false positive the builder should record whether truncation actually occurred (e.g. a boolean/overflow flag on LineageEdgeContext) and have the markdown key off that rather than the post-cap size. Note the added test render_notesWhenColumnMappingsReachTheCap uses exactly 25 and asserts this false-positive behavior.

✅ Quality: Backticks in function/FQN values can corrupt inline Markdown

📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextMarkdown.java:533-547 📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/PersonaContextMarkdown.java:603-610
Column mapping rendering wraps source → target (function) in a single inline code span, and the MCP pointer wraps entityType/fqn in backticks. A transformation function or FQN containing a backtick (or newline) will prematurely close the code span and produce malformed Markdown that MCP/LLM consumers may misparse. Sanitize these values (strip/escape backticks and collapse newlines) before embedding them, similar to how cell() escapes table delimiters.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[github-actions]

✅ TypeScript Types Auto-Updated

The generated TypeScript types have been automatically updated based on JSON schema changes in this PR.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gitar-bot]

Code Review ✅ Approved 3 resolved / 3 findings

Improves AI context rendering for MCP consumers by adding bounded column-level lineage and replacing REST pointers with tool-call references. Resolved issues related to redundant truncation notices and Markdown corruption.

✅ 3 resolved

✅ Edge Case: Cap note shown even when column mappings not truncated

📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextMarkdown.java:578-586 📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextBuilder.java:467-474
hasCappedMappings() flags an edge with >= MAX_COLUMN_MAPPINGS_PER_EDGE (25) columns, but edgeColumns() also truncates to exactly 25. An edge that genuinely has exactly 25 mappings (none dropped) still emits the misleading 'Column mappings are capped at 25 per edge…' note, telling the agent to fetch a fuller graph that doesn't exist. Since truncation is lossy, the count alone can't distinguish 'exactly 25' from 'truncated to 25'; consider tracking a truncated flag on LineageEdgeContext (e.g. set only when the source list exceeded the cap) and keying the note off that.

✅ Edge Case: Cap note emitted for edges with exactly 25 mappings

📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextMarkdown.java:578-586 📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextBuilder.java:467-473
edgeColumns() truncates only when the mapping list size is strictly greater than MAX_COLUMN_MAPPINGS_PER_EDGE (capList uses size() > max), so an edge with exactly 25 mappings is rendered in full. But hasCappedMappings() uses >= MAX_COLUMN_MAPPINGS_PER_EDGE, so it wrongly appends the 'capped, fetch full lineage graph' note even though nothing was omitted. Because the builder already discards overflow, the count cannot distinguish 'exactly 25' from 'truncated from 30' at render time; to avoid a false positive the builder should record whether truncation actually occurred (e.g. a boolean/overflow flag on LineageEdgeContext) and have the markdown key off that rather than the post-cap size. Note the added test render_notesWhenColumnMappingsReachTheCap uses exactly 25 and asserts this false-positive behavior.

✅ Quality: Backticks in function/FQN values can corrupt inline Markdown

📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextMarkdown.java:533-547 📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/PersonaContextMarkdown.java:603-610
Column mapping rendering wraps source → target (function) in a single inline code span, and the MCP pointer wraps entityType/fqn in backticks. A transformation function or FQN containing a backtick (or newline) will prematurely close the code span and produce malformed Markdown that MCP/LLM consumers may misparse. Sanitize these values (strip/escape backticks and collapse newlines) before embedding them, similar to how cell() escapes table delimiters.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ui'

Issues
1199 New issues
0 Accepted issues

Measures
13 Security Hotspots
0.0% Coverage on New Code
3.1% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

🔴 Playwright Results — 2 failure(s), 27 flaky

✅ 4540 passed · ❌ 2 failed · 🟡 27 flaky · ⏭️ 96 skipped

Shard	Passed	Failed	Flaky	Skipped
🟡 Shard 1	436	0	4	16
✅ Shard 2	11	0	0	0
🟡 Shard 3	826	0	6	8
🔴 Shard 4	820	1	2	18
🟡 Shard 5	836	0	4	5
🟡 Shard 6	786	0	2	46
🔴 Shard 7	825	1	9	3

Genuine Failures (failed on all attempts)

❌ Features/Glossary/GlossaryTermRelationsGraphNested.spec.ts › viewing a child term: parentOf edge is rendered between parent and child (shard 4)

�[31mTest timeout of 60000ms exceeded.�[39m

❌ Features/AutoPilot.spec.ts › Create Service and check the AutoPilot status (shard 7)

Error: �[2mexpect(�[22m�[31mlocator�[39m�[2m).�[22mtoBeVisible�[2m(�[22m�[2m)�[22m failed

Locator: getByText('AutoPilot agents run completed successfully.')
Expected: visible
Timeout: 60000ms
Error: element(s) not found

Call log:
�[2m  - Expect "toBeVisible" with timeout 60000ms�[22m
�[2m  - waiting for getByText('AutoPilot agents run completed successfully.')�[22m

🟡 27 flaky test(s) (passed on retry)

• Features/TeamsHierarchy.spec.ts › Add teams in hierarchy (shard 1, 2 retries)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: metric (shard 1, 1 retry)
• Pages/SearchSettings.spec.ts › Latest preview config wins when a superseded request resolves late (shard 1, 1 retry)
• Flow/SearchRBAC.spec.ts › a fully denied user sees neither asset type when browsing (shard 1, 1 retry)
• Features/BulkImportWithDotInName.spec.ts › Bulk edit existing entity with dot in service name (shard 3, 1 retry)
• Features/ContextCenterArchive.spec.ts › archive page lazy-loads more rows on scroll within its own scroll container (shard 3, 1 retry)
• Features/ContextCenterArticles.spec.ts › Article listing search filters, clears, and shows empty state (shard 3, 1 retry)
• Features/ContextCenterArticles.spec.ts › Article list cards, recently viewed widget, and pagination work (shard 3, 1 retry)
• Features/ContextCenterArticles.spec.ts › description: switching articles does not bleed unsaved content into next article (shard 3, 1 retry)
• Features/ContextCenterMemories.spec.ts › Shared memory shows the shared-with-specific-people description (shard 3, 1 retry)
• Features/Glossary/GlossaryTermRelationsGraphNested.spec.ts › viewing a child term: parent appears as a 1-hop neighbour via parentOf edge (shard 4, 1 retry)
• Features/Table.spec.ts › should persist page size (shard 4, 1 retry)
• Flow/ObservabilityAlerts.spec.ts › Pipeline Alert (shard 5, 1 retry)
• Pages/CustomProperties.spec.ts › Enum (shard 5, 1 retry)
• Pages/CustomProperties.spec.ts › Markdown (shard 5, 1 retry)
• Pages/CustomProperties.spec.ts › Should verify property name is visible for apiCollection in right panel (shard 5, 1 retry)
• Pages/Entity.spec.ts › Tier Add, Update and Remove (shard 6, 1 retry)
• Pages/ExplorePageRightPanel_KnowledgeCenter.spec.ts › Should remove user owner for knowledgeCenter (shard 6, 1 retry)
• Pages/ExplorePageRightPanel.spec.ts › Should allow Data Steward to edit tags for searchIndex (shard 7, 1 retry)
• Pages/Glossary.spec.ts › Create term with related terms, tags and owners during creation (shard 7, 1 retry)
• Pages/GlossaryImportExport.spec.ts › Check for Circular Reference in Glossary Import (shard 7, 1 retry)
• Pages/GlossaryImportExport.spec.ts › Import partial success - some terms pass, some fail (shard 7, 1 retry)
• Pages/GlossaryImportExport.spec.ts › Glossary CSV import rejects unknown relation type (shard 7, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify Impact Analysis service filter selection (shard 7, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab is NOT visible for pipelineService in platform lineage (shard 7, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab is NOT visible for apiService in platform lineage (shard 7, 1 retry)
• Pages/TestSuite.spec.ts › Logical TestSuite (shard 7, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace