Skip to content

Clarifies which LLMs we support for Obs and Sec throughout AI docs#5629

Merged
benironside merged 6 commits into
mainfrom
internal-731-obs-performance-matrix-applicability
Mar 28, 2026
Merged

Clarifies which LLMs we support for Obs and Sec throughout AI docs#5629
benironside merged 6 commits into
mainfrom
internal-731-obs-performance-matrix-applicability

Conversation

@benironside

@benironside benironside commented Mar 25, 2026

Copy link
Copy Markdown
Contributor

Fixes /elastic/docs-content-internal/issues/958

The problem was that we weren't always making it clear that certain LLMs are supported for certain use cases, while others aren't, or don't perform well. In other words, we may have oversold the extent to which users can bring any model.

The purpose of this PR is to make it clear to users who are learning about our AI powered features that we have tested models, we vouch for them, and we can only vouch for them.

It:

  • Adds references to the LLM performance matrices into pages that discuss AI Chat
  • Clarifies that the Observability LLM performance matrix applies equally to Agent Builder and to AI Assistant
  • Updates Security LLM performance matrix to match Obs updates
  • Disambiguates the Obs and Sec performance matrix page titles (now "...for Observability" / "...for Security")

Generative AI disclosure

To help us ensure compliance with the Elastic open source and documentation guidelines, please answer the following:

  1. Did you use a generative AI (GenAI) tool to assist in creating this contribution?
  • [x ] Yes
  • No
  1. If you answered "Yes" to the previous question, please specify the tool(s) and model(s) used (e.g., Google Gemini, OpenAI ChatGPT-4, etc.).

Tool(s) and model(s) used: Claude 4.6, Cursor
-->

Updated the AI Assistant section to clarify the integration with large language model (LLM) providers, emphasizing the use of connectors and providing a more specific reference to the LLM performance matrix for observability. This enhances user understanding of available models and their performance ratings.
@benironside benironside self-assigned this Mar 25, 2026
@benironside
benironside requested review from a team as code owners March 25, 2026 23:35
@github-actions

github-actions Bot commented Mar 25, 2026

Copy link
Copy Markdown
Contributor

Vale Linting Results

Summary: 1 suggestion found

💡 Suggestions (1)
File Line Rule Message
explore-analyze/ai-features/llm-guides/llm-connectors.md 32 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

@benironside benironside changed the title Internal 731 obs performance matrix applicability Clarifies which LLMs we support for Obs and Sec throughout AI docs Mar 26, 2026

@florent-leborgne florent-leborgne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice additions! How can we clarify the paths to the right performance pages too for the Search solution and the various AI features it includes? For example, is AI Assistant for Obs/Search still relevant for search? If not, is there a dedicated performance page for AB models (there's a page yes, but what is it relevant for?) cc @leemthompo in case you know more there.

I'm thinking that in addition to this, we may benefit from an orientation page if this is a regular concerns for our users like "LLM performance in the Elasticsearch platform" where we could clearly say that different models perform differently based on what feature/solution they're used and then link to the various places. Right now I feel like we have a (too) complex and incomplete web of interlinking which I'm not sure is fixing the underlying issue.

Also, similar to what is being done for EIS, specifying the (minimum) stack version for the models could be a good idea too.

Lastly, support folks seem to have particular needs/interests there, we should include them in reviewing what we produce around this issue IMO to make sure that what we deliver matches their expectations too.

@benironside

Copy link
Copy Markdown
Contributor Author

Thanks @florent-leborgne. As we discussed on Slack, I created sub-issues here. This PR will close one of them, and I'll work on the general page about LLM performance in a follow-up PR.

@lcawl lcawl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Text LGTM

@benironside
benironside enabled auto-merge (squash) March 28, 2026 00:02

@alaudazzi alaudazzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@benironside
benironside merged commit 82b47b1 into main Mar 28, 2026
7 checks passed
@benironside
benironside deleted the internal-731-obs-performance-matrix-applicability branch March 28, 2026 07:04
@leemthompo

leemthompo commented Mar 30, 2026

Copy link
Copy Markdown
Member

For example, is AI Assistant for Obs/Search still relevant for search?

It was never relevant for search, there's only two assistants, search team were focused initially on playground and now AB

If not, is there a dedicated performance page for AB models (there's a page yes, but what is it relevant for?)

https://www.elastic.co/docs/explore-analyze/ai-features/agent-builder/models#model-requirements

this is relevant for anyone using GA AB features in 9.3

the obs and sec AB functionalities are still in tech preview until 9.4, so the question of how we recommend models etc. for these different use cases needs to be taken up with security/obs PMs

@florent-leborgne

Copy link
Copy Markdown
Member

@leemthompo I'm not asking for myself :p I'm asking for anyone reading the docs - we have to think about how to make all of this clear to them

@leemthompo

leemthompo commented Mar 30, 2026

Copy link
Copy Markdown
Member

@florent-leborgne was just answering the two specific questions you pinged me about in this thread

we have to think about how to make all of this clear to them

I agree of course, was just letting folks in the thread understand that there isn't much of a story to tell for search users re. AI Assistant + the fact that how we recommend models for sec/obs in AB is TBD with PMs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants