Skip to content

Add E2E tutorial: otelcol-contrib → EDOT Collector (inside Elastic Agent) → Elasticsearch with OTel native mode #6972

Description

@theletterf

Summary

There is no single end-to-end guide for the pipeline that many self-managed users are actually building:

otelcol-contrib → EDOT Collector (embedded in Elastic Agent) → Elasticsearch (mapping_mode: otel) → waterfall traces in Kibana APM

What exists today

What's missing

A step-by-step tutorial that covers:

  1. Configuring otelcol-contrib with the OTLP exporter pointing at the EDOT endpoint
  2. Installing and configuring Elastic Agent to run the EDOT Collector (standalone or Fleet-managed)
  3. Setting mapping_mode: otel on the Elasticsearch exporter
  4. Configuring the elasticapm connector correctly (exporter in traces pipeline + receiver in metrics pipeline)
  5. Stamping required resource attributes (deployment.environment, service.name) at the collector
  6. Verifying data with waterfall traces in Kibana APM

All gotchas should be called out explicitly (see related issues for individual gotcha entries).

Related issues

  • elasticapm connector misconfiguration gotcha
  • generic.otel template troubleshooting
  • deployment.environment callout for Service Map
  • Timestamp type silent 400 troubleshooting
  • EDOT-inside-Elastic-Agent discoverability

Elastic Docs AI Scoping 🤖

Docs issue scope

Summary

The issue requests a step-by-step tutorial for the pipeline: otelcol-contrib → EDOT Collector (embedded in Elastic Agent) → Elasticsearch with OTel native mode (mapping_mode: otel), ending in waterfall traces in Kibana APM. No linked code PRs were provided, but the feature is GA (Elastic Agent 9.2+) and the issue itself precisely identifies the gap: existing pages cover either otelcol-contrib → APM Server (the upstream-opentelemetry-collectors-language-sdks page) or Elastic Agent EDOT architecture concepts (the elastic-agent-as-otel-collector reference), but no tutorial assembles the complete pipeline end-to-end with configuration details, gotchas, and a verification step.

Request accuracy

Accurate — confirmed by inspection of the three referenced pages; none covers the otelcol-contrib → Elastic Agent EDOT → Elasticsearch (native OTel mode) pipeline as a tutorial.

Next action for author

Create a new tutorial page in solutions/observability/get-started/opentelemetry/use-cases/ covering the full otelcol-contrib → Elastic Agent EDOT → ES pipeline for self-managed, and add cross-references from the two existing pages.

Impact: High

Scope boundary

Serverless and Elastic Cloud Hosted quickstart pages are not affected; this tutorial is self-managed only. The existing architecture reference (elastic-agent-as-otel-collector.md) does not need substantive rewrites — only a cross-reference added.

Recommended documentation targets

Page URL Action Impact Confidence Why this page?
otelcol-contrib → EDOT (Elastic Agent) tutorial (new) solutions/observability/get-started/opentelemetry/use-cases/upstream-collector/index.md Create new page High High No existing tutorial covers this pipeline; use-cases section already hosts Kubernetes and LLMs tutorials — same pattern
Contrib OpenTelemetry Collectors and language SDKs https://www.elastic.co/docs/solutions/observability/apm/opentelemetry/upstream-opentelemetry-collectors-language-sdks Update existing page Medium High Existing page covers otelcol-contrib → APM Server only; should add a callout or link pointing to the new tutorial for the direct-to-ES EDOT path
Elastic Agent as an OpenTelemetry Collector https://www.elastic.co/docs/reference/fleet/elastic-agent-as-otel-collector Update existing page Low High Architecture reference page; add a "Getting started" pointer to the new tutorial

Recommendations

  1. Create solutions/observability/get-started/opentelemetry/use-cases/upstream-collector/index.md as a standalone tutorial (applies_to: stack: ga 9.2+, self-managed only). Cover all six steps listed in the issue: otelcol-contrib OTLP exporter config, Elastic Agent install and EDOT endpoint setup, mapping_mode: otel on the ES exporter, elasticapm connector wiring (traces exporter → metrics receiver), deployment.environment and service.name resource attributes, and Kibana APM waterfall verification. Call out each gotcha inline.
  2. Add a note and link in upstream-opentelemetry-collectors-language-sdks.md under the existing "Send data from a contrib OpenTelemetry Collector" section pointing to the new tutorial for users who want the direct-to-Elasticsearch path through Elastic Agent EDOT.
  3. Add a "Getting started" or "Tutorial" cross-reference in the FAQ or at the bottom of elastic-agent-as-otel-collector.md.

Notes

  • Content type: The new page is a Tutorial (single-goal, multi-step, specific toolchain with verification). It should remain a standalone page and not be merged into the existing quickstart pages.
  • applies_to tagging: stack: ga 9.2+ — Elastic Agent embeds EDOT Collector starting in 9.2. Do not tag serverless.
  • mapping_mode: otel and elasticapm connector: These are documented upstream in the EDOT Collector reference (outside docs-content). The new tutorial should link to those references rather than reproduce the full config option documentation.
  • Related gotcha issues mentioned in the body (connector misconfiguration, generic.otel template, deployment.environment for Service Map, timestamp 400 error, EDOT discoverability): each likely warrants its own entry in troubleshoot/ingest/opentelemetry/edot-collector/ — scoping those is out of scope for this issue but should be tracked as follow-on.
  • TOC update: Add the new use-case entry under observability/get-started/opentelemetry/use-cases/ in solutions/toc.yml.

Generated by Issue Scope Analyzer for issue #6972 · 160.4 AIC · ⌖ 18.5 AIC · ⊞ 31K ·

Metadata

Metadata

Assignees

Labels

Planned-S24SKI work planned for Sprint 24 (Jun 7-20)Team:IngestIssues owned by the Ingest Docs Teamsource:webIssues originating from the elastic.co docs

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions