Feat/auto instrumentation service name

[prathamesh-sonpatki]

Summary

This PR adds robust auto-instrumentation support with automatic service.name and deployment.environment detection, namespace-level instrumentation management, prevents conflicts with other operators (like Datadog), enables ESM support for Node.js apps, and fixes 7 configuration/documentation issues.

Changes

1. Service Name Resolution

Automatic service.name detection from Kubernetes metadata:

1. last9.io/service annotation (explicit override)
2. app.kubernetes.io/name label (K8s standard)
3. app.kubernetes.io/component label
4. Deployment name
5. app label
6. Container name (fallback)

2. Deployment Environment Auto-Detection

Automatic deployment.environment detection (replaces hardcoded local):

1. last9.io/env annotation (explicit override)
2. environment label
3. app.kubernetes.io/environment label (K8s standard)
4. Namespace name (fallback, now supports prefixed namespaces like staging-v2, prod-us-east)

3. Namespace-Level Auto-Instrumentation

• auto-instrument=all — instrument all namespaces (excluding system)
• auto-instrument=ns1,ns2 — whitelist specific namespaces
• auto-instrument-exclude=ns1,ns2 — blacklist specific namespaces (only works with auto-instrument=all)

System namespaces auto-excluded: kube-system, kube-public, kube-node-lease, cert-manager, istio-system, linkerd, monitoring, prometheus, grafana, argocd, flux-system, last9

4. Multi-Language Support

Language	Status
Java	Enabled
Python	Enabled
Node.js	Enabled (ESM + CJS)
.NET	Enabled
PHP	Not Supported (CRD limitation)
Go	Optional (eBPF)
Apache HTTPD	Optional
Nginx	Optional

5. Node.js ESM Auto-Instrumentation

Overrides the OTel Operator's default --require injection with --import to support both ESM and CJS Node.js apps. Without this, ESM apps get no framework-level instrumentation (missing http.route, incorrect span names like bare GET/POST).

• Uses --import /otel-auto-instrumentation-nodejs/autoinstrumentation.mjs
• The .mjs entry point registers import-in-the-middle hooks for ESM
• Falls back to --require behavior for CJS apps (backward compatible)
• Requires autoinstrumentation-nodejs image >= 0.57.0

6. Bug Fixes (7 issues resolved)

Critical

• Fixed hardcoded cluster.name: "minikube" in collector values
  • Changed to "unknown-cluster" default that matches script's auto-detection
  • Script already detects cluster name from kubectl context or cluster= parameter
  • Prevents production deployments from being tagged as "minikube"

High Priority

• Removed PHP false advertising

  • OTel Operator CRD does NOT support PHP auto-instrumentation
  • Removed inject-php annotation injection from setup script
  • Updated README language table: PHP | Not Supported | CRD doesn't support PHP, use SDK
  • Previously claimed "PHP Enabled" while injecting non-functional annotations

• Fixed auto-instrument-exclude silently ignored in whitelist mode

  • Script now validates and errors with clear message:

    ERROR: auto-instrument-exclude cannot be used with whitelist mode
    Either use: auto-instrument=all auto-instrument-exclude=ns1,ns2
    Or use: auto-instrument=ns1,ns2 (without exclude)
    

  • Previously parsed the parameter but completely ignored it

Low Priority

• Deleted unused uninstrument_namespace() function

  • Dead code that was never called anywhere in the script

• Relaxed IsMatch regex for deployment.environment detection

  • Changed from strict ^(prod|staging|...)$ to prefix match ^(prod|staging|...)([-_].*)?$
  • Now matches real-world namespaces: staging-v2, prod-us-east, dev-team-a
  • Still rejects false matches: myproduction, unstaging

Documentation

• Documented annotation override behavior
  • Added warning that last9.io/service annotation unconditionally overrides SDK-set service.name
  • By design for operator control, but can surprise users when app's OTEL_SERVICE_NAME is ignored
  • Helps prevent confusion when application configuration appears to be ignored

7. Documentation Overhaul

• Added badges, "Why This Exists" section
• Documented all features with examples
• Added troubleshooting guide
• Added architecture diagram
• Documented annotation override behavior and limitations

Files Changed

File	Changes
instrumentation.yaml	Added OTEL_SERVICE_NAME via fieldRef, enabled .NET, removed hardcoded deployment.environment=local, added NODE_OPTIONS for ESM support
last9-otel-collector-values.yaml	Added transform processors for service.name and deployment.environment auto-detection, fixed hardcoded cluster.name="minikube", relaxed namespace regex
last9-otel-setup.sh	Added auto-instrument, auto-instrument-exclude options with validation, removed PHP injection, deleted dead code
README.md	Complete overhaul with updated features, troubleshooting, architecture diagram, annotation override warnings, corrected PHP status

Usage

# Full install with auto-instrumentation for all namespaces
./last9-otel-setup.sh \
  auto-instrument=all \
  token="Basic <your-token>" \
  endpoint="<your-otlp-endpoint>"

Override Examples

metadata:
  annotations:
    last9.io/service: "my-custom-service"
    last9.io/env: "production"

Test Plan

• Deploy to test cluster with auto-instrument=all
• Verify Java/Python/Node.js/.NET apps are auto-instrumented
• Confirm service.name is resolved from K8s labels
• Confirm deployment.environment is set from namespace (not hardcoded local)
• Test explicit override via last9.io/service and last9.io/env annotations
• Verify system namespaces are excluded
• Test coexistence with Datadog operator (no service name conflicts)
• Verify Node.js ESM apps produce spans with http.route (Express, Fastify, Koa, Hapi)
• Verify Node.js CJS apps still work with the --import flag
• Verify cluster.name is auto-detected (not hardcoded to "minikube")
• Test namespace patterns: staging-v2, prod-us-east-1 are detected correctly
• Verify auto-instrument-exclude with whitelist mode returns error

---

Generated with Claude Code

[ashishl9]

🔍 Test Results - PR #3 Deployment Testing

Tested the PR changes on a local Minikube cluster with real workloads. Found several critical issues that prevent the PR's features from being deployed.

---

Test Environment

• Cluster: Minikube (local)
• Test Apps: 8 running services (Java, Python, Node.js, Go, Nginx, Postgres, React)
• Operator Version: 0.129.1 (chart 0.92.1)
• Setup Command: Full installation with provided credentials

---

🔴 Critical Issues (Blockers)

Issue #1: Setup script downloads from WRONG repository ⚠️

File: last9-otel-setup.sh:38

Current:

DEFAULT_REPO="https://github.com/last9/l9-otel-operator.git"

Problem: The setup script clones from the old l9-otel-operator repository instead of the current last9-k8s-observability repository, completely overwriting all PR changes during deployment.

Evidence from script execution:

[INFO] Using repository: https://github.com/last9/l9-otel-operator.git
[INFO] Downloading configuration files from: https://github.com/last9/l9-otel-operator.git

Impact: 🚨 NONE of the PR's configuration changes actually get deployed!

Fix Required:

DEFAULT_REPO="https://github.com/last9/last9-k8s-observability.git"

---

Issue #2: Instrumentation CRD has unsupported fields ⚠️

Error when applying instrumentation.yaml:

Error from server (BadRequest): Instrumentation in version "v1alpha1" cannot be handled as a Instrumentation: 
strict decoding error: unknown field "spec.defaults.env", unknown field "spec.php"

Operator version tested: 0.129.1

Unsupported fields introduced in PR:

• spec.defaults.env - For setting OTEL_SERVICE_NAME via fieldRef
• spec.php - PHP instrumentation configuration

Impact: Cannot deploy the new instrumentation features. Backward compatibility broken.

Questions:

• Which operator version supports these fields?
• Should PR document minimum operator version requirement?
• Consider adding version compatibility matrix to README?

---

Issue #3: Collector deployed with OLD configuration ⚠️

Current deployed config (from old repo):

• ✅ Has transform/logs/last9 (but OLD version)
• ❌ Missing transform/traces/last9 processor from PR
• ❌ Traces pipeline has NO transform: receivers → k8sattributes → batch → exporter
• ❌ Still has hardcoded deployment.environment=local

Expected from PR:

• transform/traces/last9 with 6-level priority chain for service name
• Environment auto-detection (not hardcoded "local")
• Both logs and traces using consistent priority logic

Root cause: Same as Issue #1 - script downloads from wrong repo

Current deployed config:

transform/logs/last9:
  statements:
    - set(attributes["service.name"], attributes["k8s.container.name"]) where attributes["service.name"] == ""
    - set(attributes["deployment.environment"], "local")  # ❌ Still hardcoded!

PR's intended config (NOT deployed):

transform/traces/last9:
  statements:
    # Priority 1: last9.io/service annotation
    # Priority 2: resource.opentelemetry.io/service.name
    # Priority 3: app.kubernetes.io/name label
    # Priority 4: k8s workload name (deployment/statefulset/daemonset)
    # Priority 5: app label
    # Priority 6: container name
    # + deployment.environment auto-detection

---

🟡 Design Issues (Review Findings)

Issue #4: OTEL_SERVICE_NAME fieldRef will fail without label

File: instrumentation.yaml:16-19

defaults:
  env:
    - name: OTEL_SERVICE_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.labels['app.kubernetes.io/name']

Problem: If a pod doesn't have the app.kubernetes.io/name label, the container will fail to start with a field path error.

Real-world impact: Our test environment has 8 running pods, all using simple app=<name> labels (NOT app.kubernetes.io/name). This configuration would cause all of them to fail on restart.

Recommendation: Either:

1. Remove SDK-level OTEL_SERVICE_NAME entirely (rely on collector transform only) ← Recommended
2. Document that app.kubernetes.io/name is REQUIRED for all pods
3. Use init container or admission webhook to inject the env var safely

---

Issue #5: Duplicate service name resolution logic

Service name resolution happens in TWO places:

1. SDK level: instrumentation.yaml - OTEL_SERVICE_NAME from fieldRef
2. Collector level: last9-otel-collector-values.yaml - transform processor

Problems:

• These could diverge and cause confusion
• SDK-level value gets cached, requires pod restart for changes
• Collector-level is dynamic but adds processing overhead
• Which one "wins" is unclear to users

Recommendation: Choose ONE approach:

• Option A: SDK-only (faster, but requires pod restart for changes)
• Option B: Collector-only (dynamic, graceful fallback) ← Recommended

Current PR tries to do both, which may lead to unexpected behavior.

---

Issue #6: Environment fallback too permissive

File: last9-otel-collector-values.yaml:220-222

# Priority 4: namespace name (fallback)
- set(attributes["deployment.environment"], attributes["k8s.namespace.name"])

Problem: A namespace named "payment-service" would set deployment.environment=payment-service, which isn't an actual environment name.

Example scenarios:

• Namespace: microservices → environment: microservices ❌
• Namespace: team-alpha → environment: team-alpha ❌
• Namespace: production → environment: production ✅

Recommendation:

• Add validation to only accept common environment names (prod, production, staging, stage, dev, development, qa, test)
• Or default to "unknown" if namespace doesn't match expected patterns
• Or document prominently that production workloads MUST set explicit annotations

---

📋 Test Coverage Gaps

Could NOT test (due to deployment issues):

• ❌ Service name auto-detection from K8s standard labels
• ❌ Deployment environment auto-detection with priority chain
• ❌ .NET and PHP auto-instrumentation (CRD fields rejected)
• ❌ Namespace-level instrumentation (didn't use auto-instrument=all parameter)
• ❌ Priority chain fallback logic
• ❌ Annotation-based overrides (last9.io/service, last9.io/env)
• ❌ Coexistence with Datadog operator (claimed in PR description but no test evidence)

Verification needed:

• Has this been tested with the correct operator version that supports defaults.env and php fields?
• Has service name detection been tested with actual K8s standard labels (app.kubernetes.io/name)?
• Have edge cases for environment detection been tested?
• Has StatefulSet and DaemonSet service name detection been verified?
• Is there actual evidence of Datadog operator coexistence working?

---

✅ What Works

Despite the issues above, these components deployed successfully:

1. Setup script execution - Script completes end-to-end
2. All infrastructure deployed:
   • OpenTelemetry Operator (2/2 containers running)
   • Collector DaemonSet (1/1 running)
   • Prometheus monitoring stack (running)
   • Events agent (running)
3. Log collection - Collector is actively watching and collecting logs from test pods
4. Instrumentation CRD - Created successfully (with old config)

$ kubectl get pods -n last9
NAME                                                             READY   STATUS    RESTARTS   AGE
opentelemetry-operator-78f6684589-kfhvs                          2/2     Running   0          5m
last9-opentelemetry-collector-last9-otel-collector-agent-clzsc   1/1     Running   0          5m
last9-k8s-monitoring-kube-prometheus-0                           2/2     Running   0          5m
...

---

🎯 Recommended Actions Before Merge

HIGH PRIORITY (Blockers):

1. ✅ Fix DEFAULT_REPO in last9-otel-setup.sh:38 (Issue Add optional application metrics scraping via OTel Collector #1)
2. ✅ Document minimum operator version requirement for new CRD fields
3. ✅ Either:
   • Remove spec.defaults.env and spec.php until operator supports them, OR
   • Update operator version in script to one that supports these fields
4. ✅ Remove SDK-level OTEL_SERVICE_NAME or handle missing label gracefully (Issue Add GKE Autopilot support for Last9 OpenTelemetry stack #4)

MEDIUM PRIORITY:

1. ⚠️ Add validation for environment fallback logic (Issue adding gpu related setup #6)
2. ⚠️ Align logs and traces transform processors to use same logic
3. ⚠️ Document and clarify service name resolution strategy (SDK vs collector) (Issue Revert "Feat/auto instrumentation service name" #5)

TESTING:

1. 📝 Add integration test results showing features actually work
2. 📝 Test with K8s standard labels (app.kubernetes.io/name, app.kubernetes.io/environment)
3. 📝 Test fieldRef failure scenario with pods missing required labels
4. 📝 Add operator version compatibility matrix to README

---

📊 Summary

This PR introduces valuable features for auto-instrumentation and service discovery, but critical deployment issues prevent any of the new functionality from working. The main blocker is that the setup script downloads configuration from the wrong repository, negating all PR changes.

Once Issue #1 is fixed and CRD compatibility is resolved, the features should work as intended. However, additional testing is needed to verify the complete workflow, especially edge cases around missing labels and namespace naming.

Status: ⚠️ Do not merge - Critical issues must be resolved first.

cc @prathamesh-sonpatki

[prathamesh-sonpatki]

Fixes Applied for All Review Comments

Thanks for the thorough testing @ashishl9! All 6 issues have been addressed. Here's a summary:

---

Critical Issues (Blockers) — FIXED

Issue #1: Setup script downloads from WRONG repository

• Fixed DEFAULT_REPO in last9-otel-setup.sh:38 from l9-otel-operator → last9-k8s-observability
• Also fixed the curl one-liner URL in the comment at line 27
• This also resolves Issue Feat/auto instrumentation service name #3 (collector deployed with old config) since both had the same root cause

Issue #2: Instrumentation CRD has unsupported fields

• Removed spec.defaults.env — this field does NOT exist in the Defaults struct (confirmed from operator source). Defaults only has useLabelsForResourceAttributes (bool)
• Removed spec.php — PHP is NOT supported by the OTel Operator Instrumentation CRD (supported: Java, Python, Node.js, .NET, Go, Apache HTTPD, Nginx)
• Replaced with spec.defaults.useLabelsForResourceAttributes: true which auto-maps app.kubernetes.io/name → service.name
• Updated versions to latest:
  • Operator chart: 0.92.1 → 0.105.1 (latest, released 2026-02-10)
  • Collector chart: 0.126.0 → 0.145.0 (latest, released 2026-02-06)
  • Collector image: 0.126.0 → 0.145.0

Issue #4: OTEL_SERVICE_NAME fieldRef will fail without label

• Resolved by removing spec.defaults.env entirely (see Issue Detect topology of resources #2 fix above)
• spec.defaults.useLabelsForResourceAttributes: true handles this gracefully — if labels are missing, it simply doesn't set the attribute (no pod crash)

---

Medium Priority — FIXED

Issue #5: Duplicate service name resolution logic

• Resolved: SDK now uses useLabelsForResourceAttributes (labels → resource attributes), Collector handles fallback chain via transform processors
• No duplication — SDK uses labels when present, collector provides cascading fallbacks for everything else

Issue #6: Environment fallback too permissive

• Added IsMatch() validation on namespace name — only known environment patterns are accepted as deployment.environment:
  prod, production, staging, stage, dev, development, qa, test, uat, sandbox, preprod, pre-prod, demo, perf, load-test, integration, sit
• If namespace doesn't match, falls back to "unknown" instead
• Applied consistently to both transform/logs/last9 and transform/traces/last9 processors

---

Additional Improvements

Aligned logs and traces transform processors:

• Logs transform now uses the same 6-level priority chain as traces (was missing statefulset/daemonset detection and legacy last9.io/service-name annotation)

Added CI validation pipeline (.github/workflows/validate.yml):

• ShellCheck for bash script linting
• yamllint for YAML syntax validation
• kubeconform for Instrumentation CRD schema validation (would have caught Issues Detect topology of resources #2 and Add GKE Autopilot support for Last9 OpenTelemetry stack #4 automatically)
• helm template rendering validation for collector values
• otelcol-contrib validate via Docker for OTTL statement validation (the only way to validate OTTL offline)

Local validation results:

• ✅ helm template renders successfully with chart version 0.145.0
• ✅ otelcol-contrib validate passes — all OTTL statements (including IsMatch() regex patterns) are syntactically correct

---

Files Changed

File	Changes
last9-otel-setup.sh	Fixed DEFAULT_REPO URL, updated operator chart to 0.105.1, collector chart to 0.145.0
instrumentation.yaml	Replaced invalid spec.defaults.env with useLabelsForResourceAttributes: true, removed spec.php
last9-otel-collector-values.yaml	Updated collector image to 0.145.0, aligned logs/traces processors, added env validation with IsMatch() + "unknown" fallback
.github/workflows/validate.yml	New CI pipeline for automated validation

cc @ashishl9

[ashishl9]

All 7 issues from the initial review have been addressed in the Feb 17 commit. CI pipeline added, cluster.name hardcode fixed, PHP inconsistency resolved, and namespace exclude validation added. LGTM.