feat(ecs-fargate): EventBridge lifecycle forwarder + OTel collector improvements

[prathamesh-sonpatki]

Summary

• New: eventbridge-task-lifecycle/ — captures killed/stopped ECS task events (stop reason, exit codes) and forwards to Last9 as searchable logs via EventBridge → no Lambda required
• Fix: nodejs-express-large-logs/ — OTel collector config improvements for better visibility of stopped tasks

EventBridge lifecycle forwarder

When ECS kills a task (OOM, health check failure, scale-in), the OTel metrics receiver stops emitting immediately. The stop reason and exit codes are never captured. This forwarder bridges that gap.

How it works:

ECS Task stops → EventBridge "ECS Task State Change" event
  → API Destination (no Lambda) → POST to Last9 /json/v2
      Body: [{"taskArn":"...","stopCode":"EssentialContainerExited","containers":[{"exitCode":137}],...}]

Two CloudFormation templates:

• cloudformation-no-lambda.yaml ← recommended: EventBridge API Destination posts directly to Last9 /json/v2 using an input transformer [<detail>]. Zero code.
• cloudformation.yaml: Lambda variant that wraps events in OTLP format (useful if structured OTLP attributes are needed for metric correlation)

Correlation key: taskArn in log body ↔ aws_ecs_task_arn label in ECS metrics

OTel config fixes (nodejs-express-large-logs)

Change	Why
collection_interval: 60s → 10s	Captures task state during 30s ECS shutdown window before SIGKILL
Added memory_limiter (256 MiB)	Prevents collector OOM under log bursts
Added traces pipeline	OTLP receiver was declared but traces were silently dropped
Added otlp to logs pipeline	Enables OTLP log ingestion alongside FluentBit
batch.timeout: 10s → 5s	Faster flush during graceful shutdown
Pin collector to 0.149.0	Reproducible builds
Add stopTimeout: 30	Gives collector 30s to flush before SIGKILL

Validation

• Lambda handler logic validated with sample ECS stop event (Python)
• Input transformer output [<detail>] validated as valid JSON array
• Go lifecycle reporter passes go vet

🤖 Generated with Claude Code

[karthikeyangs9]

The YAML block scalar > here produces a trailing newline in the resolved string, which violates the AWS API Destination description regex constraint (Member must satisfy regular expression pattern: .*). CloudFormation fails with CREATE_FAILED on this resource.

Fix: use a single-line string instead:

Description: Posts ECS task stop events to Last9 HTTP log ingestion endpoint.

[karthikeyangs9]

The resources prefix filter here uses the cluster ARN format (arn:aws:ecs:REGION:ACCOUNT:cluster/CLUSTER-NAME), but the resources array in ECS Task State Change events contains task ARNs (arn:aws:ecs:REGION:ACCOUNT:task/CLUSTER-NAME/TASK-ID).

Since task/ ≠ cluster/, this prefix will never match and the rule silently drops all events when ECSClusterArn is provided.

Verified by deploying this template with a real ECS cluster — TriggeredRules stayed at 0 until the resources filter was removed.

Fix: filter on detail.clusterArn instead, which contains the actual cluster ARN:

      EventPattern:
        source:
          - aws.ecs
        detail-type:
          - ECS Task State Change
        detail:
          lastStatus:
            - STOPPED
          clusterArn: !If
            - FilterByCluster
            - - !Ref ECSClusterArn
            - !Ref "AWS::NoValue"

[karthikeyangs9]

The InputTransformer with template [<detail>] produces invalid JSON for Last9's /json/v2 endpoint. EventBridge serializes the <detail> variable as a JSON string (double-encoded), resulting in ["{\"taskArn\":...}"] instead of [{"taskArn":...}]. Every invocation fails silently.

Verified by deploying two rules targeting the same API Destination — one with InputTransformer and one without:

• With InputTransformer: TriggeredRules: 7, FailedInvocations: 7
• Without InputTransformer: TriggeredRules: 7, FailedInvocations: 0

Fix: remove the InputTransformer entirely. The raw EventBridge envelope (which includes source, detail-type, time, region, and detail) is a valid JSON object that /json/v2 accepts as-is. This actually provides richer data in Last9 since it includes event metadata alongside the ECS detail.