AWS Fanout Pattern (SNS -> SQS) with Terraform and Python Lambdas

This repository demonstrates an AWS fanout architecture built with Terraform and Python Lambdas. An HTTP API publishes messages to an SNS topic which fans out to multiple SQS queues. Each queue has a dedicated consumer Lambda that persists the message to DynamoDB only after successful processing. Failed messages are retried up to 3 times before landing in a Dead-Letter Queue (DLQ).

What this project provides

SNS topic (fanout) that publishes to multiple SQS queues
Publisher Lambda (API Gateway HTTP API) — only publishes to SNS, no persistence
Consumer Lambdas — one per SQS queue, each saves the processed message to DynamoDB
Dead-Letter Queues — one per main queue, receives messages after 3 failed consumer attempts
DynamoDB table — written by consumers on success (proof of processing, not just receipt)
CloudWatch Log Groups, Metric Filters and Alarms for SNS, SQS, DLQs and Lambda errors
60-second delivery delay on all queues — messages sit in SQS for 60s in Delayed state before consumers can pick them up; use make poll to watch the lifecycle in real time

Prerequisites

Terraform >= 1.0
AWS CLI configured with credentials and default region
Python 3.13
Node.js >= 18 and npm (required by Serverless Framework and its plugins)

Run make requirements to install all tool dependencies at once (see Makefile below).

Quick structure

aws_fanout_pattern/
├─ Makefile                      # All common tasks (see below)
├─ terraform/
│  ├─ main.tf                    # Terraform configuration (SNS, SQS, Lambda, IAM, CloudWatch)
│  ├─ lambda/
│  │  ├─ publisher.py            # Publisher Lambda: publishes to SNS
│  │  ├─ sqs_logger.py           # Logger Lambda: writes SQS/SNS events to CloudWatch log groups
│  │  └─ test_fail.py            # (optional) test lambda used during debugging
│  └─ outputs.json               # Generated by Terraform after apply, consumed by serverless/
├─ serverless/
│  ├─ serverless.yml             # Serverless Framework configuration
│  ├─ package.json               # Node.js deps: serverless + serverless-python-requirements
│  ├─ requirements.txt           # Python deps packaged into Lambda (boto3)
│  ├─ handlers/
│  │  ├─ publisher.py            # HTTP API Lambda handler (publishes to SNS)
│  │  └─ consumer.py             # SQS consumer Lambda handler (saves to DynamoDB)
│  ├─ publish.py                 # Local script to publish a test message to SNS
│  └─ receive_sqs.py             # Local script to poll SQS manually
└─ README.md

Makefile

All common operations are available as make targets. Run make help to see them:

Usage: make <target> [REGION=<aws-region>]

  REGION  AWS region to deploy to (default: eu-west-1)

Targets:
  requirements       Install all local dependencies (npm + pip + serverless)
  terraform-init     Initialize Terraform providers
  terraform-apply    Deploy infrastructure (SNS, SQS, IAM, CloudWatch)
  serverless-deploy  Deploy Lambda functions via Serverless Framework
  publish            Publish a test message to SNS
  poll               Watch all 3 SQS queues every 10s (Delayed/InFlight/Visible) — Ctrl+C to stop
  destroy            Remove ALL resources (Serverless + Terraform)

Examples:
  make terraform-apply REGION=eu-west-1
  make serverless-deploy REGION=eu-west-1
  make publish && make poll

Target details

Target	What it does
`make requirements`	Creates a Python venv at `serverless/.venv/`, installs Python deps, Serverless Framework v3 globally, and the `serverless-python-requirements` plugin
`make terraform-init`	Runs `terraform init` inside `terraform/`
`make terraform-apply [REGION=...]`	Runs `terraform apply` with `aws_region` and writes `outputs.json`
`make serverless-deploy [REGION=...]`	Runs `npm install` + `sls deploy --region` inside `serverless/`
`make publish [REGION=...]`	Publishes a test message to SNS using the venv Python
`make poll [REGION=...]`	Watches all 3 SQS queues every 10s showing Delayed / InFlight / Visible counts — runs until `Ctrl+C`
`make destroy [REGION=...]`	Removes all resources: Serverless stack first, then Terraform infrastructure

Region selection

All deployment targets accept a REGION parameter. If not specified, eu-west-1 is used by default.

# Deploy to Ireland (default)
make terraform-apply REGION=eu-west-1
make serverless-deploy REGION=eu-west-1

# Deploy to São Paulo
make terraform-apply REGION=sa-east-1
make serverless-deploy REGION=sa-east-1

# Deploy to Singapore
make terraform-apply REGION=ap-southeast-1
make serverless-deploy REGION=ap-southeast-1

Always use the same REGION for terraform-apply and serverless-deploy so both point to the same region.

Deploy (step-by-step)

1. Install all requirements

make requirements

This installs:

A Python virtual environment at serverless/.venv/ (avoids PEP 668 system-package conflicts)
boto3 and any other Python deps into the venv (from serverless/requirements.txt)
serverless@3 globally via npm
serverless-python-requirements plugin (from serverless/package.json)

2. Initialize and apply Terraform

make terraform-init
make terraform-apply REGION=eu-west-1

This provisions SNS, SQS queues, DLQs, DynamoDB, IAM roles, and CloudWatch Log Groups, and writes terraform/outputs.json which is consumed by the Serverless deployment.

Note: The Terraform configuration uses a local-exec provisioner to set SNS topic attributes via the AWS CLI. Ensure the AWS CLI is configured with the correct credentials and region.

3. Deploy Lambda functions

make serverless-deploy REGION=eu-west-1

Packages and deploys the publisher and consumer Lambda functions using the Serverless Framework.

4. Publish a test message and observe the fan-out

Open two terminals:

Terminal 1 — start the watcher before publishing:

make poll

Terminal 2 — publish the message:

make publish

make poll refreshes every 10 seconds and shows the full message lifecycle:

─────────────────────────────────────── 14:32:00
  Queue 1 → Delayed: 1  |  InFlight: 0  |  Visible: 0   ← message just arrived
  Queue 2 → Delayed: 1  |  InFlight: 0  |  Visible: 0
  Queue 3 → Delayed: 1  |  InFlight: 0  |  Visible: 0

─────────────────────────────────────── 14:33:00   ← after 60s delay
  Queue 1 → Delayed: 0  |  InFlight: 1  |  Visible: 0   ← Lambda processing
  Queue 2 → Delayed: 0  |  InFlight: 1  |  Visible: 0
  Queue 3 → Delayed: 0  |  InFlight: 1  |  Visible: 0

─────────────────────────────────────── 14:33:10
  Queue 1 → Delayed: 0  |  InFlight: 0  |  Visible: 0   ← processed ✓
  Queue 2 → Delayed: 0  |  InFlight: 0  |  Visible: 0
  Queue 3 → Delayed: 0  |  InFlight: 0  |  Visible: 0

Press Ctrl+C to stop polling. Then verify the 3 DynamoDB records were written:

TABLE=$(jq -r .dynamodb_table_name terraform/outputs.json)
aws dynamodb scan \
  --region eu-west-1 \
  --table-name $TABLE \
  --query "Items[*].{consumer:consumer_id.S, body:body.S, at:processed_at.N}" \
  --output table

Testing the fan-out pattern

The goal is to verify that one message published = one DynamoDB record per consumer (3 records total), and that a consumer failure routes the message to its DLQ after 3 retries.

All commands below read from terraform/outputs.json — make sure Terraform has been applied first.

# Convenience variables — run these once in your shell
REGION=eu-west-1
API_ENDPOINT=$(jq -r .api_endpoint terraform/outputs.json)
SNS_ARN=$(jq -r .sns_topic_arn terraform/outputs.json)
TABLE=$(jq -r .dynamodb_table_name terraform/outputs.json)
QUEUE_1_URL=$(jq -r '.queues[0].url' terraform/outputs.json)
QUEUE_2_URL=$(jq -r '.queues[1].url' terraform/outputs.json)
QUEUE_3_URL=$(jq -r '.queues[2].url' terraform/outputs.json)
DLQ_1_URL=$(jq -r '.dlqs[0].url' terraform/outputs.json)

Test 1 — Publish and observe the fan-out in real time

The queues have a 60-second delivery delay: messages arrive in SQS immediately in Delayed state and consumers cannot pick them up for 60 seconds. make poll watches this lifecycle automatically every 10 seconds.

Terminal 1:

make poll

Terminal 2:

# Publish via Makefile (uses publish.py + SNS ARN from outputs.json)
make publish

# Or publish via the HTTP API directly
curl -s -X POST "$API_ENDPOINT/publish" \
  -H 'Content-Type: application/json' \
  -d '{"text": "fanout-test-1"}' \
  -w '\nHTTP_STATUS:%{http_code}\n'

Watch Terminal 1 progress through three phases:

Phase	Delayed	InFlight	Meaning
0–60s	1	0	Message held in delay window
~60s	0	1	Consumer Lambda executing
done	0	0	Processed and saved to DynamoDB

After polling completes, verify 3 DynamoDB records — one per consumer:

aws dynamodb scan \
  --region $REGION \
  --table-name $TABLE \
  --filter-expression "contains(body, :v)" \
  --expression-attribute-values '{":v": {"S": "fanout-test-1"}}' \
  --query "Items[*].{consumer:consumer_id.S, message_id:message_id.S, at:processed_at.N}" \
  --output table

Expected: 3 rows, one for each consumer_1, consumer_2, consumer_3.

Test 2 — Publish directly to SNS (bypass API Gateway)

Useful to test the fan-out in isolation, without API Gateway involved.

aws sns publish \
  --region $REGION \
  --topic-arn $SNS_ARN \
  --message '{"text": "fanout-test-direct"}'

Then repeat the DynamoDB scan from Test 1 with fanout-test-direct. Expect 3 new records.

Test 3 — Verify queues are empty after processing

After a successful fan-out all messages should have been consumed. Check that no messages are stuck in the queues.

for URL in $QUEUE_1_URL $QUEUE_2_URL $QUEUE_3_URL; do
  aws sqs get-queue-attributes \
    --region $REGION \
    --queue-url "$URL" \
    --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible ApproximateNumberOfMessagesDelayed \
    --query "Attributes" \
    --output table
done

Expected: all three counters at 0 on every queue.

Test 4 — Verify DLQ behaviour (failure path)

To confirm messages route to the DLQ after 3 failed processing attempts, temporarily break consumer_1 by setting an invalid DDB_TABLE_NAME via the AWS console or CLI, then publish a message.

# 1. Check DLQ is empty before the test
aws sqs get-queue-attributes \
  --region $REGION \
  --queue-url $DLQ_1_URL \
  --attribute-names ApproximateNumberOfMessages \
  --query "Attributes.ApproximateNumberOfMessages"

# 2. Publish a message (with consumer_1 broken, it will fail 3 times)
aws sns publish \
  --region $REGION \
  --topic-arn $SNS_ARN \
  --message '{"text": "dlq-test"}'

# 3. Wait ~90 seconds (3 retries × 30s visibility timeout), then check DLQ
aws sqs get-queue-attributes \
  --region $REGION \
  --queue-url $DLQ_1_URL \
  --attribute-names ApproximateNumberOfMessages \
  --query "Attributes.ApproximateNumberOfMessages"

Expected: DLQ count goes from "0" to "1". consumer_2 and consumer_3 still succeed and write to DynamoDB normally.

# 4. Inspect the message in the DLQ (without deleting it)
aws sqs receive-message \
  --region $REGION \
  --queue-url $DLQ_1_URL \
  --max-number-of-messages 1 \
  --visibility-timeout 0 \
  --query "Messages[0].Body"

Test 5 — Check CloudWatch alarms for DLQ

After Test 4, the DLQ alarm should have triggered.

aws cloudwatch describe-alarms \
  --region $REGION \
  --alarm-name-prefix "dlq-messages-visible" \
  --query "MetricAlarms[*].{Name:AlarmName, State:StateValue, Reason:StateReason}" \
  --output table

Expected: at least one alarm in ALARM state for the queue that had the failing consumer.

Cleanup

make destroy REGION=eu-west-1

Notes

In production you might want to:
- Give each Lambda its own execution role with least-privilege permissions.
- Use structured JSON logs so CloudWatch metric filters are more precise.
- Configure SNS subscription confirmations and security settings for your environment.
- Remove or reduce the delay_seconds on the SQS queues (set to 60s here for educational purposes only).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

AWS Fanout Pattern (SNS -> SQS) with Terraform and Python Lambdas

What this project provides

Prerequisites

Quick structure

Makefile

Target details

Region selection

Deploy (step-by-step)

1. Install all requirements

2. Initialize and apply Terraform

3. Deploy Lambda functions

4. Publish a test message and observe the fan-out

Testing the fan-out pattern

Test 1 — Publish and observe the fan-out in real time

Test 2 — Publish directly to SNS (bypass API Gateway)

Test 3 — Verify queues are empty after processing

Test 4 — Verify DLQ behaviour (failure path)

Test 5 — Check CloudWatch alarms for DLQ

Cleanup

Notes

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
serverless		serverless
terraform		terraform
.gitignore		.gitignore
Makefile		Makefile
README.md		README.md
architecture.png		architecture.png

Folders and files

Latest commit

History

Repository files navigation

AWS Fanout Pattern (SNS -> SQS) with Terraform and Python Lambdas

What this project provides

Prerequisites

Quick structure

Makefile

Target details

Region selection

Deploy (step-by-step)

1. Install all requirements

2. Initialize and apply Terraform

3. Deploy Lambda functions

4. Publish a test message and observe the fan-out

Testing the fan-out pattern

Test 1 — Publish and observe the fan-out in real time

Test 2 — Publish directly to SNS (bypass API Gateway)

Test 3 — Verify queues are empty after processing

Test 4 — Verify DLQ behaviour (failure path)

Test 5 — Check CloudWatch alarms for DLQ

Cleanup

Notes

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages