📚 Multi‑Language Web Scraper Pipeline

A production‑ready web scraper that combines Bash (orchestration), Go (concurrent scraping), and Python (data cleaning) to extract book data from books.toscrape.com – a safe, dummy e‑commerce site perfect for testing.

---

📖 Table of Contents

• Why Polyglot?
• Features
• How It Works (Data Flow)
• File Overview
• Prerequisites
• Installation & Setup
• Usage
• Output Format
• Architecture Deep Dive
• Error Handling & Reliability
• Customisation
• Troubleshooting
• What I Learned

---

Why Polyglot?

Instead of forcing one language to do everything, this project uses the right tool for each subtask:

Stage	Language	Strengths applied
Orchestration	Bash	Glue code, argument parsing, exit code propagation, signal trapping (trap), temp file cleanup.
Scraping	Go	Native concurrency (goroutines), fast HTTP client, static binary, graceful shutdown via context.
Cleaning	Python	Expressive data wrangling, easy CSV/JSON handling, quick iteration for validation logic.

This separation makes each component simpler, testable, and reusable – and it’s a perfect example of the Unix philosophy: do one thing well and pipe between tools.

Inspiration

This project was directly inspired by this video – it opened my eyes to the power of mixing languages in a single codebase. Seeing how seamlessly different runtimes could collaborate through simple streams made me realise that polyglot pipelines aren’t just possible, they’re practical for solving real‑world problems. That video sent me on a quest to learn how to combine Bash, Go, and Python in a scraper that’s robust, fast, and a joy to maintain. 🙌

Features

✅ Concurrent page fetches (2 req/s) with goroutines
✅ Automatic retries (3 attempts with backoff)
✅ Graceful shutdown on Ctrl+C
✅ JSON lines output from Go → easy to inspect or replay
✅ Deduplication and data validation in Python
✅ CSV output with clean columns
✅ Full error logging to stderr (so stdout is pure data)
✅ Timed runs and exit code propagation

How It Works (Data Flow)

Bash script (run_scraper.sh)
   │
   ├─ Starts Go scraper (scraper.go)
   │    │
   │    └─ Writes raw JSON lines → temporary file
   │
   └─ After Go completes, calls Python cleaner (cleaner.py)
        │
        └─ Reads temp file, cleans data → data/books.csv

Why a temp file instead of a direct pipe?
Piping go run scraper.go | python cleaner.py hides the exit code of the Go process when it fails. The tempfile approach lets the script check each stage’s exit code independently, then feed the file to Python – cleaner and more reliable.

File Overview

File	Role
run_scraper.sh	Orchestrator: parses --max-pages, runs Go, runs Python, cleans up.
scraper.go	Concurrent HTTP scraper with rate limiting, retries, and JSONL output.
cleaner.py	Reads JSONL, transforms fields, deduplicates, writes CSV + summary.

Prerequisites

• Go 1.20 or later
• Python 3.9 or later (only standard library required)
• Bash 4+ (Linux, macOS, WSL, or Git Bash on Windows)

Installation & Setup

1. Clone or create the project folder:

   mkdir scraper && cd scraper # this creates the directory where the project goes

2. Create the three files (run_scraper.sh, scraper.go, cleaner.py) with the code from your AI assistant.

3. Make the orchestrator executable:

   chmod +x run_scraper.sh

4. Install Go dependencies (the script does this automatically on first run, but you can do it manually):

   go mod init scraper
   go get github.com/PuerkitoBio/goquery

That’s it. No Python packages needed – the script uses only csv, json, re, and sys.

Usage

./run_scraper.sh                    # scrape first 5 pages (default)
./run_scraper.sh --max-pages 20     # scrape up to 20 pages
./run_scraper.sh --max-pages 50     # be mindful of target site rate limits

During execution you’ll see (on stderr):

[2025-03-25 10:00:01] [INFO] Starting scrape for up to 5 pages...
[2025-03-25 10:00:02] [INFO] Scraped page 1: 20 books
...
[2025-03-25 10:00:05] [INFO] Cleaning data...
[2025-03-25 10:00:05] [INFO] Summary: received 85 books, 85 unique, avg price £32.15
Scraping succeeded
Duration: 0m 4.2s

Output Format

All output goes to the data/ folder (created automatically):

data/
  └── books.csv

CSV columns: title, price, rating, available

Example row:

"A Light in the Attic",51.77,3,true
"Tipping the Velvet",53.74,1,true

Architecture Deep Dive

Go Scraper (scraper.go)

• Concurrency pattern: one goroutine per page URL, but rate limited by a time.Ticker (500ms interval → max 2 requests/sec). This respects the target server while still being fast.
• Retry logic: up to 3 attempts with exponential backoff (1s, 2s, 4s). Retries happen on network errors, timeouts, or non-200 HTTP status codes.
• Cancellation: uses context.WithCancel – on SIGINT/SIGTERM the root context is cancelled, all in-flight goroutines abort and the program exits cleanly.
• Output: each book as one JSON object on stdout, e.g.:

  {"title":"A Light in the Attic","price":"£51.77","rating":5,"available":true}

Python Cleaner (cleaner.py)

• Stream processing: reads sys.stdin line by line – no memory explosion even for huge files.
• Transforms:
  • Price: "£51.77" → 51.77 (float) using regex [^0-9.]
  • Rating: "One", "Two", … "Five" → 1–5
  • Availability: string detection → boolean
• Deduplication: keeps a set of seen titles; first occurrence wins.
• Validation: skips lines with invalid price (≤0 or non‑numeric) and logs a warning.
• Final CSV: uses Python’s csv writer with quoting for safety.

Bash Orchestrator (run_scraper.sh)

• Strict mode: set -euo pipefail – fails on any error, unset variable, or broken pipe.
• Temporary file: created with mktemp to hold raw JSONL from Go.
• Exit code checking: runs Go and checks $? before feeding data to Python.
• Trap cleanup: trap cleanup EXIT INT TERM removes the temp file even if the script is interrupted.
• Timing: uses SECONDS shell variable to measure total runtime.

Error Handling & Reliability

Failure scenario	How it’s handled
Network error while fetching	Go retries up to 3 times; if still failing, skips that page and logs error.
Target site returns 5xx	Same retry logic.
Malformed HTML on a page	Go logs the error, emits no JSON for that page – pipeline continues.
Python receives invalid JSON	Cleaner logs a warning and skips the line.
Ctrl+C during Go scrape	Go receives SIGINT, cancels context, exits with code 1; temp file stays, script stops.
Disk full while writing CSV	Python raises exception → script prints failure and exits with code 1.

All errors are printed to stderr with timestamps, so they don’t contaminate the data stream. The script’s final exit code is non‑zero if any stage fails.

Customisation

Adjust concurrency / rate limit

Edit scraper.go:

ticker := time.NewTicker(500 * time.Millisecond) // 500ms = 2 req/s

Change to 1000 * time.Millisecond for 1 req/s.

Scrape a different website

Modify scraper.go – change baseURL and the CSS selectors inside fetchAndParsePage().

Add more fields

• In scraper.go: add a field to the Book struct and populate it in parsePage().
• In cleaner.py: add a new column in the CSV writer and update the cleaning logic.

Troubleshooting

Problem	Likely fix
go: command not found	Install Go from go.dev/dl
Module not found: goquery	Run go mod init scraper && go get github.com/PuerkitoBio/goquery
Python imports fail	The script uses only stdlib; you may be running a different Python (e.g., 2.x) – ensure Python 3.9+
HTTP 429 (Too Many Requests)	Reduce --max-pages or increase ticker interval in scraper.go
No data/books.csv file created	Check stderr for errors; run with ./run_scraper.sh 2> log.txt to see full logs
Go scrapes but CSV is empty	Python’s validation might be rejecting all lines – check logs for warnings

What I Learned

• Composing languages via Unix streams – instead of forcing one runtime to do everything, let each language shine and connect them with simple pipes or temp files.
• Practical goroutine rate limiting – the ticker + channel pattern prevents accidental DOS while keeping concurrency simple.
• Graceful shutdown with context – cancelling goroutines on SIGINT is essential for clean scripts.
• set -euo pipefail + trap – transforms bash from a brittle “scripting language” into a reliable orchestrator.
• Logging discipline – all errors to stderr, timestamps added, leaves stdout pristine for JSON or CSV data. That small habit makes debugging a joy.
• Separation of concerns – the Go stage does only scraping and JSON output; the Python stage does only cleaning. Each can be tested in isolation.

---

License

MIT – use freely, modify, and share.

Enjoy building polyglot pipelines! 🚀