AGENTS.md

Instructions for Claude Code and contributors working on this repository.

Project Overview
Commands
Architecture
Adding a New Feed
Deprecating a Feed
Troubleshooting
GitHub Actions

Project Overview

RSS Feed Generator creates RSS feeds for blogs that don't provide them natively. Feed generators scrape blog pages and output feed_*.xml files to the feeds/ directory. A GitHub Action runs hourly to regenerate and commit updated feeds.

Commands

# Environment setup
make env_setup            # Install dependencies (uses uv sync)
make dev_setup            # Install dev dependencies + pre-commit hooks

# Generate feeds
make feeds_generate_all   # Run all feed generators
make feeds_<name>         # Run specific feed (e.g., feeds_ollama, feeds_anthropic_news)

# Development
make dev_lint             # Check code with ruff
make dev_lint_fix         # Auto-fix and format with ruff
make dev_format           # Alias for dev_lint_fix
make dev_test_feed        # Run test feed generator

# Run single generator directly
uv run feed_generators/ollama_blog.py

# CI/CD
make ci_trigger_feeds_workflow    # Trigger GitHub Action manually
make ci_run_feeds_workflow_local  # Test workflow locally with act

Architecture

feed_generators/           # Python scripts that scrape blogs and generate RSS
  run_all_feeds.py         # Orchestrator that runs all generators
  utils.py                 # Shared utilities (setup_feed_links, get_project_root, etc.)
  <source>_blog.py         # Individual feed generators
feeds/                     # Output directory for feed_*.xml files
cache/                     # JSON cache for paginated/dynamic feeds
makefiles/                 # Modular Makefile includes (feeds.mk, env.mk, dev.mk, ci.mk)

Feed Generator Patterns

Three patterns exist based on how the target site loads content:

1. Simple Static (Default)

For blogs where all content loads on first request.

Examples: ollama_blog.py, paulgraham_blog.py, hamel_blog.py

Key functions:

fetch_blog_content(url) - HTTP request with User-Agent header
parse_blog_html(html) - BeautifulSoup parsing for posts
generate_rss_feed(posts) - Create feed using feedgen
save_rss_feed(fg, name) - Write to feeds/feed_{name}.xml

Cache: Not needed (all posts fetched each run)

2. Pagination + Caching

For blogs with "Load More" or pagination that uses URL query params (?page=2).

Examples: cursor_blog.py, dagster_blog.py

Key functions:

load_cache() / save_cache(posts) - JSON persistence in cache/<source>_posts.json
merge_posts(new, cached) - Dedupe by URL, merge, sort by date
fetch_all_pages() - Follow pagination until no next link

Cache behavior:

First run / --full flag: Fetch all pages, populate cache
Incremental (default): Fetch page 1 only, merge with cache
Dedupe: By URL, sorted by date descending

3. Selenium + Click "Load More"

For JS-heavy sites where content loads dynamically via JavaScript button clicks.

Examples: anthropic_news_blog.py (reference implementation), anthropic_research_blog.py, openai_research_blog.py, xainews_blog.py

Key functions:

setup_selenium_driver() - Headless Chrome with undetected-chromedriver
fetch_news_content(max_clicks) - Load page, click buttons, return final HTML
load_cache() / save_cache(articles) - JSON persistence in cache/<source>_posts.json
merge_articles(new, cached) - Dedupe by link, merge, sort by date

Selenium specifics:

Uses undetected-chromedriver to avoid bot detection
Clicks "See more"/"Load more" button repeatedly
Waits for content to load between clicks
max_clicks parameter controls depth (20 for full, 2-3 for incremental)

Cache behavior (see anthropic_news_blog.py for reference):

First run / --full flag: Click up to 20 times, fetch all articles, populate cache
Incremental (default): Click 2-3 times (recent articles), merge with cache
Dedupe: By URL, sorted by date descending

When to Use Each Pattern

Site Behavior	Pattern	Example	Cache?
All posts on single page	Simple Static	`ollama_blog.py`	No
URL-based pagination (`?page=2`)	Pagination + Caching	`dagster_blog.py`	Yes
JS button loads more content	Selenium + Click	`anthropic_news_blog.py`	Yes
JS-rendered page (curl returns empty shell)	Selenium + Wait	`xainews_blog.py`	Yes

Key libraries: requests, beautifulsoup4, feedgen, selenium, undetected-chromedriver

Feed Link Setup (Important)

The main <link> element must point to the original blog, not the feed URL. Use the helper:

from utils import setup_feed_links

fg = FeedGenerator()
# ... set title, description, etc.
setup_feed_links(fg, blog_url="https://example.com/blog", feed_name="example")

Why this matters: In feedgen, link order determines which URL becomes the main <link>:

rel="self" must be set first → becomes <atom:link rel="self">
rel="alternate" must be set last → becomes the main <link>

Wrong order produces <link>https://.../feed_example.xml</link> instead of the blog URL.

Adding a New Feed

Step 1: Analyze the Target Blog

Before writing code, determine which pattern to use:

Open the blog in your browser
Check for pagination:
- URL changes to ?page=2 or /page/2 → Pattern 2 (Pagination)
- No URL change but "Load More" button exists → Pattern 3 (Selenium)
- All posts visible on single page → Pattern 1 (Simple Static)
Check for JavaScript loading:
- Open DevTools → Network tab → Reload
- If posts appear after JS execution (XHR requests) → Pattern 3 (Selenium)
- If posts are in initial HTML → Pattern 1 or 2

Step 2: Download HTML Sample

# For static sites (Pattern 1 or 2)
curl -o sample.html "https://example.com/blog"

# For JS-heavy sites (Pattern 3)
# Use browser: View Page Source won't work
# Instead: DevTools → Elements → Copy outer HTML after page loads

Step 3: Generate the Feed Script

Use Claude Code with the generator prompt:

Use /cmd-rss-feed-generator to convert @sample.html to a RSS feed for https://example.com/blog

Claude will:

Analyze the HTML structure
Choose the appropriate pattern
Generate feed_generators/<source>_blog.py

Step 4: Test Locally

# Install dependencies
make env_setup

# Run the generator
uv run feed_generators/<source>_blog.py

# Verify output
cat feeds/feed_<source>.xml | head -50

# For paginated feeds, test full fetch
uv run feed_generators/<source>_blog.py --full

Verify:

Feed XML is valid (no parsing errors)
<link> points to blog URL, not feed URL
Posts have titles, dates, and links
Dates are in correct order (newest first)

Step 5: Register the Feed

Add to feeds.yaml (the feed registry):

<source>:
  script: <source>_blog.py
  type: requests  # or "selenium" for JS-heavy sites
  blog_url: https://example.com/blog

Add Make target in makefiles/feeds.mk:

.PHONY: feeds_<source>
feeds_<source>: ## Generate RSS feed for <Source Name>
	$(call check_venv)
	$(call print_info,Generating <Source Name> feed)
	$(Q)uv run feed_generators/<source>_blog.py
	$(call print_success,<Source Name> feed generated)

Update README.md table (alphabetical order):

| [Source Name](https://example.com/blog) | [feed_<source>.xml](https://raw.githubusercontent.com/Olshansk/rss-feeds/main/feeds/feed_<source>.xml) |

Step 6: PR Checklist

Before submitting your PR, verify:

make dev_format passes (code formatting)
uv run feed_generators/<source>_blog.py runs without errors
feeds/feed_<source>.xml is generated and valid
Feed registered in feeds.yaml
Make target added to makefiles/feeds.mk
README.md table updated
For paginated/dynamic feeds: cache file created in cache/ on first run
Feed <link> points to original blog (not the XML feed URL)

Deprecating a Feed

When a blog launches an official RSS feed (or we otherwise decide to retire a scraper), follow the two-stage retirement process. Stage 1 is manual and lands in a single PR. Stage 2 is automated.

Stage 1: Inject the notice and tear down the code (manual, one PR)

Inject a sunset notice into the feed XML:
```
uv run feed_generators/deprecate_feed.py \
    --feed=<name> \
    --message="Site X now publishes an official RSS feed." \
    --alternative="https://example.com/feed.xml"
```
This adds a single <item> at the top of feeds/feed_<name>.xml with a stable GUID (so repeated runs are idempotent). Subscribers see the notice in their reader the next time they poll the feed.
Remove everything except the XML, in the same PR:
- Delete feed_generators/<name>_blog.py.
- Remove the <name>: entry from feeds.yaml.
- Remove the feeds_<name> target (and any _full variant) from makefiles/feeds.mk.
- Remove the <name> row from the README table (or update it to point at the official feed only).
- cache/<name>_posts.json is gitignored; nothing to do there.
Leave feeds/feed_<name>.xml in place. It now carries the notice as its newest <item> plus the historical posts. Subscribers can read both.

Stage 2: Automatic deletion (workflow, ~90 days later)

.github/workflows/cleanup_deprecated_feeds.yml runs monthly. It invokes feed_generators/cleanup_deprecated_feeds.py --apply, which scans feeds/feed_*.xml for the deprecation-notice-<name> GUID, parses the notice's <pubDate>, and deletes any XML whose notice is older than 90 days. The deletion is committed to main directly; git history preserves the file for recovery.

To preview what would be removed without touching anything:

uv run feed_generators/cleanup_deprecated_feeds.py

To force-test deletion locally (reversible with git checkout):

uv run feed_generators/cleanup_deprecated_feeds.py --apply --threshold-days=0

Troubleshooting

"No posts found" or empty feed

HTML structure may have changed; re-download sample and update selectors
For Selenium: increase wait times or check if site blocks headless browsers

Feed <link> shows XML URL instead of blog URL

Use setup_feed_links() helper from utils.py
Ensure rel="self" is set before rel="alternate"

Selenium bot detection

undetected-chromedriver should handle most cases
Try increasing wait times between clicks
Some sites may require additional headers or cookies

Cache not updating

Delete cache/<source>_posts.json and run with --full
Check merge_posts() deduplication logic

Date parsing errors

Add the date format to the date_formats list
Use stable_fallback_date() for entries without parseable dates

Empty feed after Selenium run (0 items)

The site is JS-rendered but curl returns a minimal HTML shell — confirm with curl -sL <url> | wc -c (< 10KB = JS-rendered)
Capture Selenium page source to a file and inspect actual selectors: element classes on JS-rendered pages often differ from View Source
Always call deserialize_entries() on cached data before passing to merge_entries() — ISO strings don't sort correctly as datetimes

GitHub Actions

run_feeds.yml - Runs hourly, executes run_all_feeds.py, commits updated XML files
test_feed.yml - Tests feed generation on PRs (runs ollama_blog.py)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

AGENTS.md

Table of Contents

Project Overview

Commands

Architecture

Feed Generator Patterns

1. Simple Static (Default)

2. Pagination + Caching

3. Selenium + Click "Load More"

When to Use Each Pattern

Feed Link Setup (Important)

Adding a New Feed

Step 1: Analyze the Target Blog

Step 2: Download HTML Sample

Step 3: Generate the Feed Script

Step 4: Test Locally

Step 5: Register the Feed

Step 6: PR Checklist

Deprecating a Feed

Stage 1: Inject the notice and tear down the code (manual, one PR)

Stage 2: Automatic deletion (workflow, ~90 days later)

Troubleshooting

GitHub Actions

Uh oh!

FilesExpand file tree

AGENTS.md

Latest commit

History

AGENTS.md

File metadata and controls

AGENTS.md

Table of Contents

Project Overview

Commands

Architecture

Feed Generator Patterns

1. Simple Static (Default)

2. Pagination + Caching

3. Selenium + Click "Load More"

When to Use Each Pattern

Feed Link Setup (Important)

Adding a New Feed

Step 1: Analyze the Target Blog

Step 2: Download HTML Sample

Step 3: Generate the Feed Script

Step 4: Test Locally

Step 5: Register the Feed

Step 6: PR Checklist

Deprecating a Feed

Stage 1: Inject the notice and tear down the code (manual, one PR)

Stage 2: Automatic deletion (workflow, ~90 days later)

Troubleshooting

GitHub Actions