Crawler: one schedule table for which languages run, when, and how much

[mircealungu]

Why

Bulgarian took three fixes to go live, because the list of crawled languages was defined in several places that each rejected bg in their own way:

• the ops crontab
• LANG_CONFIG_* / DEFAULT_LANGUAGES in crawl_all_in_parallel.sh (silently skipped bg)
• ALL_LANGUAGES in crawl.py (exited with "Invalid language codes: ['bg']")
• Language.CODES_OF_LANGUAGES_BEING_CRAWLED

What

• New zeeguu/core/crawl_schedule.py: CRAWL_SCHEDULE lists every crawled language with its crawl hours, max_articles and max_minutes. It uses only the standard library, so the shell script runs it with the host's python3 and it reads the same clock cron uses.
• Everything else derives from it:
  • Language.CODES_OF_LANGUAGES_BEING_CRAWLED = list(CRAWL_SCHEDULE)
  • crawl.py's valid-language check uses that list.
  • crawl_all_in_parallel.sh asks the schedule which languages are due this hour (or looks up the limits for languages passed explicitly). The LANG_CONFIG block is gone.
• Schedule changes:
  • The three-times-a-day slot moves from 10:06/14:50/20:50 to the :37 run at 9/14/20h, after the hourly da/fr/de. 9 rather than 10 so the morning crawl still finishes (around 10:15) before the 11:00 feed_delivery_health check.
  • sv is PAUSED: still offered in the app, never due. That's the same as today.
• Deleted stale copies that had their own language lists: tools/crawler/crawl.py, tools/crawler/crawl_roundrobin.py, two_anthropic_crawl.sh, crawl_ab_test.sh.

Adding a language is now one line in CRAWL_SCHEDULE, plus its feeds.

Deploy order

1. Merge and deploy this PR first. The old crontab still works with it, because explicit language arguments are still accepted.
2. Then merge the ops PR and run cd ~/ops/cron && ./refresh. Doing it the other way round would crawl pt sv ro nl en el es it every hour until this is deployed.

Testing

• Added: zeeguu/core/test/test_crawl_schedule.py. I couldn't run it locally (the local venv is empty and there's no Docker), so CI is its first run.
• Run by hand: the schedule CLI for hours 3/9/10/14/20, explicit languages, and an unknown language (exits 1 with nothing on stdout).
• Run by hand: crawl_all_in_parallel.sh end to end with stub docker/timeout. At 9h it launches the same 11 crawl.py commands, with the same --max-articles/--max-time, that the old crontab and LANG_CONFIG produced (plus bg). The </dev/null guard is needed: without it, a docker compose run that reads stdin swallows the rest of the plan and only the first language runs.

🤖 Generated with Claude Code

[github-actions]

ArchLens - No architecturally relevant changes to the existing views