Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion schema/card.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,8 @@
"capacity-cost",
"observability",
"dependency-failure",
"backup-restore"
"backup-restore",
"model-pipeline"
]
},
"scaleAxis": {
Expand Down
1 change: 1 addition & 0 deletions schema/vocabulary.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,4 @@ The design question a record answers.
| `observability` | Detecting and diagnosing problems |
| `dependency-failure` | Behaviour when something the system relies on degrades |
| `backup-restore` | Backups, restores and recovering deleted or corrupted data. Rebooting a site and failing over are `replication-failover` |
| `model-pipeline` | How a model is built and run: model architecture, training and feature pipelines, and the inference path. A cost or sizing question about them is still `capacity-cost` |
8 changes: 7 additions & 1 deletion scripts/fetch-browser.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
One page, for a source a person has chosen to write a card from. Never used to crawl a site.
Usage: uvx --from camoufox python -I scripts/fetch-browser.py <url> <out.txt> (out must be outside the repository)
"""
import pathlib, re, sys
import pathlib, re, sys, time
from urllib.parse import urlparse
from camoufox.sync_api import Camoufox

Expand All @@ -15,6 +15,12 @@
with Camoufox(headless=True, i_know_what_im_doing=True) as browser:
page = browser.new_page()
response = page.goto(url, timeout=60000, wait_until="domcontentloaded")
# Some hosts answer 429 to a second page fetched straight after the first; wait and ask again.
for wait in (20, 60):
if response is None or response.status != 429:
break
time.sleep(wait)
response = page.goto(url, timeout=60000, wait_until="domcontentloaded")
if response is not None and response.status >= 400:
sys.exit(f"HTTP {response.status} for {url}; nothing written")
# A retired article often redirects to a home page; the article's own name must survive the redirect.
Expand Down
Loading