Skip to content

Add osu API backoff for beatmap requests#40

Open
cmyui wants to merge 9 commits into
masterfrom
fix/osu-api-backoff-beatmaps
Open

Add osu API backoff for beatmap requests#40
cmyui wants to merge 9 commits into
masterfrom
fix/osu-api-backoff-beatmaps

Conversation

@cmyui

@cmyui cmyui commented Jun 9, 2026

Copy link
Copy Markdown
Member

Summary

  • add a shared osu! API circuit breaker with fail-fast open state and one canary retry after cooldown
  • cap official osu! API egress from beatmaps-service to 600 requests/minute with burst 10, shared across v1, v2, and v2 OAuth token refreshes
  • lower generic upstream failure cooldown to 10 seconds; explicit 403/429 rate-limit responses still use Retry-After or the 60 second default probe interval
  • wire v1/v2 beatmap clients and v2 OAuth token refresh into the backoff path
  • validate fresh .osu bodies before caching so a 200 HTML/error page cannot be saved to S3
  • serve stale cached beatmap metadata/.osu data where possible while upstream osu! is unavailable
  • route Cheesegull mirror metadata fallback only through mirrors that support that metadata, and fail fast on intentional osu! API backoff instead of shifting burst traffic to osu.direct
  • stabilize the circuit under concurrent in-flight successes/failures so stale responses cannot close or extend an already-open circuit
  • temporarily allow production deploys from this feature branch for incident mitigation

Notes

  • osu!'s current public docs ask API clients to stay at no more than 60 requests/minute, while older/internal limits have historically been much higher. The 600/min cap is an operational compromise for Akatsuki's existing production traffic, not a final answer for long-term upstream dependency reduction.
  • the local cap is per beatmaps-service process. Current production runs a single uvicorn process; if this service is scaled horizontally, the limiter should move to shared state such as Redis.
  • success paths are mostly silent in current logging, so historical logs cannot directly count all successful outbound osu! API requests.

Production log check

  • baseline June 4-7 had short BAU spikes, including v2 429s, but June 8 became sustained incident-level pressure: roughly 144k 429 Too Many Requests log occurrences before the first fix, with a peak hourly bucket around 9.8k
  • deployed 5edb61a to production from fix/osu-api-backoff-beatmaps
  • first fresh window after the 5edb61a deploy: 26 .osu files saved to S3, zero invalid .osu body cache skips, zero v1 .osu fetch failures, and zero circuit opens
  • sampled recent saved .osu objects from S3; all started with osu file format v14, had [General] and [Metadata] sections, were not HTML, and MD5-matched beatmaps.beatmap_md5
  • earlier post-backoff window had zero old ValueError: osu api is down, zero ASGI exception storm signatures, zero Cheesegull NotImplementedError/fallback spam, and local limiter skips mostly under 100ms; this suggests burst smoothing under the 600/min cap rather than long outage-style starvation

Tests

  • python3 -m unittest discover tests
  • python3 -m py_compile app/usecases/osu_files.py tests/test_osu_files.py app/adapters/osu_api_v1.py
  • python3 -m py_compile app/adapters/osu_api_backoff.py app/adapters/osu_api_v1.py app/adapters/osu_api_v2/api.py app/oauth.py tests/test_osu_api_backoff.py
  • python3 -m py_compile app/usecases/cheesegull_beatmaps.py tests/test_cheesegull_beatmaps.py app/adapters/osu_mirrors/__init__.py app/adapters/osu_mirrors/backends/osu_direct.py app/repositories/beatmap_mirror_requests.py tests/test_osu_mirrors.py
  • python3 -m mypy app/adapters/osu_api_backoff.py app/adapters/osu_api_v1.py app/oauth.py app/adapters/osu_api_v2/api.py

@cmyui cmyui requested a review from infernalfire72 as a code owner June 9, 2026 01:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant