feat: retry 429 with backoff + document pagination date-range loss#178
Open
ranahaani wants to merge 2 commits into
Open
feat: retry 429 with backoff + document pagination date-range loss#178ranahaani wants to merge 2 commits into
ranahaani wants to merge 2 commits into
Conversation
…ation date-range loss - _get_news now retries HTTP 429 responses with capped exponential backoff and uniform jitter (max_retries, retry_backoff_base, retry_backoff_max constructor params). Previously a single 429 bubbled straight to the caller, which forced every downstream consumer to reinvent retry policy. - _fetch_feed and _sleep extracted as seams so retry behaviour can be unit tested without real HTTP or wall-clock waits. - _get_news_more_than_100 docstring + UserWarning now spell out that date ranges are discarded when paginating past 100 results, that the rolling window steps in 7-day chunks anchored on the earliest parsed published_date, and that per-call seen_urls dedup does not persist across calls. Helps callers building precise temporal pipelines pick the right max_results. - README updated with the new retry parameters. - tests/test_retry_backoff.py covers config validation, backoff math, retry exhaustion, retry disabled, and non-429 short-circuit (8 tests, all pass).
…changelog - New docs/usage/retries.md walks through default behaviour, tuning, disabling, the backoff formula, and what is intentionally not retried. Linked from docs/index.rst so it appears in the ReadTheDocs sidebar. - docs/reference/api.md GNews constructor signature now lists the three new retry parameters with defaults, plus a pointer to the retries guide. - setup.py version bumped to 0.8.2 (additive feature, no API removal). - docs/conf.py release synced to 0.8.2 (was stuck at 0.6.0). - docs/changelog.md prepended with a 0.8.2 entry covering both the retry feature and the pagination warning/docstring tightening.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_get_newsretries HTTP 429 with capped exponential backoff + uniform jitter. Configurable viamax_retries,retry_backoff_base,retry_backoff_maxon theGNewsconstructor. Previously a single 429 bubbled straight to the caller, forcing every downstream consumer to reinvent retry policy._get_news_more_than_100docstring +UserWarningnow spell out that date ranges are discarded when paginating past 100 results, that the rolling window steps in 7-day chunks anchored on the earliest parsedpublished_date, and that per-callseen_urlsdedup does not persist across calls. Lets callers building precise temporal pipelines pick the rightmax_results._fetch_feedand_sleepextracted as seams so retry behaviour can be unit tested without real HTTP or wall-clock waits.Why
Came up while designing
gnews-agent(persistent news memory + MCP layer built on top of GNews). The agent fetcher needed retry+backoff and precise temporal queries — both gaps belonged in the library, not every downstream user reinventing them.Compatibility
max_retries=3,retry_backoff_base=1.0,retry_backoff_max=60.0). Existing callers see a more resilient fetcher with no API changes required.max_retries=0restores the previous immediate-raise behaviour.Test plan
pytest tests/test_retry_backoff.py— 8/8 passpytest tests/test_exceptions.py tests/test_logging.py— 5/5 pass (no regression on offline tests)test_gnews.pysuite as part of CI