Skip to content

fix(scripts): report Cloudflare-blocked live-check steps as inconclusive - #2674

Open
RibatTRW wants to merge 2 commits into
lnreader:masterfrom
RibatTRW:fm/lnreader-plugins-livecheck-cf-inconclusive
Open

RibatTRW wants to merge 2 commits into
lnreader:masterfrom
RibatTRW:fm/lnreader-plugins-livecheck-cf-inconclusive

Conversation

@RibatTRW

Copy link
Copy Markdown
Contributor

Intent

Open a small separate upstream PR so the shared Plugin Live Check reports "couldn't test (Cloudflare)" (INCONCLUSIVE) instead of "failed" when page loads are blocked by Cloudflare. Royal Road PR #2665 stays as is with its Cloudflare explanation; once this checker fix merges, its check stops failing, as does every other Cloudflare-blocked source's.

Context: on GitHub's runners Cloudflare blocks every royalroad.com HTML page (home, search, listings, novel pages, chapters), with any request style. The checker marks it FAIL rather than INCONCLUSIVE because its root reachability probe (HEAD of the site root, passes on any status < 400) gets through while the real page loads do not. fetchApi (src/lib/fetch.ts) returns the Response for any status and never throws, so a Cloudflare challenge page (served as 403 or 200) parses to zero novels and the checker scores FAIL "Returned no novels"; the existing 403/Cloudflare classification applies only to thrown errors and to the root probe. Failing run: https://github.com/lnreader/lnreader-plugins/actions/runs/38055684264.

What Changed

  • scripts/live-check-plugin.js now wraps the global fetch during each step to spot Cloudflare challenge responses (cf-mitigated: challenge, or a 403/503 carrying cf-* headers) on the plugin's own page loads. A step that would otherwise FAIL after such a response is reclassified as INCONCLUSIVE with a "Blocked by Cloudflare" detail. Passing steps and failures with no observed challenge are unchanged.
  • The script now exports describeCloudflareChallenge, installChallengeDetector and checkPlugin, and only runs main() when executed directly, so it can be imported by tests.
  • Added scripts/live-check-plugin.test.mjs, which covers the new detection and reclassification logic. Updated the INCONCLUSIVE description in docs/testing.md to mention challenge pages served for a plugin's page loads.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change is confined to the live-check script and only downgrades FAIL to INCONCLUSIVE when a Cloudflare challenge response was seen during that step; its tests run the real checker against a local server.

Testing

The 6 node tests in scripts/live-check-plugin.test.mjs pass. They run the real checkPlugin against local HTTP servers that serve a challenge as 403 or 200, a 403 with Cloudflare headers, an empty result, a plain 403 and a passing page. I also ran the checker as a CLI against a local server that returns a Cloudflare challenge on its list page while the root probe succeeds. The report showed INCONCLUSIVE ("Blocked by Cloudflare…"), "no hard failures" and exit code 0. I did not test against the real royalroad.com. The worktree stayed clean.

  • Live validation: ✅ go - 6 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Challenge page (403 with cf-mitigated) on a list load gives INCONCLUSIVE ✅ pass live node --test output and CLI report
Challenge served as HTTP 200 gives INCONCLUSIVE ✅ pass live node --test
403 with Cloudflare headers gives INCONCLUSIVE ✅ pass live node --test
Adversarial: an empty result with a plain cf-ray header, or a plain 403, still FAILs ✅ pass live node --test
Adversarial: a challenged response on a step that passed stays PASS ✅ pass live node --test
CLI entry still runs and exits 0 on an inconclusive result ✅ pass live CLI transcript
Evidence: CLI report for a challenged plugin
| `popularNovels` | ⚠️ INCONCLUSIVE | Blocked by Cloudflare: HTTP 403 (Cloudflare challenge) (was: Returned no novels) |
**Result: ⚠️ Inconclusive — no hard failures.**

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 6 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Challenge page (403 with cf-mitigated) on a list load gives INCONCLUSIVE ✅ pass live node --test output and CLI report
Challenge served as HTTP 200 gives INCONCLUSIVE ✅ pass live node --test
403 with Cloudflare headers gives INCONCLUSIVE ✅ pass live node --test
Adversarial: an empty result with a plain cf-ray header, or a plain 403, still FAILs ✅ pass live node --test
Adversarial: a challenged response on a step that passed stays PASS ✅ pass live node --test
CLI entry still runs and exits 0 on an inconclusive result ✅ pass live CLI transcript
  • npm ci --ignore-scripts
  • node --test scripts/live-check-plugin.test.mjs
  • node scripts/live-check-plugin.js &lt;plugin&gt; against a local server that returns a Cloudflare challenge on its list page
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@greptile-apps

greptile-apps Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] Changes how the live-check test script reports Cloudflare blocks.

Fix response-to-step tracking before merging so a timed-out search cannot hide a real plugin failure.

Findings

  1. P1 Late responses hide failures ▶
  2. P2 New tests stay manual ▶

Summary

The checker now observes Cloudflare responses during plugin requests and changes affected failed steps to INCONCLUSIVE.

  • Plugin checks report blocked page loads as inconclusive.

Reviews (1) · Last reviewed commit: "no-mistakes(document): Document Cloudfla..." · Reviewed by Greptile

Comment on lines +52 to +53
const res = await originalFetch(...args);
const step = currentStep;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Late responses hide failures

installChallengeDetector reads currentStep after the request finishes. If searchNovels times out, its request keeps running while the checker starts parseNovel. A late Cloudflare response from that search then marks parseNovel as blocked, hiding a real failure there as INCONCLUSIVE.

Capture the step before awaiting the request, and add a test for a late response crossing steps.

Suggested change
const res = await originalFetch(...args);
const step = currentStep;
const step = currentStep;
const res = await originalFetch(...args);

@@ -0,0 +1,97 @@
// Run with: node --test scripts/live-check-plugin.test.mjs

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 New tests stay manual

No package script or workflow runs scripts/live-check-plugin.test.mjs. The live-check workflow also skips execution when only checker files change, so these six tests cannot catch regressions automatically.

Add a CI step that runs these local-server tests when the checker or its tests change.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant