feat(jshttp): optional Firefox/WebKit browser engine + ExecutablePath override#21
Open
knoellp wants to merge 2 commits into
Open
feat(jshttp): optional Firefox/WebKit browser engine + ExecutablePath override#21knoellp wants to merge 2 commits into
knoellp wants to merge 2 commits into
Conversation
Adds JSFetcherOptions.BrowserType ("chromium" default, "firefox", "webkit")
and JSFetcherOptions.ExecutablePath to override the browser binary, exposed
through scrapemateapp.WithJSBrowserType(...) and WithJSExecutablePath(...) as
WithJS sub-options.
The browser engine is selected via the matching playwright.BrowserType
(Chromium/Firefox/WebKit) and the corresponding install target. Chromium
launch flags are only applied to Chromium — Firefox and WebKit use the
Playwright engine defaults, since forwarding Chromium flags makes Firefox
hang on the first NewPage call.
Backward-compatible: the empty default maps to Chromium, so existing callers
are unaffected. Adds unit tests for the install mapping, the Chromium-only
arg handling, and the new config sub-options.
gosom
reviewed
Jun 1, 2026
| } | ||
| } | ||
|
|
||
| func newBrowser(pw *playwright.Playwright, headless, disableImages bool, proxyPool *ProxyPool, ua, browserType, executablePath string) (*browser, error) { |
Owner
There was a problem hiding this comment.
If not mistaken when the ua passed is empty then a default will be used.
See line 348:
when a Firefox is returned and ua == '' then
the user agent set will be the one from line 358.
Probably the default UA for firefox/webkit should be used there
| // Playwright engine defaults; forwarding Chromium flags hangs Firefox at | ||
| // the first NewPage. | ||
| if browserType == "" || browserType == "chromium" { | ||
| opts.Args = chromiumLaunchArgs(disableImages) |
Owner
There was a problem hiding this comment.
❓ Is there an option to disable image on firefox/webkit?
If yes maybe it can be added
(optional)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an optional Firefox / WebKit browser engine to the
jshttp(Playwright) fetcher, alongside the existing Chromium default.JSFetcherOptions.BrowserType—"chromium"(default, also for the empty string),"firefox","webkit".JSFetcherOptions.ExecutablePath— optional override for the browser binary.WithJSsub-options:scrapemateapp.WithJSBrowserType(...)andscrapemateapp.WithJSExecutablePath(...).Why
Some targets behave differently across browser engines (bot detection, TLS/JA3 fingerprint, rendering). Being able to opt into Firefox or WebKit — and to point at a custom binary — without leaving the scrapemate fetcher is useful for those cases.
How
playwright.Installinstalls the selected engine; the launch uses the matchingplaywright.BrowserType(pw.Chromium/pw.Firefox/pw.WebKit).NewPagecall.BrowserType/ExecutablePathvalues are threaded through the existing slot/runtime factories (playwrightRuntimeFactory,playwrightSlotFactory,playwrightRuntime), so both the single-page and multi-page pool paths honour them.Compatibility
Fully backward-compatible: the empty default maps to Chromium, so callers that never set
BrowserTypeare unaffected.Tests
Unit tests (no browser required) cover the install-target mapping, the Chromium-only launch-arg handling (incl.
DisableImages), and the new config sub-options.