diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f46728d..ef12d3b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ their command's method and duration, `FERRUM_LOG_FORMAT=json` writes each CDP message as JSON alone - Ferrum keeps reading what Chrome writes after startup: it's logged on the `browser` channel, the last 100 lines are in `browser.process.output` +- `browser.add_extension` adds an extension for pages and iframes attached afterwards [#663] ### Changed - The logger cuts any string value over 1024 characters in CDP messages, not just screenshots. @@ -27,6 +28,11 @@ - Behind a proxy with `:user` and `:password`, Ferrum stops continuing intercepted requests once you subscribe to `on(:request)`, like it already does without a proxy: your callbacks have to continue or abort every request, or it stays paused until the page times out [#332] +- `evaluate_on_new_document` is a `Page` method sending `Page.addScriptToEvaluateOnNewDocument` to that page, with + `world_name:` and `run_immediately:`. It returns the script id for `remove_script_on_new_document`. It used to add + an extension for every page built afterwards, use `browser.add_extension` for that [#663] +- Extension files are read in `Browser.new`, a missing one raises `Errno::ENOENT` there instead of on first + navigation [#663] - Browser details (user agent, versions) come from `Browser.getVersion` over CDP for every browser, local or remote. `/json/version` is only asked for the websocket url when `url:` is given, so `ws_url:` works against services without an HTTP endpoint and a local launch makes no HTTP call @@ -34,6 +40,10 @@ `429 Too Many Requests` for a full remote browser service, and exposes it as `#status` ### Fixed +- An out-of-process iframe kept its parent page's session when `Target.targetCreated` came before + `Target.attachedToTarget`, so commands sent through it reached the page [#663] +- `:extensions` run before the document's own scripts in popups and out-of-process iframes too. Popups used to get + them only after loading, iframes not at all, and every page ran them once more on `about:blank` [#663] - Every CDP response was scanned for Base64 screenshots to hide from the logger, even with no `:logger` set - Headers can be added on a remote browser whose `/json/version` has no `User-Agent`, such as browserless v2, instead of failing with `Invalid parameters` [rubycdp/cuprite#300] diff --git a/docs/13-javascript.md b/docs/13-javascript.md index 6dff564a..6f955711 100644 --- a/docs/13-javascript.md +++ b/docs/13-javascript.md @@ -41,20 +41,44 @@ Execute expression. Doesn't return the result page.execute(%(1 + 1)) # => true ``` -#### evaluate_on_new_document(expression) +#### evaluate_on_new_document(source, \*\*options) : `String` -Evaluate JavaScript to modify things before a page load +Runs JavaScript in every new document of the page and its in-process frames, before the document's own scripts. +Returns the script identifier. For every page and iframe of the browser, use `add_extension`. -* expression `String` should be valid JavaScript +* source `String` should be valid JavaScript +* options `Hash` + * :world_name `String` - run in an isolated world with this name instead of the page's main world + * :run_immediately `Boolean` - run in the current document as well, `false` by default ```ruby -browser.evaluate_on_new_document <<~JS +id = page.evaluate_on_new_document <<~JS Object.defineProperty(navigator, "languages", { get: function() { return ["tlh"]; } }); JS ``` +#### remove_script_on_new_document(identifier) : `Boolean` + +Stops running a script added with `evaluate_on_new_document` in new documents. + +```ruby +page.remove_script_on_new_document(id) # => true +``` + +#### add_extension(extension) + +Adds an extension, see the `:extensions` option. It runs in every page and iframe, popups and out-of-process iframes +included, attached after the call, before their own scripts. Pages already open keep the extensions they had. + +* extension `String | Hash` a path to a JS file or `{ source: "..." }` + +```ruby +browser.add_extension("/path/to/stub.js") +browser.add_extension(source: "window.secret = 'top'") +``` + #### add_script_tag(\*\*options) : `Boolean` * options `Hash` diff --git a/docs/2-customization.md b/docs/2-customization.md index 290fe5ab..e21b5e07 100644 --- a/docs/2-customization.md +++ b/docs/2-customization.md @@ -21,8 +21,8 @@ Ferrum::Browser.new(options) * `:window_size` (Array) - The dimensions of the browser window in which to test, expressed as a 2-element array, e.g. [1024, 768]. Default: [1024, 768] * `:extensions` (Array[String | Hash]) - An array of paths to files or JS - source code to be preloaded into the browser e.g.: - `["/path/to/script.js", { source: "window.secret = 'top'" }]` + source code to run in every page and iframe before it loads, e.g.: + `["/path/to/script.js", { source: "window.secret = 'top'" }]`. Files are read in `Browser.new`. * `:logger` (Object responding to `puts`, or a `Logger`) - When present, debug output is written to it, see [Logging](#logging). * `:slowmo` (Integer | Float) - Set a delay in seconds to wait before sending command. diff --git a/lib/ferrum/browser.rb b/lib/ferrum/browser.rb index 51886709..2003cf69 100644 --- a/lib/ferrum/browser.rb +++ b/lib/ferrum/browser.rb @@ -38,6 +38,7 @@ class Browser frames frame_by main_frame evaluate evaluate_on evaluate_async execute evaluate_func add_script_tag add_style_tag bypass_csp + evaluate_on_new_document remove_script_on_new_document on position position= playback_rate playback_rate= disable_javascript set_viewport resize] => :page @@ -45,7 +46,7 @@ class Browser attr_reader :client, :process, :contexts, :options delegate %i[timeout timeout= protocol_timeout protocol_timeout= - base_url base_url= default_user_agent default_user_agent= extensions] => :options + base_url base_url= default_user_agent default_user_agent= extensions add_extension] => :options delegate %i[command] => :client # @@ -191,23 +192,6 @@ def create_page(new_context: false, proxy: nil) end end - # - # Evaluate JavaScript to modify things before a page load. - # - # @param [String] expression - # The JavaScript to add to each new document. - # - # @example - # browser.evaluate_on_new_document <<~JS - # Object.defineProperty(navigator, "languages", { - # get: function() { return ["tlh"]; } - # }); - # JS - # - def evaluate_on_new_document(expression) - extensions << expression - end - # # Closes browser tabs opened by the `Browser` instance. # diff --git a/lib/ferrum/browser/options.rb b/lib/ferrum/browser/options.rb index 8bf17188..3a1c0fff 100644 --- a/lib/ferrum/browser/options.rb +++ b/lib/ferrum/browser/options.rb @@ -25,7 +25,7 @@ class Options :js_errors, :base_url, :slowmo, :pending_connection_errors, :url, :ws_url, :env, :process_timeout, :browser_name, :browser_path, :save_path, :proxy, :port, :host, :headless, :incognito, :dockerize, :browser_options, - :ignore_default_browser_options, :xvfb, :flatten + :ignore_default_browser_options, :xvfb, :flatten, :extensions attr_accessor :timeout, :protocol_timeout, :default_user_agent def initialize(options = nil) @@ -58,6 +58,7 @@ def initialize(options = nil) @base_url = parse_base_url(@options[:base_url]) if @options[:base_url] @url = @options[:url].to_s if @options[:url] @ws_url = @options[:ws_url].to_s if @options[:ws_url] + @extensions = Array(@options[:extensions]).map { |e| read_extension(e) }.freeze @options = @options.merge(window_size: @window_size).freeze @browser_options = @options.fetch(:browser_options, {}).freeze @@ -76,15 +77,24 @@ def base_url=(value) end # - # JS source to preload into the browser, read from `:extensions` option. + # Adds an extension to run in every new document, before the document's own scripts. Applies to pages and + # iframes attached after the call: pages already open keep the extensions they were attached with. + # + # @param [String, Hash] extension + # A path to a JS file, or `{ source: "..." }`. # # @return [Array] - # JS source code for each configured extension. + # JS source code of every extension. # - def extensions - @extensions ||= Array(@options[:extensions]).map do |extension| - (extension.is_a?(Hash) && extension[:source]) || File.read(extension) - end + # @raise [Errno::ENOENT] + # If the file doesn't exist. + # + # @example + # browser.add_extension("/path/to/stub.js") + # browser.add_extension(source: "window.secret = 'top'") + # + def add_extension(extension) + @extensions = [*@extensions, read_extension(extension)].freeze end # @@ -122,6 +132,10 @@ def to_h private + def read_extension(extension) + (extension.is_a?(Hash) && extension[:source]) || File.read(extension) + end + def parse_logger(logger) value = ENV.fetch("FERRUM_LOG") { "*" if ENV.key?("FERRUM_DEBUG") } settings = value ? Log.parse(value) : {} diff --git a/lib/ferrum/context.rb b/lib/ferrum/context.rb index 0f7c5123..aab93452 100644 --- a/lib/ferrum/context.rb +++ b/lib/ferrum/context.rb @@ -124,13 +124,16 @@ def create_target # # @param [String, nil] session_id # + # @param [Boolean] attached + # The session comes from `Target.attachedToTarget`, so it replaces the one a target got from its frame owner. + # # @return [Target] - def add_target(params:, session_id: nil) + def add_target(params:, session_id: nil, attached: false) new_target = Target.new(@client, session_id, params, implicit: implicit?) # `put_if_absent` returns nil if added a new value or existing if there was one already target = @targets.put_if_absent(new_target.id, new_target) || new_target # on first iteration session_id may be nil, then if session is present here we must set it to the target - target.session_id = session_id if session_id && target.session_id.nil? + target.session_id = session_id if session_id && (attached || target.session_id.nil?) @default_target ||= target new_pending = Concurrent::IVar.new diff --git a/lib/ferrum/contexts.rb b/lib/ferrum/contexts.rb index 45cc7f73..33138fb7 100644 --- a/lib/ferrum/contexts.rb +++ b/lib/ferrum/contexts.rb @@ -163,7 +163,7 @@ def subscribe_attached_target(client) context_id = info["browserContextId"] add_context(context_id) - target = @contexts[context_id]&.add_target(session_id: session_id, params: info) + target = @contexts[context_id]&.add_target(session_id: session_id, params: info, attached: true) rearm_auto_attach(session_id, info["type"]) handle_attach(target, session_id, params) @@ -225,11 +225,24 @@ def handle_attach(target, session_id, params) detach_unless_manually_attached(target, session_id) elsif target.worker? || target.shared_worker? connect_worker(target) - elsif params["waitingForDebugger"] - resume(session_id) + else + add_extensions_and_resume(session_id, params["waitingForDebugger"]) end end + # A page or iframe is paused until we resume it, so the extensions we + # register now run before any script of its first document. An + # out-of-process iframe ignores them until the Page domain is enabled. + def add_extensions_and_resume(session_id, waiting) + extensions = @client.options.extensions + session = @client.session(session_id) + session.command("Page.enable", async: true) if extensions.any? + extensions.each do |source| + session.command("Page.addScriptToEvaluateOnNewDocument", async: true, source: source) + end + resume(session_id) if waiting + end + # Auto-attach pauses every target it attaches to, including the types # we don't track: Chrome's own `browser_ui` targets, for instance. We # never resume those, so left alone they stay attached and paused for diff --git a/lib/ferrum/page.rb b/lib/ferrum/page.rb index e52a20bb..cad99476 100644 --- a/lib/ferrum/page.rb +++ b/lib/ferrum/page.rb @@ -417,6 +417,45 @@ def bypass_csp(enabled: true) enabled end + # + # Adds JavaScript to run in every new document of this page and its in-process frames, before the document's own + # scripts. For every page and iframe of the browser, see {Browser::Options#add_extension}. + # + # @param [String] source + # + # @param [String, nil] world_name + # Runs the script in an isolated world with this name instead of the page's main world. + # + # @param [Boolean] run_immediately + # Runs the script in the current document as well. + # + # @return [String] + # The script identifier, for {#remove_script_on_new_document}. + # + # @example + # page.evaluate_on_new_document <<~JS + # Object.defineProperty(navigator, "languages", { + # get: function() { return ["tlh"]; } + # }); + # JS + # + def evaluate_on_new_document(source, world_name: nil, run_immediately: false) + params = { source: source, worldName: world_name, runImmediately: run_immediately }.compact + command("Page.addScriptToEvaluateOnNewDocument", **params)["identifier"] + end + + # + # Stops running a script added with {#evaluate_on_new_document} in new documents. + # + # @param [String] identifier + # + # @return [true] + # + def remove_script_on_new_document(identifier) + command("Page.removeScriptToEvaluateOnNewDocument", identifier: identifier) + true + end + # # Activates (focuses) the target for the given page. # When you have multiple tabs you work with, and you need to switch a given one. @@ -582,12 +621,6 @@ def prepare_page downloads.set_behavior(save_path: @options.save_path) if @options.save_path - @options.extensions.each do |extension| - command("Page.addScriptToEvaluateOnNewDocument", source: extension) - end - - inject_extensions - response = command("Page.getNavigationHistory") transition_type = response.dig("entries", 0, "transitionType") return if transition_type == "auto_toplevel" @@ -600,19 +633,6 @@ def prepare_page document_node_id end - def inject_extensions - @options.extensions.each do |extension| - # https://github.com/GoogleChrome/puppeteer/issues/1443 - # https://github.com/ChromeDevTools/devtools-protocol/issues/77 - # https://github.com/cyrus-and/chrome-remote-interface/issues/319 - # We also evaluate script just in case because - # `Page.addScriptToEvaluateOnNewDocument` doesn't work in popups. - command("Runtime.evaluate", expression: extension, - executionContextId: execution_id!, - returnByValue: true) - end - end - def history_navigate(delta:) history = command("Page.getNavigationHistory") index, entries = history.values_at("currentIndex", "entries") diff --git a/sig/ferrum/browser.rbs b/sig/ferrum/browser.rbs index 411b42aa..d2b40bef 100644 --- a/sig/ferrum/browser.rbs +++ b/sig/ferrum/browser.rbs @@ -35,7 +35,7 @@ module Ferrum def extensions: () -> ::Array[String] - def evaluate_on_new_document: (String expression) -> void + def add_extension: (String | Hash[Symbol, String] extension) -> ::Array[String] def command: (String method, **untyped params) -> Hash[String, untyped] diff --git a/sig/ferrum/browser/options.rbs b/sig/ferrum/browser/options.rbs index 240aabb8..1775c2f7 100644 --- a/sig/ferrum/browser/options.rbs +++ b/sig/ferrum/browser/options.rbs @@ -95,7 +95,7 @@ module Ferrum @url: String? @ws_url: String? @browser_options: Hash[String, (String | bool | nil)] - @extensions: ::Array[String]? + @extensions: ::Array[String] @default_user_agent: String? def initialize: (?Hash[Symbol, untyped]? options) -> void @@ -104,12 +104,16 @@ module Ferrum def extensions: () -> ::Array[String] + def add_extension: (String | Hash[Symbol, String] extension) -> ::Array[String] + def validate_proxy: (Hash[Symbol, (String | ::Integer)]? options) -> Hash[Symbol, (String | ::Integer)]? def to_h: () -> Hash[Symbol, untyped] private + def read_extension: (String | Hash[Symbol, String] extension) -> String + def parse_logger: (untyped logger) -> [untyped, Log?] def parse_base_url: (String value) -> ::Addressable::URI diff --git a/sig/ferrum/context.rbs b/sig/ferrum/context.rbs index d0efb0aa..9b75c07c 100644 --- a/sig/ferrum/context.rbs +++ b/sig/ferrum/context.rbs @@ -33,7 +33,7 @@ module Ferrum def create_target: () -> Target - def add_target: (params: Hash[String, untyped], ?session_id: String?) -> Target + def add_target: (params: Hash[String, untyped], ?session_id: String?, ?attached: bool) -> Target def update_target: (String target_id, Hash[String, untyped] params) -> void diff --git a/sig/ferrum/contexts.rbs b/sig/ferrum/contexts.rbs index c575b1a1..3da8c009 100644 --- a/sig/ferrum/contexts.rbs +++ b/sig/ferrum/contexts.rbs @@ -55,6 +55,8 @@ module Ferrum def connect_worker: (Target target) -> void + def add_extensions_and_resume: (String session_id, bool? waiting) -> void + def resume: (String session_id) -> void def detach: (String session_id) -> void diff --git a/sig/ferrum/page.rbs b/sig/ferrum/page.rbs index 927c2f4f..643b0ace 100644 --- a/sig/ferrum/page.rbs +++ b/sig/ferrum/page.rbs @@ -82,6 +82,10 @@ module Ferrum def bypass_csp: (?enabled: bool) -> Hash[String, untyped] + def evaluate_on_new_document: (String source, ?world_name: String?, ?run_immediately: bool) -> String + + def remove_script_on_new_document: (String identifier) -> true + def window_id: () -> ::Integer def set_window_bounds: (Hash[Symbol, (::Integer | String)] bounds) -> Hash[String, untyped] @@ -111,8 +115,6 @@ module Ferrum def prepare_page: () -> void - def inject_extensions: () -> void - def history_navigate: (::Integer delta) -> Hash[String, untyped] def combine_url!: (String? url) -> String diff --git a/spec/browser_spec.rb b/spec/browser_spec.rb index a0487437..8f4c046a 100644 --- a/spec/browser_spec.rb +++ b/spec/browser_spec.rb @@ -334,9 +334,70 @@ def broken.puts(*) = raise(IOError, "closed stream") browser&.quit end + it "extends a popup's world before its scripts run", :launch do + browser = Ferrum::Browser.new(base_url: base_url, + extensions: [File.expand_path("support/geolocation.js", __dir__)]) + browser.go_to + + browser.execute(%(window.open("/requiring_custom_extension", "popup"))) + popup, = browser.windows(:last) + + expect(popup.at_css("#location").text).to eq("1,-1") + ensure + browser&.quit + end + + it "extends an out-of-process iframe's world before its scripts run", :launch do + report = <<~JS + if (window !== top) { + addEventListener("load", () => parent.postMessage(document.getElementById("location").textContent, "*")); + } + JS + browser = Ferrum::Browser.new(base_url: base_url, + browser_options: { "site-per-process" => nil }, + extensions: [File.expand_path("support/geolocation.js", __dir__), { source: report }]) + browser.go_to + browser.execute(<<~JS) + addEventListener("message", (event) => { window.iframeLocation = event.data }); + document.body.innerHTML = `