fix: run extensions before page scripts in popups and out-of-process iframes

CHANGELOG.md

@@ -15,6 +15,7 @@
   their command's method and duration, `FERRUM_LOG_FORMAT=json` writes each CDP message as JSON alone
 - Ferrum keeps reading what Chrome writes after startup: it's logged on the `browser` channel, the last 100 lines are in
   `browser.process.output`
+- `browser.add_extension` adds an extension for pages and iframes attached afterwards [#663]
 
 ### Changed
 - The logger cuts any string value over 1024 characters in CDP messages, not just screenshots.
@@ -27,13 +28,22 @@
 - Behind a proxy with `:user` and `:password`, Ferrum stops continuing intercepted requests once you subscribe to
   `on(:request)`, like it already does without a proxy: your callbacks have to continue or abort every request, or it
   stays paused until the page times out [#332]
+- `evaluate_on_new_document` is a `Page` method sending `Page.addScriptToEvaluateOnNewDocument` to that page, with
+  `world_name:` and `run_immediately:`. It returns the script id for `remove_script_on_new_document`. It used to add
+  an extension for every page built afterwards, use `browser.add_extension` for that [#663]
+- Extension files are read in `Browser.new`, a missing one raises `Errno::ENOENT` there instead of on first
+  navigation [#663]
 - Browser details (user agent, versions) come from `Browser.getVersion` over CDP for every browser, local or remote.
   `/json/version` is only asked for the websocket url when `url:` is given, so `ws_url:` works against services
   without an HTTP endpoint and a local launch makes no HTTP call
 - `NoWebSocketUrlError` names the HTTP status `/json/version` answered with, e.g. `401 Unauthorized` for a missing token or
   `429 Too Many Requests` for a full remote browser service, and exposes it as `#status`
 
 ### Fixed
+- An out-of-process iframe kept its parent page's session when `Target.targetCreated` came before
+  `Target.attachedToTarget`, so commands sent through it reached the page [#663]
+- `:extensions` run before the document's own scripts in popups and out-of-process iframes too. Popups used to get
+  them only after loading, iframes not at all, and every page ran them once more on `about:blank` [#663]
 - Every CDP response was scanned for Base64 screenshots to hide from the logger, even with no `:logger` set
 - Headers can be added on a remote browser whose `/json/version` has no `User-Agent`, such as browserless v2, instead of
   failing with `Invalid parameters` [rubycdp/cuprite#300]

docs/13-javascript.md

@@ -41,20 +41,44 @@ Execute expression. Doesn't return the result
 page.execute(%(1 + 1)) # => true
 ```
 
-#### evaluate_on_new_document(expression)
+#### evaluate_on_new_document(source, \*\*options) : `String`
 
-Evaluate JavaScript to modify things before a page load
+Runs JavaScript in every new document of the page and its in-process frames, before the document's own scripts.
+Returns the script identifier. For every page and iframe of the browser, use `add_extension`.
 
-* expression `String` should be valid JavaScript
+* source `String` should be valid JavaScript
+* options `Hash`
+  * :world_name `String` - run in an isolated world with this name instead of the page's main world
+  * :run_immediately `Boolean` - run in the current document as well, `false` by default
 
 ```ruby
-browser.evaluate_on_new_document <<~JS
+id = page.evaluate_on_new_document <<~JS
   Object.defineProperty(navigator, "languages", {
     get: function() { return ["tlh"]; }
   });
 JS
 ```
 
+#### remove_script_on_new_document(identifier) : `Boolean`
+
+Stops running a script added with `evaluate_on_new_document` in new documents.
+
+```ruby
+page.remove_script_on_new_document(id) # => true
+```
+
+#### add_extension(extension)
+
+Adds an extension, see the `:extensions` option. It runs in every page and iframe, popups and out-of-process iframes
+included, attached after the call, before their own scripts. Pages already open keep the extensions they had.
+
+* extension `String | Hash` a path to a JS file or `{ source: "..." }`
+
+```ruby
+browser.add_extension("/path/to/stub.js")
+browser.add_extension(source: "window.secret = 'top'")
+```
+
 #### add_script_tag(\*\*options) : `Boolean`
 
 * options `Hash`

docs/2-customization.md

@@ -21,8 +21,8 @@ Ferrum::Browser.new(options)
     * `:window_size` (Array) - The dimensions of the browser window in which to
       test, expressed as a 2-element array, e.g. [1024, 768]. Default: [1024, 768]
     * `:extensions` (Array[String | Hash]) - An array of paths to files or JS
-      source code to be preloaded into the browser e.g.:
-      `["/path/to/script.js", { source: "window.secret = 'top'" }]`
+      source code to run in every page and iframe before it loads, e.g.:
+      `["/path/to/script.js", { source: "window.secret = 'top'" }]`. Files are read in `Browser.new`.
     * `:logger` (Object responding to `puts`, or a `Logger`) - When present, debug output is
       written to it, see [Logging](#logging).
     * `:slowmo` (Integer | Float) - Set a delay in seconds to wait before sending command.

lib/ferrum/browser.rb

@@ -38,14 +38,15 @@ class Browser
                 frames frame_by main_frame
                 evaluate evaluate_on evaluate_async execute evaluate_func
                 add_script_tag add_style_tag bypass_csp
+                evaluate_on_new_document remove_script_on_new_document
                 on position position=
                 playback_rate playback_rate=
                 disable_javascript set_viewport resize] => :page
 
     attr_reader :client, :process, :contexts, :options
 
     delegate %i[timeout timeout= protocol_timeout protocol_timeout=
-                base_url base_url= default_user_agent default_user_agent= extensions] => :options
+                base_url base_url= default_user_agent default_user_agent= extensions add_extension] => :options
     delegate %i[command] => :client
 
     #
@@ -191,23 +192,6 @@ def create_page(new_context: false, proxy: nil)
       end
     end
 
-    #
-    # Evaluate JavaScript to modify things before a page load.
-    #
-    # @param [String] expression
-    #   The JavaScript to add to each new document.
-    #
-    # @example
-    #   browser.evaluate_on_new_document <<~JS
-    #     Object.defineProperty(navigator, "languages", {
-    #       get: function() { return ["tlh"]; }
-    #     });
-    #   JS
-    #
-    def evaluate_on_new_document(expression)
-      extensions << expression
-    end
-
     #
     # Closes browser tabs opened by the `Browser` instance.
     #

lib/ferrum/browser/options.rb

@@ -25,7 +25,7 @@ class Options
                   :js_errors, :base_url, :slowmo, :pending_connection_errors,
                   :url, :ws_url, :env, :process_timeout, :browser_name, :browser_path,
                   :save_path, :proxy, :port, :host, :headless, :incognito, :dockerize, :browser_options,
-                  :ignore_default_browser_options, :xvfb, :flatten
+                  :ignore_default_browser_options, :xvfb, :flatten, :extensions
       attr_accessor :timeout, :protocol_timeout, :default_user_agent
 
       def initialize(options = nil)
@@ -58,6 +58,7 @@ def initialize(options = nil)
         @base_url = parse_base_url(@options[:base_url]) if @options[:base_url]
         @url = @options[:url].to_s if @options[:url]
         @ws_url = @options[:ws_url].to_s if @options[:ws_url]
+        @extensions = Array(@options[:extensions]).map { |e| read_extension(e) }.freeze
 
         @options = @options.merge(window_size: @window_size).freeze
         @browser_options = @options.fetch(:browser_options, {}).freeze
@@ -76,15 +77,24 @@ def base_url=(value)
       end
 
       #
-      # JS source to preload into the browser, read from `:extensions` option.
+      # Adds an extension to run in every new document, before the document's own scripts. Applies to pages and
+      # iframes attached after the call: pages already open keep the extensions they were attached with.
+      #
+      # @param [String, Hash] extension
+      #   A path to a JS file, or `{ source: "..." }`.
       #
       # @return [Array<String>]
-      #   JS source code for each configured extension.
+      #   JS source code of every extension.
       #
-      def extensions
-        @extensions ||= Array(@options[:extensions]).map do |extension|
-          (extension.is_a?(Hash) && extension[:source]) || File.read(extension)
-        end
+      # @raise [Errno::ENOENT]
+      #   If the file doesn't exist.
+      #
+      # @example
+      #   browser.add_extension("/path/to/stub.js")
+      #   browser.add_extension(source: "window.secret = 'top'")
+      #
+      def add_extension(extension)
+        @extensions = [*@extensions, read_extension(extension)].freeze
       end
 
       #
@@ -122,6 +132,10 @@ def to_h
 
       private
 
+      def read_extension(extension)
+        (extension.is_a?(Hash) && extension[:source]) || File.read(extension)
+      end
+
       def parse_logger(logger)
         value = ENV.fetch("FERRUM_LOG") { "*" if ENV.key?("FERRUM_DEBUG") }
         settings = value ? Log.parse(value) : {}

lib/ferrum/context.rb

@@ -124,13 +124,16 @@ def create_target
     #
     # @param [String, nil] session_id
     #
+    # @param [Boolean] attached
+    #   The session comes from `Target.attachedToTarget`, so it replaces the one a target got from its frame owner.
+    #
     # @return [Target]
-    def add_target(params:, session_id: nil)
+    def add_target(params:, session_id: nil, attached: false)
       new_target = Target.new(@client, session_id, params, implicit: implicit?)
       # `put_if_absent` returns nil if added a new value or existing if there was one already
       target = @targets.put_if_absent(new_target.id, new_target) || new_target
       # on first iteration session_id may be nil, then if session is present here we must set it to the target
-      target.session_id = session_id if session_id && target.session_id.nil?
+      target.session_id = session_id if session_id && (attached || target.session_id.nil?)
       @default_target ||= target
 
       new_pending = Concurrent::IVar.new

lib/ferrum/contexts.rb

@@ -163,7 +163,7 @@ def subscribe_attached_target(client)
 
         context_id = info["browserContextId"]
         add_context(context_id)
-        target = @contexts[context_id]&.add_target(session_id: session_id, params: info)
+        target = @contexts[context_id]&.add_target(session_id: session_id, params: info, attached: true)
 
         rearm_auto_attach(session_id, info["type"])
         handle_attach(target, session_id, params)
@@ -225,11 +225,24 @@ def handle_attach(target, session_id, params)
         detach_unless_manually_attached(target, session_id)
       elsif target.worker? || target.shared_worker?
         connect_worker(target)
-      elsif params["waitingForDebugger"]
-        resume(session_id)
+      else
+        add_extensions_and_resume(session_id, params["waitingForDebugger"])
       end
     end
 
+    # A page or iframe is paused until we resume it, so the extensions we
+    # register now run before any script of its first document. An
+    # out-of-process iframe ignores them until the Page domain is enabled.
+    def add_extensions_and_resume(session_id, waiting)
+      extensions = @client.options.extensions
+      session = @client.session(session_id)
+      session.command("Page.enable", async: true) if extensions.any?
+      extensions.each do |source|
+        session.command("Page.addScriptToEvaluateOnNewDocument", async: true, source: source)
+      end
+      resume(session_id) if waiting
+    end
+
     # Auto-attach pauses every target it attaches to, including the types
     # we don't track: Chrome's own `browser_ui` targets, for instance. We
     # never resume those, so left alone they stay attached and paused for

lib/ferrum/page.rb

@@ -417,6 +417,45 @@ def bypass_csp(enabled: true)
       enabled
     end
 
+    #
+    # Adds JavaScript to run in every new document of this page and its in-process frames, before the document's own
+    # scripts. For every page and iframe of the browser, see {Browser::Options#add_extension}.
+    #
+    # @param [String] source
+    #
+    # @param [String, nil] world_name
+    #   Runs the script in an isolated world with this name instead of the page's main world.
+    #
+    # @param [Boolean] run_immediately
+    #   Runs the script in the current document as well.
+    #
+    # @return [String]
+    #   The script identifier, for {#remove_script_on_new_document}.
+    #
+    # @example
+    #   page.evaluate_on_new_document <<~JS
+    #     Object.defineProperty(navigator, "languages", {
+    #       get: function() { return ["tlh"]; }
+    #     });
+    #   JS
+    #
+    def evaluate_on_new_document(source, world_name: nil, run_immediately: false)
+      params = { source: source, worldName: world_name, runImmediately: run_immediately }.compact
+      command("Page.addScriptToEvaluateOnNewDocument", **params)["identifier"]
+    end
+
+    #
+    # Stops running a script added with {#evaluate_on_new_document} in new documents.
+    #
+    # @param [String] identifier
+    #
+    # @return [true]
+    #
+    def remove_script_on_new_document(identifier)
+      command("Page.removeScriptToEvaluateOnNewDocument", identifier: identifier)
+      true
+    end
+
     #
     # Activates (focuses) the target for the given page.
     # When you have multiple tabs you work with, and you need to switch a given one.
@@ -582,12 +621,6 @@ def prepare_page
 
       downloads.set_behavior(save_path: @options.save_path) if @options.save_path
 
-      @options.extensions.each do |extension|
-        command("Page.addScriptToEvaluateOnNewDocument", source: extension)
-      end
-
-      inject_extensions
-
       response = command("Page.getNavigationHistory")
       transition_type = response.dig("entries", 0, "transitionType")
       return if transition_type == "auto_toplevel"
@@ -600,19 +633,6 @@ def prepare_page
       document_node_id
     end
 
-    def inject_extensions
-      @options.extensions.each do |extension|
-        # https://github.com/GoogleChrome/puppeteer/issues/1443
-        # https://github.com/ChromeDevTools/devtools-protocol/issues/77
-        # https://github.com/cyrus-and/chrome-remote-interface/issues/319
-        # We also evaluate script just in case because
-        # `Page.addScriptToEvaluateOnNewDocument` doesn't work in popups.
-        command("Runtime.evaluate", expression: extension,
-                                    executionContextId: execution_id!,
-                                    returnByValue: true)
-      end
-    end
-
     def history_navigate(delta:)
       history = command("Page.getNavigationHistory")
       index, entries = history.values_at("currentIndex", "entries")

sig/ferrum/browser.rbs

@@ -35,7 +35,7 @@ module Ferrum
 
     def extensions: () -> ::Array[String]
 
-    def evaluate_on_new_document: (String expression) -> void
+    def add_extension: (String | Hash[Symbol, String] extension) -> ::Array[String]
 
     def command: (String method, **untyped params) -> Hash[String, untyped]
 

sig/ferrum/browser/options.rbs

@@ -95,7 +95,7 @@ module Ferrum
       @url: String?
       @ws_url: String?
       @browser_options: Hash[String, (String | bool | nil)]
-      @extensions: ::Array[String]?
+      @extensions: ::Array[String]
       @default_user_agent: String?
 
       def initialize: (?Hash[Symbol, untyped]? options) -> void
@@ -104,12 +104,16 @@ module Ferrum
 
       def extensions: () -> ::Array[String]
 
+      def add_extension: (String | Hash[Symbol, String] extension) -> ::Array[String]
+
       def validate_proxy: (Hash[Symbol, (String | ::Integer)]? options) -> Hash[Symbol, (String | ::Integer)]?
 
       def to_h: () -> Hash[Symbol, untyped]
 
       private
 
+      def read_extension: (String | Hash[Symbol, String] extension) -> String
+
       def parse_logger: (untyped logger) -> [untyped, Log?]
 
       def parse_base_url: (String value) -> ::Addressable::URI

sig/ferrum/context.rbs

@@ -33,7 +33,7 @@ module Ferrum
 
     def create_target: () -> Target
 
-    def add_target: (params: Hash[String, untyped], ?session_id: String?) -> Target
+    def add_target: (params: Hash[String, untyped], ?session_id: String?, ?attached: bool) -> Target
 
     def update_target: (String target_id, Hash[String, untyped] params) -> void
 

sig/ferrum/contexts.rbs

@@ -55,6 +55,8 @@ module Ferrum
 
     def connect_worker: (Target target) -> void
 
+    def add_extensions_and_resume: (String session_id, bool? waiting) -> void
+
     def resume: (String session_id) -> void
 
     def detach: (String session_id) -> void

sig/ferrum/page.rbs

@@ -82,6 +82,10 @@ module Ferrum
 
     def bypass_csp: (?enabled: bool) -> Hash[String, untyped]
 
+    def evaluate_on_new_document: (String source, ?world_name: String?, ?run_immediately: bool) -> String
+
+    def remove_script_on_new_document: (String identifier) -> true
+
     def window_id: () -> ::Integer
 
     def set_window_bounds: (Hash[Symbol, (::Integer | String)] bounds) -> Hash[String, untyped]
@@ -111,8 +115,6 @@ module Ferrum
 
     def prepare_page: () -> void
 
-    def inject_extensions: () -> void
-
     def history_navigate: (::Integer delta) -> Hash[String, untyped]
 
     def combine_url!: (String? url) -> String