opal 1.6.1 → 1.7.0.rc1

data/docs/cdp_common.md

@@ -0,0 +1,18 @@
+### Common CDP of Chrome, Firefox, Node and Deno
+
+cdp_common.json documents a subset of the common CDP as reported by Chrome, version 107 and Firefox, version 106.
+All entries that where marked as "experimental" or "deprecated" have been removed.
+Only entries, that where included in both browsers with equal state have been kept.
+
+The CDP as implemented by Chrome or Firefox can be retrieved directly from the browsers by starting them with the `--remote-debugging-port` option
+and visiting the (http://localhost:9222/json/protocol)[http://localhost:9222/json/protocol] endpoint.
+
+Unfortunately, Firefox advertising this protocol support doesn't mean it actually is available or even works!
+Various domains and methods, advertised as available from Firefox, present a "UnknownMethodError" or other errors.
+
+The Node and Deno CDP protocol can be inspected by starting them with the `--inspect` option and visiting the
+(http://localhost:9229/json/protocol)[http://localhost:9229/json/protocol] endpoint.
+
+The actual websocket endpoint uri can be retrieved at (http://localhost:9222/json/version)[http://localhost:9222/json/version]
+or as list of targets at (http://localhost:9222/json/list)[http://localhost:9222/json/list],
+likewise in Node or Deno with the adjusted port number.

data/docs/{headless_chrome.md → headless_browsers.md}

@@ -1,16 +1,21 @@
-# Running code in a Headless Chrome
+# Running code in a Headless Browsers
 
 ## Requirements
 
-First of all, make sure that you have Chrome at least 59.0 installed.
+First of all, make sure that you have Chrome, at least version 59.0, installed or Firefox, at least version 106.
 
-## Using the runner
+## Using the runners
 
-To run your code using headless chrome, use `-R chrome` (`--runner chrome`) option:
+To run your code using headless Chrome, use `-R chrome` (`--runner chrome`) option:
 
     $ opal -Rchrome -e "puts 'Hello, Opal'"
     Hello, Opal
 
+To run your code using headless Firefox, use `-R firefox` (`--runner firefox`) option:
+
+    $ opal -Rfirefox -e "puts 'Hello, Opal'"
+    Hello, Opal
+
 The runner also listens for any exceptions and prints formatted stracktraces back to your console:
 
     $ opal -Rchrome -e "
@@ -27,7 +32,7 @@ The runner also listens for any exceptions and prints formatted stracktraces bac
 
 ## Using exit codes
 
-By default headless chrome runner explicitly sets exit code to 1 when there was any error in the code.
+By default headless browser runner explicitly sets exit code to 1 when there was any error in the code.
 
     $ opal -Rchrome -e "42"; echo $?
     0
@@ -46,23 +51,21 @@ You can change final exit code by using `Kernel#exit`, but make sure to require
     $ opal -Rchrome -ropal/platform -e "exit(1)"; echo $?
     1
 
-Also `Kernel#exit` doesn't abort your script. It simply takes the value that was passed to the first
-invocation and persists it in `window.OPAL_EXIT_CODE`. Later headless chrome runner extracts it from the chrome runtime.
-
 ## Known limitations
 
-1. `exit` doesn't abort you script (but `raise/throw` does)
-2. When you call `console.log(one, two, three)` from your code headless chrome prints only the first passed object.
+1. When you call `console.log(one, two, three)` from your code headless chrome prints only the first passed object.
    The reason behind it is the format of the message that chrome sends to the runner.
    Opal intentionally uses a simplified method from Chrome API (`Console.messageAdded`) to catch `console.log` invocations.
    (Check `lib/opal/cli_runners/chrome.js` do get more information)
 
 ## Internals
 
+### Chrome
+
 Under the hood when you call `opal -Rchrome -e 'your code'` Opal uses chrome runner that is defined in
 `lib/opal/cli_runners/chrome.rb`. This runner tries to connect to `localhost:9222` (9222 is a default port for a headless chrome server)
-or runs the server on its own. It detects your platform and uses a default path to the chrome executable
-(`Opal::CliRunners::Chrome#chrome_executable`) but you can override it by specifying `GOOGLE_CHROME_BINARY` environment
+or runs the server on its own. It detects your platform and uses a default path to the Chrome executable
+(`Opal::CliRunners::Chrome#chrome_executable`), but you can override it by specifying `GOOGLE_CHROME_BINARY` environment
 variable.
 
 When the server is up and running it passes compiled js code to `lib/opal/cli_runners/chrome_cdp_interface.rb`
@@ -70,6 +73,18 @@ as a plain input using stdin (basically, it's a second part of the runner).
 `chrome_cdp_interface.rb` is a node js + Opal script that does the main job. It runs any provided code on the running chrome server,
 catches errors and forwards console messages.
 
+### Firefox
+
+This runner tries to connect to `localhost:9333` (9333 is the default port for a headless firefox server used by Opal to prevent accidental
+connection to a lingering Chrome at port 9222)
+or runs the server on its own. It detects your platform and uses a default path to the Firefox executable
+(`Opal::CliRunners::Firefox#firefox_executable`), but you can override it by specifying `MOZILLA_FIREFOX_BINARY` environment
+variable.
+
+When the server is up and running it passes compiled js code to `lib/opal/cli_runners/firefox_cdp_interface.rb`
+as a plain input using stdin (basically, it's a second part of the runner).
+`firefox_cdp_interface.rb` is a node js + Opal script that does the main job. It runs any provided code on the running chrome server,
+catches errors and forwards console messages.
 
 ## Using a remote chrome server
 
@@ -82,6 +97,8 @@ you can override default values by specifying `CHROME_HOST` and `CHROME_PORT` en
 
 NOTE: `CHROME_HOST` requires a chrome server to be started. You can't start remotely a server on a different host.
 
+The `CHROME_HOST` and `CHROME_PORT` environment variables are also used by the Firefox runner, the underlying 'chrome-remote-interface'
+node module depends on them.
 
 ## Additional options
 
@@ -93,3 +110,5 @@ If you need to pass additional CLI options to the Chrome executable you can do s
 Docker users may need `CHROME_OPTS="--no-sandbox"` due to the user namespaces limitations.
 
 _For a list of additional options see https://developers.google.com/web/updates/2017/04/headless-chrome_
+
+For the Firefox runner use the `FIREFOX_OPTS` environment variable instead.

data/lib/opal/ast/builder.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 require 'opal/ast/node'
-require 'parser/ruby31'
+require 'parser/ruby32'
 
 module Opal
   module AST

data/lib/opal/builder.rb

@@ -192,6 +192,11 @@ module Opal
       end
     end
 
+    # Return a list of dependent files, for watching purposes
+    def dependent_files
+      processed.map(&:abs_path).compact.select { |fn| File.exist?(fn) }
+    end
+
     private
 
     def process_requires(rel_path, requires, autoloads, options)
@@ -231,7 +236,7 @@ module Opal
 
       options = options.merge(cache: cache)
 
-      processor.new(source, rel_path, @compiler_options.merge(options))
+      processor.new(source, rel_path, abs_path, @compiler_options.merge(options))
     end
 
     def read(path, autoload)

data/lib/opal/builder_processors.rb

@@ -6,15 +6,17 @@ require 'opal/erb'
 module Opal
   module BuilderProcessors
     class Processor
-      def initialize(source, filename, options = {})
+      def initialize(source, filename, abs_path = nil, options = {})
+        options = abs_path if abs_path.is_a? Hash
+
         source += "\n" unless source.end_with?("\n")
-        @source, @filename, @options = source, filename, options.dup
+        @source, @filename, @abs_path, @options = source, filename, abs_path, options.dup
         @cache = @options.delete(:cache) { Opal.cache }
         @requires = []
         @required_trees = []
         @autoloads = []
       end
-      attr_reader :source, :filename, :options, :requires, :required_trees, :autoloads
+      attr_reader :source, :filename, :options, :requires, :required_trees, :autoloads, :abs_path
 
       def to_s
         source.to_s

data/lib/opal/cache.rb

@@ -48,13 +48,7 @@ module Opal
 
     def runtime_key
       @runtime_key ||= begin
-        # We want to ensure the compiler and any Gemfile/gemspec (for development)
-        # stays untouched
-        opal_path = File.expand_path('..', Opal.gem_dir)
-        files = Dir["#{opal_path}/{Gemfile*,*.gemspec,lib/**/*}"]
-
-        # Also check if parser wasn't changed:
-        files += $LOADED_FEATURES.grep(%r{lib/(parser|ast)})
+        files = Opal.dependent_files
 
         digest [
           files.sort.map { |f| "#{f}:#{File.size(f)}:#{File.mtime(f).to_f}" },

data/lib/opal/cli_options.rb

@@ -13,6 +13,10 @@ module Opal
 
       separator ''
 
+      on('--repl', 'Run the Opal REPL') do
+        options[:repl] = true
+      end
+
       on('-v', '--verbose', 'print version number, then turn on verbose mode') do
         print_version
         exit if ARGV.empty?
@@ -32,10 +36,6 @@ module Opal
         exit
       end
 
-      on('--repl', 'Run the Opal REPL') do
-        options[:repl] = true
-      end
-
       on('-h', '--help', 'Show this message') do
         puts self
         exit
@@ -43,11 +43,6 @@ module Opal
 
       section 'Basic Options:'
 
-      on('-I', '--include DIR', 'Append a load path (may be used more than once)') do |i|
-        options[:load_paths] ||= []
-        options[:load_paths] << i
-      end
-
       on('-e', '--eval SOURCE', String,
         'One line of script. Several -e\'s allowed. Omit [programfile]'
       ) do |source|
@@ -69,6 +64,11 @@ module Opal
         options[:rbrequires] << library
       end
 
+      on('-I', '--include DIR', 'Append a load path (may be used more than once)') do |i|
+        options[:load_paths] ||= []
+        options[:load_paths] << i
+      end
+
       on('-s', '--stub FILE', String, 'Stubbed files will be compiled as empty files') do |stub|
         options[:stubs] ||= []
         options[:stubs] << stub
@@ -86,54 +86,78 @@ module Opal
 
       section 'Running Options:'
 
-      on('--sexp', 'Show Sexps') do
-        options[:sexp] = true
+      on('-R', '--runner RUNNER', Opal::CliRunners.to_h.keys, 'Choose the runner:', "nodejs (default), #{(Opal::CliRunners.to_h.keys - %i[nodejs compiler]).join(', ')}") do |runner|
+        options[:runner] = runner.to_sym
       end
 
-      on('--debug-source-map', 'Debug source map') do
-        options[:debug_source_map] = true
+      on('--server-port PORT', 'Set the port for the server runner (default port: 3000)') do |port|
+        options[:runner_options] ||= {}
+        options[:runner_options][:port] = port.to_i
+      end
+
+      on('--runner-options JSON', 'Set options specific to the selected runner as a JSON string (e.g. port for server)') do |json_options|
+        require 'json'
+        runner_options = JSON.parse(json_options, symbolize_names: true)
+        options[:runner_options] ||= {}
+        options[:runner_options].merge!(runner_options)
       end
 
+      section 'Builder Options:'
+
       on('-c', '--compile', 'Compile to JavaScript') do
         options[:runner] = :compiler
       end
 
-      on('-R', '--runner RUNNER', Opal::CliRunners.to_h.keys, "Choose the runner: nodejs (default), #{(Opal::CliRunners.to_h.keys - [:nodejs]).join(', ')}") do |runner|
-        options[:runner] = runner.to_sym
+      on('-o', '--output FILE', 'Output JavaScript to FILE') do |file|
+        options[:output] = File.open(file, 'w')
       end
 
-      on('--runner-options JSON', 'Set options specific to the selected runner as a JSON string (e.g. port for server)') do |json_options|
-        require 'json'
-        runner_options = JSON.parse(json_options, symbolize_names: true)
+      on('-P', '--map FILE', 'Output source map to FILE') do |file|
         options[:runner_options] ||= {}
-        options[:runner_options].merge!(runner_options)
+        options[:runner_options][:map_file] = file
       end
 
-      on('--server-port PORT', '(deprecated, use --runner-options) Set the port for the server runner (default port: 3000)') do |port|
+      on('--no-source-map', "Don't append source map to a compiled file") do
         options[:runner_options] ||= {}
-        options[:runner_options][:port] = port.to_i
+        options[:runner_options][:no_source_map] = true
       end
 
-      on('-E', '--no-exit', 'Do not append a Kernel#exit at the end of file') do
-        options[:no_exit] = true
+      on('--watch', 'Run the compiler in foreground, recompiling every filesystem change') do
+        options[:runner_options] ||= {}
+        options[:runner_options][:watch] = true
       end
 
-      section 'Compilation Options:'
+      on('--no-cache', 'Disable filesystem cache') do
+        options[:no_cache] = true
+      end
 
-      on('-M', '--no-method-missing', 'Disable method missing') do
-        options[:method_missing] = false
+      on('-L', '--library', 'Compile only required libraries. Omit [programfile] and [-e]. Assumed [-cOE].') do
+        options[:lib_only] = true
+        options[:no_exit] = true
+        options[:compile] = true
+        options[:skip_opal_require] = true
       end
 
       on('-O', '--no-opal', 'Disable implicit `require "opal"`') do
         options[:skip_opal_require] = true
       end
 
-      on('-A', '--arity-check', 'Enable arity check') do
-        options[:arity_check] = true
+      on('-E', '--no-exit', 'Do not append a Kernel#exit at the end of file') do
+        options[:no_exit] = true
       end
 
-      on('-V', 'Enable inline Operators') do
-        options[:inline_operators] = true
+      section 'Compiler Options:'
+
+      on('--use-strict', 'Enables JavaScript\'s strict mode (i.e., adds \'use strict\'; statement)') do
+        options[:use_strict] = true
+      end
+
+      on('--esm', 'Wraps compiled bundle as for ES6 module') do
+        options[:esm] = true
+      end
+
+      on('-A', '--arity-check', 'Enable arity check') do
+        options[:arity_check] = true
       end
 
       dynamic_require_levels = %w[error warning ignore]
@@ -152,53 +176,43 @@ module Opal
         options[:missing_require_severity] = level.to_sym
       end
 
-      on('-P', '--map FILE', 'Output path to FILE') do |file|
-        options[:runner_options] ||= {}
-        options[:runner_options][:map_file] = file
-      end
-
-      on('--no-source-map', "Don't append source map to a compiled file") do
-        options[:runner_options] ||= {}
-        options[:runner_options][:no_source_map] = true
+      on('--enable-source-location', 'Compiles source location for each method definition.') do
+        options[:enable_source_location] = true
       end
 
-      on('-F', '--file FILE', 'Set filename for compiled code') do |file|
-        options[:file] = file
+      on('--parse-comments', 'Compiles comments for each method definition.') do
+        options[:parse_comments] = true
       end
 
-      on('-L', '--library', 'Compile only required libraries. Omit [programfile] and [-e]. Assumed [-cOE].') do
-        options[:lib_only] = true
-        options[:no_exit] = true
-        options[:compile] = true
-        options[:skip_opal_require] = true
+      on('--enable-file-source-embed', 'Embeds file sources to be accessed by applications.') do
+        options[:enable_file_source_embed] = true
       end
 
       on('--irb', 'Enable IRB var mode') do
         options[:irb] = true
       end
 
-      on('--enable-source-location', 'Compiles source location for each method definition.') do
-        options[:enable_source_location] = true
+      on('-M', '--no-method-missing', 'Disable method missing') do
+        options[:method_missing] = false
       end
 
-      on('--enable-file-source-embed', 'Embeds file sources to be accessed by applications.') do
-        options[:enable_file_source_embed] = true
+      on('-F', '--file FILE', 'Set filename for compiled code') do |file|
+        options[:file] = file
       end
 
-      on('--use-strict', 'Enables JavaScript\'s strict mode (i.e., adds \'use strict\'; statement)') do
-        options[:use_strict] = true
+      on('-V', '(deprecated; always enabled) Enable inline Operators') do
+        warn '* -V is deprecated and has no effect'
+        options[:inline_operators] = true
       end
 
-      on('--parse-comments', 'Compiles comments for each method definition.') do
-        options[:parse_comments] = true
-      end
+      section 'Debug Options:'
 
-      on('--no-cache', 'Disable filesystem cache') do
-        options[:no_cache] = true
+      on('--sexp', 'Show Sexps') do
+        options[:sexp] = true
       end
 
-      on('--esm', 'Wraps compiled bundle as for ES6 module(requires -c or -L)') do
-        options[:esm] = true
+      on('--debug-source-map', 'Debug source map') do
+        options[:debug_source_map] = true
       end
 
       separator ''

data/lib/opal/cli_runners/chrome.rb

@@ -21,16 +21,14 @@ module Opal
       end
 
       def initialize(data)
-        builder = data[:builder].call
-        options = data[:options]
-        argv    = data[:argv]
-
+        argv = data[:argv]
         if argv && argv.any?
           warn "warning: ARGV is not supported by the Chrome runner #{argv.inspect}"
         end
 
-        @output = options.fetch(:output, $stdout)
-        @builder = builder
+        options  = data[:options]
+        @output  = options.fetch(:output, $stdout)
+        @builder = data[:builder].call
       end
 
       attr_reader :output, :exit_status, :builder
@@ -43,7 +41,8 @@ module Opal
             env = {
               'CHROME_HOST' => chrome_host,
               'CHROME_PORT' => chrome_port.to_s,
-              'NODE_PATH' => File.join(__dir__, 'node_modules')
+              'NODE_PATH' => File.join(__dir__, 'node_modules'),
+              'OPAL_CDP_EXT' => builder.output_extension
             }
 
             cmd = [
@@ -53,7 +52,7 @@ module Opal
               '-I', __dir__,
               '-r', 'source-map-support-node',
               SCRIPT_PATH,
-              dir,
+              dir
             ]
 
             Kernel.exec(env, *cmd)
@@ -75,10 +74,12 @@ module Opal
         # https://groups.google.com/a/chromium.org/forum/#!topic/chromium-discuss/U5qyeX_ydBo
         # The only way is to create temporary files and pass them to chrome.
         File.binwrite("#{dir}/index.#{ext}", js)
+        File.binwrite("#{dir}/index.map", map)
         File.binwrite("#{dir}/source-map-support.js", stack)
         File.binwrite("#{dir}/index.html", <<~HTML)
           <html><head>
             <meta charset='utf-8'>
+            <link rel="shortcut icon" href="data:image/x-icon;," type="image/x-icon">
             <script src='./source-map-support.js'></script>
             <script>
             window.opalheadlesschrome = true;
@@ -115,14 +116,46 @@ module Opal
       def run_chrome_server
         raise 'Chrome server can be started only on localhost' if chrome_host != DEFAULT_CHROME_HOST
 
+        profile = mktmpprofile
+
         # Disable web security with "--disable-web-security" flag to be able to do XMLHttpRequest (see test_openuri.rb)
+        # For other options see https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/node/ChromeLauncher.ts
         chrome_server_cmd = %{#{OS.shellescape(chrome_executable)} \
+          --allow-pre-commit-input \
+          --disable-background-networking \
+          --enable-features=NetworkServiceInProcess2 \
+          --disable-background-timer-throttling \
+          --disable-backgrounding-occluded-windows \
+          --disable-breakpad \
+          --disable-client-side-phishing-detection \
+          --disable-component-extensions-with-background-pages \
+          --disable-default-apps \
+          --disable-dev-shm-usage \
+          --disable-extensions \
+          --disable-features=Translate,BackForwardCache,AcceptCHFrame,AvoidUnnecessaryBeforeUnloadCheckSync \
+          --disable-hang-monitor \
+          --disable-ipc-flooding-protection \
+          --disable-popup-blocking \
+          --disable-prompt-on-repost \
+          --disable-renderer-backgrounding \
+          --disable-sync \
+          --force-color-profile=srgb \
+          --metrics-recording-only \
+          --no-first-run \
+          --enable-automation \
+          --password-store=basic \
+          --use-mock-keychain \
+          --enable-blink-features=IdleDetection \
+          --export-tagged-pdf \
           --headless \
+          --user-data-dir=#{profile} \
+          --hide-scrollbars \
+          --mute-audio \
           --disable-web-security \
           --remote-debugging-port=#{chrome_port} \
           #{ENV['CHROME_OPTS']}}
 
-        chrome_pid = Process.spawn(chrome_server_cmd)
+        chrome_pid = Process.spawn(chrome_server_cmd, in: OS.dev_null, out: OS.dev_null, err: OS.dev_null)
 
         Timeout.timeout(30) do
           loop do
@@ -142,6 +175,7 @@ module Opal
         elsif chrome_pid
           Process.kill('HUP', chrome_pid)
         end
+        FileUtils.rm_rf(profile) if profile
       end
 
       def chrome_server_running?
@@ -181,6 +215,10 @@ module Opal
       def mktmpdir(&block)
         Dir.mktmpdir('chrome-opal-', &block)
       end
+
+      def mktmpprofile
+        Dir.mktmpdir('chrome-opal-profile-')
+      end
     end
   end
 end