rcee_precompiled 0.1.0-x64-mingw32 → 0.4.0-x64-mingw32

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e12ef62653c5915c0d72a743c49899ae029682943297ea443f13cfc7cb789ad4
-  data.tar.gz: 5fc721a6b4bce84bf2bedfc3aed6b275c2bf945c5524d5a7d005e90b9e57b8c8
+  metadata.gz: ea6f3cd835a5bed241848335027c8a8cf022833ae7026da43029743ad5f91880
+  data.tar.gz: a766319f7e9bccd4ec17ece36b32d6fb7e4b66bb7ab0d507ac152895d5d3d5ba
 SHA512:
-  metadata.gz: 94212df664d8204add58e579455d67814142ae83dc9a8c98e7bc0db924212753b1dc54033c55f97cdf048be2c0fe8234bbdeaa8f44fdb6fc8d5ff981af704948
-  data.tar.gz: 748b518e55b7dcf53f7b2c99c044d81cfe56e309c48daeb4807c79ba7122ef64da62d4012c2ee687cd5d55b5e67c51e5b93847417395f6bf5569ae674d23a2f9
+  metadata.gz: 75ee4cf1972cbba08b4a0907bb2e0dbfe1161a6a94d9be354fc430823c8600b45e42a460c996a820e665b5c6deaadb7f223dd096ebbcba46f4c24642932c7ccb
+  data.tar.gz: e70418704048aa9f987acb84d84181798fd133bec1335ef3846a78310f0521438a6a84c2f8de81dc477dcace06715b4967c1acba6484b0d2225c13296f149ec6

data/Gemfile

@@ -8,6 +8,6 @@ gemspec
 gem "rake", "~> 13.0"
 
 gem "rake-compiler"
-gem "rake-compiler-dock"
+gem "rake-compiler-dock", "~> 1.2.1"
 
 gem "minitest", "~> 5.0"

data/README.md

@@ -1,2 +1,191 @@
+# RCEE::Precompiled
 
 This gem is part of the Ruby C Extensions Explained project at https://github.com/flavorjones/ruby-c-extensions-explained
+
+## Summary
+
+Installation time is an important aspect of developer happiness, and the Packaged strategies can make your users very unhappy.
+
+One way to speed up installation time is to precompile the extension, so that during `gem install` files are simply being unpacked, and the entire configure/compile/link process is avoided completely. This can be tricky to set up, and requires that you test across all the target platforms; but makes for much happier users.
+
+Note, however, that it's necessary to _also_ ship the vanilla (platform=ruby) gem as a fallback for platforms you haven't pre-compiled for.
+
+In the nine months since Nokogiri v1.11 started shipping precompiled native gems, we've seen over 45 million gem installations, and essentially zero support issues have been opened. Twitter complaints dropped to near zero. Page views of the Nokogiri installation tutorial dropped by half.
+
+Precompiled Nokogiri gems have been an unmitigated success and have made both users and maintainers happy.
+
+
+## Details
+
+An important tool we rely on for precompiling is [`rake-compiler/rake-compiler-dock`](https://github.com/rake-compiler/rake-compiler-dock), maintained by Lars Kanis. `rake-compiler-dock` is a Docker-based build environment that uses `rake-compiler` to _cross compile_ for all of these major platforms.
+
+What this means is, I can run my normal gem build process in a docker container on my Linux machine, and it will build gems that I run on Windows and MacOS.
+
+This is really powerful stuff, and once we assume that we can cross-compile reliably, the remaining problems boil down to modifying how we build the gemfile and making sure we test gems adequately on the target platforms.
+
+First, we need to add some features to our `Rake::ExtensionTask` in `Rakefile`:
+
+``` ruby
+cross_rubies = ["3.1.0", "3.0.0", "2.7.0", "2.6.0"]
+cross_platforms = [
+  "x64-mingw32",
+  "x64-mingw-ucrt",
+  "x86-linux",
+  "x86_64-linux",
+  "aarch64-linux",
+  "x86_64-darwin",
+  "arm64-darwin",
+]
+ENV["RUBY_CC_VERSION"] = cross_rubies.join(":")
+
+Rake::ExtensionTask.new("precompiled", rcee_precompiled_spec) do |ext|
+  ext.lib_dir = "lib/rcee/precompiled"
+  ext.cross_compile = true
+  ext.cross_platform = cross_platforms
+  ext.cross_config_options << "--enable-cross-build" # so extconf.rb knows we're cross-compiling
+  ext.cross_compiling do |spec|
+    # remove things not needed for precompiled gems
+    spec.dependencies.reject! { |dep| dep.name == "mini_portile2" }
+    spec.files.reject! { |file| File.fnmatch?("*.tar.gz", file) }
+  end
+end
+```
+
+This does the following:
+
+- set up some local variables to indicate what ruby versions and which platforms we will build for
+- set an environment variable to let rake-compiler know which ruby versions we'll build
+- tell the extension task to turn on cross-compiling features, including additional rake tasks
+- signal to our `extconf.rb` when we're cross-compiling (in case its behavior needs to change)
+- finally, in a block that is only run when cross-compiling, we modify the gemspec to remove things we don't need in native gems:
+  - the tarball
+  - the dependency on `mini_portile`
+
+Next we need some new rake tasks:
+
+``` ruby
+namespace "gem" do
+  cross_platforms.each do |platform|
+    desc "build native gem for #{platform}"
+    task platform do
+      RakeCompilerDock.sh(<<~EOF, platform: platform)
+        gem install bundler --no-document &&
+        bundle &&
+        bundle exec rake gem:#{platform}:buildit
+      EOF
+    end
+
+    namespace platform do
+      # this runs in the rake-compiler-dock docker container
+      task "buildit" do
+        # use Task#invoke because the pkg/*gem task is defined at runtime
+        Rake::Task["native:#{platform}"].invoke
+        Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform)}.gem"].invoke
+      end
+    end
+  end
+end
+```
+
+The top task (or, really, _set_ of tasks) runs on the host system, and invokes the lower task within the appropriate docker container to cross-compile for that platform. So the bottom task is doing most of the work, and it's doing it inside a guest container. Please note that the worker is both building the extension *and* packaging the gem according to the gemspec that was just modified by our extensiontask cross-compiling block.
+
+Changes to the `extconf.rb`:
+
+``` ruby
+ENV["CC"] = RbConfig::CONFIG["CC"]
+```
+
+This makes sure that the cross-compiler is the compiler used within the guest container (and not the native linux compiler).
+
+``` ruby
+cross_build_p = enable_config("cross-build")
+```
+
+The cross-compile rake task signals to `extconf.rb` that it's cross-compiling by using a commandline flag that we can inspect. We'll need this for `libyaml` to make sure that set the appropriate flags during precompilation (flags which shouldn't be set when compiling natively).
+
+``` ruby
+MiniPortile.new("yaml", "0.2.5").tap do |recipe|
+  # ...
+  # configure the environment that MiniPortile will use for subshells
+  ENV.to_h.dup.tap do |env|
+    # -fPIC is necessary for linking into a shared library
+    env["CFLAGS"] = [env["CFLAGS"], "-fPIC"].join(" ")
+    env["SUBDIRS"] = "include src" # libyaml: skip tests
+
+    recipe.configure_options += env.map { |key, value| "#{key}=#{value.strip}" }
+  end
+  # ...
+end
+```
+
+The rest of the extconf changes are related to configuring libyaml at build time. We need to set the -fPIC option so we can mix static and shared libraries together. (This should probably always be set.)
+
+The "SUBDIRS" environment variable is something that's very specific to libyaml, though: it tells libyaml's autoconf build system to skip running the tests. We have to do this because although we can *generate* binaries for other platforms, we can't actually *run* them.
+
+We have one more small change we'll need to make to how the extension is required. Let's take a look at the directory structure in the packaged gem:
+
+``` text
+lib
+└── rcee
+    ├── precompiled
+    │   ├── 2.6
+    │   │   └── precompiled.so
+    │   ├── 2.7
+    │   │   └── precompiled.so
+    │   ├── 3.0
+    │   │   └── precompiled.so
+    │   ├── 3.1
+    │   │   └── precompiled.so
+    │   └── version.rb
+    └── precompiled.rb
+```
+
+You can see that we have FOUR c extensions in this gem, one for each minor version of Ruby that we support. Remember that a C extension is specific to an architecture and a version of Ruby. For example, if we're running Ruby 3.0.1, then we need to load the extension in the 3.0 directory. Let's make sure we do that.
+
+In `lib/rcee/precompiled.rb`, we'll replace the normal `require` with:
+
+``` ruby
+begin
+  # load the precompiled extension file
+  ruby_version = /(\d+\.\d+)/.match(::RUBY_VERSION)
+  require_relative "precompiled/#{ruby_version}/precompiled"
+rescue LoadError
+  # fall back to the extension compiled upon installation.
+  require "rcee/precompiled/precompiled"
+end
+```
+
+Go ahead and try it! `gem install rcee_precompiled`. If you're on windows, linux, or macos you should get a precompiled version that installs in under a second. Everyone else (hello FreeBSD people!) it'll take a few more seconds to build the vanilla gem's packaged tarball.
+
+
+## Testing
+
+See [.github/workflows/precompiled.yml](../.github/workflows/precompiled.yml)
+
+Key things to note:
+
+- matrix across all supported Rubies and platforms (for compile-from-source installation testing)
+- test native gems for a variety of platforms
+  - use rake-compiler-dock images to build the gems
+  - then install on native platforms and verify that it passes tests
+
+Note that there's additional complexity because of how we test:
+
+- see new script bin/test-gem-build which artificially bumps the VERSION string to double-check we're testing the packaged version of the gem (which the tests output)
+- see new script bin/test-gem-install which installs the gem, deletes the local source code, and runs the tests against the installed gem
+- the gemspec handles a missing version file (because we delete the local source code during testing)
+
+
+## What Can Go Wrong
+
+This strategy isn't perfect. Remember what I said earlier, that a compiled C extension is specific to
+
+- the minor version of ruby (e.g., 3.0)
+- the machine architecture (e.g., x86_64)
+- the system libraries
+
+The precompiled strategy mostly takes care of the first two, but there are still edge cases for system libraries. The big gotcha is that linux libc is not the same as linux musl, and we've had to work around this a few times in Nokogiri.
+
+I'm positive that there are more edge cases that will be found as users add more platforms and as more gems start precompiling. I'm willing to bet money that you can break this by setting some Ruby compile-time flags on your system. I'm honestly surprised it works as well as it has. (Worth noting: the `sassc` gem stopped shipping native gems for linux because of the musl incompatibilities.)
+
+So the lesson here is: make sure you have an automated test pipeline that will build a gem and test it on the target platform! This takes time to set up, but it will save you time and effort in the long run.

data/Rakefile

@@ -1,9 +1,27 @@
 # frozen_string_literal: true
 
 require "bundler/gem_tasks"
+require "rubygems/package_task"
 require "rake/testtask"
+require "rake/extensiontask"
+require "rake_compiler_dock"
+
+cross_rubies = ["3.1.0", "3.0.0", "2.7.0", "2.6.0"]
+cross_platforms = [
+  "aarch64-linux",
+  "arm-linux",
+  "arm64-darwin",
+  "x64-mingw-ucrt",
+  "x64-mingw32",
+  "x86-linux",
+  "x86_64-darwin",
+  "x86_64-linux",
+]
+ENV["RUBY_CC_VERSION"] = cross_rubies.join(":")
 
 rcee_precompiled_spec = Bundler.load_gemspec("rcee_precompiled.gemspec")
+Gem::PackageTask.new(rcee_precompiled_spec).define #packaged_tarball version of the gem for platform=ruby
+task "package" => cross_platforms.map { |p| "gem:#{p}" } # "package" task for all the native platforms
 
 Rake::TestTask.new(:test) do |t|
   t.libs << "test"
@@ -11,19 +29,6 @@ Rake::TestTask.new(:test) do |t|
   t.test_files = FileList["test/**/*_test.rb"]
 end
 
-# "gem" task for vanilla packaged (tarball)
-require "rubygems/package_task"
-Gem::PackageTask.new(rcee_precompiled_spec).define
-
-require "rake/extensiontask"
-require "rake_compiler_dock"
-
-cross_rubies = ["3.0.0", "2.7.0", "2.6.0", "2.5.0"]
-cross_platforms = ["x64-mingw32", "x86_64-linux", "x86_64-darwin", "arm64-darwin"]
-
-ENV["RUBY_CC_VERSION"] = cross_rubies.join(":")
-
-
 Rake::ExtensionTask.new("precompiled", rcee_precompiled_spec) do |ext|
   ext.lib_dir = "lib/rcee/precompiled"
   ext.cross_compile = true
@@ -31,7 +36,7 @@ Rake::ExtensionTask.new("precompiled", rcee_precompiled_spec) do |ext|
   ext.cross_config_options << "--enable-cross-build" # so extconf.rb knows we're cross-compiling
   ext.cross_compiling do |spec|
     # remove things not needed for precompiled gems
-    spec.dependencies.reject! { |dep| dep.name == 'mini_portile2' }
+    spec.dependencies.reject! { |dep| dep.name == "mini_portile2" }
     spec.files.reject! { |file| File.fnmatch?("*.tar.gz", file) }
   end
 end
@@ -52,7 +57,7 @@ namespace "gem" do
       task "buildit" do
         # use Task#invoke because the pkg/*gem task is defined at runtime
         Rake::Task["native:#{platform}"].invoke
-        Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform).to_s}.gem"].invoke
+        Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform)}.gem"].invoke
       end
     end
   end
@@ -61,9 +66,36 @@ namespace "gem" do
   multitask "all" => [cross_platforms, "gem"].flatten
 end
 
+desc "Temporarily set VERSION to a unique timestamp"
+task "set-version-to-timestamp" do
+  # this task is used by bin/test-gem-build
+  # to test building, packaging, and installing a precompiled gem
+  version_constant_re = /^\s*VERSION\s*=\s*["'](.*)["']$/
+
+  version_file_path = File.join(__dir__, "lib/rcee/precompiled/version.rb")
+  version_file_contents = File.read(version_file_path)
+
+  current_version_string = version_constant_re.match(version_file_contents)[1]
+  current_version = Gem::Version.new(current_version_string)
+
+  fake_version = Gem::Version.new(format("%s.test.%s", current_version.bump, Time.now.strftime("%Y.%m%d.%H%M")))
+
+  unless version_file_contents.gsub!(version_constant_re, "    VERSION = \"#{fake_version}\"")
+    raise("Could not hack the VERSION constant")
+  end
+
+  File.open(version_file_path, "w") { |f| f.write(version_file_contents) }
+
+  puts "NOTE: wrote version as \"#{fake_version}\""
+end
+
+task default: [:clobber, :compile, :test]
 
+CLEAN.add("{ext,lib}/**/*.{o,so}", "pkg")
 CLOBBER.add("ports")
-CLEAN.add("{ext,lib}/**/*.{o,so}")
 
-task build: :compile
-task default: %i[clobber compile test]
+# when packaging the gem, if the tarball isn't cached, we need to fetch it. the easiest thing to do
+# is to run the compile phase to invoke the extconf and have mini_portile download the file for us.
+# this is wasteful and in the future I would prefer to separate mini_portile from the extconf to
+# allow us to download without compiling.
+Rake::Task["package"].prerequisites.prepend("compile")

data/ext/precompiled/extconf.rb

@@ -6,7 +6,6 @@ package_root_dir = File.expand_path(File.join(File.dirname(__FILE__), "..", ".."
 RbConfig::CONFIG["CC"] = RbConfig::MAKEFILE_CONFIG["CC"] = ENV["CC"] if ENV["CC"]
 ENV["CC"] = RbConfig::CONFIG["CC"]
 
-# not needed for libyaml, but generally useful to know whether we're cross-compiling
 cross_build_p = enable_config("cross-build")
 
 MiniPortile.new("yaml", "0.2.5").tap do |recipe|
@@ -17,12 +16,14 @@ MiniPortile.new("yaml", "0.2.5").tap do |recipe|
   recipe.target = File.join(package_root_dir, "ports")
 
   # configure the environment that MiniPortile will use for subshells
-  ENV.to_h.dup.tap do |env|
-    # -fPIC is necessary for linking into a shared library
-    env["CFLAGS"] = [env["CFLAGS"], "-fPIC"].join(" ")
-    env["SUBDIRS"] = "include src" # libyaml: skip tests
-
-    recipe.configure_options += env.map { |key, value| "#{key}=#{value.strip}" }
+  if cross_build_p
+    ENV.to_h.tap do |env|
+      # -fPIC is necessary for linking into a shared library
+      env["CFLAGS"] = [env["CFLAGS"], "-fPIC"].join(" ")
+      env["SUBDIRS"] = "include src" # libyaml: skip tests
+
+      recipe.configure_options += env.map { |key, value| "#{key}=#{value.strip}" }
+    end
   end
 
   unless File.exist?(File.join(recipe.target, recipe.host, recipe.name, recipe.version))

data/lib/rcee/precompiled/version.rb

@@ -2,6 +2,6 @@
 
 module RCEE
   module Precompiled
-    VERSION = "0.1.0"
+    VERSION = "0.4.0"
   end
 end

data/rcee_precompiled.gemspec

@@ -1,17 +1,21 @@
 # frozen_string_literal: true
 
-require_relative "lib/rcee/precompiled/version"
+begin
+  require_relative "lib/rcee/precompiled/version"
+rescue LoadError
+  puts "WARNING: Could not load RCEE::Precompiled::VERSION"
+end
 
 Gem::Specification.new do |spec|
   spec.name          = "rcee_precompiled"
-  spec.version       = RCEE::Precompiled::VERSION
+  spec.version       = defined?(RCEE::Precompiled::VERSION) ? RCEE::Precompiled::VERSION : "0.0.0"
   spec.authors       = ["Mike Dalessio"]
   spec.email         = ["mike.dalessio@gmail.com"]
 
   spec.summary       = "Example gem demonstrating a basic C extension."
   spec.description   = "Part of a project to explain how Ruby C extensions work."
   spec.homepage      = "https://github.com/flavorjones/ruby-c-extensions-explained"
-  spec.required_ruby_version = ">= 2.5.0"
+  spec.required_ruby_version = ">= 2.6.0"
   spec.license = "MIT"
 
   # Specify which files should be added to the gem when it is released.

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rcee_precompiled
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.4.0
 platform: x64-mingw32
 authors:
 - Mike Dalessio
-autorequire: 
+autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-09-02 00:00:00.000000000 Z
+date: 2022-05-19 00:00:00.000000000 Z
 dependencies: []
 description: Part of a project to explain how Ruby C extensions work.
 email:
@@ -25,7 +25,6 @@ files:
 - ext/precompiled/precompiled.c
 - ext/precompiled/precompiled.h
 - lib/rcee/precompiled.rb
-- lib/rcee/precompiled/2.5/precompiled.so
 - lib/rcee/precompiled/2.6/precompiled.so
 - lib/rcee/precompiled/2.7/precompiled.so
 - lib/rcee/precompiled/3.0/precompiled.so
@@ -35,7 +34,7 @@ homepage: https://github.com/flavorjones/ruby-c-extensions-explained
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -43,7 +42,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.5'
+      version: '2.6'
   - - "<"
     - !ruby/object:Gem::Version
       version: 3.1.dev
@@ -53,8 +52,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.3
-signing_key: 
+rubygems_version: 3.3.4
+signing_key:
 specification_version: 4
 summary: Example gem demonstrating a basic C extension.
 test_files: []