rcee_precompiled 0.1.0-x86_64-darwin → 0.2.0-x86_64-darwin

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ff2a0fb01dfe21ea765be928b094b9e3372e85d57c6886d3e98e1753d8f34602
-  data.tar.gz: '0999c42935488a6033b3524015cd843efc874892bc92cd74b409ccd140dfd974'
+  metadata.gz: c4cbbc4cd72bc77def66ab41486f15bb9dd86cc8e22e28d8c0a0d056ad2398d1
+  data.tar.gz: de2c8877cf4413177fe5fbd73e942b7fc8498d058e22edbf9a7b26174ae38df7
 SHA512:
-  metadata.gz: bc2ff74a9040da02dd3fd2fae273ebb38aad5fefafa95a57531253d73bc2a379a10ca321be3e09e26ccc9d2e61d4b663a24636c3138df0832a3921e12c76cef4
-  data.tar.gz: fff9b7d04dbfbf7bc256ebd0c015db1fbfef991fb7fc262d5a8c4b9ba00b22a6065e4f438512f9014d9565039bec00329e4b0a52617826524623846298e0498a
+  metadata.gz: 41925f02c420be44d4a36219b73484f0bc8a03ec888c917369341adfe3ac10027f9aeeb81033c7842187a2eee757369d56fd506c049eb9d2dd69c28d65d3b297
+  data.tar.gz: 04c612d43440579679a5ec1a700a5d9c60c28af6394612eca01d24ab5424b6b9ce74d115d4a66ce9b570efafe7316d5aa517ed1033508f7c26649aaa53a9a890

data/README.md

@@ -1,2 +1,165 @@
+# RCEE::Precompiled
 
 This gem is part of the Ruby C Extensions Explained project at https://github.com/flavorjones/ruby-c-extensions-explained
+
+## Summary
+
+Installation time is an important aspect of developer happiness, and the Packaged strategies can make your users very unhappy.
+
+One way to speed up installation time is to precompile the extension, so that during `gem install` files are simply being unpacked, and the entire configure/compile/link process is avoided completely. This can be tricky to set up, and requires that you test across all the target platforms; but makes for much happier users.
+
+Note, however, that it's necessary to _also_ ship the vanilla (platform=ruby) gem as a fallback for platforms you haven't pre-compiled for.
+
+In the nine months since Nokogiri v1.11 started shipping precompiled native gems, we've seen over 45 million gem installations, and essentially zero support issues have been opened. Twitter complaints dropped to near zero. Page views of the Nokogiri installation tutorial dropped by half.
+
+Precompiled Nokogiri gems have been an unmitigated success and have made both users and maintainers happy.
+
+
+## Details
+
+An important tool we rely on for precompiling is [`rake-compiler/rake-compiler-dock`](https://github.com/rake-compiler/rake-compiler-dock), maintained by Lars Kanis. `rake-compiler-dock` is a Docker-based build environment that uses `rake-compiler` to _cross compile_ for all of these major platforms.
+
+What this means is, I can run my normal gem build process in a docker container on my Linux machine, and it will build gems that I run on Windows and MacOS.
+
+This is really powerful stuff, and once we assume that we can cross-compile reliably, the remaining problems boil down to modifying how we build the gemfile and making sure we test gems adequately on the target platforms.
+
+First, we need to add some features to our `Rake::ExtensionTask` in `Rakefile`:
+
+``` ruby
+cross_rubies = ["3.0.0", "2.7.0", "2.6.0", "2.5.0"]
+cross_platforms = ["x64-mingw32", "x86_64-linux", "x86_64-darwin", "arm64-darwin"]
+ENV["RUBY_CC_VERSION"] = cross_rubies.join(":")
+
+Rake::ExtensionTask.new("precompiled", rcee_precompiled_spec) do |ext|
+  ext.lib_dir = "lib/rcee/precompiled"
+  ext.cross_compile = true
+  ext.cross_platform = cross_platforms
+  ext.cross_config_options << "--enable-cross-build" # so extconf.rb knows we're cross-compiling
+  ext.cross_compiling do |spec|
+    # remove things not needed for precompiled gems
+    spec.dependencies.reject! { |dep| dep.name == "mini_portile2" }
+    spec.files.reject! { |file| File.fnmatch?("*.tar.gz", file) }
+  end
+end
+```
+
+This does the following:
+
+- set up some local variables to indicate what ruby versions and which platforms we will build for
+- set an environment variable to let rake-compiler know which ruby versions we'll build
+- tell the extension task to turn on cross-compiling features, including additional rake tasks
+- signal to our `extconf.rb` when we're cross-compiling (in case its behavior needs to change)
+- finally, in a block that is only run when cross-compiling, we modify the gemspec to remove things we don't need in native gems:
+  - the tarball
+  - the dependency on `mini_portile`
+
+Next we need some new rake tasks:
+
+``` ruby
+namespace "gem" do
+  cross_platforms.each do |platform|
+    desc "build native gem for #{platform}"
+    task platform do
+      RakeCompilerDock.sh(<<~EOF, platform: platform)
+        gem install bundler --no-document &&
+        bundle &&
+        bundle exec rake gem:#{platform}:buildit
+      EOF
+    end
+
+    namespace platform do
+      # this runs in the rake-compiler-dock docker container
+      task "buildit" do
+        # use Task#invoke because the pkg/*gem task is defined at runtime
+        Rake::Task["native:#{platform}"].invoke
+        Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform)}.gem"].invoke
+      end
+    end
+  end
+end
+```
+
+The top task (or, really, _set_ of tasks) runs on the host system, and invokes the lower task within the appropriate docker container to cross-compile for that platform. So the bottom task is doing most of the work, and it's doing it inside a guest container. Please note that the worker is both building the extension *and* packaging the gem according to the gemspec that was just modified by our extensiontask cross-compiling block.
+
+Changes to the `extconf.rb`:
+
+``` ruby
+ENV["CC"] = RbConfig::CONFIG["CC"]
+```
+
+This makes sure that the cross-compiler is the compiler used within the guest container (and not the native linux compiler).
+
+``` ruby
+cross_build_p = enable_config("cross-build")
+```
+
+We don't need this for `libyaml`, but the technique is useful: the cross-compile rake task signals to `extconf.rb` that it's cross-compiling by using a commandline flag that we can inspect. As an example, `extconf.rb` may need to set specific configuration options on the third-party library when cross-compiling.
+
+``` ruby
+MiniPortile.new("yaml", "0.2.5").tap do |recipe|
+  # ...
+  # configure the environment that MiniPortile will use for subshells
+  ENV.to_h.dup.tap do |env|
+    # -fPIC is necessary for linking into a shared library
+    env["CFLAGS"] = [env["CFLAGS"], "-fPIC"].join(" ")
+    env["SUBDIRS"] = "include src" # libyaml: skip tests
+
+    recipe.configure_options += env.map { |key, value| "#{key}=#{value.strip}" }
+  end
+  # ...
+end
+```
+
+The rest of the extconf changes are related to configuring libyaml at build time. We need to set the -fPIC option so we can mix static and shared libraries together. (This should probably always be set.)
+
+The "SUBDIRS" environment variable is something that's very specific to libyaml, though: it tells libyaml's autoconf build system to skip running the tests. We have to do this because although we can *generate* binaries for other platforms, we can't actually *run* them.
+
+We have one more small change we'll need to make to how the extension is required. Let's take a look at the directory structure in the packaged gem:
+
+``` text
+lib
+└── rcee
+    ├── precompiled
+    │   ├── 2.5
+    │   │   └── precompiled.so
+    │   ├── 2.6
+    │   │   └── precompiled.so
+    │   ├── 2.7
+    │   │   └── precompiled.so
+    │   ├── 3.0
+    │   │   └── precompiled.so
+    │   └── version.rb
+    └── precompiled.rb
+```
+
+You can see that we have FOUR c extensions in this gem, one for each minor version of Ruby that we support. Remember that a C extension is specific to an architecture and a version of Ruby. For example, if we're running Ruby 3.0.1, then we need to load the extension in the 3.0 directory. Let's make sure we do that.
+
+In `lib/rcee/precompiled.rb`, we'll replace the normal `require` with:
+
+``` ruby
+begin
+  # load the precompiled extension file
+  ruby_version = /(\d+\.\d+)/.match(::RUBY_VERSION)
+  require_relative "precompiled/#{ruby_version}/precompiled"
+rescue LoadError
+  # fall back to the extension compiled upon installation.
+  require "rcee/precompiled/precompiled"
+end
+```
+
+Go ahead and try it! `gem install rcee_precompiled`. If you're on windows, linux, or macos you should get a precompiled version that installs in under a second. Everyone else (hello FreeBSD people!) it'll take a few more seconds to build the vanilla gem's packaged tarball.
+
+
+## What Can Go Wrong
+
+This strategy isn't perfect. Remember what I said earlier, that a compiled C extension is specific to
+
+- the minor version of ruby (e.g., 3.0)
+- the machine architecture (e.g., x86_64)
+- the system libraries
+
+The precompiled strategy mostly takes care of the first two, but there are still edge cases for system libraries. The big gotcha is that linux libc is not the same as linux musl, and we've had to work around this a few times in Nokogiri.
+
+I'm positive that there are more edge cases that will be found as users add more platforms and as more gems start precompiling. I'm willing to bet money that you can break this by setting some Ruby compile-time flags on your system. I'm honestly surprised it works as well as it has.
+
+So the lesson here is: make sure you have an automated test pipeline that will build a gem and test it on the target platform! This takes time to set up, but it will save you time and effort in the long run.

data/Rakefile

@@ -1,9 +1,18 @@
 # frozen_string_literal: true
 
 require "bundler/gem_tasks"
+require "rubygems/package_task"
 require "rake/testtask"
+require "rake/extensiontask"
+require "rake_compiler_dock"
+
+cross_rubies = ["3.0.0", "2.7.0", "2.6.0", "2.5.0"]
+cross_platforms = ["x64-mingw32", "x86_64-linux", "x86_64-darwin", "arm64-darwin"]
+ENV["RUBY_CC_VERSION"] = cross_rubies.join(":")
 
 rcee_precompiled_spec = Bundler.load_gemspec("rcee_precompiled.gemspec")
+Gem::PackageTask.new(rcee_precompiled_spec).define #packaged_tarball version of the gem for platform=ruby
+task "package" => cross_platforms.map { |p| "gem:#{p}" } # "package" task for all the native platforms
 
 Rake::TestTask.new(:test) do |t|
   t.libs << "test"
@@ -11,19 +20,6 @@ Rake::TestTask.new(:test) do |t|
   t.test_files = FileList["test/**/*_test.rb"]
 end
 
-# "gem" task for vanilla packaged (tarball)
-require "rubygems/package_task"
-Gem::PackageTask.new(rcee_precompiled_spec).define
-
-require "rake/extensiontask"
-require "rake_compiler_dock"
-
-cross_rubies = ["3.0.0", "2.7.0", "2.6.0", "2.5.0"]
-cross_platforms = ["x64-mingw32", "x86_64-linux", "x86_64-darwin", "arm64-darwin"]
-
-ENV["RUBY_CC_VERSION"] = cross_rubies.join(":")
-
-
 Rake::ExtensionTask.new("precompiled", rcee_precompiled_spec) do |ext|
   ext.lib_dir = "lib/rcee/precompiled"
   ext.cross_compile = true
@@ -31,7 +27,7 @@ Rake::ExtensionTask.new("precompiled", rcee_precompiled_spec) do |ext|
   ext.cross_config_options << "--enable-cross-build" # so extconf.rb knows we're cross-compiling
   ext.cross_compiling do |spec|
     # remove things not needed for precompiled gems
-    spec.dependencies.reject! { |dep| dep.name == 'mini_portile2' }
+    spec.dependencies.reject! { |dep| dep.name == "mini_portile2" }
     spec.files.reject! { |file| File.fnmatch?("*.tar.gz", file) }
   end
 end
@@ -52,7 +48,7 @@ namespace "gem" do
       task "buildit" do
         # use Task#invoke because the pkg/*gem task is defined at runtime
         Rake::Task["native:#{platform}"].invoke
-        Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform).to_s}.gem"].invoke
+        Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform)}.gem"].invoke
       end
     end
   end
@@ -61,9 +57,13 @@ namespace "gem" do
   multitask "all" => [cross_platforms, "gem"].flatten
 end
 
+task default: [:clobber, :compile, :test]
 
+CLEAN.add("{ext,lib}/**/*.{o,so}", "pkg")
 CLOBBER.add("ports")
-CLEAN.add("{ext,lib}/**/*.{o,so}")
 
-task build: :compile
-task default: %i[clobber compile test]
+# when packaging the gem, if the tarball isn't cached, we need to fetch it. the easiest thing to do
+# is to run the compile phase to invoke the extconf and have mini_portile download the file for us.
+# this is wasteful and in the future I would prefer to separate mini_portile from the extconf to
+# allow us to download without compiling.
+Rake::Task["package"].prerequisites.prepend("compile")

data/lib/rcee/precompiled/version.rb

@@ -2,6 +2,6 @@
 
 module RCEE
   module Precompiled
-    VERSION = "0.1.0"
+    VERSION = "0.2.0"
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rcee_precompiled
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: x86_64-darwin
 authors:
 - Mike Dalessio
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2021-09-02 00:00:00.000000000 Z
+date: 2021-09-10 00:00:00.000000000 Z
 dependencies: []
 description: Part of a project to explain how Ruby C extensions work.
 email: