cibuildgem 0.2.2 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9a435af5eac0a64303ce3cc170330faa6d10c7baef2af54c7c7df99713766fd4
-  data.tar.gz: b2e59e31200a2a3490a15d033730156da13e1cf42a54ad6e175d5f8475717df3
+  metadata.gz: 2d98ae7e4d9c1c2bd075ee978c39d292fa995d5162d018f399e217e58ac90501
+  data.tar.gz: e497b8a81dc7b395b20f7c9f94db94a937b4c55de4e331fbccd7e172d3cf3807
 SHA512:
-  metadata.gz: cf049b9d6bcbddaca238f847077dd3b3bf335df62674236b4c36a44a68931a315f32b344bc100d3871cc8fc5161c101c29cfc2e2b2195870aa18d1268cfec5ab
-  data.tar.gz: c7782b3afcb6f1c8a871c18f0199fb95fe9e4b77b74ab8e5b6aa2e1c75b872bd4a994077a0460bbbe40c9b4e93cc1633b6c58e11b48e4109538b65ffc22a9fe5
+  metadata.gz: 5255e73551d7a8237dce25300b64f00845f196348443684c7c4fc9fc42e3b57818d3bf30eee5989cd84fac6e610f96dfdf75db4353b0ed4d54c141f3eb124abc
+  data.tar.gz: f15c1eab3cbc400375a9e561a452f5ad70f10e09b068433a4eb4589a10dac3cedf37b6e20f6b49b38ab286a65847f8b1371c853b78a483ce4e19abca7568368e

data/README.md

@@ -22,49 +22,22 @@ As noted by [@flavorjones](https://github.com/flavorjones), this toolchain works
 
 ## 💻 cibuildgem
 
-> [!NOTE]
-> cibuildgem is for now not able to compile projects that needs to link on external libraries. Unless the project vendors those libraries or uses [mini_portile](https://github.com/flavorjones/mini_portile).
-
-### How to use it
-
-While cibuildgem is generally **not** meant to be used locally, it provides a command to generate the right GitHub workflow for your project:
-
-1. Install cibuildgem: `gem install cibuildgem`
-2. Generate the workflow: `cd` in your gem's folder and run `cibuildgem ci_template`
-3. Commit the `.github/workflows/cibuildgem.yaml` file.
-
-### Triggering the workflow
-
-Once pushed in your repository **default** branch, the workflow that we just generated is actionable manually on the GitHub action page. It will run in sequence:
-
-1. Compile the gem on the target platform (defaults to MacOS ARM, MacOS Intel, Windows, Ubuntu 24)
-2. Once the compilation succeeds on all platform, it proceeds to run the test suite on the target platform. This will trigger many CI steps as the testing matrix is big.
-3. Once the test suite passes for all platforms and all Ruby versions the gem is compatible with, the action proceeds to installing the gem we just packaged. This step ensure that the gem is actually installable.
-4. [OPTIONAL] When trigering the workflow manually, you can tick the box to automatically release the gems that were packaged. This works using the RubyGems trusted publisher feature (documentation to write later). If you do no want the tool to make the release, you can download all the GitHub artifacts that were uploaded. It will contain all the gems with precompiled binaries in the `pkg` folder. You are free to download them locally and release them yourself from your machine.
+Head to the [documentation Wiki](https://github.com/Shopify/cibuildgem/wiki) to setup and configure cibuildgem for your gem.
 
+## Working examples
 
-### Changes to make in your gem to support precompiled binaries
+Here are some working examples on gem that have setup cibuildgem
 
-Due to the RubyGems specification, we can't release a gem with precompiled binaries for a specific Ruby version. Because the Ruby ABI is incompatible between minor versions, Rake Compiler (the tool underneath cibuildgem), compiles the binary for every minor Ruby versions your gem supports. All those binaries will be packaged in the gem (called a fat gem) in different folder such as `3.0/date.so`, `3.1/date.so` etc...
-At runtime, your gem need to require the right binary based on the running ruby version.
+| Name      | Example Run | Published Gem |
+|-----------|-------------|---------------|
+| [Rubydex](https://github.com/shopify/rubydex) | [CI Run](https://github.com/Shopify/rubydex/actions/runs/21880161517) | [Gem](https://rubygems.org/gems/rubydex/versions/0.1.0.beta4-arm64-darwin) |
+| [Heap Profiler](https://github.com/shopify/heap-profiler) | [CI Run](https://github.com/Shopify/heap-profiler/actions/runs/20996043558) | [Gem](https://rubygems.org/gems/heap-profiler/versions/0.8.0.rc1-x86_64-linux) |
+| [djb2](https://github.com/Shopify/djb2) | [CI Run](https://github.com/Shopify/djb2/actions/runs/22188688549) | [Gem](https://rubygems.org/gems/djb2/versions/0.1.1-x86_64-linux) |
+| [Blake3](https://github.com/Shopify/blake3-rb) | [CI Run](https://github.com/Shopify/blake3-rb/actions/runs/21253662535) | [Gem](https://rubygems.org/gems/blake3-rb/versions/1.5.6.rc1-x86_64-linux) |
+| [Raindrops (fork)](https://github.com/Edouard-chin/raindrops) | [CI Run](https://github.com/Edouard-chin/raindrops/actions/runs/22045845221) | [Gem](https://rubygems.org/gems/precompiled-raindrops) |
+| [Stack Frames](https://github.com/Shopify/stack_frames) | [CI Run](https://github.com/Shopify/stack_frames/actions/runs/19969899178) | [Gem](https://rubygems.org/gems/stack_frames/versions/0.1.4-x86_64-linux) |
 
-```ruby
-# Before
-
-require 'date_core.so'
-
-# After
-
-begin
-  ruby_version = /(\d+\.\d+)/.match(::RUBY_VERSION)
-  require "#{ruby_version}/date_core"
-rescue LoadError
-  # It's important to leave for users that can not or don't want to use the gem with precompiled binaries.
-  require "date_core"
-end
-```
-
-### Supported platforms/Ruby versions
+## Supported platforms/Ruby versions
 
 |         | MacOS Intel  | MacOS ARM | Windows x64 UCRT | Linux GNU x86_64|Linux AARCH64 |
 |---------|------------- | --------- | ------------|-----------------|-----------------|
@@ -72,7 +45,4 @@ end
 | Ruby 3.2| 🟢           | 🟢        | 🟢          | 🟢             | 🟢             |
 | Ruby 3.3| 🟢           | 🟢        | 🟢          | 🟢             | 🟢             |
 | Ruby 3.4| 🟢           | 🟢        | 🟢          | 🟢             | 🟢             |
-
-## 🧪 Development
-
-If you'd like to run a end-to-end test, the `date` gem is vendored in this project. You can trigger a manual run to do the whole compile, test, install dance from the GitHub action menu.
+| Ruby 4.0| 🟢           | 🟢        | 🟠 (not tested)| 🟢             | 🟢             |

data/lib/cibuildgem/cli.rb

@@ -3,6 +3,7 @@
 require "thor"
 require "rake/extensiontask"
 require "prism"
+require "open3"
 
 module Cibuildgem
   class CLI < Thor
@@ -10,6 +11,10 @@ module Cibuildgem
 
     source_root(File.expand_path("templates", __dir__))
 
+    # RubyGems already restricts a gem's name, version and platform to these characters, so every filename
+    # `cibuildgem package` can legitimately produce matches. Anything else was not produced by the package task.
+    RELEASABLE_GEM_FILENAME = /\A[A-Za-z0-9._-]+\.gem\z/
+
     class << self
       def exit_on_failure?
         true
@@ -31,7 +36,7 @@ module Cibuildgem
       run_rake_tasks!("cibuildgem:setup", :compile)
     end
 
-    desc "package", "Compile and package a 'fat gem'.", hide: true
+    desc "package", "Compile and package a 'multi-ABI gem'.", hide: true
     long_desc <<~MSG
       This command should normally run on CI, using the cibuildgem workflow. It will not work locally unless
       the environment is properly setup.
@@ -41,13 +46,20 @@ module Cibuildgem
       - A gem without precompiled binary (Ruby platform).
 
       The gem with precompiled binaries will be packaged with multiple binaries compatible for different
-      Ruby ABI (depending on what Ruby version the gem supports).
+      Ruby ABI (depending on what Ruby version the gem supports). If `--include-single-abi` is passed,
+      cibuildgem will also compile a set of single-ABI gems per Ruby version supported by the gem.
     MSG
     method_option "gemspec", type: "string", required: false, desc: "The gemspec to use. Defaults to the gemspec from the current working directory."
+    method_option "include-single-abi", type: :boolean, required: false, default: false, desc: "Whether to also compile a set of single-ABI gems. Defaults to false."
     def package
       ENV["RUBY_CC_VERSION"] ||= compilation_task.ruby_cc_version
-
-      run_rake_tasks!("cibuildgem:setup", :cross, :native, :gem)
+      run_rake_tasks!(
+        "cibuildgem:setup",
+        :cross,
+        :native,
+        :gem,
+        env: { "CIBUILDGEM_CONTENT_ADDRESSABLE" => options["include-single-abi"] == true ? "true" : "false" },
+      )
     end
 
     desc "test", "Run the test suites of the target gem"
@@ -112,14 +124,25 @@ module Cibuildgem
         pathname = Pathname(file)
         next if pathname.directory? || pathname.extname != ".gem"
 
-        Kernel.system("gem push #{file}", exception: true)
+        verify_releasable_filename!(pathname)
+
+        out, status = Open3.capture2e("gem", "push", file)
+        next if status.success?
+
+        if out =~ /Repushing of gem versions is not allowed/
+          puts "Gem #{file} already exists on RubyGems.org, skipping..."
+
+          next
+        end
+
+        raise out
       end
     end
 
     desc "print_ruby_cc_version", "Output the cross compile ruby version needed for the gem. For internal usage", hide: true
     method_option "gemspec", type: "string", required: false, desc: "The gemspec to use. If the option is not passed, a gemspec file from the current working directory will be used."
     def print_ruby_cc_version
-      print(compilation_task.ruby_cc_version)
+      print(ENV["RUBY_CC_VERSION"] || compilation_task.ruby_cc_version)
     end
 
     desc "normalized_platform", "The platform name for compilation purposes", hide: true
@@ -129,8 +152,20 @@ module Cibuildgem
 
     private
 
-    def run_rake_tasks!(*tasks)
-      all_tasks = tasks.join(" ")
+    def verify_releasable_filename!(pathname)
+      basename = pathname.basename.to_s
+      return if RELEASABLE_GEM_FILENAME.match?(basename)
+
+      raise(<<~MSG)
+        Refusing to publish #{basename.inspect}.
+
+        A gem filename may only contain letters, digits, dots, dashes and underscores, so this file was not
+        produced by `cibuildgem package`. Artifacts reach the release job from the compile job, which runs
+        without publishing credentials, so an unexpected filename is treated as tampering rather than a gem.
+      MSG
+    end
+
+    def run_rake_tasks!(*tasks, env: {})
       rakelibdir = [File.expand_path("tasks", __dir__), "rakelib"].join(File::PATH_SEPARATOR)
       rake_compiler_path = Gem.loaded_specs["rake-compiler"].full_require_paths
       rake_specs = Gem.loaded_specs["rake"]
@@ -141,8 +176,15 @@ module Cibuildgem
       patch = File.expand_path("../extension_patch.rb", __FILE__)
 
       system(
-        { "RUBYLIB" => load_paths },
-        "bundle exec #{RbConfig.ruby} #{rake_executable} #{all_tasks} -R#{rakelibdir} -r #{patch}",
+        { "RUBYLIB" => load_paths }.merge(env),
+        "bundle",
+        "exec",
+        RbConfig.ruby,
+        rake_executable,
+        *tasks.map(&:to_s),
+        "-R#{rakelibdir}",
+        "-r",
+        patch,
         exception: true,
       )
     end

data/lib/cibuildgem/compilation_tasks.rb

@@ -7,18 +7,21 @@ require_relative "create_makefile_finder"
 
 module Cibuildgem
   class CompilationTasks
-    attr_reader :gemspec, :native, :create_packaging_task, :extension_task
+    attr_reader :gemspec, :native, :create_packaging_task, :extension_task, :content_addressable
 
-    def initialize(create_packaging_task = false, gemspec = nil)
+    def initialize(
+      create_packaging_task = false,
+      gemspec = nil,
+      content_addressable: ENV["CIBUILDGEM_CONTENT_ADDRESSABLE"] == "true"
+    )
       @gemspec = Bundler.load_gemspec(gemspec || find_gemspec)
       verify_gemspec!
 
       @create_packaging_task = create_packaging_task
+      @content_addressable = content_addressable
     end
 
     def setup
-      Rake::ExtensionTask.enable!
-
       gemspec.extensions.each do |path|
         binary_name = parse_extconf(path)
         define_task(path, binary_name)
@@ -54,20 +57,35 @@ module Cibuildgem
     end
 
     def define_task(path, binary_name)
-      @extension_task = Rake::ExtensionTask.new do |ext|
-        ext.name = File.basename(binary_name)
-        ext.config_script = File.basename(path)
-        ext.ext_dir = File.dirname(path)
-        ext.lib_dir = binary_lib_dir(binary_name) if binary_lib_dir(binary_name)
-        ext.gem_spec = gemspec
-        ext.cross_platform = normalized_platform
-        ext.cross_compile = true
-        ext.no_native = true
+      @extension_task = Rake::ExtensionTask.current || Rake::ExtensionTask.new
+
+      @extension_task.name = File.basename(binary_name)
+      @extension_task.config_script = File.basename(path)
+      @extension_task.ext_dir = File.dirname(path)
+      @extension_task.lib_dir = binary_lib_dir(binary_name) if binary_lib_dir(binary_name)
+      @extension_task.gem_spec = gemspec
+      @extension_task.cross_platform = normalized_platform
+      @extension_task.cross_compile = true
+      @extension_task.no_native = true
+      if content_addressable
+        if @extension_task.respond_to?(:content_addressable=)
+          @extension_task.content_addressable = true
+        else
+          warn("WARNING: content-addressable packaging was requested, but rake-compiler #{rake_compiler_version} " \
+            "does not support it. Upgrade rake-compiler to >= 1.3.2 to build single-ABI gems.")
+        end
       end
 
+      Rake::ExtensionTask.enable!
+      @extension_task.define
+
       disable_shared unless Gem.win_platform?
     end
 
+    def rake_compiler_version
+      Gem.loaded_specs["rake-compiler"]&.version&.to_s || "unknown"
+    end
+
     def binary_lib_dir(binary_name)
       dir = File.dirname(binary_name)
       return if dir == "."

data/lib/cibuildgem/extension_patch.rb

@@ -7,7 +7,7 @@ module Cibuildgem
     class << self
       def prepended(mod)
         class << mod
-          attr_accessor :enabled
+          attr_accessor :enabled, :current
 
           def enable!
             @enabled = true
@@ -16,6 +16,12 @@ module Cibuildgem
       end
     end
 
+    def initialize(*)
+      super
+
+      self.class.current = self
+    end
+
     def define
       super if self.class.enabled
     end

data/lib/cibuildgem/tasks/wrapper.rake

@@ -12,11 +12,9 @@ task "cibuildgem:setup" do
 end
 
 task "copy:stage:lib" do
-  version = RUBY_VERSION.match(/(\d\.\d)/)[1]
-  dest = File.join(task.extension_task.lib_dir, version)
-  src = File.join("tmp", task.extension_task.cross_platform, "stage", dest)
+  src = File.join("tmp", task.extension_task.cross_platform, "stage")
 
-  cp_r(src, dest, remove_destination: true)
+  cp_r(Dir.glob("#{src}/*"), ".", remove_destination: true)
 end
 
 unless Rake::Task.task_defined?(:test)

data/lib/cibuildgem/templates/github/workflows/cibuildgem.yaml.tt

@@ -7,6 +7,8 @@ on:
         required: false
         type: boolean
         default: false
+env:
+  CIBUILDGEM: 1
 jobs:
   compile:
     timeout-minutes: 20

data/lib/cibuildgem/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Cibuildgem
-  VERSION = "0.2.2"
+  VERSION = "0.4.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: cibuildgem
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.4.0
 platform: ruby
 authors:
 - Shopify