rcee_precompiled 0.1.0-x86_64-linux → 0.2.0-x86_64-linux

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 07ee801da8b64f612aee78f3c51ac5b083a62d49c68992e3a089e33eb31dadf4
4
- data.tar.gz: 733e1652eae9fd74d98fd184c562c160ee2508f694134b790384be47a0bf810d
3
+ metadata.gz: '0818480101cccae68bf8c0864bcfe3a2bcb834c6109a2ea0de8fb20b0b857839'
4
+ data.tar.gz: 2c5895c09102a334bd504bafff4ea219cff158b4a59eb7206c113ff9a706d951
5
5
  SHA512:
6
- metadata.gz: 51a35bc755af82e00cd3050ed5fbfe49cd83f03f6184eb9edb2a216e6176d2b27eba34fcdf3a8f635856e8fdc5f50345e30bd786dffef3daabdf2a0211b61815
7
- data.tar.gz: 5d601a193c724b65788728a8ecc47a3ef84073575ed06bd1df4ae73475a1d8dfbea2b20c63a3d07b5bd4a3db10e66b67cd747874d57e07feef59dfd6adab2ea4
6
+ metadata.gz: 00efa25e3e7109ea75c6e163d2ff59d6b5d55278b0f35b888cfe9befd1bd965cc4697146766ab740bd4a8942f4d3b56b7daeb33165b4fe3c9073fe58c84798da
7
+ data.tar.gz: b7d2027f2b8673254ab8ae573c641f954756d8ad7a8a649890f3bc9d5d1f0db5197e063494c5fd839d1a3021ecf0f53ae9b47b3cafa4c96344d10af0d0f6a186
data/README.md CHANGED
@@ -1,2 +1,165 @@
1
+ # RCEE::Precompiled
1
2
 
2
3
  This gem is part of the Ruby C Extensions Explained project at https://github.com/flavorjones/ruby-c-extensions-explained
4
+
5
+ ## Summary
6
+
7
+ Installation time is an important aspect of developer happiness, and the Packaged strategies can make your users very unhappy.
8
+
9
+ One way to speed up installation time is to precompile the extension, so that during `gem install` files are simply being unpacked, and the entire configure/compile/link process is avoided completely. This can be tricky to set up, and requires that you test across all the target platforms; but makes for much happier users.
10
+
11
+ Note, however, that it's necessary to _also_ ship the vanilla (platform=ruby) gem as a fallback for platforms you haven't pre-compiled for.
12
+
13
+ In the nine months since Nokogiri v1.11 started shipping precompiled native gems, we've seen over 45 million gem installations, and essentially zero support issues have been opened. Twitter complaints dropped to near zero. Page views of the Nokogiri installation tutorial dropped by half.
14
+
15
+ Precompiled Nokogiri gems have been an unmitigated success and have made both users and maintainers happy.
16
+
17
+
18
+ ## Details
19
+
20
+ An important tool we rely on for precompiling is [`rake-compiler/rake-compiler-dock`](https://github.com/rake-compiler/rake-compiler-dock), maintained by Lars Kanis. `rake-compiler-dock` is a Docker-based build environment that uses `rake-compiler` to _cross compile_ for all of these major platforms.
21
+
22
+ What this means is, I can run my normal gem build process in a docker container on my Linux machine, and it will build gems that I run on Windows and MacOS.
23
+
24
+ This is really powerful stuff, and once we assume that we can cross-compile reliably, the remaining problems boil down to modifying how we build the gemfile and making sure we test gems adequately on the target platforms.
25
+
26
+ First, we need to add some features to our `Rake::ExtensionTask` in `Rakefile`:
27
+
28
+ ``` ruby
29
+ cross_rubies = ["3.0.0", "2.7.0", "2.6.0", "2.5.0"]
30
+ cross_platforms = ["x64-mingw32", "x86_64-linux", "x86_64-darwin", "arm64-darwin"]
31
+ ENV["RUBY_CC_VERSION"] = cross_rubies.join(":")
32
+
33
+ Rake::ExtensionTask.new("precompiled", rcee_precompiled_spec) do |ext|
34
+ ext.lib_dir = "lib/rcee/precompiled"
35
+ ext.cross_compile = true
36
+ ext.cross_platform = cross_platforms
37
+ ext.cross_config_options << "--enable-cross-build" # so extconf.rb knows we're cross-compiling
38
+ ext.cross_compiling do |spec|
39
+ # remove things not needed for precompiled gems
40
+ spec.dependencies.reject! { |dep| dep.name == "mini_portile2" }
41
+ spec.files.reject! { |file| File.fnmatch?("*.tar.gz", file) }
42
+ end
43
+ end
44
+ ```
45
+
46
+ This does the following:
47
+
48
+ - set up some local variables to indicate what ruby versions and which platforms we will build for
49
+ - set an environment variable to let rake-compiler know which ruby versions we'll build
50
+ - tell the extension task to turn on cross-compiling features, including additional rake tasks
51
+ - signal to our `extconf.rb` when we're cross-compiling (in case its behavior needs to change)
52
+ - finally, in a block that is only run when cross-compiling, we modify the gemspec to remove things we don't need in native gems:
53
+ - the tarball
54
+ - the dependency on `mini_portile`
55
+
56
+ Next we need some new rake tasks:
57
+
58
+ ``` ruby
59
+ namespace "gem" do
60
+ cross_platforms.each do |platform|
61
+ desc "build native gem for #{platform}"
62
+ task platform do
63
+ RakeCompilerDock.sh(<<~EOF, platform: platform)
64
+ gem install bundler --no-document &&
65
+ bundle &&
66
+ bundle exec rake gem:#{platform}:buildit
67
+ EOF
68
+ end
69
+
70
+ namespace platform do
71
+ # this runs in the rake-compiler-dock docker container
72
+ task "buildit" do
73
+ # use Task#invoke because the pkg/*gem task is defined at runtime
74
+ Rake::Task["native:#{platform}"].invoke
75
+ Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform)}.gem"].invoke
76
+ end
77
+ end
78
+ end
79
+ end
80
+ ```
81
+
82
+ The top task (or, really, _set_ of tasks) runs on the host system, and invokes the lower task within the appropriate docker container to cross-compile for that platform. So the bottom task is doing most of the work, and it's doing it inside a guest container. Please note that the worker is both building the extension *and* packaging the gem according to the gemspec that was just modified by our extensiontask cross-compiling block.
83
+
84
+ Changes to the `extconf.rb`:
85
+
86
+ ``` ruby
87
+ ENV["CC"] = RbConfig::CONFIG["CC"]
88
+ ```
89
+
90
+ This makes sure that the cross-compiler is the compiler used within the guest container (and not the native linux compiler).
91
+
92
+ ``` ruby
93
+ cross_build_p = enable_config("cross-build")
94
+ ```
95
+
96
+ We don't need this for `libyaml`, but the technique is useful: the cross-compile rake task signals to `extconf.rb` that it's cross-compiling by using a commandline flag that we can inspect. As an example, `extconf.rb` may need to set specific configuration options on the third-party library when cross-compiling.
97
+
98
+ ``` ruby
99
+ MiniPortile.new("yaml", "0.2.5").tap do |recipe|
100
+ # ...
101
+ # configure the environment that MiniPortile will use for subshells
102
+ ENV.to_h.dup.tap do |env|
103
+ # -fPIC is necessary for linking into a shared library
104
+ env["CFLAGS"] = [env["CFLAGS"], "-fPIC"].join(" ")
105
+ env["SUBDIRS"] = "include src" # libyaml: skip tests
106
+
107
+ recipe.configure_options += env.map { |key, value| "#{key}=#{value.strip}" }
108
+ end
109
+ # ...
110
+ end
111
+ ```
112
+
113
+ The rest of the extconf changes are related to configuring libyaml at build time. We need to set the -fPIC option so we can mix static and shared libraries together. (This should probably always be set.)
114
+
115
+ The "SUBDIRS" environment variable is something that's very specific to libyaml, though: it tells libyaml's autoconf build system to skip running the tests. We have to do this because although we can *generate* binaries for other platforms, we can't actually *run* them.
116
+
117
+ We have one more small change we'll need to make to how the extension is required. Let's take a look at the directory structure in the packaged gem:
118
+
119
+ ``` text
120
+ lib
121
+ └── rcee
122
+ ├── precompiled
123
+ │   ├── 2.5
124
+ │   │   └── precompiled.so
125
+ │   ├── 2.6
126
+ │   │   └── precompiled.so
127
+ │   ├── 2.7
128
+ │   │   └── precompiled.so
129
+ │   ├── 3.0
130
+ │   │   └── precompiled.so
131
+ │   └── version.rb
132
+ └── precompiled.rb
133
+ ```
134
+
135
+ You can see that we have FOUR c extensions in this gem, one for each minor version of Ruby that we support. Remember that a C extension is specific to an architecture and a version of Ruby. For example, if we're running Ruby 3.0.1, then we need to load the extension in the 3.0 directory. Let's make sure we do that.
136
+
137
+ In `lib/rcee/precompiled.rb`, we'll replace the normal `require` with:
138
+
139
+ ``` ruby
140
+ begin
141
+ # load the precompiled extension file
142
+ ruby_version = /(\d+\.\d+)/.match(::RUBY_VERSION)
143
+ require_relative "precompiled/#{ruby_version}/precompiled"
144
+ rescue LoadError
145
+ # fall back to the extension compiled upon installation.
146
+ require "rcee/precompiled/precompiled"
147
+ end
148
+ ```
149
+
150
+ Go ahead and try it! `gem install rcee_precompiled`. If you're on windows, linux, or macos you should get a precompiled version that installs in under a second. Everyone else (hello FreeBSD people!) it'll take a few more seconds to build the vanilla gem's packaged tarball.
151
+
152
+
153
+ ## What Can Go Wrong
154
+
155
+ This strategy isn't perfect. Remember what I said earlier, that a compiled C extension is specific to
156
+
157
+ - the minor version of ruby (e.g., 3.0)
158
+ - the machine architecture (e.g., x86_64)
159
+ - the system libraries
160
+
161
+ The precompiled strategy mostly takes care of the first two, but there are still edge cases for system libraries. The big gotcha is that linux libc is not the same as linux musl, and we've had to work around this a few times in Nokogiri.
162
+
163
+ I'm positive that there are more edge cases that will be found as users add more platforms and as more gems start precompiling. I'm willing to bet money that you can break this by setting some Ruby compile-time flags on your system. I'm honestly surprised it works as well as it has.
164
+
165
+ So the lesson here is: make sure you have an automated test pipeline that will build a gem and test it on the target platform! This takes time to set up, but it will save you time and effort in the long run.
data/Rakefile CHANGED
@@ -1,9 +1,18 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require "bundler/gem_tasks"
4
+ require "rubygems/package_task"
4
5
  require "rake/testtask"
6
+ require "rake/extensiontask"
7
+ require "rake_compiler_dock"
8
+
9
+ cross_rubies = ["3.0.0", "2.7.0", "2.6.0", "2.5.0"]
10
+ cross_platforms = ["x64-mingw32", "x86_64-linux", "x86_64-darwin", "arm64-darwin"]
11
+ ENV["RUBY_CC_VERSION"] = cross_rubies.join(":")
5
12
 
6
13
  rcee_precompiled_spec = Bundler.load_gemspec("rcee_precompiled.gemspec")
14
+ Gem::PackageTask.new(rcee_precompiled_spec).define #packaged_tarball version of the gem for platform=ruby
15
+ task "package" => cross_platforms.map { |p| "gem:#{p}" } # "package" task for all the native platforms
7
16
 
8
17
  Rake::TestTask.new(:test) do |t|
9
18
  t.libs << "test"
@@ -11,19 +20,6 @@ Rake::TestTask.new(:test) do |t|
11
20
  t.test_files = FileList["test/**/*_test.rb"]
12
21
  end
13
22
 
14
- # "gem" task for vanilla packaged (tarball)
15
- require "rubygems/package_task"
16
- Gem::PackageTask.new(rcee_precompiled_spec).define
17
-
18
- require "rake/extensiontask"
19
- require "rake_compiler_dock"
20
-
21
- cross_rubies = ["3.0.0", "2.7.0", "2.6.0", "2.5.0"]
22
- cross_platforms = ["x64-mingw32", "x86_64-linux", "x86_64-darwin", "arm64-darwin"]
23
-
24
- ENV["RUBY_CC_VERSION"] = cross_rubies.join(":")
25
-
26
-
27
23
  Rake::ExtensionTask.new("precompiled", rcee_precompiled_spec) do |ext|
28
24
  ext.lib_dir = "lib/rcee/precompiled"
29
25
  ext.cross_compile = true
@@ -31,7 +27,7 @@ Rake::ExtensionTask.new("precompiled", rcee_precompiled_spec) do |ext|
31
27
  ext.cross_config_options << "--enable-cross-build" # so extconf.rb knows we're cross-compiling
32
28
  ext.cross_compiling do |spec|
33
29
  # remove things not needed for precompiled gems
34
- spec.dependencies.reject! { |dep| dep.name == 'mini_portile2' }
30
+ spec.dependencies.reject! { |dep| dep.name == "mini_portile2" }
35
31
  spec.files.reject! { |file| File.fnmatch?("*.tar.gz", file) }
36
32
  end
37
33
  end
@@ -52,7 +48,7 @@ namespace "gem" do
52
48
  task "buildit" do
53
49
  # use Task#invoke because the pkg/*gem task is defined at runtime
54
50
  Rake::Task["native:#{platform}"].invoke
55
- Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform).to_s}.gem"].invoke
51
+ Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform)}.gem"].invoke
56
52
  end
57
53
  end
58
54
  end
@@ -61,9 +57,13 @@ namespace "gem" do
61
57
  multitask "all" => [cross_platforms, "gem"].flatten
62
58
  end
63
59
 
60
+ task default: [:clobber, :compile, :test]
64
61
 
62
+ CLEAN.add("{ext,lib}/**/*.{o,so}", "pkg")
65
63
  CLOBBER.add("ports")
66
- CLEAN.add("{ext,lib}/**/*.{o,so}")
67
64
 
68
- task build: :compile
69
- task default: %i[clobber compile test]
65
+ # when packaging the gem, if the tarball isn't cached, we need to fetch it. the easiest thing to do
66
+ # is to run the compile phase to invoke the extconf and have mini_portile download the file for us.
67
+ # this is wasteful and in the future I would prefer to separate mini_portile from the extconf to
68
+ # allow us to download without compiling.
69
+ Rake::Task["package"].prerequisites.prepend("compile")
@@ -2,6 +2,6 @@
2
2
 
3
3
  module RCEE
4
4
  module Precompiled
5
- VERSION = "0.1.0"
5
+ VERSION = "0.2.0"
6
6
  end
7
7
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: rcee_precompiled
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.2.0
5
5
  platform: x86_64-linux
6
6
  authors:
7
7
  - Mike Dalessio
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2021-09-02 00:00:00.000000000 Z
11
+ date: 2021-09-10 00:00:00.000000000 Z
12
12
  dependencies: []
13
13
  description: Part of a project to explain how Ruby C extensions work.
14
14
  email: