rake-compiler-dock 0.7.2 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +4 -0
- data/.github/workflows/ci.yml +168 -0
- data/.gitignore +14 -0
- data/Dockerfile.jruby +12 -7
- data/Dockerfile.mri.erb +271 -0
- data/History.md +63 -0
- data/README.md +138 -27
- data/Rakefile +71 -4
- data/{lib/rake_compiler_dock → build}/gem_helper.rb +0 -15
- data/build/parallel_docker_build.rb +106 -0
- data/build/patches/{ruby-2.5.3 → ruby-2.5.9}/no_sendfile.patch +1 -1
- data/build/patches/ruby-3.1.0/no_sendfile.patch +24 -0
- data/build/patches2/rake-compiler-1.1.6/0004-Enable-build-of-static-libruby.patch +28 -0
- data/build/rcd-env.sh +6 -0
- data/build/runas +1 -6
- data/build/sudoers +1 -1
- data/lib/rake_compiler_dock/docker_check.rb +12 -6
- data/lib/rake_compiler_dock/starter.rb +69 -53
- data/lib/rake_compiler_dock/version.rb +2 -2
- data/lib/rake_compiler_dock.rb +10 -0
- data/mingw64-ucrt/Dockerfile +62 -0
- data/mingw64-ucrt/binutils-mingw-w64-ignore-check-errors.patch +13 -0
- data/mingw64-ucrt/gcc-mingw-w64-only-c-c++.patch +13 -0
- data/mingw64-ucrt/mingw-w64-enable-ucrt.patch +22 -0
- data/rake-compiler-dock.gemspec +2 -2
- data/test/env/Dockerfile.alpine +17 -0
- data/test/env/Dockerfile.centos +17 -0
- data/test/env/Dockerfile.debian +24 -0
- data/test/rcd_test/Gemfile +11 -0
- data/test/rcd_test/Rakefile +74 -0
- data/test/rcd_test/ext/java/RcdTestExtService.java +19 -0
- data/test/rcd_test/ext/java/RubyRcdTest.java +16 -0
- data/test/rcd_test/ext/mri/extconf.rb +20 -0
- data/test/rcd_test/ext/mri/rcd_test_ext.c +35 -0
- data/test/rcd_test/ext/mri/rcd_test_ext.h +6 -0
- data/test/rcd_test/lib/rcd_test.rb +6 -0
- data/test/rcd_test/rcd_test.gemspec +27 -0
- data/test/rcd_test/test/test_basic.rb +22 -0
- data/test/test_environment_variables.rb +56 -31
- data/test/test_parallel_docker_build.rb +70 -0
- data.tar.gz.sig +1 -0
- metadata +83 -14
- metadata.gz.sig +0 -0
- data/Dockerfile.mri +0 -122
- data/build/patches/rake-compiler-1.0.7/enable-static.diff +0 -13
data/README.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# rake-compiler-dock
|
|
2
2
|
|
|
3
|
-
**Easy to use and reliable cross compiler environment for building Windows, Linux and JRuby binary gems.**
|
|
3
|
+
**Easy to use and reliable cross compiler environment for building Windows, Linux, Mac and JRuby binary gems.**
|
|
4
4
|
|
|
5
|
-
It provides cross compilers and Ruby environments for 2.
|
|
5
|
+
It provides cross compilers and Ruby environments for 2.4 and newer versions of the [RubyInstaller](http://rubyinstaller.org/) and Linux runtime environments.
|
|
6
6
|
They are prepared for use with [rake-compiler](https://github.com/rake-compiler/rake-compiler).
|
|
7
7
|
It is used by [many gems with C or JRuby extentions](https://github.com/rake-compiler/rake-compiler-dock/wiki/Projects-using-rake-compiler-dock).
|
|
8
8
|
|
|
@@ -12,11 +12,7 @@ It is also more reliable, since the underlying docker images are versioned and i
|
|
|
12
12
|
|
|
13
13
|
## Installation
|
|
14
14
|
|
|
15
|
-
Install docker
|
|
16
|
-
|
|
17
|
-
$ sudo apt-get install docker.io
|
|
18
|
-
|
|
19
|
-
... or install [docker-toolbox for Windows and OSX](https://www.docker.com/docker-toolbox) or boot2docker on [Windows](https://github.com/boot2docker/windows-installer/releases) or [OS X](https://github.com/boot2docker/osx-installer/releases) .
|
|
15
|
+
Install docker [following the instructions on the docker website](https://docs.docker.com/engine/install/) ... or install [docker-toolbox for Windows and OSX](https://github.com/docker/toolbox/releases) or boot2docker on [Windows](https://github.com/boot2docker/windows-installer/releases) or [OS X](https://github.com/boot2docker/osx-installer/releases) .
|
|
20
16
|
|
|
21
17
|
Install rake-compiler-dock as a gem. The docker image is downloaded later on demand:
|
|
22
18
|
|
|
@@ -34,16 +30,23 @@ Rake-compiler-dock provides the necessary tools to build Ruby extensions for Win
|
|
|
34
30
|
It is intended to be used in conjunction with [rake-compiler's](https://github.com/rake-compiler/rake-compiler) cross build capability.
|
|
35
31
|
Your Rakefile should enable cross compilation like so:
|
|
36
32
|
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
33
|
+
```ruby
|
|
34
|
+
exttask = Rake::ExtensionTask.new('my_extension', my_gem_spec) do |ext|
|
|
35
|
+
ext.cross_compile = true
|
|
36
|
+
ext.cross_platform = %w[x86-mingw32 x64-mingw-ucrt x64-mingw32 x86-linux x86_64-linux x86_64-darwin arm64-darwin]
|
|
37
|
+
end
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
See below, how to invoke cross builds in your Rakefile.
|
|
41
41
|
|
|
42
|
-
|
|
42
|
+
Additionally it may also be used to build ffi based binary gems like [libusb](https://github.com/larskanis/libusb), but currently doesn't provide any additional build helpers for this use case, beyond docker invocation and cross compilers.
|
|
43
43
|
|
|
44
44
|
### Interactive Usage
|
|
45
45
|
|
|
46
46
|
Rake-compiler-dock offers the shell command `rake-compiler-dock` and a [ruby API](http://www.rubydoc.info/gems/rake-compiler-dock/RakeCompilerDock) for issuing commands within the docker image, described below.
|
|
47
|
+
There are dedicated images for `x86-mingw32`, `x64-mingw-ucrt`, `x64-mingw32`, `x86-linux`, `x86_64-linux`, `x86_64-darwin`, `arm64-darwin` and `jruby` targets.
|
|
48
|
+
The images contain all supported cross ruby versions, with the exception of `x64-mingw32`, which has versions before 3.1 only, and `x64-mingw-ucrt`, which has only ruby-3.1+.
|
|
49
|
+
This is to match the [changed platform of RubyInstaller-3.1](https://rubyinstaller.org/2021/12/31/rubyinstaller-3.1.0-1-released.html).
|
|
47
50
|
|
|
48
51
|
`rake-compiler-dock` without arguments starts an interactive shell session.
|
|
49
52
|
This is best suited to try out and debug a build.
|
|
@@ -55,16 +58,22 @@ But note, that all other changes to the file system of the container are dropped
|
|
|
55
58
|
All commands are executed with the same user and group of the host.
|
|
56
59
|
This is done by copying user account data into the container and sudo to it.
|
|
57
60
|
|
|
58
|
-
To build x86
|
|
61
|
+
To build x86 Windows and x86_64 Linux binary gems interactively, it can be called like this:
|
|
59
62
|
|
|
60
63
|
user@host:$ cd your-gem-dir/
|
|
61
|
-
user@host:$ rake-compiler-dock # this enters a container with an interactive shell
|
|
64
|
+
user@host:$ rake-compiler-dock # this enters a container with an interactive shell for x86 Windows (default)
|
|
62
65
|
user@5b53794ada92:$ bundle
|
|
63
66
|
user@5b53794ada92:$ rake cross native gem
|
|
64
67
|
user@5b53794ada92:$ exit
|
|
65
68
|
user@host:$ ls pkg/*.gem
|
|
66
|
-
your-gem-1.0.0.gem your-gem-1.0.0-
|
|
67
|
-
|
|
69
|
+
your-gem-1.0.0.gem your-gem-1.0.0-x86-mingw32.gem
|
|
70
|
+
|
|
71
|
+
user@host:$ RCD_PLATFORM=x86_64-linux rake-compiler-dock # this enters a container for amd64 Linux target
|
|
72
|
+
user@adc55b2b92a9:$ bundle
|
|
73
|
+
user@adc55b2b92a9:$ rake cross native gem
|
|
74
|
+
user@adc55b2b92a9:$ exit
|
|
75
|
+
user@host:$ ls pkg/*.gem
|
|
76
|
+
your-gem-1.0.0.gem your-gem-1.0.0-x86_64-linux.gem
|
|
68
77
|
|
|
69
78
|
Or non-interactive:
|
|
70
79
|
|
|
@@ -72,13 +81,64 @@ Or non-interactive:
|
|
|
72
81
|
|
|
73
82
|
The environment variable `RUBY_CC_VERSION` is predefined as described [below](#environment-variables).
|
|
74
83
|
|
|
75
|
-
If necessary, additional software
|
|
76
|
-
This is local to the running session, only
|
|
84
|
+
If necessary, additional software can be installed, prior to the build command.
|
|
85
|
+
This is local to the running session, only.
|
|
86
|
+
|
|
87
|
+
For Windows and Mac:
|
|
77
88
|
|
|
78
89
|
sudo apt-get update && sudo apt-get install your-package
|
|
79
90
|
|
|
91
|
+
For Linux:
|
|
92
|
+
|
|
93
|
+
sudo yum install your-package
|
|
94
|
+
|
|
80
95
|
You can also choose between different executable ruby versions by `rvm use <version>` .
|
|
81
|
-
The current default is
|
|
96
|
+
The current default is 3.1.
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
### As a CI System Container
|
|
100
|
+
|
|
101
|
+
The OCI images provided by rake-compiler-dock can be used without the `rake-compiler-dock` gem or wrapper. This may be useful if your CI pipeline is building native gems.
|
|
102
|
+
|
|
103
|
+
For example, a Github Actions job might look like this:
|
|
104
|
+
|
|
105
|
+
``` yaml
|
|
106
|
+
jobs:
|
|
107
|
+
native-gem:
|
|
108
|
+
name: "native-gem"
|
|
109
|
+
runs-on: ubuntu-latest
|
|
110
|
+
container:
|
|
111
|
+
image: "larskanis/rake-compiler-dock-mri-x86_64-linux:1.2.0"
|
|
112
|
+
steps:
|
|
113
|
+
- uses: actions/checkout@v2
|
|
114
|
+
- run: bundle install && bundle exec rake gem:x86_64-linux:rcd
|
|
115
|
+
- uses: actions/upload-artifact@v2
|
|
116
|
+
with:
|
|
117
|
+
name: native-gem
|
|
118
|
+
path: gems
|
|
119
|
+
retention-days: 1
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
Where the referenced rake task might be defined by:
|
|
123
|
+
|
|
124
|
+
``` ruby
|
|
125
|
+
cross_platforms = ["x64-mingw32", "x86_64-linux", "x86_64-darwin", "arm64-darwin"]
|
|
126
|
+
|
|
127
|
+
namespace "gem" do
|
|
128
|
+
cross_platforms.each do |platform|
|
|
129
|
+
namespace platform do
|
|
130
|
+
task "rcd" do
|
|
131
|
+
Rake::Task["native:#{platform}"].invoke
|
|
132
|
+
Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform)}.gem"].invoke
|
|
133
|
+
end
|
|
134
|
+
end
|
|
135
|
+
end
|
|
136
|
+
end
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
For an example of rake tasks that support this style of invocation, visit https://github.com/flavorjones/ruby-c-extensions-explained/tree/main/precompiled
|
|
141
|
+
|
|
82
142
|
|
|
83
143
|
### JRuby support
|
|
84
144
|
|
|
@@ -102,16 +162,61 @@ To build java binary gems interactively, it can be called like this:
|
|
|
102
162
|
To make the build process reproducible for other parties, it is recommended to add rake-compiler-dock to your Rakefile.
|
|
103
163
|
This can be done like this:
|
|
104
164
|
|
|
105
|
-
|
|
165
|
+
```ruby
|
|
166
|
+
task 'gem:native' do
|
|
167
|
+
require 'rake_compiler_dock'
|
|
168
|
+
sh "bundle package --all" # Avoid repeated downloads of gems by using gem files from the host.
|
|
169
|
+
%w[ x86-mingw32 x64-mingw-ucrt x64-mingw32 x86-linux x86_64-linux arm-linux aarch64-linux x86_64-darwin arm64-darwin ].each do |plat|
|
|
170
|
+
RakeCompilerDock.sh "bundle --local && rake native:#{plat} gem", platform: plat
|
|
171
|
+
end
|
|
172
|
+
RakeCompilerDock.sh "bundle --local && rake java gem", rubyvm: :jruby
|
|
173
|
+
end
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
This runs the `bundle` and `rake` commands once for each platform.
|
|
177
|
+
That is once for the jruby gems and 6 times for the specified MRI platforms.
|
|
178
|
+
|
|
179
|
+
### Run builds in parallel
|
|
180
|
+
|
|
181
|
+
rake-compiler-dock uses dedicated docker images per build target (since rake-compiler-dock-1.0).
|
|
182
|
+
Because each target runs in a separate docker container, it is simple to run all targets in parallel.
|
|
183
|
+
The following example defines `rake gem:native` as a multitask and separates the preparation which should run only once.
|
|
184
|
+
It also shows how gem signing can be done with parallel builds.
|
|
185
|
+
Please note, that parallel builds only work reliable, if the specific platform gem is requested (instead of just "rake gem").
|
|
186
|
+
|
|
187
|
+
```ruby
|
|
188
|
+
namespace "gem" do
|
|
189
|
+
task 'prepare' do
|
|
106
190
|
require 'rake_compiler_dock'
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
191
|
+
require 'io/console'
|
|
192
|
+
sh "bundle package --all"
|
|
193
|
+
sh "cp ~/.gem/gem-*.pem build/gem/ || true"
|
|
194
|
+
ENV["GEM_PRIVATE_KEY_PASSPHRASE"] = STDIN.getpass("Enter passphrase of gem signature key: ")
|
|
195
|
+
end
|
|
196
|
+
|
|
197
|
+
exttask.cross_platform.each do |plat|
|
|
198
|
+
desc "Build all native binary gems in parallel"
|
|
199
|
+
multitask 'native' => plat
|
|
200
|
+
|
|
201
|
+
desc "Build the native gem for #{plat}"
|
|
202
|
+
task plat => 'prepare' do
|
|
203
|
+
RakeCompilerDock.sh <<-EOT, platform: plat
|
|
204
|
+
(cp build/gem/gem-*.pem ~/.gem/ || true) &&
|
|
205
|
+
bundle --local &&
|
|
206
|
+
rake native:#{plat} pkg/#{exttask.gem_spec.full_name}-#{plat}.gem
|
|
207
|
+
EOT
|
|
208
|
+
end
|
|
110
209
|
end
|
|
210
|
+
end
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
### Add to your Gemfile
|
|
111
214
|
|
|
112
215
|
Rake-compiler-dock uses [semantic versioning](http://semver.org/), so you should add it into your Gemfile, to make sure, that future changes will not break your build.
|
|
113
216
|
|
|
114
|
-
|
|
217
|
+
```ruby
|
|
218
|
+
gem 'rake-compiler-dock', '~> 1.2'
|
|
219
|
+
```
|
|
115
220
|
|
|
116
221
|
See [the wiki](https://github.com/rake-compiler/rake-compiler-dock/wiki/Projects-using-rake-compiler-dock) for projects which make use of rake-compiler-dock.
|
|
117
222
|
|
|
@@ -122,14 +227,18 @@ Rake-compiler-dock makes use of several environment variables.
|
|
|
122
227
|
|
|
123
228
|
The following variables are recognized by rake-compiler-dock:
|
|
124
229
|
|
|
125
|
-
* `RCD_RUBYVM` - The
|
|
230
|
+
* `RCD_RUBYVM` - The ruby VM and toolchain to be used.
|
|
126
231
|
Must be one of `mri`, `jruby`.
|
|
232
|
+
* `RCD_PLATFORM` - The target rubygems platform.
|
|
233
|
+
Must be a space separated list out of `x86-mingw32`, `x64-mingw-ucrt`, `x64-mingw32`, `x86-linux`, `x86_64-linux`, `arm-linux`, `aarch64-linux`, `x86_64-darwin` and `arm64-darwin`.
|
|
234
|
+
It is ignored when `rubyvm` is set to `:jruby`.
|
|
127
235
|
* `RCD_IMAGE` - The docker image that is downloaded and started.
|
|
128
236
|
Defaults to "larskanis/rake-compiler-dock:IMAGE_VERSION" with an image version that is determined by the gem version.
|
|
129
237
|
|
|
130
238
|
The following variables are passed through to the docker container without modification:
|
|
131
239
|
|
|
132
240
|
* `http_proxy`, `https_proxy`, `ftp_proxy` - See [Frequently asked questions](https://github.com/rake-compiler/rake-compiler-dock/wiki/FAQ) for more details.
|
|
241
|
+
* `GEM_PRIVATE_KEY_PASSPHRASE` - To avoid interactive password prompts in the container.
|
|
133
242
|
|
|
134
243
|
The following variables are provided to the running docker container:
|
|
135
244
|
|
|
@@ -137,7 +246,7 @@ The following variables are provided to the running docker container:
|
|
|
137
246
|
* `RCD_HOST_RUBY_PLATFORM` - The `RUBY_PLATFORM` of the host ruby.
|
|
138
247
|
* `RCD_HOST_RUBY_VERSION` - The `RUBY_VERSION` of the host ruby.
|
|
139
248
|
* `RUBY_CC_VERSION` - The target ruby versions for rake-compiler.
|
|
140
|
-
The default is defined in the [Dockerfile](https://github.com/rake-compiler/rake-compiler-dock/blob/
|
|
249
|
+
The default is defined in the [Dockerfile](https://github.com/rake-compiler/rake-compiler-dock/blob/94770238d68d71df5f70abe76097451a575ce46c/Dockerfile.mri.erb#L229), but can be changed as a parameter to rake.
|
|
141
250
|
* `RCD_MOUNTDIR` - The directory which is mounted into the docker container.
|
|
142
251
|
Defaults to pwd.
|
|
143
252
|
* `RCD_WORKDIR` - The working directory within the docker container.
|
|
@@ -145,7 +254,9 @@ The following variables are provided to the running docker container:
|
|
|
145
254
|
|
|
146
255
|
Other environment variables can be set or passed through to the container like this:
|
|
147
256
|
|
|
148
|
-
|
|
257
|
+
```ruby
|
|
258
|
+
RakeCompilerDock.sh "rake cross native gem OPENSSL_VERSION=#{ENV['OPENSSL_VERSION']}"
|
|
259
|
+
```
|
|
149
260
|
|
|
150
261
|
|
|
151
262
|
## More information
|
data/Rakefile
CHANGED
|
@@ -1,11 +1,65 @@
|
|
|
1
|
+
require 'erb'
|
|
2
|
+
require "rake/clean"
|
|
1
3
|
require "rake_compiler_dock"
|
|
2
|
-
|
|
4
|
+
require_relative "build/gem_helper"
|
|
5
|
+
require_relative "build/parallel_docker_build"
|
|
3
6
|
|
|
4
7
|
RakeCompilerDock::GemHelper.install_tasks
|
|
5
8
|
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
+
DOCKERHUB_USER = ENV['DOCKERHUB_USER'] || "larskanis"
|
|
10
|
+
docker_build_cmd = Shellwords.split(ENV['RCD_DOCKER_BUILD'] || "docker build")
|
|
11
|
+
|
|
12
|
+
platforms = [
|
|
13
|
+
["x86-mingw32", "i686-w64-mingw32"],
|
|
14
|
+
["x64-mingw32", "x86_64-w64-mingw32"],
|
|
15
|
+
["x64-mingw-ucrt", "x86_64-w64-mingw32"],
|
|
16
|
+
["x86-linux", "i686-redhat-linux"],
|
|
17
|
+
["x86_64-linux", "x86_64-redhat-linux"],
|
|
18
|
+
["x86_64-darwin", "x86_64-apple-darwin"],
|
|
19
|
+
["arm64-darwin", "aarch64-apple-darwin"],
|
|
20
|
+
["arm-linux", "arm-linux-gnueabihf"],
|
|
21
|
+
["aarch64-linux", "aarch64-linux-gnu"],
|
|
22
|
+
]
|
|
23
|
+
|
|
24
|
+
namespace :build do
|
|
25
|
+
|
|
26
|
+
platforms.each do |platform, target|
|
|
27
|
+
sdf = "Dockerfile.mri.#{platform}"
|
|
28
|
+
|
|
29
|
+
desc "Build image for platform #{platform}"
|
|
30
|
+
task platform => sdf
|
|
31
|
+
task sdf do
|
|
32
|
+
sh(*docker_build_cmd, "-t", "#{DOCKERHUB_USER}/rake-compiler-dock-mri-#{platform}:#{RakeCompilerDock::IMAGE_VERSION}", "-f", "Dockerfile.mri.#{platform}", ".")
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
df = ERB.new(File.read("Dockerfile.mri.erb"), trim_mode: ">").result(binding)
|
|
36
|
+
File.write(sdf, df)
|
|
37
|
+
CLEAN.include(sdf)
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
desc "Build image for JRuby"
|
|
41
|
+
task :jruby => "Dockerfile.jruby"
|
|
42
|
+
task "Dockerfile.jruby" do
|
|
43
|
+
sh(*docker_build_cmd, "-t", "#{DOCKERHUB_USER}/rake-compiler-dock-jruby:#{RakeCompilerDock::IMAGE_VERSION}", "-f", "Dockerfile.jruby", ".")
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
RakeCompilerDock::ParallelDockerBuild.new(platforms.map{|pl, _| "Dockerfile.mri.#{pl}" } + ["Dockerfile.jruby"], workdir: "tmp/docker", docker_build_cmd: docker_build_cmd)
|
|
47
|
+
|
|
48
|
+
desc "Build images for all MRI platforms in parallel"
|
|
49
|
+
multitask :mri => platforms.map(&:first)
|
|
50
|
+
|
|
51
|
+
desc "Build images for all platforms in parallel"
|
|
52
|
+
multitask :all => platforms.map(&:first) + ["jruby"]
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
task :build => "build:all"
|
|
56
|
+
|
|
57
|
+
namespace :prepare do
|
|
58
|
+
desc "Build cross compiler for x64-mingw-ucrt aka RubyInstaller-3.1+"
|
|
59
|
+
task "mingw64-ucrt" do
|
|
60
|
+
sh(*docker_build_cmd, "-t", "#{DOCKERHUB_USER}/mingw64-ucrt:20.04", ".",
|
|
61
|
+
chdir: "mingw64-ucrt")
|
|
62
|
+
end
|
|
9
63
|
end
|
|
10
64
|
|
|
11
65
|
desc "Run tests"
|
|
@@ -34,3 +88,16 @@ task :update_lists do
|
|
|
34
88
|
EOT
|
|
35
89
|
end
|
|
36
90
|
end
|
|
91
|
+
|
|
92
|
+
namespace :release do
|
|
93
|
+
desc "push all docker images"
|
|
94
|
+
task :images do
|
|
95
|
+
jimg = "#{DOCKERHUB_USER}/rake-compiler-dock-jruby:#{RakeCompilerDock::IMAGE_VERSION}"
|
|
96
|
+
sh "docker", "push", jimg
|
|
97
|
+
|
|
98
|
+
platforms.each do |platform, _|
|
|
99
|
+
img = "#{DOCKERHUB_USER}/rake-compiler-dock-mri-#{platform}:#{RakeCompilerDock::IMAGE_VERSION}"
|
|
100
|
+
sh "docker", "push", img
|
|
101
|
+
end
|
|
102
|
+
end
|
|
103
|
+
end
|
|
@@ -1,18 +1,3 @@
|
|
|
1
|
-
# This file is part of Libusb for Ruby.
|
|
2
|
-
#
|
|
3
|
-
# Libusb for Ruby is free software: you can redistribute it and/or modify
|
|
4
|
-
# it under the terms of the GNU Lesser General Public License as published by
|
|
5
|
-
# the Free Software Foundation, either version 3 of the License, or
|
|
6
|
-
# (at your option) any later version.
|
|
7
|
-
#
|
|
8
|
-
# Libusb for Ruby is distributed in the hope that it will be useful,
|
|
9
|
-
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
10
|
-
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
11
|
-
# GNU Lesser General Public License for more details.
|
|
12
|
-
#
|
|
13
|
-
# You should have received a copy of the GNU Lesser General Public License
|
|
14
|
-
# along with Libusb for Ruby. If not, see <http://www.gnu.org/licenses/>.
|
|
15
|
-
|
|
16
1
|
require "bundler/gem_helper"
|
|
17
2
|
|
|
18
3
|
module RakeCompilerDock
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
require "fileutils"
|
|
2
|
+
require "rake"
|
|
3
|
+
|
|
4
|
+
module RakeCompilerDock
|
|
5
|
+
# Run docker builds in parallel, but ensure that common docker layers are reused
|
|
6
|
+
class ParallelDockerBuild
|
|
7
|
+
include Rake::DSL
|
|
8
|
+
|
|
9
|
+
attr_reader :docker_build_cmd
|
|
10
|
+
|
|
11
|
+
def initialize(dockerfiles, workdir: "tmp/docker", inputdir: ".", task_prefix: "common-",
|
|
12
|
+
docker_build_cmd: %w["docker", "build"])
|
|
13
|
+
@docker_build_cmd = docker_build_cmd
|
|
14
|
+
FileUtils.mkdir_p(workdir)
|
|
15
|
+
|
|
16
|
+
files = parse_dockerfiles(dockerfiles, inputdir)
|
|
17
|
+
# pp files
|
|
18
|
+
|
|
19
|
+
vcs = find_commons(files)
|
|
20
|
+
# pp vcs
|
|
21
|
+
|
|
22
|
+
define_common_tasks(vcs, workdir, task_prefix)
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Read given dockerfiles from inputdir and split into a list of commands.
|
|
26
|
+
#
|
|
27
|
+
# Returns:
|
|
28
|
+
# {"File0"=>[" FROM a\n", " RUN a\n", " RUN d\n"],
|
|
29
|
+
# "File1"=>[" FROM a\n",
|
|
30
|
+
# ...
|
|
31
|
+
def parse_dockerfiles(dockerfiles, inputdir)
|
|
32
|
+
dockerfiles.map do |fn|
|
|
33
|
+
[fn, File.read(File.join(inputdir, fn))]
|
|
34
|
+
end.map do |fn, f|
|
|
35
|
+
# Split file contant in lines unless line ends with backslash
|
|
36
|
+
fc = f.each_line.with_object([]) do |line, lines|
|
|
37
|
+
if lines.last=~/\\\n\z/
|
|
38
|
+
lines.last << line
|
|
39
|
+
else
|
|
40
|
+
lines << line
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
[fn, fc]
|
|
44
|
+
end.to_h
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
# Build a tree of common parts of given files.
|
|
48
|
+
#
|
|
49
|
+
# Returns:
|
|
50
|
+
# {["File0", "File1", "File2", "File3"]=>
|
|
51
|
+
# [[" FROM a\n"],
|
|
52
|
+
# {["File0", "File1"]=>
|
|
53
|
+
# [[" RUN a\n", " RUN d\n"], {["File1"]=>[[" RUN f\n"], {}]}],
|
|
54
|
+
# ["File2", "File3"]=>[[" RUN b\n", " RUN c\n", " RUN d\n"], {}]}]}
|
|
55
|
+
def find_commons(files, vmask=nil, li=0)
|
|
56
|
+
vmask ||= files.keys
|
|
57
|
+
vcs = Hash.new { [] }
|
|
58
|
+
files.each do |fn, lines|
|
|
59
|
+
next unless vmask.include?(fn)
|
|
60
|
+
vcs[lines[li]] += [fn]
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
vcs.map do |line, vc|
|
|
64
|
+
next unless line
|
|
65
|
+
nvcs = find_commons(files, vc, li+1)
|
|
66
|
+
if nvcs.first && nvcs.first[0] == vc
|
|
67
|
+
# Append lines that are equal between file(s)
|
|
68
|
+
nl = [[line] + nvcs.first[1][0], nvcs.first[1][1]]
|
|
69
|
+
else
|
|
70
|
+
nl = [[line], nvcs]
|
|
71
|
+
end
|
|
72
|
+
[vc, nl]
|
|
73
|
+
end.compact.to_h
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
# Write intermediate dockerfiles to workdir and define rake tasks
|
|
77
|
+
#
|
|
78
|
+
# The rake tasks are named after the dockerfiles given to #new .
|
|
79
|
+
# This also adds dependant intermediate tasks as prerequisites.
|
|
80
|
+
def define_common_tasks(vcs, workdir, task_prefix, plines=[])
|
|
81
|
+
vcs.map do |files, (lines, nvcs)|
|
|
82
|
+
fn = "#{task_prefix}#{files.join}"
|
|
83
|
+
File.write(File.join(workdir, fn), (plines + lines).join)
|
|
84
|
+
task fn do
|
|
85
|
+
docker_build(fn, workdir)
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
nfn = define_common_tasks(nvcs, workdir, task_prefix, plines + lines)
|
|
89
|
+
nfn.each do |file|
|
|
90
|
+
task file => fn
|
|
91
|
+
end
|
|
92
|
+
files.each do |file|
|
|
93
|
+
task file => fn
|
|
94
|
+
end
|
|
95
|
+
fn
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
# Run an intermediate dockerfile without tag
|
|
100
|
+
#
|
|
101
|
+
# The layers will be reused in subsequent builds, even if they run in parallel.
|
|
102
|
+
def docker_build(filename, workdir)
|
|
103
|
+
sh(*docker_build_cmd, "-f", File.join(workdir, filename), ".")
|
|
104
|
+
end
|
|
105
|
+
end
|
|
106
|
+
end
|
|
@@ -17,7 +17,7 @@ index 82c5940538..a8d3661ec8 100644
|
|
|
17
17
|
@@ -10721,7 +10721,6 @@ nogvl_copy_stream_wait_write(struct copy_stream_struct *stp)
|
|
18
18
|
}
|
|
19
19
|
|
|
20
|
-
#if defined __linux__ && defined __NR_copy_file_range
|
|
20
|
+
#if defined HAVE_COPY_FILE_RANGE || (defined __linux__ && defined __NR_copy_file_range)
|
|
21
21
|
-# define USE_COPY_FILE_RANGE
|
|
22
22
|
#endif
|
|
23
23
|
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
diff --git a/configure b/configure
|
|
2
|
+
index ebe3d8c..a336b73 100755
|
|
3
|
+
--- a/configure
|
|
4
|
+
+++ b/configure
|
|
5
|
+
@@ -18943,7 +18943,6 @@ do :
|
|
6
|
+
ac_fn_c_check_func "$LINENO" "sendfile" "ac_cv_func_sendfile"
|
|
7
|
+
if test "x$ac_cv_func_sendfile" = xyes; then :
|
|
8
|
+
cat >>confdefs.h <<_ACEOF
|
|
9
|
+
-#define HAVE_SENDFILE 1
|
|
10
|
+
_ACEOF
|
|
11
|
+
|
|
12
|
+
fi
|
|
13
|
+
diff --git a/io.c b/io.c
|
|
14
|
+
index 48592ac51a..2635ac8ec6 100644
|
|
15
|
+
--- a/io.c
|
|
16
|
+
+++ b/io.c
|
|
17
|
+
@@ -11111,6 +11111,7 @@ nogvl_copy_stream_wait_write(struct copy_stream_struct *stp)
|
|
18
|
+
return 0;
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
+#undef USE_COPY_FILE_RANGE
|
|
22
|
+
#ifdef USE_COPY_FILE_RANGE
|
|
23
|
+
|
|
24
|
+
static ssize_t
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
diff --git a/tasks/bin/cross-ruby.rake b/tasks/bin/cross-ruby.rake
|
|
2
|
+
index 8317a2a..5a1b3ea 100644
|
|
3
|
+
--- a/tasks/bin/cross-ruby.rake
|
|
4
|
+
+++ b/tasks/bin/cross-ruby.rake
|
|
5
|
+
@@ -116,10 +116,22 @@ RUBY_CC_VERSIONS.split(":").each do |ruby_cc_version|
|
|
6
|
+
"--host=#{mingw_host}",
|
|
7
|
+
"--target=#{mingw_target}",
|
|
8
|
+
"--build=#{RUBY_BUILD}",
|
|
9
|
+
- '--enable-shared',
|
|
10
|
+
+ '--enable-install-static-library',
|
|
11
|
+
+ '--disable-jit-support',
|
|
12
|
+
'--disable-install-doc',
|
|
13
|
+
'--with-ext=',
|
|
14
|
+
]
|
|
15
|
+
+ if mingw_host=~/darwin/
|
|
16
|
+
+ options += [
|
|
17
|
+
+ '--enable-static',
|
|
18
|
+
+ '--disable-shared',
|
|
19
|
+
+ ]
|
|
20
|
+
+ else
|
|
21
|
+
+ options += [
|
|
22
|
+
+ '--enable-static',
|
|
23
|
+
+ '--enable-shared',
|
|
24
|
+
+ ]
|
|
25
|
+
+ end
|
|
26
|
+
|
|
27
|
+
# Force Winsock2 for Ruby 1.8, 1.9 defaults to it
|
|
28
|
+
options << "--with-winsock2" if major == "1.8"
|
data/build/rcd-env.sh
ADDED
data/build/runas
CHANGED
|
@@ -4,9 +4,4 @@ groupadd -o -g "$GID" "$GROUP"
|
|
|
4
4
|
mkdir -p /tmp/home
|
|
5
5
|
useradd -o -g "$GID" -u "$UID" -G rvm,sudo -p "" -b /tmp/home -m "$USER"
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
ln -s /usr/local/rake-compiler "$HOME"/.rake-compiler
|
|
9
|
-
|
|
10
|
-
sudo -u "$USER" --set-home \
|
|
11
|
-
BASH_ENV=/etc/rubybashrc \
|
|
12
|
-
-- "$@"
|
|
7
|
+
/usr/bin/sudo -u "$USER" -H BASH_ENV=/etc/rubybashrc -- "$@"
|
data/build/sudoers
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
Defaults env_keep += "http_proxy https_proxy ftp_proxy RCD_HOST_RUBY_PLATFORM RCD_HOST_RUBY_VERSION RCD_IMAGE RUBY_CC_VERSION"
|
|
1
|
+
Defaults env_keep += "http_proxy https_proxy ftp_proxy GEM_PRIVATE_KEY_PASSPHRASE RCD_HOST_RUBY_PLATFORM RCD_HOST_RUBY_VERSION RCD_IMAGE RUBY_CC_VERSION LD_LIBRARY_PATH DEVTOOLSET_ROOTPATH"
|
|
@@ -8,6 +8,7 @@ module RakeCompilerDock
|
|
|
8
8
|
|
|
9
9
|
attr_reader :io
|
|
10
10
|
attr_reader :pwd
|
|
11
|
+
attr_reader :docker_command
|
|
11
12
|
attr_accessor :machine_name
|
|
12
13
|
|
|
13
14
|
def initialize(io, pwd, machine_name="rake-compiler-dock")
|
|
@@ -53,12 +54,17 @@ module RakeCompilerDock
|
|
|
53
54
|
end
|
|
54
55
|
end
|
|
55
56
|
|
|
57
|
+
COMMANDS = %w[docker podman]
|
|
56
58
|
def docker_version
|
|
57
|
-
|
|
59
|
+
COMMANDS.find do |command|
|
|
60
|
+
@docker_version_text, @docker_version_status = run("#{command} version")
|
|
61
|
+
@docker_command = command
|
|
62
|
+
@docker_version_status == 0
|
|
63
|
+
end
|
|
58
64
|
end
|
|
59
65
|
|
|
60
66
|
def ok?
|
|
61
|
-
@docker_version_status == 0 && @docker_version_text =~ /version/ && doma_pwd_ok?
|
|
67
|
+
@docker_version_status == 0 && @docker_version_text =~ /version/i && doma_pwd_ok?
|
|
62
68
|
end
|
|
63
69
|
|
|
64
70
|
def docker_client_avail?
|
|
@@ -216,16 +222,16 @@ module RakeCompilerDock
|
|
|
216
222
|
help << yellow("Please download and install the docker-toolbox:")
|
|
217
223
|
help << yellow(" https://www.docker.com/docker-toolbox")
|
|
218
224
|
when /linux/
|
|
219
|
-
help << red("Docker is
|
|
225
|
+
help << red("Neither Docker nor Podman is available.")
|
|
220
226
|
help << ""
|
|
221
|
-
help << yellow("Install on Ubuntu/Debian:")
|
|
227
|
+
help << yellow("Install Docker on Ubuntu/Debian:")
|
|
222
228
|
help << " sudo apt-get install docker.io"
|
|
223
229
|
help << ""
|
|
224
|
-
help << yellow("Install on Fedora/Centos/RHEL")
|
|
230
|
+
help << yellow("Install Docker on Fedora/Centos/RHEL")
|
|
225
231
|
help << " sudo yum install docker"
|
|
226
232
|
help << " sudo systemctl start docker"
|
|
227
233
|
help << ""
|
|
228
|
-
help << yellow("Install on SuSE")
|
|
234
|
+
help << yellow("Install Docker on SuSE")
|
|
229
235
|
help << " sudo zypper install docker"
|
|
230
236
|
help << " sudo systemctl start docker"
|
|
231
237
|
when /darwin/
|