rake-compiler-dock 0.7.2 → 1.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +4 -0
- data/.github/workflows/ci.yml +168 -0
- data/.gitignore +14 -0
- data/Dockerfile.jruby +12 -7
- data/Dockerfile.mri.erb +271 -0
- data/History.md +63 -0
- data/README.md +138 -27
- data/Rakefile +71 -4
- data/{lib/rake_compiler_dock → build}/gem_helper.rb +0 -15
- data/build/parallel_docker_build.rb +106 -0
- data/build/patches/{ruby-2.5.3 → ruby-2.5.9}/no_sendfile.patch +1 -1
- data/build/patches/ruby-3.1.0/no_sendfile.patch +24 -0
- data/build/patches2/rake-compiler-1.1.6/0004-Enable-build-of-static-libruby.patch +28 -0
- data/build/rcd-env.sh +6 -0
- data/build/runas +1 -6
- data/build/sudoers +1 -1
- data/lib/rake_compiler_dock/docker_check.rb +12 -6
- data/lib/rake_compiler_dock/starter.rb +69 -53
- data/lib/rake_compiler_dock/version.rb +2 -2
- data/lib/rake_compiler_dock.rb +10 -0
- data/mingw64-ucrt/Dockerfile +62 -0
- data/mingw64-ucrt/binutils-mingw-w64-ignore-check-errors.patch +13 -0
- data/mingw64-ucrt/gcc-mingw-w64-only-c-c++.patch +13 -0
- data/mingw64-ucrt/mingw-w64-enable-ucrt.patch +22 -0
- data/rake-compiler-dock.gemspec +2 -2
- data/test/env/Dockerfile.alpine +17 -0
- data/test/env/Dockerfile.centos +17 -0
- data/test/env/Dockerfile.debian +24 -0
- data/test/rcd_test/Gemfile +11 -0
- data/test/rcd_test/Rakefile +74 -0
- data/test/rcd_test/ext/java/RcdTestExtService.java +19 -0
- data/test/rcd_test/ext/java/RubyRcdTest.java +16 -0
- data/test/rcd_test/ext/mri/extconf.rb +20 -0
- data/test/rcd_test/ext/mri/rcd_test_ext.c +35 -0
- data/test/rcd_test/ext/mri/rcd_test_ext.h +6 -0
- data/test/rcd_test/lib/rcd_test.rb +6 -0
- data/test/rcd_test/rcd_test.gemspec +27 -0
- data/test/rcd_test/test/test_basic.rb +22 -0
- data/test/test_environment_variables.rb +56 -31
- data/test/test_parallel_docker_build.rb +70 -0
- data.tar.gz.sig +1 -0
- metadata +83 -14
- metadata.gz.sig +0 -0
- data/Dockerfile.mri +0 -122
- data/build/patches/rake-compiler-1.0.7/enable-static.diff +0 -13
data/README.md
CHANGED
@@ -1,8 +1,8 @@
|
|
1
1
|
# rake-compiler-dock
|
2
2
|
|
3
|
-
**Easy to use and reliable cross compiler environment for building Windows, Linux and JRuby binary gems.**
|
3
|
+
**Easy to use and reliable cross compiler environment for building Windows, Linux, Mac and JRuby binary gems.**
|
4
4
|
|
5
|
-
It provides cross compilers and Ruby environments for 2.
|
5
|
+
It provides cross compilers and Ruby environments for 2.4 and newer versions of the [RubyInstaller](http://rubyinstaller.org/) and Linux runtime environments.
|
6
6
|
They are prepared for use with [rake-compiler](https://github.com/rake-compiler/rake-compiler).
|
7
7
|
It is used by [many gems with C or JRuby extentions](https://github.com/rake-compiler/rake-compiler-dock/wiki/Projects-using-rake-compiler-dock).
|
8
8
|
|
@@ -12,11 +12,7 @@ It is also more reliable, since the underlying docker images are versioned and i
|
|
12
12
|
|
13
13
|
## Installation
|
14
14
|
|
15
|
-
Install docker
|
16
|
-
|
17
|
-
$ sudo apt-get install docker.io
|
18
|
-
|
19
|
-
... or install [docker-toolbox for Windows and OSX](https://www.docker.com/docker-toolbox) or boot2docker on [Windows](https://github.com/boot2docker/windows-installer/releases) or [OS X](https://github.com/boot2docker/osx-installer/releases) .
|
15
|
+
Install docker [following the instructions on the docker website](https://docs.docker.com/engine/install/) ... or install [docker-toolbox for Windows and OSX](https://github.com/docker/toolbox/releases) or boot2docker on [Windows](https://github.com/boot2docker/windows-installer/releases) or [OS X](https://github.com/boot2docker/osx-installer/releases) .
|
20
16
|
|
21
17
|
Install rake-compiler-dock as a gem. The docker image is downloaded later on demand:
|
22
18
|
|
@@ -34,16 +30,23 @@ Rake-compiler-dock provides the necessary tools to build Ruby extensions for Win
|
|
34
30
|
It is intended to be used in conjunction with [rake-compiler's](https://github.com/rake-compiler/rake-compiler) cross build capability.
|
35
31
|
Your Rakefile should enable cross compilation like so:
|
36
32
|
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
33
|
+
```ruby
|
34
|
+
exttask = Rake::ExtensionTask.new('my_extension', my_gem_spec) do |ext|
|
35
|
+
ext.cross_compile = true
|
36
|
+
ext.cross_platform = %w[x86-mingw32 x64-mingw-ucrt x64-mingw32 x86-linux x86_64-linux x86_64-darwin arm64-darwin]
|
37
|
+
end
|
38
|
+
```
|
39
|
+
|
40
|
+
See below, how to invoke cross builds in your Rakefile.
|
41
41
|
|
42
|
-
|
42
|
+
Additionally it may also be used to build ffi based binary gems like [libusb](https://github.com/larskanis/libusb), but currently doesn't provide any additional build helpers for this use case, beyond docker invocation and cross compilers.
|
43
43
|
|
44
44
|
### Interactive Usage
|
45
45
|
|
46
46
|
Rake-compiler-dock offers the shell command `rake-compiler-dock` and a [ruby API](http://www.rubydoc.info/gems/rake-compiler-dock/RakeCompilerDock) for issuing commands within the docker image, described below.
|
47
|
+
There are dedicated images for `x86-mingw32`, `x64-mingw-ucrt`, `x64-mingw32`, `x86-linux`, `x86_64-linux`, `x86_64-darwin`, `arm64-darwin` and `jruby` targets.
|
48
|
+
The images contain all supported cross ruby versions, with the exception of `x64-mingw32`, which has versions before 3.1 only, and `x64-mingw-ucrt`, which has only ruby-3.1+.
|
49
|
+
This is to match the [changed platform of RubyInstaller-3.1](https://rubyinstaller.org/2021/12/31/rubyinstaller-3.1.0-1-released.html).
|
47
50
|
|
48
51
|
`rake-compiler-dock` without arguments starts an interactive shell session.
|
49
52
|
This is best suited to try out and debug a build.
|
@@ -55,16 +58,22 @@ But note, that all other changes to the file system of the container are dropped
|
|
55
58
|
All commands are executed with the same user and group of the host.
|
56
59
|
This is done by copying user account data into the container and sudo to it.
|
57
60
|
|
58
|
-
To build x86
|
61
|
+
To build x86 Windows and x86_64 Linux binary gems interactively, it can be called like this:
|
59
62
|
|
60
63
|
user@host:$ cd your-gem-dir/
|
61
|
-
user@host:$ rake-compiler-dock # this enters a container with an interactive shell
|
64
|
+
user@host:$ rake-compiler-dock # this enters a container with an interactive shell for x86 Windows (default)
|
62
65
|
user@5b53794ada92:$ bundle
|
63
66
|
user@5b53794ada92:$ rake cross native gem
|
64
67
|
user@5b53794ada92:$ exit
|
65
68
|
user@host:$ ls pkg/*.gem
|
66
|
-
your-gem-1.0.0.gem your-gem-1.0.0-
|
67
|
-
|
69
|
+
your-gem-1.0.0.gem your-gem-1.0.0-x86-mingw32.gem
|
70
|
+
|
71
|
+
user@host:$ RCD_PLATFORM=x86_64-linux rake-compiler-dock # this enters a container for amd64 Linux target
|
72
|
+
user@adc55b2b92a9:$ bundle
|
73
|
+
user@adc55b2b92a9:$ rake cross native gem
|
74
|
+
user@adc55b2b92a9:$ exit
|
75
|
+
user@host:$ ls pkg/*.gem
|
76
|
+
your-gem-1.0.0.gem your-gem-1.0.0-x86_64-linux.gem
|
68
77
|
|
69
78
|
Or non-interactive:
|
70
79
|
|
@@ -72,13 +81,64 @@ Or non-interactive:
|
|
72
81
|
|
73
82
|
The environment variable `RUBY_CC_VERSION` is predefined as described [below](#environment-variables).
|
74
83
|
|
75
|
-
If necessary, additional software
|
76
|
-
This is local to the running session, only
|
84
|
+
If necessary, additional software can be installed, prior to the build command.
|
85
|
+
This is local to the running session, only.
|
86
|
+
|
87
|
+
For Windows and Mac:
|
77
88
|
|
78
89
|
sudo apt-get update && sudo apt-get install your-package
|
79
90
|
|
91
|
+
For Linux:
|
92
|
+
|
93
|
+
sudo yum install your-package
|
94
|
+
|
80
95
|
You can also choose between different executable ruby versions by `rvm use <version>` .
|
81
|
-
The current default is
|
96
|
+
The current default is 3.1.
|
97
|
+
|
98
|
+
|
99
|
+
### As a CI System Container
|
100
|
+
|
101
|
+
The OCI images provided by rake-compiler-dock can be used without the `rake-compiler-dock` gem or wrapper. This may be useful if your CI pipeline is building native gems.
|
102
|
+
|
103
|
+
For example, a Github Actions job might look like this:
|
104
|
+
|
105
|
+
``` yaml
|
106
|
+
jobs:
|
107
|
+
native-gem:
|
108
|
+
name: "native-gem"
|
109
|
+
runs-on: ubuntu-latest
|
110
|
+
container:
|
111
|
+
image: "larskanis/rake-compiler-dock-mri-x86_64-linux:1.2.0"
|
112
|
+
steps:
|
113
|
+
- uses: actions/checkout@v2
|
114
|
+
- run: bundle install && bundle exec rake gem:x86_64-linux:rcd
|
115
|
+
- uses: actions/upload-artifact@v2
|
116
|
+
with:
|
117
|
+
name: native-gem
|
118
|
+
path: gems
|
119
|
+
retention-days: 1
|
120
|
+
```
|
121
|
+
|
122
|
+
Where the referenced rake task might be defined by:
|
123
|
+
|
124
|
+
``` ruby
|
125
|
+
cross_platforms = ["x64-mingw32", "x86_64-linux", "x86_64-darwin", "arm64-darwin"]
|
126
|
+
|
127
|
+
namespace "gem" do
|
128
|
+
cross_platforms.each do |platform|
|
129
|
+
namespace platform do
|
130
|
+
task "rcd" do
|
131
|
+
Rake::Task["native:#{platform}"].invoke
|
132
|
+
Rake::Task["pkg/#{rcee_precompiled_spec.full_name}-#{Gem::Platform.new(platform)}.gem"].invoke
|
133
|
+
end
|
134
|
+
end
|
135
|
+
end
|
136
|
+
end
|
137
|
+
|
138
|
+
```
|
139
|
+
|
140
|
+
For an example of rake tasks that support this style of invocation, visit https://github.com/flavorjones/ruby-c-extensions-explained/tree/main/precompiled
|
141
|
+
|
82
142
|
|
83
143
|
### JRuby support
|
84
144
|
|
@@ -102,16 +162,61 @@ To build java binary gems interactively, it can be called like this:
|
|
102
162
|
To make the build process reproducible for other parties, it is recommended to add rake-compiler-dock to your Rakefile.
|
103
163
|
This can be done like this:
|
104
164
|
|
105
|
-
|
165
|
+
```ruby
|
166
|
+
task 'gem:native' do
|
167
|
+
require 'rake_compiler_dock'
|
168
|
+
sh "bundle package --all" # Avoid repeated downloads of gems by using gem files from the host.
|
169
|
+
%w[ x86-mingw32 x64-mingw-ucrt x64-mingw32 x86-linux x86_64-linux arm-linux aarch64-linux x86_64-darwin arm64-darwin ].each do |plat|
|
170
|
+
RakeCompilerDock.sh "bundle --local && rake native:#{plat} gem", platform: plat
|
171
|
+
end
|
172
|
+
RakeCompilerDock.sh "bundle --local && rake java gem", rubyvm: :jruby
|
173
|
+
end
|
174
|
+
```
|
175
|
+
|
176
|
+
This runs the `bundle` and `rake` commands once for each platform.
|
177
|
+
That is once for the jruby gems and 6 times for the specified MRI platforms.
|
178
|
+
|
179
|
+
### Run builds in parallel
|
180
|
+
|
181
|
+
rake-compiler-dock uses dedicated docker images per build target (since rake-compiler-dock-1.0).
|
182
|
+
Because each target runs in a separate docker container, it is simple to run all targets in parallel.
|
183
|
+
The following example defines `rake gem:native` as a multitask and separates the preparation which should run only once.
|
184
|
+
It also shows how gem signing can be done with parallel builds.
|
185
|
+
Please note, that parallel builds only work reliable, if the specific platform gem is requested (instead of just "rake gem").
|
186
|
+
|
187
|
+
```ruby
|
188
|
+
namespace "gem" do
|
189
|
+
task 'prepare' do
|
106
190
|
require 'rake_compiler_dock'
|
107
|
-
|
108
|
-
|
109
|
-
|
191
|
+
require 'io/console'
|
192
|
+
sh "bundle package --all"
|
193
|
+
sh "cp ~/.gem/gem-*.pem build/gem/ || true"
|
194
|
+
ENV["GEM_PRIVATE_KEY_PASSPHRASE"] = STDIN.getpass("Enter passphrase of gem signature key: ")
|
195
|
+
end
|
196
|
+
|
197
|
+
exttask.cross_platform.each do |plat|
|
198
|
+
desc "Build all native binary gems in parallel"
|
199
|
+
multitask 'native' => plat
|
200
|
+
|
201
|
+
desc "Build the native gem for #{plat}"
|
202
|
+
task plat => 'prepare' do
|
203
|
+
RakeCompilerDock.sh <<-EOT, platform: plat
|
204
|
+
(cp build/gem/gem-*.pem ~/.gem/ || true) &&
|
205
|
+
bundle --local &&
|
206
|
+
rake native:#{plat} pkg/#{exttask.gem_spec.full_name}-#{plat}.gem
|
207
|
+
EOT
|
208
|
+
end
|
110
209
|
end
|
210
|
+
end
|
211
|
+
```
|
212
|
+
|
213
|
+
### Add to your Gemfile
|
111
214
|
|
112
215
|
Rake-compiler-dock uses [semantic versioning](http://semver.org/), so you should add it into your Gemfile, to make sure, that future changes will not break your build.
|
113
216
|
|
114
|
-
|
217
|
+
```ruby
|
218
|
+
gem 'rake-compiler-dock', '~> 1.2'
|
219
|
+
```
|
115
220
|
|
116
221
|
See [the wiki](https://github.com/rake-compiler/rake-compiler-dock/wiki/Projects-using-rake-compiler-dock) for projects which make use of rake-compiler-dock.
|
117
222
|
|
@@ -122,14 +227,18 @@ Rake-compiler-dock makes use of several environment variables.
|
|
122
227
|
|
123
228
|
The following variables are recognized by rake-compiler-dock:
|
124
229
|
|
125
|
-
* `RCD_RUBYVM` - The
|
230
|
+
* `RCD_RUBYVM` - The ruby VM and toolchain to be used.
|
126
231
|
Must be one of `mri`, `jruby`.
|
232
|
+
* `RCD_PLATFORM` - The target rubygems platform.
|
233
|
+
Must be a space separated list out of `x86-mingw32`, `x64-mingw-ucrt`, `x64-mingw32`, `x86-linux`, `x86_64-linux`, `arm-linux`, `aarch64-linux`, `x86_64-darwin` and `arm64-darwin`.
|
234
|
+
It is ignored when `rubyvm` is set to `:jruby`.
|
127
235
|
* `RCD_IMAGE` - The docker image that is downloaded and started.
|
128
236
|
Defaults to "larskanis/rake-compiler-dock:IMAGE_VERSION" with an image version that is determined by the gem version.
|
129
237
|
|
130
238
|
The following variables are passed through to the docker container without modification:
|
131
239
|
|
132
240
|
* `http_proxy`, `https_proxy`, `ftp_proxy` - See [Frequently asked questions](https://github.com/rake-compiler/rake-compiler-dock/wiki/FAQ) for more details.
|
241
|
+
* `GEM_PRIVATE_KEY_PASSPHRASE` - To avoid interactive password prompts in the container.
|
133
242
|
|
134
243
|
The following variables are provided to the running docker container:
|
135
244
|
|
@@ -137,7 +246,7 @@ The following variables are provided to the running docker container:
|
|
137
246
|
* `RCD_HOST_RUBY_PLATFORM` - The `RUBY_PLATFORM` of the host ruby.
|
138
247
|
* `RCD_HOST_RUBY_VERSION` - The `RUBY_VERSION` of the host ruby.
|
139
248
|
* `RUBY_CC_VERSION` - The target ruby versions for rake-compiler.
|
140
|
-
The default is defined in the [Dockerfile](https://github.com/rake-compiler/rake-compiler-dock/blob/
|
249
|
+
The default is defined in the [Dockerfile](https://github.com/rake-compiler/rake-compiler-dock/blob/94770238d68d71df5f70abe76097451a575ce46c/Dockerfile.mri.erb#L229), but can be changed as a parameter to rake.
|
141
250
|
* `RCD_MOUNTDIR` - The directory which is mounted into the docker container.
|
142
251
|
Defaults to pwd.
|
143
252
|
* `RCD_WORKDIR` - The working directory within the docker container.
|
@@ -145,7 +254,9 @@ The following variables are provided to the running docker container:
|
|
145
254
|
|
146
255
|
Other environment variables can be set or passed through to the container like this:
|
147
256
|
|
148
|
-
|
257
|
+
```ruby
|
258
|
+
RakeCompilerDock.sh "rake cross native gem OPENSSL_VERSION=#{ENV['OPENSSL_VERSION']}"
|
259
|
+
```
|
149
260
|
|
150
261
|
|
151
262
|
## More information
|
data/Rakefile
CHANGED
@@ -1,11 +1,65 @@
|
|
1
|
+
require 'erb'
|
2
|
+
require "rake/clean"
|
1
3
|
require "rake_compiler_dock"
|
2
|
-
|
4
|
+
require_relative "build/gem_helper"
|
5
|
+
require_relative "build/parallel_docker_build"
|
3
6
|
|
4
7
|
RakeCompilerDock::GemHelper.install_tasks
|
5
8
|
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
+
DOCKERHUB_USER = ENV['DOCKERHUB_USER'] || "larskanis"
|
10
|
+
docker_build_cmd = Shellwords.split(ENV['RCD_DOCKER_BUILD'] || "docker build")
|
11
|
+
|
12
|
+
platforms = [
|
13
|
+
["x86-mingw32", "i686-w64-mingw32"],
|
14
|
+
["x64-mingw32", "x86_64-w64-mingw32"],
|
15
|
+
["x64-mingw-ucrt", "x86_64-w64-mingw32"],
|
16
|
+
["x86-linux", "i686-redhat-linux"],
|
17
|
+
["x86_64-linux", "x86_64-redhat-linux"],
|
18
|
+
["x86_64-darwin", "x86_64-apple-darwin"],
|
19
|
+
["arm64-darwin", "aarch64-apple-darwin"],
|
20
|
+
["arm-linux", "arm-linux-gnueabihf"],
|
21
|
+
["aarch64-linux", "aarch64-linux-gnu"],
|
22
|
+
]
|
23
|
+
|
24
|
+
namespace :build do
|
25
|
+
|
26
|
+
platforms.each do |platform, target|
|
27
|
+
sdf = "Dockerfile.mri.#{platform}"
|
28
|
+
|
29
|
+
desc "Build image for platform #{platform}"
|
30
|
+
task platform => sdf
|
31
|
+
task sdf do
|
32
|
+
sh(*docker_build_cmd, "-t", "#{DOCKERHUB_USER}/rake-compiler-dock-mri-#{platform}:#{RakeCompilerDock::IMAGE_VERSION}", "-f", "Dockerfile.mri.#{platform}", ".")
|
33
|
+
end
|
34
|
+
|
35
|
+
df = ERB.new(File.read("Dockerfile.mri.erb"), trim_mode: ">").result(binding)
|
36
|
+
File.write(sdf, df)
|
37
|
+
CLEAN.include(sdf)
|
38
|
+
end
|
39
|
+
|
40
|
+
desc "Build image for JRuby"
|
41
|
+
task :jruby => "Dockerfile.jruby"
|
42
|
+
task "Dockerfile.jruby" do
|
43
|
+
sh(*docker_build_cmd, "-t", "#{DOCKERHUB_USER}/rake-compiler-dock-jruby:#{RakeCompilerDock::IMAGE_VERSION}", "-f", "Dockerfile.jruby", ".")
|
44
|
+
end
|
45
|
+
|
46
|
+
RakeCompilerDock::ParallelDockerBuild.new(platforms.map{|pl, _| "Dockerfile.mri.#{pl}" } + ["Dockerfile.jruby"], workdir: "tmp/docker", docker_build_cmd: docker_build_cmd)
|
47
|
+
|
48
|
+
desc "Build images for all MRI platforms in parallel"
|
49
|
+
multitask :mri => platforms.map(&:first)
|
50
|
+
|
51
|
+
desc "Build images for all platforms in parallel"
|
52
|
+
multitask :all => platforms.map(&:first) + ["jruby"]
|
53
|
+
end
|
54
|
+
|
55
|
+
task :build => "build:all"
|
56
|
+
|
57
|
+
namespace :prepare do
|
58
|
+
desc "Build cross compiler for x64-mingw-ucrt aka RubyInstaller-3.1+"
|
59
|
+
task "mingw64-ucrt" do
|
60
|
+
sh(*docker_build_cmd, "-t", "#{DOCKERHUB_USER}/mingw64-ucrt:20.04", ".",
|
61
|
+
chdir: "mingw64-ucrt")
|
62
|
+
end
|
9
63
|
end
|
10
64
|
|
11
65
|
desc "Run tests"
|
@@ -34,3 +88,16 @@ task :update_lists do
|
|
34
88
|
EOT
|
35
89
|
end
|
36
90
|
end
|
91
|
+
|
92
|
+
namespace :release do
|
93
|
+
desc "push all docker images"
|
94
|
+
task :images do
|
95
|
+
jimg = "#{DOCKERHUB_USER}/rake-compiler-dock-jruby:#{RakeCompilerDock::IMAGE_VERSION}"
|
96
|
+
sh "docker", "push", jimg
|
97
|
+
|
98
|
+
platforms.each do |platform, _|
|
99
|
+
img = "#{DOCKERHUB_USER}/rake-compiler-dock-mri-#{platform}:#{RakeCompilerDock::IMAGE_VERSION}"
|
100
|
+
sh "docker", "push", img
|
101
|
+
end
|
102
|
+
end
|
103
|
+
end
|
@@ -1,18 +1,3 @@
|
|
1
|
-
# This file is part of Libusb for Ruby.
|
2
|
-
#
|
3
|
-
# Libusb for Ruby is free software: you can redistribute it and/or modify
|
4
|
-
# it under the terms of the GNU Lesser General Public License as published by
|
5
|
-
# the Free Software Foundation, either version 3 of the License, or
|
6
|
-
# (at your option) any later version.
|
7
|
-
#
|
8
|
-
# Libusb for Ruby is distributed in the hope that it will be useful,
|
9
|
-
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
10
|
-
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
11
|
-
# GNU Lesser General Public License for more details.
|
12
|
-
#
|
13
|
-
# You should have received a copy of the GNU Lesser General Public License
|
14
|
-
# along with Libusb for Ruby. If not, see <http://www.gnu.org/licenses/>.
|
15
|
-
|
16
1
|
require "bundler/gem_helper"
|
17
2
|
|
18
3
|
module RakeCompilerDock
|
@@ -0,0 +1,106 @@
|
|
1
|
+
require "fileutils"
|
2
|
+
require "rake"
|
3
|
+
|
4
|
+
module RakeCompilerDock
|
5
|
+
# Run docker builds in parallel, but ensure that common docker layers are reused
|
6
|
+
class ParallelDockerBuild
|
7
|
+
include Rake::DSL
|
8
|
+
|
9
|
+
attr_reader :docker_build_cmd
|
10
|
+
|
11
|
+
def initialize(dockerfiles, workdir: "tmp/docker", inputdir: ".", task_prefix: "common-",
|
12
|
+
docker_build_cmd: %w["docker", "build"])
|
13
|
+
@docker_build_cmd = docker_build_cmd
|
14
|
+
FileUtils.mkdir_p(workdir)
|
15
|
+
|
16
|
+
files = parse_dockerfiles(dockerfiles, inputdir)
|
17
|
+
# pp files
|
18
|
+
|
19
|
+
vcs = find_commons(files)
|
20
|
+
# pp vcs
|
21
|
+
|
22
|
+
define_common_tasks(vcs, workdir, task_prefix)
|
23
|
+
end
|
24
|
+
|
25
|
+
# Read given dockerfiles from inputdir and split into a list of commands.
|
26
|
+
#
|
27
|
+
# Returns:
|
28
|
+
# {"File0"=>[" FROM a\n", " RUN a\n", " RUN d\n"],
|
29
|
+
# "File1"=>[" FROM a\n",
|
30
|
+
# ...
|
31
|
+
def parse_dockerfiles(dockerfiles, inputdir)
|
32
|
+
dockerfiles.map do |fn|
|
33
|
+
[fn, File.read(File.join(inputdir, fn))]
|
34
|
+
end.map do |fn, f|
|
35
|
+
# Split file contant in lines unless line ends with backslash
|
36
|
+
fc = f.each_line.with_object([]) do |line, lines|
|
37
|
+
if lines.last=~/\\\n\z/
|
38
|
+
lines.last << line
|
39
|
+
else
|
40
|
+
lines << line
|
41
|
+
end
|
42
|
+
end
|
43
|
+
[fn, fc]
|
44
|
+
end.to_h
|
45
|
+
end
|
46
|
+
|
47
|
+
# Build a tree of common parts of given files.
|
48
|
+
#
|
49
|
+
# Returns:
|
50
|
+
# {["File0", "File1", "File2", "File3"]=>
|
51
|
+
# [[" FROM a\n"],
|
52
|
+
# {["File0", "File1"]=>
|
53
|
+
# [[" RUN a\n", " RUN d\n"], {["File1"]=>[[" RUN f\n"], {}]}],
|
54
|
+
# ["File2", "File3"]=>[[" RUN b\n", " RUN c\n", " RUN d\n"], {}]}]}
|
55
|
+
def find_commons(files, vmask=nil, li=0)
|
56
|
+
vmask ||= files.keys
|
57
|
+
vcs = Hash.new { [] }
|
58
|
+
files.each do |fn, lines|
|
59
|
+
next unless vmask.include?(fn)
|
60
|
+
vcs[lines[li]] += [fn]
|
61
|
+
end
|
62
|
+
|
63
|
+
vcs.map do |line, vc|
|
64
|
+
next unless line
|
65
|
+
nvcs = find_commons(files, vc, li+1)
|
66
|
+
if nvcs.first && nvcs.first[0] == vc
|
67
|
+
# Append lines that are equal between file(s)
|
68
|
+
nl = [[line] + nvcs.first[1][0], nvcs.first[1][1]]
|
69
|
+
else
|
70
|
+
nl = [[line], nvcs]
|
71
|
+
end
|
72
|
+
[vc, nl]
|
73
|
+
end.compact.to_h
|
74
|
+
end
|
75
|
+
|
76
|
+
# Write intermediate dockerfiles to workdir and define rake tasks
|
77
|
+
#
|
78
|
+
# The rake tasks are named after the dockerfiles given to #new .
|
79
|
+
# This also adds dependant intermediate tasks as prerequisites.
|
80
|
+
def define_common_tasks(vcs, workdir, task_prefix, plines=[])
|
81
|
+
vcs.map do |files, (lines, nvcs)|
|
82
|
+
fn = "#{task_prefix}#{files.join}"
|
83
|
+
File.write(File.join(workdir, fn), (plines + lines).join)
|
84
|
+
task fn do
|
85
|
+
docker_build(fn, workdir)
|
86
|
+
end
|
87
|
+
|
88
|
+
nfn = define_common_tasks(nvcs, workdir, task_prefix, plines + lines)
|
89
|
+
nfn.each do |file|
|
90
|
+
task file => fn
|
91
|
+
end
|
92
|
+
files.each do |file|
|
93
|
+
task file => fn
|
94
|
+
end
|
95
|
+
fn
|
96
|
+
end
|
97
|
+
end
|
98
|
+
|
99
|
+
# Run an intermediate dockerfile without tag
|
100
|
+
#
|
101
|
+
# The layers will be reused in subsequent builds, even if they run in parallel.
|
102
|
+
def docker_build(filename, workdir)
|
103
|
+
sh(*docker_build_cmd, "-f", File.join(workdir, filename), ".")
|
104
|
+
end
|
105
|
+
end
|
106
|
+
end
|
@@ -17,7 +17,7 @@ index 82c5940538..a8d3661ec8 100644
|
|
17
17
|
@@ -10721,7 +10721,6 @@ nogvl_copy_stream_wait_write(struct copy_stream_struct *stp)
|
18
18
|
}
|
19
19
|
|
20
|
-
#if defined __linux__ && defined __NR_copy_file_range
|
20
|
+
#if defined HAVE_COPY_FILE_RANGE || (defined __linux__ && defined __NR_copy_file_range)
|
21
21
|
-# define USE_COPY_FILE_RANGE
|
22
22
|
#endif
|
23
23
|
|
@@ -0,0 +1,24 @@
|
|
1
|
+
diff --git a/configure b/configure
|
2
|
+
index ebe3d8c..a336b73 100755
|
3
|
+
--- a/configure
|
4
|
+
+++ b/configure
|
5
|
+
@@ -18943,7 +18943,6 @@ do :
|
6
|
+
ac_fn_c_check_func "$LINENO" "sendfile" "ac_cv_func_sendfile"
|
7
|
+
if test "x$ac_cv_func_sendfile" = xyes; then :
|
8
|
+
cat >>confdefs.h <<_ACEOF
|
9
|
+
-#define HAVE_SENDFILE 1
|
10
|
+
_ACEOF
|
11
|
+
|
12
|
+
fi
|
13
|
+
diff --git a/io.c b/io.c
|
14
|
+
index 48592ac51a..2635ac8ec6 100644
|
15
|
+
--- a/io.c
|
16
|
+
+++ b/io.c
|
17
|
+
@@ -11111,6 +11111,7 @@ nogvl_copy_stream_wait_write(struct copy_stream_struct *stp)
|
18
|
+
return 0;
|
19
|
+
}
|
20
|
+
|
21
|
+
+#undef USE_COPY_FILE_RANGE
|
22
|
+
#ifdef USE_COPY_FILE_RANGE
|
23
|
+
|
24
|
+
static ssize_t
|
@@ -0,0 +1,28 @@
|
|
1
|
+
diff --git a/tasks/bin/cross-ruby.rake b/tasks/bin/cross-ruby.rake
|
2
|
+
index 8317a2a..5a1b3ea 100644
|
3
|
+
--- a/tasks/bin/cross-ruby.rake
|
4
|
+
+++ b/tasks/bin/cross-ruby.rake
|
5
|
+
@@ -116,10 +116,22 @@ RUBY_CC_VERSIONS.split(":").each do |ruby_cc_version|
|
6
|
+
"--host=#{mingw_host}",
|
7
|
+
"--target=#{mingw_target}",
|
8
|
+
"--build=#{RUBY_BUILD}",
|
9
|
+
- '--enable-shared',
|
10
|
+
+ '--enable-install-static-library',
|
11
|
+
+ '--disable-jit-support',
|
12
|
+
'--disable-install-doc',
|
13
|
+
'--with-ext=',
|
14
|
+
]
|
15
|
+
+ if mingw_host=~/darwin/
|
16
|
+
+ options += [
|
17
|
+
+ '--enable-static',
|
18
|
+
+ '--disable-shared',
|
19
|
+
+ ]
|
20
|
+
+ else
|
21
|
+
+ options += [
|
22
|
+
+ '--enable-static',
|
23
|
+
+ '--enable-shared',
|
24
|
+
+ ]
|
25
|
+
+ end
|
26
|
+
|
27
|
+
# Force Winsock2 for Ruby 1.8, 1.9 defaults to it
|
28
|
+
options << "--with-winsock2" if major == "1.8"
|
data/build/rcd-env.sh
ADDED
data/build/runas
CHANGED
@@ -4,9 +4,4 @@ groupadd -o -g "$GID" "$GROUP"
|
|
4
4
|
mkdir -p /tmp/home
|
5
5
|
useradd -o -g "$GID" -u "$UID" -G rvm,sudo -p "" -b /tmp/home -m "$USER"
|
6
6
|
|
7
|
-
|
8
|
-
ln -s /usr/local/rake-compiler "$HOME"/.rake-compiler
|
9
|
-
|
10
|
-
sudo -u "$USER" --set-home \
|
11
|
-
BASH_ENV=/etc/rubybashrc \
|
12
|
-
-- "$@"
|
7
|
+
/usr/bin/sudo -u "$USER" -H BASH_ENV=/etc/rubybashrc -- "$@"
|
data/build/sudoers
CHANGED
@@ -1 +1 @@
|
|
1
|
-
Defaults env_keep += "http_proxy https_proxy ftp_proxy RCD_HOST_RUBY_PLATFORM RCD_HOST_RUBY_VERSION RCD_IMAGE RUBY_CC_VERSION"
|
1
|
+
Defaults env_keep += "http_proxy https_proxy ftp_proxy GEM_PRIVATE_KEY_PASSPHRASE RCD_HOST_RUBY_PLATFORM RCD_HOST_RUBY_VERSION RCD_IMAGE RUBY_CC_VERSION LD_LIBRARY_PATH DEVTOOLSET_ROOTPATH"
|
@@ -8,6 +8,7 @@ module RakeCompilerDock
|
|
8
8
|
|
9
9
|
attr_reader :io
|
10
10
|
attr_reader :pwd
|
11
|
+
attr_reader :docker_command
|
11
12
|
attr_accessor :machine_name
|
12
13
|
|
13
14
|
def initialize(io, pwd, machine_name="rake-compiler-dock")
|
@@ -53,12 +54,17 @@ module RakeCompilerDock
|
|
53
54
|
end
|
54
55
|
end
|
55
56
|
|
57
|
+
COMMANDS = %w[docker podman]
|
56
58
|
def docker_version
|
57
|
-
|
59
|
+
COMMANDS.find do |command|
|
60
|
+
@docker_version_text, @docker_version_status = run("#{command} version")
|
61
|
+
@docker_command = command
|
62
|
+
@docker_version_status == 0
|
63
|
+
end
|
58
64
|
end
|
59
65
|
|
60
66
|
def ok?
|
61
|
-
@docker_version_status == 0 && @docker_version_text =~ /version/ && doma_pwd_ok?
|
67
|
+
@docker_version_status == 0 && @docker_version_text =~ /version/i && doma_pwd_ok?
|
62
68
|
end
|
63
69
|
|
64
70
|
def docker_client_avail?
|
@@ -216,16 +222,16 @@ module RakeCompilerDock
|
|
216
222
|
help << yellow("Please download and install the docker-toolbox:")
|
217
223
|
help << yellow(" https://www.docker.com/docker-toolbox")
|
218
224
|
when /linux/
|
219
|
-
help << red("Docker is
|
225
|
+
help << red("Neither Docker nor Podman is available.")
|
220
226
|
help << ""
|
221
|
-
help << yellow("Install on Ubuntu/Debian:")
|
227
|
+
help << yellow("Install Docker on Ubuntu/Debian:")
|
222
228
|
help << " sudo apt-get install docker.io"
|
223
229
|
help << ""
|
224
|
-
help << yellow("Install on Fedora/Centos/RHEL")
|
230
|
+
help << yellow("Install Docker on Fedora/Centos/RHEL")
|
225
231
|
help << " sudo yum install docker"
|
226
232
|
help << " sudo systemctl start docker"
|
227
233
|
help << ""
|
228
|
-
help << yellow("Install on SuSE")
|
234
|
+
help << yellow("Install Docker on SuSE")
|
229
235
|
help << " sudo zypper install docker"
|
230
236
|
help << " sudo systemctl start docker"
|
231
237
|
when /darwin/
|