appengine 0.4.3 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/.yardopts +2 -1
- data/CHANGELOG.md +28 -0
- data/README.md +67 -17
- data/data/exec_standard_entrypoint.rb.erb +116 -0
- data/lib/appengine.rb +12 -5
- data/lib/appengine/env.rb +4 -5
- data/lib/appengine/exec.rb +627 -192
- data/lib/appengine/railtie.rb +4 -9
- data/lib/appengine/tasks.rb +219 -142
- data/lib/appengine/util/gcloud.rb +25 -17
- data/lib/appengine/version.rb +5 -5
- metadata +97 -22
- data/Rakefile +0 -42
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
|
-
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 8985bf9ca824b3567577328e9c844eabcab5691907b0eb8a3385c181c91d2f13
|
|
4
|
+
data.tar.gz: 04d8e4d73ee85d186798b69748191ec14423aecabfa1cbc5446d6e79e29b438c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: fa651f55f75abf3d21813891a34c26d9c27ffe0656f7b77e54f440a2bb8d747f751317327cdc250a72fc3e286e4a76b79ac4f44ace1f70f01d8d46d3a27866ec
|
|
7
|
+
data.tar.gz: a03743e4a99da424fb1c588f83767389c36b82ddcb5271cba0462a5c7433bd113d3db1b668159dad5520eaa7fcb11ffcb88694133a0f46feb557ac69ada0bab2
|
data/.yardopts
CHANGED
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,34 @@
|
|
|
2
2
|
|
|
3
3
|
This is the change history for the appengine gem.
|
|
4
4
|
|
|
5
|
+
## v0.6.0 (2020-12-02)
|
|
6
|
+
|
|
7
|
+
* Fix failure in the appengine:exec cloud_build strategy when App Engine doesn't provide the image
|
|
8
|
+
* Fix exception when appengine:exec is provided a shell command rather than a command array (tpbowden)
|
|
9
|
+
* Update stackdriver dependency to 0.20 and google-cloud-env dependency to 1.4
|
|
10
|
+
|
|
11
|
+
## v0.5.0 (2019-07-15)
|
|
12
|
+
|
|
13
|
+
* appengine:exec supports the App Engine standard environment.
|
|
14
|
+
* appengine:exec supports setting the project via `GAE_PROJECT`.
|
|
15
|
+
* Support for an alternate appengine:exec strategy for flexible environment apps that talk to a database via a private IP.
|
|
16
|
+
* Fix crash when the gcloud path includes directories with spaces.
|
|
17
|
+
* Escape `$` symbols in environment configs. (tpbowden)
|
|
18
|
+
|
|
19
|
+
## v0.4.6 (2018-09-17)
|
|
20
|
+
|
|
21
|
+
* Use gcloud builds submit instead of gcloud container builds submit. (tbpg)
|
|
22
|
+
* Update stackdriver dependency to 0.15.
|
|
23
|
+
|
|
24
|
+
## v0.4.5 (2017-12-04)
|
|
25
|
+
|
|
26
|
+
* Ensure tempfile is required when needed.
|
|
27
|
+
* Update stackdriver dependency to 0.11.
|
|
28
|
+
|
|
29
|
+
## v0.4.4 (2017-10-03)
|
|
30
|
+
|
|
31
|
+
* Windows compatibility for appengine:exec task. (gkaykck)
|
|
32
|
+
|
|
5
33
|
## v0.4.3 (2017-09-20)
|
|
6
34
|
|
|
7
35
|
* Fixed incorrect namespace in the handler for gcloud errors.
|
data/README.md
CHANGED
|
@@ -1,6 +1,9 @@
|
|
|
1
1
|
Google App Engine Integration Tools
|
|
2
2
|
===================================
|
|
3
3
|
|
|
4
|
+
[](https://circleci.com/gh/GoogleCloudPlatform/appengine-ruby)
|
|
5
|
+
[](https://badge.fury.io/rb/appengine)
|
|
6
|
+
|
|
4
7
|
This repository contains the "appengine" gem, a collection of libraries and
|
|
5
8
|
plugins for integrating Ruby apps with Google App Engine. It is not required
|
|
6
9
|
for deploying a ruby application to Google App Engine, but it provides a
|
|
@@ -18,7 +21,7 @@ Currently, it includes:
|
|
|
18
21
|
* Convenient access to environment information such as project ID and VM
|
|
19
22
|
properties.
|
|
20
23
|
|
|
21
|
-
|
|
24
|
+
Potential future directions:
|
|
22
25
|
|
|
23
26
|
* Tools for generating "app.yaml" configuration files for Ruby applications.
|
|
24
27
|
* Streamlined implementation of health checks and other lifecycle hooks.
|
|
@@ -32,6 +35,10 @@ To install, include the "appengine" gem in your Gemfile. e.g.
|
|
|
32
35
|
|
|
33
36
|
gem "appengine"
|
|
34
37
|
|
|
38
|
+
## Quick Start
|
|
39
|
+
|
|
40
|
+
### Rails Quick Start
|
|
41
|
+
|
|
35
42
|
If you are running [Ruby On Rails](http://rubyonrails.org/) 4.0 or later, this
|
|
36
43
|
gem will automatically install a Railtie that provides its capabilities. You
|
|
37
44
|
may need to include the line:
|
|
@@ -41,10 +48,48 @@ may need to include the line:
|
|
|
41
48
|
in your `config/application.rb` file if you aren't already requiring all
|
|
42
49
|
bundled gems.
|
|
43
50
|
|
|
44
|
-
|
|
45
|
-
|
|
51
|
+
### Rack Quick Start
|
|
52
|
+
|
|
53
|
+
If you are running a different Rack-based web framework, include the following
|
|
54
|
+
line in your main Ruby file or `config.ru`:
|
|
55
|
+
|
|
56
|
+
require "appengine"
|
|
57
|
+
|
|
58
|
+
Then, to activate Stackdriver instrumentation, add the following middleware:
|
|
59
|
+
|
|
60
|
+
use Google::Cloud::Logging::Middleware
|
|
61
|
+
use Google::Cloud::ErrorReporting::Middleware
|
|
62
|
+
use Google::Cloud::Trace::Middleware
|
|
63
|
+
use Google::Cloud::Debugger::Middleware
|
|
64
|
+
|
|
65
|
+
You can add the Rake tasks to your application by adding the following to your
|
|
66
|
+
Rakefile:
|
|
67
|
+
|
|
68
|
+
require "appengine/tasks"
|
|
69
|
+
|
|
70
|
+
### Setting up appengine:exec remote execution
|
|
71
|
+
|
|
72
|
+
This gem is commonly used for its `appengine:exec` Rake task that provides a
|
|
73
|
+
way to run production tasks such as database migrations in the cloud. If you
|
|
74
|
+
are getting started with this feature, you should read the documentation
|
|
75
|
+
(available on the
|
|
76
|
+
[AppEngine::Exec module](http://www.rubydoc.info/gems/appengine/AppEngine/Exec))
|
|
77
|
+
carefully, for important tips. In particular:
|
|
78
|
+
|
|
79
|
+
* The strategy used by the gem is different depending on whether your app is
|
|
80
|
+
deployed to the App Engine standard environment or flexible environment.
|
|
81
|
+
It is important to understand which strategy is in use, because it affects
|
|
82
|
+
which version of your application code is used to run the task, and various
|
|
83
|
+
other factors.
|
|
84
|
+
* You may need to grant additional permissions to the service account that
|
|
85
|
+
runs the task. Again, the documentation will describe this in detail.
|
|
86
|
+
* If your app is running on the flexible environment and uses a VPC (and
|
|
87
|
+
connects to your database via a private IP address), then you will need to
|
|
88
|
+
use a special configuration for the task.
|
|
89
|
+
|
|
90
|
+
## Using this library
|
|
46
91
|
|
|
47
|
-
|
|
92
|
+
### Logging and monitoring
|
|
48
93
|
|
|
49
94
|
This library automatically installs the "stackdriver" gem, which instruments
|
|
50
95
|
your application to report logs, unhandled exceptions, and latency traces to
|
|
@@ -54,16 +99,21 @@ monitoring features of Google App Engine, see:
|
|
|
54
99
|
* [google-cloud-logging instrumentation](http://googlecloudplatform.github.io/google-cloud-ruby/#/docs/google-cloud-logging/latest/guides/instrumentation)
|
|
55
100
|
* [google-cloud-error_reporting instrumentation](http://googlecloudplatform.github.io/google-cloud-ruby/#/docs/google-cloud-error_reporting/latest/guides/instrumentation)
|
|
56
101
|
* [google-cloud-trace instrumentation](http://googlecloudplatform.github.io/google-cloud-ruby/#/docs/google-cloud-trace/latest/guides/instrumentation)
|
|
102
|
+
* [google-cloud-debugger instrumentation](http://googlecloudplatform.github.io/google-cloud-ruby/#/docs/google-cloud-debugger/latest/guides/instrumentation)
|
|
57
103
|
|
|
58
104
|
Rails applications automatically activate this instrumentation when the gem
|
|
59
105
|
is present. You may opt out of individual services by providing appropriate
|
|
60
|
-
Rails configuration. See
|
|
106
|
+
Rails configuration. See
|
|
107
|
+
[AppEngine::Railtie](http://www.rubydoc.info/gems/appengine/AppEngine/Railtie)
|
|
108
|
+
for more information.
|
|
61
109
|
|
|
62
110
|
Non-Rails applications must provide initialization code to activate this
|
|
63
|
-
instrumentation, typically by installing a Rack middleware.
|
|
64
|
-
|
|
111
|
+
instrumentation, typically by installing a Rack middleware. You can find the
|
|
112
|
+
basic code for installing these middlewares in the Rack Quick Start section
|
|
113
|
+
above. See the individual service documentation links above for more
|
|
114
|
+
information and configuration options.
|
|
65
115
|
|
|
66
|
-
|
|
116
|
+
### App Engine remote execution
|
|
67
117
|
|
|
68
118
|
This library provides rake tasks for App Engine remote execution, allowing
|
|
69
119
|
App Engine applications to perform on-demand tasks in the App Engine
|
|
@@ -73,17 +123,17 @@ example, you could run a production database migration in a Rails app using:
|
|
|
73
123
|
|
|
74
124
|
bundle exec rake appengine:exec -- bundle exec rake db:migrate
|
|
75
125
|
|
|
76
|
-
The migration would be run in
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
is often much easier and safer than running the task on a local workstation and
|
|
80
|
-
granting that workstation direct access to those Cloud SQL instances.
|
|
126
|
+
The migration would be run in containers on Google Cloud infrastructure, which
|
|
127
|
+
is much easier and safer than running the task on a local workstation and
|
|
128
|
+
granting that workstation direct access to your production database.
|
|
81
129
|
|
|
82
|
-
See
|
|
130
|
+
See [AppEngine::Exec](http://www.rubydoc.info/gems/appengine/AppEngine/Exec)
|
|
131
|
+
for more information on App Engine remote execution.
|
|
83
132
|
|
|
84
|
-
See
|
|
85
|
-
|
|
86
|
-
|
|
133
|
+
See [AppEngine::Tasks](http://www.rubydoc.info/gems/appengine/AppEngine/Tasks)
|
|
134
|
+
for more information on running the Rake tasks. The tasks are available
|
|
135
|
+
automatically in Rails applications when the gem is present. Non-Rails
|
|
136
|
+
applications may install the tasks by adding the line
|
|
87
137
|
`require "appengine/tasks"` to the `Rakefile`.
|
|
88
138
|
|
|
89
139
|
## Development and support
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
# Copyright 2019 Google LLC
|
|
2
|
+
#
|
|
3
|
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
4
|
+
# you may not use this file except in compliance with the License.
|
|
5
|
+
# You may obtain a copy of the License at
|
|
6
|
+
#
|
|
7
|
+
# http://www.apache.org/licenses/LICENSE-2.0
|
|
8
|
+
#
|
|
9
|
+
# Unless required by applicable law or agreed to in writing, software
|
|
10
|
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
11
|
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
12
|
+
# See the License for the specific language governing permissions and
|
|
13
|
+
# limitations under the License.
|
|
14
|
+
|
|
15
|
+
require "webrick"
|
|
16
|
+
require "monitor"
|
|
17
|
+
require "json"
|
|
18
|
+
|
|
19
|
+
SECRET = <%= secret %>
|
|
20
|
+
COMMAND = Array(<%= command %>)
|
|
21
|
+
|
|
22
|
+
port = Integer ENV["PORT"]
|
|
23
|
+
server = ::WEBrick::HTTPServer.new Port: port
|
|
24
|
+
|
|
25
|
+
Status = ::Struct.new :out_lines, :err_lines, :exit_status, :pid, :start_time
|
|
26
|
+
$status = Status.new [], [], nil, nil, nil
|
|
27
|
+
$status.extend ::MonitorMixin
|
|
28
|
+
|
|
29
|
+
def do_start
|
|
30
|
+
$status.synchronize do
|
|
31
|
+
return unless $status.pid.nil?
|
|
32
|
+
end
|
|
33
|
+
$stdout.puts "Executing: #{COMMAND.inspect}"
|
|
34
|
+
$stdout.flush
|
|
35
|
+
rout, wout = ::IO.pipe
|
|
36
|
+
rerr, werr = ::IO.pipe
|
|
37
|
+
::Thread.new do
|
|
38
|
+
rout.each_line do |line|
|
|
39
|
+
$status.synchronize { $status.out_lines << line }
|
|
40
|
+
$stdout.puts line
|
|
41
|
+
$stdout.flush
|
|
42
|
+
end
|
|
43
|
+
end
|
|
44
|
+
::Thread.new do
|
|
45
|
+
rerr.each_line do |line|
|
|
46
|
+
$status.synchronize { $status.err_lines << line }
|
|
47
|
+
$stderr.puts line
|
|
48
|
+
$stderr.flush
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
start_time = Time.now.to_i
|
|
52
|
+
pid = ::Process.spawn *COMMAND, err: werr, out: wout
|
|
53
|
+
werr.close
|
|
54
|
+
wout.close
|
|
55
|
+
$status.synchronize do
|
|
56
|
+
$status.pid = pid
|
|
57
|
+
$status.start_time = start_time
|
|
58
|
+
end
|
|
59
|
+
::Thread.new do
|
|
60
|
+
_pid, status = ::Process.wait2 pid
|
|
61
|
+
$status.synchronize do
|
|
62
|
+
$status.exit_status = status.exitstatus
|
|
63
|
+
end
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def get_status outpos: 0, errpos: 0
|
|
68
|
+
outlines, errlines, status, start_time =
|
|
69
|
+
$status.synchronize do
|
|
70
|
+
[
|
|
71
|
+
$status.out_lines[outpos..-1],
|
|
72
|
+
$status.err_lines[errpos..-1],
|
|
73
|
+
$status.exit_status,
|
|
74
|
+
$status.start_time
|
|
75
|
+
]
|
|
76
|
+
end
|
|
77
|
+
{
|
|
78
|
+
"outpos" => outpos + outlines.size,
|
|
79
|
+
"errpos" => errpos + errlines.size,
|
|
80
|
+
"outlines" => outlines,
|
|
81
|
+
"errlines" => errlines,
|
|
82
|
+
"status" => status,
|
|
83
|
+
"time" => Time.now.to_i - start_time
|
|
84
|
+
}
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
server.mount_proc "/_ah/start" do |req, res|
|
|
88
|
+
do_start
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
server.mount_proc "/#{SECRET}" do |req, res|
|
|
92
|
+
do_start
|
|
93
|
+
status = get_status outpos: req.query["outpos"].to_i,
|
|
94
|
+
errpos: req.query["errpos"].to_i
|
|
95
|
+
res.body = JSON.dump status
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
server.mount_proc "/#{SECRET}/kill" do |req, res|
|
|
99
|
+
unless req.request_method == "POST"
|
|
100
|
+
res.status = 404
|
|
101
|
+
return
|
|
102
|
+
end
|
|
103
|
+
$status.synchronize do
|
|
104
|
+
if $status.pid.nil?
|
|
105
|
+
res.status = 404
|
|
106
|
+
return
|
|
107
|
+
end
|
|
108
|
+
::Process.kill "SIGTERM", $status.pid
|
|
109
|
+
end
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
begin
|
|
113
|
+
server.start
|
|
114
|
+
ensure
|
|
115
|
+
server.shutdown
|
|
116
|
+
end
|
data/lib/appengine.rb
CHANGED
|
@@ -1,4 +1,6 @@
|
|
|
1
|
-
#
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Copyright 2019 Google LLC
|
|
2
4
|
#
|
|
3
5
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
4
6
|
# you may not use this file except in compliance with the License.
|
|
@@ -11,17 +13,22 @@
|
|
|
11
13
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
12
14
|
# See the License for the specific language governing permissions and
|
|
13
15
|
# limitations under the License.
|
|
14
|
-
;
|
|
15
16
|
|
|
16
|
-
|
|
17
|
+
|
|
18
|
+
##
|
|
19
|
+
# ## Google AppEngine integration
|
|
17
20
|
#
|
|
18
21
|
# The AppEngine module includes optional tools helping Ruby applications to
|
|
19
22
|
# integrate more closely with the Google App Engine environment.
|
|
20
|
-
|
|
23
|
+
#
|
|
21
24
|
module AppEngine
|
|
25
|
+
##
|
|
26
|
+
# Internal utilities
|
|
27
|
+
#
|
|
28
|
+
module Util
|
|
29
|
+
end
|
|
22
30
|
end
|
|
23
31
|
|
|
24
|
-
|
|
25
32
|
require "appengine/version"
|
|
26
33
|
require "appengine/env"
|
|
27
34
|
require "appengine/exec"
|
data/lib/appengine/env.rb
CHANGED
|
@@ -1,4 +1,6 @@
|
|
|
1
|
-
#
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Copyright 2019 Google LLC
|
|
2
4
|
#
|
|
3
5
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
4
6
|
# you may not use this file except in compliance with the License.
|
|
@@ -11,13 +13,11 @@
|
|
|
11
13
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
12
14
|
# See the License for the specific language governing permissions and
|
|
13
15
|
# limitations under the License.
|
|
14
|
-
;
|
|
15
16
|
|
|
16
|
-
require "google/cloud/env"
|
|
17
17
|
|
|
18
|
+
require "google/cloud/env"
|
|
18
19
|
|
|
19
20
|
module AppEngine
|
|
20
|
-
|
|
21
21
|
##
|
|
22
22
|
# A convenience object that provides information on the Google Cloud
|
|
23
23
|
# hosting environment. For example, you can call
|
|
@@ -34,5 +34,4 @@ module AppEngine
|
|
|
34
34
|
# directly instead. See the documentation for the `google-cloud-env` gem.
|
|
35
35
|
#
|
|
36
36
|
Env = ::Google::Cloud.env
|
|
37
|
-
|
|
38
37
|
end
|
data/lib/appengine/exec.rb
CHANGED
|
@@ -1,4 +1,6 @@
|
|
|
1
|
-
#
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Copyright 2019 Google LLC
|
|
2
4
|
#
|
|
3
5
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
4
6
|
# you may not use this file except in compliance with the License.
|
|
@@ -11,16 +13,20 @@
|
|
|
11
13
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
12
14
|
# See the License for the specific language governing permissions and
|
|
13
15
|
# limitations under the License.
|
|
14
|
-
;
|
|
15
16
|
|
|
16
|
-
|
|
17
|
+
|
|
18
|
+
require "date"
|
|
19
|
+
require "erb"
|
|
17
20
|
require "json"
|
|
21
|
+
require "net/http"
|
|
22
|
+
require "securerandom"
|
|
23
|
+
require "shellwords"
|
|
24
|
+
require "tempfile"
|
|
25
|
+
require "yaml"
|
|
18
26
|
|
|
19
27
|
require "appengine/util/gcloud"
|
|
20
28
|
|
|
21
|
-
|
|
22
29
|
module AppEngine
|
|
23
|
-
|
|
24
30
|
##
|
|
25
31
|
# # App Engine remote execution
|
|
26
32
|
#
|
|
@@ -31,90 +37,219 @@ module AppEngine
|
|
|
31
37
|
#
|
|
32
38
|
# ## About App Engine execution
|
|
33
39
|
#
|
|
34
|
-
# App Engine execution spins up
|
|
35
|
-
#
|
|
36
|
-
#
|
|
37
|
-
#
|
|
38
|
-
#
|
|
40
|
+
# App Engine execution spins up a one-off copy of an App Engine app, and runs
|
|
41
|
+
# a command against it. For example, if your app runs on Ruby on Rails, then
|
|
42
|
+
# you might use App Engine execution to run a command such as
|
|
43
|
+
# `bundle exec bin/rails db:migrate` in production infrastructure (to avoid
|
|
44
|
+
# having to connect directly to your production database from a local
|
|
45
|
+
# workstation).
|
|
46
|
+
#
|
|
47
|
+
# App Engine execution provides two strategies for generating that "one-off
|
|
48
|
+
# copy":
|
|
49
|
+
#
|
|
50
|
+
# * A `deployment` strategy, which deploys a temporary version of your app
|
|
51
|
+
# to a single backend instance and runs the command there.
|
|
52
|
+
# * A `cloud_build` strategy, which deploys your application image to
|
|
53
|
+
# Google Cloud Build and runs the command there.
|
|
39
54
|
#
|
|
40
|
-
#
|
|
41
|
-
#
|
|
55
|
+
# Both strategies are generally designed to emulate the App Engine runtime
|
|
56
|
+
# environment on cloud VMs similar to those used by actual deployments of
|
|
57
|
+
# your app. Both provide your application code and environment variables, and
|
|
58
|
+
# both provide access to Cloud SQL connections used by your app. However,
|
|
59
|
+
# they differ in what *version* of your app code they run against, and in
|
|
60
|
+
# certain other constraints and performance characteristics. More detailed
|
|
61
|
+
# information on using the two strategies is provided in the sections below.
|
|
42
62
|
#
|
|
43
|
-
#
|
|
44
|
-
#
|
|
45
|
-
#
|
|
63
|
+
# Apps deployed to the App Engine *flexible environment* will use the
|
|
64
|
+
# `cloud_build` strategy by default. However, you can force an app to use the
|
|
65
|
+
# `deployment` strategy instead. (You might do so if you need to connect to a
|
|
66
|
+
# Cloud SQL database on a VPC using a private IP, because the `cloud_build`
|
|
67
|
+
# strategy does not support private IPs.) To force use of `deployment`, set
|
|
68
|
+
# the `strategy` parameter in the {AppEngine::Exec} constructor (or the
|
|
69
|
+
# corresponding `GAE_EXEC_STRATEGY` parameter in the Rake task). Note that
|
|
70
|
+
# the `deployment` strategy is usually significantly slower than
|
|
71
|
+
# `cloud_build` for apps in the flexible environment.
|
|
46
72
|
#
|
|
47
|
-
#
|
|
48
|
-
#
|
|
49
|
-
#
|
|
73
|
+
# Apps deployed to the App Engine *standard environment* will *always* use
|
|
74
|
+
# the `deployment` strategy. You cannot force use of the `cloud_build`
|
|
75
|
+
# strategy.
|
|
50
76
|
#
|
|
51
77
|
# ## Prerequisites
|
|
52
78
|
#
|
|
53
79
|
# To use App Engine remote execution, you will need:
|
|
54
80
|
#
|
|
55
81
|
# * An app deployed to Google App Engine, of course!
|
|
56
|
-
# * The gcloud SDK
|
|
82
|
+
# * The [gcloud SDK](https://cloud.google.com/sdk/) installed and configured.
|
|
57
83
|
# * The `appengine` gem.
|
|
58
84
|
#
|
|
59
|
-
# You may also need to grant the Cloud Container Builder service account
|
|
60
|
-
# any permissions needed by your command. Often, Project Editor permissions
|
|
61
|
-
# will be sufficient for most tasks. You can find the service account
|
|
62
|
-
# configuration in the IAM tab in the Cloud Console under the name
|
|
63
|
-
# `[your-project-number]@cloudbuild.gserviceaccount.com`.
|
|
64
|
-
#
|
|
65
85
|
# You may use the `AppEngine::Exec` class to run commands directly. However,
|
|
66
86
|
# in most cases, it will be easier to run commands via the provided rake
|
|
67
|
-
# tasks
|
|
87
|
+
# tasks (see {AppEngine::Tasks}).
|
|
88
|
+
#
|
|
89
|
+
# ## Using the "deployment" strategy
|
|
90
|
+
#
|
|
91
|
+
# The `deployment` strategy deploys a temporary version of your app to a
|
|
92
|
+
# single backend App Engine instance, runs the command there, and then
|
|
93
|
+
# deletes the temporary version when it is finished.
|
|
94
|
+
#
|
|
95
|
+
# This is the default strategy (and indeed the only option) for apps running
|
|
96
|
+
# on the App Engine standard environment. It can also be used for flexible
|
|
97
|
+
# environment apps, but this is not commonly done because deployment of
|
|
98
|
+
# flexible environment apps can take a long time.
|
|
99
|
+
#
|
|
100
|
+
# Because the `deployment` strategy deploys a temporary version of your app,
|
|
101
|
+
# it runs against the *current application code* present where the command
|
|
102
|
+
# was initiated (i.e. the code currently on your workstation if you run the
|
|
103
|
+
# rake task from your workstation, or the current code on the branch if you
|
|
104
|
+
# are running from a CI/CD system.) This may be different from the code
|
|
105
|
+
# actually running in production, so it is important that you run from a
|
|
106
|
+
# compatible code branch.
|
|
68
107
|
#
|
|
69
|
-
#
|
|
108
|
+
# ### Specifying the host application
|
|
70
109
|
#
|
|
71
|
-
#
|
|
72
|
-
# to
|
|
110
|
+
# The `deployment` strategy works by deploying a temporary version of your
|
|
111
|
+
# app, so that it has access to your app's project and settings in App
|
|
112
|
+
# Engine. In most cases, it can determine this automatically, but depending
|
|
113
|
+
# on how your app or environment is structured, you may need to give it some
|
|
114
|
+
# help.
|
|
73
115
|
#
|
|
74
|
-
#
|
|
75
|
-
#
|
|
76
|
-
#
|
|
77
|
-
#
|
|
78
|
-
# created deployment version for that service. That deployment version then
|
|
79
|
-
# provides the application image that runs your command.
|
|
116
|
+
# By default, your Google Cloud project is taken from the current gcloud
|
|
117
|
+
# project. If you need to override this, set the `:project` parameter in the
|
|
118
|
+
# {AppEngine::Exec} constructor (or the corresponding `GAE_PROJECT`
|
|
119
|
+
# parameter in the Rake task).
|
|
80
120
|
#
|
|
81
|
-
#
|
|
82
|
-
#
|
|
83
|
-
#
|
|
84
|
-
#
|
|
85
|
-
#
|
|
121
|
+
# By default, the service name is taken from the App Engine config file.
|
|
122
|
+
# App Engine execution will assume this file is called `app.yaml` in the
|
|
123
|
+
# current directory. To use a different config file, set the `config_path`
|
|
124
|
+
# parameter in the {AppEngine::Exec} constructor (or the corresponding
|
|
125
|
+
# `GAE_CONFIG` parameter in the Rake task). You may also set the service name
|
|
126
|
+
# directly, using the `service` parameter (or `GAE_SERVICE` in Rake).
|
|
86
127
|
#
|
|
87
|
-
#
|
|
88
|
-
#
|
|
89
|
-
#
|
|
90
|
-
#
|
|
91
|
-
# by
|
|
128
|
+
# ### Providing credentials
|
|
129
|
+
#
|
|
130
|
+
# Your command will effectively be a deployment of your App Engine app
|
|
131
|
+
# itself, and will have access to the same credentials. For example, App
|
|
132
|
+
# Engine provides a service account by default for your app, or your app may
|
|
133
|
+
# be making use of its own service account key. In either case, make sure the
|
|
134
|
+
# service account has sufficient access for the command you want to run
|
|
135
|
+
# (such as database admin credentials).
|
|
136
|
+
#
|
|
137
|
+
# ### Other options
|
|
92
138
|
#
|
|
93
139
|
# You may also provide a timeout, which is the length of time that App
|
|
94
140
|
# Engine execution will allow your command to run before it is considered to
|
|
95
141
|
# have stalled and is terminated. The timeout should be a string of the form
|
|
96
142
|
# `2h15m10s`. The default is `10m`.
|
|
97
143
|
#
|
|
98
|
-
#
|
|
144
|
+
# The timeout is set via the `timeout` parameter to the {AppEngine::Exec}
|
|
145
|
+
# constructor, or by setting the `GAE_TIMEOUT` environment variable when
|
|
146
|
+
# invoking using Rake.
|
|
147
|
+
#
|
|
148
|
+
# ### Resource usage and billing
|
|
149
|
+
#
|
|
150
|
+
# The `deployment` strategy deploys to a temporary instance of your app in
|
|
151
|
+
# order to run the command. You may be billed for that usage. However, the
|
|
152
|
+
# cost should be minimal, because it will then immediately delete that
|
|
153
|
+
# instance in order to minimize usage.
|
|
154
|
+
#
|
|
155
|
+
# If you interrupt the execution (or it crashes), it is possible that the
|
|
156
|
+
# temporary instance may not get deleted properly. If you suspect this may
|
|
157
|
+
# have happened, go to the App Engine tab in the cloud console, under
|
|
158
|
+
# "versions" of your service, and delete the temporary version manually. It
|
|
159
|
+
# will have a name matching the pattern `appengine-exec-<timestamp>`.
|
|
160
|
+
#
|
|
161
|
+
# ## Using the "cloud_build" strategy
|
|
162
|
+
#
|
|
163
|
+
# The `cloud_build` strategy takes the application image that App Engine is
|
|
164
|
+
# actually using to run your app, and uses it to spin up a copy of your app
|
|
165
|
+
# in [Google Cloud Build](https://cloud.google.com/cloud-build) (along with
|
|
166
|
+
# an emulation layer that emulates certain App Engine services such as Cloud
|
|
167
|
+
# SQL connection sockets). The command then gets run in the Cloud Build
|
|
168
|
+
# environment.
|
|
169
|
+
#
|
|
170
|
+
# This is the default strategy for apps running on the App Engine flexible
|
|
171
|
+
# environment. (It is not available for standard environment apps.) Note that
|
|
172
|
+
# the `cloud_build` strategy cannot be used if your command needs to connect
|
|
173
|
+
# to a database over a [VPC](https://cloud.google.com/vpc/) private IP
|
|
174
|
+
# address. This is because it runs on virtual machines provided by the Cloud
|
|
175
|
+
# Build service, which are not part of your VPC. If your database can be
|
|
176
|
+
# accessed only over a private IP, you should use the `deployment` strategy
|
|
177
|
+
# instead.
|
|
178
|
+
#
|
|
179
|
+
# The Cloud Build log is output to the directory specified by
|
|
180
|
+
# "CLOUD_BUILD_GCS_LOG_DIR". (ex. "gs://BUCKET-NAME/FOLDER-NAME")
|
|
181
|
+
# By default, log directory name is
|
|
182
|
+
# "gs://[PROJECT_NUMBER].cloudbuild-logs.googleusercontent.com/".
|
|
183
|
+
#
|
|
184
|
+
# ### Specifying the host application
|
|
185
|
+
#
|
|
186
|
+
# The `cloud_build` strategy needs to know exactly which app, service, and
|
|
187
|
+
# version of your app, to identify the application image to use.
|
|
188
|
+
#
|
|
189
|
+
# By default, your Google Cloud project is taken from the current gcloud
|
|
190
|
+
# project. If you need to override this, set the `:project` parameter in the
|
|
191
|
+
# {AppEngine::Exec} constructor (or the corresponding `GAE_PROJECT`
|
|
192
|
+
# parameter in the Rake task).
|
|
193
|
+
#
|
|
194
|
+
# By default, the service name is taken from the App Engine config file.
|
|
195
|
+
# App Engine execution will assume this file is called `app.yaml` in the
|
|
196
|
+
# current directory. To use a different config file, set the `config_path`
|
|
197
|
+
# parameter in the {AppEngine::Exec} constructor (or the corresponding
|
|
198
|
+
# `GAE_CONFIG` parameter in the Rake task). You may also set the service name
|
|
199
|
+
# directly, using the `service` parameter (or `GAE_SERVICE` in Rake).
|
|
200
|
+
#
|
|
201
|
+
# By default, the image of the most recently deployed version of your app is
|
|
202
|
+
# used. (Note that this most recently deployed version may not be the same
|
|
203
|
+
# version that is currently receiving traffic: for example, if you deployed
|
|
204
|
+
# with `--no-promote`.) To use a different version, set the `version`
|
|
205
|
+
# parameter in the {AppEngine::Exec} constructor (or the corresponding
|
|
206
|
+
# `GAE_VERSION` parameter in the Rake task).
|
|
99
207
|
#
|
|
100
|
-
#
|
|
101
|
-
#
|
|
102
|
-
#
|
|
103
|
-
#
|
|
104
|
-
#
|
|
208
|
+
# ### Providing credentials
|
|
209
|
+
#
|
|
210
|
+
# By default, the `cloud_build` strategy uses your project's Cloud Build
|
|
211
|
+
# service account for its credentials. Unless your command provides its own
|
|
212
|
+
# service account key, you may need to grant the Cloud Build service account
|
|
213
|
+
# any permissions needed to execute your command (such as access to your
|
|
214
|
+
# database). For most tasks, it is sufficient to grant Project Editor
|
|
215
|
+
# permissions to the service account. You can find the service account
|
|
216
|
+
# configuration in the IAM tab in the Cloud Console under the name
|
|
217
|
+
# `[your-project-number]@cloudbuild.gserviceaccount.com`.
|
|
218
|
+
#
|
|
219
|
+
# ### Other options
|
|
220
|
+
#
|
|
221
|
+
# You may also provide a timeout, which is the length of time that App
|
|
222
|
+
# Engine execution will allow your command to run before it is considered to
|
|
223
|
+
# have stalled and is terminated. The timeout should be a string of the form
|
|
224
|
+
# `2h15m10s`. The default is `10m`.
|
|
225
|
+
#
|
|
226
|
+
# The timeout is set via the `timeout` parameter to the {AppEngine::Exec}
|
|
227
|
+
# constructor, or by setting the `GAE_TIMEOUT` environment variable when
|
|
228
|
+
# invoking using Rake.
|
|
229
|
+
#
|
|
230
|
+
# You can also set the wrapper image used to emulate the App Engine runtime
|
|
231
|
+
# environment, by setting the `wrapper_image` parameter to the constructor,
|
|
232
|
+
# or by setting the `GAE_EXEC_WRAPPER_IMAGE` environment variable. Generally,
|
|
233
|
+
# you will not need to do this unless you are testing a new wrapper image.
|
|
234
|
+
#
|
|
235
|
+
# ### Resource usage and billing
|
|
236
|
+
#
|
|
237
|
+
# The `cloud_build` strategy uses virtual machine resources provided by
|
|
238
|
+
# Google Cloud Build. Generally, a certain number of usage minutes per day is
|
|
239
|
+
# covered under a free tier, but additional compute usage beyond that time is
|
|
240
|
+
# billed to your Google Cloud account. For more details,
|
|
241
|
+
# see https://cloud.google.com/cloud-build/pricing
|
|
105
242
|
#
|
|
106
243
|
# If your command makes API calls or utilizes other cloud resources, you may
|
|
107
|
-
# also be billed for that usage. However,
|
|
108
|
-
# actual App Engine instances, and
|
|
109
|
-
# Engine instance usage.
|
|
244
|
+
# also be billed for that usage. However, the `cloud_build` strategy (unlike
|
|
245
|
+
# the `deployment` strategy) does not use actual App Engine instances, and
|
|
246
|
+
# you will not be billed for additional App Engine instance usage.
|
|
110
247
|
#
|
|
111
248
|
class Exec
|
|
112
|
-
|
|
113
|
-
@
|
|
114
|
-
@
|
|
115
|
-
@
|
|
116
|
-
@default_wrapper_image = "gcr.io/google-appengine/exec-wrapper".freeze
|
|
117
|
-
|
|
249
|
+
@default_timeout = "10m"
|
|
250
|
+
@default_service = "default"
|
|
251
|
+
@default_config_path = "./app.yaml"
|
|
252
|
+
@default_wrapper_image = "gcr.io/google-appengine/exec-wrapper:latest"
|
|
118
253
|
|
|
119
254
|
##
|
|
120
255
|
# Base class for exec-related usage errors.
|
|
@@ -122,6 +257,41 @@ module AppEngine
|
|
|
122
257
|
class UsageError < ::StandardError
|
|
123
258
|
end
|
|
124
259
|
|
|
260
|
+
##
|
|
261
|
+
# Unsupported strategy
|
|
262
|
+
#
|
|
263
|
+
class UnsupportedStrategy < UsageError
|
|
264
|
+
def initialize strategy, app_env
|
|
265
|
+
@strategy = strategy
|
|
266
|
+
@app_env = app_env
|
|
267
|
+
super "Strategy \"#{strategy}\" not supported for the #{app_env}" \
|
|
268
|
+
" environment"
|
|
269
|
+
end
|
|
270
|
+
attr_reader :strategy
|
|
271
|
+
attr_reader :app_env
|
|
272
|
+
end
|
|
273
|
+
|
|
274
|
+
##
|
|
275
|
+
# Exception raised when a parameter is malformed.
|
|
276
|
+
#
|
|
277
|
+
class BadParameter < UsageError
|
|
278
|
+
def initialize param, value
|
|
279
|
+
@param_name = param
|
|
280
|
+
@value = value
|
|
281
|
+
super "Bad value for #{param}: #{value}"
|
|
282
|
+
end
|
|
283
|
+
attr_reader :param_name
|
|
284
|
+
attr_reader :value
|
|
285
|
+
end
|
|
286
|
+
|
|
287
|
+
##
|
|
288
|
+
# Exception raised when gcloud has no default project.
|
|
289
|
+
#
|
|
290
|
+
class NoDefaultProject < UsageError
|
|
291
|
+
def initialize
|
|
292
|
+
super "No default project set."
|
|
293
|
+
end
|
|
294
|
+
end
|
|
125
295
|
|
|
126
296
|
##
|
|
127
297
|
# Exception raised when the App Engine config file could not be found.
|
|
@@ -150,7 +320,7 @@ module AppEngine
|
|
|
150
320
|
# versions at all could be found for the given service.
|
|
151
321
|
#
|
|
152
322
|
class NoSuchVersion < UsageError
|
|
153
|
-
def initialize service, version=nil
|
|
323
|
+
def initialize service, version = nil
|
|
154
324
|
@service = service
|
|
155
325
|
@version = version
|
|
156
326
|
if version
|
|
@@ -163,25 +333,7 @@ module AppEngine
|
|
|
163
333
|
attr_reader :version
|
|
164
334
|
end
|
|
165
335
|
|
|
166
|
-
##
|
|
167
|
-
# Exception raised when an explicitly-specified service name conflicts with
|
|
168
|
-
# a config-specified service name.
|
|
169
|
-
#
|
|
170
|
-
class ServiceNameConflict < UsageError
|
|
171
|
-
def initialize service_name, config_name, config_path
|
|
172
|
-
@service_name = service_name
|
|
173
|
-
@config_name = config_name
|
|
174
|
-
@config_path = config_path
|
|
175
|
-
super "Service name conflicts with config file"
|
|
176
|
-
end
|
|
177
|
-
attr_reader :service_name
|
|
178
|
-
attr_reader :config_name
|
|
179
|
-
attr_reader :config_path
|
|
180
|
-
end
|
|
181
|
-
|
|
182
|
-
|
|
183
336
|
class << self
|
|
184
|
-
|
|
185
337
|
## @return [String] Default command timeout.
|
|
186
338
|
attr_accessor :default_timeout
|
|
187
339
|
|
|
@@ -205,78 +357,138 @@ module AppEngine
|
|
|
205
357
|
# the service name from the config file.
|
|
206
358
|
# @param config_path [String,nil] App Engine config file to get the
|
|
207
359
|
# service name from if the service name is not provided directly.
|
|
208
|
-
#
|
|
209
|
-
#
|
|
210
|
-
#
|
|
211
|
-
#
|
|
212
|
-
#
|
|
213
|
-
#
|
|
360
|
+
# If omitted, defaults to the value returned by
|
|
361
|
+
# {AppEngine::Exec.default_config_path}.
|
|
362
|
+
# @param version [String,nil] Version string. If omitted, defaults to the
|
|
363
|
+
# most recently created version of the given service (which may not
|
|
364
|
+
# be the one currently receiving traffic).
|
|
365
|
+
# @param timeout [String,nil] Timeout string. If omitted, defaults to the
|
|
366
|
+
# value returned by {AppEngine::Exec.default_timeout}.
|
|
367
|
+
# @param wrapper_image [String,nil] The fully qualified name of the
|
|
368
|
+
# wrapper image to use. (Applies only to the "cloud_build" strategy.)
|
|
369
|
+
# @param strategy [String,nil] The execution strategy to use, or `nil` to
|
|
370
|
+
# choose a default based on the App Engine environment (flexible or
|
|
371
|
+
# standard). Allowed values are `nil`, `"deployment"` (which is the
|
|
372
|
+
# default for Standard), and `"cloud_build"` (which is the default
|
|
373
|
+
# for Flexible).
|
|
374
|
+
# @param gcs_log_dir [String,nil] GCS bucket name of the cloud build log
|
|
375
|
+
# when strategy is "cloud_build". (ex. "gs://BUCKET-NAME/FOLDER-NAME")
|
|
214
376
|
#
|
|
215
377
|
def new_rake_task name, args: [], env_args: [],
|
|
216
378
|
service: nil, config_path: nil, version: nil,
|
|
217
|
-
timeout: nil
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
if escaped_args.empty?
|
|
222
|
-
name_with_args = name
|
|
223
|
-
else
|
|
224
|
-
name_with_args = "#{name}[#{escaped_args.join ','}]"
|
|
379
|
+
timeout: nil, project: nil, wrapper_image: nil,
|
|
380
|
+
strategy: nil, gcs_log_dir: nil
|
|
381
|
+
escaped_args = args.map do |arg|
|
|
382
|
+
arg.gsub(/[,\[\]]/) { |m| "\\#{m}" }
|
|
225
383
|
end
|
|
384
|
+
name_with_args =
|
|
385
|
+
if escaped_args.empty?
|
|
386
|
+
name
|
|
387
|
+
else
|
|
388
|
+
"#{name}[#{escaped_args.join ','}]"
|
|
389
|
+
end
|
|
226
390
|
new ["bundle", "exec", "rake", name_with_args] + env_args,
|
|
227
391
|
service: service, config_path: config_path, version: version,
|
|
228
|
-
timeout: timeout
|
|
392
|
+
timeout: timeout, project: project, wrapper_image: wrapper_image,
|
|
393
|
+
strategy: strategy, gcs_log_dir: gcs_log_dir
|
|
229
394
|
end
|
|
230
|
-
|
|
231
395
|
end
|
|
232
396
|
|
|
233
|
-
|
|
234
397
|
##
|
|
235
398
|
# Create an execution for the given command.
|
|
236
399
|
#
|
|
237
400
|
# @param command [Array<String>] The command in array form.
|
|
401
|
+
# @param project [String,nil] ID of the project. If omitted, obtains
|
|
402
|
+
# the project from gcloud.
|
|
238
403
|
# @param service [String,nil] Name of the service. If omitted, obtains
|
|
239
404
|
# the service name from the config file.
|
|
240
405
|
# @param config_path [String,nil] App Engine config file to get the
|
|
241
406
|
# service name from if the service name is not provided directly.
|
|
242
|
-
#
|
|
243
|
-
#
|
|
244
|
-
#
|
|
245
|
-
#
|
|
246
|
-
#
|
|
247
|
-
#
|
|
407
|
+
# If omitted, defaults to the value returned by
|
|
408
|
+
# {AppEngine::Exec.default_config_path}.
|
|
409
|
+
# @param version [String,nil] Version string. If omitted, defaults to the
|
|
410
|
+
# most recently created version of the given service (which may not be
|
|
411
|
+
# the one currently receiving traffic).
|
|
412
|
+
# @param timeout [String,nil] Timeout string. If omitted, defaults to the
|
|
413
|
+
# value returned by {AppEngine::Exec.default_timeout}.
|
|
414
|
+
# @param wrapper_image [String,nil] The fully qualified name of the wrapper
|
|
415
|
+
# image to use. (Applies only to the "cloud_build" strategy.)
|
|
416
|
+
# @param strategy [String,nil] The execution strategy to use, or `nil` to
|
|
417
|
+
# choose a default based on the App Engine environment (flexible or
|
|
418
|
+
# standard). Allowed values are `nil`, `"deployment"` (which is the
|
|
419
|
+
# default for Standard), and `"cloud_build"` (which is the default for
|
|
420
|
+
# Flexible).
|
|
421
|
+
# @param gcs_log_dir [String,nil] GCS bucket name of the cloud build log
|
|
422
|
+
# when strategy is "cloud_build". (ex. "gs://BUCKET-NAME/FOLDER-NAME")
|
|
248
423
|
#
|
|
249
424
|
def initialize command,
|
|
250
|
-
|
|
425
|
+
project: nil, service: nil, config_path: nil, version: nil,
|
|
426
|
+
timeout: nil, wrapper_image: nil, strategy: nil, gcs_log_dir: nil
|
|
251
427
|
@command = command
|
|
252
428
|
@service = service
|
|
253
429
|
@config_path = config_path
|
|
254
430
|
@version = version
|
|
255
431
|
@timeout = timeout
|
|
256
|
-
@
|
|
432
|
+
@project = project
|
|
433
|
+
@wrapper_image = wrapper_image
|
|
434
|
+
@strategy = strategy
|
|
435
|
+
@gcs_log_dir = gcs_log_dir
|
|
257
436
|
|
|
258
437
|
yield self if block_given?
|
|
259
438
|
end
|
|
260
439
|
|
|
440
|
+
##
|
|
441
|
+
# @return [String] The project ID.
|
|
442
|
+
# @return [nil] if the default gcloud project should be used.
|
|
443
|
+
#
|
|
444
|
+
attr_accessor :project
|
|
261
445
|
|
|
262
|
-
##
|
|
446
|
+
##
|
|
447
|
+
# @return [String] The service name.
|
|
448
|
+
# @return [nil] if the service should be obtained from the app config.
|
|
449
|
+
#
|
|
263
450
|
attr_accessor :service
|
|
264
451
|
|
|
265
|
-
##
|
|
452
|
+
##
|
|
453
|
+
# @return [String] Path to the config file.
|
|
454
|
+
# @return [nil] if the default of `./app.yaml` should be used.
|
|
455
|
+
#
|
|
266
456
|
attr_accessor :config_path
|
|
267
457
|
|
|
268
|
-
##
|
|
458
|
+
##
|
|
459
|
+
# @return [String] Service version of the image to use.
|
|
460
|
+
# @return [nil] if the most recent should be used.
|
|
461
|
+
#
|
|
269
462
|
attr_accessor :version
|
|
270
463
|
|
|
271
|
-
##
|
|
464
|
+
##
|
|
465
|
+
# @return [String] The command timeout, in `1h23m45s` format.
|
|
466
|
+
# @return [nil] if the default of `10m` should be used.
|
|
467
|
+
#
|
|
272
468
|
attr_accessor :timeout
|
|
273
469
|
|
|
274
|
-
##
|
|
470
|
+
##
|
|
471
|
+
# The command to run.
|
|
472
|
+
#
|
|
473
|
+
# @return [String] if the command is a script to be run in a shell.
|
|
474
|
+
# @return [Array<String>] if the command is a posix command to be run
|
|
475
|
+
# directly without a shell.
|
|
476
|
+
#
|
|
275
477
|
attr_accessor :command
|
|
276
478
|
|
|
277
|
-
##
|
|
479
|
+
##
|
|
480
|
+
# @return [String] Custom wrapper image to use.
|
|
481
|
+
# @return [nil] if the default should be used.
|
|
482
|
+
#
|
|
278
483
|
attr_accessor :wrapper_image
|
|
279
484
|
|
|
485
|
+
##
|
|
486
|
+
# @return [String] The execution strategy to use. Allowed values are
|
|
487
|
+
# `"deployment"` and `"cloud_build"`.
|
|
488
|
+
# @return [nil] to choose a default based on the App Engine environment
|
|
489
|
+
# (flexible or standard).
|
|
490
|
+
#
|
|
491
|
+
attr_accessor :strategy
|
|
280
492
|
|
|
281
493
|
##
|
|
282
494
|
# Executes the command synchronously. Streams the logs back to standard out
|
|
@@ -284,29 +496,15 @@ module AppEngine
|
|
|
284
496
|
#
|
|
285
497
|
def start
|
|
286
498
|
resolve_parameters
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
config = build_config command, image, env_variables, cloud_sql_instances
|
|
295
|
-
file = ::Tempfile.new ["cloudbuild_", ".json"]
|
|
296
|
-
begin
|
|
297
|
-
::JSON.dump config, file
|
|
298
|
-
file.flush
|
|
299
|
-
Util::Gcloud.execute [
|
|
300
|
-
"container", "builds", "submit",
|
|
301
|
-
"--no-source",
|
|
302
|
-
"--config=#{file.path}",
|
|
303
|
-
"--timeout=#{@timeout}"]
|
|
304
|
-
ensure
|
|
305
|
-
file.close!
|
|
499
|
+
app_info = version_info @service, @version
|
|
500
|
+
resolve_strategy app_info["env"]
|
|
501
|
+
if @strategy == "cloud_build"
|
|
502
|
+
start_build_strategy app_info
|
|
503
|
+
else
|
|
504
|
+
start_deployment_strategy app_info
|
|
306
505
|
end
|
|
307
506
|
end
|
|
308
507
|
|
|
309
|
-
|
|
310
508
|
private
|
|
311
509
|
|
|
312
510
|
##
|
|
@@ -314,61 +512,55 @@ module AppEngine
|
|
|
314
512
|
# Resolves and canonicalizes all the parameters.
|
|
315
513
|
#
|
|
316
514
|
def resolve_parameters
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
config_service = config_path = nil
|
|
322
|
-
if @config_path || !@service
|
|
323
|
-
config_service = begin
|
|
324
|
-
config_path = @config_path || Exec.default_config_path
|
|
325
|
-
::YAML.load_file(config_path)["service"] || Exec.default_service
|
|
326
|
-
rescue ::Errno::ENOENT
|
|
327
|
-
raise ConfigFileNotFound.new config_path
|
|
328
|
-
rescue
|
|
329
|
-
raise BadConfigFileFormat.new config_path
|
|
330
|
-
end
|
|
331
|
-
end
|
|
332
|
-
if @service && config_service && @service != config_service
|
|
333
|
-
raise ServiceNameConflict.new @service, config_service, config_path
|
|
334
|
-
end
|
|
335
|
-
|
|
336
|
-
@service ||= config_service
|
|
515
|
+
@timestamp_suffix = ::Time.now.strftime "%Y%m%d%H%M%S"
|
|
516
|
+
@command = ::Shellwords.split @command.to_s unless @command.is_a? Array
|
|
517
|
+
@project ||= default_project
|
|
518
|
+
@service ||= service_from_config || Exec.default_service
|
|
337
519
|
@version ||= latest_version @service
|
|
338
520
|
@timeout ||= Exec.default_timeout
|
|
521
|
+
@timeout_seconds = parse_timeout @timeout
|
|
339
522
|
@wrapper_image ||= Exec.default_wrapper_image
|
|
523
|
+
self
|
|
340
524
|
end
|
|
341
525
|
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
# @param command [Array<String>] The command in array form.
|
|
347
|
-
# @param image [String] The fully qualified image path.
|
|
348
|
-
# @param env_variables[Hash<String,String>] Environment variables.
|
|
349
|
-
# @param cloud_sql_instances[String,Array<String>] Names of cloud sql
|
|
350
|
-
# instances to connect to.
|
|
351
|
-
#
|
|
352
|
-
def build_config command, image, env_variables, cloud_sql_instances
|
|
353
|
-
args = ["-i", image]
|
|
354
|
-
env_variables.each do |k, v|
|
|
355
|
-
args << "-e" << "#{k}=#{v}"
|
|
526
|
+
def resolve_strategy app_env
|
|
527
|
+
@strategy = @strategy.to_s.downcase
|
|
528
|
+
if @strategy.empty?
|
|
529
|
+
@strategy = app_env == "flexible" ? "cloud_build" : "deployment"
|
|
356
530
|
end
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
args << "-s" << sql
|
|
361
|
-
end
|
|
531
|
+
if app_env == "standard" && @strategy == "cloud_build" ||
|
|
532
|
+
@strategy != "cloud_build" && @strategy != "deployment"
|
|
533
|
+
raise UnsupportedStrategy.new @strategy, app_env
|
|
362
534
|
end
|
|
363
|
-
|
|
364
|
-
|
|
535
|
+
@strategy
|
|
536
|
+
end
|
|
365
537
|
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
538
|
+
def service_from_config
|
|
539
|
+
return nil if !@config_path && @service
|
|
540
|
+
@config_path ||= Exec.default_config_path
|
|
541
|
+
::YAML.load_file(config_path)["service"]
|
|
542
|
+
rescue ::Errno::ENOENT
|
|
543
|
+
raise ConfigFileNotFound, @config_path
|
|
544
|
+
rescue ::StandardError
|
|
545
|
+
raise BadConfigFileFormat, @config_path
|
|
546
|
+
end
|
|
547
|
+
|
|
548
|
+
def default_project
|
|
549
|
+
result = Util::Gcloud.execute \
|
|
550
|
+
["config", "get-value", "project"],
|
|
551
|
+
capture: true, assert: false
|
|
552
|
+
result.strip!
|
|
553
|
+
raise NoDefaultProject if result.empty?
|
|
554
|
+
result
|
|
555
|
+
end
|
|
556
|
+
|
|
557
|
+
def parse_timeout timeout_str
|
|
558
|
+
matched = timeout_str =~ /^(?:(\d+)h)?(?:(\d+)m)?(?:(\d+)s?)?$/
|
|
559
|
+
raise BadParameter.new "timeout", timeout_str unless matched
|
|
560
|
+
hours = ::Regexp.last_match(1).to_i
|
|
561
|
+
minutes = ::Regexp.last_match(2).to_i
|
|
562
|
+
seconds = ::Regexp.last_match(3).to_i
|
|
563
|
+
hours * 3600 + minutes * 60 + seconds
|
|
372
564
|
end
|
|
373
565
|
|
|
374
566
|
##
|
|
@@ -380,15 +572,18 @@ module AppEngine
|
|
|
380
572
|
# @return [String] Name of the most recent version.
|
|
381
573
|
#
|
|
382
574
|
def latest_version service
|
|
383
|
-
result = Util::Gcloud.execute
|
|
575
|
+
result = Util::Gcloud.execute \
|
|
576
|
+
[
|
|
384
577
|
"app", "versions", "list",
|
|
385
|
-
"--
|
|
386
|
-
"--
|
|
387
|
-
"--
|
|
388
|
-
"--
|
|
389
|
-
|
|
578
|
+
"--project", @project,
|
|
579
|
+
"--service", service,
|
|
580
|
+
"--format", "get(version.id)",
|
|
581
|
+
"--sort-by", "~version.createTime",
|
|
582
|
+
"--limit", "1"
|
|
583
|
+
],
|
|
584
|
+
capture: true, assert: false
|
|
390
585
|
result = result.split.first
|
|
391
|
-
raise NoSuchVersion
|
|
586
|
+
raise NoSuchVersion, service unless result
|
|
392
587
|
result
|
|
393
588
|
end
|
|
394
589
|
|
|
@@ -400,23 +595,263 @@ module AppEngine
|
|
|
400
595
|
# "default" is used.
|
|
401
596
|
# @param version [String] Name of the version. If omitted, the most
|
|
402
597
|
# recently deployed is used.
|
|
403
|
-
# @return [Hash
|
|
404
|
-
#
|
|
405
|
-
#
|
|
598
|
+
# @return [Hash] A collection of fields parsed from the JSON representation
|
|
599
|
+
# of the version
|
|
600
|
+
# @return [nil] if the requested version doesn't exist.
|
|
406
601
|
#
|
|
407
602
|
def version_info service, version
|
|
408
603
|
service ||= "default"
|
|
409
604
|
version ||= latest_version service
|
|
410
|
-
result = Util::Gcloud.execute
|
|
605
|
+
result = Util::Gcloud.execute \
|
|
606
|
+
[
|
|
411
607
|
"app", "versions", "describe", version,
|
|
412
|
-
"--
|
|
413
|
-
"--
|
|
414
|
-
|
|
608
|
+
"--project", @project,
|
|
609
|
+
"--service", service,
|
|
610
|
+
"--format", "json"
|
|
611
|
+
],
|
|
612
|
+
capture: true, assert: false
|
|
415
613
|
result.strip!
|
|
416
614
|
raise NoSuchVersion.new(service, version) if result.empty?
|
|
417
615
|
::JSON.parse result
|
|
418
616
|
end
|
|
419
617
|
|
|
420
|
-
|
|
618
|
+
##
|
|
619
|
+
# @private
|
|
620
|
+
# Performs exec on a GAE standard app.
|
|
621
|
+
#
|
|
622
|
+
def start_deployment_strategy app_info
|
|
623
|
+
describe_deployment_strategy
|
|
624
|
+
entrypoint_file = app_yaml_file = temp_version = nil
|
|
625
|
+
begin
|
|
626
|
+
puts "\n---------- DEPLOY COMMAND ----------"
|
|
627
|
+
secret = create_secret
|
|
628
|
+
entrypoint_file = copy_entrypoint secret
|
|
629
|
+
app_yaml_file = copy_app_yaml app_info, entrypoint_file
|
|
630
|
+
temp_version = deploy_temp_app app_yaml_file
|
|
631
|
+
puts "\n---------- EXECUTE COMMAND ----------"
|
|
632
|
+
puts "COMMAND: #{@command.inspect}\n\n"
|
|
633
|
+
exit_status = track_status temp_version, secret
|
|
634
|
+
puts "\nEXIT STATUS: #{exit_status}"
|
|
635
|
+
ensure
|
|
636
|
+
puts "\n---------- CLEANUP ----------"
|
|
637
|
+
::File.unlink entrypoint_file if entrypoint_file
|
|
638
|
+
::File.unlink app_yaml_file if app_yaml_file
|
|
639
|
+
delete_temp_version temp_version
|
|
640
|
+
end
|
|
641
|
+
end
|
|
642
|
+
|
|
643
|
+
def describe_deployment_strategy
|
|
644
|
+
puts "\nUsing the `deployment` strategy for appengine:exec"
|
|
645
|
+
puts "(i.e. deploying a temporary version of your app)"
|
|
646
|
+
puts "PROJECT: #{@project}"
|
|
647
|
+
puts "SERVICE: #{@service}"
|
|
648
|
+
puts "TIMEOUT: #{@timeout}"
|
|
649
|
+
end
|
|
650
|
+
|
|
651
|
+
def create_secret
|
|
652
|
+
::SecureRandom.alphanumeric 20
|
|
653
|
+
end
|
|
654
|
+
|
|
655
|
+
def copy_entrypoint secret
|
|
656
|
+
entrypoint_template =
|
|
657
|
+
::File.join(::File.dirname(::File.dirname(__dir__)),
|
|
658
|
+
"data", "exec_standard_entrypoint.rb.erb")
|
|
659
|
+
entrypoint_file = "appengine_exec_entrypoint_#{@timestamp_suffix}.rb"
|
|
660
|
+
erb = ::ERB.new ::File.read entrypoint_template
|
|
661
|
+
data = {
|
|
662
|
+
secret: secret.inspect, command: command.inspect
|
|
663
|
+
}
|
|
664
|
+
result = erb.result_with_hash data
|
|
665
|
+
::File.open entrypoint_file, "w" do |file|
|
|
666
|
+
file.write result
|
|
667
|
+
end
|
|
668
|
+
entrypoint_file
|
|
669
|
+
end
|
|
670
|
+
|
|
671
|
+
def copy_app_yaml app_info, entrypoint_file
|
|
672
|
+
yaml_data = {
|
|
673
|
+
"runtime" => app_info["runtime"],
|
|
674
|
+
"service" => @service,
|
|
675
|
+
"entrypoint" => "ruby #{entrypoint_file}",
|
|
676
|
+
"env_variables" => app_info["envVariables"],
|
|
677
|
+
"manual_scaling" => { "instances" => 1 }
|
|
678
|
+
}
|
|
679
|
+
if app_info["env"] == "flexible"
|
|
680
|
+
complete_flex_app_yaml yaml_data, app_info
|
|
681
|
+
else
|
|
682
|
+
complete_standard_app_yaml yaml_data, app_info
|
|
683
|
+
end
|
|
684
|
+
app_yaml_file = "appengine_exec_config_#{@timestamp_suffix}.yaml"
|
|
685
|
+
::File.open app_yaml_file, "w" do |file|
|
|
686
|
+
::Psych.dump yaml_data, file
|
|
687
|
+
end
|
|
688
|
+
app_yaml_file
|
|
689
|
+
end
|
|
690
|
+
|
|
691
|
+
def complete_flex_app_yaml yaml_data, app_info
|
|
692
|
+
yaml_data["env"] = "flex"
|
|
693
|
+
orig_path = (app_info["betaSettings"] || {})["module_yaml_path"]
|
|
694
|
+
return unless orig_path
|
|
695
|
+
orig_yaml = ::YAML.load_file orig_path
|
|
696
|
+
copy_keys = ["skip_files", "resources", "network", "runtime_config",
|
|
697
|
+
"beta_settings"]
|
|
698
|
+
copy_keys.each do |key|
|
|
699
|
+
yaml_data[key] = orig_yaml[key] if orig_yaml[key]
|
|
700
|
+
end
|
|
701
|
+
end
|
|
702
|
+
|
|
703
|
+
def complete_standard_app_yaml yaml_data, app_info
|
|
704
|
+
yaml_data["instance_class"] = app_info["instanceClass"].sub(/^F/, "B")
|
|
705
|
+
end
|
|
706
|
+
|
|
707
|
+
def deploy_temp_app app_yaml_file
|
|
708
|
+
temp_version = "appengine-exec-#{@timestamp_suffix}"
|
|
709
|
+
Util::Gcloud.execute [
|
|
710
|
+
"app", "deploy", app_yaml_file,
|
|
711
|
+
"--project", @project,
|
|
712
|
+
"--version", temp_version,
|
|
713
|
+
"--no-promote", "--quiet"
|
|
714
|
+
]
|
|
715
|
+
temp_version
|
|
716
|
+
end
|
|
717
|
+
|
|
718
|
+
def track_status temp_version, secret
|
|
719
|
+
host = "#{temp_version}.#{@service}.#{@project}.appspot.com"
|
|
720
|
+
::Net::HTTP.start host do |http|
|
|
721
|
+
outpos = errpos = 0
|
|
722
|
+
delay = 0.0
|
|
723
|
+
loop do
|
|
724
|
+
sleep delay
|
|
725
|
+
uri = URI("http://#{host}/#{secret}")
|
|
726
|
+
uri.query = ::URI.encode_www_form outpos: outpos, errpos: errpos
|
|
727
|
+
response = http.request_get uri
|
|
728
|
+
data = ::JSON.parse response.body
|
|
729
|
+
data["outlines"].each { |line| puts "[STDOUT] #{line}" }
|
|
730
|
+
data["errlines"].each { |line| puts "[STDERR] #{line}" }
|
|
731
|
+
outpos = data["outpos"]
|
|
732
|
+
errpos = data["errpos"]
|
|
733
|
+
return data["status"] if data["status"]
|
|
734
|
+
if data["time"] > @timeout_seconds
|
|
735
|
+
http.request_post "/#{secret}/kill", ""
|
|
736
|
+
return "timeout"
|
|
737
|
+
end
|
|
738
|
+
if data["outlines"].empty? && data["errlines"].empty?
|
|
739
|
+
delay += 0.1
|
|
740
|
+
delay = 1.0 if delay > 1.0
|
|
741
|
+
else
|
|
742
|
+
delay = 0.0
|
|
743
|
+
end
|
|
744
|
+
end
|
|
745
|
+
end
|
|
746
|
+
end
|
|
747
|
+
|
|
748
|
+
def delete_temp_version temp_version
|
|
749
|
+
Util::Gcloud.execute [
|
|
750
|
+
"app", "versions", "delete", temp_version,
|
|
751
|
+
"--project", @project,
|
|
752
|
+
"--service", @service,
|
|
753
|
+
"--quiet"
|
|
754
|
+
]
|
|
755
|
+
end
|
|
756
|
+
|
|
757
|
+
##
|
|
758
|
+
# @private
|
|
759
|
+
# Performs exec on a GAE flexible app.
|
|
760
|
+
#
|
|
761
|
+
def start_build_strategy app_info
|
|
762
|
+
env_variables = app_info["envVariables"] || {}
|
|
763
|
+
beta_settings = app_info["betaSettings"] || {}
|
|
764
|
+
cloud_sql_instances = beta_settings["cloud_sql_instances"] || []
|
|
765
|
+
container = app_info["deployment"]["container"]
|
|
766
|
+
image = container ? container["image"] : image_from_build(app_info)
|
|
767
|
+
|
|
768
|
+
describe_build_strategy
|
|
769
|
+
|
|
770
|
+
config = build_config command, image, env_variables, cloud_sql_instances
|
|
771
|
+
file = ::Tempfile.new ["cloudbuild_", ".json"]
|
|
772
|
+
begin
|
|
773
|
+
::JSON.dump config, file
|
|
774
|
+
file.flush
|
|
775
|
+
execute_command = [
|
|
776
|
+
"builds", "submit",
|
|
777
|
+
"--project", @project,
|
|
778
|
+
"--no-source",
|
|
779
|
+
"--config", file.path,
|
|
780
|
+
"--timeout", @timeout
|
|
781
|
+
]
|
|
782
|
+
execute_command.concat ["--gcs-log-dir", @gcs_log_dir] unless @gcs_log_dir.nil?
|
|
783
|
+
Util::Gcloud.execute execute_command
|
|
784
|
+
ensure
|
|
785
|
+
file.close!
|
|
786
|
+
end
|
|
787
|
+
end
|
|
788
|
+
|
|
789
|
+
##
|
|
790
|
+
# @private
|
|
791
|
+
# Workaround for https://github.com/GoogleCloudPlatform/appengine-ruby/issues/33
|
|
792
|
+
# Determines the image by looking it up in Cloud Build
|
|
793
|
+
#
|
|
794
|
+
def image_from_build app_info
|
|
795
|
+
create_time = ::DateTime.parse(app_info["createTime"]).to_time.utc
|
|
796
|
+
after_time = (create_time - 3600).strftime "%Y-%m-%dT%H:%M:%SZ"
|
|
797
|
+
before_time = (create_time + 3600).strftime "%Y-%m-%dT%H:%M:%SZ"
|
|
798
|
+
partial_uri = "gcr.io/#{@project}/appengine/#{@service}.#{@version}"
|
|
799
|
+
filter = "createTime>#{after_time} createTime<#{before_time} images[]:#{partial_uri}"
|
|
800
|
+
result = Util::Gcloud.execute \
|
|
801
|
+
[
|
|
802
|
+
"builds", "list",
|
|
803
|
+
"--project", @project,
|
|
804
|
+
"--filter", filter,
|
|
805
|
+
"--format", "json"
|
|
806
|
+
],
|
|
807
|
+
capture: true, assert: false
|
|
808
|
+
result.strip!
|
|
809
|
+
raise NoSuchVersion.new(@service, @version) if result.empty?
|
|
810
|
+
build_info = ::JSON.parse(result).first
|
|
811
|
+
build_info["images"].first
|
|
812
|
+
end
|
|
813
|
+
|
|
814
|
+
def describe_build_strategy
|
|
815
|
+
puts "\nUsing the `cloud_build` strategy for appengine:exec"
|
|
816
|
+
puts "(i.e. running your app image in Cloud Build)"
|
|
817
|
+
puts "PROJECT: #{@project}"
|
|
818
|
+
puts "SERVICE: #{@service}"
|
|
819
|
+
puts "VERSION: #{@version}"
|
|
820
|
+
puts "TIMEOUT: #{@timeout}"
|
|
821
|
+
puts ""
|
|
822
|
+
end
|
|
823
|
+
|
|
824
|
+
##
|
|
825
|
+
# @private
|
|
826
|
+
# Builds a cloudbuild config as a data structure.
|
|
827
|
+
#
|
|
828
|
+
# @param command [Array<String>] The command in array form.
|
|
829
|
+
# @param image [String] The fully qualified image path.
|
|
830
|
+
# @param env_variables [Hash<String,String>] Environment variables.
|
|
831
|
+
# @param cloud_sql_instances [String,Array<String>] Names of cloud sql
|
|
832
|
+
# instances to connect to.
|
|
833
|
+
#
|
|
834
|
+
def build_config command, image, env_variables, cloud_sql_instances
|
|
835
|
+
args = ["-i", image]
|
|
836
|
+
env_variables.each do |k, v|
|
|
837
|
+
v = v.gsub "$", "$$"
|
|
838
|
+
args << "-e" << "#{k}=#{v}"
|
|
839
|
+
end
|
|
840
|
+
unless cloud_sql_instances.empty?
|
|
841
|
+
cloud_sql_instances = Array(cloud_sql_instances)
|
|
842
|
+
cloud_sql_instances.each do |sql|
|
|
843
|
+
args << "-s" << sql
|
|
844
|
+
end
|
|
845
|
+
end
|
|
846
|
+
args << "--"
|
|
847
|
+
args += command
|
|
421
848
|
|
|
849
|
+
{
|
|
850
|
+
"steps" => [
|
|
851
|
+
"name" => @wrapper_image,
|
|
852
|
+
"args" => args
|
|
853
|
+
]
|
|
854
|
+
}
|
|
855
|
+
end
|
|
856
|
+
end
|
|
422
857
|
end
|