lambda_punch 0.0.1 → 0.0.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/workflows/test.yml +17 -0
- data/.gitignore +2 -1
- data/CHANGELOG.md +5 -1
- data/Dockerfile +2 -0
- data/Gemfile +2 -7
- data/Gemfile.lock +30 -0
- data/README.md +102 -19
- data/Rakefile +0 -2
- data/bin/_setup +6 -0
- data/bin/_test +4 -0
- data/bin/bootstrap +6 -0
- data/bin/console +5 -14
- data/bin/run +6 -0
- data/bin/setup +5 -7
- data/bin/test +6 -0
- data/docker-compose.yml +10 -0
- data/lambda_punch.gemspec +3 -1
- data/lib/lambda_punch.rb +47 -5
- data/lib/lambda_punch/api.rb +93 -0
- data/lib/lambda_punch/error.rb +7 -0
- data/lib/lambda_punch/extensions/lambdapunch +12 -0
- data/lib/lambda_punch/logger.rb +24 -0
- data/lib/lambda_punch/notifier.rb +49 -0
- data/lib/lambda_punch/queue.rb +41 -0
- data/lib/lambda_punch/railtie.rb +12 -0
- data/lib/lambda_punch/server.rb +26 -0
- data/lib/lambda_punch/tasks/install.rake +11 -0
- data/lib/lambda_punch/version.rb +1 -3
- data/lib/lambda_punch/worker.rb +112 -0
- metadata +64 -4
- data/.github/workflows/main.yml +0 -18
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 415dae29a1a4d52820fbd86aa6dc1a03dc4eb3129098851a2db9ed94385c257f
|
4
|
+
data.tar.gz: 06a861cc3d854392ad2fcb049b4e2acce49536cbb6ec41ba4ea3021b4e43c994
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e3fc61d1620bf1676c96c6fb37221bdc9138797feeebecae40c20300c001a62b560231ccc69b936514d5846220066175a757af24b3ba42aaccbf054377c3971c
|
7
|
+
data.tar.gz: 92ef0f6674b3d42c5f35fc2912653db1b8cb15506e8dd9085fe34b84be305b742bc12decb575ab7f61befc412e5903c9116ea57a9a9838482aacba475a640c17
|
@@ -0,0 +1,17 @@
|
|
1
|
+
name: Test
|
2
|
+
on: [push]
|
3
|
+
jobs:
|
4
|
+
test:
|
5
|
+
runs-on: ubuntu-latest
|
6
|
+
steps:
|
7
|
+
- name: Checkout
|
8
|
+
uses: actions/checkout@v2
|
9
|
+
- name: Bootstrap
|
10
|
+
run: |
|
11
|
+
./bin/bootstrap
|
12
|
+
- name: Setup
|
13
|
+
run: |
|
14
|
+
./bin/setup
|
15
|
+
- name: Test
|
16
|
+
run: |
|
17
|
+
./bin/test
|
data/.gitignore
CHANGED
data/CHANGELOG.md
CHANGED
data/Dockerfile
ADDED
data/Gemfile
CHANGED
data/Gemfile.lock
ADDED
@@ -0,0 +1,30 @@
|
|
1
|
+
PATH
|
2
|
+
remote: .
|
3
|
+
specs:
|
4
|
+
lambda_punch (0.0.2)
|
5
|
+
concurrent-ruby
|
6
|
+
rb-inotify
|
7
|
+
timeout
|
8
|
+
|
9
|
+
GEM
|
10
|
+
remote: https://rubygems.org/
|
11
|
+
specs:
|
12
|
+
concurrent-ruby (1.1.9)
|
13
|
+
ffi (1.15.3)
|
14
|
+
minitest (5.14.4)
|
15
|
+
rake (13.0.3)
|
16
|
+
rb-inotify (0.10.1)
|
17
|
+
ffi (~> 1.0)
|
18
|
+
timeout (0.1.1)
|
19
|
+
|
20
|
+
PLATFORMS
|
21
|
+
x86_64-darwin-19
|
22
|
+
x86_64-linux
|
23
|
+
|
24
|
+
DEPENDENCIES
|
25
|
+
lambda_punch!
|
26
|
+
minitest
|
27
|
+
rake
|
28
|
+
|
29
|
+
BUNDLED WITH
|
30
|
+
2.2.21
|
data/README.md
CHANGED
@@ -1,43 +1,126 @@
|
|
1
|
-
|
1
|
+
![LambdaPunch](https://user-images.githubusercontent.com/2381/123561512-c23fb580-d776-11eb-9780-71d606cd8f2c.png)
|
2
2
|
|
3
|
-
|
3
|
+
![Test](https://github.com/customink/lambda_punch/workflows/Test/badge.svg)
|
4
4
|
|
5
|
-
|
5
|
+
# 👊 LambdaPunch
|
6
6
|
|
7
|
-
|
7
|
+
<a href="https://lamby.custominktech.com"><img src="https://user-images.githubusercontent.com/2381/59363668-89edeb80-8d03-11e9-9985-2ce14361b7e3.png" alt="Lamby: Simple Rails & AWS Lambda Integration using Rack." align="right" width="300" /></a>Asynchronous background job processing for AWS Lambda with Ruby using [Lambda Extensions](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-extensions-api.html). Inspired by the [SuckerPunch](https://github.com/brandonhilkert/sucker_punch) gem but specifically tooled to work with Lambda's invoke model.
|
8
8
|
|
9
|
-
|
9
|
+
**For a more robust background job solution, please consider using AWS SQS with the [Lambdakiq](https://github.com/customink/lambdakiq) gem. A drop-in replacement for [Sidekiq](https://github.com/mperham/sidekiq) when running Rails in AWS Lambda using the [Lamby](https://lamby.custominktech.com/) gem.**
|
10
|
+
|
11
|
+
## 🏗 Architecture
|
12
|
+
|
13
|
+
Because AWS Lambda [freezes the execution environment](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html) after each invoke, there is no "process" that continues to run after the handler's response. However, thanks to Lambda Extensions along with its "early return", we can do two important things. First, we leverage the [rb-inotify](https://github.com/guard/rb-inotify) gem to send the extension process a simulated `POST-INVOKE` event. We then use [Distributed Ruby](https://ruby-doc.org/stdlib-3.0.1/libdoc/drb/rdoc/DRb.html) (DRb) from the extension to signal your application to work jobs off a queue. Both of these are synchronous calls. Once complete the LambdaPunch extensions signals it is done and your function is ready for the next request.
|
14
|
+
|
15
|
+
<img src="https://user-images.githubusercontent.com/2381/123647632-408f6c80-d7f6-11eb-8e39-fb4ee92b1ffa.png" width="100%" alt="AWS Lambda Extensions with LambdaPunch async job queue processing." >
|
16
|
+
|
17
|
+
The LambdaPunch extension process is very small and lean. It only requires a few Ruby libraries and needed gems from your application's bundle. Its only job is to send signals back to your application on the runtime. It does this within a few milliseconds and adds no noticeable overhead to your function.
|
18
|
+
|
19
|
+
## 🎁 Installation
|
20
|
+
|
21
|
+
Add this line to your project's `Gemfile` and then make sure to `bundle install` afterward.
|
10
22
|
|
11
23
|
```ruby
|
12
24
|
gem 'lambda_punch'
|
13
25
|
```
|
14
26
|
|
15
|
-
|
27
|
+
Now, within your application's handler file, make sure to start the LambdaPunch DRb server outside of your handler method. Within the handler method, add an `ensure` section that lets the extension process know the request is done.
|
16
28
|
|
17
|
-
|
29
|
+
```ruby
|
30
|
+
LambdaPunch.start_server!
|
18
31
|
|
19
|
-
|
32
|
+
def handler(event:, context:)
|
33
|
+
# ...
|
34
|
+
ensure
|
35
|
+
LambdaPunch.handled!(context)
|
36
|
+
end
|
37
|
+
```
|
20
38
|
|
21
|
-
|
39
|
+
Within your project or [Rails application's](https://lamby.custominktech.com/docs/anatomy) `Dockerfile`, after you copy your code, add this `RUN` command to install the extension within your container's `/opt/extensions` directory.
|
40
|
+
|
41
|
+
```dockerfile
|
42
|
+
RUN bundle exec rake lambda_punch:install
|
43
|
+
```
|
44
|
+
|
45
|
+
## 🧰 Usage
|
46
|
+
|
47
|
+
Anywhere in your application's code, use the `LambdaPunch.push` method to add blocks of code to your jobs queue.
|
48
|
+
|
49
|
+
```ruby
|
50
|
+
LambdaPunch.push do
|
51
|
+
# ...
|
52
|
+
end
|
53
|
+
```
|
54
|
+
|
55
|
+
For example, if you are using Rails with AWS Lambda via the [Lamby](https://lamby.custominktech.com/) gem along with [New Relic APM](https://dev.to/aws-heroes/using-new-relic-apm-with-rails-on-aws-lambda-51gi) here is how your handler function might appear to ensure their metrics are flushed after each request.
|
56
|
+
|
57
|
+
```ruby
|
58
|
+
def handler(event:, context:)
|
59
|
+
Lamby.handler $app, event, context
|
60
|
+
ensure
|
61
|
+
LambdaPunch.push { NewRelic::Agent.agent.flush_pipe_data }
|
62
|
+
LambdaPunch.handled!(context)
|
63
|
+
end
|
64
|
+
```
|
22
65
|
|
23
|
-
|
66
|
+
### ActiveJob
|
24
67
|
|
25
|
-
|
68
|
+
🚧 COMING SOON 🚧 - A simple ActiveJob adapter...
|
26
69
|
|
27
|
-
|
70
|
+
### Timeouts
|
28
71
|
|
29
|
-
|
72
|
+
Your function's timeout is the max amount to handle the request and process all extension's invoke events. If your function times out, it is possible that queued jobs will not be processed until the next invoke.
|
73
|
+
|
74
|
+
If your application integrates with API Gateway (which has a 30 second timeout) then it is possible your function can be set with a higher timeout to perform background work. Since work is done after each invoke, the LambdaPunch queue should be empty when your function receives the `SHUTDOWN` event. If jobs are in the queue when this happens they will have two seconds max to work them down before being lost.
|
75
|
+
|
76
|
+
**For a more robust background job solution, please consider using AWS SQS with the [Lambdakiq](https://github.com/customink/lambdakiq) gem. A drop-in replacement for [Sidekiq](https://github.com/mperham/sidekiq) when running Rails in AWS Lambda using the [Lamby](https://lamby.custominktech.com/) gem.**
|
77
|
+
|
78
|
+
### Logging
|
79
|
+
|
80
|
+
The default log level is `fatal`, so you will not see any LambdaPunch lines in your logs. However, if you want some low level debugging information on how LambdaPunch is working, you can use this environment variable to change the log level.
|
81
|
+
|
82
|
+
```yaml
|
83
|
+
Environment:
|
84
|
+
Variables:
|
85
|
+
LAMBDA_PUNCH_LOG_LEVEL: debug
|
86
|
+
```
|
87
|
+
|
88
|
+
## 📊 CloudWatch Metrics
|
89
|
+
|
90
|
+
When using Extensions, your function's CloudWatch `Duration` metrics will be the sum of your response time combined with your extension's execution time. For example, if your request takes `200ms` to respond but your background task takes `1000ms` your duration will be a combined `1200ms`. For more details see the _"Performance impact and extension overhead"_ section of the [Lambda Extensions API
|
91
|
+
](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-extensions-api.html)
|
92
|
+
|
93
|
+
Thankfully, when using Lambda Extensions, CloudWatch will create a `PostRuntimeExtensionsDuration` metric that you can use to isolate your true response times `Duration` [using some metric math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html). Here is an example where the metric math above is used in the first "Duration (response)" widget.
|
94
|
+
|
95
|
+
![metric-math](https://user-images.githubusercontent.com/2381/123561591-4eea7380-d777-11eb-8682-c20b9460f112.png)
|
96
|
+
|
97
|
+
![durations](https://user-images.githubusercontent.com/2381/123561590-4e51dd00-d777-11eb-96b2-d886c91aedb0.png)
|
98
|
+
|
99
|
+
## 👷🏽♀️ Development
|
100
|
+
|
101
|
+
After checking out the repo, run the following commands to build a Docker image and install dependencies.
|
102
|
+
|
103
|
+
```shell
|
104
|
+
$ ./bin/bootstrap
|
105
|
+
$ ./bin/setup
|
106
|
+
```
|
107
|
+
|
108
|
+
Then, to run the tests use the following command.
|
109
|
+
|
110
|
+
```shell
|
111
|
+
$ ./bin/test
|
112
|
+
```
|
30
113
|
|
31
|
-
|
114
|
+
You can also run the `./bin/console` command for an interactive prompt within the development Docker container. Likewise you can use `./bin/run ...` followed by any command which would be executed within the same container.
|
32
115
|
|
33
|
-
## Contributing
|
116
|
+
## 💖 Contributing
|
34
117
|
|
35
|
-
Bug reports and pull requests are welcome on GitHub at https://github.com/
|
118
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/customink/lambda_punch. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/customink/lambda_punch/blob/main/CODE_OF_CONDUCT.md).
|
36
119
|
|
37
|
-
## License
|
120
|
+
## 👩⚖️ License
|
38
121
|
|
39
122
|
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
40
123
|
|
41
|
-
## Code of Conduct
|
124
|
+
## 🤝 Code of Conduct
|
42
125
|
|
43
|
-
Everyone interacting in the LambdaPunch project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/
|
126
|
+
Everyone interacting in the LambdaPunch project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/customink/lambda_punch/blob/main/CODE_OF_CONDUCT.md).
|
data/Rakefile
CHANGED
data/bin/_setup
ADDED
data/bin/_test
ADDED
data/bin/bootstrap
ADDED
data/bin/console
CHANGED
@@ -1,15 +1,6 @@
|
|
1
|
-
#!/
|
2
|
-
|
1
|
+
#!/bin/sh
|
2
|
+
set -e
|
3
3
|
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
# You can add fixtures and/or initialization code here to make experimenting
|
8
|
-
# with your gem easier. You can also use a different console, if you like.
|
9
|
-
|
10
|
-
# (If you use this, don't forget to add pry to your Gemfile!)
|
11
|
-
# require "pry"
|
12
|
-
# Pry.start
|
13
|
-
|
14
|
-
require "irb"
|
15
|
-
IRB.start(__FILE__)
|
4
|
+
docker-compose run \
|
5
|
+
lambdapunch \
|
6
|
+
bundle console
|
data/bin/run
ADDED
data/bin/setup
CHANGED
data/bin/test
ADDED
data/docker-compose.yml
ADDED
data/lambda_punch.gemspec
CHANGED
@@ -18,5 +18,7 @@ Gem::Specification.new do |spec|
|
|
18
18
|
spec.bindir = "exe"
|
19
19
|
spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
|
20
20
|
spec.require_paths = ["lib"]
|
21
|
-
|
21
|
+
spec.add_dependency "concurrent-ruby"
|
22
|
+
spec.add_dependency "rb-inotify"
|
23
|
+
spec.add_dependency "timeout"
|
22
24
|
end
|
data/lib/lambda_punch.rb
CHANGED
@@ -1,8 +1,50 @@
|
|
1
|
-
|
2
|
-
|
3
|
-
|
1
|
+
require 'uri'
|
2
|
+
require 'drb'
|
3
|
+
require 'json'
|
4
|
+
require 'tmpdir'
|
5
|
+
require 'logger'
|
6
|
+
require 'net/http'
|
7
|
+
require 'singleton'
|
8
|
+
require 'lambda_punch/api'
|
9
|
+
require 'lambda_punch/error'
|
10
|
+
require 'lambda_punch/logger'
|
11
|
+
require 'lambda_punch/queue'
|
12
|
+
require 'lambda_punch/server'
|
13
|
+
require 'lambda_punch/worker'
|
14
|
+
require 'lambda_punch/version'
|
15
|
+
require 'lambda_punch/notifier'
|
16
|
+
require 'lambda_punch/railtie' if defined?(Rails)
|
4
17
|
|
5
18
|
module LambdaPunch
|
6
|
-
|
7
|
-
|
19
|
+
|
20
|
+
def push(&block)
|
21
|
+
Queue.push(block)
|
22
|
+
end
|
23
|
+
|
24
|
+
def register!
|
25
|
+
Api.register!
|
26
|
+
end
|
27
|
+
|
28
|
+
def loop
|
29
|
+
Api.loop
|
30
|
+
end
|
31
|
+
|
32
|
+
def start_server!
|
33
|
+
Server.start!
|
34
|
+
end
|
35
|
+
|
36
|
+
def start_worker!
|
37
|
+
Worker.start!
|
38
|
+
end
|
39
|
+
|
40
|
+
def logger
|
41
|
+
@logger ||= Logger.new.logger
|
42
|
+
end
|
43
|
+
|
44
|
+
def handled!(context)
|
45
|
+
Notifier.handled!(context)
|
46
|
+
end
|
47
|
+
|
48
|
+
extend self
|
49
|
+
|
8
50
|
end
|
@@ -0,0 +1,93 @@
|
|
1
|
+
module LambdaPunch
|
2
|
+
# Interface to Lambda's Extensions API using simple `Net::HTTP` calls.
|
3
|
+
#
|
4
|
+
# Lambda Extensions API
|
5
|
+
# https://docs.aws.amazon.com/lambda/latest/dg/runtimes-extensions-api.html
|
6
|
+
#
|
7
|
+
class Api
|
8
|
+
|
9
|
+
EXTENSION_NAME = 'lambdapunch'
|
10
|
+
|
11
|
+
include Singleton
|
12
|
+
|
13
|
+
class << self
|
14
|
+
|
15
|
+
def register!
|
16
|
+
instance.register!
|
17
|
+
end
|
18
|
+
|
19
|
+
def loop
|
20
|
+
instance.loop
|
21
|
+
end
|
22
|
+
|
23
|
+
end
|
24
|
+
|
25
|
+
def register!
|
26
|
+
return if @registered
|
27
|
+
uri = URI.parse "#{base_uri}/register"
|
28
|
+
http = Net::HTTP.new uri.host, uri.port
|
29
|
+
request = Net::HTTP::Post.new uri.request_uri
|
30
|
+
request['Content-Type'] = 'application/vnd.aws.lambda.extension+json'
|
31
|
+
request['Lambda-Extension-Name'] = EXTENSION_NAME
|
32
|
+
request.body = %q|{"events":["INVOKE","SHUTDOWN"]}|
|
33
|
+
http.request(request).tap do |r|
|
34
|
+
logger.debug "Api#register! => #{r.class.name.inspect}, body: #{r.body}"
|
35
|
+
@registered = true
|
36
|
+
@extension_id = r.each_header.to_h['lambda-extension-identifier']
|
37
|
+
logger.debug "Api::ExtensionId => #{@extension_id}"
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
def loop
|
42
|
+
resp = event_next
|
43
|
+
event_payload = JSON.parse(resp.body)
|
44
|
+
case event_payload['eventType']
|
45
|
+
when 'INVOKE' then invoke(event_payload)
|
46
|
+
when 'SHUTDOWN' then shutdown
|
47
|
+
else
|
48
|
+
event_type_error(event_payload)
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
private
|
53
|
+
|
54
|
+
def event_next
|
55
|
+
uri = URI.parse "#{base_uri}/event/next"
|
56
|
+
http = Net::HTTP.new uri.host, uri.port
|
57
|
+
request = Net::HTTP::Get.new uri.request_uri
|
58
|
+
request['Content-Type'] = 'application/vnd.aws.lambda.extension+json'
|
59
|
+
request['Lambda-Extension-Identifier'] = @extension_id
|
60
|
+
http.request(request).tap do |r|
|
61
|
+
logger.debug "Api#event_next => #{r.class.name.inspect}, body: #{r.body}"
|
62
|
+
end
|
63
|
+
end
|
64
|
+
|
65
|
+
def invoke(event_payload)
|
66
|
+
logger.debug "Api#invoke => #{JSON.dump(event_payload)}" if logger.debug?
|
67
|
+
Worker.call(event_payload)
|
68
|
+
end
|
69
|
+
|
70
|
+
def shutdown
|
71
|
+
logger.info 'Api#shutdown...'
|
72
|
+
DRb.stop_service rescue true
|
73
|
+
exit
|
74
|
+
end
|
75
|
+
|
76
|
+
private
|
77
|
+
|
78
|
+
def base_uri
|
79
|
+
"http://#{ENV['AWS_LAMBDA_RUNTIME_API']}/2020-01-01/extension"
|
80
|
+
end
|
81
|
+
|
82
|
+
def logger
|
83
|
+
LambdaPunch.logger
|
84
|
+
end
|
85
|
+
|
86
|
+
def event_type_error(event_payload)
|
87
|
+
message = "Unknown event type: #{event_payload['eventType'].inspect}"
|
88
|
+
logger.fatal(message)
|
89
|
+
raise EventTypeError.new(message)
|
90
|
+
end
|
91
|
+
|
92
|
+
end
|
93
|
+
end
|
@@ -0,0 +1,24 @@
|
|
1
|
+
module LambdaPunch
|
2
|
+
class Logger
|
3
|
+
|
4
|
+
def logger
|
5
|
+
@logger ||= ::Logger.new(STDOUT).tap do |l|
|
6
|
+
l.level = level
|
7
|
+
l.formatter = proc { |_s, _d, _p, m| "[LambdaPunch] #{m}\n" }
|
8
|
+
end
|
9
|
+
end
|
10
|
+
|
11
|
+
def level=(value)
|
12
|
+
@level = value.to_s
|
13
|
+
@logger = nil
|
14
|
+
end
|
15
|
+
|
16
|
+
private
|
17
|
+
|
18
|
+
def level
|
19
|
+
l = (@level || ENV['LAMBDA_PUNCH_LOG_LEVEL'] || 'fatal').upcase.to_sym
|
20
|
+
::Logger.const_defined?(l) ? ::Logger.const_get(l) : ::Logger::FATAL
|
21
|
+
end
|
22
|
+
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,49 @@
|
|
1
|
+
module LambdaPunch
|
2
|
+
class Notifier
|
3
|
+
|
4
|
+
FILE = "#{Dir.tmpdir}/lambdapunch-handled"
|
5
|
+
File.open(FILE, 'w') { |f| f.write('') }
|
6
|
+
|
7
|
+
class << self
|
8
|
+
|
9
|
+
def handled!(context)
|
10
|
+
File.open(FILE, 'w') do |f|
|
11
|
+
f.write context.aws_request_id
|
12
|
+
end
|
13
|
+
end
|
14
|
+
|
15
|
+
def request_id
|
16
|
+
File.read(FILE)
|
17
|
+
end
|
18
|
+
|
19
|
+
end
|
20
|
+
|
21
|
+
def initialize
|
22
|
+
@notifier = INotify::Notifier.new
|
23
|
+
end
|
24
|
+
|
25
|
+
def watch
|
26
|
+
@notifier.watch(FILE, :modify, :oneshot) { yield(request_id) }
|
27
|
+
end
|
28
|
+
|
29
|
+
def process
|
30
|
+
@notifier.process
|
31
|
+
end
|
32
|
+
|
33
|
+
def close
|
34
|
+
logger.debug "Notifier#close"
|
35
|
+
@notifier.close rescue true
|
36
|
+
end
|
37
|
+
|
38
|
+
def request_id
|
39
|
+
self.class.request_id
|
40
|
+
end
|
41
|
+
|
42
|
+
private
|
43
|
+
|
44
|
+
def logger
|
45
|
+
LambdaPunch.logger
|
46
|
+
end
|
47
|
+
|
48
|
+
end
|
49
|
+
end
|
@@ -0,0 +1,41 @@
|
|
1
|
+
module LambdaPunch
|
2
|
+
class Queue
|
3
|
+
|
4
|
+
class << self
|
5
|
+
|
6
|
+
def push(block)
|
7
|
+
jobs << block
|
8
|
+
end
|
9
|
+
|
10
|
+
def jobs
|
11
|
+
@jobs ||= Concurrent::Array.new
|
12
|
+
end
|
13
|
+
|
14
|
+
end
|
15
|
+
|
16
|
+
def call
|
17
|
+
jobs.each do |job|
|
18
|
+
begin
|
19
|
+
job.call
|
20
|
+
rescue => e
|
21
|
+
logger.error "Queue#call::error => #{e.message}"
|
22
|
+
# ...
|
23
|
+
end
|
24
|
+
end
|
25
|
+
true
|
26
|
+
ensure
|
27
|
+
jobs.clear
|
28
|
+
end
|
29
|
+
|
30
|
+
private
|
31
|
+
|
32
|
+
def jobs
|
33
|
+
self.class.jobs
|
34
|
+
end
|
35
|
+
|
36
|
+
def logger
|
37
|
+
LambdaPunch.logger
|
38
|
+
end
|
39
|
+
|
40
|
+
end
|
41
|
+
end
|
@@ -0,0 +1,26 @@
|
|
1
|
+
module LambdaPunch
|
2
|
+
class Server
|
3
|
+
|
4
|
+
include Singleton
|
5
|
+
|
6
|
+
class << self
|
7
|
+
|
8
|
+
def uri
|
9
|
+
'druby://127.0.0.1:9030'
|
10
|
+
end
|
11
|
+
|
12
|
+
def start!
|
13
|
+
require 'concurrent'
|
14
|
+
LambdaPunch.logger.info "Server.start!..."
|
15
|
+
instance
|
16
|
+
end
|
17
|
+
|
18
|
+
end
|
19
|
+
|
20
|
+
def initialize
|
21
|
+
@queue = Queue.new
|
22
|
+
DRb.start_service self.class.uri, @queue
|
23
|
+
end
|
24
|
+
|
25
|
+
end
|
26
|
+
end
|
@@ -0,0 +1,11 @@
|
|
1
|
+
namespace :lambda_punch do
|
2
|
+
|
3
|
+
desc "Install the LambdaPunch Lambda Extension."
|
4
|
+
task :install do
|
5
|
+
require 'fileutils'
|
6
|
+
FileUtils.mkdir_p '/opt/extensions'
|
7
|
+
extension = File.expand_path "#{__dir__}/../extensions/lambdapunch"
|
8
|
+
FileUtils.cp extension '/opt/extensions/'
|
9
|
+
end
|
10
|
+
|
11
|
+
end
|
data/lib/lambda_punch/version.rb
CHANGED
@@ -0,0 +1,112 @@
|
|
1
|
+
module LambdaPunch
|
2
|
+
# This `LambdaPunch::Worker` has a few responsibilities:
|
3
|
+
#
|
4
|
+
# 1. Maintain a class level DRb reference to your function's `LambdaPunch::Queue` object.
|
5
|
+
# 2. Process extension `INVOKE` events by waiting for your function to complete.
|
6
|
+
# 3. Triggering your application to perform work after each request.
|
7
|
+
#
|
8
|
+
class Worker
|
9
|
+
|
10
|
+
class << self
|
11
|
+
|
12
|
+
# Method to lazily require rb-inotify and start the DRb service.
|
13
|
+
#
|
14
|
+
def start!
|
15
|
+
LambdaPunch.logger.info "Worker.start!..."
|
16
|
+
require 'timeout'
|
17
|
+
require 'rb-inotify'
|
18
|
+
DRb.start_service
|
19
|
+
@queue = DRbObject.new_with_uri(Server.uri)
|
20
|
+
end
|
21
|
+
|
22
|
+
# Creates a new instance of this object with the event payload from the `LambdaPunch::Api#invoke`
|
23
|
+
# method and immediately performs the `call` method which waits for the function's handler to complete.
|
24
|
+
#
|
25
|
+
def call(event_payload)
|
26
|
+
new(event_payload).call
|
27
|
+
end
|
28
|
+
|
29
|
+
# The `@queue` object is the local process' reference to the application `LambdaPunch::Queue`
|
30
|
+
# instance which does all the work in the applciation's scope.
|
31
|
+
#
|
32
|
+
def queue
|
33
|
+
@queue
|
34
|
+
end
|
35
|
+
|
36
|
+
end
|
37
|
+
|
38
|
+
def initialize(event_payload)
|
39
|
+
@invoked = false
|
40
|
+
@event_payload = event_payload
|
41
|
+
@notifier = Notifier.new
|
42
|
+
@notifier.watch { |request_id| notified(request_id) }
|
43
|
+
@request_id_notifier = nil
|
44
|
+
end
|
45
|
+
|
46
|
+
# Here we wait for the application's handler to signal it is done via the `LambdaPunch::Notifier` or if the
|
47
|
+
# function has timed out. In either event there may be work to perform in the `LambdaPunch::Queue`. This method
|
48
|
+
# also ensures any clean up is done. For example, closing file notifications.
|
49
|
+
#
|
50
|
+
def call
|
51
|
+
Timeout.timeout(timeout) { @notifier.process }
|
52
|
+
rescue Timeout::Error
|
53
|
+
logger.debug "Worker#call => Function timeout reached."
|
54
|
+
ensure
|
55
|
+
@notifier.close
|
56
|
+
self.class.queue.call
|
57
|
+
end
|
58
|
+
|
59
|
+
private
|
60
|
+
|
61
|
+
# The Notifier's watch handler would set this instance variable to `true`. We also return `true`
|
62
|
+
# if the extension's invoke palyload event has a `requestId` matching what the handler has written
|
63
|
+
# to the `LambdaPunch::Notifier` file location. See also `request_ids_match?` method.
|
64
|
+
#
|
65
|
+
def invoked?
|
66
|
+
@invoked || request_ids_match?
|
67
|
+
end
|
68
|
+
|
69
|
+
# The unique AWS reqeust id that both the extension and handler receive for each invoke. This one
|
70
|
+
# represents the extension's side.
|
71
|
+
#
|
72
|
+
def request_id_payload
|
73
|
+
@event_payload['requestId']
|
74
|
+
end
|
75
|
+
|
76
|
+
# Set via the `LambdaPunch::Notifier` watch event from the your function's handler.
|
77
|
+
#
|
78
|
+
def request_id_notifier
|
79
|
+
@request_id_notifier
|
80
|
+
end
|
81
|
+
|
82
|
+
# Check if notified via inotify or in some rare case the function's handler has already completed
|
83
|
+
# and written the matching request id via the context object to the `LambdaPunch::Notifier` file.
|
84
|
+
#
|
85
|
+
def request_ids_match?
|
86
|
+
request_id_payload == (request_id_notifier || Notifier.request_id)
|
87
|
+
end
|
88
|
+
|
89
|
+
# The function's timeout in seconds using the `INVOKE` event payload's `deadlineMs` value.
|
90
|
+
#
|
91
|
+
def timeout
|
92
|
+
deadline_milliseconds = @event_payload['deadlineMs']
|
93
|
+
deadline = Time.at(deadline_milliseconds / 1000.0)
|
94
|
+
deadline_timeout = deadline - Time.now
|
95
|
+
deadline_timeout > 0 ? deadline_timeout : 0
|
96
|
+
end
|
97
|
+
|
98
|
+
# Our `LambdaPunch::Notifier` instance callback.
|
99
|
+
#
|
100
|
+
def notified(request_id)
|
101
|
+
@invoked = true
|
102
|
+
@request_id_notifier = request_id
|
103
|
+
end
|
104
|
+
|
105
|
+
def logger
|
106
|
+
LambdaPunch.logger
|
107
|
+
end
|
108
|
+
|
109
|
+
def noop ; end
|
110
|
+
|
111
|
+
end
|
112
|
+
end
|
metadata
CHANGED
@@ -1,15 +1,57 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: lambda_punch
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ken Collins
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-06-
|
12
|
-
dependencies:
|
11
|
+
date: 2021-06-28 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: concurrent-ruby
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '0'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '0'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: rb-inotify
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '0'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: timeout
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '0'
|
13
55
|
description: 'LambdaPunch: Async Processing using Lambda Extensions'
|
14
56
|
email:
|
15
57
|
- ken@metaskills.net
|
@@ -17,19 +59,37 @@ executables: []
|
|
17
59
|
extensions: []
|
18
60
|
extra_rdoc_files: []
|
19
61
|
files:
|
20
|
-
- ".github/workflows/
|
62
|
+
- ".github/workflows/test.yml"
|
21
63
|
- ".gitignore"
|
22
64
|
- CHANGELOG.md
|
23
65
|
- CODE_OF_CONDUCT.md
|
66
|
+
- Dockerfile
|
24
67
|
- Gemfile
|
68
|
+
- Gemfile.lock
|
25
69
|
- LICENSE.txt
|
26
70
|
- README.md
|
27
71
|
- Rakefile
|
72
|
+
- bin/_setup
|
73
|
+
- bin/_test
|
74
|
+
- bin/bootstrap
|
28
75
|
- bin/console
|
76
|
+
- bin/run
|
29
77
|
- bin/setup
|
78
|
+
- bin/test
|
79
|
+
- docker-compose.yml
|
30
80
|
- lambda_punch.gemspec
|
31
81
|
- lib/lambda_punch.rb
|
82
|
+
- lib/lambda_punch/api.rb
|
83
|
+
- lib/lambda_punch/error.rb
|
84
|
+
- lib/lambda_punch/extensions/lambdapunch
|
85
|
+
- lib/lambda_punch/logger.rb
|
86
|
+
- lib/lambda_punch/notifier.rb
|
87
|
+
- lib/lambda_punch/queue.rb
|
88
|
+
- lib/lambda_punch/railtie.rb
|
89
|
+
- lib/lambda_punch/server.rb
|
90
|
+
- lib/lambda_punch/tasks/install.rake
|
32
91
|
- lib/lambda_punch/version.rb
|
92
|
+
- lib/lambda_punch/worker.rb
|
33
93
|
homepage: https://github.com/customink/lambda_punch
|
34
94
|
licenses:
|
35
95
|
- MIT
|
data/.github/workflows/main.yml
DELETED
@@ -1,18 +0,0 @@
|
|
1
|
-
name: Ruby
|
2
|
-
|
3
|
-
on: [push,pull_request]
|
4
|
-
|
5
|
-
jobs:
|
6
|
-
build:
|
7
|
-
runs-on: ubuntu-latest
|
8
|
-
steps:
|
9
|
-
- uses: actions/checkout@v2
|
10
|
-
- name: Set up Ruby
|
11
|
-
uses: ruby/setup-ruby@v1
|
12
|
-
with:
|
13
|
-
ruby-version: 2.7.0
|
14
|
-
- name: Run the default task
|
15
|
-
run: |
|
16
|
-
gem install bundler -v 2.2.15
|
17
|
-
bundle install
|
18
|
-
bundle exec rake
|