sfn-lambda 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 687938d6c0855b30342de6b52eb87a837043aabe
4
+ data.tar.gz: e38d4701d422e4a108a045b9fbeb5c2c8e439a98
5
+ SHA512:
6
+ metadata.gz: 5c76cb6f390565b252cbd115de9073cf2033171617c754fd65c2f99e4dca2cd30ce76841cd5dae27f1c15643ff0a2b6e8d2bd014dae45705229b50d06f9325c9
7
+ data.tar.gz: 3135cd6699333754d1825d00ddc1633e4d702e118ae737e1b6dfbd89f75c42c392d157ee70883fe738b48d300c11833bf8bdb202abf785635da62a82a45886ed
@@ -0,0 +1,2 @@
1
+ # v0.1.0
2
+ * Initial release
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2016 Chris Roberts
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
@@ -0,0 +1,215 @@
1
+ # AWS Lambda for SparkleFormation
2
+
3
+ Lets make lambda functions easier to manage in CloudFormation!
4
+
5
+ ## Design
6
+
7
+ This SparkleFormation Callback adds a new helper method to AWS based
8
+ SparkleFormation templates called `lambda!`. This helper method will
9
+ insert an `AWS::Lambda::Function` resource into the template using
10
+ source files contained within configured directories.
11
+
12
+ ## Features
13
+
14
+ * Individual source files for lambda functions
15
+ * Automatic resource creation within templates
16
+ * Automatic code setup for resource
17
+ * Acceptable functions will be defined inline
18
+ * S3 storage will be used when inline is unacceptable
19
+ * S3 versioning will be used when bucket configured for versioning
20
+ * Automatic asset builds (for `java8` runtime targets)
21
+
22
+ ## Usage
23
+
24
+ ### Setup
25
+
26
+ First add the `sfn-lambda` gem to the local bundle (in the `./Gemfile`):
27
+
28
+ ```ruby
29
+ group :sfn do
30
+ gem 'sfn-lambda'
31
+ end
32
+ ```
33
+
34
+ Now enable the `sfn-lambda` callback in the `.sfn` configuration file:
35
+
36
+ ```ruby
37
+ Configuration.new do
38
+ ...
39
+ callbacks do
40
+ default ['lambda']
41
+ end
42
+ ...
43
+ end
44
+ ```
45
+
46
+ _NOTE: If using the `java8` runtime for lambda functions, `maven` must
47
+ be installed with `mvn` being available within the user's PATH._
48
+
49
+ ### Configuration
50
+
51
+ #### Lambda function files directory
52
+
53
+ By default the `sfn-lambda` callback will search the `./lambda` directory
54
+ for lambda function files. A custom directory path can be used by modifying
55
+ the configuration:
56
+
57
+ ```ruby
58
+ Configuration.new do
59
+ lambda do
60
+ directory './my-lambdas'
61
+ end
62
+ end
63
+ ```
64
+
65
+ #### S3 lambda function file storage
66
+
67
+ By default the `sfn-lambda` callback will use the bucket name provided by
68
+ the `nesting_bucket` configuration item. This can be customized to use a
69
+ different bucket by modifying the configuration:
70
+
71
+ ```ruby
72
+ Configuration.new do
73
+ lambda do
74
+ upload do
75
+ bucket 'my-custom-bucket'
76
+ end
77
+ end
78
+ end
79
+ ```
80
+
81
+ ### Lambda function files
82
+
83
+ The path of lambda function files is important. The path is used to determine
84
+ the proper handler for running the lambda function, as well as providing the
85
+ identifier to reference the function. The path structure is as follows:
86
+
87
+ ```
88
+ ./lambda/RUNTIME/FUNCTION_NAME.extension
89
+ ```
90
+
91
+ The `RUNTIME` defines the runtime used for handling the lambda function. At
92
+ the time of writing this, that value can be one of:
93
+
94
+ * `nodejs`
95
+ * `nodejs4.3`
96
+ * `java8`
97
+ * `python2.7`
98
+
99
+ _NOTE: Runtime values are not validated which allows new runtimes to be used
100
+ as they are made available._
101
+
102
+ The `FUNCTION_NAME` is used for two purposes:
103
+
104
+ 1. It identifies the function name lambda should use
105
+ 2. It is used in combination with the `RUNTIME` to identify the lambda in the template
106
+
107
+ ### Example
108
+
109
+ Using the python example described within the lambda documentation:
110
+
111
+ * http://docs.aws.amazon.com/lambda/latest/dg/python-programming-model-handler-types.html
112
+
113
+ we can define our handler code:
114
+
115
+ * `./lambda/python2.7/my_handler.py`
116
+
117
+ ```python
118
+ def my_handler(event, context):
119
+ message = 'Hello {} {}!'.format(event['first_name'], event['last_name'])
120
+ return {
121
+ 'message' : message
122
+ }
123
+ ```
124
+
125
+ Now, using a new helper method, lambda resources can be created within a SparkleFormation template using
126
+ the newly created file:
127
+
128
+ * `./sparkleformation/lambda_test.rb`
129
+
130
+ ```ruby
131
+ SparkleFormation.new(:lambda_test) do
132
+ lambda!(:my_handler)
133
+ end
134
+ ```
135
+
136
+ When the template is printed a lambda resource is shown with the function properly inlined:
137
+
138
+ ```
139
+ $ sfn print --file lambda_test
140
+ {
141
+ "Resources": {
142
+ "MyHandlerLambdaFunction": {
143
+ "Type": "AWS::Lambda::Function",
144
+ "Properties": {
145
+ "Handler": "python2.7",
146
+ "FunctionName": "my_handler",
147
+ "ZipFile": "def my_handler(event, context):\n message = 'Hello {} {}!'.format(event['first_name'], event['last_name'])\n return {\n 'message' : message\n }\n\n"
148
+ }
149
+ }
150
+ }
151
+ }
152
+ ```
153
+
154
+ If the name of a lambda function is shared across multiple runtimes, the desired runtime
155
+ can be specified within the call:
156
+
157
+ ```ruby
158
+ SparkleFormation.new(:lambda_test) do
159
+ lambda!(:my_handler, :runtime => 'python2.7')
160
+ end
161
+ ```
162
+
163
+ If a lambda function is to be used for creating multiple resources within a template, a
164
+ custom name can be added as well:
165
+
166
+ ```ruby
167
+ SparkleFormation.new(:lambda_test) do
168
+ lambda!(:my_handler, :first, :runtime => 'python2.7')
169
+ lambda!(:my_handler, :second, :runtime => 'python2.7')
170
+ end
171
+ ```
172
+
173
+ ### Special Cases
174
+
175
+ ### S3 Storage
176
+
177
+ When the size of the lambda function is greater than the defined max size (4096 default),
178
+ the function will be stored on S3. If the bucket configured for storage has versioning
179
+ enabled, versioning information will be automatically set within the resource. If no
180
+ versioning information is available, a checksum will be attached to the generated key name.
181
+
182
+ ### Builds
183
+
184
+ For lambda functions utilizing the `java8` runtime, the `sfn-lambda` callback will behave
185
+ slightly different. When discovering available lambda functions, the directory names under the
186
+ `./lambda/java8` directory will be used. This allows for the collection of required files to
187
+ be stored within the directory.
188
+
189
+ Using the example here: http://docs.aws.amazon.com/lambda/latest/dg/java-create-jar-pkg-maven-no-ide.html
190
+
191
+ The defined directory structure would be:
192
+
193
+ ```
194
+ $ cd ./lambda
195
+ $ tree
196
+ .
197
+ |____java8
198
+ | |____hello
199
+ | | |____src
200
+ | | | |____main
201
+ | | | | |____java
202
+ | | | | | |____example
203
+ | | | | | | |____Hello.java
204
+ | | |____pom.xml
205
+ ```
206
+
207
+ When the `hello` lambda function is used within a template, `sfn-lambda` will automatically generate
208
+ the required jar file using Maven and store the resulting asset on S3.
209
+
210
+ _NOTE: Maven is required to be installed when using the `java8` runtime_
211
+
212
+ ## Info
213
+
214
+ * Repository: https://github.com/sparkleformation/sfn-lambda
215
+ * IRC: Freenode @ #sparkleformation
@@ -0,0 +1,5 @@
1
+ require 'sfn'
2
+ require 'sfn-lambda/setup'
3
+ require 'sfn-lambda/control'
4
+ require 'sfn-lambda/inject'
5
+ require 'sfn-lambda/version'
@@ -0,0 +1,234 @@
1
+ require 'singleton'
2
+
3
+ module SfnLambda
4
+
5
+ # @return [Control]
6
+ def self.control
7
+ Control.instance
8
+ end
9
+
10
+ class Control
11
+
12
+ include Singleton
13
+
14
+ DEFAULTS = {
15
+ :INLINE_MAX_SIZE => 4096,
16
+ :INLINE_RESTRICTED => ['java8'].freeze,
17
+ :BUILD_REQUIRED => {
18
+ 'java8' => {
19
+ :build_command => 'mvn package',
20
+ :output_directory => './target',
21
+ :asset_extension => '.jar'
22
+ }.freeze
23
+ }.freeze
24
+ }.freeze
25
+
26
+ attr_reader :functions
27
+ attr_accessor :callback
28
+
29
+ # Create a new control instance
30
+ #
31
+ # @return [self]
32
+ def initialize
33
+ @functions = Smash.new
34
+ end
35
+
36
+ # @return [Array<String>] paths to lambda storage directories
37
+ def lambda_directories
38
+ paths = [callback.config.fetch(:lambda, :directory, 'lambda')].flatten.compact.uniq.map do |path|
39
+ File.expand_path(path)
40
+ end
41
+ invalids = paths.find_all do |path|
42
+ !File.directory?(path)
43
+ end
44
+ unless(invalids.empty?)
45
+ raise "Invalid lambda directory paths provided: #{invalids.join(', ')}"
46
+ end
47
+ paths
48
+ end
49
+
50
+ # Get path to lambda function
51
+ #
52
+ # @param name [String] name of lambda function
53
+ # @param runtime [String] runtime of lambda function
54
+ # @return [Hash] {:path, :runtime}
55
+ def get(name, runtime=nil)
56
+ unless(runtime)
57
+ runtime = functions.keys.find_all do |r_name|
58
+ functions[r_name].keys.include?(name.to_s)
59
+ end
60
+ if(runtime.empty?)
61
+ raise "Failed to locate requested lambda function `#{name}`"
62
+ elsif(runtime.size > 1)
63
+ raise "Multiple lambda function matches for `#{name}`. (In runtimes: `#{runtime.sort.join('`, `')}`)"
64
+ end
65
+ runtime = runtime.first
66
+ end
67
+ result = functions.get(runtime, name)
68
+ if(result.nil?)
69
+ raise "Failed to locate requested lambda function `#{name}`"
70
+ else
71
+ Smash.new(:path => result, :runtime => runtime, :name => name)
72
+ end
73
+ end
74
+
75
+ # Format lambda function content to use within template. Will provide raw
76
+ # source when function can be inlined within the template. If inline is not
77
+ # available, it will store within S3
78
+ #
79
+ # @param info [Hash] content information
80
+ # @option info [String] :path path to lambda function
81
+ # @option info [String] :runtime runtime of lambda function
82
+ # @option info [String] :name name of lambda function
83
+ # @return [Smash] content information
84
+ def format_content(info)
85
+ if(can_inline?(info))
86
+ Smash.new(:raw => File.read(info[:path]))
87
+ else
88
+ apply_build!(info)
89
+ key_name = generate_key_name(info)
90
+ io = File.open(info[:path], 'rb')
91
+ file = bucket.files.build
92
+ file.name = key_name
93
+ file.body = io
94
+ file.save
95
+ io.close
96
+ if(versioning_enabled?)
97
+ result = s3.request(
98
+ :path => s3.file_path(file),
99
+ :endpoint => s3.bucket_endpoint(file.bucket),
100
+ :method => :head
101
+ )
102
+ version = result[:headers][:x_amz_version_id]
103
+ end
104
+ Smash(:bucket => storage_bucket, :key => key_name, :version => version)
105
+ end
106
+ end
107
+
108
+ # Build the lambda asset if building is a requirement
109
+ #
110
+ # @param info [Hash]
111
+ # @return [TrueClass, FalseClass] build was performed
112
+ def apply_build!(info)
113
+ if(build_info = self[:build_required][info[:runtime]])
114
+ Open3.popen3(build_info[:build_command], :chdir => info[:path]) do |stdin, stdout, stderr, wait_thread|
115
+ exit_status = wait_thread.value
116
+ unless(exit_status.success?)
117
+ callback.ui.error "Failed to build lambda assets for storage from path: #{info[:path]}"
118
+ callback.ui.debug "Build command used which generated failure: `#{build_info[:build_command]}`"
119
+ callback.ui.debug "STDOUT: #{stdout.read}"
120
+ callback.ui.debug "STDERR: #{stderr.read}"
121
+ raise "Failed to build lambda asset for storage! (path: `#{info[:path]}`)"
122
+ end
123
+ end
124
+ file = Dir.glob(File.join(info[:path], build_info[:output_directory], "*.#{build_config[:asset_extension]}")).first
125
+ if(file)
126
+ info[:path] = file
127
+ true
128
+ else
129
+ debug "Glob pattern used for build asset detection: `#{File.join(info[:path], build_info[:output_directory], "*.#{build_config[:asset_extension]}")}`"
130
+ raise "Failed to locate generated build asset for storage! (path: `#{info[:path]}`)"
131
+ end
132
+ else
133
+ false
134
+ end
135
+ end
136
+
137
+ # @return [Miasma::Models::Storage::Bucket]
138
+ def bucket
139
+ storage_bucket = callback.config.fetch(:lambda, :upload, :bucket, callback.config[:nesting_bucket])
140
+ if(storage_bucket)
141
+ s3 = api.connection.api_for(:storage)
142
+ l_bucket = s3.buckets.get(storage_bucket)
143
+ end
144
+ unless(l_bucket)
145
+ raise "Failed to locate configured bucket for lambda storage (#{storage_bucket})"
146
+ else
147
+ l_bucket
148
+ end
149
+ end
150
+
151
+ # @return [TrueClass, FalseClass] bucket has versioning enabled
152
+ def versioning_enabled?
153
+ unless(@versioned.nil?)
154
+ s3 = api.connection.api_for(:storage)
155
+ result = s3.request(
156
+ :path => '/',
157
+ :params => {
158
+ :versioning => true
159
+ },
160
+ :endpoint => s3.bucket_endpoint(bucket)
161
+ )
162
+ @versioned = result.get(:body, 'VersioningConfiguration', 'Status') == 'Enabled'
163
+ end
164
+ @versioned
165
+ end
166
+
167
+ # Generate key name based on state
168
+ #
169
+ # @param info [Hash]
170
+ # @return [String] key name
171
+ def generate_key_name(info)
172
+ if(versioning_enabled?)
173
+ "sfn.lambda/#{info[:runtime]}/#{File.basename(info[:path])}"
174
+ else
175
+ checksum = Digest::SHA256.new
176
+ File.open(info[:path], 'rb') do |file|
177
+ while(content = file.read(2048))
178
+ checksum << content
179
+ end
180
+ end
181
+ "sfn.lambda/#{info[:runtime]}/#{File.basename(info[:path])}-#{checksum.base64digest}"
182
+ end
183
+ end
184
+
185
+ # Determine if function can be defined inline within template
186
+ #
187
+ # @param info [Hash]
188
+ # @return [TrueClass, FalseClass]
189
+ def can_inline?(info)
190
+ !self[:inline_restricted].include?(info[:runtime]) && File.size(info[:path]) <= self[:inline_max_size]
191
+ end
192
+
193
+ # Get configuration value for control via sfn configuration and fall back
194
+ # to defined defaults if not set
195
+ #
196
+ # @return [Object] configuration value
197
+ def [](key)
198
+ callback.config.fetch(:lambda, :config, key.to_s.downcase, DEFAULTS[key.to_s.upcase.to_sym])
199
+ end
200
+
201
+ # Discover all defined lambda functions available in directories provided
202
+ # via configuration
203
+ #
204
+ # @return [NilClass]
205
+ def discover_functions!
206
+ core_paths = lambda_directories
207
+ core_paths.each do |path|
208
+ Dir.new(path).each do |dir_item|
209
+ next if dir_item.start_with?('.')
210
+ next unless File.directory?(File.join(path, dir_item))
211
+ if(self[:build_required].keys.include?(dir_item))
212
+ Dir.new(File.join(path, dir_item)).each do |item|
213
+ next if item.start_with?('.')
214
+ full_item = File.join(path, dir_item, item)
215
+ next unless File.directory?(full_item)
216
+ functions.set(dir_item, item, full_item)
217
+ end
218
+ else
219
+ items = Dir.glob(File.join(path, dir_item, '**', '**', '*'))
220
+ items.each do |full_item|
221
+ next unless File.file?(full_item)
222
+ item = full_item.sub(File.join(path, dir_item, ''), '').gsub(File::SEPARATOR, '')
223
+ item = item.sub(/#{Regexp.escape(File.extname(item))}$/, '')
224
+ functions.set(dir_item, item, full_item)
225
+ end
226
+ end
227
+ end
228
+ end
229
+ nil
230
+ end
231
+
232
+ end
233
+
234
+ end
@@ -0,0 +1,42 @@
1
+ class SparkleFormation
2
+ module SparkleAttribute
3
+ module Aws
4
+
5
+ def _lambda(*fn_args)
6
+ if(fn_args.size > 2)
7
+ fn_name, fn_uniq_name, fn_opts = fn_args
8
+ else
9
+ fn_name, fn_uniq_name = fn_args
10
+ end
11
+ __t_stringish(fn_name)
12
+ __t_stringish(fn_uniq_name) unless fn_uniq_name.is_a?(::NilClass)
13
+ if(fn_opts)
14
+ fn_runtime = fn_opts[:runtime] if fn_opts[:runtime]
15
+ end
16
+ unless(fn_runtime.is_a?(::NilClass))
17
+ __t_stringish(fn_runtime)
18
+ end
19
+ lookup = ::SfnLambda.control.get(fn_name, fn_runtime)
20
+ new_fn = _dynamic(:aws_lambda_function,
21
+ [fn_name, fn_uniq_name].compact.map(&:to_s).join('_'),
22
+ :resource_name_suffix => :lambda_function
23
+ )
24
+ new_fn.properties.handler lookup[:runtime]
25
+ new_fn.properties.function_name fn_name
26
+ content = ::SfnLambda.control.format_content(lookup)
27
+ if(content[:raw])
28
+ new_fn.properties.zip_file content[:raw]
29
+ else
30
+ new_fn.properties.s3_bucket content[:bucket]
31
+ new_fn.properties.s3_key content[:key]
32
+ if(content[:version])
33
+ new_fn.properties.s3_object_version content[:version]
34
+ end
35
+ end
36
+ new_fn
37
+ end
38
+ alias_method :lambda!, :_lambda
39
+
40
+ end
41
+ end
42
+ end
@@ -0,0 +1,18 @@
1
+ require 'sfn-lambda'
2
+
3
+ module Sfn
4
+ class Callback
5
+ class Lambda < Callback
6
+
7
+ def quiet
8
+ ENV['DEBUG']
9
+ end
10
+
11
+ def after_config(*_)
12
+ SfnLambda.control.callback = self
13
+ SfnLambda.control.discover_functions!
14
+ end
15
+
16
+ end
17
+ end
18
+ end
@@ -0,0 +1,3 @@
1
+ module SfnLambda
2
+ VERSION = Gem::Version.new('0.1.0')
3
+ end
@@ -0,0 +1,15 @@
1
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__)) + '/lib/'
2
+ require 'sfn-lambda/version'
3
+ Gem::Specification.new do |s|
4
+ s.name = 'sfn-lambda'
5
+ s.version = SfnLambda::VERSION.to_s
6
+ s.summary = 'AWS Lambda integration for SparkleFormation'
7
+ s.author = 'Chris Roberts'
8
+ s.email = 'chrisroberts.code@gmail.com'
9
+ s.homepage = 'http://github.com/spox/sfn-lambda'
10
+ s.description = 'AWS Lambda integration for SparkleFormation'
11
+ s.license = 'MIT'
12
+ s.require_path = 'lib'
13
+ s.add_runtime_dependency 'sparkle_formation', '>= 2.1.0'
14
+ s.files = Dir['{lib,docs}/**/*'] + %w(sfn-lambda.gemspec README.md CHANGELOG.md LICENSE)
15
+ end
metadata ADDED
@@ -0,0 +1,66 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: sfn-lambda
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Chris Roberts
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2016-10-05 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: sparkle_formation
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: 2.1.0
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: 2.1.0
27
+ description: AWS Lambda integration for SparkleFormation
28
+ email: chrisroberts.code@gmail.com
29
+ executables: []
30
+ extensions: []
31
+ extra_rdoc_files: []
32
+ files:
33
+ - CHANGELOG.md
34
+ - LICENSE
35
+ - README.md
36
+ - lib/sfn-lambda.rb
37
+ - lib/sfn-lambda/control.rb
38
+ - lib/sfn-lambda/inject.rb
39
+ - lib/sfn-lambda/setup.rb
40
+ - lib/sfn-lambda/version.rb
41
+ - sfn-lambda.gemspec
42
+ homepage: http://github.com/spox/sfn-lambda
43
+ licenses:
44
+ - MIT
45
+ metadata: {}
46
+ post_install_message:
47
+ rdoc_options: []
48
+ require_paths:
49
+ - lib
50
+ required_ruby_version: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - ">="
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
55
+ required_rubygems_version: !ruby/object:Gem::Requirement
56
+ requirements:
57
+ - - ">="
58
+ - !ruby/object:Gem::Version
59
+ version: '0'
60
+ requirements: []
61
+ rubyforge_project:
62
+ rubygems_version: 2.4.8
63
+ signing_key:
64
+ specification_version: 4
65
+ summary: AWS Lambda integration for SparkleFormation
66
+ test_files: []