paramsync 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 37d39c92a6c66de87ac34ede59db549949cd7385035fb6459024c0feb3a5efb4
4
+ data.tar.gz: d6f3655096d15d5843bf4606835f27f6675c2ecd6237643deafa93d1ac3e9d08
5
+ SHA512:
6
+ metadata.gz: 5d2e34c7ee05e6985348214c92ecc5b05eb7fb966b0b95288f1237e45ed7126e24a3d9337c260f0d409501efd40e4d5f264747aca4f6fb8a5040e3416ee157ec
7
+ data.tar.gz: 53399f07cfe95f581835517af20cf517e8c89673ca20f90cf2951622172fb00c5a5ebc2e1bdfe8bf2b65a5c4642d14de91fdcc91bc0ee1436a9ca03183b3deb7
data/LICENSE ADDED
@@ -0,0 +1,12 @@
1
+ This work is dedicated to the public domain. No rights reserved.
2
+
3
+ I, the copyright holder of this work, hereby release it into the public
4
+ domain. This applies worldwide.
5
+
6
+ I grant any entity the right to use this work for any purpose, without
7
+ any conditions, unless such conditions are required by law.
8
+
9
+ If you require a fuller legal statement, please refer to the Creative
10
+ Commons Zero license:
11
+
12
+ http://creativecommons.org/publicdomain/zero/1.0/
data/README.md ADDED
@@ -0,0 +1,334 @@
1
+ # paramsync
2
+
3
+ Paramsync is a simple, straightforward CLI tool for synchronizing data from the
4
+ filesystem to the AWS parameter store KV store and vice-versa.
5
+
6
+ ## Basic Usage
7
+
8
+ Run `paramsync check` to see what differences exist, and `paramsync push` to
9
+ synchronize the changes from the filesystem to parameter store.
10
+
11
+ $ paramsync check
12
+ =====================================================================================
13
+ myapp-private
14
+ local:path/private => ssm:us-east-1:private/myapp
15
+ Keys scanned: 37
16
+ No changes to make for this sync target.
17
+
18
+ =====================================================================================
19
+ myapp-config
20
+ local:path/config => ssm:us-east-1:config/myapp
21
+ Keys scanned: 80
22
+
23
+ UPDATE config/myapp/prod/ip-allowlist.json
24
+ -------------------------------------------------------------------------------------
25
+ -["10.8.0.0/16"]
26
+ +["10.8.0.0/16","10.9.10.0/24"]
27
+ -------------------------------------------------------------------------------------
28
+
29
+ Keys to update: 1
30
+ ~ config/myapp/prod/ip-allowlist.json
31
+
32
+ You can also limit your command to specific synchronization targets by using
33
+ the `--target` flag:
34
+
35
+ $ paramsync push --target myapp-config
36
+ =====================================================================================
37
+ myapp-config
38
+ local:path/config => ssm:us-east-1:config/myapp
39
+ Keys scanned: 80
40
+
41
+ UPDATE config/myapp/prod/ip-allowlist.json
42
+ -------------------------------------------------------------------------------------
43
+ -["10.8.0.0/16"]
44
+ +["10.8.0.0/16","10.9.10.0/24"]
45
+ -------------------------------------------------------------------------------------
46
+
47
+ Keys to update: 1
48
+ ~ config/myapp/prod/ip-allowlist.json
49
+
50
+ Do you want to push these changes?
51
+ Enter 'yes' to continue: yes
52
+
53
+ UPDATE config/myapp/prod/ip-allowlist.json OK
54
+
55
+ Run `paramsync --help` for additional options and commands.
56
+
57
+ ## Pull Mode
58
+
59
+ Paramsync can also sync _from_ parameter store to the local filesystem. This can be
60
+ particularly useful for seeding a git repo with the current contents of a parameter
61
+ store config.
62
+
63
+ Run `paramsync check --pull` to get a summary of changes, and `paramsync pull`
64
+ to actually sync the changes to the local filesystem. Additional arguments such
65
+ as `--target <name>` work in pull mode as well.
66
+
67
+
68
+ ## Configuration
69
+
70
+ Paramsync will automatically configure itself using the first `paramsync.yml`
71
+ file it comes across when searching backwards through the directory tree from
72
+ the current working directory. So, typically you may wish to place the config
73
+ file in the root of your git repository or the base directory of your config
74
+ file tree.
75
+
76
+ You can also specify a config file using the `--config <filename>` command line
77
+ argument.
78
+
79
+
80
+ ### Configuration file structure
81
+
82
+ The configuration file is a Hash represented in YAML format with three possible
83
+ top-level keys: `paramsync`, `ssm`, and `sync`. The `paramsync` section sets
84
+ global defaults and app options. The `ssm` section specifies the roles to
85
+ assume for aws. And the `sync` section lists the directories and
86
+ ssm prefixes you wish to synchronize. Only the `sync` section is strictly
87
+ required. An example `paramsync.yml` is below including explanatory comments:
88
+
89
+ # paramsync.yml
90
+
91
+ paramsync:
92
+ # verbose - defaults to `false`
93
+ # Set this to `true` for more verbose output.
94
+ verbose: false
95
+
96
+ # chomp - defaults to `true`
97
+ # Automatically runs `chomp` on the strings read in from files to
98
+ # eliminate a single trailing newline character (commonly inserted
99
+ # by text editors). Set to `false` to disable this by default for
100
+ # all sync targets (it can be overridden on a per-target basis).
101
+ chomp: true
102
+
103
+ # delete - defaults to `false`
104
+ # Set this to `true` to make the default for all sync targets to
105
+ # delete any keys found in parameter store that do not have a corresponding
106
+ # file on disk. By default, extraneous remote keys will be ignored.
107
+ # If `verbose` is set to `true` the extraneous keys will be named
108
+ # in the output.
109
+ delete: false
110
+
111
+ # color - defaults to `true`
112
+ # Set this to `false` to disable colorized output (eg when running
113
+ # with an automated tool).
114
+ color: true
115
+
116
+ ssm:
117
+ accounts:
118
+ account1:
119
+ role: arn:aws:iam::123456789012:role/admin
120
+
121
+ sync:
122
+ # sync is an array of hashes of sync target configurations
123
+ # Fields:
124
+ # name - The arbitrary friendly name of the sync target. Only
125
+ # required if you wish to target specific sync targets using
126
+ # the `--target` CLI flag.
127
+ # prefix - (required) The parameter store prefix to synchronize to.
128
+ # type - (default: "dir") The type of local file storage. Either
129
+ # 'dir' to indicate a directory tree of files corresponding to
130
+ # parameter store keys; or 'file' to indicate a single YAML file with a
131
+ # map of relative key paths to values.
132
+ # region - (required) The aws region to synchronize to
133
+ # path - (required) The relative filesystem path to either the
134
+ # directory containing the files with content to synchronize
135
+ # to parameter store if this sync target has type=dir, or the local file
136
+ # containing a hash of remote keys if this sync target has
137
+ # type=file. This path is calculated relative to the directory
138
+ # containing the configuration file.
139
+ # account - (account) The account from the ssm block to use
140
+ # delete - Whether or not to delete remote keys that do not exist
141
+ # in the local filesystem. This inherits the setting from the
142
+ # `paramsync` section, or if not specified, defaults to `false`.
143
+ # chomp - Whether or not to chomp a single newline character off
144
+ # the contents of local files before synchronizing to parameter store.
145
+ # This inherits the setting from the `paramsync` section, or if
146
+ # not specified, defaults to `true`.
147
+ # exclude - An array of parameter store paths to exclude from the
148
+ # sync process. These exclusions will be noted in output if the
149
+ # verbose mode is in effect, otherwise they will be silently
150
+ # ignored. At this time there is no provision for specifying
151
+ # prefixes or patterns. Each key must be fully and explicitly
152
+ # specified.
153
+ # erb_enabled - Whether or not to run the local content through
154
+ # ERB parsing before attempting to sync to the remote. Defaults
155
+ # to `false`.
156
+ - name: myapp-config
157
+ prefix: config/myapp
158
+ region: us-east-1
159
+ path: path/config
160
+ exclude:
161
+ - config/myapp/beta.cowboy-yolo
162
+ - config/myapp/prod.cowboy-yolo
163
+ account: account1
164
+ - name: myapp-private
165
+ prefix: private/myapp
166
+ type: dir
167
+ region: us-east-1
168
+ path: path/private
169
+ account: account1
170
+ delete: true
171
+ - name: yourapp-config
172
+ prefix: config/yourapp
173
+ type: file
174
+ region: us-east-1
175
+ path: path/yourapp.yml
176
+ delete: true
177
+ erb_enabled: true
178
+
179
+ You can run `paramsync config` to get a summary of the defined configuration
180
+ and to double-check config syntax.
181
+
182
+ ### File sync targets
183
+
184
+ When using `type: file` for a sync target (see example above), the local path
185
+ should be a YAML (or JSON) file containing a hash of relative key paths to the
186
+ contents of those keys. So for example, given this configuration:
187
+
188
+ sync:
189
+ - name: config
190
+ prefix: config/yourapp
191
+ type: file
192
+ region: us-east-1
193
+ path: yourapp.yml
194
+
195
+ If the file `yourapp.yml` has the following content:
196
+
197
+ ---
198
+ prod/dbname: yourapp
199
+ prod/message: |
200
+ Hello, world. This is a multiline message.
201
+ Thanks.
202
+ prod/app/config.json: |
203
+ {
204
+ "port": 8080,
205
+ "enabled": true
206
+ }
207
+
208
+ Then `paramsync push` will attempt to create and/or update the following keys
209
+ with the corresponding content from `yourapp.yml`:
210
+
211
+ | Key | Value |
212
+ |:-----|:-------|
213
+ | `config/yourapp/prod/dbname` | `yourapp` |
214
+ | `config/yourapp/prod/message` | `Hello, world. This is a multiline message.\nThanks.` |
215
+ | `config/yourapp/prod/app/config.json` | `{\n "port": 8080,\n "enabled": true\n}` |
216
+
217
+ In addition to specifying the entire relative path in each key, you may also
218
+ reference paths via your file's YAML structure directly. For example:
219
+
220
+ ---
221
+ prod:
222
+ redis:
223
+ port: 6380
224
+ host: redis.example.com
225
+
226
+ When pushed, this document will create and/or update the following keys:
227
+
228
+ | Key | Value |
229
+ |:-----|:-------|
230
+ | `config/yourapp/prod/redis/port` | `6380` |
231
+ | `config/yourapp/prod/redis/host` | `redis.example.com` |
232
+
233
+ You may mix and match relative paths and document hierarchy to build paths as
234
+ you would like. And you may also use the special key `_` to embed a value for
235
+ a particular prefix while also nesting values underneath it. For example, given
236
+ this local file target content:
237
+
238
+ ---
239
+ prod/postgres:
240
+ host: db.myproject.example.com
241
+ port: 10001
242
+
243
+ prod:
244
+ redis:
245
+ _: Embedded Value
246
+ port: 6380
247
+
248
+ prod/redis/host: cache.myproject.example.com
249
+
250
+ This file target content would correspond to the following values, when pushed:
251
+
252
+ | Key | Value |
253
+ |:-----|:-------|
254
+ | `config/yourapp/prod/postgres/host` | `db.myproject.example.com` |
255
+ | `config/yourapp/prod/postgres/port` | `10001` |
256
+ | `config/yourapp/prod/redis` | `Embedded Value` |
257
+ | `config/yourapp/prod/redis/port` | `6380` |
258
+ | `config/yourapp/prod/redis/host` | `cache.myproject.example.com` |
259
+
260
+ A `paramsync pull` operation against a file type target will work in reverse,
261
+ and pull values from any keys under `config/yourapp/` into the file
262
+ `yourapp.yml`, overwriting whatever values are there.
263
+
264
+ **NOTE**: Values in local file targets are converted to strings before comparing
265
+ with or uploading to the parameter store. However, because YAML parsing
266
+ converts some values (such as `yes` or `no`) to boolean types, the effective
267
+ value of a key with a value of a bare `yes` will be `true` when converted to a
268
+ string. If you need the actual values `yes` or `no`, use quotes around the value
269
+ to force the YAML parser to interpret it as a string.
270
+
271
+
272
+ #### IMPORTANT NOTES ABOUT PULL MODE WITH FILE TARGETS
273
+
274
+ Against a file target, the structure of the local file can vary in a number
275
+ of ways while still producing the same remote structure. Thus, in pull mode,
276
+ Paramsync must necessarily choose one particular rendering format, and will not
277
+ be able to retain the exact structure of the local file if you alternate push
278
+ and pull operations.
279
+
280
+ Specifically, the following caveats are important to note, when pulling a target
281
+ to a local file:
282
+
283
+ * The local file will be written out as YAML, even if it was originally
284
+ provided locally as a JSON file, and even if the extension is `.json`.
285
+
286
+ * Any existing comments in the local file will be lost.
287
+
288
+ * The document structure will be that of a flat hash will fully-specified
289
+ relative paths as the keys.
290
+
291
+ Future versions of Paramsync may provide options to modify the behavior for pull
292
+ operations on a per-target basis. Pull requests are always welcome.
293
+
294
+
295
+ ### Dynamic configuration
296
+
297
+ The configuration file will be rendered through ERB before being parsed as
298
+ YAML. This can be useful for avoiding repetitive configuration across multiple
299
+ prefixes or regions, eg:
300
+
301
+ sync:
302
+ <% %w( us-east-1 us-west-2 ).each do |region| %>
303
+ - name: <%= dc %>:myapp-private
304
+ prefix: private/myapp
305
+ region: <%= region %>
306
+ path: path/<%= region %>/private
307
+ delete: true
308
+ <% end %>
309
+
310
+ It's a good idea to sanity-check your ERB by running `paramsync config` after
311
+ making any changes.
312
+
313
+
314
+ ### Dynamic content
315
+
316
+ You can also choose to enable ERB parsing for local content as well, by setting
317
+ `erb_enabled: true` on any sync targets you wish to populate in this way.
318
+
319
+
320
+ ### Environment configuration
321
+
322
+ Paramsync may be partially configured using environment variables:
323
+ * `PARAMSYNC_VERBOSE` - set this variable to any value to enable verbose mode
324
+
325
+
326
+ ## Contributing
327
+
328
+ I'm happy to accept suggestions, bug reports, and pull requests through Github.
329
+
330
+
331
+ ## License
332
+
333
+ This software is public domain. No rights are reserved. See LICENSE for more
334
+ information.
data/bin/paramsync ADDED
@@ -0,0 +1,8 @@
1
+ #!/usr/bin/env ruby
2
+ #
3
+ # This software is public domain. No rights are reserved. See LICENSE for more information.
4
+ #
5
+
6
+ require_relative '../lib/paramsync/cli'
7
+
8
+ Paramsync::CLI.run
data/lib/paramsync.rb ADDED
@@ -0,0 +1,73 @@
1
+ # This software is public domain. No rights are reserved. See LICENSE for more information.
2
+
3
+ require 'erb'
4
+ require 'aws-sdk-kms'
5
+ require 'aws-sdk-ssm'
6
+ require 'fileutils'
7
+ require 'ostruct'
8
+ require 'yaml'
9
+
10
+ require_relative 'paramsync/version'
11
+ require_relative 'paramsync/config'
12
+ require_relative 'paramsync/diff'
13
+ require_relative 'paramsync/sync_target'
14
+
15
+ class Paramsync
16
+ class InternalError < RuntimeError; end
17
+
18
+ class << self
19
+ @@config = nil
20
+
21
+ def config
22
+ @@config ||= Paramsync::Config.new
23
+ end
24
+
25
+ def configure(path: nil, targets: nil)
26
+ @@config = Paramsync::Config.new(path: path, targets: targets)
27
+ end
28
+
29
+ def configured?
30
+ not @@config.nil?
31
+ end
32
+ end
33
+ end
34
+
35
+ # monkeypatch String for display prettiness
36
+ class String
37
+ # trim_path replaces the HOME directory on an absolute path with '~'
38
+ def trim_path
39
+ self.sub(%r(^#{ENV['HOME']}), '~')
40
+ end
41
+
42
+ def colorize(s,e=0)
43
+ Paramsync.config.color? ? "\e[#{s}m#{self}\e[#{e}m" : self
44
+ end
45
+
46
+ def red
47
+ colorize(31)
48
+ end
49
+
50
+ def green
51
+ colorize(32)
52
+ end
53
+
54
+ def blue
55
+ colorize(34)
56
+ end
57
+
58
+ def magenta
59
+ colorize(35)
60
+ end
61
+
62
+ def cyan
63
+ colorize(36)
64
+ end
65
+
66
+ def gray
67
+ colorize(37)
68
+ end
69
+
70
+ def bold
71
+ colorize(1,22)
72
+ end
73
+ end
@@ -0,0 +1,148 @@
1
+ # This software is public domain. No rights are reserved. See LICENSE for more information.
2
+
3
+ require_relative '../paramsync'
4
+ require 'diffy'
5
+ require_relative 'cli/check_command'
6
+ require_relative 'cli/encrypt_command'
7
+ require_relative 'cli/push_command'
8
+ require_relative 'cli/pull_command'
9
+ require_relative 'cli/config_command'
10
+ require_relative 'cli/targets_command'
11
+
12
+ class Paramsync
13
+ class CLI
14
+ class << self
15
+ attr_accessor :command, :cli_mode, :config_file, :extra_args, :targets
16
+
17
+ def parse_args(args)
18
+ self.print_usage if args.count < 1
19
+ self.command = nil
20
+ self.config_file = nil
21
+ self.extra_args = []
22
+ self.cli_mode = :command
23
+
24
+ while arg = args.shift
25
+ case arg
26
+ when "--help"
27
+ self.cli_mode = :help
28
+
29
+ when "--config"
30
+ self.config_file = args.shift
31
+
32
+ when "--target"
33
+ self.targets = (args.shift||'').split(",")
34
+
35
+ when /^-/
36
+ # additional option, maybe for the command
37
+ self.extra_args << arg
38
+
39
+ else
40
+ if self.command.nil?
41
+ # if command is not set, this is probably the command
42
+ self.command = arg
43
+ else
44
+ # otherwise, pass it thru to the child command
45
+ self.extra_args << arg
46
+ end
47
+ end
48
+ end
49
+ end
50
+
51
+ def print_usage
52
+ STDERR.puts <<USAGE
53
+ Usage:
54
+ #{File.basename($0)} <command> [options]
55
+
56
+ Commands:
57
+ check Print a summary of changes to be made
58
+ push Push changes from filesystem to parameter store
59
+ pull Pull changes from parameter store to filesystem
60
+ encrypt Encrypt a string for the file
61
+ config Print a summary of the active configuration
62
+ targets List target names
63
+
64
+ General options:
65
+ --help Print help for the given command
66
+ --config <file> Use the specified config file
67
+ --target <tgt> Only apply to the specified target name or names (comma-separated)
68
+
69
+ Options for 'check' command:
70
+ --pull Perform dry run in pull mode
71
+
72
+ Options for 'pull' command:
73
+ --yes Skip confirmation prompt
74
+
75
+ Options for 'push' command:
76
+ --yes Skip confirmation prompt
77
+
78
+ USAGE
79
+ exit 1
80
+ end
81
+
82
+ def configure
83
+ return if Paramsync.configured?
84
+
85
+ begin
86
+ Paramsync.configure(path: self.config_file, targets: self.targets)
87
+
88
+ rescue Paramsync::ConfigFileNotFound
89
+ if self.config_file.nil?
90
+ STDERR.puts "paramsync: ERROR: No configuration file found"
91
+ else
92
+ STDERR.puts "paramsync: ERROR: Configuration file '#{self.config_file}' was not found"
93
+ end
94
+ exit 1
95
+
96
+ rescue Paramsync::ConfigFileInvalid => e
97
+ if self.config_file.nil?
98
+ STDERR.puts "paramsync: ERROR: Configuration file is invalid:"
99
+ else
100
+ STDERR.puts "paramsync: ERROR: Configuration file '#{self.config_file}' is invalid:"
101
+ end
102
+ STDERR.puts " #{e}"
103
+ exit 1
104
+
105
+ end
106
+
107
+ if Paramsync.config.sync_targets.count < 1
108
+ if self.targets.nil?
109
+ STDERR.puts "paramsync: WARNING: No sync targets are defined"
110
+ else
111
+ STDERR.puts "paramsync: WARNING: No sync targets were found that matched the specified list"
112
+ end
113
+ end
114
+ end
115
+
116
+ def run
117
+ self.parse_args(ARGV)
118
+
119
+ case self.cli_mode
120
+ when :help
121
+ # TODO: per-command help
122
+ self.print_usage
123
+
124
+ when :command
125
+ case self.command
126
+ when 'check' then Paramsync::CLI::CheckCommand.run(self.extra_args)
127
+ when 'push' then Paramsync::CLI::PushCommand.run(self.extra_args)
128
+ when 'pull' then Paramsync::CLI::PullCommand.run(self.extra_args)
129
+ when 'encrypt' then Paramsync::CLI::EncryptCommand.run(self.extra_args)
130
+ when 'config' then Paramsync::CLI::ConfigCommand.run
131
+ when 'targets' then Paramsync::CLI::TargetsCommand.run
132
+ when nil then self.print_usage
133
+
134
+ else
135
+ STDERR.puts "paramsync: ERROR: unknown command '#{self.command}'"
136
+ STDERR.puts
137
+ self.print_usage
138
+ end
139
+
140
+ else
141
+ STDERR.puts "paramsync: ERROR: unknown CLI mode '#{self.cli_mode}'"
142
+ exit 1
143
+
144
+ end
145
+ end
146
+ end
147
+ end
148
+ end