kubes 0.4.0 → 0.4.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3d309f1eb882767efd06378aa5bff9fbe8e162db4ebcd19b9ffec2637929d8b0
-  data.tar.gz: a60c4ae73920a38cda6780f435475922f318409e34aed8dec827df0b8483d6c5
+  metadata.gz: bbdaabc58ed6ae8e1c2941431c61a3e7edf7708b7782f1d456b4bcf2b01ef777
+  data.tar.gz: '04290becf20d593ba00d214dd2e790eac5501e3e8f5e3b62d636b3e882b7a5e1'
 SHA512:
-  metadata.gz: 600df214d8b09d66bbea43b3114dc8dd3b41011025e76e2741ce2a076fa4af1ab8f07b68e54f1aeaaa72d166a1c9c2970c34f22f314b14b99384c00d8818aee4
-  data.tar.gz: 491bf32c8f78ce324aa500d57976345e28696d2fa483b6c8671d92c62191a3c84ffe11048b743251903c4b9b2ff1a0dc7c92b86211b9b909f8fcb5d813c79b7a
+  metadata.gz: 302a4a216bc5007de836e335cc38e4fcd8d4c4db8d873f4333d862f84dbcfd32c302ba72c8123392f2a1679ca419be15ef660818f2b75a06b05cb0250a469198
+  data.tar.gz: dbbf8a825b893490963db61c6b08d2adf45ef62be0bad19d44235b1b822077ebb4e8728d45d119dddfb6f33e39ecb3d69ccb89ceda938275b8371306ca72efaa

data/CHANGELOG.md

@@ -3,6 +3,24 @@
 All notable changes to this project will be documented in this file.
 This project *loosely tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [0.4.5]
+- #31 kubes AWS helpers
+
+## [0.4.4]
+- #30 friendly message for rendered erb yaml and dsl errors
+- fix backtrace_reject pattern
+
+## [0.4.3]
+- #29 fix edge case when user provides hook on option for non-kubectl hooks
+
+## [0.4.2]
+- #28 base64 helper
+
+## [0.4.1]
+- kubes init: default namespace now includes Kubes.env
+- fix kubes deploy: compile it gets called once and output folder kept
+- include kubes_google dependency helpers
+
 ## [0.4.0]
 - #26 features: kubes vs kubectl hooks, prune, etc
 - hooks: kubes, kubectl, docker breaking changes.

data/docs/_docs/config/hooks/docker.md

@@ -58,7 +58,7 @@ Results in:
 
 ## exit_on_fail option
 
-By default, if the hook commands fail, then terraspace will exit with the original hook error code.  You can change this behavior with the `exit_on_fail` option.
+By default, if the hook commands fail, then kubes will exit with the original hook error code.  You can change this behavior with the `exit_on_fail` option.
 
 ```ruby
 before("build"

data/docs/_docs/config/hooks/kubectl.md

@@ -70,7 +70,7 @@ The `on` option is used to match the path the gets applied: .kubes/resources/**w
 
 ## exit on fail
 
-By default, if the hook commands fail, then terraspace will exit with the original hook error code.  You can change this behavior with the `exit_on_fail` option.
+By default, if the hook commands fail, then kubes will exit with the original hook error code.  You can change this behavior with the `exit_on_fail` option.
 
 ```ruby
 before("apply"

data/docs/_docs/config/hooks/ruby.md

@@ -18,7 +18,7 @@ When the Ruby object is a class with an instance method `call`, Kubes creates a
 ```ruby
 class EnvExporter
   def call
-    ENV['SECRET_FOO'] = Base64.encode64("hi").strip
+    ENV['SECRET_FOO'] = Base64.strict_encode64("hi").strip
   end
 end
 
@@ -46,9 +46,11 @@ metadata:
 apiVersion: v1
 kind: Secret
 data:
-  foo: <%= ENV['SECRET_FOO']
+  foo: <%= ENV['SECRET_FOO'] %>
 ```
 
+Note, the example above is used to explain how Ruby can be used as the execute option. For secrets, kubes supports secrets with some helpers. See: [Secrets Pattern Docs]({% link _docs/patterns/secrets.md %})
+
 ## Ruby Object
 
 When the Ruby object, Kubes expects it to have a `call` method and will run it.  Example:
@@ -57,7 +59,7 @@ When the Ruby object, Kubes expects it to have a `call` method and will run it.
 
 ```ruby
 before("compile",
-  execute: lambda { ENV['SECRET_FOO'] = Base64.encode64("hi2").strip }
+  execute: lambda { ENV['SECRET_FOO'] = Base64.strict_encode64("hi2").strip }
 )
 ```
 

data/docs/_docs/helpers.md

@@ -7,13 +7,21 @@ Kubes provides some helper methods to help write Kubernetes YAML files.  Here's
 Helper | Description
 --- | ---
 built_image | Method refers to the latest Docker image built by Kubes. This spares you from having to update the image manually in the deployment resource.
+decode64 | Basey64d decode a string.
 dockerfile_port	| Exposed port extracted from the Dockerfile of the project.
+encode64 | Basey64 encode a string. Also available as `base64` method.
 extra | The `KUBES_EXTRA` value.
 with_extra | Appends the `KUBES_EXTRA` value to a string if it's set. It's covered in the [Extra Env Docs]({% link _docs/extra-env.md %}).
 
-Here's also the source code with the helpers: [helpers.rb](https://github.com/boltops-tools/kubes/blob/master/lib/kubes/compiler/shared/helpers.rb).
-
+Here's also the source code with most of the helpers: [helpers.rb](https://github.com/boltops-tools/kubes/blob/master/lib/kubes/compiler/shared/helpers.rb).
 
 ## DSL Specific Methods
 
-Each DSL resource has it's own specific methods. Refer to the [DSL Docs]({% link _docs/dsl.md %}) for their methods.
+Each DSL resource has it's own specific methods. Refer to the [DSL Docs]({% link _docs/dsl.md %}) for their methods.
+
+## Provider Helpers
+
+There are also provider-specific helpers:
+
+* [AWS Helpers]({% link _docs/helpers/aws.md %})
+* [Google Helpers]({% link _docs/helpers/google.md %})

data/docs/_docs/helpers/aws.md

@@ -0,0 +1,14 @@
+---
+title: AWS Helpers
+---
+
+List of AWS helpers:
+
+{% assign docs = site.docs | where: "categories","helpers-aws" %}
+{% for doc in docs -%}
+  * [{{ doc.nav_text }}]({{ doc.url }})
+{% endfor %}
+
+## Notes
+
+* By default, `KubeGoogle.logger = Kubes.logger`. This means, you can set `logger.level = "debug"` in `.kubes/config.rb` to see more details.

data/docs/_docs/helpers/aws/iam-role.md

@@ -0,0 +1,91 @@
+---
+title: AWS IAM Role
+nav_text: IAM Role
+categories: helpers-aws
+---
+
+You can automatically create the IAM Role associated with the Kubernetes Service Account, covered in [Introducing fine-grained IAM roles for service accounts](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/).
+
+Here's a Kubes hook that creates an IAM Role:
+
+.kubes/config/hooks/kubes.rb
+
+```ruby
+iam_role = KubesAws::IamRole.new(
+  app: "demo",
+  namespace: "demo-#{Kubes.env}", # defaults to APP-ENV when not set. IE: demo-dev
+  managed_policies: ["AmazonS3ReadOnlyAccess", "AmazonSSMReadOnlyAccess"], # defaults to empty when not set
+  inline_policies: [:secrets_read_only], # See Secrets Read Only Inline Policy at the bottom
+)
+before("apply",
+  label: "create iam role",
+  execute: iam_role,
+)
+KubesAws::IamRole.role_arn = iam_role.arn # used in .kubes/resources/shared/service_account.yaml
+```
+
+The corresponding Kubernetes Service account looks like this:
+
+.kubes/resources/shared/service_account.yaml
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  annotations:
+    eks.amazonaws.com/role-arn: <%= KubesAws::IamRole.role_arn %>
+  name: demo
+  labels:
+    app: demo
+```
+
+The role policy permissions are currently always added to the existing permissions. So removing roles that were previously added does not remove them.
+
+IamRole#initialize options:
+
+Variable | Description | Default
+---|---|---
+app | The app name. It's used to set other variables conventionally. This is required. | nil
+ksa | The Kubernetes Service Account name. The conventional name is APP. IE: demo | APP
+namespace | The Kubernetes namespace. Defaults to the APP-ENV. IE: demo-dev. | APP-ENV
+policies | IAM policies to add. This adds permissions to the IAM Role. | []
+role_name | The IAM Role name. The conventional name is APP-ENV. IE: demo-dev. | APP-ENV
+
+## OpenID Connect Provider
+
+The `KubesAws::IamRole` class also automatically creates the OpenID Connect Provider if it doesn't already exist.
+
+## Secrets Read-Only Inline Policy
+
+Note the the `:secrets_read_only` is a way to generate an Inline Policy that represents read-only access for Secrets. Kubes does this since there's no managed policy for this yet. For example:
+
+```ruby
+inline_policies: [:secrets_read_only]
+```
+
+Is the same as:
+
+```ruby
+inline_secrets_read_only = {
+  policy_document: {
+    Version: "2012-10-17",
+    Statement: {
+      Effect: "Allow",
+      Action: [
+        "secretsmanager:Describe*",
+        "secretsmanager:Get*",
+        "secretsmanager:List*"
+      ],
+      Resource: "*"
+    }
+  },
+  policy_name: "SecretsReadOnly",
+}
+iam_role = KubesAws::IamRole.new(
+  app: "rails",
+  cluster: "dev-cluster",
+  namespace: "rails-#{Kubes.env}", # defaults to APP-ENV when not set. IE: rails-dev
+  managed_policies: ["AmazonS3ReadOnlyAccess", "AmazonSSMReadOnlyAccess"], # defaults to empty when not set
+  inline_policies: [inline_secrets_read_only],
+)
+```

data/docs/_docs/helpers/aws/secrets.md

@@ -0,0 +1,129 @@
+---
+title: AWS Secrets
+nav_text: Secrets
+categories: helpers-aws
+---
+
+## Simple Values
+
+For example if you have these secret values:
+
+    $ aws secretsmanager get-secret-value --secret-id demo/dev/db_user | jq '.SecretString'
+    user
+    $ aws secretsmanager get-secret-value --secret-id demo/dev/db_pass | jq '.SecretString'
+    pass
+
+Set up a [Kubes hook](https://kubes.guru/docs/config/hooks/kubes/).
+
+.kubes/config/hooks/kubes.rb
+
+```ruby
+secrets = KubesAws::Secrets.new(upcase: true, prefix: "demo/dev/")
+before("compile",
+  label: "Get secrets from AWS Secrets Manager",
+  execute: secrets,
+)
+```
+
+Then set the secrets in the YAML:
+
+.kubes/resources/shared/secret.yaml
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: demo
+  labels:
+    app: demo
+data:
+<% KubesAws::Secrets.data.each do |k,v| -%>
+  <%= k %>: <%= base64(v) %>
+<% end -%>
+```
+
+This results in AWS secrets with the prefix the `demo/dev/` being added to the Kubernetes secret data.  The values are automatically base64 encoded. Produces:
+
+.kubes/output/shared/secret.yaml
+
+```yaml
+metadata:
+  namespace: demo
+  name: demo-2a78a13682
+  labels:
+    app: demo
+apiVersion: v1
+kind: Secret
+data:
+  db_pass: dGVzdDEK
+  db_user: dGVzdDIK
+```
+
+## JSON Values
+
+For example if you have these secret values:
+
+    $ aws secretsmanager get-secret-value --secret-id demo/dev/k2 | jq '.SecretString'
+    {\"a\":1,\"b\":2}"
+
+Set up a [Kubes hook](https://kubes.guru/docs/config/hooks/kubes/).
+
+.kubes/config/hooks/kubes.rb
+
+```ruby
+secrets = KubesAws::Secrets.new(prefix: "rails/dev/")
+before("compile",
+  label: "Get secrets from AWS Secrets Manager",
+  execute: secrets,
+)
+```
+
+Then set the secrets in the YAML:
+
+.kubes/resources/shared/secret.yaml
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: demo
+  labels:
+    app: demo
+data:
+<% k2 = JSON.load(KubesAws::Secrets.data["k2"]) %>
+  a: <%= base64(k2["a"]) %>
+  b: <%= base64(k2["b"]) %>
+```
+
+Produces:
+
+```yaml
+metadata:
+  namespace: demo-dev
+  name: demo-a4cd604a95
+  labels:
+    app: demo
+apiVersion: v1
+kind: Secret
+data:
+  a: MQ==
+  b: Mg==
+```
+
+## Variables
+
+These environment variables can be set:
+
+Name | Description
+---|---
+AWS_SECRET_PREFIX | Prefixed used to list and filter AWS secrets. IE: `demo/dev/`.
+
+Secrets#initialize options:
+
+Variable | Description | Default
+---|---|---
+base64 | Automatically base64 encode the values. | false
+upcase | Automatically upcase the Kubernetes secret data keys. | false
+prefix | Prefixed used to list and filter AWS secrets. IE: `demo/dev/`. Can also be set with the `AWS_SECRET_PREFIX` env variable. The env variable takes the highest precedence. | nil
+
+{% include helpers/base64.md %}

data/docs/_docs/helpers/aws/ssm.md

@@ -0,0 +1,76 @@
+---
+title: AWS SSM Parameters
+nav_text: SSM
+categories: helpers-aws
+---
+
+For example if you have these secret values:
+
+    $ aws ssm get-parameter --name /demo/development/db_user --with-decryption | jq '.Parameter.Value'
+    user
+    $ aws ssm get-parameter --name /demo/development/db_pass --with-decryption | jq '.Parameter.Value'
+    pass
+
+Set up a [Kubes hook](https://kubes.guru/docs/config/hooks/kubes/).
+
+.kubes/config/hooks/kubes.rb
+
+```ruby
+ssm = KubesAws::SSM.new(upcase: true, prefix: "/demo/development/")
+before("compile",
+  label: "Get secrets from AWS SSM Manager",
+  execute: ssm,
+)
+```
+
+Then set the secrets in the YAML:
+
+.kubes/resources/shared/secret.yaml
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: demo
+  labels:
+    app: demo
+data:
+<% KubesAws::SSM.data.each do |k,v| -%>
+  <%= k %>: <%= base64(v) %>
+<% end -%>
+```
+
+This results in AWS secrets with the prefix the `demo/dev/` being added to the Kubernetes secret data.  The values are automatically base64 encoded. Produces:
+
+.kubes/output/shared/secret.yaml
+
+```yaml
+metadata:
+  namespace: demo
+  name: demo-2a78a13682
+  labels:
+    app: demo
+apiVersion: v1
+kind: Secret
+data:
+  db_pass: dGVzdDEK
+  db_user: dGVzdDIK
+```
+
+## Variables
+
+These environment variables can be set:
+
+Name | Description
+---|---
+AWS_SSM_PREFIX | Prefixed used to list and filter AWS SSM Parameters. IE: `demo/dev/`.
+
+Secrets#initialize options:
+
+Variable | Description | Default
+---|---|---
+base64 | Automatically base64 encode the values. | false
+upcase | Automatically upcase the Kubernetes secret data keys. | false
+prefix | Prefixed used to list and filter AWS secrets. IE: `demo/dev/`. Can also be set with the `AWS_SECRET_PREFIX` env variable. The env variable takes the highest precedence. | nil
+
+{% include helpers/base64.md %}

data/docs/_docs/helpers/google.md

@@ -0,0 +1,17 @@
+---
+title: Google Helpers
+---
+
+List of Google helpers:
+
+{% assign docs = site.docs | where: "categories","helpers-aws" %}
+{% for doc in docs -%}
+  * [{{ doc.nav_text }}]({{ doc.url }})
+{% endfor %}
+
+## Notes
+
+* By default, `KubeGoogle.logger = Kubes.logger`. This means, you can set `logger.level = "debug"` in `.kubes/config.rb` to see more details.
+* The `gcloud` cli is used to create IAM roles. So `gcloud` is required.
+* Note: Would like to use the google sdk, but it wasn't obvious how to do so. PRs are welcomed.
+

data/docs/_docs/helpers/google/secrets.md

@@ -0,0 +1,76 @@
+---
+title: Google Secrets
+nav_text: Secrets
+categories: helpers-google
+---
+
+Set up a [Kubes hook](https://kubes.guru/docs/config/hooks/kubes/).
+
+.kubes/config/hooks/kubes.rb
+
+```ruby
+before("compile",
+  execute: KubesGoogle::Secrets.new(upcase: true, prefix: 'projects/686010496118/secrets/demo-dev-')
+)
+```
+
+Then set the secrets in the YAML:
+
+.kubes/resources/shared/secret.yaml
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: demo
+  labels:
+    app: demo
+data:
+<% KubesGoogle::Secrets.data.each do |k,v| -%>
+  <%= k %>: <%= base64(v) %>
+<% end -%>
+```
+
+This results in Google secrets with the prefix the `demo-dev-` being added to the Kubernetes secret data.  The values are automatically base64 encoded.
+
+For example if you have these secret values:
+
+    $ gcloud secrets versions access latest --secret demo-dev-db_user
+    test1
+    $ gcloud secrets versions access latest --secret demo-dev-db_pass
+    test2
+    $
+
+.kubes/output/shared/secret.yaml
+
+```yaml
+metadata:
+  namespace: demo
+  name: demo-2a78a13682
+  labels:
+    app: demo
+apiVersion: v1
+kind: Secret
+data:
+  db_pass: dGVzdDEK
+  db_user: dGVzdDIK
+```
+
+## Variables
+
+These environment variables can be set:
+
+Name | Description
+---|---
+GCP_SECRET_PREFIX | Prefixed used to list and filter Google secrets. IE: `projects/686010496118/secrets/demo-dev-`.
+GOOGLE_PROJECT | Google project id.
+
+Secrets#initialize options:
+
+Variable | Description | Default
+---|---|---
+base64 | Automatically base64 encode the values. | false
+upcase | Automatically upcase the Kubernetes secret data keys. | false
+prefix | Prefixed used to list and filter Google secrets. IE: `projects/686010496118/secrets/demo-dev-`. Can also be set with the `GCP_SECRET_PREFIX` env variable. The env variable takes the highest precedence. | nil
+
+{% include helpers/base64.md %}

data/docs/_docs/helpers/google/service-account.md

@@ -0,0 +1,52 @@
+---
+title: Google Service Account
+nav_text: Service Account
+categories: helpers-google
+---
+
+## Service Accounts
+
+You can automatically create the Google Service Account associated with the [GKE Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).
+
+Here's a Kubes hook that creates a service account:
+
+.kubes/config/hooks/kubes.rb
+
+```ruby
+service_account = KubesGoogle::ServiceAccount.new(
+  app: "demo",
+  namespace: "demo-#{Kubes.env}", # defaults to APP-ENV when not set. IE: demo-dev
+  roles: ["cloudsql.client", "secretmanager.viewer"], # defaults to empty when not set
+)
+before("apply",
+  label: "create service account",
+  execute: service_account,
+)
+```
+
+The corresponding Kubernetes Service account looks like this:
+
+.kubes/resources/shared/service_account.yaml
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  annotations:
+    iam.gke.io/gcp-service-account: demo-<%= Kubes.env %>@<%= ENV['GOOGLE_PROJECT'] %>.iam.gserviceaccount.com
+  name: demo
+  labels:
+    app: demo
+```
+
+The role permissions are currently always added to the existing permissions. So removing roles that were previously added does not remove them.
+
+ServiceAccount#initialize options:
+
+Variable | Description | Default
+---|---|---
+app | The app name. It's used to set other variables conventionally. This is required. | nil
+gsa | The Google Service Account name. The conventional name is APP-ENV. IE: demo-dev. | APP-ENV
+ksa | The Kubernetes Service Account name. The conventional name is APP. IE: demo | APP
+namespace | The Kubernetes namespace. Defaults to the APP-ENV. IE: demo-dev. | APP-ENV
+roles | Google IAM roles to add. This adds permissions to the Google service account. | []

data/docs/_docs/learn/dsl/review-project.md

@@ -11,10 +11,12 @@ We'll create a namespace for the app resources:
 .kubes/resources/shared/namespace.rb
 
 ```ruby
-name "demo"
+name "demo-#{Kubes.env}"
 labels(app: "demo")
 ```
 
+Notice, the `#{Kubes.env}`. Kubes adds the env to the namespace by default. You can change this with the `init --namespace` option.
+
 ## Deployment
 
 The `web/deployment.rb` file is a little more interesting:
@@ -39,7 +41,7 @@ Also let's check the files in the base folder.
 .kubes/resources/base/all.rb
 
 ```ruby
-namespace "default"
+namespace "demo-#{Kubes.env}"
 labels(app: "demo")
 ```
 

data/docs/_docs/learn/yaml/review-project.md

@@ -14,11 +14,13 @@ We'll create a namespace for the app resources:
 apiVersion: v1
 kind: Namespace
 metadata:
-  name: demo
+  name: demo-<%= Kubes.env %>
   labels:
     app: demo
 ```
 
+Notice, the `<%= Kubes.env %>`. Kubes adds the env to the namespace by default. You can change this with the `init --namespace` option.
+
 ## Deployment
 
 The `web/deployment.yaml` file is a little more interesting:
@@ -57,7 +59,7 @@ Also let's check the files in the base folder.
 
 ```yaml
 metadata:
-  namespace: demo
+  namespace: demo-<%= Kubes.env %>
 ```
 
 .kubes/resources/base/deployment.yaml

data/docs/_docs/patterns.md

@@ -4,4 +4,7 @@ title: Patterns
 
 We'll cover some common deployment patterns here:
 
-* [Clock Web Worker]({% link _docs/patterns/clock-web-worker.md %})
+{% assign docs = site.docs | where: "categories","patterns" %}
+{% for doc in docs -%}
+* [{{ doc.title }}]({{ doc.url }})
+{% endfor %}

data/docs/_docs/patterns/clock-web-worker.md

@@ -1,5 +1,7 @@
 ---
 title: Clock Web Worker Pattern
+nav_text: Clock Web Worker
+categories: patterns
 ---
 
 A common pattern is to use the same code to run different types of processes like clock, web, worker. Kubes is flexible enough to support this pattern.

data/docs/_docs/patterns/migrations.md

@@ -1,5 +1,7 @@
 ---
 title: Database Migrations
+nav_text: Database Migrations
+categories: patterns
 ---
 
 A common task is to run database migrations. You can use Kubes hooks to achieve this as part of the `kubes deploy` process.

data/docs/_docs/patterns/secrets.md

@@ -0,0 +1,82 @@
+---
+title: Secrets
+nav_text: Secrets
+categories: patterns
+---
+
+A Google Secrets helper is currently supported.
+
+## Set Up Kubes Hook
+
+Set up a [Kubes hook]({% link _docs/config/hooks/kubes.md %}).
+
+.kubes/config/hooks/kubes.rb
+
+```ruby
+before("compile",
+  execute: KubesGoogle::Secrets.new(upcase: true, prefix: 'projects/686010496118/secrets/demo-dev-')
+)
+```
+
+Then set the secrets in the YAML:
+
+.kubes/resources/shared/secret.yaml
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: demo
+  labels:
+    app: demo
+data:
+<% KubesGoogle::Secrets.data.each do |k,v| -%>
+  <%= k %>: <%= base64(v) %>
+<% end -%>
+```
+
+This results in Google secrets with the prefix the `demo-dev-` being added to the Kubernetes secret data.  The values are base64 encoded.
+
+For example if you have these secret values:
+
+    $ gcloud secrets versions access latest --secret demo-dev-db_user
+    test1
+    $ gcloud secrets versions access latest --secret demo-dev-db_pass
+    test2
+    $
+
+The compiled secrets.yaml looks like this:
+
+.kubes/output/shared/secret.yaml
+
+```yaml
+metadata:
+  namespace: demo
+  name: demo-2a78a13682
+  labels:
+    app: demo
+apiVersion: v1
+kind: Secret
+data:
+  db_pass: dGVzdDEK
+  db_user: dGVzdDIK
+```
+
+These environment variables can be set:
+
+Name | Description
+---|---
+GCP_SECRET_PREFIX | Prefixed used to list and filter Google secrets. IE: `projects/686010496118/secrets/demo-dev-`.
+GOOGLE_PROJECT | Google project id.
+
+Secrets#initialize options:
+
+Variable | Description | Default
+---|---|---
+base64 | Automatically base64 encode the values. | false
+upcase | Automatically upcase the Kubernetes secret data keys. | false
+prefix | Prefixed used to list and filter Google secrets. IE: `projects/686010496118/secrets/demo-dev-`. Can also be set with the `GCP_SECRET_PREFIX` env variable. The env variable takes the highest precedence. | nil
+
+Note, Kubernetes secrets are only base64 encoded. So users who have access to read Kubernetes secrets will be able to decode and get the value trivially. Depending on your security posture requirements, this may or may not suffice.
+
+The Google helpers are provided by the [boltops-tools/kubes_google](https://github.com/boltops-tools/kubes_google) library. For more details, check out its README.

data/docs/_includes/config/hooks/options.md

@@ -10,7 +10,7 @@ The command name corresponds to the `{{ include.command }}` commands: apply, del
 
 Name | Description
 ---|---
-label | A human-friendly label so you can see what hooks is being run during `kubes apply`
+label | A human-friendly label so you can see what hooks is being run.
 execute | The script or command to run. IE: path/to/some/script.sh
 exit_on_fail | Whether or not to continue process if the script returns an failed exit code.
 {% if include.command == "kubectl" %}on | What resource to run the hook on. IE: shared/namespace, web/deployment, web/service. Note: This option is only used by kubectl hooks.{% endif %}

data/docs/_includes/helpers/base64.md

@@ -0,0 +1 @@
+Note, Kubernetes secrets are only base64 encoded. So users who have access to read Kubernetes secrets will be able to decode and get the value trivially. Depending on your security posture requirements, this may or may not suffice.

data/docs/_includes/layering/layers.md

@@ -27,7 +27,7 @@ To explain the layering, here's the general processing order that Kubes takes.
 
 Note, both YAML and DSL forms support layering.
 
-Layering only combines resources definitions with the same form. For example, `base/all.rb` will not be combined with `web/deployment.yaml`.
+Layering only combines resources definitions with the same form. For example, the DSL form `base/all.rb` will not be combined with YAML form `web/deployment.yaml`.
 
 ## Full Layering
 

data/docs/_includes/sidebar.html

@@ -97,11 +97,32 @@
           <li><a href="{% link _docs/dsl/multiple-resources.md %}">Multiple Resources</a>
         </ul>
       </li>
-      <li><a href="{% link _docs/helpers.md %}">Helpers</a></li>
+      <li><a href="{% link _docs/helpers.md %}">Helpers</a>
+        <ul>
+          <li><a href="{% link _docs/helpers/aws.md %}">AWS</a>
+            <ul>
+              {% assign docs = site.docs | where: "categories","helpers-aws" %}
+              {% for doc in docs -%}
+                <li><a href="{{ doc.url }}">{{ doc.nav_text }}</a></li>
+              {% endfor %}
+            </ul>
+          </li>
+          <li><a href="{% link _docs/helpers/google.md %}">Google</a>
+            <ul>
+              {% assign docs = site.docs | where: "categories","helpers-google" %}
+              {% for doc in docs -%}
+                <li><a href="{{ doc.url }}">{{ doc.nav_text }}</a></li>
+              {% endfor %}
+            </ul>
+          </li>
+        </ul>
+      </li>
       <li><a href="{% link _docs/patterns.md %}">Patterns</a>
         <ul>
-          <li><a href="{% link _docs/patterns/migrations.md %}">Database Migrations</a></li>
-          <li><a href="{% link _docs/patterns/clock-web-worker.md %}">Clock Web Worker</a></li>
+          {% assign docs = site.docs | where: "categories","patterns" %}
+          {% for doc in docs -%}
+            <li><a href="{{ doc.url }}">{{ doc.nav_text }}</a></li>
+          {% endfor %}
         </ul>
       </li>
       <li><a href="{% link _docs/extra-env.md %}">Extra Env</a>

data/kubes.gemspec

@@ -28,6 +28,10 @@ Gem::Specification.new do |spec|
   spec.add_dependency "thor"
   spec.add_dependency "zeitwerk"
 
+  # core helper libs
+  spec.add_dependency "kubes_aws"
+  spec.add_dependency "kubes_google"
+
   spec.add_development_dependency "bundler"
   spec.add_development_dependency "byebug"
   spec.add_development_dependency "cli_markdown"

data/lib/kubes.rb

@@ -14,7 +14,11 @@ require "memoist"
 require "rainbow/ext/string"
 require "yaml"
 
-DslEvaluator.backtrace_reject = ".kubes"
+# core helper libraries
+require "kubes_aws"
+require "kubes_google"
+
+DslEvaluator.backtrace_reject = "lib/kubes"
 
 require "kubes/autoloader"
 Kubes::Autoloader.setup

data/lib/kubes/cli/compile.rb

@@ -1,8 +1,16 @@
 class Kubes::CLI
   class Compile < Base
+    # Separate command like prune can call compile. Apply also calls Prune.
+    # Instead of moving Compile out of Prune, will use this class variable.
+    # In case we have other cases where compile is called in another area.
+    # We only want compiled to be called once so hooks only fire once.
+    # Done here so we don't clean and remove the .kubes/output folder.
+    @@compiled = false
     def run
+      return if @@compiled
       Clean.new(@options.merge(mute: true)).run
       Kubes::Compiler.new(@options).run
+      @@compiled = true
     end
   end
 end

data/lib/kubes/cli/init.rb

@@ -6,7 +6,7 @@ class Kubes::CLI
         [:force, type: :boolean, desc: "Bypass overwrite are you sure prompt for existing files"],
         [:type, aliases: ["t"], default: "yaml", desc: "Type: dsl or yaml"],
         [:repo, required: true, desc: "Docker repo name. Example: user/repo. Configures .kubes/config.rb"],
-        [:namespace, aliases: ["n"], desc: "Namespace to use, defaults to the app option"],
+        [:namespace, aliases: ["n"], desc: "Namespace to use, defaults to APP-ENV. IE: demo-dev"],
       ]
     end
 
@@ -19,7 +19,12 @@ class Kubes::CLI
     end
 
     def namespace
-      @options[:namespace] || @options[:app]
+      @options[:namespace] || default_namespace
+    end
+
+    def default_namespace
+      env = @options[:type] == "yaml" ? '<%= Kubes.env %>' : '#{Kubes.env}'
+      "#{app}-#{env}"
     end
 
     def excludes

data/lib/kubes/compiler.rb

@@ -8,13 +8,7 @@ module Kubes
       @options = options
     end
 
-    # Separate command like prune can call compile. Apply also calls Prune.
-    # Instead of moving Compile out of Prune, will use this class variable.
-    # In case we have other cases where compile is called in another area.
-    # We only want compiled to be called once so hooks only fire once.
-    @@compiled = false
     def run
-      return if @@compiled
       Kubes.config # trigger config load. So can set ENV['VAR'] in config/envs/dev.rb etc
       run_hooks("kubes.rb", name: "compile") do
         results = resources.map do |path|
@@ -28,7 +22,6 @@ module Kubes
       end
 
       puts "Compiled  .kubes/resources files to .kubes/output" if show_compiled_message?
-      @@compiled = true
     end
 
     def resources

data/lib/kubes/compiler/shared/helpers.rb

@@ -24,8 +24,13 @@ module Kubes::Compiler::Shared
       extra&.strip&.empty? ? nil : extra # if blank string then also return nil
     end
 
-    def base64(v)
-      Base64.encode64(v).strip
+    def encode64(v)
+      Base64.strict_encode64(v.to_s).strip
+    end
+    alias_method :base64, :encode64
+
+    def decode64(v)
+      Base64.strict_decode64(v)
     end
   end
 end

data/lib/kubes/compiler/strategy/base.rb

@@ -1,13 +1,12 @@
 class Kubes::Compiler::Strategy
   class Base
     include Kubes::Logging
+    include Kubes::Compiler::Util::SaveFile
 
     def initialize(options={})
       @options = options
       @path = options[:path]
-
-      @filename = @path.sub(%r{.*\.kubes/resources/},'') # IE: web/deployment.rb or web/deployment.yaml
-      @save_file = @filename.sub('.yml','.yaml').sub('.rb','.yaml')
+      @save_file = save_file(@path)
     end
   end
 end

data/lib/kubes/compiler/strategy/dsl.rb

@@ -17,7 +17,7 @@ class Kubes::Compiler::Strategy
     end
 
     def syntax_class
-      klass_name = normalize_kind(@filename)
+      klass_name = normalize_kind(@save_file)
       "Kubes::Compiler::Dsl::Syntax::#{klass_name}".constantize
     rescue NameError
       logger.debug "Using default resource for: #{klass_name}"
@@ -25,7 +25,7 @@ class Kubes::Compiler::Strategy
     end
 
     def block_form?
-      type = extract_type(@filename)
+      type = extract_type(@save_file)
       type.pluralize == type
     end
   end

data/lib/kubes/compiler/strategy/erb.rb

@@ -32,11 +32,18 @@ class Kubes::Compiler::Strategy
     def render_result(path)
       if File.exist?(path)
         yaml = RenderMePretty.result(path, context: self)
-        result = YAML.load(yaml)
+        result = yaml_load(path, yaml)
         result.is_a?(Hash) ? result : {} # in case of blank yaml doc a Boolean false is returned
       else
         {}
       end
     end
+
+    def yaml_load(path, yaml)
+      YAML.load(yaml)
+    rescue Psych::SyntaxError
+      YamlError.new(path, yaml).show
+      exit 1
+    end
   end
 end

data/lib/kubes/compiler/strategy/erb/yaml_error.rb

@@ -0,0 +1,60 @@
+class Kubes::Compiler::Strategy::Erb
+  class YamlError
+    include Kubes::Compiler::Util::SaveFile
+
+    def initialize(path, rendered_yaml)
+      @path, @rendered_yaml = path, rendered_yaml
+    end
+
+    def show
+      FileUtils.mkdir_p(File.dirname(dest))
+      IO.write(dest, @rendered_yaml)
+      show_error(dest)
+    end
+
+    def dest
+      save_file = save_file(@path)
+      "#{Kubes.root}/.kubes/output/#{save_file}"
+    end
+
+    def show_error(path)
+      text = IO.read(path)
+      begin
+        YAML.load(text)
+      rescue Psych::SyntaxError => e
+        handle_yaml_syntax_error(e, path)
+      end
+    end
+
+    def handle_yaml_syntax_error(e, path)
+      io = StringIO.new
+      io.puts "Invalid yaml. Output written for debugging: #{path}".color(:red)
+      io.puts "ERROR: #{e.message}".color(:red)
+
+      # Grab line info.  Example error:
+      #   ERROR: (<unknown>): could not find expected ':' while scanning a simple key at line 2 column 1
+      md = e.message.match(/at line (\d+) column (\d+)/)
+      line = md[1].to_i
+
+      lines = IO.read(path).split("\n")
+      context = 5 # lines of context
+      top, bottom = [line-context-1, 0].max, line+context-1
+      spacing = lines.size.to_s.size
+      lines[top..bottom].each_with_index do |line_content, index|
+        line_number = top+index+1
+        if line_number == line
+          io.printf("%#{spacing}d %s\n".color(:red), line_number, line_content)
+        else
+          io.printf("%#{spacing}d %s\n", line_number, line_content)
+        end
+      end
+
+      if ENV['KUBES_TEST']
+        io.string
+      else
+        puts io.string
+        exit 1
+      end
+    end
+  end
+end

data/lib/kubes/compiler/util/normalize.rb

@@ -5,7 +5,7 @@ module Kubes::Compiler::Util
     end
 
     def extract_type(path)
-      File.basename(path).sub('.rb','').sub(/-.*/,'')
+      File.basename(path).sub('.yaml','').sub('.yml','').sub('.rb','').sub(/-.*/,'')
     end
   end
 end

data/lib/kubes/compiler/util/save_file.rb

@@ -0,0 +1,8 @@
+module Kubes::Compiler::Util
+  module SaveFile
+    def save_file(path)
+      filename = path.sub(%r{.*\.kubes/resources/},'') # IE: web/deployment.rb or web/deployment.yaml
+      filename.sub('.yml','.yaml').sub('.rb','.yaml')
+    end
+  end
+end

data/lib/kubes/hooks/builder.rb

@@ -50,7 +50,7 @@ module Kubes::Hooks
     def run?(hook)
       return false unless hook["execute"]
       return true  unless hook["on"]
-      @output_file.include?(hook["on"])
+      @output_file && @output_file.include?(hook["on"]) # output file is only passed
     end
   end
 end

data/lib/kubes/version.rb

@@ -1,3 +1,3 @@
 module Kubes
-  VERSION = "0.4.0"
+  VERSION = "0.4.5"
 end

data/spec/fixtures/decorators/deployment/both/valueFrom.yaml

@@ -0,0 +1,33 @@
+---
+metadata:
+  namespace: default
+  labels:
+    app: demo
+    role: web
+  name: demo-web
+spec:
+  selector:
+    matchLabels:
+      app: demo
+      role: web
+  template:
+    metadata:
+      labels:
+        app: demo
+        role: web
+    spec:
+      containers:
+      - name: demo-web
+        image: gcr.io/project/demo-web:kubes-2020-06-23T00-07-54
+        env:
+        - name: MYSQL_ROOT_PASSWORD
+          valueFrom:
+            configMapKeyRef:
+              name: demo-config-map
+              key: password
+            secretKeyRef:
+              name: demo-secret
+              key: password
+  replicas: 1
+apiVersion: apps/v1
+kind: Deployment

data/spec/fixtures/decorators/deployment/both/volumes.yaml

@@ -0,0 +1,40 @@
+---
+metadata:
+  namespace: default
+  labels:
+    app: demo
+    role: web
+  name: demo-web
+spec:
+  selector:
+    matchLabels:
+      app: demo
+      role: web
+  template:
+    metadata:
+      labels:
+        app: demo
+        role: web
+    spec:
+      containers:
+      - name: demo-web
+        image: gcr.io/project/demo-web:kubes-2020-06-23T00-07-54
+        volumeMounts:
+        - mountPath: /config-map
+          name: config-map-volume
+      volumes:
+      - configMap:
+          name: demo-config-map
+          items:
+          - key: k1
+            path: config-map.conf
+        name: config-map-volume
+      - secret:
+          secretName: demo-secret
+          items:
+          - key: k1
+            path: secrets.conf
+        name: secrets-volume
+  replicas: 1
+apiVersion: apps/v1
+kind: Pod

data/spec/kubes/compiler/decorator/post/deployment_spec.rb

@@ -5,6 +5,7 @@ describe Kubes::Compiler::Decorator::Post do
     YAML.load_file("spec/fixtures/decorators/deployment/#{name}.yaml")
   end
   before(:each) do
+    allow(Kubes::Compiler::Decorator::Hashable::Storage).to receive(:fetch).and_return("fakehash")
     allow(Kubes::Compiler::Decorator::Hashable::Storage).to receive(:fetch).with("Secret", "demo-secret").and_return("fakehash")
     allow(Kubes::Compiler::Decorator::Hashable::Storage).to receive(:fetch).with("ConfigMap", "demo-config-map").and_return("fakehash-config")
     allow(Kubes::Compiler::Decorator::Hashable::Storage).to receive(:fetch).with("ConfigMap", "demo-config-map-2").and_return("fakehash-config2")
@@ -107,5 +108,31 @@ describe Kubes::Compiler::Decorator::Post do
         expect(name).to eq("demo-config-map-2-fakehash-config2")
       end
     end
+
+    describe "valueFrom" do
+      let(:data) { fixture("both/valueFrom") }
+      it "run" do
+        decorator.run
+        data = decorator.data
+        valueFrom = data['spec']['template']['spec']['containers'][0]['env'][0]['valueFrom']
+        name = valueFrom['configMapKeyRef']['name']
+        expect(name).to eq("demo-config-map-fakehash-config")
+        name = valueFrom['secretKeyRef']['name']
+        expect(name).to eq("demo-secret-fakehash")
+      end
+    end
+
+    describe "volumes" do
+      let(:data) { fixture("both/volumes") }
+      it "run" do
+        decorator.run
+        data = decorator.data
+        volumes = data['spec']['template']['spec']['volumes']
+        name = volumes[0]['configMap']['name']
+        expect(name).to eq("demo-config-map-fakehash-config")
+        name = volumes[1]['secret']['secretName']
+        expect(name).to eq("demo-secret-fakehash")
+      end
+    end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kubes
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.4.5
 platform: ruby
 authors:
 - Tung Nguyen
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-10-11 00:00:00.000000000 Z
+date: 2020-10-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -136,6 +136,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: kubes_aws
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: kubes_google
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -271,6 +299,13 @@ files:
 - docs/_docs/extra-env/dsl.md
 - docs/_docs/extra-env/yaml.md
 - docs/_docs/helpers.md
+- docs/_docs/helpers/aws.md
+- docs/_docs/helpers/aws/iam-role.md
+- docs/_docs/helpers/aws/secrets.md
+- docs/_docs/helpers/aws/ssm.md
+- docs/_docs/helpers/google.md
+- docs/_docs/helpers/google/secrets.md
+- docs/_docs/helpers/google/service-account.md
 - docs/_docs/intro.md
 - docs/_docs/intro/concepts.md
 - docs/_docs/intro/how-kubes-works.md
@@ -309,6 +344,7 @@ files:
 - docs/_docs/patterns.md
 - docs/_docs/patterns/clock-web-worker.md
 - docs/_docs/patterns/migrations.md
+- docs/_docs/patterns/secrets.md
 - docs/_docs/resources.md
 - docs/_docs/resources/base.md
 - docs/_docs/resources/role.md
@@ -324,6 +360,7 @@ files:
 - docs/_includes/footer.html
 - docs/_includes/google_analytics.html
 - docs/_includes/header.html
+- docs/_includes/helpers/base64.md
 - docs/_includes/intro/install.md
 - docs/_includes/js.html
 - docs/_includes/kubes-steps.md
@@ -525,9 +562,11 @@ files:
 - lib/kubes/compiler/strategy/base.rb
 - lib/kubes/compiler/strategy/dsl.rb
 - lib/kubes/compiler/strategy/erb.rb
+- lib/kubes/compiler/strategy/erb/yaml_error.rb
 - lib/kubes/compiler/strategy/pass.rb
 - lib/kubes/compiler/strategy/result.rb
 - lib/kubes/compiler/util/normalize.rb
+- lib/kubes/compiler/util/save_file.rb
 - lib/kubes/compiler/util/yaml_dump.rb
 - lib/kubes/completer.rb
 - lib/kubes/completer/script.rb
@@ -588,6 +627,8 @@ files:
 - spec/fixtures/artifacts/demo-web/service.yaml
 - spec/fixtures/blocks/deployments.rb
 - spec/fixtures/decorators/deployment/both/envFrom.yaml
+- spec/fixtures/decorators/deployment/both/valueFrom.yaml
+- spec/fixtures/decorators/deployment/both/volumes.yaml
 - spec/fixtures/decorators/deployment/configMap/envFrom.yaml
 - spec/fixtures/decorators/deployment/configMap/valueFrom.yaml
 - spec/fixtures/decorators/deployment/configMap/volumes.yaml
@@ -661,6 +702,8 @@ test_files:
 - spec/fixtures/artifacts/demo-web/service.yaml
 - spec/fixtures/blocks/deployments.rb
 - spec/fixtures/decorators/deployment/both/envFrom.yaml
+- spec/fixtures/decorators/deployment/both/valueFrom.yaml
+- spec/fixtures/decorators/deployment/both/volumes.yaml
 - spec/fixtures/decorators/deployment/configMap/envFrom.yaml
 - spec/fixtures/decorators/deployment/configMap/valueFrom.yaml
 - spec/fixtures/decorators/deployment/configMap/volumes.yaml