kubes 0.3.5 → 0.4.4

data/docs/_docs/config/kubectl.md

@@ -6,5 +6,5 @@ title: Kubectl Customizations
 
 Kubes calls out the `kubectl` command. You can customize the command.
 
-* [Args]({% link _docs/config/kubectl/args.md %}): Customize the CLI args.
-* [Hooks]({% link _docs/config/kubectl/hooks.md %}): Run hooks before and after the kubectl commands.
+* [Args]({% link _docs/config/args/kubectl.md %}): Customize the CLI args.
+* [Hooks]({% link _docs/config/hooks/kubectl.md %}): Run hooks before and after the kubectl commands.

data/docs/_docs/config/reference.md

@@ -0,0 +1,20 @@
+---
+title: Config Reference
+---
+
+Name | Description | Default
+---|---|---
+auto_prune | Prune and delete old hashed resources like Secret and ConfigMap. | true
+builder | What docker build command to use. Can use `docker` or `gcloud` to build the Docker image. | docker
+kubectl.context | What kubectl context to auto-switch to. | nil
+kubectl.context_keep | Whether or not to keep the context switched | true
+kubectl.exit_on_fail.apply  | Whether or not continue if the `kubectl apply` fails. Note, can use `KUBES_EXIT_ON_FAIL=0` env var to set to false. | true
+kubectl.exit_on_fail.delete | Whether or not continue if the `kubectl delete` fails. | false
+kubectl.order.kinds | Change ordering for Kubernetes Kinds. | See [source code](https://github.com/boltops-tools/kubes/blob/master/lib/kubes/config.rb#L52)
+kubectl.order.roles | Change ordering for Kubes Roles. | See [source code](https://github.com/boltops-tools/kubes/blob/master/lib/kubes/config.rb#L44)
+logger | Logger object | Logger.new($stdout)
+logger.level | Logger level. Can also be set with `KUBES_LOG_LEVEL` env var | info
+repo | The Docker repo to use. Required to be set. | nil
+skip | List of resources to skip. Can also be set with the `KUBES_SKIP` env var. `KUBES_SKIP` should be a list of strings separated by spaces. It adds onto the `config.skip` option. | []
+state.docker_image_path | Where to store the state file with the last build Docker image. | .kubes/state/docker_image.txt
+suffix_hash | Whether or not to append suffix hash to ConfigMap and Secret | true

data/docs/_docs/config/skip.md

@@ -0,0 +1,58 @@
+---
+title: Skip Option
+---
+
+You can tell Kubes to skip resources to deploy. This can useful if you want to still resources with Kubes and have it compile `.kubes/output` files, but wish to deploy them outside of Kubes manually.
+
+## Example
+
+Here's an example with a Job.
+
+.kubes/resources/cleanup/job.yaml:
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: cleanup
+spec:
+  template:
+    spec:
+      containers:
+      - name: cleanup
+        image: <%= built_image %>
+        command: ["bin/cleanup.sh"]
+      restartPolicy: Never
+```
+
+To skip the cleanup job, use the `config.skip` option:
+
+```ruby
+Kubes.configure do |config|
+  config.skip = ["cleanup/job"]
+end
+```
+
+Now when you deploy, the `cleanup/job` resource will not be deployed:
+
+    kubes deploy # deploys everything except cleanup/job
+
+## Deploy Outside of Kubes
+
+Then to deploy outside of kubes.
+
+    $ kubes compile # not necessary if already ran: kubes deploy
+    Compiled  .kubes/resources files to .kubes/output
+    $ kubectl apply -f .kubes/output/cleanup/job.yaml
+    job.batch/cleanup created
+    $ kubectl delete -f .kubes/output/cleanup/job.yaml
+    job.batch "cleanup" deleted
+    $
+
+## Env Var KUBES_SKIP
+
+You can also us ethe `KUBES_SKIP` env var. It takes list of strings separated by a space. It adds onto the `config.skip` option. Example:
+
+    KUBES_SKIP="cleanup/job" kubes delete
+
+This can be useful for one-off use cases.

data/docs/_docs/dsl/resources.md

@@ -6,7 +6,7 @@ Here's a list of the resources supported by the Kubes DSL.
 
 {% assign docs = site.docs | where: "categories","dsl" %}
 {% for doc in docs -%}
-* [{{ doc.title }}]({{ doc.url }})
+* [{{ doc.nav_text }}]({{ doc.url }})
 {% endfor %}
 
 For resources, that are not supported, you can use the [Generic resource]({% link _docs/dsl/resources/generic.md %}) or use [YAML]({% link _docs/yaml.md %}) instead. You can use a mix of DSL and YAML definitions in the `.kubes/resources` folder.

data/docs/_docs/helpers.md

@@ -7,12 +7,13 @@ Kubes provides some helper methods to help write Kubernetes YAML files.  Here's
 Helper | Description
 --- | ---
 built_image | Method refers to the latest Docker image built by Kubes. This spares you from having to update the image manually in the deployment resource.
+decode64 | Basey64d decode a string.
 dockerfile_port	| Exposed port extracted from the Dockerfile of the project.
+encode64 | Basey64 encode a string. Also available as `base64` method.
 extra | The `KUBES_EXTRA` value.
 with_extra | Appends the `KUBES_EXTRA` value to a string if it's set. It's covered in the [Extra Env Docs]({% link _docs/extra-env.md %}).
 
-Here's also the source code with the helpers: [helpers.rb](https://github.com/boltops-tools/kubes/blob/master/lib/kubes/compiler/shared/helpers.rb).
-
+Here's also the source code with most of the helpers: [helpers.rb](https://github.com/boltops-tools/kubes/blob/master/lib/kubes/compiler/shared/helpers.rb).
 
 ## DSL Specific Methods
 

data/docs/_docs/intro.md

@@ -4,12 +4,14 @@ title: What is Kubes?
 
 {% include reference.md %}
 
+<div class="video-box"><div class="video-container"><iframe src="https://www.youtube.com/embed/M4zHL0mfKNU" frameborder="0" allowfullscreen=""></iframe></div></div>
+
 ## Features:
 
 * Automation: [Builds the Docker image]({% link _docs/config/docker.md %}) and updates the compiled YAML files
 * Syntactic Sugar: Use an [ERB/YAML]({% link _docs/yaml.md %}) or a [DSL]({% link _docs/dsl.md %}) to write your Kubernetes YAML files. You can use a mix of DSL and YAML definitions in the `.kubes/resources` folder.
 * Layering: Use the same Kubernetes YAML to build multiple environments like dev and prod with [layering]({% link _docs/layering.md %}).
-* CLI Customizations: You can customize the [cli args]({% link _docs/config/kubectl/args.md %}). You can also run [hooks]({% link _docs/config/kubectl/hooks.md %}) before and after kubectl commands.
+* CLI Customizations: You can customize the [cli args]({% link _docs/config/args/kubectl.md %}). You can also run [hooks]({% link _docs/config/hooks/kubectl.md %}) before and after kubectl commands.
 * Automated Suffix Hashes: Automatically appends a suffix hash to ConfigMap and Secret resources. More details in [ConfigMap]({% link _docs/dsl/resources/config_map.md %}) and [Secret]({% link _docs/dsl/resources/secret.md %}) docs.
 * Kustomize Support: If you’re a kustomization user, you can use it with Kubes. More details in [Kustomize Support Docs]({% link _docs/misc/kustomize.md %}).
 * Auto Context Switching: Map dev to a specific kubectl context and prod to another kubectl context and Kubes can switch them automatically so you won't have to remember. More details in [Auto Context Docs]({% link _docs/misc/auto-context.md %}).

data/docs/_docs/learn/dsl/review-project.md

@@ -11,10 +11,12 @@ We'll create a namespace for the app resources:
 .kubes/resources/shared/namespace.rb
 
 ```ruby
-name "demo"
+name "demo-#{Kubes.env}"
 labels(app: "demo")
 ```
 
+Notice, the `#{Kubes.env}`. Kubes adds the env to the namespace by default. You can change this with the `init --namespace` option.
+
 ## Deployment
 
 The `web/deployment.rb` file is a little more interesting:
@@ -39,7 +41,7 @@ Also let's check the files in the base folder.
 .kubes/resources/base/all.rb
 
 ```ruby
-namespace "default"
+namespace "demo-#{Kubes.env}"
 labels(app: "demo")
 ```
 

data/docs/_docs/learn/yaml/review-project.md

@@ -14,11 +14,13 @@ We'll create a namespace for the app resources:
 apiVersion: v1
 kind: Namespace
 metadata:
-  name: demo
+  name: demo-<%= Kubes.env %>
   labels:
     app: demo
 ```
 
+Notice, the `<%= Kubes.env %>`. Kubes adds the env to the namespace by default. You can change this with the `init --namespace` option.
+
 ## Deployment
 
 The `web/deployment.yaml` file is a little more interesting:
@@ -57,7 +59,7 @@ Also let's check the files in the base folder.
 
 ```yaml
 metadata:
-  namespace: demo
+  namespace: demo-<%= Kubes.env %>
 ```
 
 .kubes/resources/base/deployment.yaml

data/docs/_docs/patterns.md

@@ -4,4 +4,7 @@ title: Patterns
 
 We'll cover some common deployment patterns here:
 
-* [Clock Web Worker]({% link _docs/patterns/clock-web-worker.md %})
+{% assign docs = site.docs | where: "categories","patterns" %}
+{% for doc in docs -%}
+* [{{ doc.title }}]({{ doc.url }})
+{% endfor %}

data/docs/_docs/patterns/clock-web-worker.md

@@ -1,5 +1,7 @@
 ---
 title: Clock Web Worker Pattern
+nav_text: Clock Web Worker
+categories: patterns
 ---
 
 A common pattern is to use the same code to run different types of processes like clock, web, worker. Kubes is flexible enough to support this pattern.

data/docs/_docs/patterns/migrations.md

@@ -0,0 +1,123 @@
+---
+title: Database Migrations
+nav_text: Database Migrations
+categories: patterns
+---
+
+A common task is to run database migrations. You can use Kubes hooks to achieve this as part of the `kubes deploy` process.
+
+1. Create Migrate Job YAML
+2. Set up Kubes Hooks
+
+## 1. Create Migrate Job YAML
+
+First, let's create the migrate job YAML. Here's a starter example:
+
+.kubes/resources/migrate/job.yaml
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: migrate
+spec:
+  template:
+    spec:
+      containers:
+      - name: migrate
+        image: <%= built_image %>
+        command: ["bin/job/migrate.sh"]
+      restartPolicy: Never
+  backoffLimit: 4
+```
+
+The Kubernetes job calls a `job/migrate.sh` script. Something like this:
+
+bin/job/migrate.sh
+
+    #!/bin/bash
+    rails db:migrate
+
+## 2. Set up Kubes Hooks
+
+Set up the [kubes hooks]({% link _docs/config/hooks/kubectl.md %}) to help the migrate job run properly.
+
+.kubes/config/hooks/kubectl.rb
+
+```ruby
+before("apply",
+  on: "migrate/job",
+  execute: "bin/hooks/migrate/delete.sh",
+  exit_on_fail: false,
+)
+
+after("apply",
+  on: "migrate/job",
+  execute: "bin/hooks/migrate/wait.sh",
+)
+```
+
+Here's what the `bin/hook/migrate` scripts could look like:
+
+bin/hooks/migrate/delete.sh
+
+    #!/bin/bash
+    kubectl delete job/migrate
+
+bin/hooks/migrate/wait.sh
+
+    #!/bin/bash
+    kubectl wait --for=condition=Complete job/migrate --timeout=300s
+
+The `migrate/delete.sh` script first cleans up old migrate jobs that may have been previously created.
+
+The `migrate/wait.sh` script waits until the migration job finishes before continuing. Note, the default timeout is 30s, which may not be long enough for your migrations to finish, so we set it to 300s.  The `kubectl wait` only returns if the migrate job finishes successfully. If the job fails after it exhausts all its retries, default 6, then you'll see an error like this:
+
+    + kubectl wait --for=condition=Complete job/migrate --timeout=30s
+    error: timed out waiting for the condition on jobs/migrate
+    ERROR: running bin/hooks/migrate.sh
+
+There is also an [migration-example](https://github.com/boltops-tools/kubes-examples/tree/master/yaml/migration-example) repo with a smarter version of the wait script.
+
+## Example Deploy
+
+Once that is set up, a `kubes deploy` will automatically run migrations. Here's an example deploy:
+
+    $ kubes deploy
+    => kubectl apply -f .kubes/output/shared/namespace.yaml
+    => bin/hooks/migrate/delete.sh
+    job.batch "migrate" deleted
+    => kubectl apply -f .kubes/output/migrate/job.yaml
+    job.batch/migrate created
+    Running hook: after apply on: migrate/job
+    => bin/hooks/migrate/wait.sh
+    Sun Oct 11 03:22:35 UTC 2020
+    Migration complete
+    => kubectl apply -f .kubes/output/web/service.yaml
+    service/web unchanged
+    => kubectl apply -f .kubes/output/web/deployment.yaml
+    deployment.apps/web configured
+    $
+
+## To Couple or Not to Couple?
+
+While some companies prefer running the migration step as a part of the app deploy, some prefer to separate it out as a discrete step. Usually, the separate step is still called as part of a pipeline.
+
+In practice, the decision usually comes down to:
+
+* The size of your database. If your database is large and the migrations take a long time to run. It makes sense to separate it out.
+* The risk tolerance of database migration operations. If it's quite risky to run DB migrations, you may want to separate it as discrete step so a human can review it.
+
+For small apps and databases, it's often pragmatic to just run everything in a single step for simplicity.
+
+## Migration as Separate Step
+
+If you would like it to run it as a discrete step, remove the hook in `.kubes/config/hooks/kubectl.rb`, and run it as a separate script like so:
+
+bin/run/migrate.sh
+
+    #!/bin/bash
+    kubes compile
+    bin/hooks/migrate/delete.sh
+    bin/job/migrate.sh
+    bin/hooks/migrate/wait.sh

data/docs/_docs/patterns/secrets.md

@@ -0,0 +1,82 @@
+---
+title: Secrets
+nav_text: Secrets
+categories: patterns
+---
+
+A Google Secrets helper is currently supported.
+
+## Set Up Kubes Hook
+
+Set up a [Kubes hook]({% link _docs/config/hooks/kubes.md %}).
+
+.kubes/config/hooks/kubes.rb
+
+```ruby
+before("compile",
+  execute: KubesGoogle::Secrets.new(upcase: true, prefix: 'projects/686010496118/secrets/demo-dev-')
+)
+```
+
+Then set the secrets in the YAML:
+
+.kubes/resources/shared/secret.yaml
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: demo
+  labels:
+    app: demo
+data:
+<% KubesGoogle::Secrets.data.each do |k,v| -%>
+  <%= k %>: <%= base64(v) %>
+<% end -%>
+```
+
+This results in Google secrets with the prefix the `demo-dev-` being added to the Kubernetes secret data.  The values are base64 encoded.
+
+For example if you have these secret values:
+
+    $ gcloud secrets versions access latest --secret demo-dev-db_user
+    test1
+    $ gcloud secrets versions access latest --secret demo-dev-db_pass
+    test2
+    $
+
+The compiled secrets.yaml looks like this:
+
+.kubes/output/shared/secret.yaml
+
+```yaml
+metadata:
+  namespace: demo
+  name: demo-2a78a13682
+  labels:
+    app: demo
+apiVersion: v1
+kind: Secret
+data:
+  db_pass: dGVzdDEK
+  db_user: dGVzdDIK
+```
+
+These environment variables can be set:
+
+Name | Description
+---|---
+GCP_SECRET_PREFIX | Prefixed used to list and filter Google secrets. IE: `projects/686010496118/secrets/demo-dev-`.
+GOOGLE_PROJECT | Google project id.
+
+Secrets#initialize options:
+
+Variable | Description | Default
+---|---|---
+base64 | Automatically base64 encode the values. | false
+upcase | Automatically upcase the Kubernetes secret data keys. | false
+prefix | Prefixed used to list and filter Google secrets. IE: `projects/686010496118/secrets/demo-dev-`. Can also be set with the `GCP_SECRET_PREFIX` env variable. The env variable takes the highest precedence. | nil
+
+Note, Kubernetes secrets are only base64 encoded. So users who have access to read Kubernetes secrets will be able to decode and get the value trivially. Depending on your security posture requirements, this may or may not suffice.
+
+The Google helpers are provided by the [boltops-tools/kubes_google](https://github.com/boltops-tools/kubes_google) library. For more details, check out its README.

data/docs/_includes/config/hooks/options.md

@@ -0,0 +1,20 @@
+## General Form
+
+```ruby
+before(COMMAND_NAME, OPTIONS)
+````
+
+The command name corresponds to the `{{ include.command }}` commands: apply, delete, etc.
+
+## Hook Options
+
+Name | Description
+---|---
+label | A human-friendly label so you can see what hooks is being run.
+execute | The script or command to run. IE: path/to/some/script.sh
+exit_on_fail | Whether or not to continue process if the script returns an failed exit code.
+{% if include.command == "kubectl" %}on | What resource to run the hook on. IE: shared/namespace, web/deployment, web/service. Note: This option is only used by kubectl hooks.{% endif %}
+
+## Ruby Hooks
+
+Instead of using a script for the hook `execute` option, you can also use a Ruby object. This provides some more control over the current process. See: [Ruby Hooks]({% link _docs/config/hooks/ruby.md %})

data/docs/_includes/layering/layers.md

@@ -27,7 +27,7 @@ To explain the layering, here's the general processing order that Kubes takes.
 
 Note, both YAML and DSL forms support layering.
 
-Layering only combines resources definitions with the same form. For example, `base/all.rb` will not be combined with `web/deployment.yaml`.
+Layering only combines resources definitions with the same form. For example, the DSL form `base/all.rb` will not be combined with YAML form `web/deployment.yaml`.
 
 ## Full Layering
 

data/docs/_includes/sidebar.html

@@ -53,18 +53,37 @@
       </li>
       <li><a href="{% link _docs/config.md %}">Config</a>
         <ul>
-          <li><a href="{% link _docs/config/docker.md %}">Docker</a></li>
-          <li><a href="{% link _docs/config/env.md %}">Env</a></li>
-          <li><a href="{% link _docs/config/kubectl.md %}">Kubectl</a>
+          <li><a href="{% link _docs/config/args.md %}">Args</a>
             <ul>
-              <li><a href="{% link _docs/config/kubectl/args.md %}">Args</a></li>
-              <li><a href="{% link _docs/config/kubectl/hooks.md %}">Hooks</a></li>
+            {% assign docs = site.docs | where: "categories","args" %}
+            {% for doc in docs -%}
+              <li><a href="{{ doc.url }}">{{ doc.nav_text }}</a></li>
+            {% endfor %}
             </ul>
           </li>
+          <li><a href="{% link _docs/config/hooks.md %}">Hooks</a>
+            <ul>
+            {% assign docs = site.docs | where: "categories","hooks" %}
+            {% for doc in docs -%}
+              <li><a href="{{ doc.url }}">{{ doc.nav_text }}</a></li>
+            {% endfor %}
+            </ul>
+          </li>
+          <li><a href="{% link _docs/config/docker.md %}">Docker</a></li>
+          <li><a href="{% link _docs/config/env.md %}">Env</a></li>
           <li><a href="{% link _docs/config/builder.md %}">Builder</a></li>
+          <li><a href="{% link _docs/config/skip.md %}">Skip Option</a></li>
+          <li><a href="{% link _docs/config/reference.md %}">Reference</a></li>
         </ul>
       </li>
       <li><a href="{% link _docs/yaml.md %}">YAML</a></li>
+      <li><a href="{% link _docs/layering.md %}">Layering</a>
+        <ul>
+          <li><a href="{% link _docs/layering/yaml.md %}">YAML</a></li>
+          <li><a href="{% link _docs/layering/dsl.md %}">DSL</a></li>
+          <li><a href="{% link _docs/layering/merge.md %}">Merge Behavior</a></li>
+        </ul>
+      </li>
       <li><a href="{% link _docs/dsl.md %}">DSL</a>
         <ul>
           <li><a href="{% link _docs/dsl/resources.md %}">Resources</a>
@@ -78,17 +97,13 @@
           <li><a href="{% link _docs/dsl/multiple-resources.md %}">Multiple Resources</a>
         </ul>
       </li>
-      <li><a href="{% link _docs/layering.md %}">Layering</a>
-        <ul>
-          <li><a href="{% link _docs/layering/yaml.md %}">YAML</a></li>
-          <li><a href="{% link _docs/layering/dsl.md %}">DSL</a></li>
-          <li><a href="{% link _docs/layering/merge.md %}">Merge Behavior</a></li>
-        </ul>
-      </li>
       <li><a href="{% link _docs/helpers.md %}">Helpers</a></li>
       <li><a href="{% link _docs/patterns.md %}">Patterns</a>
         <ul>
-          <li><a href="{% link _docs/patterns/clock-web-worker.md %}">Clock Web Worker</a></li>
+          {% assign docs = site.docs | where: "categories","patterns" %}
+          {% for doc in docs -%}
+            <li><a href="{{ doc.url }}">{{ doc.nav_text }}</a></li>
+          {% endfor %}
         </ul>
       </li>
       <li><a href="{% link _docs/extra-env.md %}">Extra Env</a>