kubes_google 0.1.2 → 0.3.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 64cbb6d5f91c9d194e47f9695b79802c5615367afe6464933b83bb59d9dfc418
-  data.tar.gz: 6d3d98f419a73dd8117dc87cfd0adf43e40e2e9feef447c8bbcfe5867b1410c9
+  metadata.gz: 36b12dca084f1a7011be11085f1e91b3e2551ae1814ddc33acec6172da1b5a66
+  data.tar.gz: 1fb93979efa46e903873fb18f9bad7affce32c73bc52bfe4c68c35b642033c60
 SHA512:
-  metadata.gz: 500425e494045b16a68672ad80c6d2157fc3a0dcd501c627850c6b3432dd95bec3dba727d31a7bed59d17be128366c4056bccb788d398659032b57afeb6b13e1
-  data.tar.gz: 679203c3c148448bfab87162c5267d1403e4f37c5f3dfd6e01bcf042dc7d4453e1919199037092651c807bb774b7a4afb6348f63603146463f552983c5e2c29e
+  metadata.gz: 7494108d3c48d710449494be1c4e3c772b13a5341becdaa76afd5cb02c3b390fd715bc6f48b46b62f0b3335731df6cf1e308a7e1851acba9c33544548a5e9252
+  data.tar.gz: 38fade8f5865c6355153b1ff18c4c37e28532d318b802503629b49553a0f5fe69e4e0d99fbbab53423a231b8ab5909d87c6ec55d91269086b9a1a76cf5c892b4

data/CHANGELOG.md

@@ -3,6 +3,22 @@
 All notable changes to this project will be documented in this file.
 This project *loosely tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [0.3.3] - 2020-11-12
+- [#6](https://github.com/boltops-tools/kubes_google/pull/6) sdk and gcloud secrets fetcher strategy: secrets.fetcher option
+
+## [0.3.2] - 2020-11-11
+- [#5](https://github.com/boltops-tools/kubes_google/pull/5) config.base64 option
+
+## [0.3.1] - 2020-11-11
+- [#4](https://github.com/boltops-tools/kubes_google/pull/4) get_credentials hook
+
+## [0.3.0]
+- #3 gke hook to whitelist ip
+
+## [0.2.0]
+- #2 add google_secret helper and register plugin
+- fix GOOGLE_PROJECT check
+
 ## [0.1.2]
 - #1 base64 option
 

data/README.md

@@ -8,119 +8,7 @@
 
 ## Usage
 
-The helpers include:
-
-* Secrets
-* Service Accounts
-
-## Secrets
-
-Set up a [Kubes hook](https://kubes.guru/docs/config/hooks/kubes/).
-
-.kubes/config/hooks/kubes.rb
-
-```ruby
-before("compile",
-  execute: KubesGoogle::Secrets.new(upcase: true, prefix: 'projects/686010496118/secrets/demo-dev-')
-)
-```
-
-Then set the secrets in the YAML:
-
-.kubes/resources/shared/secret.yaml
-
-```yaml
-apiVersion: v1
-kind: Secret
-metadata:
-  name: demo
-  labels:
-    app: demo
-data:
-<% KubesGoogle::Secrets.data.each do |k,v| -%>
-  <%= k %>: <%= base64(v) %>
-<% end -%>
-```
-
-This results in Google secrets with the prefix the `demo-dev-` being added to the Kubernetes secret data.  The values are automatically base64 encoded.
-
-For example if you have these secret values:
-
-    $ gcloud secrets versions access latest --secret demo-dev-db_user
-    test1
-    $ gcloud secrets versions access latest --secret demo-dev-db_pass
-    test2
-    $
-
-.kubes/output/shared/secret.yaml
-
-```yaml
-metadata:
-  namespace: demo
-  name: demo-2a78a13682
-  labels:
-    app: demo
-apiVersion: v1
-kind: Secret
-data:
-  db_pass: dGVzdDEK
-  db_user: dGVzdDIK
-```
-
-These environment variables can be set:
-
-Name | Description
----|---
-GCP_SECRET_PREFIX | Prefixed used to list and filter Google secrets. IE: `projects/686010496118/secrets/demo-dev-`.
-GOOGLE_PROJECT | Google project id.
-
-Secrets#initialize options:
-
-Variable | Description | Default
----|---|---
-base64 | Automatically base64 encode the values. | false
-upcase | Automatically upcase the Kubernetes secret data keys. | false
-prefix | Prefixed used to list and filter Google secrets. IE: `projects/686010496118/secrets/demo-dev-`. Can also be set with the `GCP_SECRET_PREFIX` env variable. The env variable takes the highest precedence. | nil
-
-Note, Kubernetes secrets are only base64 encoded. So users who have access to read Kubernetes secrets will be able to decode and get the value trivially. Depending on your security posture requirements, this may or may not suffice.
-
-## Service Accounts
-
-This library can also be used to automatically create Google Service Accounts associated with the [GKE Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).
-
-Here's a Kubes hook that creates a service account:
-
-.kubes/config/hooks/kubes.rb
-
-```ruby
-service_account = KubesGoogle::ServiceAccount.new(
-  app: "demo",
-  namespace: "demo-#{Kubes.env}", # defaults to APP-ENV when not set. IE: demo-dev
-  roles: ["cloudsql.client", "secretmanager.viewer"], # defaults to empty when not set
-)
-before("apply",
-  label: "create service account",
-  execute: service_account,
-)
-```
-
-The role permissions are currently always added to the existing permissions. So removing roles that were previously added does not remove them.
-
-ServiceAccount#initialize options:
-
-Variable | Description | Default
----|---|---
-app | The app name. It's used to conventionally set other variables. This is required. | nil
-gsa | The Google Service Account name. The conventional name is APP-ENV. IE: demo-dev. | APP-ENV
-ksa | The Kubernetes Service Account name. The conventional name is APP. IE: demo | APP
-namespace | The Kubernetes namespace. Defaults to the APP-ENV. IE: demo-dev. | APP-ENV
-roles | Google IAM roles to add. This adds permissions to the Google service account. | []
-
-Notes:
-
-* By default, `KubeGoogle.logger = Kubes.logger`. This means, you can set `logger.level = "debug"` in `.kubes/config.rb` to see more details.
-* The `gcloud` cli is used to create IAM roles. So `gcloud` is required.
-* Note: Would like to use the google sdk, but it wasn't obvious how to do so. PRs are welcomed.
+For more detailed usage instructions refer to the [Kubes Helpers docs](https://kubes.guru/docs/helpers/google/).
 
 ## Contributing
 

data/kubes_google.gemspec

@@ -23,7 +23,10 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.add_dependency "activesupport"
+  spec.add_dependency "google-cloud-container"
   spec.add_dependency "google-cloud-secret_manager"
   spec.add_dependency "memoist"
   spec.add_dependency "zeitwerk"
+
+  spec.add_development_dependency "kubes"
 end

data/lib/hooks/kubes.rb

@@ -0,0 +1,22 @@
+gke = KubesGoogle::Gke.new(
+  cluster_name: KubesGoogle.config.gke.cluster_name,
+  google_region: KubesGoogle.config.gke.google_region,
+  google_project: KubesGoogle.config.gke.google_project,
+  enable_get_credentials: KubesGoogle.config.gke.enable_get_credentials,
+  whitelist_ip: KubesGoogle.config.gke.whitelist_ip,
+)
+
+before("apply",
+  label: "gke get-credentials hook",
+  execute: gke.method(:get_credentials).to_proc,
+) if gke.get_credentials_enabled?
+
+before("apply",
+  label: "gke whitelist hook",
+  execute: gke.method(:allow).to_proc,
+) if gke.enabled?
+
+after("apply",
+  label: "gke whitelist hook",
+  execute: gke.method(:deny).to_proc,
+) if gke.enabled?

data/lib/kubes_google.rb

@@ -16,5 +16,26 @@ module KubesGoogle
     @@logger = v
   end
 
+  # Friendlier method configure.
+  #
+  #    .kubes/config/env/dev.rb
+  #    .kubes/config/plugins/google.rb # also works
+  #
+  # Example:
+  #
+  #     KubesGoogle.configure do |config|
+  #       config.hooks.gke_whitelist = true
+  #     end
+  #
+  def configure(&block)
+    Config.instance.configure(&block)
+  end
+
+  def config
+    Config.instance.config
+  end
+
   extend self
 end
+
+Kubes::Plugin.register(KubesGoogle)

data/lib/kubes_google/config.rb

@@ -0,0 +1,29 @@
+module KubesGoogle
+  class Config
+    include Singleton
+
+    def defaults
+      c = ActiveSupport::OrderedOptions.new
+      c.gke = ActiveSupport::OrderedOptions.new
+      c.gke.cluster_name = nil
+      c.gke.enable_get_credentials = nil
+      c.gke.enable_hooks = nil # nil since need cluster_name also. setting to false will explicitly disable hooks
+      c.gke.google_project = nil
+      c.gke.google_region = nil
+      c.gke.whitelist_ip = nil # default will auto-detect IP
+      c.secrets = ActiveSupport::OrderedOptions.new
+      c.secrets.fetcher = "sdk"
+      c.secrets.base64 = true
+      c
+    end
+
+    @@config = nil
+    def config
+      @@config ||= defaults
+    end
+
+    def configure
+      yield(config)
+    end
+  end
+end

data/lib/kubes_google/gke.rb

@@ -0,0 +1,120 @@
+require 'open-uri'
+
+module KubesGoogle
+  class Gke
+    extend Memoist
+    include Logging
+    include Services
+    include Util::Sh
+
+    def initialize(cluster_name:,
+                   enable_get_credentials: false,
+                   google_project: nil,
+                   google_region: "us-central1",
+                   whitelist_ip: nil)
+      @cluster_name = cluster_name
+      @enable_get_credentials = enable_get_credentials
+      @google_project = ENV['GOOGLE_PROJECT'] || google_project
+      @google_region = ENV['GOOGLE_REGION'] || google_region
+      @whitelist_ip = whitelist_ip
+    end
+
+    def allow
+      logger.debug "Updating cluster. Adding IP: #{ip}"
+      update_cluster(cidr_blocks(:with_whitelist))
+    end
+
+    def deny
+      logger.debug "Updating cluster. Removing IP: #{ip}"
+      update_cluster(cidr_blocks(:without_whitelist))
+    end
+
+    def get_credentials
+      return unless get_credentials_enabled?
+      sh "gcloud container clusters get-credentials --project=#{@google_project} --region=#{@google_region} #{@cluster_name}"
+    end
+
+    def full_name
+      "projects/#{@google_project}/locations/#{@google_region}/clusters/#{@cluster_name}"
+    end
+
+    def enabled?
+      enable = KubesGoogle.config.gke.enable_hooks
+      enable = enable.nil? ? true : enable
+      # gke = KubesGoogle::Gke.new(name: KubesGoogle.config.gke.cluster_name)
+      # so @name = KubesGoogle.config.gke.cluster_name
+      !!(enable && @cluster_name)
+    end
+
+    def get_credentials_enabled?
+      enable = KubesGoogle.config.gke.enable_get_credentials
+      enable = enable.nil? ? false : enable
+      !!(enable && full_name)
+    end
+
+    def update_cluster(cidr_blocks)
+      resp = cluster_manager.update_cluster(
+        name: full_name,
+        update: {
+          desired_master_authorized_networks_config: {
+            cidr_blocks: cidr_blocks,
+            enabled: true,
+          }
+        }
+      )
+      operation_name = resp.self_link.sub(/.*projects/,'projects')
+      wait_for(operation_name)
+    end
+
+    def wait_for(operation_name)
+      resp = cluster_manager.get_operation(name: operation_name)
+      until resp.status != :RUNNING do
+        sleep 5
+        resp = cluster_manager.get_operation(name: operation_name)
+      end
+    end
+
+    def cidr_blocks(type)
+      # so we dont keep adding duplicates
+      old = old_cidrs.reject do |x|
+        x[:display_name] == new_cidr[:display_name] &&
+        x[:cidr_block] == new_cidr[:cidr_block]
+      end
+      if type == :with_whitelist
+        old + [new_cidr]
+      else
+        old
+      end
+    end
+
+    def old_cidrs
+      resp = cluster_manager.get_cluster(name: full_name)
+      config = resp.master_authorized_networks_config.to_h
+      config[:cidr_blocks]
+    end
+    memoize :old_cidrs
+
+    def new_cidr
+      {
+        display_name: "added-by-kubes-google",
+        cidr_block: ip,
+      }
+    end
+    memoize :new_cidr
+
+    def ip
+      @whitelist_ip || current_ip
+    end
+
+    def current_ip
+      resp = URI.open("http://ifconfig.me")
+      ip = resp.read
+      "#{ip}/32"
+    rescue SocketError => e
+      logger.info "WARN: #{e.message}"
+      logger.info "Unable to detect current ip. Will use 0.0.0.0/0"
+      "0.0.0.0/0"
+    end
+    memoize :current_ip
+  end
+end

data/lib/kubes_google/helpers.rb

@@ -0,0 +1,11 @@
+module KubesGoogle
+  module Helpers
+    extend Memoist
+    include Services
+
+    def google_secret(name, options={})
+      fetcher = Secrets::Fetcher.new(options)
+      fetcher.fetch(name)
+    end
+  end
+end

data/lib/kubes_google/hooks.rb

@@ -0,0 +1,7 @@
+module KubesGoogle
+  class Hooks
+    def path
+      File.expand_path("../hooks", __dir__)
+    end
+  end
+end

data/lib/kubes_google/secrets.rb

@@ -4,8 +4,8 @@ module KubesGoogle
   class Secrets
     def initialize(upcase: false, base64: false, prefix: nil)
       @upcase, @base64 = upcase, base64
-      @prefix = ENV['GCP_SECRET_PREFIX'] || prefix || raise("GOOGLE_PROJECT env variable is not set. It's required.")
-      @project_id = ENV['GOOGLE_PROJECT']
+      @prefix = ENV['GCP_SECRET_PREFIX'] || prefix
+      @project_id = ENV['GOOGLE_PROJECT'] || raise("GOOGLE_PROJECT env variable is not set. It's required.")
       # IE: prefix: projects/686010496118/secrets/demo-dev-
     end
 
@@ -13,7 +13,7 @@ module KubesGoogle
       client = Google::Cloud::SecretManager.secret_manager_service
 
       parent = "projects/#{@project_id}"
-      resp = client.list_secrets(parent: parent, page_size: 1)
+      resp = client.list_secrets(parent: parent) # note: page_size doesnt seem to get respected
       resp.each do |secret|
         next unless secret.name.include?(@prefix)
         version = client.access_secret_version(name: "#{secret.name}/versions/latest")

data/lib/kubes_google/secrets/fetcher.rb

@@ -0,0 +1,22 @@
+class KubesGoogle::Secrets
+  class Fetcher
+    extend Memoist
+
+    def initialize(options={})
+      @options = options
+    end
+
+    def fetch(short_name)
+      fetcher.fetch(short_name)
+    end
+
+    def fetcher
+      if Kubes.config.secrets_fetcher == "sdk"
+        Sdk.new(@options)
+      else
+        Gcloud.new(@options)
+      end
+    end
+    memoize :fetcher
+  end
+end

data/lib/kubes_google/secrets/fetcher/base.rb

@@ -0,0 +1,15 @@
+class KubesGoogle::Secrets::Fetcher
+  class Base
+    include KubesGoogle::Logging
+
+    def initialize(options={})
+      @options = options
+      @base64 = options[:base64]
+      @project_id = options[:google_project] || ENV['GOOGLE_PROJECT'] || raise("GOOGLE_PROJECT env variable is not set. It's required.")
+    end
+
+    def base64?
+      @base64.nil? ? KubesGoogle.config.secrets.base64 : @base64
+    end
+  end
+end

data/lib/kubes_google/secrets/fetcher/gcloud.rb

@@ -0,0 +1,22 @@
+class KubesGoogle::Secrets::Fetcher
+  class Gcloud < Base
+    include KubesGoogle::Util::Sh
+
+    def fetch(short_name, version="latest")
+      puts "gcloud fetch #{short_name}"
+      value = gcloud("secrets versions access #{version} --secret #{short_name}")
+      if value.include?("ERROR") && value.include?("NOT_FOUND")
+        logger.info "WARN: secret #{short_name} not found".color(:yellow)
+        logger.info e.message
+        "NOT FOUND #{short_name}" # simple string so Kubernetes YAML is valid
+      else
+        value = Base64.strict_encode64(value).strip if base64?
+        value
+      end
+    end
+
+    def gcloud(args)
+      capture("gcloud --project #{@project_id} #{args}")
+    end
+  end
+end

data/lib/kubes_google/secrets/fetcher/sdk.rb

@@ -0,0 +1,34 @@
+class KubesGoogle::Secrets::Fetcher
+  class Sdk < Base
+    include KubesGoogle::Services
+
+    def fetch(short_name, version="latest")
+      value = fetch_value(short_name, version)
+      value = Base64.strict_encode64(value).strip if base64?
+      value
+    end
+
+    def fetch_value(short_name, version="latest")
+      name = "projects/#{project_number}/secrets/#{short_name}/versions/#{version}"
+      version = secret_manager_service.access_secret_version(name: name)
+      version.payload.data
+    rescue Google::Cloud::NotFoundError => e
+      logger.info "WARN: secret #{name} not found".color(:yellow)
+      logger.info e.message
+      "NOT FOUND #{name}" # simple string so Kubernetes YAML is valid
+    end
+
+    # TODO: Get the project from the list project api instead. Unsure where the docs are for this.
+    # If someone knows, let me know.
+    # Right now grabbing the first secret to then be able to get the google project number
+    @@project_number = nil
+    def project_number
+      return @@project_number if @@project_number
+
+      parent = "projects/#{@project_id}"
+      resp = secret_manager_service.list_secrets(parent: parent) # note: page_size doesnt seem to get respected
+      name = resp.first.name # IE: projects/686010496118/secrets/demo-dev-db_host
+      @@project_number = name.split('/')[1]
+    end
+  end
+end

data/lib/kubes_google/service_account.rb

@@ -4,6 +4,7 @@ require "json"
 module KubesGoogle
   class ServiceAccount
     include Logging
+    include Util::Sh
 
     def initialize(app:, namespace:nil, roles: [], gsa: nil, ksa: nil)
       @app, @roles = app, roles
@@ -71,25 +72,5 @@ module KubesGoogle
           --member=serviceAccount:#{@service_account} \
           --role=#{role} > /dev/null".squish
     end
-
-  private
-    def sh(command)
-      logger.debug "=> #{command}"
-      success = system(command)
-      unless success
-        logger.info "WARN: Running #{command}"
-      end
-      success
-    end
-
-    def capture(command)
-      out = `#{command}`
-      unless $?.exitstatus == 0
-        logger.info "ERROR: Running #{command}"
-        logger.info out
-        exit 1
-      end
-      out
-    end
   end
 end

data/lib/kubes_google/services.rb

@@ -0,0 +1,19 @@
+require "google-cloud-secret_manager"
+require "google/cloud/container"
+
+module KubesGoogle
+  module Services
+    extend Memoist
+
+    def cluster_manager
+      Google::Cloud::Container.cluster_manager
+    end
+    memoize :cluster_manager
+
+    def secret_manager_service
+      Google::Cloud::SecretManager.secret_manager_service
+    end
+    memoize :secret_manager_service
+  end
+end
+

data/lib/kubes_google/util/sh.rb

@@ -0,0 +1,23 @@
+module KubesGoogle::Util
+  module Sh
+  private
+    def sh(command)
+      logger.debug "=> #{command}"
+      success = system(command)
+      unless success
+        logger.info "WARN: Running #{command}"
+      end
+      success
+    end
+
+    def capture(command)
+      out = `#{command}`
+      unless $?.exitstatus == 0
+        logger.info "ERROR: Running #{command}"
+        logger.info out
+        exit 1
+      end
+      out
+    end
+  end
+end

data/lib/kubes_google/version.rb

@@ -1,3 +1,3 @@
 module KubesGoogle
-  VERSION = "0.1.2"
+  VERSION = "0.3.3"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kubes_google
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.3.3
 platform: ruby
 authors:
 - Tung Nguyen
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-10-14 00:00:00.000000000 Z
+date: 2020-11-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -24,6 +24,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: google-cloud-container
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: google-cloud-secret_manager
   requirement: !ruby/object:Gem::Requirement
@@ -66,6 +80,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: kubes
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: 
 email:
 - tung@boltops.com
@@ -81,11 +109,22 @@ files:
 - README.md
 - Rakefile
 - kubes_google.gemspec
+- lib/hooks/kubes.rb
 - lib/kubes_google.rb
 - lib/kubes_google/autoloader.rb
+- lib/kubes_google/config.rb
+- lib/kubes_google/gke.rb
+- lib/kubes_google/helpers.rb
+- lib/kubes_google/hooks.rb
 - lib/kubes_google/logging.rb
 - lib/kubes_google/secrets.rb
+- lib/kubes_google/secrets/fetcher.rb
+- lib/kubes_google/secrets/fetcher/base.rb
+- lib/kubes_google/secrets/fetcher/gcloud.rb
+- lib/kubes_google/secrets/fetcher/sdk.rb
 - lib/kubes_google/service_account.rb
+- lib/kubes_google/services.rb
+- lib/kubes_google/util/sh.rb
 - lib/kubes_google/version.rb
 homepage: https://github.com/boltops-tools/kubes_google
 licenses:
@@ -107,7 +146,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.2
+rubygems_version: 3.1.4
 signing_key: 
 specification_version: 4
 summary: Kubes Google Helpers Library