hetzner-k3s 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 600b092e02f2bca4fd0be7e830a13913d2bbf82d2e3b98226ab52a2b5df4e859
+  data.tar.gz: f595dda56d1ca9aeaa611a77d82c168a63d300a83c5f3e8edc44b3da5790d46a
+SHA512:
+  metadata.gz: 6024a0b99ecc6d97d56e50f39e9e89f9cd2d92db7261604162ec37628df6bd4b942f50be64c4f05d9a745e55a3e18e567a759f20626671e81746c4160a280a9a
+  data.tar.gz: 17f2095befd0035555adc902dc52baed71f77bd82ef26ecb9bff3df5e7b5ec14068cdd16923fd6a2297e22badaab922b3fbc745939285e41fed8699922e26017

data/.gitignore

@@ -0,0 +1,13 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+
+# rspec failure tracking
+.rspec_status
+/kubeconfig
+/cluster_config.yaml

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,6 @@
+---
+language: ruby
+cache: bundler
+rvm:
+  - 2.7.2
+before_install: gem install bundler -v 2.1.4

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at vito@botta.me. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [https://contributor-covenant.org/version/1/4][version]
+
+[homepage]: https://contributor-covenant.org
+[version]: https://contributor-covenant.org/version/1/4/

data/Gemfile

@@ -0,0 +1,7 @@
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in k3s.gemspec
+gemspec
+
+gem "rake", "~> 12.0"
+gem "rspec", "~> 3.0"

data/Gemfile.lock

@@ -0,0 +1,111 @@
+PATH
+  remote: .
+  specs:
+    hetzner-k3s (0.1.0)
+      http
+      k8s-ruby
+      net-ssh
+      sshkey
+      thor
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    addressable (2.8.0)
+      public_suffix (>= 2.0.2, < 5.0)
+    concurrent-ruby (1.1.9)
+    diff-lcs (1.4.4)
+    domain_name (0.5.20190701)
+      unf (>= 0.0.5, < 1.0.0)
+    dry-configurable (0.12.1)
+      concurrent-ruby (~> 1.0)
+      dry-core (~> 0.5, >= 0.5.0)
+    dry-container (0.8.0)
+      concurrent-ruby (~> 1.0)
+      dry-configurable (~> 0.1, >= 0.1.3)
+    dry-core (0.7.1)
+      concurrent-ruby (~> 1.0)
+    dry-equalizer (0.3.0)
+    dry-inflector (0.2.1)
+    dry-logic (0.6.1)
+      concurrent-ruby (~> 1.0)
+      dry-core (~> 0.2)
+      dry-equalizer (~> 0.2)
+    dry-struct (0.5.1)
+      dry-core (~> 0.4, >= 0.4.3)
+      dry-equalizer (~> 0.2)
+      dry-types (~> 0.13)
+      ice_nine (~> 0.11)
+    dry-types (0.13.4)
+      concurrent-ruby (~> 1.0)
+      dry-container (~> 0.3)
+      dry-core (~> 0.4, >= 0.4.4)
+      dry-equalizer (~> 0.2)
+      dry-inflector (~> 0.1, >= 0.1.2)
+      dry-logic (~> 0.4, >= 0.4.2)
+    excon (0.85.0)
+    ffi (1.15.3)
+    ffi-compiler (1.0.1)
+      ffi (>= 1.0.0)
+      rake
+    hashdiff (1.0.1)
+    http (4.4.1)
+      addressable (~> 2.3)
+      http-cookie (~> 1.0)
+      http-form_data (~> 2.2)
+      http-parser (~> 1.2.0)
+    http-cookie (1.0.4)
+      domain_name (~> 0.5)
+    http-form_data (2.3.0)
+    http-parser (1.2.3)
+      ffi-compiler (>= 1.0, < 2.0)
+    ice_nine (0.11.2)
+    jsonpath (0.9.9)
+      multi_json
+      to_regexp (~> 0.2.1)
+    k8s-ruby (0.10.5)
+      dry-struct (~> 0.5.0)
+      dry-types (~> 0.13.0)
+      excon (~> 0.71)
+      hashdiff (~> 1.0.0)
+      jsonpath (~> 0.9.5)
+      recursive-open-struct (~> 1.1.0)
+      yajl-ruby (~> 1.4.0)
+      yaml-safe_load_stream (~> 0.1)
+    multi_json (1.15.0)
+    net-ssh (6.1.0)
+    public_suffix (4.0.6)
+    rake (12.3.3)
+    recursive-open-struct (1.1.3)
+    rspec (3.10.0)
+      rspec-core (~> 3.10.0)
+      rspec-expectations (~> 3.10.0)
+      rspec-mocks (~> 3.10.0)
+    rspec-core (3.10.1)
+      rspec-support (~> 3.10.0)
+    rspec-expectations (3.10.1)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.10.0)
+    rspec-mocks (3.10.2)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.10.0)
+    rspec-support (3.10.2)
+    sshkey (2.0.0)
+    thor (1.1.0)
+    to_regexp (0.2.1)
+    unf (0.1.4)
+      unf_ext
+    unf_ext (0.0.7.7)
+    yajl-ruby (1.4.1)
+    yaml-safe_load_stream (0.1.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  hetzner-k3s!
+  rake (~> 12.0)
+  rspec (~> 3.0)
+
+BUNDLED WITH
+   2.1.4

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2021 Vito Botta
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,221 @@
+# Create production grade Kubernetes clusters in Hetzner Cloud in a couple of minutes or less
+
+This is a CLI tool - based on a Ruby gem - to quickly create and manage Kubernetes clusters in [Hetzner Cloud](https://www.hetzner.com/cloud) using the lightweight Kubernetes distribution [k3s](https://k3s.io/) from [Rancher](https://rancher.com/).
+
+Hetzner Cloud is an awesome cloud provider which offers a truly great service with the best performance/cost ratio in the market. I highly recommend them if European locations (Germany and Finland) are OK for your projects (the Nuremberg data center has decent latency for US users as well). With Hetzner's Cloud Controller Manager and CSI driver you can provision load balancers and persistent volumes very easily.
+
+k3s is my favorite Kubernetes distribution now because it uses much less memory and CPU, leaving more resources to workloads. It is also super quick to deploy because it's a single binary.
+
+Using this tool, creating a highly available k3s cluster with 3 masters for the control plane and 3 worker nodes takes about **a couple of minutes** only. This includes
+
+- creating the infra resources (servers, private network, firewall, load balancer for the API server for HA clusters)
+- deploying k3s to the nodes
+- installing the [Hetzner Cloud Controller Manager](https://github.com/hetznercloud/hcloud-cloud-controller-manager) to provision load balancers right away
+- installing the [Hetzner CSI Driver](https://github.com/hetznercloud/csi-driver) to provision persistent volumes using Hetzner's block storage
+- installing the [Rancher System Upgrade Controller](https://github.com/rancher/system-upgrade-controller) to make upgrades to a newer version of k3s easy and quick
+
+
+## Requirements
+
+All that is needed to use this tool is
+
+- an Hetzner Cloud account
+- an Hetzner Cloud token: for this you need to create a project from the cloud console, and then an API token with **both read and write permissions** (sidebar > Security > API Tokens); you will see the token only once, so ensure you take note of it somewhere safe
+- a recent Ruby runtime installed (see [this page](https://www.ruby-lang.org/en/documentation/installation/) for instructions if you are not familiar with Ruby). I am also going to try and create single binaries for this tool that will include the Ruby runtime, for easier installation.
+
+## Installation
+
+Once you have the Ruby runtime up and running, you just need to install the gem:
+
+```bash
+gem install hetzner-k3s
+```
+
+This will install the `hetzner-k3s` executable in your PATH.
+
+## Creating a cluster
+
+The tool requires a simple configuration file in order to create/upgrade/delete clusters, in the YAML format like in the example below:
+
+```yaml
+---
+hetzner_token: <your token>
+cluster_name: test
+kubeconfig_path: "./kubeconfig"
+k3s_version: v1.21.3+k3s1
+ssh_key_path: "~/.ssh/id_rsa.pub"
+location: nbg1
+masters:
+  instance_type: cpx21
+  instance_count: 3
+worker_node_pools:
+- name: small
+  instance_type: cpx21
+  instance_count: 4
+- name: big
+  instance_type: cp321
+  instance_count: 2
+```
+
+It should hopefully be self explanatory; you can run `hetzner-k3s releases` to see a list of the available releases from the most recent to the oldest available.
+
+If you set `masters.instance_count` to 1 then the tool will create a non highly available control plane; for production clusters you may want to set it to a number greater than 1. This number must be odd to avoid split brain issues with etcd and the recommended number is 3.
+
+You can specify any number of worker node pools for example to have mixed nodes with different specs for different workloads.
+
+At the moment Hetzner Cloud has three locations: two in Germany (`nbg1`, Nuremberg and `fsn1`, Falkensteing) and one in Finland (`hel1`, Helsinki).
+
+For the available instance types and their specs, either check from inside a project when adding a server manually or run the following with your Hetzner token:
+
+```bash
+curl \
+	-H "Authorization: Bearer $API_TOKEN" \
+	'https://api.hetzner.cloud/v1/server_types'
+```
+
+
+Finally, to create the cluster run:
+
+```bash
+hetzner-k3s create-cluster --config-file cluster_config.yaml
+```
+
+This will take a couple of minutes or less depending on the number of masters and worker nodes.
+
+If you are creating an HA cluster and see the following in the output you can safely ignore it - it happens when additional masters are joining the first one:
+
+```
+Job for k3s.service failed because the control process exited with error code.
+See "systemctl status k3s.service" and "journalctl -xe" for details.
+```
+
+
+### Idempotency
+
+The `create-cluster` command can be run any number of times with the same configuration without causing any issue, since the process is idempotent. This means that if for some reason the create process gets stuck or throws errors (for example if the Hetzner API is unavailable or there are timeouts etc), you can just stop the current command, and re-run it with the same configuration to continue from where it left.
+
+### Adding nodes
+
+To add one or more nodes to a node pool, just change the instance count in the configuration file for that node pool and re-run the create command.
+
+### Scaling down a node pool
+
+To make a node pool smaller:
+
+- decrease the instance count for the node pool in the configuration file so that those extra nodes are not recreated in the future
+- delete the nodes from Kubernetes (`kubectl delete node <name>`)
+- delete the instances from the cloud console (make sure you delete the correct ones :p)
+
+In a future relese I will add some automation for the cleanup.
+
+### Replacing a problematic node
+
+- delete the node from Kubernetes (`kubectl delete node <name>`)
+- delete the correct instance from the cloud console
+- re-run the create script. This will re-create the missing node and have it join to the cluster
+
+
+### Converting a non-HA cluster to HA
+
+It's easy to convert a non-HA with a single master cluster to HA with multiple masters. Just change the masters instance count and re-run the create command. This will create a load balancer for the API server and update the kubeconfig so that all the API requests go through the load balancer.
+
+## Upgrading to a new version of k3s
+
+If it's the first time you upgrade the cluster, all you need to do to upgrade it to a newer version of k3s is run the following command:
+
+```bash
+hetzner-k3s upgrade-cluster --config-file cluster_config.yaml --new-k3s-version v1.21.3+k3s1
+```
+
+So you just need to specify the new k3s version as an additional parameter and the configuration file will be updated with the new version automatically during the upgrade. To see the list of available k3s releases run the command `hetzner-k3s releases`.
+
+Note that the API server will briefly be unavailable during the upgrade of the controlplane.
+
+To check the upgrade progress, run `watch kubectl get nodes -owide`. You will see the masters being upgraded one per time, followed by the worker nodes.
+
+
+### What to do if the upgrade doesn't go smoothly
+
+If the upgrade gets stuck for some reason, or it doesn't upgrade all the nodes:
+
+1. Clean up the existing upgrade plans and jobs, and restart the upgrade controller
+
+```bash
+kubectl -n system-upgrade delete job --all
+kubectl -n system-upgrade delete plan --all
+
+kubectl label node --all plan.upgrade.cattle.io/k3s-server- plan.upgrade.cattle.io/k3s-agent-
+
+kubectl -n system-upgrade rollout restart deployment system-upgrade-controller
+kubectl -n system-upgrade rollout status deployment system-upgrade-controller
+```
+
+I recommend running the above commands also when upgrading a cluster that has already been upgraded at least once previously, since the upgrade leaves some stuff behind that needs to be cleaned up.
+
+2. Re-run the `upgrade-cluster` command with an additiona parameter `--force true`.
+
+I have noticed that sometimes I need to re-run the upgrade command a couple of times to complete an upgrade successfully. Must be some bug in the system upgrade controller but I haven't investigated further.
+
+You can also check the logs of the system upgrade controller's pod:
+
+```bash
+kubectl -n system-upgrade logs -f $(kubectl -n system-upgrade get pod -l pod-template-hash -o jsonpath="{.items[0].metadata.name}")
+```
+
+A final note about upgrades is that if for some reason the upgrade gets stuck after upgrading the masters and before upgrading the worker nodes, just cleaning up the resources as described above might not be enough. In that case also try running the following to tell the upgrade job for the workers that the masters have already been upgraded, so the upgrade can continue for the workers:
+
+```bash
+kubectl label node <master1> <master2> <master2> plan.upgrade.cattle.io/k3s-server=upgraded
+```
+
+## Deleting a cluster
+
+To delete a cluster, running
+
+```bash
+hetzner-k3s delete-cluster --config-file cluster_config.yam
+```
+
+This will delete all the resources in the Hetzner Cloud project for the cluster being deleted.
+
+
+## Additional info
+
+### Load balancers
+
+Once the cluster is ready, you can already provision services of type LoadBalancer for your workloads (such as the Nginx ingress controller for example) thanks to the Hetzner Cloud Controller Manager that is installed automatically.
+
+There are some annotations that you can add to your services to configure the load balancers. I personally use the following:
+
+```yaml
+  service:
+    annotations:
+      load-balancer.hetzner.cloud/hostname: <a valid fqdn>
+      load-balancer.hetzner.cloud/http-redirect-https: 'false'
+      load-balancer.hetzner.cloud/location: nbg1
+      load-balancer.hetzner.cloud/name: <lb name>
+      load-balancer.hetzner.cloud/uses-proxyprotocol: 'true'
+      load-balancer.hetzner.cloud/use-private-ip: "true"
+```
+
+I set `load-balancer.hetzner.cloud/hostname` to a valid hostname that I configure (after creating the load balancer) with the IP of the load balancer; I use this together with the annotation `load-balancer.hetzner.cloud/uses-proxyprotocol: 'true'` to enable the proxy protocol. Reason: I enable the proxy protocol on the load balancers so that my ingress controller and applications can "see" the real IP address of the client. However when this is enabled, there is a problem where [cert-manager](https://cert-manager.io/docs/) fails http01 challenges; you can find an explanation of why [here](https://github.com/compumike/hairpin-proxy) but the easy fix provided by some providers - including Hetzner - is to configure the load balancer so that it uses a hostname instead of an IP. Again, read the explanation for the reason but if you care about seeing the actual IP of the client then I recommend you use these two annotations.
+
+The annotation `load-balancer.hetzner.cloud/use-private-ip: "true"` ensures that the communication between the load balancer and the nodes happens through the private network, so we don't have to open any ports on the nodes (other than the port 6443 for the Kubernetes API server).
+
+The other annotations should be self explanatory. You can find a list of the available annotations here.
+
+## Persistent volumes
+
+Once the cluster is ready you can create persistent volumes out of the box with the default storage class `hcloud-volumes`, since the Hetzner CSI driver is installed automatically. This will use Hetzner's block storage (based on Ceph so it's replicated and highly available) for your persistent volumes. Note that the minimum size of a volume is 10Gi. If you specify a smaller size for a volume, the volume will be created with a capacity of 10Gi anyway.
+
+## Contributing and support
+
+Please create a PR if you want to propose any changes, or open an issue if you are having trouble with the tool - I will do my best to help if I can.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the hetzner-k3s project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/vitobotta/k3s/blob/master/CODE_OF_CONDUCT.md).