Kubernetes Tooling For TechOps And Support (Local Kubernetes Clusters)

Guest post originally published on Oteemo’s blog by Tom Halligan

Introduction:

This is the second in a series of posts aimed at providing information on Kubernetes tooling (I will use the abbreviation K8s at times) to IT, DevOps and engineering teams that help support K8s deployments. These will not be deep dives into K8s architecture or extensive reviews of specific products: just the data needed to get teams started with the tools.

In the previous installment of this series, I reviewed some basic K8s support tooling including a local Kubernetes deployment solution K3d. In this article, I expand on local Kubernetes solutions as these tools are critical for helping engineering and support teams test solutions on their workstations. There are a number of advantages in the use of local K8s solutions.

Local K8s Advantages:

• No need to purchase additional hardware or increase cloud instance spend
• Multi-OS support
• Quick to deploy and quick to tear-down
• Can deploy and kick the tires on new solutions quickly
• Allow engineers to move away from Docker and Docker-Compose based workflows when they are working with apps destined for Kubernetes clusters

I went through a number of local cluster tools when starting with K8s and eventually landed on using a small subset of three solutions that I still switch between to this day, depending on the work I am doing.

I thought it might be helpful to give folks a leg up in setup of some of these solutions and provide my thoughts on the tools I use. I solution on the following three:

1. K3d (Dockerized version of K3s https://k3s.io/)
2. Minikube (Oldest localized Kubernetes cluster creation software VM based https://minikube.sigs.k8s.io/)
3. Kind (Light-weight Dockerized Kubeadm https://kind.sigs.k8s.io/)

Before we start we will need to install kubectl and Docker on your workstation see Appendix A below for links to install documents.

Note:

You do not need kubectl installed for minikube as the tool comes with its own version, but you will need the tool locally for the other solutions. The local kubectl tool will be needed as you work more and more with K8s, local or remote. A good resource for the basic usage can be found here https://kubernetes.io/docs/reference/kubectl/cheatsheet/

K3d:

The Good:

• Very fast cluster deploy  (seconds)
• Multi node clusters
• Multiple clusters on a single local system
• Stop\start clusters
• Very minimal local resource usage

The Not So Good:

• Ingress can be a bit tricky for novice

Setup & Teardown:

Install K3d  https://k3d.io/#installation
Start your cluster and ensure kubectl switches to context
k3d cluster create –servers 3
kubectl config use-context k3d-k3s-default

Teardown your cluster when done
k3d cluster rm -a (This will remove all k3d clusters on your system, just FYI if you have more than one)

Verdict:
My favorite solution due to simplicity of deployment, blazing speed and low overhead.

Minikube:

The Good: 

• Multi node clusters
• Good set of addons (minikube addons list)
• Stop\start clusters
• Great documentation (https://minikube.sigs.k8s.io/docs/)
• Multiple local hypervisor support (https://minikube.sigs.k8s.io/docs/drivers/)
• Lots of ways to push images to the cluster (https://minikube.sigs.k8s.io/docs/handbook/pushing/)

The Not So Good:

• Virtual Box based cluster installs can get borked forcing and delete recreate, so try to avoid Virtualbox
• Bit of a resource hog compared to K3d and Kind
• Slightly slower startup time compared to other solutions reviewed (See Appendix B)

Setup & Teardown:
Install Minikube https://minikube.sigs.k8s.io/docs/start/
Start your cluster kubectl will switch to minikube context by default.
minikube start –nodes=3 –vm-driver=docker 

Teardown your cluster when done
minikube delete

Verdict:
Good choice for teams starting out with Kubernetes. Also given solutions age there is a lot of supporting online documentation for some edge case solutions.

Kind:

The Good: 

• Fast cluster deploy
• Kubernetes-style API configuration for cluster creation via Kubeadm
• Kind is how Kubernetes is tested in development so great solution for certification training or activity needing vanilla K8s
• Very minimal local resource usage
• Local image registry integration https://kind.sigs.k8s.io/docs/user/local-registry/

The Not So Good:

• Not as novice friendly as other solutions
• Just the facts K8s with limited extras (can also be a good point)
• Multi-node, or other advanced configurations, needs to use config file and can not be done via imperative cli command (This is a more cloud native stance but not as friendly to novice Kubernetes users so good or bad depending on your perspective)
• Hard-coded container-runtime (containerd)

Setup & Teardown:

Install kind https://kind.sigs.k8s.io/docs/user/quick-start/#installation
Start your cluster kubectl will switch to kind-kind context by default
kind create cluster

Teardown your cluster when done
kind delete cluster

Verdict:
Good for CKA,CKAD and CKS certification preparation preparation (Kubeadm and etcd under the hood) and use by more advanced users.

Extra Credit:

Not a local K8s solution but a great tool for quickly spinning up a Linux VM on your local system (much less trouble then Vagrant or Virtualbox) is Multipass https://multipass.run/. The tool runs on Windows, MAC and Linux (install how-tos in the link).

You can use Multipass to setup 2-3 local linux nodes which you can use to perform a manual K8s build using Kubeadm (run the bash script from Appendix C to install kubectl and docker on the new multipass node).

Installing kubeadm  https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm

Creating a cluster with kubeadm  https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm

Conclusion:

Running Kubernetes clusters on your local system opens up an enormous landscape of technology testing capabilities. The above synopsis does not do justice to the broad feature sets that each tool offers. Minikube and its add-ons are a great way to take a first look at service meshes and other complex K8s tools. K3d offers a ton of features available via more advanced config files (a recent feature) and is so so fast. Lastly, kind is great for working in a more advanced and at the same time non-opinionated Kubernetes space.

Like me, I think once you look at the various local K8s cluster providers you will settle on a few for your work and personal uses, perhaps the ones above or some of the other solutions that I did not include in this introductory review.

Appendix A:

kubectl install info https://kubernetes.io/docs/tasks/tools/
Docker Desktop on Windows https://docs.docker.com/docker-for-windows/install/
Docker Desktop on MAC https://docs.docker.com/docker-for-mac/install/
Docker Engine (Various Linux Distros) https://docs.docker.com/engine/install/

Appendix B:

Below is a comparison of the startup times for each of the solutions on the same system (My workstation). Images need to be pulled on fresh start so your startup time will be greater if that is the case. All deployments were simple one node clusters, with Docker backend, no additional configurations. Of course times will vary on the basis of resources free on the system you are working with.

1. K3d 1 minute 45 seconds for all components to come online.
2. Minikube 2 minutes 44 seconds for all components to come online.
3. Kind 2 minutes 3 seconds for all components to come online.

Appendix C:

#!/bin/bash
# Install kubectl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update && sudo apt-get install -y kubectl

# Remove any old Docker items
sudo apt remove docker docker-engine docker.io containerd runc

# Install all pre-reqs for Docker
sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common

# Add the Docker repository, we are installing from Docker and not the Ubuntu APT repo.
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# Install Docker
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io

# Add your base user to the Docker group so that you do not need sudo to run docker commands
sudo usermod -aG docker ubuntu

kubectl create deployment nginxgood —-image=nginx