Projects:Kubernetes

According to Wikipedia, Kubernetes (commonly stylized as k8s) is an open-source container orchestration system for automating application deployment, scaling, and management. I've heard a lot about it, and it seems to solve some problems I'm encountering sometimes, so I'd like to get to know it better. While doing so, I wanted to make a write-up of what I found out.

This page is not intended as a tutorial, but it will link to various tutorials I found useful. In this way, I hope that it can be used as a reference for someone setting up Kubernetes to be able to learn about it more quickly.

Disclaimer: I'm a beginner in the Kubernetes area, so this tutorial will contain errors. Please correct errors where you find them, and if possible, write some explanation! The "I" in this article is Sjors, but parts are possibly written by other people.

Problem

We start with a Linux desktop, running some version of Arch Linux. Someone wants to run an application on it for which only binaries are available, but those binaries were compiled on Ubuntu. Now, system libraries are different between Arch Linux and Ubuntu, and while it's possible to create binaries that run independent of system libraries (called "static binaries"), in this particular situation let's assume the binaries aren't static, but you still want to run them on your Arch installation.

You can get a second machine and run Ubuntu on it, or similarly you could use a Virtual Machine. But, there's a simpler and more efficient solution: Docker allows you to install, within your Linux distribution (the "host"), another Linux distribution (let's call it "guest" for now). The host and the guest may be completely different – technically, only the kernel of the host is used, and of course the host and guest must have compatible processor architectures. You run the guest environment within Docker (a "Docker Container") and run that binary in it.

This is how easy that is:

sjors@somebox:~$ lsb_release -a
LSB Version: 1.4
Distributor ID: Arch
Description: Arch Linux
Release: rolling
Codename: n/a
sjors@somebox:~$ docker run -ti ubuntu:bionic bash
root@a5b210d251c2:/# apt-get update && apt-get -y install lsb-release
[....]
root@a5b210d251c2:/# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic

The -it flag makes this an interactive container that can run bash; just as easily, you can run detached containers running webservers, and use the integrated port-forwarding features to make them accessible to the outside.

Now, Docker has some problems of its own:

• You start Docker containers by accessing the Docker daemon; the daemon runs containers as root and allows you to start a container with a bind-mount of "/". Basically, having access to the Docker daemon means you have root on the system, but you need access to do anything. It's all-or-nothing.
• When your Docker machine goes down, all containers are gone. You'll have to either restart all containers manually, or have boot-scripts that set them up, but there's no automatic restart mechanism.
• When you want to run more Docker containers than fit on one machine, there's no horizontal scaling mechanism built-in.
• When you want to run a service multiple times, e.g. for redundancy, you need to schedule them manually multiple times and roll your own method of load-balancing them.

Kubernetes provides a solution for these problems, while otherwise looking very much like Docker. In fact, when you're familiar with Docker some of the commands below will also be very familiar to you.

Concepts

In this section, I'll explain some of Kubernetes' concepts quickly, and add links if you want to know more.

• Container: Like with Docker, this is one 'guest environment' in which you can run anything. Usually, Kubernetes containers are, in fact, Docker containers.
• Pod: The basic unit you actually schedule in Kubernetes. Usually, a Pod contains one Container, but a Pod can consist of multiple Containers which can be a very useful feature. More on that later. A Pod isn't durable: when anything causes a Pod to stop (e.g. the machine it runs on has a power outage), it won't be restarted automatically.
• Deployment: An indication of "desired state" of the cluster in terms of the pods you always expect to have. The Kubernetes system will always try to match these Deployments to what it's actually running. Basically, a Deployment is the way to start a Pod and ensure it stays running.
• Job: An indication that you want to run some command to completion. When the cluster has a Job, it will keep re-creating its Pods until a given number of them succeed successfully. Basically, a Job is the way to start a Pod and ensure it finishes once.
• Volume: As in Docker, changes to containers are temporary and will be gone when the container stops. If you want to keep those changes after a restart, like in Docker, you make a Volume. They are also useful to share data between containers. In Kubernetes, Volumes are kept over restarts of Containers, but not over restarts of Pods, unless they are Persistent. More on that later.
• Service: When your Pod contains some application, such as a webserver, you can make its TCP port available as a Service so that people (inside or outside the cluster) can connect to it. For an application you want to run redundantly, multiple Pods can be started; you'll configure them to share the same Service. This way, when you connect to the Service, you'll get one of the running Pods behind it. Instant redundancy!
• Namespace: Kubernetes supports running multiple "virtual clusters" on the infrastructure of one "physical cluster". Those virtual clusters are called "namespaces", and you can restrict access to certain namespaces. Normally, you're only working with the "default" namespace.

Those are some concepts that allow you to use a Kubernetes cluster. In this guide, we'll also be setting up the infrastructure behind that:

• Node: A machine that actually runs the Containers. Can be bare-metal or a virtual machine, or even an embedded IoT device. A Node runs a process called "Kubelet" which interacts with the Docker daemon (usually) to set everything up, but normally you never communicate directly with it.
• Control Plane: A set of some applications (API server, etcd, controller manager, scheduler...) that make sure the cluster is "healthy". For example, it starts Pods when you request it to, but also when a Node goes down that was running Pods for some Deployment, it restarts those Pods elsewhere. These master applications themselves also run in Pods, in a separate namespace called "kube-system".
• Master node: Otherwise a normal Node, but it runs the Control Plane applications. By default, a Master node will only run Pods for these applications, but you can configure it to allow normal Pods too. There can be multiple Master nodes, for redundancy of the cluster.

Understanding networking

Networking within a Kubernetes cluster isn't difficult, but requires some specific explanation. Applications within Kubernetes containers need to be able to access each other, regardless of whether they run on the same Node or not. To make it worse, Nodes may be running behind firewalls or may be running in different subnets. Luckily, Kubernetes has some mechanisms that make it very robust against this.

When setting up a Kubernetes cluster, there are two important internal IP ranges throughout the cluster:

• The Pod network range. This internal range is automatically split over Nodes, and Pods get individual addresses from it.
  • For example, you can set this to 10.123.0.0/16; the master node will likely get 10.123.0.0/24 and the second Node you add after that gets 10.123.1.0/24 and so on. A Pod running on this second node may have 10.123.1.55 as an IP address. (If the Pod has multiple containers, all of them will have the same IP address.)
• The service network range. When you register a Service, such as "my-fun-webserver", it automatically gets an IP address within this range. An application called the 'kube-proxy', running automatically on every Node, then takes care that any communication with this IP address is forwarded to one of the actual Pods providing that service (by configuring iptables). Fun fact: the Kubernetes API server registers itself as a service and is always available at the first host address of the range you chose.
  • For example, your service network range may be 10.96.0.0/16; the Kubernetes API service makes itself available at 10.96.0.1. When you communicate with this IP address, the communication is automatically translated (by iptables) to be sent to the Pod IP address of the Kubernetes API service, e.g. 10.123.1.55.

It's important that these ranges don't overlap, and they also both shouldn't overlap with any relevant IP ranges within your existing network! The Kubernetes folks suggest you use something within 10.0.0.0/8 if your local network range is within 192.168.0.0/16 and vice-versa.

Since there is no one-size-fits-all solution to networking between Nodes, Kubernetes allows that to be done by specific networking plugins, called CNI. There are a number of such plugins, but a friend of mine experienced with running Kubernetes clusters explained why you should probably go for Weave:

• It uses the vxlan standard. vxlan allows tunneling traffic between nodes automatically. When a packet should travel from one Node to another, a small UDP/IP header is prepended to it, so it is sent to the right node, where the header is taken off and routing continues.
• This method allows it to cross almost all difficult network setups as long as you can have one UDP port forwarded between the machines.
• Weave is smart enough to figure out the most efficient way to use vxlan given your Linux kernel version.
• It's also pretty simple: just a single Go binary.

Kubernetes takes care that the pod network range and service network range is not only usable within pods, but also on the nodes. So, using the example values above, `https://10.96.0.1/` will reach the Kubernetes API server within pods and on nodes, also highly-available if you have multiple masters, which is pretty convenient.

Some more important features of Kubernetes networking:

• A Kubernetes cluster automatically runs a "CoreDNS" pod, which provides DNS to all other pods. It forwards requests outside the cluster to an upstream DNS server, but most importantly, provides an internal `cluster.local` DNS zone that you can use to look up other pods or services. For example, `kubernetes.default.svc.cluster.local` resolves to 10.96.0.1, as above. (In that hostname, 'kubernetes' is the service name, 'default' is the namespace.)
• When a pod is listening on some TCP port, you don't need to use Services to reach them externally: kubectl port-forward pod/foobarbaz 8080:80 forwards local port 8080 to port 80 of a pod called 'foobarbaz', and for this to work your kubectl can run on any machine with credentials to access the API server, it doesn't need to be part of the cluster.

Setting it all up

With that all behind us, let's start setting up our first cluster!

For a useful cluster, you'll need at least one machine to be the master, but of course we'll use at least two so we can call it an actual cluster. The master node is strongly recommended to have at least two CPUs and 2 GB of RAM. The other nodes can be a bit smaller.

For my tests, I've used four machines:

• Kubetest1 - 1 CPU, 1 GB RAM - 145.131.6.177 - Ubuntu 16.04
• Kubetest2 - 1 CPU, 1 GB RAM - 145.131.8.75 - Ubuntu 16.04
• Kubetest3 - 1 CPU, 1 GB RAM - 145.131.5.151 - Ubuntu 16.04
• Kubetest4 - 2 CPU, 2 GB RAM - 145.131.6.179 - Ubuntu 16.04

(I initially wanted a three-machine cluster, and forced the master on a 1 CPU/1 GB RAM machine, but processes were being killed because of going out-of-memory. So, kubetest4 became a bit bigger and will be my master.)

Now, the machines in your cluster should be able to access each other on all ports used by Kubernetes (see this list here). Important note: the machines should also be able to access themselves on their external IP address! I was initially having problems because of this, where pods on a worker node could reach the master normally, but pods on the master node couldn't. It was because the external IP address of the master were used in some communications, which was ironically impossible just on the master itself.

So, once you have your machines up, and you know they can reach each other, we'll start installation.

• First, we choose our Pod network range and Service range. See previous section. I used 10.107.0.0/16 for the network range and 10.16.0.0/16 for the service range.
• Optionally, we disable swap on the machine, as Kubernetes prefers to run without it. To do this, check /proc/swaps for your swap locations and run swapoff on them, then remove the entries from /etc/fstab. Or, leave them enabled and later disable the check in Kubernetes.
• We install the Docker daemon. On Ubuntu, sudo apt-get install docker.io does it.
• We install the basic Kubernetes tools: kubeadm, kubelet and kubecfg. Here's a nice guide for it.
• We use kubeadm to create our cluster. Just follow this guide, until the "Installing a pod network add-on" step.
  • I used: kubeadm init --pod-network-cidr "10.107.0.0/16" --service-cidr "10.16.0.0/16"
  • After this, kubectl get nodes should already respond with your master node, and kubectl get pods --all-namespaces should mention the kube-system pods that make up the Control Plane as discussed above!
• We install a pod network add-on. As discussed before, we use Weave, but YMMV. See this guide on Weave setup, with an important note: if you don't use Kubernetes' default Pod network CIDR, pay attention to the exact kubectl apply step.
  • I used: kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')&env.IPALLOC_RANGE=10.107.0.0/16"
  • After this, kubectl get pods --all-namespaces should show some additional pods. Most importantly, it should now show that CoreDNS is running.
• Now, we can join our other nodes. kubeadm init will show a kubeadm join command at the end of its output, which you can run on the other nodes. If you don't have that output anymore, see this page on how to recreate a valid join command.
  • I also copied the /etc/kubernetes/admin.conf from the master to the other nodes (root-readable only), and set KUBECONFIG=/etc/kubernetes/admin.conf, so I could run kubectl on them. But that's up to you!

Hopefully, after understanding all the concepts discussed earlier, this process was a matter of mere minutes!

Your node list should be complete now:

root@kubetest4:~# kubectl get nodes
NAME        STATUS   ROLES    AGE    VERSION
kubetest1   Ready    <none>   2d2h   v1.13.4
kubetest2   Ready    <none>   2d2h   v1.13.4
kubetest3   Ready    <none>   2d5h   v1.13.4
kubetest4   Ready    master   2d5h   v1.13.4

By the way, you can run kubectl commands from your own machine as well by copying /etc/kubernetes/admin.conf from the master to your $HOME/.kube/config, and possibly after some firewall config to allow communications.

TO DO: It should be possible to generate user credentials instead of taking admin credentials. But how?

Creating some basic pods

You'll see your cluster has no pods in the default namespace:

$ kubectl get pods
No resources found.

The simplest thing we can do, now that we have a cluster, is run a random image (which is a Docker image, in this case ubuntu:bionic, pulled from Docker Hub) with a random command, like before:

$ kubectl run --restart=Never -ti --image=ubuntu:bionic bash
If you don't see a command prompt, try pressing enter.
root@bash:/# apt-get update && apt-get -y install lsb-release
[....]
root@bash:/# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic

Indeed, your pod will be shown now:

$ kubectl get pods
NAME   READY   STATUS    RESTARTS   AGE
bash   1/1     Running   0          56s

In the command above, we gave --restart=Never. There are three parameters to this option:

• Never: When the pod exits, it will not be recreated. If this is given, kubectl will just create a Pod.
• OnFailure: When the pod exits with failure, it will be recreated, otherwise not. In other words, kubectl will start a Job to create the Pod. (If that doesn't make sense, quickly check up on Jobs in the Concepts section above!)
• Always: When the pod exits, it will be recreated. You guessed it: in this case, kubectl will create a Deployment. (If you didn't guess it, re-check the Concepts section!)

The default is --restart=Always so you'll see the container is recreated like this:

$ kubectl run -ti --image=ubuntu:bionic bash 
If you don't see a command prompt, try pressing enter.
root@bash-58654c7f4b-9bhcq:/# touch foobarbaz
root@bash-58654c7f4b-9bhcq:/# exit
Session ended, resume using 'kubectl attach bash-58654c7f4b-9bhcq -c bash -i -t' command when the pod is running

[...wait for a bit until the pod comes back up...]

$ kubectl attach bash-58654c7f4b-9bhcq -c bash -i -t
If you don't see a command prompt, try pressing enter.
root@bash-58654c7f4b-9bhcq:/# ls foobarbaz
ls: cannot access 'foobarbaz': No such file or directory

As you can see, the container restarted, as /foobarbaz did not exist anymore when re-attaching after the exit. Any state in the filesystem of the container/pod will be gone upon restart.

If you tried this, you can check and remove the deployment like this:

$ kubectl get deployments      
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
bash   1/1     1            1           3s
$ kubectl delete deployment bash
deployment.extensions "bash" deleted

Storage using Volumes

So let's try adding a Volume to our pod, to see if we can make some changes persistent. Kubernetes supports many types of volumes; in this case we use emptyDir which is just a locally stored disk (initially empty).

There is no command-line parameter to kubectl run to add volumes. Internally, kubectl run translates your commandline to a JSON request to the Kubernetes API server; we'd have to add any additional requests directly into the JSON. This can be done with the --overrides flag, but at this point, it is probably easier to switch to sending those commands ourselves. We can use JSON for this too, but many users use YAML for this, so we will too.

The command above, kubectl run --restart=Never -ti --image=ubuntu:bionic bash, translates to the following YAML:

apiVersion: v1
kind: Pod
metadata:
  name: bash
spec:
  containers:
  - name: bash
    image: ubuntu:bionic
    stdin: true
    stdinOnce: true
    tty: true

As you can see, this is a Pod named "bash" and with one container, also called "bash", running ubuntu:bionic.

To create this Pod and attach to it, we write the code above to bash.yaml and run these commands:

$ kubectl create -f bash.yaml
pod/bash created
$ kubectl attach -ti bash -c bash
If you don't see a command prompt, try pressing enter.
root@bash:/# exit 0
exit
$ kubectl delete pod bash

Now, we will recreate the pod with an emptyDir volume mounted at /foo.

$ cat bash.yaml
apiVersion: v1
kind: Pod
metadata:
  name: bash
spec:
  volumes:
  - name: testing-volume
    emptyDir: {}
  containers:
  - name: bash
    image: ubuntu:bionic
    stdin: true
    stdinOnce: true
    tty: true
    volumeMounts:
    - mountPath: /foo
      name: testing-volume
$ kubectl create -f bash.yaml
pod/bash created
$ kubectl attach -ti bash -c bash
If you don't see a command prompt, try pressing enter.
root@bash:/# mount | grep foo
/dev/mapper/ubuntu--vg-root on /foo type ext4 (rw,relatime,errors=remount-ro,data=ordered)
root@bash:/# exit 0
exit
$ kubectl delete pod bash

Of course, this volume isn't really persistent just yet; restarts of the pod will cause it to be recreated (the Volume has the "lifetime of the Pod") so it actually doesn't serve our purpose.

There's two ways to get around this:

• Instead of using an emptyDir volume, use a volume type that stores its contents somewhere
• Or, alternatively, make a PersistentVolume that exists outside our Pod, and then mount it.

A volume type with persistency

As seen before, Kubernetes supports many volume types, some of which are naturally persistent because they store the data on an external service.

The quickest way to set up persistent storage is to set up a NFS server. (Remember that in production, you'll want to go for something redundant, such as Ceph or Gluster or clustered NFS.)

root@kubetest4:~# apt-get install -y nfs-kernel-server
[...]
root@kubetest4:~# mkdir /persistent
root@kubetest4:~# chmod 0777 /persistent
root@kubetest4:~# cat <<EOF >>/etc/exports
/persistent *(rw,sync,no_subtree_check)
EOF
root@kubetest4:~# exportfs -ra

Make sure the other nodes can access it, by checking that mkdir /persistent && mount -t nfs <ipaddress>:/persistent /persistent works. Also, create a file /persistent/helloworld.txt with some contents.

Now we can go ahead and create our pod with this volume (replacing <ipaddress> with the address of your NFS host again):

$ cat bash.yaml
apiVersion: v1
kind: Pod
metadata:
  name: bash
spec:
  volumes:
  - name: testing-volume
    nfs:
      path: /persistent
      server: <ipaddress>
  containers:
  - name: bash
    image: ubuntu:bionic
    stdin: true
    stdinOnce: true
    tty: true
    volumeMounts:
    - mountPath: /foo
      name: testing-volume
$ kubectl create -f bash.yaml
pod/bash created
$ kubectl attach -ti bash -c bash
If you don't see a command prompt, try pressing enter.
root@bash:/# mount | grep foo
145.131.6.179:/persistent on /foo type nfs4 (rw,relatime,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=145.131.6.179,local_lock=none,addr=145.131.6.179)
root@bash:/# cat /foo/helloworld.txt
Hello world!

Make a PersistentVolume with a claim

The Pod YAML file above has persistency, but with the downside that the definition describes where the files should be stored. Most of the time, definitions don't care about where the files are stored, but only what properties the storage has:

• How much storage can we use?
• Is it optimized for small files or for large files?
• Is it highly available?
• Is it backed up?

To more accurately implement this use-case, Kubernetes has two object types called PersistentVolume and PersistentVolumeClaim. The idea is that a cluster administrator creates PersistentVolumes (abbreviated pv) that know what kind of storage they represent and where to find it; then, users create PersistentVolumeClaim (abbreviated pvc) asking for storage with constraints like "at least 10 GB". When a PVC is created, it is matched to its closest PV and a link is created. If no PV is available to fulfill a PVC, the PVC stays in "Pending" state. In a way, PersistentVolumes are like Nodes in the sense that they provide capacity, and PersistentVolumeClaims are like Pods in the sense that they use that capacity if it is available anywhere in the cluster.

In this section, we'll rewrite our Pod from before to use a PersistentVolumeClaim with its constraints, and then have that PersistentVolumeClaim automatically matched to a PersistentVolume that provides the same NFS share. The first thing we'll do is create the PersistentVolumeClaim:

$ cat testing-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: testing-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
$ kubectl apply -f testing-pvc.yaml 
persistentvolumeclaim/testing-pvc created
$ kubectl get pvc
NAME             STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
testing-pvc      Pending                                                     6s

As you see, the testing-pvc PVC is created, but is Pending, because haven't supplied any PersistentVolumes to the cluster yet that can fulfill it. So, we create a PersistentVolume:

$ cat nfs-storage.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-storage
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: /persistent
    server: <IP address>
$ kubectl apply -f nfs-storage.yaml
$ kubectl get pv        
NAME          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
nfs-storage   10Gi       RWO            Retain           Bound    default/testing-pvc                              18s

It might take a moment for the PVC to be bound, but sure enough, after a bit, it is:

$ kubectl get pvc
NAME             STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
testing-pvc      Bound    nfs-storage   10Gi       RWO                           3m15s

Note that we asked for only 2 GB, but since the volume provides 10 GB, we got the lowest capacity volume that could be bound.

And let's create a Pod to use it:

$ cat bash-pvc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: bash
spec:
  volumes:
  - name: testing-volume
    persistentVolumeClaim:
      claimName: testing-pvc
  containers:
  - name: bash
    image: ubuntu:bionic
    stdin: true
    stdinOnce: true
    tty: true
    volumeMounts:
    - mountPath: /foo
      name: testing-volume
$ kubectl create -f bash.yaml
pod/bash created
$ kubectl attach -ti bash -c bash
If you don't see a command prompt, try pressing enter.
root@bash:/# mount | grep foo
145.131.6.179:/persistent on /foo type nfs4 (rw,relatime,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=145.131.6.179,local_lock=none,addr=145.131.6.179)
root@bash:/# cat /foo/helloworld.txt
Hello world!

This might seem like an insignificant victory, since we're getting the same end result, but it's a crucial step forward: our Pods, Jobs and Deployments don't need to care anymore where their storage comes from, only that it is persistent.

There's some more benefits of Persistent Volume Claims:

• If a PVC is deleted while there is still a Pod using it, it will switch state to Terminating but will not disappear until the Pods are gone, too.
• PVCs have a reclaim policy that can be set to Delete or Recycle, allowing the PV's contents to be deleted as well.
• When StorageClasses are used, PVC's can only be bound to a PV with the same Storage Class. When no free PV exists within the Storage Class, while a PVC is waiting to be bound, a PV can be automatically created. This is called Dynamic provisioning.
• It is possible to make a PV with a "Node Affinity", causing any Pods using that PV to run on a specific node. This combines very well with the HostPath volume type, as this allows a bind-mount of some directory on a Node to be accessible within a Pod.
  • But, it should also be obvious that this is a security risk if you allow untrusted users to create PV's and use them in Pods. TO DO: Add a section on protecting this.

Running a Deployment using this volume

Now that we know how to persistently store data and access it from Pods, it's time to create an actual Deployment with a real-world application in it.

$ cat nginx.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-html-storage
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: /persistent/html
    server: <IP of nfs server>
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-webfiles
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
      - name: webfiles
        persistentVolumeClaim:
          claimName: nginx-webfiles
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: webfiles

Now, before starting this deployment, make an example web page by creating /persistent/html/index.html with something like:

<h1>It's working!</h1>

Then, start the PV, PVC and deployment:

$ kubectl apply -f nginx.yaml

A note about kubectl apply ("declarative management") versus kubectl create ("imperative management"): in this case, apply and create would do the same thing as the Deployment described in nginx.yaml doesn't exist yet. However, would you change nginx.yaml and run kubectl create again, you'd get an error. "Imperative management" (create, delete, replace) means you're telling kubectl what action is necessary, while "declarative management" means you're telling kubectl what the state of the cluster should be, and it will perform the correct action for you. Both are fine in a production context; from now on, this page will be using apply where possible since that seems to be the community consensus in tutorials.

Let's check if the deployment has been created and the pods as well:

$ kubectl get deployments
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   1/2     2            1           11h
$ kubectl get pods                                      
NAME                                READY   STATUS              RESTARTS   AGE
nginx-deployment-58b6c946d5-fnqr6   1/1     Running             0          49s
nginx-deployment-58b6c946d5-p2nlm   0/1     ContainerCreating   0          49s

One pod has been created, the other one is still in ContainerCreating state. Let's check why...

$ kubectl describe pod nginx-deployment-58b6c946d5-p2nlm
[....]
  Warning  FailedMount  15s  kubelet, kubetest1  (combined from similar events): MountVolume.SetUp failed for volume "websource" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/fac670f9-47d7-11e9-a977-001dd8b7660c/volumes/kubernetes.io~nfs/websource --scope -- mount -t nfs 145.131.6.179:/persistent/html /var/lib/kubelet/pods/fac670f9-47d7-11e9-a977-001dd8b7660c/volumes/kubernetes.io~nfs/websource
Output: Running scope as unit run-r368fd7089b0a46139882e708a89f8926.scope.
mount: wrong fs type, bad option, bad superblock on 145.131.6.179:/persistent/html,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

In other words, it is failing to start because the volume websource (our NFS mount) cannot be started. You can see from the output that this is running on the kubelet for kubetest1, and the error comes from mount -t nfs 145.131.6.179:/persistent/html/.... The error is correct: we need an NFS mount helper tool that isn't installed on kubetest1. I run apt-get install nfs-common on it, and sure enough, the pod is soon running:

$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-58b6c946d5-fnqr6   1/1     Running   0          6m42s
nginx-deployment-58b6c946d5-p2nlm   1/1     Running   0          6m42s

Making services accessible over TCP

Accessing Pods and Deployment using port-forward

Now, we'd like to see nginx in action! The pod is listening on an internal port 80 (according to its configuration and the default nginx config). If we want to access this from outside the cluster, there's three ways:

• kubectl port-forward, which listens to a local port on the machine where you run kubectl port-forward, and then forwards all connections to a pod. (I've explained above how to configure kubectl so it can run on your own machine when the cluster runs elsewhere.)
• Creating a Service, in the next section
• Creating an Ingress, in the section after that

Let's try creating a port-forward to the pod first:

$ kubectl port-forward pod/nginx-deployment-58b6c946d5-fnqr6 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

Now, visit http://127.0.0.1:8080 in your browser and presto!

You can also do a port-forward to a deployment:

$ kubectl port-forward deployment/nginx-deployment 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

When you start a port-forward to a deployment, the deployment is resolved to a random one of the Running pods in the deployment. So, in this case, you'll get a response from either of the two pods; if one of them is down, you'll get it from the one that's up. (If both are down, you're out of luck.) However, when that resolved pod goes down for whatever reason, the port-forward is not restarted to another pod in the deployment. You can try this out by running kubectl delete pod nginx-deployment-... on your pods: the deployment will cause them to be restarted, but the port-forward will cease to work once you've deleted the right one.

Accessing a deployment using a Service

So, what should we do if we want the application to be reachable even if its Pods go down? The first method is to create a Service. A Service describes how an application should be accessible. There's multiple types of Services, corresponding with multiple interpretations of "accessible":

• ClusterIP is a service type indicating that the application should be only internally accessible using a "virtual service IP" (as described above). This service IP will be allocated by Kubernetes and distributed to all nodes and pods, so that a connection to the virtual service IP on the correct port will automatically end up on one of its running Pods.
• NodePort is a service type indicating that the application should be externally accessible using a "service port" on all Nodes. The service port will be allocated by Kubernetes (you can choose it, but that's not recommended) and distributed to all nodes, so that a connection to any node on the service port will automatically end up on one of its running Pods. A NodePort service also automatically gets a ClusterIP, so you can use that, too.
• LoadBalancer is a service type indicating that the application should be externally accessible using a provided load balancer. By default, this works like the NodePort but on specific cloud providers you'll also get an allocated external IP address, on which a wanted port is listening and end up on one of the running Pods. I'm running this on my own cluster, not one hosted by a cloud provider, so I won't create a LoadBalancer service. If you'd like to, this page explains how they work.
• ExternalName doesn't actually set up any forwarding, but allows you to register an internal name that forwards to a given name in DNS elsewhere. This allows migration to/from Kubernetes.
• Not a service type, but if your service uses HTTP, you can use Ingress instead of Service to make your service externally accessible. More on that later.

Since we want our service to be externally accessible, we'll make a NodePort service:

$ cat service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - name: nginx
      port: 80
      protocol: TCP
  type: NodePort
$ kubectl apply -f service.yaml
$ kubectl get services
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes      ClusterIP   10.16.0.1       <none>        443/TCP        8d
nginx-service   NodePort    10.16.247.178   <none>        80:31106/TCP   4s

When we define the Service, we provide a Selector which defines the Pods supplying this Service. You can supply arbitrary labels to Pods; the app=nginx label was initially set in our Deployment (see the metadata section) and is inherited by all Pods created by it, so the Service will use them directly.

As you can see, our nginx-service service is now up and it got external port 31106. Indeed, when we go to http://<IP>:31106/ (replacing <IP> with the IP of any of our nodes) we can see the page again! Also, when we kubectl delete pod ... arbitrary pods within the deployment, they are restarted automatically, and accesses to the external IP/port keep working as long as at least one pod is Running.

There is an alternative way to make a Service externally reachable that can be convenient: you can set an externalIP on a ClusterIP type Service and any Node with that IP will listen on the indicated Service Port -- but that comes with a big fat warning: it introduces a single point of failure into your Service, as it will be unreachable if that Node is down! Yet, it can be very convenient especially for bare-metal clusters, so I'll show you how to do it:

$ cat service-externalip.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service-externalip
spec:
  selector:
    app: nginx
  ports:
    - name: nginx
      port: 80
      protocol: TCP
  type: ClusterIP
  externalIPs:
  - "145.131.6.177" # Replace this with one of your node's IPs, of course!
$ kubectl apply -f service-externalip.yaml
$ kubectl get services
NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
kubernetes                 ClusterIP   10.16.0.1       <none>          443/TCP        19d
nginx-service              NodePort    10.16.247.178   <none>          80:31106/TCP   11d
nginx-service-externalip   ClusterIP   10.16.165.143   145.131.6.177   80/TCP         3s

Sure enough, if you visit your external IP on port 80, you should see the same page served by Nginx appear!

A summary so far

We've talked about:

• Nodes, the machines (usually physical or VMs) that together form the Kubernetes cluster
  • Master nodes are nothing special, except they (also) run Pods that together form the Kubernetes Control Plane
• Pods, the basic unit of scheduling; they run on Nodes and consist of at least one Container running an actual Docker Image
• Deployments, which are a way to tell Kubernetes to always have some type of Pod running
• Jobs, which are a way to tell Kubernetes to keep running some type of Pod until it finishes successfully
• Services, which are a way to make some application in Pods accessible over TCP (inside and/or outside the cluster)
• Volumes, which provide various kinds of storage to Pods
  • Persistent Volumes are provided by the cluster administrator to allow storage
  • Persistent Volume Claims claim such volumes for some user
  • Pods can have a Persistent Volume Claim attached to them, making the contents of the volume actually usable

Accessing a Deployment using an Ingress

Ingresses are like Services, but for HTTP only. This specialisation allows adding a number of additional features, such as having multiple applications behind one URL or hostname (e.g. micro-services), SSL termination and splitting load between different versions of the same service (canarying).

Ingress is currently in beta (v1beta1), meaning that the feature is well-tested and will continue to exist, but details may change. Consider this before using it in production.

Like LoadBalancer Services, creating an Ingress does not immediately change anything in the cluster. You need to have an Ingress Controller for anything to change in the cluster after you create an Ingress. There's many Ingress Controller plugins to choose from; I will try Traefik since it supports Let's Encrypt out of the box. (Some cloud providers may provide an Ingress Controller out of the box.)

First of all, we set up Traefik. For this, we'll need to create some service types we haven't seen before: service accounts, cluster role bindings and config maps. Bear with me for a bit while we set up Traefik:

$ cat traefik-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system
$ kubectl apply -f traefik-account.yaml
serviceaccount/traefik-ingress-controller created
clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created

Now we create a ConfigMap for Traefik's configuration:

$ cat traefik-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-configmap
  namespace: kube-system
data:
  traefik.toml: |
    defaultEntryPoints = ["http", "https"]
    insecureSkipVerify = true

    [entryPoints]
      [entryPoints.http]
        address = ":80"
      [entryPoints.https]
        address = ":443"
        [entryPoints.https.tls]
      [entryPoints.admin]
        address = ":8080"

    [kubernetes]
      [kubernetes.ingressEndpoint]
        publishedService = "kube-system/traefik-ingress-service-external"

    [api]
    entryPoint = "admin"
$ kubectl apply -f traefik-configmap.yaml
configmap/traefik-configmap created

That being done, we now start the Traefik deployment:

$ cat traefik.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
    name: traefik-ingress
    namespace: kube-system
    labels:
        k8s-app: traefik-ingress-lb
spec:
    replicas: 1
    selector:
        matchLabels:
            k8s-app: traefik-ingress-lb
    template:
        metadata:
            labels:
                k8s-app: traefik-ingress-lb
                name: traefik-ingress-lb
        spec:
            volumes:
            - name: traefik-configmap
              configMap:
                name: traefik-configmap
            serviceAccountName: traefik-ingress-controller
            terminationGracePeriodSeconds: 60
            containers:
            - image: traefik
              name: traefik-ingress-lb
              ports:
              - name: web
                containerPort: 80
              - name: https
                containerPort: 443
              - name: admin
                containerPort: 8080
              volumeMounts:
              - mountPath: "/config"
                name: "traefik-configmap"
              args:
              - --loglevel=INFO
              - --configfile=/config/traefik.toml
$ kubectl apply -f traefik.yaml
deployment.extensions/traefik-ingress created

What did this do?

• We created the service account and privileges Traefik needs to find Ingresses, Services and Endpoints.
• We created a ConfigMap, a hard-coded type of Volume that is commonly used to supply configuration inside Pods. This ConfigMap causes Traefik to listen on ports 80, 443 and 8080.
• Then, we created a Deployment that runs the Traefik image with the given configmap and service account.
• Note that you won't find these deployments and pods using the normal kubectl get pods (etc) commands unless you give -n kube-system to select the kube-system namespace.

You should see a traefik-ingress-... pod with status Running when you run kubectl get pods -n kube-system; if that's not the case, you should stop here and investigate what's wrong.

To use Traefik, we'll configure two things:

• External connections end up at it
• It reads the hostname and path of requests, and sends them onwards to the correct Service

The first thing we've already discussed before: it requires setting up a LoadBalancer Service if you're running on a cloud provider; if you're not, like me, you can set up a ClusterIP Service with an ExternalIP and the side-note of a single-point-of-failure applies here as well. (Note that we expose only ports 80 and 443, not 8080; this is the administrator port of Traefik.)

$ cat traefik-service-external.yaml
apiVersion: v1
kind: Service
metadata:
  name: traefik-ingress-service-external
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 443
      name: https
  externalIPs:
  - "145.131.8.75"

The externalIPs mentioned here should be the external IP of one of your Nodes. At this point you can also create a record in DNS to point to this IP address if you want; I created kubetest.sjorsgielen.nl IN A 145.131.8.75.

Having this set up should cause http://kubetest.sjorsgielen.nl/ to end up within Traefik. It will give a "404 page not found" result, as Traefik doesn't know about any Ingresses yet to forward your request to.

Creating your own images

So far, we've set up the standard container ubuntu:bionic. It's pulled from the Docker Hub at https://hub.docker.com/_/ubuntu. Docker Hub is a central registry for images. In the same way you can pull many images from there, such as the minimal Linux image alpine.

But, if we want to run our own Docker images inside Kubernetes, it will need to be able to pull them as well. This can be done by uploading our images to Docker Hub, but for our own experimentation, let's set up our own registry and plug Kubernetes into it.

To do: Set up Docker Registry. Push images to it. Start them in a pod.

To do

• Play with auto-HTTPS ingresses (e.g. Traefik)
• Kubectl set image for rolling release?
• Kubernetes Dashboard
• Attempt Kubernetes upgrade from 1.13 to 1.14
• Try getting information on a pod from inside it using the Kubernetes API
• Play with native cronjobs
• Play with Statefulset / Daemonset
• Security Contexts
  • Refuse pods with host networking
  • Refuse PVs with hostpath mounts
  • Allow K8s API communication from a pod, but only to receive information about itself
  • Basically: Make it impossible to root a node even with "broad" privileges on the Kubernetes API server
• Limiting pods in memory, CPU, I/O
• Limiting pods in network communication