AWS offers a great service called “Amazon Elastic Container Service for Kubernetes” (AWS EKS).
The setup guide can be found here: Offical AWS EKS getting started guide
If you overload such a cluster it easily happens that your Kubelet gets “Out of Memory” (OOM) errors and stops working.
Once the Kubelet is down you can see
kubectl get nodes that node is in state “NotReady”.
In addition if you describe your node
kubectl describe $NODE you can see the status description is: “System OOM encountered”.
If you look on your pods
kubectl get pods --all-namespaces you can see that pods are in state “Unknown” or in “NodeLost”.
Kubelet OOM errors should be avoided by all costs.
It causes to stop all pods on that node and its quite complicated for K8s to maintain high availability for applications in some cases.
For example for stateful sets with a single replica k8s cannot immediately move that pod to another node.
The reason is that k8s does not know how long the node with all its pods stays unavailable.
Therefore i like to share some best practice to avoid OOM problems in your AWS EKS clusters.
Recently, I stumbled on a situation where I wanted to add a couple of values to an OpenShift deployment configuration. Previously I had modified or added a single attribute in a yaml file with
oc patch. So I started to wonder whether it is possible to update multiple attributes with
oc patch as well. To get right to the result: Yes, it is possible. This article will show you which features
oc patch and likewise
kubectl patch really have, beside a simple modification of one attribute.
Kubernetes and OpenShift have a lot in common. Actually OpenShift is more or less Kubernetes with some additions. But what exactly is the difference?
It’s not so easy to tell as both products are moving targets. The delta changes with every release - be it of Kubernetes or OpenShift. I tried to find out and stumbled across a few blog posts here and there. But they all where based on not so recent versions - thus not really up-to-date.
So I took the effort to compare the most recent versions of Kubernetes and OpenShift. At the time of writing v1.13 of Kubernetes and v3.11 of OpenShift. I plan to update this article as new versions become available.
[Prometheus][prometheus] is a popular monitoring tool based on time series data. One of the strengths of Prometheus is its deep integration with [Kubernetes][kubernetes]. Kubernetes components provide Prometheus metrics out of the box, and Prometheus’s service discovery integrates well with dynamic deployments in Kubernetes.
There are multiple ways how to set up Prometheus in a Kubernetes cluster. There’s an official [Prometheus Docker image][promdock], so you could use that and create the Kubernetes YAML files from scratch (which according to Joe Beda is [not totally crazy][crazy]). There is also a [helm chart][helmchart]. And there is the [Prometheus Operator][promop], which is built on top of the CoreOS [operator framework][operator].
This blog post shows how to get the [Prometheus Operator][promop] up and running in a Kubernetes cluster set up with [kubeadm][kubeadm]. We use [Ansible][ansible] to automate the deployment.
[Kubeadm][kubeadm] is a basic toolkit that helps you bootstrap a simple [Kubernetes][kubernetes] cluster. It is intended as a [basis for higher-level deployment tools][kubeadm-scope], like [Ansible][ansible] playbooks. A typical Kubernetes cluster set-up with
kubeadm consists of a single Kubernetes master, which is the machine coordinating the cluster, and multiple Kubernetes nodes, which are the machines running the actual workload.
Dealing with node failure is simple: When a node fails, the master will detect the failure and re-schedule the workload to other nodes. To get back to the desired number of nodes, you can simply create a new node and add it to the cluster. In order to add a new node to an existing cluster, you first create a token on the master with
kubeadm token create, then you use that token on the new node to join the cluster with
Dealing with master failure is more complicated. Good news is: Master failure is not as bad as it sounds. The cluster and all workloads will continue running with exactly the same configuration as before the failure. Applications running in the Kubernetes cluster will still be usable. However, it will not be possible to create new deployments or to recover from node failures without the master.
This post shows how to backup and restore a Kubernetes master in a
This blog post shows how to use [CIFS] (a.k.a. SMB, Samba, Windows Share) network filesystems as [Kubernetes volumes].
Docker containers running in Kubernetes have an ephemeral file system: Once a container is terminated, all files are gone. In order to store persistent data in Kubernetes, you need to mount a [Persistent Volume] into your container. Kubernetes has built-in support for network filesystems found in the most common cloud providers, like [Amazon’s EBS], [Microsoft’s Azure disk], etc. However, some cloud hosting services, like the [Hetzner cloud], provide network storage using the CIFS (SMB, Samba, Windows Share) protocol, which is not natively supported in Kubernetes.
Fortunately, Kubernetes provides [Flexvolume], which is a plugin mechanism enabling users to write their own drivers. There are a few flexvolume drivers for CIFS out there, but for different reasons none of them seemed to work for me. So I wrote my own, which can be found on [github.com/fstab/cifs].
This blog post shows how to use the
fstab/cifs plugin for mounting CIFS volumes in Kubernetes.
Docker Headless VNC Container 1.3.0 has been released today. The different Docker images contains a complete VNC based, headless UI environment for testautomation like Sakuli does or simply for web browsing and temporary work in a throw-away UI container. The functionality is pretty near to a VM based image, but can be started in seconds instead of minutes. Each Docker image has therefore installed the following components:
Sowohl End-2-End-Testing als auch End-2-End-Monitoring folgen dem gleichen Paradigma – sie betrachten eine Applikation aus der Sicht des End-Users. Hier darf es keine Rolle spielen, in welcher Oberflächentechnologie die Applikation geschrieben ist oder in welcher Art sie mit dem End-User in Verbindung tritt. Genau an diesem Punkt setzt das Open-Source-Tool Sakuli an.
Getting started with Kubernetes can be intimidating at first. Installing Kubernetes is not the easiest of tasks and can get quite frustrating.[^1] Luckily, there is an out-of-the box distribution called Minikube which makes toying around with Kubernetes a bliss.
Kiel, 24 Grad, 50 Mann an Bord. Bei unerwartet schönstem Sommerwetter wurde in der Kieler Fachhochschule am 7. und 8. September der elfte Workshop der Monitoring-Community veranstaltet. Das ConSol-Monitoringteam trug mit acht Vorträgen zum Gelingen der Veranstaltung bei. Eine kurze Zusammenfassung:
Bereits mit dem erstem Vortrag nach der Begrüßung, “E2E-Monitoring mit Sakuli”, sorgte Simon Meggle für einen würdigen und technisch anspruchsvollen Auftakt der Veranstaltung. Die Möglichkeit, Sakuli in Docker-Containern einzusetzen und End-to-End-Tests somit praktisch beliebig zu parallelisieren, sorgte für viel Gesprächsstoff.
Damit es jeder zu Hause nachmachen kann, führte Simon dann am zweiten Tag die Teilnehmer in einer Live-Demo durch sein Tutorial “Sakuli-Tests im Docker-Container”.