No pod to pod communication on CentOS 8, Kubernetes with Calico

When I build clusters I have one goal — make it secure. And I usually prefer to get something ready before I build it completely on my own. The reasoning for the latter is simple — behind ready products is always the community. And I am for sure not as smart and as fast as the whole community. It is usually much better to fetch something ready and modify it slightly to suit your needs. Even better — your modifications can always be played back to the community and enhance the product. Therefore my first-stop when building a cluster is RedHat OpenShift — even if it is in its community version — OKD. It’s just there and comes with so many important features — SELinux enabled, Fedora CoreOS, multi-tenancy support, pod isolation, integrated registry, operator hub, nice GUI, BuildConfigs and ImageSets, etc.

Of course, having all of this costs — compute resources. Not every organization can afford to spawn a full blown OpenShift cluster with 3 master and 3 worker nodes, each of them with at least 4 vCPUs and 16 GB RAM. Sometimes people need less. Therefore I went back to the basics for one of the last clusters I built privately — vanilla Kubernetes on a single node. It’s not for productive use, but for learning purposes, so it needed to be cheap and efficient. An option was even to go to K3s, but we decided to stick to vanilla Kubernetes.

My criteria:

• SELinux enabled;
• As secure as possible;
• CRI-O;
• Network policies for namespace isolation;
• eBPF;
• HAproxy ingress controller for Layer 4 passthrough communication.

The answers to these points were: vanilla Kubernetes with Calico CNI with activated eBPF and CentOS Stream 8 with CRI-O. CentOS Stream 8 is a stable, free distribution from the RedHat family, which comes with a lot of security in mind (it just has a very good basic setup on SELinux). It leaves almost nothing to touch on it and you can still sleep calmly in the night, knowing this layer of your system is well secured.

The installation and setup of CentOS 8, Kubernetes and Calico was as straight as possible — just follow the tutorials.

Once we installed the HAproxy ingress controller and the first application, there we noticed the problems — HAproxy was unable to establish connection to the pod of the application. Communication from pods to services worked, but communication to the outer world and from pod to pod always failed.

We started digging and after several hours of reading, we stumbled upon this issue on GitHub — Networking: Pods are unable to communicate with: other pods, the API Server and the outside world. Reading through it, it forwarded us to the following issue on GitHub — kube-proxy currently incompatible with `iptables >= 1.8`.

To summarize the problem — back in time “iptables” was the firewall of the day on every Linux system. Over the years people got used to it, its cumbersome command-line syntax, its tricks, its problems and all the good and the bad it comes with. In the last couple of years there’s a movement to improve the firewall system in the Linux kernel (and to combine several features of it distributed across 4 different systems into one). This new system is called “nftables”. A great blog post about it written by Eric Garver is this one — Using nftables in Red Hat Enterprise Linux 8.

In order to ease the transition from “iptables” to “nftables” the RedHat family of distributions still have the “iptables” command and it is still used by a lot of software around — even by Kubernetes. The problem is — this doesn’t modify the “iptables”-parts of the Linux kernel, but the “nftables”-parts of it. Some distributions offer “iptables-legacy”, with which you could modify the old “iptables”-structures. This is not the case for CentOS Stream 8. A good blog post about the two structures and their differences is “iptables: The two variants and their relationship with nftables” by … let’s guess — Eric Garver.

Digging deeper into the problem we found the following issue on GitHub — Calico autoselects iptables-legacy mode if ip_tables.ko preloaded on CentOS 8. At least there were good news —setting the environment variable FELIX_IPTABLESBACKEND to “NFT” on the “calico-node” DaemonSet’s pods would force Calico to use “nftables” instead of “iptables”.

So the solution is as follows:

• Create a file called “/etc/modprobe.d/10-blacklist-iptables.conf” with the following line: “blacklist ip_tables”;
• Run “dracut -f”;
• Reboot the machine;
• Modify the DaemonSet “calico-node” by adding an environment variable to the “calico-node” container:

- name: FELIX_IPTABLESBACKEND
value: NFT

Kill all your “calico-node” pods (just to be sure) and wait for them to restart. That’s it — now you can be sure your Kubernetes system uses “nftables” instead of “iptables”.

Disclaimer

While I am not a representative of Red Hat and my views about Red Hat are my own, I am a member of Red Hat Accelerators community which gives me a connection to Red Hat and through which I engage with other Red Hat Accelerators.
Learn more about the Red Hat Accelerators