How to setup Kubernetes Cluster using Cilium CNI Network Plugin
Here in this article we will be setting up kubernetes cluster using kubeadm. We will be using CRI-O as the container runtime implementation and cilium as the networking provider for implementing the networking infrastructure in the kubernetes cluster.
Test Environment
Fedora 36 server (kubemaster)
Fedora 36 server (kubenode)
Procedure
Step1: Ensure the below pre-requisite
As a first make sure the following kernel modules are loaded. These are required for the iptables to see the bridge traffic and for the usage union filesystem for the containers.
Master Node
[admin@kubemaster ~]$ cat /etc/modules-load.d/k8s.conf
overlay
br_netfilter
[admin@kubemaster ~]$ sudo modprobe overlay
[admin@kubemaster ~]$ sudo modprobe br_netfilter
[admin@kubemaster ~]$ lsmod | grep overlay
overlay 155648 0
[admin@kubemaster ~]$ lsmod | grep br_netfilter
br_netfilter 32768 0
bridge 352256 1 br_netfilter
[admin@kubemaster ~]$ cat /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
[admin@kubemaster ~]$ sudo sysctl --system
...
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
...Worker Node
[admin@kubenode ~]$ cat /etc/modules-load.d/k8s.conf
overlay
br_netfilter
[admin@kubenode ~]$ sudo modprobe overlay
[admin@kubenode ~]$ sudo modprobe br_netfilter
[admin@kubenode ~]$ lsmod | grep overlay
overlay 151552 0
[admin@kubenode ~]$ lsmod | grep br_netfilter
br_netfilter 32768 0
bridge 344064 1 br_netfilter
[admin@kubenode ~]$ cat /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
[admin@kubenode ~]$ sudo sysctl --system
...
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
...Step2: Install CRI-O Container Runtime
Container Runtime is the required component on each of the kubernetes cluster nodes (ie control plane or worker nodes) to run pods. Here we are going to install and use CRI-O container runtime as the runtime for kubernetes cluster.
Master Node
[admin@kubemaster ~]$ dnf module list cri-o
...
Fedora Modular 36 - x86_64 - Updates
Name Stream Profiles Summary
cri-o 1.20 default [d] Kubernetes Container Runtime Interface for OCI-based containers
cri-o 1.21 default [d] Kubernetes Container Runtime Interface for OCI-based containers
cri-o 1.22 default [d] Kubernetes Container Runtime Interface for OCI-based containers
cri-o 1.23 default [d] Kubernetes Container Runtime Interface for OCI-based containers
cri-o 1.24 default Kubernetes Container Runtime Interface for OCI-based containers
Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled
[admin@kubemaster ~]$ VERSION=1.24
[admin@kubemaster ~]$ sudo dnf module enable cri-o:$VERSION
[admin@kubemaster ~]$ sudo dnf install cri-o
...
Installed:
aardvark-dns-1.0.3-1.fc36.x86_64 conmon-2:2.1.4-2.fc36.x86_64 container-selinux-2:2.190.0-1.fc36.noarch containers-common-4:1-59.fc36.noarch
cri-o-1.24.0-1.module_f36+14441+e1130b47.x86_64 criu-3.17.1-2.fc36.x86_64 fuse-common-3.10.5-2.fc36.x86_64 fuse-overlayfs-1.9-1.fc36.x86_64
fuse3-3.10.5-2.fc36.x86_64 fuse3-libs-3.10.5-2.fc36.x86_64 libbsd-0.10.0-9.fc36.x86_64 libnet-1.2-5.fc36.x86_64
libslirp-4.6.1-3.fc36.x86_64 netavark-1.0.3-3.fc36.x86_64 runc-2:1.1.3-1.fc36.x86_64 slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
socat-1.7.4.2-2.fc36.x86_64 Worker Node
[admin@kubenode ~]$ dnf module list cri-o
...
Fedora Modular 36 - x86_64 - Updates
Name Stream Profiles Summary
cri-o 1.20 default [d] Kubernetes Container Runtime Interface for OCI-based containers
cri-o 1.21 default [d] Kubernetes Container Runtime Interface for OCI-based containers
cri-o 1.22 default [d] Kubernetes Container Runtime Interface for OCI-based containers
cri-o 1.23 default [d] Kubernetes Container Runtime Interface for OCI-based containers
cri-o 1.24 default Kubernetes Container Runtime Interface for OCI-based containers
[admin@kubenode ~]$ VERSION=1.24
[admin@kubenode ~]$ sudo dnf module enable cri-o:$VERSION
[admin@kubenode ~]$ sudo dnf install cri-o
...
Installed:
aardvark-dns-1.0.3-1.fc36.x86_64 conmon-2:2.1.4-2.fc36.x86_64 container-selinux-2:2.190.0-1.fc36.noarch containers-common-4:1-59.fc36.noarch
cri-o-1.24.0-1.module_f36+14441+e1130b47.x86_64 criu-3.17.1-2.fc36.x86_64 fuse-common-3.10.5-2.fc36.x86_64 fuse-overlayfs-1.9-1.fc36.x86_64
fuse3-3.10.5-2.fc36.x86_64 fuse3-libs-3.10.5-2.fc36.x86_64 libbsd-0.10.0-9.fc36.x86_64 libnet-1.2-5.fc36.x86_64
libslirp-4.6.1-3.fc36.x86_64 netavark-1.0.3-3.fc36.x86_64 runc-2:1.1.3-1.fc36.x86_64 slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
socat-1.7.4.2-2.fc36.x86_64
Complete!Step3: Check Network Adapters and Open Ports
Ensure the required ports for the kubernetes master node and worked node are available.
Master Node
[admin@kubemaster ~]$ sudo firewall-cmd --permanent --add-port=6443/tcp
[admin@kubemaster ~]$ sudo firewall-cmd --permanent --add-port=2379-2380/tcp
[admin@kubemaster ~]$ sudo firewall-cmd --permanent --add-port=10250/tcp
[admin@kubemaster ~]$ sudo firewall-cmd --permanent --add-port=10259/tcp
[admin@kubemaster ~]$ sudo firewall-cmd --permanent --add-port=10257/tcp
[admin@kubemaster ~]$ sudo systemctl restart firewalld
[admin@kubemaster ~]$ nc 127.0.0.1 6443
Ncat: Connection refused.Worker Node
[admin@kubenode ~]$ sudo firewall-cmd --permanent --add-port=10250/tcp
[admin@kubenode ~]$ sudo firewall-cmd --permanent --add-port=30000-32767/tcp
[admin@kubenode ~]$ sudo systemctl restart firewalld
[admin@kubenode ~]$ nc 127.0.0.1 6443
Ncat: Connection refused.Step4: Install Kubeadm, Kubelet and Kubectl
In this step we are going to setup the repository and install the kubeadm, kubectl and kubelet components as shown below.
Master Node
[admin@kubemaster ~]$ cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
[admin@kubemaster ~]$ sudo setenforce 0
[admin@kubemaster ~]$ sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
[admin@kubemaster ~]$ sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
...
Installed:
conntrack-tools-1.4.6-2.fc36.x86_64 containernetworking-plugins-1.1.0-1.fc36.x86_64 cri-tools-1.22.0-3.module_f36+14441+e1130b47.x86_64 kubeadm-1.25.2-0.x86_64
kubectl-1.25.2-0.x86_64 kubelet-1.25.2-0.x86_64 libnetfilter_cthelper-1.0.0-21.fc36.x86_64 libnetfilter_cttimeout-1.0.0-19.fc36.x86_64
libnetfilter_queue-1.0.5-2.fc36.x86_64
Complete!
[admin@kubemaster ~]$ sudo systemctl enable --now kubeletWorker Node
[admin@kubenode ~]$ cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
[admin@kubenode ~]$ sudo setenforce 0
[admin@kubenode ~]$ sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
[admin@kubenode ~]$ sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
...
Installed:
conntrack-tools-1.4.6-2.fc36.x86_64 containernetworking-plugins-1.1.0-1.fc36.x86_64 cri-tools-1.22.0-3.module_f36+14441+e1130b47.x86_64 kubeadm-1.25.2-0.x86_64
kubectl-1.25.2-0.x86_64 kubelet-1.25.2-0.x86_64 libnetfilter_cthelper-1.0.0-21.fc36.x86_64 libnetfilter_cttimeout-1.0.0-19.fc36.x86_64
libnetfilter_queue-1.0.5-2.fc36.x86_64
Complete!
[admin@kubenode ~]$ sudo systemctl enable --now kubeletNOTE: CRI-O by default uses systemd as the cgroup driver and from v1.22, if the user is not setting the cgroupDriver field under KubeletConfiguration, kubeadm will default it to systemd for kubelet.
Step5: Create Cluster
Disable Swap
Fedora now enables swap on ZRAM by default. To disable it permanently, we need to remove the package which generates its configuration and restart the managed nodes.
[admin@kubemaster ~]$ sudo dnf remove zram-generator-defaults
[admin@kubenode ~]$ sudo dnf remove zram-generator-defaultsAlso if we have any other swap mount available that can disabled as shown below.
[admin@kubemaster ~]$ sudo swapoff -a
[admin@kubemaster ~]$ sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
[admin@kubenode ~]$ sudo swapoff -a
[admin@kubenode ~]$ sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstabEnable CRI-O runtime
[admin@kubemaster ~]$ sudo systemctl start crio.service
[admin@kubemaster ~]$ sudo systemctl enable crio.service
[admin@kubenode ~]$ sudo systemctl start crio.service
[admin@kubenode ~]$ sudo systemctl enable crio.service Initialize Cluster using kubeadm on Master node
[admin@kubemaster ~]$ IPADDR="192.168.122.45"
[admin@kubemaster ~]$ NODENAME=$(hostname -s)
[admin@kubemaster ~]$ sudo kubeadm init --apiserver-advertise-address=$IPADDR --apiserver-cert-extra-sans=$IPADDR --pod-network-cidr=10.1.1.0/24 --node-name=$NODENAME
[sudo] password for admin:
[init] Using Kubernetes version: v1.25.2
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[WARNING FileExisting-tc]: tc not found in system path
[WARNING SystemVerification]: missing optional cgroups: blkio
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubemaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.122.45]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kubemaster localhost] and IPs [192.168.122.45 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kubemaster localhost] and IPs [192.168.122.45 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.506959 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node kubemaster as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node kubemaster as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 3otirl.zqvioyl8wz7iob7a
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.122.45:6443 --token 3otirl.zqvioyl8wz7iob7a \
--discovery-token-ca-cert-hash sha256:a762494c43228c18524398420cfd7cb8d8f2b49912aadba4e999099d08a94db9Configure the kubectl configuration
[admin@kubemaster ~]$ mkdir -p $HOME/.kube
[admin@kubemaster ~]$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[admin@kubemaster ~]$ sudo chown $(id -u):$(id -g) $HOME/.kube/configStep6: Install Cilium for Networking
Here we are going use Cilium as the networking provider for implementing the networking infrastructure in the kubernetes cluster.
Install cilium cli
[admin@kubemaster ~]$ CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
[admin@kubemaster ~]$ CLI_ARCH=amd64
[admin@kubemaster ~]$ if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
[admin@kubemaster ~]$ curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
[admin@kubemaster ~]$ sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
cilium-linux-amd64.tar.gz: OK
[admin@kubemaster ~]$ sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
cilium
[admin@kubemaster ~]$ rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}Install cilium
[admin@kubemaster ~]$ cilium install
ℹ️ Using Cilium version 1.12.2
🔮 Auto-detected cluster name: kubernetes
🔮 Auto-detected datapath mode: tunnel
🔮 Auto-detected kube-proxy has been installed
ℹ️ helm template --namespace kube-system cilium cilium/cilium --version 1.12.2 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
ℹ️ Storing helm values file in kube-system/cilium-cli-helm-values Secret
🔑 Created CA in secret cilium-ca
🔑 Generating certificates for Hubble...
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🚀 Creating ConfigMap for Cilium version 1.12.2...
🚀 Creating Agent DaemonSet...
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed and ready...
♻️ Restarting unmanaged pods...
♻️ Restarted unmanaged pod kube-system/coredns-565d847f94-759h2
✅ Cilium was successfully installed! Run 'cilium status' to view installation healthCheck cilium status
[admin@kubemaster ~]$ cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Hubble: disabled
\__/¯¯\__/ ClusterMesh: disabled
\__/
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 1
cilium-operator Running: 1
Cluster Pods: 2/2 managed by Cilium
Image versions cilium quay.io/cilium/cilium:v1.12.2@sha256:986f8b04cfdb35cf714701e58e35da0ee63da2b8a048ab596ccb49de58d5ba36: 1
cilium-operator quay.io/cilium/operator-generic:v1.12.2@sha256:00508f78dae5412161fa40ee30069c2802aef20f7bdd20e91423103ba8c0df6e: 1Check the Pod status
[admin@kubemaster ~]$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-operator-bc4d5b54-s6f79 1/1 Running 0 2m39s
kube-system cilium-sgr95 1/1 Running 0 2m39s
kube-system coredns-565d847f94-bl8ps 0/1 Running 0 74s
kube-system coredns-565d847f94-fm47f 0/1 Running 0 82s
kube-system etcd-kubemaster 1/1 Running 0 9m34s
kube-system kube-apiserver-kubemaster 1/1 Running 0 9m34s
kube-system kube-controller-manager-kubemaster 1/1 Running 0 9m34s
kube-system kube-proxy-fq2v9 1/1 Running 0 9m20s
kube-system kube-scheduler-kubemaster 1/1 Running 0 9m34sStep7: Join the Worker Node
As a last step we will be joining the worker node to the kubernetes cluster as shown below.
[admin@kubenode ~]$ sudo kubeadm join 192.168.122.45:6443 --token udwwv0.vocoxu7r79m8c53d --discovery-token-ca-cert-hash sha256:a762494c43228c18524398420cfd7cb8d8f2b49912aadba4e999099d08a94db9 --node-name kubenodeWe can also generate the command for joining the worker node to cluster at a later stage by using the below command on master node to regenerate the join command.
Regeneration Join token
[admin@kubemaster ~]$ sudo kubeadm token create --print-join-command
kubeadm join 192.168.122.45:6443 --token udwwv0.vocoxu7r79m8c53d --discovery-token-ca-cert-hash sha256:a762494c43228c18524398420cfd7cb8d8f2b49912aadba4e999099d08a94db9In case you find any issue with the kubernetes pods not able to resolve the DNS names of the service or pod. You can try to disable the firewall completely.
[admin@kubemaster ~]$ sudo systemctl stop firewalld
[admin@kubemaster ~]$ sudo systemctl disable firewalld
[admin@kubenode ~]$ sudo systemctl stop firewalld
[admin@kubenode ~]$ sudo systemctl disable firewalldHope you enjoyed reading this article. Thank you..
Leave a Reply
You must be logged in to post a comment.