How to manage Kubernetes cluster using Gitlab CICD workflow

Gitlab can be used to manage infrastructure as code. Its integration with Kubernetes helps to install, configure, manage, deploy, and troubleshoot cluster applications.
Gitlab can be connected to the Kubernetes cluster using the Gitlab agent for Kubernetes.

Table of Contents

Test Environment

fedres.stack.com – gitlab v16.9.2-ee
k8master.stack.com – master node – v1.28.5
k8node.stack.com – worker node – v1.28.5

What is GitLab agent server for Kubernetes (kas)

The GitLab-side component of GitLab that handles operations and logic for the Kubernetes agent integration. Manages the connection and communication between GitLab and Kubernetes clusters.

What is agentk

The cluster-side component that maintains a secure connection to GitLab for Kubernetes management and deployment automation.

What is Push-based deployment

A deployment method where updates are sent from GitLab CI/CD pipelines to the Kubernetes cluster.

For more details, please follow Gitlab documentation on “Manage your infrastructure“.

If you are interested in watching the video. Here is the YouTube video on the same step by step procedure outlined below.

Procedure

Step1: Ensure a working Kubernetes cluster

We need to have a working kubernetes cluster for us to manage the cluster using gitlab cicd. Here i have a cluster with 1 master and 1 worker node as shown below.

admin@k8master:~$ kubectl get nodes
NAME                 STATUS   ROLES           AGE    VERSION
k8master.stack.com   Ready    control-plane   292d   v1.28.5
k8node.stack.com     Ready    <none>          292d   v1.28.5

Step2: Ensure a working gitlab server with gitlab runner

Next we need to ensure that we have a working gitlab server with gitlab runner which can be used to execute the gitlab cicd workflow. In my case i have a self managed secured gitlab server installed with self signed certificate.

URL - https://fedres.stack.com

admin@fedres:~$ sudo gitlab-ctl status
run: alertmanager: (pid 1131) 0s; run: log: (pid 1116) 0s
run: gitaly: (pid 1083) 0s; run: log: (pid 1078) 0s
run: gitlab-exporter: (pid 1129) 0s; run: log: (pid 1119) 0s
run: gitlab-kas: (pid 1086) 0s; run: log: (pid 1082) 0s
run: gitlab-pages: (pid 1135) 0s; run: log: (pid 1122) 0s
run: gitlab-workhorse: (pid 1118) 0s; run: log: (pid 1102) 0s
run: logrotate: (pid 1101) 0s; run: log: (pid 1091) 0s
run: nginx: (pid 1093) 0s; run: log: (pid 1092) 0s
run: node-exporter: (pid 1130) 0s; run: log: (pid 1121) 0s
run: postgres-exporter: (pid 1132) 0s; run: log: (pid 1115) 0s
run: postgresql: (pid 1085) 0s; run: log: (pid 1080) 0s
run: prometheus: (pid 1134) 0s; run: log: (pid 1133) 0s
run: puma: (pid 1097) 0s; run: log: (pid 1089) 0s
run: redis: (pid 1081) 0s; run: log: (pid 1077) 0s
run: redis-exporter: (pid 1120) 0s; run: log: (pid 1114) 0s
run: sidekiq: (pid 1094) 0s; run: log: (pid 1087) 0s

admin@fedres:~$ sudo gitlab-runner status
Runtime platform                                    arch=amd64 os=linux pid=1827 revision=b92ee590 version=17.4.0
gitlab-runner: Service is running

Step3: Setup Gitlab Kubernetes agent server

In an RPM based installation method for self managed Gitlab server by default, the agent server is installed, enabled and available at wss://fedres.stack.com/-/kubernetes-agent/ where “fedres.stack.com” is FQDN for my gitlab server.

Navigate to “/var/log/gitlab/gitlab-kas” to check the current logs for more details.

root@fedres:/var/log/gitlab/gitlab-kas# tail -f current 
2024-10-22_05:59:33.99747 {"level":"info","time":"2024-10-22T11:29:33.997+0530","msg":"Running KAS gitlab-kas/v16.9.2/d5b98591"}
2024-10-22_05:59:34.01609 {"level":"info","time":"2024-10-22T11:29:34.015+0530","msg":"Using own private API URL","url":"grpc://localhost:8155"}
2024-10-22_05:59:34.03582 {"level":"info","time":"2024-10-22T11:29:34.035+0530","msg":"Private API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8155"}
2024-10-22_05:59:34.03685 {"level":"info","time":"2024-10-22T11:29:34.036+0530","msg":"Kubernetes API endpoint is up","mod_name":"kubernetes_api","net_network":"tcp","net_address":"127.0.0.1:8154"}
2024-10-22_05:59:34.04023 {"level":"info","time":"2024-10-22T11:29:34.040+0530","msg":"API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8153"}
2024-10-22_05:59:34.04050 {"level":"info","time":"2024-10-22T11:29:34.040+0530","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"127.0.0.1:8151"}
2024-10-22_05:59:34.04440 {"level":"info","time":"2024-10-22T11:29:34.044+0530","msg":"Agentk API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8150","is_websocket":true}

Ensure that you copy the self sign certificate on the gitlab server to “/opt/gitlab/embedded/ssl/certs/” and enable the following setting in gitlab.rb and reconfigure the gitlab cluster. This is to ensure that the KAS server is able to communication with the Kubernetes agent.

root@fedres:~# ls -ltr /opt/gitlab/embedded/ssl/certs/
total 224
-rw-r--r--. 1 root root 224369 Mar  6  2024 cacert.pem
-rw-r--r--. 1 root root    147 Mar  9  2024 README
lrwxrwxrwx. 1 root root     32 Sep 30 18:51 bf8674ac.0 -> /etc/gitlab/trusted-certs/ca.crt
lrwxrwxrwx. 1 root root     46 Oct 20 12:29 e2b33be7.0 -> /etc/gitlab/trusted-certs/fedres.stack.com.pem

root@fedres:/etc/gitlab# cat gitlab.rb 
...
gitlab_kas['env'] = {
   'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/"
 }
...

root@fedres:/etc/gitlab# gitlab-ctl reconfigure

Step4: Create an agent configuration file for a repository

Here we are going to create a new blank project repository for this demo named “kubernetes_iac” in the user “dev” namespace as shown below.

For configuration settings, the agent uses a YAML file in the GitLab project. In the repository, in the default branch, create an agent configuration file at “.gitlab/agents/fedreskub/config.yaml” as shown below and push the changes. Unprotect the main branch for this demo to push changes to main branch repository.

admin@fedres:~$ git clone git@fedres.stack.com:dev/kubernetes_iac.git
admin@fedres:~$ cd kubernetes_iac
admin@fedres:~/kubernetes_iac$ git switch --create main
admin@fedres:~/kubernetes_iac$ touch README.md
admin@fedres:~/kubernetes_iac$ git add README.md
admin@fedres:~/kubernetes_iac$ git commit -m "add README"
admin@fedres:~/kubernetes_iac$ git push --set-upstream origin main

admin@fedres:~/kubernetes_iac$ mkdir -p .gitlab/agents/fedreskub/
admin@fedres:~/kubernetes_iac$ touch .gitlab/agents/fedreskub/config.yaml
admin@fedres:~/kubernetes_iac$ git add .gitlab/
admin@fedres:~/kubernetes_iac$ git commit -m "create gitlab agent config file"
admin@fedres:~/kubernetes_iac$ git push -u origin main

We can leave the file blank for now, and configure it later.

Step5: Register the agent with Gitlab

We need to first register an agent before you can install the agent in your cluster. To register an agent.

Go to hellomaven2 project
Select Operate > Kubernetes clusters
Select Connect a cluster (agent)
As we already have an agent configuration file created, we can select it from the list. Select “fedreskub” from the dropdown list.
Click on Register an agent

GitLab generates an access token for the agent along with the helm command to install the agent on the cluster. You need this token to install the agent in your cluster.

Here is the sample output in the pop up window.

Agent access token:

glagent-V9cpgnKqoCpApR41HQU8zfgc4n3WQNioUH3i4PPjZb9E3tnHyQ

The agent uses the token to connect with GitLab.

You cannot see this token again after you close this window.
Install using Helm (recommended)

From a terminal, connect to your cluster and run this command. The token is included in the command. Use a Helm version compatible with your Kubernetes version (see Helm version support policy ).

helm repo add gitlab https://charts.gitlab.io
helm repo update
helm upgrade --install fedreskub gitlab/gitlab-agent \
    --namespace gitlab-agent-fedreskub \
    --create-namespace \
    --set image.tag=v16.9.2 \
    --set config.token=glagent-V9cpgnKqoCpApR41HQU8zfgc4n3WQNioUH3i4PPjZb9E3tnHyQ \
    --set config.kasAddress=wss://fedres.stack.com/-/kubernetes-agent/

Step6: Install Gitlab agent for kubernetes on the cluster

First we need to ensure that helm is installed on our kubernets cluster.

admin@k8master:~$ sudo dnf install helm

We can go ahead and install the agent on cluster using the provided commands in the output above.

admin@k8master:~$ helm repo add gitlab https://charts.gitlab.io
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/admin/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/admin/.kube/config
"gitlab" has been added to your repositories

admin@k8master:~$ helm repo update
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/admin/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/admin/.kube/config
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "gitlab" chart repository
Update Complete. ⎈Happy Helming!⎈

admin@k8master:~$ helm upgrade --install fedreskub gitlab/gitlab-agent \
    --namespace gitlab-agent-fedreskub \
    --create-namespace \
    --set image.tag=v16.9.2 \
    --set config.token=glagent-V9cpgnKqoCpApR41HQU8zfgc4n3WQNioUH3i4PPjZb9E3tnHyQ \
    --set config.kasAddress=wss://fedres.stack.com/-/kubernetes-agent/
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/admin/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/admin/.kube/config
Release "fedreskub" does not exist. Installing it now.
NAME: fedreskub
LAST DEPLOYED: Sat Oct 19 14:07:02 2024
NAMESPACE: gitlab-agent-fedreskub
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing gitlab-agent.

Your release is named fedreskub.

## Changelog

### 1.17.0

- The default replica count has been increased from `1` to `2` to allow a zero-downtime upgrade experience.
  You may use `--set replicas=1` to restore the old default behavior.

Check the status of the agent pods in new namespace “gitlab-agent-fedreskub”.

admin@k8master:~$ k get pods -n gitlab-agent-fedreskub
NAME                                         READY   STATUS    RESTARTS   AGE
fedreskub-gitlab-agent-v2-59d57f9bcb-76wzw   1/1     Running   3          47h
fedreskub-gitlab-agent-v2-59d57f9bcb-z6n4n   1/1     Running   3          47h

If you have your KAS server behind a self-signed certificate, you might encounter the following errors in your gitlab agent pods.

{"level":"error","time":"2024-10-20T03:57:54.711Z","msg":"Failed to register agent pod. Please make sure the agent version matches the server version","mod_name":"agent_registrar","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to WebSocket dial: failed to send handshake request: Get \\\"https://fedres.stack.com/-/kubernetes-agent/\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority\""}

In this case you can uninstall the agent deployment.

admin@k8master:~$ helm uninstall fedreskub -n gitlab-agent-fedreskub

Then reinstall with the value of config.kasCaCert set to the self sign certificate or CA cert chain for the self sign certificate. Here is the updated helm command that we need to execute. Ensure that your gitlab self sign certificate is present at the location specified in the field “config.kasCaCert” as shown below.

admin@k8master:~$ helm upgrade --install fedreskub gitlab/gitlab-agent \
    --namespace gitlab-agent-fedreskub \
    --create-namespace \
    --set image.tag=v16.9.2 \
    --set config.token=glagent-V9cpgnKqoCpApR41HQU8zfgc4n3WQNioUH3i4PPjZb9E3tnHyQ \
    --set config.kasAddress=wss://fedres.stack.com/-/kubernetes-agent/ \
    --set-file config.kasCaCert=/home/admin/fedres.stack.com.pem

Step7: Validate the agent status

The agent opens a bidirectional channel to KAS for communication. This channel is used for all communication between the agent and KAS.

Step8: Update gitlab cicd workflow

Here in this step we will update the .gitlab-ci.yml pipeline with deploy stage for execute. In this stage we first configure the kubectl client to use a CA certificate when connecting to the cluster. We need to convert our “fedres.stack.com.pem” self sign certificate into base64 format and use in the yml file.

admin@fedres:~$ cat /home/admin/fedres.stack.com.pem | base64 -i | tr -d '\n'
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ3RENDQXRTZ0F3SUJBZ0lVWWZOR1IyS3p5OUtvcjNxOEkva1pBeDhZRVlzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1p6RUxNQWtHQTFVRUJoTUNTVTR4Q3pBSkJnTlZCQWdNQWsxSU1ROHdEUVlEVlFRSERBWk5kVzFpWVdreApEakFNQmdOVkJBb01CWE4wWVdOck1ROHdEUVlEVlFRTERBWmtaWFp2Y0hNeEdUQVhCZ05WQkFNTUVHWmxaSEpsCmN5NXpkR0ZqYXk1amIyMHdIaGNOTWpRd09UTXdNVEEwTXpNNVdoY05Nall3T1RNd01UQTBNek01V2pCbk1Rc3cKQ1FZRFZRUUdFd0pKVGpFTE1Ba0dBMVVFQ0F3Q1RVZ3hEekFOQmdOVkJBY01CazExYldKaGFURU9NQXdHQTFVRQpDZ3dGYzNSaFkyc3hEekFOQmdOVkJBc01CbVJsZG05d2N6RVpNQmNHQTFVRUF3d1FabVZrY21WekxuTjBZV05yCkxtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNb1EzQklBOFY4MjNiRGMKMFkyRmJad1FCOTNycXRtZU5iWDRKZ0t6dmhVK280NlRxRm9FZGRQY2g1WmdiSmNrSGZMZEZJSm1hQjBFLzFKaQpPS0pQMjdZd1RLdHpZRTBKK2dFVmoxaFVFYUdFbGhvY0p5SHYrK2drcEd1R0RORUc1Ykc3b0s1emcrbjJyVjVuCmtKUXBGZU5KYkU0ZkJVWHAvZm9Zd2FVSGJwUC9ESWNSMzZKN08vU040djNZaXRTeWRRcHdTS09kekJmNU1yelYKNXFWT2hRa1pXT3hvZEdQeUtoSm5lYXZaSGVZa3RCWndlTXNLcnFZNUVTVG9rSUVvSHI4dDVaRFp5eHVCZlhSZApaSkFHNDhGWVFneVRKYjBDd0pHK3luVm5QUFRhYkFXZXRMVHVlRzFTMFlsMGp2SkpxbzdnSVJVNlpsNGlrckZZClNyckpOcU1DQXdFQUFhT0JqekNCakRBTEJnTlZIUThFQkFNQ0JMQXdIUVlEVlIwbEJCWXdGQVlJS3dZQkJRVUgKQXdFR0NDc0dBUVVGQndNQ01EOEdBMVVkRVFRNE1EYUNFR1psWkhKbGN5NXpkR0ZqYXk1amIyMkNEM0JoWjJWegpMbk4wWVdOckxtTnZiWUlSS2k1d1lXZGxjeTV6ZEdGamF5NWpiMjB3SFFZRFZSME9CQllFRlBtMHh6c3lTOXJuCktTMEEyeXE0UHFBUk9SaEZNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUFYZ0tJbWNqcGFNUTlBdm0veHZpejQKNDVFZGtJdy9nODBYVUlyQkpDNWdRbE1lRUFoK2MzUTduaVJldXNKMlhwd0doT0VyZmp5R3NiK3dWQkh4bmMvRgo2VXNFL3lUMDY3NlF0S0g5eWhSTjA2di9HZ2tEenZNbVBvUGlHOFIwVUx5QjFPVWpYS0tsNjBNaTZUQy9FM1IyClo4SndpRzd1VG5yNGh2WVV4aVFoVE5MdkJYdUdoM1NrUjNoVmVZWktkZTlBTzlRRmorcERBTTNuZEZqMUZNY1MKQzF2ekhzdFVNT2hxSHNCZXFMbkI3NHp5NmV6V3hGWEZLMkdDcTlXRm9RSG90cDJxQ3pCeEpiU3psdXN1bm5MSgpZaGdVTGtKS1R6Y1UzVlh3SExrRldaNFJZZ3Vmc3ozRGc1ZGFwUmpoTzhuSXVJb29CNWdnamFtWEVNRlBqNFh6Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

Once that is done we are setting the kubectl context to “dev/hellomaven2:fedreskub” which is the context related to our registered agent.

As in my case i have a gitlab runner with shell executor, we need to install the kubectl tool on the gitlab runner server for the gitlab cicd pipeline to use in the workflow. Run the following curl command to install kubectl on gitlab runner server.

admin@fedres:~$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

Now we are ready to manage our kubernetes cluster from gitlab. Here we are listing the pods in out kubernetes cluster default namespace.

admin@fedres:~/kubernetes_iac$ cat .gitlab-ci.yml 
#variables:
#  MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
#  MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
#
#cache:
#  paths:
#    - .m2/repository/
#    - target/

stages:
  - deploy

deploy:
  stage: deploy
  script:
    - kubectl config set clusters.gitlab.certificate-authority-data LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ3RENDQXRTZ0F3SUJBZ0lVWWZOR1IyS3p5OUtvcjNxOEkva1pBeDhZRVlzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1p6RUxNQWtHQTFVRUJoTUNTVTR4Q3pBSkJnTlZCQWdNQWsxSU1ROHdEUVlEVlFRSERBWk5kVzFpWVdreApEakFNQmdOVkJBb01CWE4wWVdOck1ROHdEUVlEVlFRTERBWmtaWFp2Y0hNeEdUQVhCZ05WQkFNTUVHWmxaSEpsCmN5NXpkR0ZqYXk1amIyMHdIaGNOTWpRd09UTXdNVEEwTXpNNVdoY05Nall3T1RNd01UQTBNek01V2pCbk1Rc3cKQ1FZRFZRUUdFd0pKVGpFTE1Ba0dBMVVFQ0F3Q1RVZ3hEekFOQmdOVkJBY01CazExYldKaGFURU9NQXdHQTFVRQpDZ3dGYzNSaFkyc3hEekFOQmdOVkJBc01CbVJsZG05d2N6RVpNQmNHQTFVRUF3d1FabVZrY21WekxuTjBZV05yCkxtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNb1EzQklBOFY4MjNiRGMKMFkyRmJad1FCOTNycXRtZU5iWDRKZ0t6dmhVK280NlRxRm9FZGRQY2g1WmdiSmNrSGZMZEZJSm1hQjBFLzFKaQpPS0pQMjdZd1RLdHpZRTBKK2dFVmoxaFVFYUdFbGhvY0p5SHYrK2drcEd1R0RORUc1Ykc3b0s1emcrbjJyVjVuCmtKUXBGZU5KYkU0ZkJVWHAvZm9Zd2FVSGJwUC9ESWNSMzZKN08vU040djNZaXRTeWRRcHdTS09kekJmNU1yelYKNXFWT2hRa1pXT3hvZEdQeUtoSm5lYXZaSGVZa3RCWndlTXNLcnFZNUVTVG9rSUVvSHI4dDVaRFp5eHVCZlhSZApaSkFHNDhGWVFneVRKYjBDd0pHK3luVm5QUFRhYkFXZXRMVHVlRzFTMFlsMGp2SkpxbzdnSVJVNlpsNGlrckZZClNyckpOcU1DQXdFQUFhT0JqekNCakRBTEJnTlZIUThFQkFNQ0JMQXdIUVlEVlIwbEJCWXdGQVlJS3dZQkJRVUgKQXdFR0NDc0dBUVVGQndNQ01EOEdBMVVkRVFRNE1EYUNFR1psWkhKbGN5NXpkR0ZqYXk1amIyMkNEM0JoWjJWegpMbk4wWVdOckxtTnZiWUlSS2k1d1lXZGxjeTV6ZEdGamF5NWpiMjB3SFFZRFZSME9CQllFRlBtMHh6c3lTOXJuCktTMEEyeXE0UHFBUk9SaEZNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUFYZ0tJbWNqcGFNUTlBdm0veHZpejQKNDVFZGtJdy9nODBYVUlyQkpDNWdRbE1lRUFoK2MzUTduaVJldXNKMlhwd0doT0VyZmp5R3NiK3dWQkh4bmMvRgo2VXNFL3lUMDY3NlF0S0g5eWhSTjA2di9HZ2tEenZNbVBvUGlHOFIwVUx5QjFPVWpYS0tsNjBNaTZUQy9FM1IyClo4SndpRzd1VG5yNGh2WVV4aVFoVE5MdkJYdUdoM1NrUjNoVmVZWktkZTlBTzlRRmorcERBTTNuZEZqMUZNY1MKQzF2ekhzdFVNT2hxSHNCZXFMbkI3NHp5NmV6V3hGWEZLMkdDcTlXRm9RSG90cDJxQ3pCeEpiU3psdXN1bm5MSgpZaGdVTGtKS1R6Y1UzVlh3SExrRldaNFJZZ3Vmc3ozRGc1ZGFwUmpoTzhuSXVJb29CNWdnamFtWEVNRlBqNFh6Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    - kubectl config view
    - kubectl config get-contexts
    - kubectl config use-context dev/kubernetes_iac:fedreskub
    - kubectl get pods
    #- kubectl --insecure-skip-tls-verify get pods

Step9: Validate the gitlab cicd pipeline execution

Once we push the gitlab cicd pipeline changes to the repository and execute the pipeline we should be able to see the list of pods in our kubernetes default namespace as shown below.

Hope you enjoyed reading this article. Thank you..

How to manage Kubernetes cluster using Gitlab CICD workflow