How to setup Gitlab EE server on Fedora using ansible

CICD

How to setup Gitlab EE server on Fedora using ansible

By novicejava1 0
gitlab_install_configure

Here in this article we will try to install and configure GitLab EE server using ansible on a Fedora OS. We will also to try to update the default password for the root user and secure the GitLab instance using self signed TLS certificate and key.

Test Environment

Fedora 39 server

As per the “Download and Install GitLab“, Fedora OS is not the offically supported OS for installing the GitLab server as it does not have a LTS version of the OS release. But it does support RHEL 8 and 9 servers and also its derivative OS AlmaLinux. So we will be trying to use the RPM package repositories for these OS to install GitLab on the Fedora OS.

If you are interested in watching the video. Here is the YouTube video on the same step by step procedure outlined below.

Here is the project structure for GitLab EE server setup.

admin@fedser:gitlabserver$ tree .
.
├── inventory
│   ├── hosts
├── linux_setup_gitlabserver.yml
├── README.md
└── roles
    ├── linux_configure_gitlabserver
    │   ├── defaults
    │   │   └── main.yml
    │   ├── tasks
    │   │   └── main.yml
    │   └── templates
    │       ├── fedres.stack.com.crt
    │       ├── fedres.stack.com.key
    │       ├── gitlab.rb
    │       └── sslcert.conf
    ├── linux_expose_gitlabserver
    │   └── tasks
    │       └── main.yml
    ├── linux_install_gitlabserver
    │   └── tasks
    │       └── main.yml
    ├── linux_ping
    │   └── tasks
    │       └── main.yml
    ├── linux_restart_gitlabserver
    │   └── tasks
    │       └── main.yml
    ├── linux_start_gitlabserver
    │   └── tasks
    │       └── main.yml
    └── linux_stop_gitlabserver
        └── tasks
            └── main.yml

NOTE: Role “linux_ping” can be used to validate the ssh connectivity with the managed hosts as per the inventory file. Here are the details.

admin@fedser:gitlabserver$ cat roles/linux_ping/tasks/main.yml 
- name: ansible ping pong validation
  ping:
admin@fedser:gitlabserver$ cat inventory/hosts
[gitlabserver]
fedres.stack.com

Procedure

Step1: Install GitLab server

The following role “linux_install_gitlabserver” will setup the GitLab EE server repository and accept the gpg key for the repository. Once repository setup is done it will carry out the GitLab EE server package installation as shown below.

admin@fedser:gitlabserver$ cat roles/linux_install_gitlabserver/tasks/main.yml 
---
- name: add gitlab ee server repository
  yum_repository:
    name: gitlabee
    description: gitlabee-repo
    baseurl: https://packages.gitlab.com/gitlab/gitlab-ee/el/9/$basearch
    gpgkey: https://packages.gitlab.com/gitlab/gitlab-ee/gpgkey
            https://packages.gitlab.com/gitlab/gitlab-ee/gpgkey/gitlab-gitlab-ee-3D645A26AB9FBD22.pub.gpg
    gpgcheck: true
    repo_gpgcheck: true
    enabled: true
    sslverify: true
    sslcacert: /etc/pki/tls/certs/ca-bundle.crt

- name: install pre-requisites
  dnf:
    name:
      - curl
      - policycoreutils
      - openssh-server
      - perl
    state: present

- name: install gitlab ee server
  dnf:
    name: gitlab-ee
    state: present

Step2: Configure GitLab Server

We first need to generate the self signed certificate to secure our GitLab server. Let us first create a sslcert.conf file which will be to request for a self signed certificate as shown below.

admin@fedser:gitlabserver$ cat roles/linux_configure_gitlabserver/templates/sslcert.conf 
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = IN
ST = MH
L = Mumbai
O = stack
OU = devops
CN = fedres.stack.com
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = fedres.stack.com

Now we are ready to generated our self signed certificate using the openssl command as shown below.

# Generate self signed certificate for server locally
openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout fedres.stack.com.key -out fedres.stack.com.crt -config sslcert.conf -extensions 'v3_req'

Now the following role “linux_configure_gitlabserver” will help us upload the template file “gitlab.rb” to the following default location “/etc/gitlab”. We are also going to create a new directory “/etc/gitlab/ssl” which is the default location wherein the GitLab server looks the SSL cert and key file to use. The names of the these two file should be with the FQDN of the server as shown below.

Once the GitLab server configuration and ssl certificate files are uploaded, we can reconfigure the GitLab server to make the configuration changes effective.

Here are the details of the configuration that we have updated. Please note, the prometheus listen address has been changed from default “9090” to “9191” as that is used by the systemd service in fedora.

You can refer the the default gitlab.rb configuration file at the following GitLab repository.

admin@fedser:gitlabserver$ cat roles/linux_configure_gitlabserver/templates/gitlab.rb | grep -v "^#" | sed '/^[[:space:]]*$/d'
...
external_url 'https://fedres.stack.com'
gitlab_rails['initial_root_password'] = "iyP/CTBhi8GGMVtrJg/CU5jQW0ENwj7mflr/fUNPE7I="
gitlab_rails['display_initial_root_password'] = true
gitlab_rails['store_initial_root_password'] = true
prometheus['listen_address'] = 'localhost:9191'
...
admin@fedser:gitlabserver$ cat roles/linux_configure_gitlabserver/defaults/main.yml 
---
gitlabserver_config_directory: /etc/gitlab
gitlabserver_config_template: gitlab.rb
gitlabserver_config_host: fedres.stack.com
gitlabserver_ssl_key: fedres.stack.com.key
gitlabserver_ssl_crt: fedres.stack.com.crt

admin@fedser:gitlabserver$ cat roles/linux_configure_gitlabserver/tasks/main.yml 
- name: "upload gitlab config"
  template:
    src: "{{gitlabserver_config_template}}"
    dest: "{{gitlabserver_config_directory}}/gitlab.rb"
    owner: "root"
    group: "root"
    mode: 0600

- name: ensure ssl cert directory exists
  file:
    path: "{{gitlabserver_config_directory}}/ssl"
    state: directory
    owner: "root"
    group: "root"
    mode: 0644

- name: "upload gitlab key file"
  template:
    src: "{{gitlabserver_ssl_key}}"
    dest: "{{gitlabserver_config_directory}}/ssl/{{gitlabserver_ssl_key}}"
    owner: "root"
    group: "root"
    mode: 0755

- name: "upload gitlab crt file"
  template:
    src: "{{gitlabserver_ssl_crt}}"
    dest: "{{gitlabserver_config_directory}}/ssl/{{gitlabserver_ssl_crt}}"
    owner: "root"
    group: "root"
    mode: 0755

- name: "reconfigure gitlab config"
  command: "gitlab-ctl reconfigure"

- name: "check gitlab status"
  command: "gitlab-ctl status"

Step3: Expose HTTP(S) protocols

Here in this step we are going to update our firewall setting to allow for HTTP and HTTPs service traffic for the GitLab server and reload the firewall daemon.

admin@fedser:gitlabserver$ cat roles/linux_expose_gitlabserver/tasks/main.yml 
---
- name: expose http service
  firewalld:
    service: http
    permanent: true
    immediate: true
    state: enabled

- name: expose https service
  firewalld:
    service: https
    permanent: true
    immediate: true
    state: enabled

- name: restart firewalld service
  service:
    name: firewalld
    state: restarted

Step4: Start GitLab Server

This role “linux_start_gitlabserver” can be used to startup the GitLab server instance and check its status.

admin@fedser:gitlabserver$ cat roles/linux_start_gitlabserver/tasks/main.yml 
- name: "ensure gitlab service started"
  command: "gitlab-ctl start"

- name: "check gitlab status"
  command: "gitlab-ctl status"

Step5: Stop GitLab Server

This role “linux_stop_gitlabserver” can be used to shutdown the GitLab server.

admin@fedser:gitlabserver$ cat roles/linux_stop_gitlabserver/tasks/main.yml 
- name: "ensure gitlab service stopped"
  command: "gitlab-ctl stop"

Step6: Restart GitLab Server

This role “linux_restart_gitlabserver” can be used to restart the GitLab server.

admin@fedser:gitlabserver$ cat roles/linux_restart_gitlabserver/tasks/main.yml 
- name: "ensure gitlab serivce restarted"
  command: "gitlab-ctl restart"

- name: "check gitlab status"
  command: "gitlab-ctl status"

Step7: GitLab Server Playbook

Here is the main playbook from which the required roles can be triggered based on the tag information that we pass to the ansible playbook. The instructions to execute are provided in the README.md file below.

admin@fedser:gitlabserver$ cat linux_setup_gitlabserver.yml 
---
- hosts: "gitlabserver"
  serial: 1
  become: true
  become_user: root
  roles:
  - { role: "linux_ping", tags: "linux_ping" }
  - { role: "linux_install_gitlabserver", tags: "linux_install_gitlabserver" }
  - { role: "linux_configure_gitlabserver", tags: "linux_configure_gitlabserver" }
  - { role: "linux_expose_gitlabserver", tags: "linux_expose_gitlabserver" }
  - { role: "linux_restart_gitlabserver", tags: "linux_restart_gitlabserver" }
  - { role: "linux_stop_gitlabserver", tags: "linux_stop_gitlabserver" }
  - { role: "linux_start_gitlabserver", tags: "linux_start_gitlabserver" }

Step8: README Instructions

admin@fedser:gitlabserver$ cat README.md 
# Instructions for execution

ansible-playbook linux_setup_gitlabserver.yml -i inventory/hosts --tags "linux_ping" -v
ansible-playbook linux_setup_gitlabserver.yml -i inventory/hosts --tags "linux_install_gitlabserver" -v 
ansible-playbook linux_setup_gitlabserver.yml -i inventory/hosts --tags "linux_configure_gitlabserver" -v
ansible-playbook linux_setup_gitlabserver.yml -i inventory/hosts --tags "linux_expose_gitlabserver" -v
ansible-playbook linux_setup_gitlabserver.yml -i inventory/hosts --tags "linux_stop_gitlabserver" -v
ansible-playbook linux_setup_gitlabserver.yml -i inventory/hosts --tags "linux_start_gitlabserver" -v
ansible-playbook linux_setup_gitlabserver.yml -i inventory/hosts --tags "linux_restart_gitlabserver" -v

Step9: Execute Playbook

We can execute specific role by mentioning the “–tags” option as shown in the instructions. But if we want to execute all the roles in a playbook we can run the playbook command without providing the “–tags” option as shown below.

Here let us disable the below two roles for the execution as we are going to install, configure and startup the gitlab server instance.

admin@fedser:gitlabserver$ cat linux_setup_gitlabserver.yml 
---
- hosts: "gitlabserver"
  serial: 1
  become: true
  become_user: root
  roles:
  - { role: "linux_ping", tags: "linux_ping" }
  - { role: "linux_install_gitlabserver", tags: "linux_install_gitlabserver" }
  - { role: "linux_configure_gitlabserver", tags: "linux_configure_gitlabserver" }
  - { role: "linux_expose_gitlabserver", tags: "linux_expose_gitlabserver" }
#  - { role: "linux_restart_gitlabserver", tags: "linux_restart_gitlabserver" }
#  - { role: "linux_stop_gitlabserver", tags: "linux_stop_gitlabserver" }
  - { role: "linux_start_gitlabserver", tags: "linux_start_gitlabserver" }

Now let us try to execute our playbook.

admin@fedser:gitlabserver$ ansible-playbook linux_setup_gitlabserver.yml -i inventory/hosts

PLAY [gitlabserver] ******************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************
ok: [fedres.stack.com]

TASK [linux_ping : ansible ping pong validation] *************************************************************************************************************
ok: [fedres.stack.com]

TASK [linux_install_gitlabserver : add gitlab ee server repository] ******************************************************************************************
ok: [fedres.stack.com]

TASK [linux_install_gitlabserver : install pre-requisites] ***************************************************************************************************
ok: [fedres.stack.com]

TASK [linux_install_gitlabserver : install gitlab ee server] *************************************************************************************************
ok: [fedres.stack.com]

TASK [linux_configure_gitlabserver : upload gitlab config] ***************************************************************************************************
ok: [fedres.stack.com]

TASK [linux_configure_gitlabserver : ensure ssl cert directory exists] ***************************************************************************************
ok: [fedres.stack.com]

TASK [linux_configure_gitlabserver : upload gitlab key file] *************************************************************************************************
ok: [fedres.stack.com]

TASK [linux_configure_gitlabserver : upload gitlab crt file] *************************************************************************************************
ok: [fedres.stack.com]

TASK [linux_configure_gitlabserver : reconfigure gitlab config] **********************************************************************************************
changed: [fedres.stack.com]

TASK [linux_configure_gitlabserver : check gitlab status] ****************************************************************************************************
changed: [fedres.stack.com]

TASK [linux_expose_gitlabserver : expose http service] *******************************************************************************************************
ok: [fedres.stack.com]

TASK [linux_expose_gitlabserver : expose https service] ******************************************************************************************************
ok: [fedres.stack.com]

TASK [linux_expose_gitlabserver : restart firewalld service] *************************************************************************************************
changed: [fedres.stack.com]

TASK [linux_start_gitlabserver : ensure gitlab service started] **********************************************************************************************
changed: [fedres.stack.com]

TASK [linux_start_gitlabserver : check gitlab status] ********************************************************************************************************
changed: [fedres.stack.com]

PLAY RECAP ***************************************************************************************************************************************************
fedres.stack.com           : ok=16   changed=5    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The order of stop/start roles can be changed as per your requirements.

Step10: Validate results

We can validate status of all the GitLab service using the below command on the GitLab instance.

admin@fedres:~$ sudo gitlab-ctl status
run: alertmanager: (pid 1115) 0s; run: log: (pid 1100) 0s
run: gitaly: (pid 1074) 0s; run: log: (pid 1072) 0s
run: gitlab-exporter: (pid 1114) 0s; run: log: (pid 1099) 0s
run: gitlab-kas: (pid 1080) 0s; run: log: (pid 1077) 0s
run: gitlab-workhorse: (pid 1120) 0s; run: log: (pid 1111) 0s
run: logrotate: (pid 1071) 0s; run: log: (pid 1070) 0s
run: nginx: (pid 1092) 0s; run: log: (pid 1087) 0s
run: node-exporter: (pid 1121) 0s; run: log: (pid 1112) 0s
run: postgres-exporter: (pid 1116) 0s; run: log: (pid 1101) 0s
run: postgresql: (pid 1083) 0s; run: log: (pid 1076) 0s
run: prometheus: (pid 1106) 0s; run: log: (pid 1105) 0s
run: puma: (pid 1119) 0s; run: log: (pid 1109) 0s
run: redis: (pid 1075) 0s; run: log: (pid 1073) 0s
run: redis-exporter: (pid 1103) 0s; run: log: (pid 1102) 0s
run: sidekiq: (pid 1118) 0s; run: log: (pid 1108) 0s

Also, you can access the GitLab portal using the following URL with https prtocol and the FQDN of the GitLab server.

URL - https://fedres.stack.com/users/sign_in

IMPORTANT

SELINUX is in enforcing mode by default in Fedora OS. If you are having issues with cloning the git repository using SSH. You need to fix the permissions issue due to selinux as below.

Error

Mar 22 21:38:26 fedres.stack.com sshd[7634]: Could not open user 'git' authorized keys '/var/opt/gitlab/.ssh/authorized_keys': Permission denied
Mar 22 21:38:26 fedres.stack.com sshd[7634]: Could not open user 'git' authorized keys '/var/opt/gitlab/.ssh/authorized_keys': Permission denied

Resolution

# semanage fcontext -a -t sshd_exec_t '/var/opt/gitlab/.ssh/authorized_keys'
# restorecon -v '/var/opt/gitlab/.ssh/authorized_keys'

QUICK HACK

If you have installed and configured gitlab using linux package and ever messed up with your /var/opt/gitlab permissions and ownership and the services are not coming up. You can execute the following script to fix the permissions issues.

#!/bin/bash

set -x

# chown_if_exists
# input: matches `chown` command, see `man chown`
# Simply, this checks that the file you're trying to chown actually exists
# before making the chown call. DRY'ing the rest of this script's checks.
chown_if_exists()
{
	# the last argument of chown is the file or path
	path="${@:${#@}}"
	if [ -e "$path" ]; then
		chown $@
	else
		echo "skipping, path does not exist: $path"
	fi
}

chmod_if_exists()
{
	# the last argument of chown is the file or path
	path="${@:${#@}}"
	if [ -e "$path" ]; then
		chmod $@
	else
		echo "skipping, path does not exist: $path"
	fi
}

# Fix GitLab permissions
if id -u git; then
	# Fix data storage
	chown_if_exists -R git:git /var/opt/gitlab/.ssh
	chown_if_exists -R git:git /var/opt/gitlab/.gitconfig
	chown_if_exists -R git:git /var/opt/gitlab/git-data
	chmod_if_exists 2770 /var/opt/gitlab/git-data/repositories
	chown_if_exists -R git:git /var/opt/gitlab/gitlab-ci/builds
	chown_if_exists -R git:git /var/opt/gitlab/gitlab-rails
	chown_if_exists -R git:git /var/opt/gitlab/gitlab-shell
	if id -g gitlab-www; then
		chown_if_exists -R git:gitlab-www /var/opt/gitlab/gitlab-workhorse
	fi

	# Fix log storage
	chown_if_exists git /var/log/gitlab/gitlab-workhorse
	chown_if_exists git /var/log/gitlab/gitlab-rails
	chown_if_exists git /var/log/gitlab/gitlab-shell
	chown_if_exists git /var/log/gitlab/sidekiq
	chown_if_exists git /var/log/gitlab/puma
	chown_if_exists git /var/log/gitlab/unicorn
	chown_if_exists git /var/log/gitlab/gitaly

	# Update log files
	chown_if_exists -R git:git /var/log/gitlab/gitlab-rails/*.log
	chown_if_exists -R git:git /var/log/gitlab/gitlab-shell/*.log
	chown_if_exists -R git:git /var/log/gitlab/puma/*.log
	chown_if_exists -R git:git /var/log/gitlab/unicorn/*.log
	chown_if_exists -R git:git /var/log/gitlab/gitaly/*.log
fi

# Fix nginx buffering & www directory permission
if id -u gitlab-www; then
	chown_if_exists -R gitlab-www:gitlab-www /var/opt/gitlab/nginx/*_temp
	chown_if_exists -R gitlab-www:gitlab-www /var/opt/gitlab/nginx/*_cache
	chown_if_exists -R root:root /var/opt/gitlab/nginx/www
fi

# Fix database storage and logs
if id -u gitlab-psql; then
	chown_if_exists -R gitlab-psql:gitlab-psql /var/opt/gitlab/postgresql
	chown_if_exists gitlab-psql /var/log/gitlab/postgresql
fi

# Fix prometheus storage and logs
if id -u gitlab-prometheus; then
	chown_if_exists -R gitlab-prometheus:gitlab-prometheus /var/opt/gitlab/prometheus
	chown_if_exists gitlab-prometheus /var/log/gitlab/prometheus
	chown_if_exists -R gitlab-prometheus:gitlab-prometheus /var/opt/gitlab/alertmanager
	chown_if_exists gitlab-prometheus /var/log/gitlab/alertmanager
fi

# Fix redis storage and logs
if id -u gitlab-redis; then
	chown_if_exists -R gitlab-redis:gitlab-redis /var/opt/gitlab/redis
	if id -g git ; then
		chown_if_exists gitlab-redis:git /var/opt/gitlab/redis
	fi
	chown_if_exists gitlab-redis /var/log/gitlab/redis
fi

# Fix registry storage
if id -u registry; then
	if [ -e "/var/opt/gitlab/gitlab-rails/shared/registry" ]; then
		find /var/opt/gitlab/gitlab-rails/shared/registry -type d -exec chmod 755 {} \;
		find /var/opt/gitlab/gitlab-rails/shared/registry -type f -exec chmod 744 {} \;
	fi
	chown_if_exists -R registry:git /var/opt/gitlab/gitlab-rails/shared/registry
fi

# Fix mattermost storage
if id -u mattermost; then
    chown_if_exists -R mattermost /var/opt/gitlab/mattermost
fi

Also even after fixing these ownership and permissions i was having the below error which was blocking the postgresql service to come up

2024-03-23_03:42:56.28371 FATAL:  data directory "/var/opt/gitlab/postgresql/data" has invalid permissions
2024-03-23_03:42:56.28372 DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).

Fixed it as below.

chmod -R 700 /var/opt/gitlab/postgresql/data

Hope you enjoyed reading this article. Thank you..