Are you tired of manually pushing code to production? Are you always searching through your BASH history to find the commands you used to test your code? Do you wish the process to merge code into production had a defined process? Well I have the solution for you! Introducing Gitlab CI/CD pipelines! With Gitlab you can setup Gitlab runners to create a CI/CD pipeline. A CI/CD pipeline will revolutionize your workflow to push code to production.

The purpose of this blog post is to provide instructions on how to setup the necessary components (Gitlab and Gitlab runners) to create a CI/CD pipeline. One of the deliverables from this blog post is Docker composes for Swarm and non-swarm deployments of Gitlab. Additionally, there are manual instructions on how to setup Gitlab runners on Ubuntu 20.04, Ubuntu 20.04 with Docker, Windows 10, Windows 10 with Docker, and macOS Big Sur. In addition, a Docker Registry is setup and integrated into the CI/CD pipeline for custom Docker images. The instructions and the infra-as-code provided in this post will create the foundation for future blogs that will contain a CI/CD component.

Goals

• Setup Gitlab stack with Docker Swarm
• Setup Gitlab runner on Windows
• Setup Gitlab runner on Linux
• Setup Gitlab runner on macOS
• Setup Gitlab runner on Docker Windows
• Setup Gitlab runner on Docker Linux
• Setup a Docker Registry with a self-signed HTTPS certificate

Update log

• September 24th 2021 – Updated Docker from Gitlab v13.8.1 to v14.3.0

Assumptions

• The means to generate DNS A records for each service.
  • These services require DNS records and will NOT work without it!!!
• Docker Swarm or machine running Docker to run Gitlab
• The ability to run VMs but not required
• Access to a macOS machine or VM but not required

Background

What is Gitlab?

GitLab is a web-based DevOps lifecycle tool that provides a Git-repository manager providing wiki, issue-tracking and continuous integration and deployment pipeline features, using an open-source license, developed by GitLab Inc.

What is a Gitlab runner?

GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline. You can choose to install the GitLab Runner application on infrastructure that you own or manage. You can install GitLab Runner on several different supported operating systems or GitLab Runner can also run inside a Docker container. A more simplified explanation is a Gitlab runner is essentially a tool that executes the instructions defined in .gitlab-ci.yml and sends the results back to Gitlab.

What is CI/CD?

_{Credit for the creation of this diagram goes to Valentin Despa: Udemy course: GitLab CI: Pipelines, CI/CD and DevOps for Beginners.}

A CI/CD pipeline automates the process of delivering code from a developers machine to production. Obviously that statement is an over simplification of the process because the digram above illustrates numerous steps. This section is going to provide a high overview of the process to help you understand the general process needed for this blog post. CI/CD stands for continuous integration and continuous deployment, which as the acronym and the digram above illustrates is two distinct phases. Continuous integration is the process of integrating new code changes, validating the new code changes can still build/compile the application, and ensuring the new code passes a set of tests.

For example, let’s say you have a web application written in GoLang. As a developer you make some changes to the existing application on your local development machine and push the changes to Gitlab. Next, Gitlab will attempt to compile the existing code base with your changes. Assuming the compilation is successful, Gitlab will perform several tests on the newly compiled application to ensure the application is functioning as intended. If the tests are successful, the developer can merge the changes into the MAIN branch.

Now you might be asking what happens if this phase is unsuccessful? Using the example above, let’s say you initialize a variable that is not used. If you’re a GoLang developer you already know this will fail to compile but for this example let’s say the code is pushed to Gitlab. Gitlab will once again attempt to compile the code that contains your changes. However, the compilation will fail and typically the pipeline will stop running on the first occurrence of an error. Gitlab will provide the developer the ability to review the error produced. Until this issue is resolved Gitlab will not allow the new code to merged.

Continuous deployment is the process of again evaluating/testing the newly committed code, pushing the application to QA for further evaluation, and finally upon manual human interaction the code is pushed to production. Pushing to prod (production) means pushing your code to the environment so that your new code can be utilized by users. Again, as the digram above illustrates there is more to this process but hopefully this provided a high overview of the process. For a more in-depth explanation, I highly recommend checking out the following Udemy course: GitLab CI: Pipelines, CI/CD and DevOps for Beginners.

.gitlab-ci.yml stages

There really aren’t any official stages but the ones listed below outline a typical flow you might see:

• Pre-build – A set of actions to perform before building your application with the newly committed code. During this stage you might install the necessary tools, libraries, or dependencies to build your application.
• Build – A set of actions to build/compile your application with the newly committed code
• Test – A set of actions to run against your newly compiled/built application to ensure everything is functioning as intended
• Deploy – A set of actions that will only run when the build and test stages have successful completed their tasks without any errors. Upon completion, this stage will push the newly committed code to appropriate environment.

Network diagram

Generate OpenSSL private key and public cert

1. git clone https://github.com/CptOfEvilMinions/Gitlab-Automation
2. cd Gitlab-Automation
3. mv conf/tls/tls.conf.example conf/tls/tls.conf
4. vim conf/tls/openssl.conf and set:
   1. Replace {{ base_domain }} with your domain
      1. My base_domain is hackinglab.local
   2. Set the location information under [ my_req_distinguished_name ]
      1. C – Set Coutry
      2. ST – Set state
      3. L – Set City
      4. O – Enter organization name
5. openssl req -x509 -new -nodes -keyout conf/tls/tls.key -out conf/tls/tls.crt -config conf/tls/tls.conf
   1. 1. Generate TLS private key and public certificate

Install Gitlab with Docker-compose v2.x

WARNING

The Docker-compose v2.x setup is for development use ONLY. The setup contains hard-coded credentials in configs and environment variables. For a more secure Docker deployment please skip to the next section to use Docker Swarm which implements Docker secrets.

WARNING

1. vim .env and set
   1. GITLAB_VERSION – OPTIONAL – Set the version of Gitlab to use – Community edition or Enterprise edition
   2. GITLAB_ROOT_PASSWORD – Set the Gitlab root user password
   3. POSTGRES_GITLAB_PASSWORD – Set Postgres Gitlab user password
   4. BASE_DOMAIN – Set this to your domain
2. docker-compose -f docker-compose.yml build
3. docker-compose -f docker-compose.yml up -d

Install Gitlab with Docker-compose v3.x (Swarm)

Create secrets

1. openssl rand -base64 32 | tr -cd '[:alnum:]' | docker secret create gitlab-postgres-gitlab-password -
   1. Create password for Gitlab Postgres password
2. GITLAB_ROOT_PASSWORD=$(openssl rand -base64 32 | tr -cd '[:alnum:]')
   1. Generate Gitlab root password
3. echo $GITLAB_ROOT_PASSWORD
   1. Print Gitlab root password – record for later
4. echo -n $GITLAB_ROOT_PASSWORD | docker secret create gitlab-root-password -
   1. Create Gitlab root password

Docker start stack

1. docker stack deploy -c docker-compose-swarm.yml gitlab
2. docker service logs -f gitlab_nginx
   1. Monitor logs until NGINX prints /docker-entrypoint.sh: Configuration complete; ready for start up

Login into Gitlab WebGUI

1. Open web browser to https://<Docker IP addr>:8443
   1. Enter root as username
   2. Enter <Gitlab root password> for password
   3. Select “Sign in”

Use Docker Registry

Push image to Registry

1. cd Gitlab-Automation
2. Add self-signed certificate for the Docker Registry to your certificate store
   1. MacOS
   2. Windows
   3. Linux
3. Instructions for macOS
   1. security add-trusted-cert -d -r trustRoot -k ~/Library/Keychains/login.keychain conf/tls/tls.crt
      1. Add Registry certificate to the user’s local keychain
      2. Enter password
      3. Restart Docker Desktop on macOS
4. docker build -f docker/Dockerfile-ubuntu-custom -t ubuntu-custom .
5. docker image ls | grep ubuntu-custom
   1. Grab image ID
6. docker tag <custom Ubuntu image ID> registry.hackinglab.local:5000/custom-ubuntu
7. docker push registry.hackinglab.local:5000/custom-ubuntu

Pull image from Registry

Since the certificate for the Registry is self-signed Docker will NOT pull the image. The instructions below are how to add the self-signed certificate for the Docker Registry to the OS root cert store.

1. SSH into Docker Swarm node
2. sudo su
3. openssl s_client -connect registry.<base_domain>:5000 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /usr/local/share/ca-certificates/ca.crt
   1. Obtain the public certificate from the remote server
4. update-ca-certificates --fresh
   1. Instruct Ubuntu to add the new cert to the cert root store
5. systemctl restart docker
6. docker pull registry.<base_domain>:5000/custom-ubuntu

Install/Setup Gitlab runner

This section will cover how to setup a Gitlab runner on Windows 10, Windows 10 with Docker, Ubuntu 20.04, Ubuntu 20.04 with Docker, and macOS Big Sur. It should be noted that setting up a Gitlab runner on Docker is the optimal setup. Running a Gitlab runner on a VM will not provide a clean state between CI/CD runs. Meaning that if you have an Ubuntu 20.04 VM and the first CI/CD run installs Java 8 but the second run requires Java 7 you will have to uninstall Java 8 first to avoid dependency conflicts.

However, the other side of this coin is you can have a Gitlab runner with a pre-defined environment. Therefore the runs can assume that the correct Java version is installed and you just need to build and run the code. Docker containers provide the advantage that each container is a blank canvas ready to be crafted to the need of the run.

Obtain Gitlab runner register token

1. Login in as a Gitlab admin
2. Admin area > Overview > Runners
3. Copy runners token
4. Copy Gitlab runner URL

Install/Setup Gitlab runner on Docker

1. GITLAB_RUNNER_DOCKER_TOKEN=$(curl -k -s -X POST https://gitlab.<base_domain>:8443/api/v4/runners --form "token=<Gitlab runner registration token>" --form "description=Docker" | jq -r .token | tr -d '\n' )
   1. Generate Gitlab runner token
2. echo $GITLAB_RUNNER_DOCKER_TOKEN
3. cp conf/gitlab-runner/docker-gitlab-runner.toml.example conf/gitlab-runner/docker-gitlab-runner.toml
4. cat conf/gitlab-runner/docker-gitlab-runner.toml | sed 's#{{ gitlab_runner_url }}#https://gitlab.<base_domain>:8443/#g' | sed "s#{{ gitlab_runner_token }}#${GITLAB_RUNNER_DOCKER_TOKEN}#g" | docker secret create gitlab-runner-config -
   1. Create Gitlab runner config containing Gitlab runner token
5. docker stack deploy -c docker-compose-swarm-gitlab-runners.yml gitlab-runner
   
6. docker service logs -f gitlab-runner_gitlab-runner
   
7. Admin area > Overview > Runners
8. Select “Edit” for the runner
   1. Enter linux,docker into the tags field
   2. Select “Save changes”

Install/Setup Gitlab runner on Windows 10

Install GIT

1. Log into Windows
2. Open Powershell as Administrator
3. Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
   1. Install Chocolately 
4. choco install git
   1. Install GIT

Install/Setup Gitlab runner

1. Log into Windows
2. Open Powershell as Administrator
3. mkdir 'C:\Program Files\gitlab-runner'
   1. Make a directory for Gitlab
4. cd 'C:\Program Files\gitlab-runner'
5. $ProgressPreference = 'SilentlyContinue'
   1. Download executable without status bar – faster download
6. Invoke-WebRequest -Uri https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-windows-amd64.exe -OutFile gitlab-runner.exe
   1. Download Gitlab runner
7. Open Chrome
8. Browse to Gitlab homepage
9. Select the “Not secure” button next to the address bar then “Certificate”
   1. Select. “Details” tab
   2. Select “Copy to file”
   3. Follow the instructions to export the certificate to the Desktop
      1. Select “Base-64 encoded X.509 (.CER)” for format
10. Move the public certificate to  C:\Program Files\gitlab-runner
11. Return to Powershell terminal
12. .\gitlab-runner.exe register --tls-ca-file .\gitlab.cer
    1. Register Gitlab runner
    2. Enter https://gitlab.<base_domain>:8443
    3. Enter <registration token>
    4. Enter WindowsVM for description
    5. Enter windows, win10_1909 for tags
    6. Enter shell for executor
13. .\gitlab-runner.exe install
    1. Install Gitlab runner as a service
14. .\gitlab-runner.exe start
    1. Start Gitlab runner service

Install/Setup Gitlab runner on Windows 10 with Docker

Install Hyper-V

1. Log into Windows
2. Open Powershell as Administrator
3. Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All
   1. Install Hyper-V
4. Reboot

Install WSL v2 with Linux kernel

1. Log into Windows
2. Open Powershell as Administrator
3. dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
   1. Enable the Windows Subsystem for Linux
4. Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform -NoRestart
   1. Enable Virtual Machine feature
5. cd $ENV:TMP
6. $ProgressPreference = 'SilentlyContinue'
7. Invoke-WebRequest -Uri https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi -OutFile wsl_update_x64.msi
   1. Download the Linux kernel update package
8. msiexec.exe /i "wsl_update_x64.msi" /quiet /qn /norestart
   1. Install updated Linux kernel
9. wsl --set-default-version 2
   1. Set WSL 2 as default

Install Docker

1. Log into Windows
2. Open Powershell as Administrator
3. cd $ENV:TEMP
   1. Enter user’s temporary directory
4. $ProgressPreference = 'SilentlyContinue'
   1. Download executable without status bar – faster download
5. Invoke-WebRequest -Uri https://desktop.docker.com/win/stable/Docker%20Desktop%20Installer.exe -OutFile DockerDesktopInstaller.exe
   1. Download Docker
6. .\DockerDesktopInstaller.exe install --quiet
   1. Install Docker
7. Logout and Sign back in
8. Start Docker if has no started on it’s own
   1. & 'C:\Program Files\Docker\Docker\Docker Desktop.exe'

Install and register Gitlab-runner

1. mkdir 'C:\Program Files\gitlab-runner'
   1. Make a directory for Gitlab
2. cd 'C:\Program Files\gitlab-runner'
3. Invoke-WebRequest -Uri https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-windows-amd64.exe -OutFile gitlab-runner.exe
   1. Download Gitlab runner
4. Open Chrome
5. Browse to Gitlab homepage
6. Select the “Not secure” button next to the address bar then “Certificate”
7. Select. “Details” tab
8. Select “Copy to file”
9. Follow the instructions to export the certificate to the Desktop
   1. Select “Base-64 encoded X.509 (.CER)” for format
10. Move the public certificate to  C:\Program Files\gitlab-runner
11. Return to Powershell terminal
12. .\gitlab-runner.exe register --tls-ca-file .\gitlab.cer
    1. Register Gitlab runner
    2. Enter https://gitlab.<base_domain>:8443
    3. Enter <registration token>
    4. Accept the VM’s hostname as the Gitlab runner’s name
    5. Enter windows, docker for tags
    6. Enter docker-windows for executor
    7. Accept default Windows image
13. docker pull mcr.microsoft.com/windows/servercore:1809
14. docker pull gitlab/gitlab-runner-helper: x86_64-775dd39d-servercore1909 
    1. Download Windows image
15. In the Windows system tray right-click Docker and select “Switch to Windows Containers”
    1. Select “Switch” on the pop-up
16. .\gitlab-runner.exe install
    1. Install Gitlab runner as a service
17. .\gitlab-runner.exe start
    1. Start Gitlab runner service

Install/Setup Gitlab runner on Ubuntu 20.04

1. SSH into Ubuntu
2. sudo su
3. apt update -y && apt install curl openssl -y
4. cd /tmp && curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb"
   1. Download Gitlab runner
5. dpkg -i gitlab-runner_amd64.deb
   1. Install Gitlab runner
6. mkdir /etc/gitlab-runner/certs
   1. Create a directory to store the Gitlab cert
7. openssl s_client -connect gitlab.<base_domain>:8443 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /etc/gitlab-runner/certs/gitlab.crt
   1. Grab a copy of the public cert for Gitlab
8. gitlab-runner register --tls-ca-file /etc/gitlab-runner/certs/gitlab.crt
   1. Register Gitlab runner
   2. Enter https://gitlab.<base_domain>:8443
   3. Enter <registration token>
   4. Enter ubuntuvm
   5. Enter linux, ubuntu2004 for tags
   6. Enter shell for executor
9. rm -rd /home/gitlab-runner/* 
   1. Delete all the dotfiles which are executed on every job run
10. systemctl restart gitlab-runner
11. systemctl enable gitlab-runner

Install/Setup Gitlab runner on macOS Big Sur

1. Log into macOS
2. Open terminal
3. sudo su
4. cd /tmp && curl https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-amd64 --output gitlab-runner-darwin-amd64.bin
   1. Download Gitlab runner
5. mv gitlab-runner-darwin-amd64.bin /usr/local/bin/gitlab-runner
6. chmod +x /usr/local/bin/gitlab-runner
   1. Install Gitlab runner
7. mkdir -p /etc/gitlab-runner/certs
8. cd /etc/gitlab-runner/certs
9. openssl s_client -connect gitlab.<base_domain>:8443 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /etc/gitlab-runner/certs/gitlab.crt
   1. Grab a copy of the public cert for Gitlab
10. gitlab-runner register --tls-ca-file /etc/gitlab-runner/certs/gitlab.crt
    1. Register Gitlab runner
    2. Enter https://gitlab.<base_domain>:8443
    3. Enter <registration token>
    4. Enter macosvm for description
    5. Enter macos, macos11for tags
    6. Enter shell for executor
11. gitlab-runner install
12. gitlab-runner start
    

Ensure all agents have checked in

1. Login in as a Gitlab admin
2. Admin area > Overview > Runners

Create Gitlab pipeline

Step 1: Create example repo

1. From the Gitlab user homepage
2. Select “New Project”
3. Select “Create blank project”
   1. Enter Test-gitlab-runner as Project name
   2. Select “Create project”

Step 2: Create .gitlab-ci.yml

1. Select “Project overview” in the top left
2. Select “New file” and select it again
3. Enter .gitlab-ci.yml as the file name
4. Open a web browser to https://github.com/CptOfEvilMinions/Gitlab-Automation/blob/main/gitlab-ci-example.yml
   1. Copy the contents
5. Paste the contents into .gitlab-ci.yml
6. Select “Commit” in bottom left
7. Select “Commit” again in the bottom left

Step 3: Review pipeline jobs

Pipeline jobs

1. Go to the Test-gitlab-runner repo
2. CI/CD > Pipelines
3. Select the latest pipeline run

test_linux_runner

test_macos_runner

test_win_runner

test_docker_runner

test_win_docker_runner

Shoutout

A personal shout out to Valentin Despa’s Gitlab course on Udemy: GitLab CI: Pipelines, CI/CD and DevOps for Beginners. I would highly recommend taking this class to learn more about Gitlab, Gitlab runners, and creating Gitlab CI/CD pipelines in-depth.

Lessons learned

New skills/knowledge

• Learned how to use environment variables in NGINX configs per this StackOverFlow post
• Learned about how to the utility sv to interact with services
• How to use Windows Docker containers
• Learned how to setup Gitlab runners
• Learned how to create a Gitlab CI/CD pipeline
• Learned how to push and pull images to the Docker Registry

What You’d Do Differently

• Build our Ansible playbooks for each Gitlab runner
• In an enterprise environment I would implement PKI to ensure all certificates are trusted

References

• Substitute environment variables in NGINX config from docker-compose
• Install GitLab Runner manually on GNU/Linux
• Install Hyper-V on Windows 10
• Powershell – Why is Using Invoke-WebRequest Much Slower Than a Browser Download?
• Silent installation of a MSI package
• How to Install WSL 2 on Windows 10 (Updated)
• Get started: Prep Windows for containers
• Docker Registry HTTP API V2
• Using openssl to get the certificate from a server
• Self-signed certificates or custom Certification Authorities