Mar 04 2021
Leave a comment
By spartan2194 DevOps, Tools

Gitlab CI/CD pipeline with Vault secrets

 

The purpose of this blog post is to provide instructions on how to setup Gitlab and Vault to use secrets during a CI/CD pipeline build. In addition, I will break down the JWT authorization process with an explanation of the process for Gitlab + Vault. The explanation, step-by-step instructions, and the infra-as-code provided in this post will create the foundation for future blogs that will contain a CI/CD component with Vault secrets.

Goals

  • Enable JWT authentication method on Vault
  • Utilize JWT authorization to read secrets from Vault for Gitlab CI/CD jobs

Background

What is Gitlab?

GitLab is a web-based DevOps lifecycle tool that provides a Git-repository manager providing wiki, issue-tracking and continuous integration and deployment pipeline features, using an open-source license, developed by GitLab Inc.

What is Vault?

HashiCorp Vault is an open-source tool for managing secrets. Application identity management with Vault enables applications and machines to automatically create, change, and rotate secrets needed for communications, services, scripts, etc. Additionally, Vault enables administrators to manage applications and machines by providing access control over different secrets.

What is CI/CD?

Credit for the creation of this diagram goes to Valentin Despa: Udemy course: GitLab CI: Pipelines, CI/CD and DevOps for Beginners.

A CI/CD pipeline automates the process of delivering code from a developers machine to production. Obviously that statement is an over simplification of the process because the digram above illustrates numerous steps. This section is going to provide a high overview of the process to help you understand the general process needed for this blog post. CI/CD stands for continuous integration and continuous deployment, which as the acronym and the digram above illustrates is two distinct phases. Continuous integration is the process of integrating code changes, validating the new code changes can still build/compile the application, and ensuring the new code passes a set of tests.

For example, let’s say you have a web application written in GoLang. As a developer you make some changes to the existing application on your local development machine and push the changes to Gitlab. Next, Gitlab will attempt to compile the existing code base with your changes. Assuming the compilation is successful, Gitlab will perform several tests on the newly compiled code to ensure the application is functioning as intended. If the tests are successful, the developer can merge the changes into the MAIN branch.

Now you might be asking what happens if this phase is unsuccessful? Using the example above, let’s say you initialize a variable that is not used. If you’re a GoLang developer you already know this will fail to compile but for this example let’s say the code is pushed to Gitlab. Gitlab will once again attempt to compile the code that contains your changes. However, the compilation will fail and typically the pipeline will stop running on the first occurrence of an error. Gitlab will provide the developer the ability to review the error produced. Until this issue is resolved Gitlab will not allow the new code to merged.

Continuous deployment is the process of again evaluating/testing the newly committed code, pushing the application to QA for further evaluation, and finally upon manual human interaction the code is pushed to production. Pushing to prod (production) means pushing your code to the environment so that your new code can be utilized by users. Again, as the digram above illustrates there is more to this process but hopefully this provided a high overview of the process. For a more in-depth explanation, I highly recommend checking out the following Udemy course: GitLab CI: Pipelines, CI/CD and DevOps for Beginners.

Understanding JWT authorization

Credit: What Is JWT and Why Should You Use JWT

Step 1: Authentication/Authorization

JWT stands for JSON web tokens. It should be noted that JWT is used for authorization and not authentication. So you might be asking what’s the difference?!?!? Authentication is when you first approach a web server and login with a username and password. By providing a username and password that is correct you are authenticating as that user.

Once you have been authenticated the user is granted a secret. This secret is used for every subsequent action with the web server. Therefore, on the next request, you would send your secret to the web server and it would verify the secret is valid and if it has the permissions to perform the requested action, which is authorization. As we move forward, I will use the term authentication when referring to Vault because it specifies JWT as an auth method but really its authorization.

Step 2: Generating JWT

Upon successful authentication, a JWT token is generated for the user and signed by the server. This JWT is sent to the user to be stored in their browser’s cache. Now to this point you might be thinking this is no different than the typical process that is currently used which is session ID.

The biggest difference between JWT and session ID implementations is JWT does not require the server to remember/store anything. With a session ID the server must remember/store that session for all future requests. The JWT contains all the necessary information about the user for the server to verify the JWT token for all future requests.

Step 3: Authorization using JWT

A JWT is a base64 encoded blob that contains all the necessary information to authorize a user. If you decode a JWT it will contain the sections listed in the screenshot above, which are the header, payload, and signature. The header section defines the type (typ) which is JWT and the encryption/decryption algorithm (alg) used which is HS256. The payload section contains a subject (sub) key-value pair which is typically the user ID, iat which stands for issued at which is a timestamp for when the token was generated, and lastly this section may contain any data relevant (name) to the application such as the user’s name.

For example, this section could include a key-value pair for the username. However, the key-values pairs mentioned are the most common. Lastly, we have the verify section which is used to ensure the JWT being sent is valid. For more information see this blog post by Auth0 or this Youtube video.

Understanding Gitlab + Vault + JWT authorization

Credit: HashiCorp Vault GitLab integration: why and how?

Step 0: Create Gitlab project/repo

As you we will see in the step-by-step instructions below, you need to create a Gitlab repo/project first. The project_id generated by Gitlab (screenshot below) for the repo will be used to instruct Vault as to which secrets that project has access too.

Step 1: Configure Vault and secrets

This may sound intuitive but to access secrets they must first exist in Vault. Therefore, a user must first create a Vault secrets path and place all the necessary secrets for a project there. Next, a Vault policy must be created to grant permissions to these secrets.

As you scan see in the screenshot below the policy allows reading and listing of the secrets stored at the following path: secrets/gitlab/project_1. Lastly, this policy must be attached to a Vault role that specifies an auth mechanism (JWT) and a bound claim (more on this later) that can access those secrets defined in the attached policy.

Step 2a: Setup Gitlab pipeline

Next, we need to configure a Gitlab CI/CD pipeline using a .gitlab-ci.yml config. First, this file is defined in the project repo. While this config can come in many shapes and sizes the basic outline will look like the screenshot below. The config starts by defining the Vault address, Vault secret path, and the Vault role in the variable section. When the CI/CD pipeline is triggered Gitlab will generate a JWT that is passed to the pipeline as the following environment variable CI_JOB_JWT.

In the sections to follow, I will do a deeper dive into various aspects of this config. However, at a high-level, this config starts by requesting a Vault token from Vault. Next, if the request is successful, a Vault token is returned, which the Gitlab runner can use to request secrets

Step 2b: Generate JWT

As I stated in the previous section when the CI/CD pipeline is triggered a JWT is generated. Let’s take a look at this JSON document to see how it is composed. Typically, a Gitlab JWT will include information about the the Gitlab server, the user excuting the pipeline job, Gitlab project ID, Vault secret path, and Gitlab job details. The screenshot below shows an example JWT payload that is generated and sent to Vault.

Step 3 and 4: Authenticate to Vault + Vault verify JWT

In this section, I am going to combine step 3 (Authenticate to Vault) and step 4 (Vault verify JWT) into one section. When Vault receives a JWT payload from Gitlab with a request for secrets it needs to verify the JWT. In the step-by-step instructions below, we will enable JWT auth on Vault which requires a bounder_issuer and a JKWS endpoint to be supplied. The bound_issuer is typically set to Gitlab’s FQDN to instruct Vault “who” can use this auth method and a URL to the JKWS endpoint to instruct Vault “how” to verify the JWT. Therefore, when Vault receives a JWT request it will ensure that the requestor has been approved to use this auth method and can verify the creator of the JWT.

To verify a JWT, Vault extracts the the bound_issuer/iss field which is typically the FQDN of Gitlab from the JWT payload. Next, Vault does a lookup on the extracted field to get the JWKS endpoint for that bound_issuer. Using the JKWS endpoint, Vault verifies the signed JWT payload to ensure that Gitlab was the creator of the request.

So for example, my Homelab Gitlab instance lives at the following FQDN: gitlab.hackinglab.local. As you will see in the step-by-step instructions below, I set bound_issuer to gitlab.hackinglab.local and the JWKS endpoint is set to jwks_url="https://gitlab.hackinglab.local/-/jwks". So what does this all mean? It means when Vault receives a JWT requesting access to a secret is will extract the iss field which should match the bound_issuer supplied to Vault. Next, Vault will verify the JWT signature by comparing it to the signature located at the JWKS endpoint provided.

Step 5: Vault checks bounded claims and attached policies

After verifying the JWT payload, Vault will extract the Vault role trying to be assumed. Below is a screenshot of the role we are going to use for this blog post. The Vault role below states that it can be used for JWT authentication, it has the gitlab-vault-readonly policy attached, and a bound_claims section which defines additional parameters for “who” can assume this role.

The bound_claims section states that following criteria must be meet which are: only the Gitlab project/repo with a project_id of 2 can assume this role and only a CI/CD pipeline build for the master/main branch can assume this role. If the bound_claims contained in the JWT payload match this criteria then this role can be assumed. Lastly, assuming the JWT contains the necessary criteria for this role, Vault will generate a token. This Vault token will be granted the ability to assume this role.

Step 6 and 7: Vault returns token + Gitlab runner reads secret from Vault

In this section, I am going to combine step 6 (Vault returns token) and step 7 (Gitlab runner reads secret from Vault) into one section. In the previous section, a Vault token is generated and returned to the Gitlab runner. This Vault token can be used by the Gitlab runner to request secrets from Vault. As I stated above, the Vault token is granted a role which has a defined set of policies. When Vault receives a request to access a secret it will check the policies attached to the role to ensure it has been granted permissions to access this secret.

Based on the screenshot below, the policy attached to this role grants READ and LIST permissions for secrets at the following path: secrets/gitlab/project_1. SO what does this mean? Let’s say the Gitlab runner is attempting to read the following secret: secrets/gitlab/project_1/password, per the policy it can read that secret. But let’s say the Gitlab runner is attempting to read the following secret: secrets/mysql_db/super_secret_db_passwrd, per the policy it does NOT have the permissions to read that secret. For more information on this process please watch the following Youtube video: HashiCorp Vault GitLab integration: why and how?

Assumptions

Create a Gitlab repo

As I stated above in the “Assumptions” section this blog post assumes you have a Gitlab EE instance running

  1. Login into Gitlab
  2. Select “Projects” in the top left
  3. Select “Create blank project”
    1. Enter VaultGitlab as project name
    2. Select “Create project”
  4. Select “Project overview” in the top left
  5. Record the project ID

 

Setup Vault

Enable JWT auth

  1. Log into Vault using the CLI tool
  2. curl -s -k -X GET https://gitlab.<base_domain>:8443/-/jwks
    1. Test Gitlab JWT auth
  3. Log into Vault via the CLI as an admin
  4. echo | openssl s_client -connect gitlab.<base_domain>:<port> 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/gitlab.crt
    1. If Gitlab is being served by a self-signed certificate pull down the cert
  5. vault auth enable jwt
  6. vault write auth/jwt/config jwks_url="https://gitlab.<base_domain>:<port>/-/jwks" bound_issuer="gitlab.<base_domain>" jwks_ca_pem="$( cat /tmp/gitlab.crt)"
  7. vault read auth/jwt/config
    1. Test Vault JWT auth

Create secrets

  1. vault secrets enable -version=2 -path=secrets kv
    1. Enable the Vault secrets engine
  3. vault kv put secrets/<secret path> <secret_key>=<secret_value>
  4. vault kv get secrets/gitlab/project_1
    1. Retrieve password

Setup Vault to provide Gitlab access to secrets

Vault policy

  1. cd GitlabVaultCICD
  2. cp conf/vault/policies/gitlab-vault-policy.hcl.example conf/vault/policies/gitlab-vault-policy.hcl
  3. vim conf/vault/policies/gitlab-vault-policy.hcl and set:
    1. {{ vault_secret_path }} – Set this to the Vault path where the secrets are located
  4. vault policy write <policy name> conf/vault/policies/gitlab-vault-policy.hcl
    1. Example policy name: gitlab-vault-readonly

Vault role

  1. Login into Gitlab
  2. Browse to the Gitlab repo you want to grant access to Vault secrets
  3. Select “Project overview” in the top left
  4. Obtain the Project ID
  5. cp conf/vault/roles/gitlab-jwt-role.json.example conf/vault/roles/gitlab-jwt-role.json
  6. vim conf/vault/roles/gitlab-jwt-role.json and set:
    1. {{ gitlab_project_id }} – Set this to the project ID
      1. The screenshot above shows a project ID of 2
    2. {{ gitlab_vault_role }} –
      1. Name of the Vault policy from the previous section – I used gitlab-vault-readonly
  7. cat conf/vault/roles/gitlab-jwt-role.json | vault write auth/jwt/role/gitlab-vault-readonly -

.gitlab-ci.yml – Get Vault secrets

  1. Login into Gitlab
  2. Browse to the Gitlab repo you want to grant access to Vault secrets
  3. Select “Project overview” in the top left
  4. Select “New file”
    1. Enter .gitlab-ci.yml as file name
    2. Copy the contents of conf/gitlab/gitlab-ci-example.yml.example into .gitlab-ci.yml and set:
      1. VAULT_ADDR – Set to the HTTP URL location of Vault
      2. {{ vault_secret_path }} – Set to the Vault secret path created above
      3. {{ gitlab_vault_role }} – Set to the Vault role created for Gitlab above
      4. Save and exit
  5. Select CI/CD > pipelines on the left
  6. Select the latest pipeline job
  7. Select vault_secrets stage

Lessons learned

New skills

  • Learned how to implement environment variables in NGINX configs per this StackOverFlow post
  • Learned how JWT authorization works
  • Learned the difference between authorization and authentication
  • How to request secrets from Vault for a Gitlab CI/CD pipeline
  • How to setup Vault JWT auth method
  • How to create a .gitlab-ci.yml config

References

Leave a Reply

Your email address will not be published. Required fields are marked *