Feb 16 2021
Leave a comment
By spartan2194 DevOps, Tools

My development server for Vault

During the COVID19 lock down instead of playing videos games to consume my free time, I decided to be proactive. I started taking Udemy courses and one of the courses was on Vault and ever since I have been incorporating Vault into my blog posts. However, each blog post requires a unique setup and I prefer to start from a clean slate for each blog post. But the turn over of new keys and adding a new root CA to my local cert store became extremely tedious. Below is my Vault development setup where I address these issues.

WARNING – WARNING – WARNING

WARNING – WARNING – WARNING

WARNING – WARNING – WARNING

This blog post is outlining my DEVELOPMENT setup and the implementation to automatically unlock Vault SHOULD NOT be used in a production setting!!! The information contained in this blog post is for educational purposes ONLY! HoldMyBeerSecurity.com/HoldMyBeer.xyz and its authors DO NOT hold any responsibility for any misuse or damage of the information provided in blog posts, discussions, activities, or exercises.

If you would like to learn more about Vault and how to implement it properly please see my Getting Started with Vault blog post.

WARNING – WARNING – WARNING

WARNING – WARNING – WARNING

WARNING – WARNING – WARNING

Assumptions

  • You have a Docker Swarm
  • You have an FreeIPA/LDAP server
  • Knowledge on how to use Docker
  • Knowledge on how to use Vault
  • Knowledge on how to use FreeIPA/LDA

Goals

  • Setup Vault development server
  • Have a consistent root CA
  • Have a consistent login with LDAP

Problem statement

As I stated above there are several roadblocks for me when spinning up a new Vault instance. For me, the first hurdle is authentication to administrate and configure Vault. The second hurdle is not having a consistent root CA between each instance.

The first hurdle is solved by using an authentication backend which for me is LDAP. I have an LDAP account that is part of the LDAP administrators group and I created a Vault policy and role that says any user part of the LDAP administrators group has “root” privileges on Vault. This means the root token becomes an obsolete secret to maintain.

The second hurdle is solved by using Docker secrets and generating a root CA with OpenSSL that is imported into Vault. First, I start by generating a rootCA with OpenSSL. Next, when I spin up Vault for the first time I import this root CA as Vault’s root CA. This allows me to docker-compose up and docker-compose down Vault instances care free and still have the same root CA.

Create a service account on FreeIPA

  1. SSH into FreeIPA as root
  2. curl https://raw.githubusercontent.com/CptOfEvilMinions/Vault-Development-Server/main/conf/ldap/vault-sysaccount.ldif --output vault-sysaccount.ldif
  3. sed -i 's/changeme/<vault service account password>/g  vault-sysaccount.ldif
  4. sed - i 's/dc=example,dc=com/dc=<FreeIPA domain>,dc=<tld>/g'  vault-sysaccount.ldif
  5. cat vault-sysaccount.ldif
  6. ldapmodify -h localhost -p 389 -x -D "cn=Directory Manager" -W -f vault-sysaccount.ldif
    1. Enter FreeIPA directory service password
  7. rm vault-sysaccount.ldif
  8. ldapsearch -D "cn=Directory Manager" -x uid=vault -W

Generate root CA and NGINX TLS with OpenSSL

Generate root CA

  1. brew install openssl
    1. macOS uses LibreSSL which doesn’t have the necessary functionality for this blog post
  2. https://github.com/CptOfEvilMinions/Vault-Development-Server
  3. cd Vault-Development-Server
  4. cp conf/tls/openssl.conf.example conf/tls/openssl.conf
    1. Copy template
  5. vim conf/tls/openssl.conf and set:
    1. Set the location information under [ my_req_distinguished_name ]
      1. C – Set Country
      2. ST – Set state
      3. L – Set city
      4. O – Enter organization name
  6. openssl genrsa -out conf/tls/root_ca.key 8192
    1. Generate root CA RSA private key
  7. COMMON_NAME="<base_domain>" /usr/local/opt/openssl/bin/openssl req -new -x509 -days 3650 -sha512 -extensions v3_ca -key conf/tls/root_ca.key -out conf/tls/root_ca.crt -config conf/tls/openssl.conf
    1. Generate root CA public certificate
  8. openssl x509 -text -noout -in conf/tls/root_ca.crt | head -n 14
    1. Verify the creation details of the root CA

Generate NGINX private key and public cert

  1. openssl genrsa -out conf/tls/nginx.key 4096
    1. Generate NGINX private key
  2. COMMON_NAME="vault.<base_domain>" /usr/local/opt/openssl/bin/openssl req -new -sha512 -extensions v3_req -key conf/tls/nginx.key -out conf/tls/nginx.csr -config conf/tls/openssl.conf
    1. Generate NGINX certificate signing request
  3. COMMON_NAME="vault.<base_domain>" /usr/local/opt/openssl/bin/openssl x509 -req -days 365 -sha512 -extensions v3_req -in conf/tls/nginx.csr -CA conf/tls/root_ca.crt -CAkey conf/tls/root_ca.key -CAcreateserial -out conf/tls/nginx.crt -extfile conf/tls/openssl.conf
    1. Generate NGINX public certificate and sign it
  4. openssl x509 -text -noout -in conf/tls/nginx.crt | sed "/Modulus:/,/X509v3 Key Usage:/d" | head -n 18
    1. Verify the creation details of the server certificate

Add dev root CA to user’s macOS keychain

  1. Open Finder to the project folder
  2. Enter the conf/root_ca directory
  3. Double click root_ca.crt
  4. Find {{ base_domain}} root CA in the list of items in macOS key chain
  5. Double click the entry
  6. Expand the “Trust” section
    1. Set Secure Socket Layer (SSL) to “Always Trust”
    2. Set X.509 Basic Policy to “Always Trust”
    3. Close windows
  7. Enter password for changes to take effect

Spin up Vault-dev-server with Docker-compose v3.x (Swarm) with NGINX

Generate secrets

  1. https://github.com/CptOfEvilMinions/Vault-Development-Server
  2. cd Vault-Development-Server
  3. cat conf/tls/root_ca.key | docker secret create vault-dev-server-rootCA-key -
    1. Create a Docker secret containing the contents of the root CA private key
  4. cat conf/tls/root_ca.crt| docker secret create vault-dev-server-rootCA-cert -
    1. Create a Docker secret containing the contents of the root CA public cert
  5. echo 'ldaps://freeipa.<base_domain>' | docker secret create vault-dev-server-ldap-bind-url -
    1. Create Docker secret containing LDAP URL to access the LDAP server
  6. echo '<ldap-bind-username>' | docker secret create vault-dev-server-ldap-bind-username -
    1. Create Docker secret containing the LDAP username used to bind to LDAP
  7. echo '<ldap-bind-password>' | docker secret create vault-dev-server-ldap-bind-password -
    1. Create Docker secret containing the LDAP password used to bind to LDAP

Spin up stack

  1. docker stack deploy -c docker-compose-swarm.yml vault-dev-server
  2. docker service logs -f vault-dev-server_vault
  3. docker exec -it $(docker ps | grep vault-dev-server_vault | awk '{print $1}') cat /vault/data/vault_keys.txt | head -n 4
    1. Record this root token
    2. Record the UNseal key

Login into Vault via webGUI

  1. Open browser to https://vault.<base_domain>:8443
  2. Select “LDAP” for method
    1. Enter LDAP username
    2. Enter LDAP password
    3. Select “Sign in”

Login into Vault via CLI

  1. export VAULT_ADDR=https://vault.<base_domain>:<port>/
  2. vault login -method=ldap username=<LDAP username>
    1. Enter LDAP password

Demystifying the magic: vault_entrypoint_setup.sh

Start from the top

First, the script starts by installing the necessary tools which are curl and jq, since they are not installed by default. Next, the script has a while statement that waits until Consul is operational. The same operation is repeated but waiting for Vault to start. Once Vault has started, there is a check that is performed to see if Vault has been initialized or not.

Initialing Vault

If Vault is not initialized, the following command vault operator init -key-shares=1 -key-threshold=1 > /vault/data/vault_keys.txt (line 28) is executed which will initializes Vault. This process also generates an UNseal key(s) and root token. These secrets (unseal key and root token) are written to a text file located at /vault/data/vault_keys.txt. Next, the Vault unseal key (line 32) and the root token (line 33) are extracted from this text file. The extracted unseal key is used to unseal Vault (line 37). Once Vault has been unsealed, the root token is used to login as root (line 41) to perform admin operations. As I have noted several times throughout this blog post, this method of writing the Vault UNseal keys and root token to a text file is extremely insecure and should NEVER be used in a production setting!

Admin policy

Once logged in as the root user, a policy is created which contains all the permissions necessary to perform any task on Vault (line 44). Essentially, any user granted this policy is assuming the role of root on the Vault development server.

Enable LDAP auth

Next, LDAP authentication is enabled (line 48) and configured using Docker secrets (lines 51-56). Once Vault has been binded to the LDAP server, the script grants all LDAP users in the LDAP admins group the Vault admin policy (line 60). Remember this admin policy essentially provides the same privileges as root to all LDAP users in the admins group. This association means you no longer need to maintain/rotate the root token to login. You can simply login with your LDAP credentials.

Enable PKI – root CA

Next, the script enables the PKI secrets engine for certificates (line 64). However, before we upload the root CA to Vault we use the openssl command to convert the certificate and private key into a bundle (lines 68-70). Once the bundled is created it is uploaded to Vault (line 75). This means you can safely add this root CA to your systems trusted root cert store. This means you can destroy (docker stack rm) and create (docker stack deploy) your Vault development server as many times as you like.

It should be noted that the role created (line 76-81) allows intermediate and leaf certificates to be generated and signed by the root CA. The role name is generated by the common name contained within the root CA. My root CA, contains the following common name: hackinglab.local but the command below (line 76) will name the role: hackinglab-local.

Initialized Vault

If the Vault has already been initialized then we skip all the steps discussed above. First, we extract the Vault unseal key (line 90) from the text file located at /vault/data/vault_keys.txt. Next, Vault is unsealed (line 93) which puts it into the ready state to handle requests.

Tear down

Shutdown vault

If you want to shutdown Vault but persist the data run the following commands listed below. If you run docker volume ls and docker secret ls you will notice those items persisted. By not deleting the volumes any settings/roles/users/etc that were created/modified/deleted will persist.

  1. docker stack rm vault-dev-server

Shutdown vault and delete data

If you want to shutdown Vault and delete the persistent data run the commands listed below. If you run docker volume ls and docker secret ls you will notice the docker volumes were deleted but not the secrets. Deleting the volumes ONLY wipes any settings/roles/users/etc that were created/modified/deleted. If you re-deploy the stack it will be a clean fresh instance with LDAP auth enabled and the same root CA

  1. docker stack rm vault-dev-server
  2. docker volume rm vault-dev-server_consul-data vault-dev-server_vault-data vault-dev-server_vault-logs vault-dev-server_vault-policies

Discussion

Why LDAP and not userpass?

So you might be asking “wouldn’t it be easier to setup a static userpass account instead of LDAP?”. The simple answer is yes that would have been simpler. However, I am probably more likely to forget the password for that account. Additionally, I wanted to cover how to properly create a FreeIPA service account for Vault.

Traefik

This repo also contains  docker-compose-swarm-traefik.yml which is a template to use Vault with Traefik. In my environment, I have a dedicated Traefik reverse proxy that I use for TLS termination so I don’t need to use NGINX.

Lessons learned

I am currently reading a book called “Cracking the Coding Interview” and it is a great book. One interesting part of the book is their matrix to describe projects you worked on and the matrix contains the following sections which are: challenges, mistakes/failures, enjoyed, leadership, conflicts, and what would you do differently. I am going to try and use this model at the end of my blog posts to summarize and reflect on the things I learn. I don’t blog to post things that I know, I blog to learn new things and to share the knowledge of my security research.

New skills/knowledge

  • Learned how to generate root CA with OpenSSL
  • Learned how to generate a certificate signed by a root CA with OpenSSL

What You’d Do Differently

  • Learn how to generate private keys and certificates with certtool
  • Would have preferred to use the built-in OpenSSL/LibreSSL on macOS
  • Included more methods like Github for authentication

References

Leave a Reply

Your email address will not be published. Required fields are marked *