Aug 24 2020
Leave a comment
By spartan2194 Incident Response, Logging, PoC, System Administration, Tools

Setting up Kolide and Osquery with client certificates for mutual TLS (mTLS)

Do you know if your Osquery client is connecting to the right server? Do you know if your Kolide server is accepting requests from rogue devices? If you have answered “I don’t know” to either of these questions then this blog post is for you. This blog post will be demonstrating how to setup Kolide + Osquery with mutual TLS (mTLS). Mutual TLS is a mechanism that can be implemented to verify the identity of the server and clients.

DISCLAIMER

This blog post is a proof of concept (POC) for a homelab and does implement best practices for an enterprise environment. Please review the Hashicorp Vault, NGINX, and Kolide documetation for best practices.

DISCLAIMER

Background

What is Mutual TLS?

As stated by Cloudflare, “Mutual TLS (mTLS) authentication ensures that traffic is both secure and trusted in both directions between a client and server.” For a more thorough deep dive on mutual TLS please visit this blog post. However, I think for the purpose of this blog post the graphic below provides a good overview of the mutual TLS communication and why it is a powerful mechanism in information security.

What is Osquery?

Osquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.

What is Hashicorp Vault?

Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log. For this blog post, Vault will act as our certificate authority for our organization.

Explain the magic

Network diagram

Walkthrough

Mutual TLS security

In order to understand how mutal TLS (mTLS) works, I am going to discuss how it works in the context of Osquery (the client) and Kolide + NGINX (the server). When you browse the web you are only performing a one-way authentication which is verifying the server is who they claim to be. For example, any server on the internet could say they are your bank and host a copy of the website. Certificates allow us to verify that claim (identity) to ensure we aren’t entering our username and password for our bank account into a fake website.

With mutual TLS, the client is verifying the server’s identity and the server is verifying the client’s identity. NGINX is performing TLS termination and enforcing mTLS for all communcation for the Kolide service. First, Osquery attempts to connect to NGINX + Kolide (the server) and verifies the certificate being served by NGINX to verify the server’s identity. Once Osquery has verified the server’s identity, it sends its clients certificate and the server checks the client’s identity. Not only is the server checking the client’s identity but it also ensures that the client is authroized to access that service. Once the client (Osquery) has been verified, it completes the TLS handshake and communication between Osquery and Kolide can occur.

You might be asking what is the point of all this work?!?!? A very likely scenario is your organization is deciding whether to expose Kolide to the internet for when an endpoint is off the local network. The risk of exposing a Kolide instance to the internet is the fear of someone compromising it. With mutual TLS you can ensure that only authorized corporate devices are connecting to your NGINX + Kolide setup. Furthermore, mutal TLS provides protection if Kolide has a vulnerability but NGINX is enforcing client certificates. Assuming NGINX is also not vulnerable, an attacker would not be able to exploit this vulnerability without a client certificate. If you remember your networking 101 days, HTTP connections start with a TCP handshake before a client can communicate with a service. With HTTPS there is a TCP handshake and then a TLS handshake. If an attacker does not have a client certificate they can not complete a TLS handshake; therefore, they are unable to exploit the vulnerable Kolide server.

Generate certificates with Hashicorp Vault

This section assumes you have a certificate manager like Vault. If you do not have a certificate manager, in place take a look at my blog post: Install/Setup Vault for PKI + NGINX + Docker – Becoming your own CA.

Connect to Vault via command line

  1. git clone https://github.com/CptOfEvilMinions/BlogProjects.git
  2. cd BlogProjects/kolide-mutual-tls
  3. Obtain a Vault token
  4. brew install vault
  5. export VAULT_ADDR="http[s]://<Vault IP address>:<port>"
  6. vault login
    1. Enter token

Obtain ROOT CA public certificate

  1. vault read -format=json pki/cert/ca | jq -r '.data.certificate' > conf/tls/root_ca/root_CA.crt

Generate a leaf cert for Kolide web GUI

  1. vault write -format=json pki_int/issue/<role> common_name="kolide.<domain>" ttl=8760h > conf/tls/kolide/kolide_<domain>.json
  2. cat conf/tls/kolide/kolide_<domain>.json | jq -r '.data.private_key' > conf/tls/kolide/kolide_<domain>.key
    1. Extract private key from JSON blob
  3. cat conf/tls/kolide/kolide_<domain>.json | jq -r '.data.certificate' > conf/tls/kolide/kolide_<domain>.crt
  4. cat conf/tls/kolide/kolide_<domain>.json | jq -r '.data.ca_chain[]' >> conf/tls/kolide/kolide_<domain>.crt
    1. Extract public certificate from JSON blob
  5. Verify the public server certificate and private server key
    1. openssl rsa -noout -modulus -in conf/tls/kolide/*.key | openssl md5
    2. openssl x509 -noout -modulus -in conf/tls/kolide/*.crt | openssl md5
    3. Both commands should return the same MD5 hash

Generate an intermediate for devices

  1. vault secrets enable -path=devices_<domain>_pki_int pki
    1. Enable the PKI secrets engine at devices_<domain>_pki_int path
  2. vault secrets tune -max-lease-ttl=43800h devices_<domain>_pki_int
    1. Tune the devices_<domain>_pki_int secrets engine to issue certificates with a maximum time-to-live (TTL) of 26280 hours
    2. 26280 hours = 5 years
  3. vault write -format=json devices_<domain>_pki_int/intermediate/generate/internal common_name="devices.<domain> Intermediate Authority" | jq -r '.data.csr' > devices_<domain>_pki_int.csr
    1. Generate an intermediate certificate and save the certificate signing request (CSR) to disk
  4. vault write -format=json pki_int/root/sign-intermediate csr=@devices_<domain>_pki_int.csr format=pem_bundle ttl="43800h" | jq -r '.data.certificate' > devices_<domain>_pki_int.crt
    1. Sign the device intermediate certificate (devices.hackinglab.local) with the intermediate certificate (hackinglab.local) and write the generated certificate to disk
  5. rm devices_<domain>_pki_int.csr
    1. Delete signing
  6. vault write devices_<domain>_pki_int/intermediate/set-signed certificate=@devices_<domain>_pki_int.crt
    1. Import the signed certificate by the intermediate certificate into Vault

Create a role for device cert

  1. vault write devices_<domain>_pki_int/roles/devices-<domain> allowed_domains="devices.<domain>" allow_subdomains=true max_ttl=26280h
    1. Create a role to allow the generation of certificates with subdomains and a max TTL of 26280 hours
    2. 26280 hours = 5 years

Generate a leaf certificate for device/Osquery client

  1. vault write --format=json devices_<domain>_pki_int/issue/devices-<domain> common_name="<hostname>.devices.<domain>" ttl="26279h" > conf/tls/client/<hostname_devices_<domain>.json
    1. Request a device certificate
  2. cat conf/tls/client/<hostname>_devices_<domain>.json | jq -r '.data.private_key' > conf/tls/client/<hostname>_devices_<domain>.key
    1. Extract private key from JSON blob
  3. cat conf/tls/client/<hostname>_devices_<domain>.json | jq -r '.data.certificate' > conf/tls/client/<hostname>_devices_<domain>.crt
    1. Extract public certificate from JSON blob
  4. cat conf/tls/client/device01_devices_hackinglab_local.json | jq -r '.data.ca_chain[]' > conf/tls/client/server_cert_chain.crt
  5. cat conf/tls/root_ca/root_CA.crt >> conf/tls/client/server_cert_chain.crt
  6. rm conf/tls/client/<hostname>_devices_<domain>.json

Convert PEM cert bundle to PKCS12

  1. openssl pkcs12 -export -out conf/tls/client/<hostname>_devices_<domain>.p12 -inkey conf/tls/client/<hostname>_devices_<domain>.key -in conf/tls/client/<hostname>_devices_<domain>.crt -certfile conf/tls/root_ca/root_CA.crt
    1. Enter a password to protect PKCS12 cert bundle

Spin up Kolide with Docker

Docker-compose up

  1. openssl rand -base64 32
    1. Copy output
  2. Paste the key from the previous command in conf/kolide/kolide.yml as the value for jwt_key
  3. docker-compose build
  4. docker-compose run --rm kolide fleet prepare db --config /etc/kolide/kolide.yml
  5. docker-compose up -d
  6. docker stats

Verify mutual TLS is working

  1. Open Firefox
  2. Browse to https://<Docker IP addr>:8443 which will prompt you with a warning
  3. Settings > Preferences > Privacy & Security > Certificates
  4. Select “View certificates”
    1. Select the “Authorities” tab
    2. Select “Import”
    3. Import .../BlogProjects/kolide-mutual-tls/conf/tls/root_ca/root_CA.crt
    4. Select “Open”
      1. Select “Trust this CA to identify websites”
  5. Select “View certificates”
    1. Select the “Your certificates” tab
    2. Select Import ..../BlogProjects/kolide-mutual-tls/conf/tls/client/device01_devices_hackinglab_local.p12
    3. Select “Open”
      1. Enter password
  6. Browse to https://<Docker IP addr>:8443
    1. Select the new client certificate

Setup Kolide

  1. Set username and password
    1. Enter the username for admin
    2. Enter password
    3. Enter e-mail
    4. Select “Submit”
  2. Set Organization Details
    1. Enter org name
    2. Enter org URL
    3. Select “Submit”
  3. Set Kolide URL
    1. Enter https://<FQDN for Kolide>:8443
    2. Select “Submit”
  4. Select “Finish”

Install/Setup Osquery 4.4.0 on Ubuntu 20.04 with client certificate

Import client certificate

  1. Create an Ubuntu 20.04 VM
  2. scp conf/tls/client/<hostname>_devices_<domain>.crt <user>@<ubuntu VM IP addr>:~/<hostname>_devices_<domain>.crt
  3. scp conf/tls/client/<hostname>_devices_<domain>.key <user>@<ubuntu VM IP addr>:~/<hostname>_devices_<domain>.key
  4. scp conf/tls/client/server_cert_chain.crt <user>@<ubuntu VM IP addr>:~/server_cert_chain.crt
  5. SSH into VM: ssh <username>@<ubuntu VM IP addr>
  6. sudo mkdir -p /etc/pki/tls/{certs,private}
  7. mv server_cert_chain.crt /etc/pki/tls/certs/osquery_server_cert_chain.crt
    1. Move Osquery server CA chain to PKI location
  8. mv <hostname>_devices_<domain>.crt /etc/pki/tls/certs/<hostname>_devices_<domain>.crt
    1. Move device public certificate to PKI location
  9. mv <hostname>_devices_<domain>.key /etc/pki/tls/private/<hostname>_devices_<domain>.key
    1. Move device private key to PKI location
  10. sudo chown root:root /etc/pki/tls/certs/osquery_server_cert_chain.crt /etc/pki/tls/certs/<hostname>_devices_<domain>.crt /etc/pki/tls/private/<hostname>_devices_<domain>.key
  11. sudo chmod 640 /etc/pki/tls/certs/osquery_server_cert_chain.crt /etc/pki/tls/certs/<hostname>_devices_<domain>.crt /etc/pki/tls/private/<hostname>_devices_<domain>.key

Install Osquery 4.4.0

  1. export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
  2. sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys $OSQUERY_KEY
  3. sudo add-apt-repository 'deb [arch=amd64] https://pkg.osquery.io/deb deb main'
  4. sudo apt-get update -y
  5. sudo apt-get install osquery -y

Configure Osquery

  1. curl https://raw.githubusercontent.com/CptOfEvilMinions/BlogProjects/master/kolide-mutual-tls/conf/osquery/osquery.flags --output /etc/osquery/osquery.flags
  2. sed -i 's#kolide_tls_hostname#kolide.<domain>:8443#g' /etc/osquery/osquery.flags
    1. Set Kolide hostname and port
  3. sed -i 's#kolide_tls_cert_file_path#/etc/pki/tls/certs/osquery_server_cert_chain.crt#g' /etc/osquery/osquery.flags
    1. Set Kolide server certificate file path
  4. sed -i 's#device_tls_cert_file_path#/etc/pki/tls/certs/<hostname>_devices_<domain>_local.crt#g' /etc/osquery/osquery.flags
    1. Set device certificate file path
  5. sed -i 's#device_tls_key_file_path#/etc/pki/tls/private/device01_devices_hackinglab_local.key#g' /etc/osquery/osquery.flags
    1. Set device private key file path

Register and start Osquery

  1. Back to Kolide WebGUI
  2. Select “Add New Host” in the top right
  3. Select “Reveal Secret” and copy it
  4. Back to SSH session
  5. echo '<Kolide enroll secret>' > /etc/osquery/enroll.key
  6. systemctl restart osqueryd
  7. systemctl enable osqueryd
  8. Back to Kolide WebGUI

Future plans

This blog post is a precursor for another project I am working on. My next project is creating a web application that will allow incident responders to use the Osquery file carve feature to upload files. However, this application relies on the implementation of mutual TLS to ensure only authorized machines can upload data to the server.

Lessons learned

I am currently reading a book called “Cracking the Coding Interview” and it is a great book. One interesting part of the book is their matrix to describe projects you worked on and the matrix contains the following sections which are: challenges, mistakes/failures, enjoyed, leadership, conflicts, and what would you do differently. I am going to try and use this model at the end of my blog posts to summarize and reflect on the things I learn. I don’t blog to post things that I know, I blog to learn new things and to share the knowledge of my security research.

New skills/knowledge

  • Improved my understanding of Hashicorp Vault
  • Improved my understanding of mutal TLS and how to implement it
  • Learned how to use the Hashicorp Vault command line tool
  • Learned how to use OpenSSL to verify certificate chains and public cert + private key pairs.

Challenges

  • Was not able to find best practices documents and tutorials
  • Osquery’s --tls_client_cert option only accepts a single certificate. If a device certificate is part of a certificate chain with multiple intermediates, all intermediate certs and the root CA must be placed in --tls_server_certs.

What You’d Do Differently

  • Use Traefik to load balance Kolide traffic over multiple instances
  • Enforce that only device certificates can talk to Kolide to report results and pull down updates. At the current moment, any user with a device certificate can access the Kolide admin console.

References

Leave a Reply

Your email address will not be published. Required fields are marked *