Mar 31 2021
Leave a comment
By spartan2194 Logging, Network Security, System Administration, Tools

Implementing Logstash and Filebeat with mutual TLS (mTLS)

Do you know if your Filebeat client is connecting to a rogue Logstash server? Do you know if your Logstash server is accepting random logs from random devices? If you have answered “I don’t know” to either of these questions then this blog post is for you. The purpose of this blog post is to provide instructions on how to setup Logstash and Filebeat with mutual TLS (mTLS). The step-by-step instructions in this post, will demonstrate how to create the certificate chain of trust using Vault. Lastly, I will cover the Python script I created to automate constructing this logging certificate chain of trust.

Goals

  • Generate an intermediate certificate for Logstash
  • Generate leaf certificates for logging clients
  • Setup Logstash to use the intermediate certificate to authenticate clients
  • Setup Filebeat to use client/leaf certificate to authenticate itself to Logstash
  • Use mTLS for communication between Logstash and Filebeat

DISCLAIMER

This blog post is a proof of concept (POC) for a homelab and does NOT implement best practices for an enterprise environment. Please review the Hashicorp Vault documentation for best practices. 

DISCLAIMER

Background

What is Mutual TLS?

As stated by Cloudflare, “Mutual TLS (mTLS) authentication ensures that traffic is both secure and trusted in both directions between a client and server.” For a more thorough deep dive on mutual TLS please visit this blog post. However, I think for the purpose of this blog post the graphic below provides a good overview of the mutual TLS communication and why it is a powerful mechanism in information security.

What is Hashicorp Vault?

Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log. For this blog post, Vault will act as our certificate authority for our organization.

Assumptions

Network diagram

 

Certificate chain of trust diagram

The diagram below illustrates the certificate chain of trust that we are going to implement with Vault. For this blog post, I decided to do as much segmentation as possible when creating this certificate chain. As you can see from the diagram below, clients and servers have their own intermediate certificates. This means for servers, in our case Logstash, a certificate will need to be generated from the server intermediate certificate. For clients, a certificate will need to be generated from the clients intermediate certificate.

This segmentation enforces the flow of data from clients to servers. On the Filebeat endpoint we specify the client certificate, client private key, and the server intermediate certificate as a trusted authority. On the Logstash endpoint, we specify the server certificate, server private key, and the client intermediate certificate as a trusted authority. By doing this we are enforcing that only Filebeat client leaf certificates can send data to Logstash servers and Logstash server leaf certificates can ingest data from Filebeat clients.

 

Generate logging certificate chain using Vault CLI

Step 1: Generate logging intermediate certificate

  1. git clone https://github.com/CptOfEvilMinions/BlogProjects.git
  2. cd BlogProjects/logstash-mutual-tls
  3. Log into Vault
  4. export LOGGING_DOMAIN=<domain>
    1. Define the domain you want to generate certificate for
  5. vault secrets enable -path=logging_pki_int pki
    1. Enable the pki secrets engine at the logging_pki_int/ path
  6. vault secrets tune -max-lease-ttl=61320h logging_pki_int
    1. Tune the logging_pki_int/ secrets engine to issue certificates with a maximum time-to-live (TTL) of 61320 hours
    2. 61320 hours = 7 years
  7. vault write -format=json logging_pki_int/intermediate/generate/internal common_name="logging.${LOGGING_DOMAIN} Intermediate Authority" | jq -r '.data.csr' > conf/tls/loggint_int/logging_int.csr
    1.  Generate an intermediate certificate and save the certificate signing request (CSR) to disk
  8. vault write -format=json pki/root/sign-intermediate csr=@conf/tls/loggint_int/logging_int.csr format=pem_bundle ttl="61320h" | jq -r '.data.certificate' > conf/tls/loggint_int/logging_int.crt
    1. Sign the intermediate certificate with the root certificate and write the generated certificate to disk
  9. openssl x509 -in conf/tls/loggint_int/logging_int.crt -text -noout | grep 'Subject:'
  10. vault write logging_pki_int/intermediate/set-signed certificate=@conf/tls/loggint_int/logging_int.crt
    1. Import the signed certificate by the ROOT CA into Vault
  11. vault write logging_pki_int/roles/logging-servers allowed_domains=servers.logging.${LOGGING_DOMAIN},clients.logging.${LOGGING_DOMAIN} allow_subdomains=false max_ttl=43800h
    1. Specify that the only domains allowed to be used are clients.logging.${LOGGING_DOMAIN} and servers.logging.${LOGGING_DOMAIN}
    2. Create a role that grants the ability for the server and client intermediate certificate to be generated
    3. 43800 hours = 5 years
  12. rm conf/tls/loggint_int/logging_int.csr

Step 2: Generate server logging intermediate certificate

  1. vault secrets enable -path=servers_logging_pki_int pki
  2. vault secrets tune -max-lease-ttl=43800h servers_logging_pki_int
    1. Tune the servers_logging_pki_int/ secrets engine to issue certificates with a maximum time-to-live (TTL) of 43800 hours
      43800 hours = 5 years
  3. vault write -format=json servers_logging_pki_int/intermediate/generate/internal common_name="servers.logging.${LOGGING_DOMAIN} Intermediate Authority" | jq -r '.data.csr' > conf/tls/server_logging_int/server_logging_int.csr
    1.  Generate an intermediate certificate and save the certificate signing request (CSR) to disk
  4. vault write -format=json logging_pki_int/root/sign-intermediate csr=@conf/tls/server_logging_int/server_logging_int.csr format=pem_bundle ttl="43800h" | jq -r '.data.certificate' > conf/tls/server_logging_int/server_logging_int.crt
    1. Sign the intermediate certificate with the logging intermediate certificate and write the generated certificate to disk
  5. openssl x509 -in conf/tls/server_logging_int/server_logging_int.crt -text -noout | grep 'Subject:'
  6. vault write servers_logging_pki_int/intermediate/set-signed certificate=@conf/tls/server_logging_int/server_logging_int.crt
    1. Import the signed certificate by the logging intermediate into Vault
  7. vault write servers_logging_pki_int/roles/logging-servers allowed_domains=servers.logging.${LOGGING_DOMAIN} allow_subdomains=true max_ttl=26280h
    1. Specify that the only domains allowed to be used by servers.*
    2. Create a role that grants the ability for server leaf certificates to be generated
    3. 26280h hours = 3 years

Step 3: Generate client logging intermediate certificate

  1. vault secrets enable -path=clients_logging_pki_int pki
  2. vault secrets tune -max-lease-ttl=43800h clients_logging_pki_int
    1. Tune the clients_logging_pki_int/ secrets engine to issue certificates with a maximum time-to-live (TTL) of 43800 hours
      43800 hours = 5 years
  3. vault write -format=json clients_logging_pki_int/intermediate/generate/internal common_name="clients.logging.${LOGGING_DOMAIN} Intermediate Authority" | jq -r '.data.csr' > conf/tls/client_logging_int/clients_logging_int.csr
    1.  Generate an intermediate certificate and save the certificate signing request (CSR) to disk
  4. vault write -format=json logging_pki_int/root/sign-intermediate csr=@conf/tls/client_logging_int/clients_logging_int.csr format=pem_bundle ttl="43800h" | jq -r '.data.certificate' > conf/tls/client_logging_int/clients_logging_int.crt
    1. Sign the intermediate certificate with the logging intermediate certificate and write the generated certificate to disk
  5. openssl x509 -in conf/tls/client_logging_int/clients_logging_int.crt -text -noout | grep 'Subject:'
  6. vault write clients_logging_pki_int/intermediate/set-signed certificate=@conf/tls/client_logging_int/clients_logging_int.crt
    1. Import the signed certificate by the logging intermediate into Vault
  7. vault write clients_logging_pki_int/roles/logging-clients allowed_domains=clients.logging.${LOGGING_DOMAIN} allow_subdomains=true max_ttl=26280h
    1. Specify that the only domains allowed to be used by clients.*
    2. Create a role that grants the ability for client leaf certificates to be generated
    3. 26280h hours = 3 years

Step 4: Generate Logstash leaf certificate

  1. vault write -format=json servers_logging_pki_int/issue/logging-servers common_name="<logstash server hostname>.servers.logging.${LOGGING_DOMAIN}" ttl="26280h" private_key_format=pkcs8 > conf/tls/<logstash server hostname>_servers_logging_${LOGGING_DOMAIN}.json
    1. Request a new leaf certificate for the common_name specified above
    2. 26280 hours = 3 years
    3. Force the private key format to be PKCS8,  which is the only format accepted by Logstash v7.0+
  2. cat conf/tls/<logstash server hostname>_servers_logging_${LOGGING_DOMAIN}.json | jq -r '.data.private_key' > conf/tls/<logstash server hostname>_servers_logging_${LOGGING_DOMAIN}.key
    1. Extract private key from JSON blob
  3. cat conf/tls/<logstash server hostname>_servers_logging_${LOGGING_DOMAIN}.json | jq -r '.data.certificate' > conf/tls/<logstash server hostname>_servers_logging_${LOGGING_DOMAIN}.crt
  4. cat conf/tls/<logstash server hostname>_servers_logging_${LOGGING_DOMAIN}.json | jq -r '.data.ca_chain[]' >> conf/tls/<logstash server hostname>_servers_logging_${LOGGING_DOMAIN}.crt
    1. Extract public certificate chain from JSON blob
  5. rm <logging client hostname>_servers_logging_<domain>.json

Step 5: Verify Logstash leaf certificate chain

  1. curl -s -k https://vault.<domain>/v1/pki/ca/pem > /tmp/root_ca.crt
    1. Download the Vault root CA
  2. openssl verify -verbose -CAfile /tmp/root_ca.crt conf/tls/logging_int.crt
    1. Verify logging intermediate with root CA
  3. openssl verify -verbose -CAfile /tmp/root_ca.crt -untrusted conf/tls/loggint_int/logging_int.crt conf/tls/server_logging_int/server_logging_int.crt
    1. Verify the server logging intermediate, the logging intermediate, and the root CA
  4. openssl verify -verbose -CAfile /tmp/root_ca.crt -untrusted <(cat conf/tls/loggint_int/logging_int.crt conf/tls/server_logging_int/server_logging_int.crt) conf/tls/<logstash server hostname>_servers_logging_${LOGGING_DOMAIN}.crt
    1. Verify the Logstash server leaf certificate, the server logging intermediate, the logging intermediate, and the root CA

Step 6: Generate logging client leaf certificate

  1. vault write -format=json clients_logging_pki_int/issue/logging-clients common_name="<filebeat client hostname>.clients.logging.${LOGGING_DOMAIN}" ttl="26280h" private_key_format=pkcs8 > conf/tls/<filebeat client hostname>_clients_logging_${LOGGING_DOMAIN}.json
    1. Request a new leaf certificate for the common_name specified above
    2. 26280 hours = 3 years
    3. Force the private key format to be PKCS8,  which is the only format accepted by Filebeat v7.0+
  2. cat conf/tls/<filebeat client hostname>_clients_logging_${LOGGING_DOMAIN}.json | jq -r '.data.private_key' > conf/tls/<filebeat client hostname>_clients_logging_${LOGGING_DOMAIN}.key
    1. Extract private key from JSON blob
  3. cat conf/tls/<filebeat client hostname>_clients_logging_${LOGGING_DOMAIN}.json | jq -r '.data.certificate' > conf/tls/<filebeat client hostname>_clients_logging_${LOGGING_DOMAIN}.crt
  4. cat conf/tls/<filebeat client hostname>_clients_logging_${LOGGING_DOMAIN}.json | jq -r '.data.ca_chain[]' >> conf/tls/<filebeat client hostname>_clients_logging_${LOGGING_DOMAIN}.crt
    1. Extract public certificate chain from JSON blob
  5. rm conf/tls/<filebeat client hostname>_clients_logging_${LOGGING_DOMAIN}.json

Step 7: Verify Filebeat leaf certificate chain

  1. openssl verify -verbose -CAfile /tmp/root_ca.crt conf/tls/loggint_int/logging_int.crt
    1. Verify logging intermediate with root CA
  2. openssl verify -verbose -CAfile /tmp/root_ca.crt -untrusted conf/tls/loggint_int/logging_int.crt conf/tls/client_logging_int/clients_logging_int.crt
    1. Verify the clients logging intermediate, the logging intermediate, and the root CA
  3. openssl verify -verbose -CAfile /tmp/root_ca.crt -untrusted <( cat conf/tls/loggint_int/logging_int.crt conf/tls/client_logging_int/clients_logging_int.crt) conf/tls/<filebeat client hostname>_clients_logging_hackinglab.local.crt
    1. Verify the Logstash clients leaf certificate, the clients logging intermediate, the logging intermediate, and the root CA

Automate setting up Vault logging PKI

  1. vault login
    1. Log into Vault
  2. virtualenv -p python3 venv
  3. source venv/bin/activate
  4. pip3 install -r requirements.txt
  5. python3 setup_logging_pki_int.py --setup_logging_pki --vault_addr https://vault.<domain> --logging_domain <logging domain>
    1. Setup logging PKI
  6. python3 setup_logging_pki_int.py --create_leaf_cert --vault_addr https://vault.<domain> --logging_domain <logging domain> --server_leaf_cert_hostname <logstash server hostname>
  7. python3 setup_logging_pki_int.py --create_leaf_cert --vault_addr https://vault.<domain> --logging_domain <logging domain> --client_leaf_cert_hostname <logstash server hostname>

Install/Setup Logstash v7.11 on Ubuntu 20.04

Install Logstash

  1. wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
  2. sudo apt-get install apt-transport-https -y
  3. echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
  4. sudo apt-get update -y && sudo apt-get install logstash -y

Setup Logstash

  1. COPY conf/tls/client_logging_int/client_logging_pki_int.crt to /etc/ssl/certs/<domain>_client_logging_int.crt
  2. COPY conf/tls/fielbeat01_clients_logging_<domain>.crt to /etc/ssl/certs/conf/tls/fielbeat01_clients_logging_<domain>.crt
  3. COPY conf/tls/fielbeat01_clients_logging_hackinglab.local.key to /etc/ssl/private/fielbeat01_clients_logging_<domain>.key
  4. chmod 644 /etc/ssl/certs/<domain>_client_logging_int.crt /etc/ssl/certs/conf/tls/fielbeat01_clients_logging_<domain>.crt
    1. Set the proper permissions for the certificates
  5. chmod 600 /etc/ssl/private/fielbeat01_clients_logging_<domain>.key
    1. Set the proper permissions for the Logstash private key
  6. curl https://raw.githubusercontent.com/CptOfEvilMinions/BlogProjects/master/logstash-mutual-tls/conf/logstash/02-input-beats.conf --output /etc/logstash/pipeline/02-input-beats.conf
    1. Download Beats input config
  7. sed -i 's#{{ logstash_server_logging_crt_file_path }}#/etc/ssl/certs/fielbeat01_clients_logging_<domain>.crt#g' /etc/logstash/pipeline/02-input-beats.conf
    1. Set the file path for the Logstash server certificate
  8. sed -i 's#{{ logstash_server_logging_key_file_path }}#/etc/ssl/private/fielbeat01_clients_logging_<domain>.key#g' /etc/logstash/pipeline/02-input-beats.conf
    1. Set the file path for the Logstash server private key
  9. sed -i 's#{{ client_logging_crt_file_path }}#/etc/ssl/certs/<domain>_client_logging_int.crt#g' /etc/logstash/pipeline/02-input-beats.conf
    1. Set the file path for the client intermediate certificate
  10. systemctl restart logstash

Install/Setup Filebeat v7.11 on Ubuntu 20.04

Install Filebeat

  1. wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
  2. sudo apt-get install apt-transport-https -y
  3. echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
  4. sudo apt-get update -y && sudo apt-get install filebeat -y

Setup Filebeat

  1. cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.bak
    1. Backup config
  2. curl https://raw.githubusercontent.com/CptOfEvilMinions/BlogProjects/master/logstash-mutual-tls/conf/filebeat/filebeat.yml --output /etc/filebeat/filebeat.yml
    1. Download config
  3. COPY conf/tls/server_logging_int/server_logging_int.crt to /etc/ssl/certs/<domain>_server_logging_int.crt
  4. COPY conf/tls/fielbeat01_clients_logging_<domain>.crt to /etc/ssl/certs/conf/tls/fielbeat01_clients_logging_<domain>.crt
  5. COPY conf/tls/fielbeat01_clients_logging_hackinglab.local.key to /etc/ssl/private/fielbeat01_clients_logging_<domain>.key
  6. chmod 644 /etc/ssl/certs/<domain>_server_logging_int.crt /etc/ssl/certs/conf/tls/fielbeat01_clients_logging_<domain>.crt
    1. Set the proper permissions for the certificates
  7. chmod 600 /etc/ssl/private/fielbeat01_clients_logging_<domain>.key
    1. Set the proper permissions for the Filebeat private key
  8. sed -i 's#{{ logstash_addr }}#<Logstash IP addr or FQDN> #g' /etc/filebeat/filebeat.yml
    1. Set the Logstash IP server IP address or FQDN
  9. sed -i 's#{{ logstash_port }}#<Logstash Beats port>#g' /etc/filebeat/filebeat.yml
    1. Set the Logstash Beats port
  10. sed -i 's#{{ server_logging_crt_file_path }}#/etc/ssl/certs/<domain>_server_logging_int.crt#g' /etc/filebeat/filebeat.yml
  11. sed -i 's#{{ filebeat_client_logging_crt_file_path }}#/etc/ssl/certs/conf/tls/fielbeat01_clients_logging_<domain>.crt#g' /etc/filebeat/filebeat.yml
  12. sed -i 's#{{ filebeat_client_logging_key_file_path }}#/etc/ssl/private/fielbeat01_clients_logging_<domain>.key#g' /etc/filebeat/filebeat.yml

Test Filebeat -> Logstash connection

  1. filebeat test output
  2. systemctl restart filebeat

Lessons learned

I am currently reading a book called “Cracking the Coding Interview” and it is a great book. One interesting part of the book is their matrix to describe projects you worked on and the matrix contains the following sections which are: challenges, mistakes/failures, enjoyed, leadership, conflicts, and what would you do differently. I am going to try and use this model at the end of my blog posts to summarize and reflect on the things I learn. I don’t blog to post things that I know, I blog to learn new things and to share the knowledge of my security research.

New skills/knowledge

  • Learned how to generate intermediate and leaf certificates for a service
  • How to connect to Vault using client certificate
  • How to implement mTLS with Logstash and Filebeat
  • Learned how to implement Python requests sessions
  • Learned how to use a custom HTTP method with python requests
  • Learned how to use Python requests with the Vault API

Challenges

What You’d Do Differently

  • Set up a Vault agent to auto-request and re-new a client certificate from Vault.

References

Leave a Reply

Your email address will not be published. Required fields are marked *