The industry has gravitated towards using Jupyter notebooks for automating incident response and threat hunting. However, one of the biggest barriers for any application/automation is the ability to store secrets (username+passwords, API keys, etc) to access other services. This blog post will demonstrate how to use Vault to store secrets and integrate the ability to retrieve secrets from Vault with Jupyter Notebooks to assist in automating your security operations.

Goals

Store secrets in a safe location, which is Vault
Centralize the ability to access secrets with Vault
Set up Vault policies for your incident response team
Set up Vauly groups for your incident response team
Create a Jupyter Notebook that requests secrets from Vault
Review the code of a notebook to request secrets form Vault
Demonstrate how to use Jupyter Notebooks with the runbooks provided

Background

What are Jupyter Notebooks?

Think of a notebook as a document that you can access via a web interface that allows you to save input (i.e. live code) and output (i.e. code execution results / evaluated code output) of interactive sessions as well as important notes needed to explain the methodology and steps taken to perform specific tasks (i.e data analysis).

Threat Hunter Playbook project

What is a runbook?

Throughout this blog post, I will use the term Jupyter Notebook, notebook, and runbook inter-changeably. A Runbook consists of a series of conditional steps to perform actions, such as data enrichment, threat containment, and sending notifications, automatically as part of the incident response or security operations process.

What is Hasicorp Vault?

HashiCorp Vault is an open-source tool for managing secrets. Application identity management with Vault enables applications and machines to automatically create, change, and rotate secrets needed for communications, services, scripts, etc. Additionally, Vault enables administrators to manage applications and machines by providing access control over different secrets.

Spin up/configure Vault

Step 0: Spin up Vault with Docker

This blog post assumes you have Vault setup, if you do not, take a look at my blog post: Install/Setup Vault for PKI + NGINX + Docker – Becoming your own CA, specifically steps 0-3. Below commands to set up a test instance of Vault with Docker.

git clone https://github.com/CptOfEvilMinions/Docker-Vault.git
cd Docker-Vault
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout conf/tls/<name>.key -out conf/tls/<name>.crt
docker-compose build
docker-compose up -d

Step 1: Login into Vault as root using CLI

export VAULT_ADDR=https://<Vault IP addr or Vault FQDN>
vault login
1. Enter root token

Step 2: Create KV secrets store for incident response

vault secrets enable -version=2 -path=secrets kv
1. Create a KV (key-value) store at the following path secrets/

Step 3: Create Vault policies

For this blog post, we are going to keep things simple. We are going to create two policies with the names ir-read-only and ir-admins. The ir-read-only policy grants all users who have this policy list and read-only access to secrets. The ir-admins policy grants all users who have this policy the ability to create, update, delete, list, and read access to secrets.

vault policy write ir-admin conf/vault/ir-admin.hcl
vault policy write ir-read-only conf/vault/ir-read-only.hcl

Step 4: Create Vault incident response group

vault write identity/group name="incident-response-juniors" policies=ir-read-only metadata=team="Incident response"
1. Save the group ID for later
vault write identity/group name="incident-response-seniors" policies=ir-admin metadata=team="Incident response"
1. Save the group ID for later

Step 5: Create Vault users and associated entity

To keep this blog post simple and brief I am going to use the userpass authentication and create an entity for that single auth backend. For enterprise deployments, I would recommend integrating Vault with your single-sign-on solution for federated access or LDAP server. If you would like more information on integrating these auth backends with Vault, please see my blog post here.

We are going to create two users which are Alice and Bob. Alice is a junior on the incident response team and will be added to the incident-response-junior group which has the read-only policy attached, which ONLY allows users to read the secrets. Bob is a senior on the incident response team and will be added to the incident-response-senior group to have the admin policy which allows users to create, update, delete, list, and read secrets. Lastly, the passwords below are weak for the sake of this blog post, please implement better passwords and/or authentication methods.

vault auth enable userpass
1. Enable userpass as an authentication method for Vault
vault auth list
1. Copy Accessor IDs for userpass auth backend
vault write auth/userpass/users/alice password=foo
1. Create a user account with the username of alice
vault write identity/entity name="alice" metadata=team="Incident response"
1. Create an entity with a metadata field containing the team’s name
2. Copy the entity’s ID
vault write identity/entity-alias name="alice" canonical_id="<ENTITY_ID>" mount_accessor="<userpass accessor ID>"
1. Add the userpass auth backend account to the entity
vault write auth/userpass/users/bob group=incident-response-seniors password=foo
1. Create a user account with the username of bob
vault write identity/entity name="bob" metadata=team="Incident response"
1. Create an entity with a metadata field containing the team’s name
2. Copy the entity’s ID
vault write identity/entity-alias name="bob" canonical_id="<ENTITY_ID>" mount_accessor="<userpass accessor ID>"
1. Add the userpass auth backend account to the entity

Step 6: Create the incident response groups

vault write identity/group name="incident-response-juniors" policies="ir-read-only" metadata=team="Incident Response"
1. Create a group for junior incident responders with the read-only policy
vault write identity/group name="incident-response-juniors" member_entity_ids="<ENTITY_ID>"
1. Add the entity to the group
vault write identity/group name="incident-response-seniors" policies="ir-admin" metadata=team="Incident Response"
1. Create a group for junior incident responders with the admin policy
vault write identity/group name="incident-response-seniors" member_entity_ids="<ENTITY_ID>"
1. Add the entity to the group

Testing user accounts and policies

Log in as a new user with Vault CLI

vault login -method=userpass username=bob
1. Enter password
vault kv put secrets/incident-response/jupyter-notebooks vti-api-key="abc123"
vault kv get secrets/incident-response/jupyter-notebooks
vault login -method=userpass username=alice
1. Enter password
vault kv get secrets/incident-response/jupyter-notebooks
vault kv patch secrets/incident-response/jupyter-notebooks vti-api-key="xyz789"
1. Attempt to update secret
2. The attempt should fail as expected

Jupyter Notebooks

Code review: Retrieveieng secrets from Vault

This blog post and all this work to setup Vault boils down to these 14 lines of Python code, which provides the ability to retrieve secrets from Vault. This Python function GetVaultSecrets() takes in the following arguments: Vault Address which is the URL location of Vault, Vault secret path which specifies the location of where the secrets are stored, and a Vault token which authenticates the request. First, we craft an HTTP header which contains the Vault token to authenticate the request. Second, the URL is crafted which is the combination of the Vault URL address and the path to the secrets. Third, an HTTP POST request is made to Vault to request a list of secrets. Lastly, if the Vault token provided has the permissions to access these secrets, it will return a list of secrets.

Code review: Using Vault secret

The code in the screenshot above calls the Python function GetVaultSecrets(), which is the function we discussed in the previous section. The return from this function is stored in a variable named vault_secrets. Next, the script prompts the user to enter a SHA256 file hash. Next, the script executes the function to obtain results from VirusTotal which takes the following arguments: a VirusTotal API key (vault_secrets['vti-api-key']) extracted from vault_secrets and the SHA256 file hash provided by the user. The result from Virustotal is stored into vti_results which is used to calculate the malicious score for the SAH256 file hash.

Init

git clone https://github.com/CptOfEvilMinions/Vault-Jupyter-Notebooks
cd Vault-Jupyter-Notebooks/playbooks
virtualenv -p python3 venv
source venv/bin/activate
pip3 install -r requirements.txt

Run VirusTotal Jupyter Notebook

All of these notebooks/runbooks are demonstrating with Python code how to integrate Vault and Jupyet Notebooks to retrieve secrets but also accomplish an incident response or threat hunting related goal. The goal of this first runbook is to automate the process of obtaining an API key for Virustotal from Vault to request the malicious score of a SHA256 file hash provided by the user.

jupyter-notebook vti_playbook.ipynb
Select “Run” at the top
Enter an SHA 256 file hash
1. Hash: 423a0799efe41b28a8b765fa505699183c8278d5a7bf07658b3bd507bfa5346f

Run Elasticsearch runbook

The goal of this second runbook is to automate the process of obtaining credentials for Elasticsearch to search an index for indicators of Powershell Empire.

vault kv patch secrets/incident-response/jupyter-notebooks elasticsearch-hostname="<ES hostname>"
vault kv patch secrets/incident-response/jupyter-notebooks elasticsearch-username="<ES username>"
vault kv patch secrets/incident-response/jupyter-notebooks elasticsearch-password="<ES password>"
1. 1. Submit the Elasticsearch hostname, username, and password to Vault
jupyter-notebook elastic_playbook.ipynb
Enter an index to search

Run threat hunting runbook

The purpose of this third runbook is to demonstrate a more complex runbook to demonstrate the power of integrating Vault and Jupyter Notebooks. The goal of this runbook is to query a Windows environment for rogue Windows services using Kolide + Osquery and Virustotal. First, as I will discuss later, the run book requests your Kolide username + password and the URL where Kolide is hosted. Second, the notebook utilizes the Kolide credentials provided to generate a Kolide JWT (JSON web token). Thirdly, the notebook uses the Kolide API to: creates a live query (using the query provided in the runbook) and retrieves the results of the live query. Lastly, the runbook lookups each SHA256 hash for each service binary in Virustotal. If that hash is NOT known to Virustotal or has a malicious score that exceeds the threshold (default is 20) it is printed to the console for further analysis.

I will elaborate more on this in the discussion section but sometimes you need to assess the risk of storing a secret in Vault. Kolide has authentication but it doesn’t have role-based access control (RBAC) to limit what a user can do. Essentially all users on the Kolide Fleet platform are admins. Therefore, if an unauthorized user was able to obtain the credentials/API key from Vault for Kolide they could perform adverse actions. Jupyter Notebooks can be configured to accommodate these scenarios.

For this runbook, I request the user’s username + password for Kolide and URL for where Kolide is hosted. The existence of these secrets is only during the runtime of this runbook. Again, the goal of this is to demonstrate that if it is too risky to store a secret in Vault, alternative methods such as environment variable, requesting the user to enter the secret manually, or reading it from a file can be used.

jupyter-notebook osquery_kolide_th_playbook.ipynb
Select “Run” at the top
Ente Kolide URL
Enter Kolide username
Enter Kolide password

Discussion

Alternatives to a local Vault instance?

Unfortunately, hosting your own instance of Vault may not be the appropriate solution for everyone. Below I have listed some alternative solutions:

Hasicorp Vault Cloud – Hasicorp provides Vault as a service
Azure vault – Provides Vault as a service
LastPass command line – LastPass has a command-line tool similar to Vault’s command-line tool that can provide similar functionality.

Service accounts vs. user accounts

Another thing to consider before loading Vault full of secrets is the type of secret. What I mean by type is the difference between a service account and a user account. Typically, a user account is created for a specific user and granted specific permissions for that user. Service accounts may be granted broader access because it is an account that will be used by several users or services. Also, users will typically leave a company after X amount of time but a service account may prevail till the end of time.

In this scenario, Vault will “service account” API keys and credentials for service accounts because multiple users will be using the same secrets. It’s important to analyze the risk if an unauthorized party obtains access to these secrets. Questions such as “will this unauthorized party have visibility to sensitive information – like password command-line monitoring that contains passwords” or “could this access be used to distribute ransomware”.

JupyterHub

Typically, Jupyter notebooks will be run from a user’s machine but JupyterHub allows the notebooks to be served by a server. This means instead of having to have a git repo everyone on the team needs to keep updated with new releases, you can host the latest releases on JupyterHub for all user’s to access. This blog post did not cover this functionality but it is something to consider for large teams or SOCs.

Lessons learned

I am currently reading a book called “Cracking the Coding Interview” and it is a great book. One interesting part of the book is their matrix to describe projects you worked on and the matrix contains the following sections which are: challenges, mistakes/failures, enjoyed, leadership, conflicts, and what would you do differently. I am going to try and use this model at the end of my blog posts to summarize and reflect on the things I learn. I don’t blog to post things that I know, I blog to learn new things and to share the knowledge of my security research.

New skills/knowledge

First time writing a Jupyter Notebook
Learned how to implement the Vault agent but it didn’t support my authentication methods
Used the latest version of Metasploit to pop a shell on a Windows machine and create a malicious service
Re-learned how to use the Python Elasticsearch client library

Challenges

The Vault agent doesn’t support LDAP as an auto auth method

HoldMyBeer

Cause every great story starts with "Hold my beer"

Integrating Vault secrets into Jupyter Notebooks for Incident Response and Threat Hunting