A common question in the #Kolide channel in the Osquery Slack is how to use the Kolide Fleet API. Kolide Fleet is written in GoLang and utilizes the GoKit framework to build the application. Therefore, almost every action that can be performed via the WebGUI is an API call. This blog post is going to demonstrate how to use the Kolide Fleet API to perform actions such as creating a live query and obtaining the results using Python websockets, obtaining the Osquery enroll secret, and creating a saved query. In addition to the API, this blog post will demonstrate how to use the Fleetctl command-line tool to perform the same actions. Lastly, this blog post includes an Ansible playbook to automate deploying Oquery agents and registering them with Kolide.

WARNING

This blog post is demonstrating how to use the Kolde API but it does not supersede the official documentation released by Kolide. Please use this blog post as a guide and not as official documentation.

WARNING

Background

What is GoKit?

Go kit is a programming toolkit for building microservices (or elegant monoliths) in Go. We solve common problems in distributed systems and application architecture so you can focus on delivering business value.

Spin up Kolide with Docker

1. git clone https://github.com/CptOfEvilMinions/Kolide-Docker.git
2. cd Kolide-Docker
3. docker-compose build
4. docker-compose run --rm kolide fleet prepare db --config /etc/kolide/kolide.yml
5. docker-compose up -d
6. For the remainder of this setup please refer to this README

API overview

Since Kolide Fleet uses the GoKit framework, you can think of the handler.go file as your API guide. At first, this file looks intimidating but once you understand how to read the code it becomes pretty simple. To start, scroll down to line 418 which is the start of the following function: func attachKolideAPIRoutes(r *mux.Router, h *kolideHandlers). In this function Kolide inits every API URL endpoint.

On line 419 there is the HTTP handler for the login function (h.Login), which requires a POST request (Methods("POST")), and the HTTP request should be sent to the following URI: /api/v1/kolide/login. At line 454 (screenshot below) is the HTTP handler to get information about a query (h.GetQuery), which requires a GET request (Methods("GET")), the HTTP request should be sent to the following URL: /api/v1/kolide/queries/{id}, and the URL requires an ID to be specified. If you click h.GetQuery it will generate a pop-up like the screenshot below and you want to select the section that has a GoLang function that starts with func (c *Client).

This will redirect your browser to the GoLang code (screenshot below) responsible for handling requests for the following URI: /api/v1/kolide/queries/{id}. The GetQuery function takes in a string parameter which is the ID specified in the URL. Next, the Kolide API generates the URL and queries that endpoint. Before the request makes it to the specified URI, the request is first authenticated. If the request is successful, the result of the request is returned to the user: return responseBody.Spec, nil. Hopefully, this provided a high overview of how to navigate the handler.go file, which is your guide to the Kolide API.

Authenticating to Kolide

Intro

Kolide implements JSON web tokens (JWT) to authenticate each request to perform an action with the Kolide Fleet API.  This blog post is not going to cover how JWTs work but essentially think of it as an “API key”. Retrieving a JWT is the first roadblock users have when they want to use the Kolide API. Below, I am going to demonstrate how to obtain a JWT with a CURL request.

Request a JWT

1. curl -k -X POST https://<Kolide FQDN>:<Port>/api/v1/kolide/login -d '{"Username": "<admin username>", "Password": "<admin password>"}'
2. export KOLIDE_TOKEN=$(curl -k -X POST https://<Kolide FQDN>:<Port>/api/v1/kolide/login -d '{"Username": "<admin username>", "Password": "<admin password>"}' | jq -r '.token')
   1. Save the Kolide token to an environment variable
3. echo $KOLIDE_TOKEN

Kolide API actions

Request Osquery enroll secret

1. curl -k -X GET https://<Kolide FQDN>:<Port>/api/v1/kolide/spec/enroll_secret -H "Authorization: Bearer ${KOLIDE_TOKEN}"
   

Create a query with Kolide API

1. curl -k -X POST https://<Kolide FQDN>:<port>/api/v1/kolide/queries -H "Authorization: Bearer ${KOLIDE_TOKEN}" -d '{"query": "SELECT * FROM osquery_info", "name": "Test"}'
   

Execute saved query with Kolide API

The first CURL statement will request a saved query to be executed by using Kolide label IDs and Kolide hosts IDs. The second CURL statement will request a saved query to be executed by using Kolide label-friendly names and Kolide hosts friendly names.

1. curl -k -X POST https://<Kolide FQDN>:<port>/api/v1/kolide/queries/run_by_names -H "Authorization: Bearer ${KOLIDE_TOKEN}" -d '{"query": "SELECT * FROM osquery_info", "selected": {"Labels": ["<Kolide label friendly names comma seperated>"], "Hosts": ["<Kolide label friendly names comma seperated>"] }}'
   1. Using IDs
2. curl -k -X POST https://<Kolide FQDN>:<port>/api/v1/kolide/queries/run -H "Authorization: Bearer ${KOLIDE_TOKEN}" -d '{"query": "<Name of saved query>", "selected": {"Labels": [<Kolide label IDS commaa seperated>], "Hosts": [<Kolide host IDS comma seperated>] }}'
   1. Using Kolide friendly names

Execute live/distributed query with Kolide API

The first CURL statement will create a live query by using Kolide label IDs and Kolide hosts IDs. The second CURL statement will create a live query by using Kolide label-friendly names and Kolide hosts friendly names.

1. curl -k -X POST https://<Kolide FQDN>:<port>/api/v1/kolide/queries/run -H "Authorization: Bearer ${KOLIDE_TOKEN}" -d '{"query": "SELECT * FROM osquery_info", "selected": {"Labels": [<Kolide label IDS commaa seperated>], "Hosts": [<Kolide host IDS comma seperated>] }}'
   1. Using IDs
2. curl -k -X POST https://<Kolide FQDN>:<port>/api/v1/kolide/queries/run_by_names -H "Authorization: Bearer ${KOLIDE_TOKEN}" -d '{"query": "SELECT * FROM osquery_info", "selected": {"Labels": ["<Kolide label friendly names comma seperated>"], "Hosts": ["<Kolide label friendly names comma seperated>"] }}'
   1. Using Kolide friendly names

Get query results with Kolide API

Getting the results is not as simple as sending an HTTP GET request because the endpoint for retrieving the results is a WebSocket. At first, it didn’t make sense why Kolide did it this way but once you understand the reasoning it makes sense. When you submit a query to Kolide all the endpoints are checking in at different times to get the latest tasks therefore all the results will come in at different times. A WebSocket allows the client to continually “poll” the Kolide server for the latest results as they come in.

Unfortunately, as stated above, it’s not a simple CURL request to get these results. To demonstrate how this works I created a Python script that is thoroughly commented to help engineers understand how to create automations with the Kolide API. I will describe at a high level the process:

1. Generate a Kolide JWT as demonstrated above
   1. line: 127: def authenticateToKolide()
2. Create a live query as demonstrated above
   1. line 103: def createKolideLiveQuery()
3. Create a Python WebSocket, which will be used for the remainder of the process
   1. line 57: ws = create_connection(kolide_websocket_uri, sslopt={"cert_reqs": ssl.CERT_NONE})
4. Send our Kolide JWT to the server to authenticate our WebSocket
   1. line 61: ws.send(generateJSONAuthHeader(kolide_token))
5. Send the Kolide query campaign ID, which identifies the query we want to retrieve results from
   1. line 65: ws.send(generateKoldieQueryCampaignIDJSONpayload(koldie_query_campaign_id))
6. Python script continually “polls” the server for new results until the server returns '{"status": "finished"}'
   1. line 92: if result.get("type") == "status" and result.get("data").get("status") == "finished":
7. Close WebSocket
   1. line 95: ws.close()
8. Print results to console for user
   1. line 98: print (result_list)

Next, I will demonstrate how to use the Python script provided to obtain results of a query

1. git clone https://github.com/CptOfEvilMinions/BlogProjects
2. cd BlogProjects/kolide-api-ansible
3. virtualenv -p python3 venv
4. source venv/bin/activate
5. pip3 install -r requirements.txt
6. python3 kolide_websocket_client.py --campaign_id <X>
   1. Enter Kolide username
   2. Enter Kolide password
   3. Enter Kolide URL

Submit a live query and get results with Python script

1. python3 kolide_websocket_client.py
   1. Enter Kolide username
   2. Enter Kolide password
   3. Enter Kolide URL
   4. Enter a list of hosts that are comma-separated
   5. Enter a list of labels that are comma-separated

Fleetctl

Download and init Fleetctl

1. wget https://github.com/kolide/fleet/releases/download/3.2.0/fleet.zip
2. unzip fleet.zip
3. mv <OS - macOS:darwin, linux:linux>/fleetctl /usr/local/bin/fleetctl
4. fleetctl config set --address https://<Kolide FQDN>:<port> --tls-skip-verify
   1. Set the Fleet API address
   2. Only specify --tls-skip-verify if you have a self-signed certificate
5. fleet login
   1. Enter Kolide user e-mail
   2. Enter Kolide user password

Create a live query

1. fleetctl query --query "<Osquery query>" --hosts <Kolide friendly name>

Creating a pack

1. wget https://raw.githubusercontent.com/osquery/osquery/master/packs/ossec-rootkit.conf
2. fleetctl convert -f ossec-rootkit.conf > ossec-rootkit--pack-fleet.yaml
   1. Convert JSON to YAML
3. fleetctl apply -f ossec-rootkit--pack-fleet.yaml
   

GOquery

This is probably one of the coolest features of Fleetctl. GOquery allows you to connect the endpoint in real-time and run queries as if you were at the console using osqueryi. To demonstrate this feature I will connect to a Windows host, move around the file system, and request information about the Osquery instance running.

1. fleetctl goquery
2. .connect <Osquery UUID>
3. cd C:\Windows\debug
4. ls
5. .query SELECT * FROM SELECT * FROM osquery_info

Setup Fleetctl with BURP

To determine some of the API calls above, I ran the Fleetctl tool and captured the HTTP payloads with BURP.

1. Download and install BURP
2. Start BURP and create an intercepting proxy on localhost:8080
3. HTTP_PROXY=http://127.0.0.1:8080 fleetctl query --query "SELECT * FROM osquery_info" --hosts ubuntuvm
4. Go to Burp > Proxy > WebSockets history

Automate enrolling Osquery endpoints

Code break down

This section will demonstrate how to use Ansible to automate enrolling Osquery endpoints to Kolide. First, Ansible starts by installing Osquery on a set of machines. Second, Ansible will copy the Osquery.flags file. Third, Ansible reads the controller’s (machine running the Ansible playbook) environment variables for a variable named KOLIDE_TOKEN. Ansible will use the JWT assigned to this environment variable to authenticate the request to request the Osquery enroll secret from Kolide. Lastly, Ansible will start the OsqueryD service, which will call back to Kolide to register itself.

Configure Ansible playbook

1. git clone https://github.com/CptOfEvilMinions/BlogProjects
2. cd BlogProjects/kolide-api-ansible
3. mv group_vars/all.yml.example group_vars/all.yml
4. vim group_vars/all.yml and set:
   1. kolide_hostname – Set to the FQDN or IP address of Kolide
   2. kolide_port – Set to the port Kolide is listening on
5. vim host.ini add a list of IP addresses to install Osquery on under [osquery_linux]

Configure Ansible playbook for Windows

1. mv group_vars/windows.yml.example group_vars/windows.yml
2. vim group_vars/windows.yml and set:
   1. ansible_user – Set to a Windows admin user
   2. ansible_password – Set password for user
3. vim host.ini add a list of IP addresses to install Osquery on under [osquery_windows]

Run Ansible playbook

1. KOLIDE_TOKEN=$(curl -k -X POST https://<Kolide FQDN>:<Port>/api/v1/kolide/login -d '{"Username": "<admin username>", "Password": "<admin password>"}' | jq '.token')
2. ansible-playbook -i hosts.ini deploy_osquery_linux.yml -u superadmin -K
   1. Deploy Osquery on Linux
3. ansible-playbook -i hosts.ini deploy_osquery_windows.yml
   1. Deploy Osquery on Windows

Discussions

NO RBAC support

Kolide Fleet does not support role-based access control (RBAC). This means all users are essentially admin users. This means you need to be careful with the operation and handling of Kolide credentials and JWTs.

Lessons learned

I am currently reading a book called “Cracking the Coding Interview” and it is a great book. One interesting part of the book is their matrix to describe projects you worked on and the matrix contains the following sections which are: challenges, mistakes/failures, enjoyed, leadership, conflicts, and what would you do differently. I am going to try and use this model at the end of my blog posts to summarize and reflect on the things I learn. I don’t blog to post things that I know, I blog to learn new things and to share the knowledge of my security research.

New skills/knowledge

• Learned how to utilize the Kolide API
• Learned more about WebSockets
• How to implement a Python client that uses WebSockets

Challenges

• Learning how to interact with WebSockets. I had to use BURP to intercept the traffic from the Fleetctl CLI tool to Kolide to understand the payloads.

What You’d Do Differently

• Created CURL statements or Python code for creating/updating Osquery query packs. Osquery packs are written in JSON and Kolide Fleet only accepts YAML files. The Fleetctl CLI tool can convert the pack from JSON to YAML but at that point, you might as well use the Fleetctl tool to upload the file as well.

References

• Github – Koldie: query_mocks.js
• Osquery schema
• Github – Kolide – handler.go
• Github – Kolide – pack_mocks.js 
• ossec-rookit-pack.json
• JSON Web Token (JWT) explained
• Github – Kolide – client_live_query.go