Getting started with FleetDM v3.6.0

The purpose of this blog post is to provide multiple methods on how to install/setup FleetDM, how to deploy Osquery, and demonstrate how to use features of FleetDM + FleetCTL. This blog post generated an Ansible playbook, Docker-composes for Swarm and non-swarm, and manual instructions for installing FleetDM on Ubuntu 20.04. Additionally, there are Ansible playbooks for deploying the Osquery agent on Windows and Ubuntu with manual instructions as well. Lastly, I will end by demonstrating how to use the FleetDM WebGUI and FleetCTL tool to manage FleetDM and interact with your Osquery agents.

Background

What is Fleet FleetDM?

Fleet is the most widely used open source osquery manager. Deploying osquery with Fleet enables programmable live queries, streaming logs, and effective management of osquery across 50,000+ servers, containers, and laptops. It’s especially useful for talking to multiple devices at the same time.

Important note: Common name match for Osquery cert check

This blog post will assume you have the knowledge and capability to create a DNS A record that points to where FleetDM is being hosted. When the Osquery agent connects to FleetDM it will verify that the common name in the public certificate being served by FleetDM matches the common name specified in the osquery.flags file: --tls_hostname=<FleetDM FQDN>. Below I have included a screen to demonstrate what I mean. Not only does the common name have to match but the public certificate being served by FleetDM and the local certificate saved on the endpoint must match as well. If this criterion is not met then Osquery will generate the following error: Request error: certificate verify failed.

Install FleetDM with Docker-compose v2.x

WARNING

The Docker-compose v2.x setup is for development use ONLY. The setup contains hard-coded credentials in configs and environment variables. For a more secure Docker deployment please skip to the next section to use Docker Swarm which implements Docker secrets.

WARNING

  1. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout conf/tls/fleet.key -out conf/tls/fleet.crt
    1. Generate TLS private key and public certificate
  2. git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
  3. cd FleetDM-Automation
  4. vim .env and set
    1. FLEETDM_VERSION – Version of FleetDM to use – Only change if you want to use a different version
    2. FLEETDM_MYSQL_USER – Name of the MySQL user for FleetDM
    3. FLEETDM_MYSQL_PASSWORD – Password for the MySQL user for FleetDM
    4. FLEETDM_MYSQL_DATABASE – Name of the MySQL database for FleetDM
    5. FLEETDM_MYSQL_PASSWORD – Password for the MySQL root user
    6. FLEETDM_JWT_KEY – The JWT token
      1. Generate a random key with openssl rand -base64 32
  5. docker-compose build
  6. docker-compose run --rm fleet fleet prepare db --config /etc/fleet/fleet.yml
  7. docker-compose up -d

Install FleetDM with Docker-compose v3.x (Swarm)

Unfortunately, Docker-compose version 3.X (DockerSwarm) doesn’t interact with the .env file the same as v2.X. The trade-off is a more secure deployment of the FleetDM stack because secrets are not hardcoded in configs or stored in environment variables. Below we create Docker secrets that will contain these sensitive secrets to be used by the Docker containers.

Furthermore, any changes such as changing the container version or the MySQL database name in the .env file will have no effect. These settings need to be changed in the docker-compose-swarn.yml file and the same changes may need to be carried over to the FleetDM config located: conf/docker/fleetdm/fleetdm-swarm.yml. Lastly, another benefit of Docker Swarm is we can have multiple instances (replicas) of FleetDM running for high-availability.

  1. git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
  2. cd FleetDM-Automation
  3. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout conf/tls/fleet.key -out conf/tls/fleet.crt
    1. Generate TLS private key and public certificate
  4. openssl rand -base64 32 | tr -cd '[:alnum:]' | docker secret create fleetdm-jwt-key -
    1. Generate a JWT key and create a secret named fleetdm-jwt-key
  5. openssl rand -base64 32 | tr -cd '[:alnum:]' | docker secret create mysql-root-password -
    1. Generate random password containing only alphanumeric characters for the MySQL root password
  6. openssl rand -base64 32 | tr -cd '[:alnum:]' | docker secret create mysql-fleetdm-password -
    1. Generate random password containing only alphanumeric characters for the MySQL FleetDM user password
  7. docker stack deploy -c docker-compose-swarm.yml fleetdm
  8. docker service logs -f fleetdm_fleet
    1. The first time these containers are started they will error out until the MySQL database initializes

Install FleetDM on Ubuntu 20.04 with Ansible

Init playbook

  1. git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
  2. cd FleetDM-Automation
  3. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout conf/tls/fleet.key -out conf/tls/fleet.crt
    1. Generate TLS private key and public certificate
  4. pip3 install ansible
  5. vim hosts.ini and add IP address under [fleetdm]
  6. vim all.yml and set:
    1. base_domain – The domain for your network and the base domain of the FQDN
    2. timezone – OPTIONAL – Change the default timezone of UTC +0
  7. openssl rand -base64 32
  8. Copy the output from the command
  9. vim fleetdm.yml and set:
    1. fleetdm_jwt –  Set this to the random string generated by the OpenSSL command
    2. mysql_root_password – Set the root password for MySQL
    3. mysql_fleetdm_password –  Set the password for FleetDM MySQL user

Run playbook

  1. ansible-playbook -i hosts.ini deploy_fleetdm.yml -u <user> -K
    1. Enter the password for the user

Manual install of FleetDM 3.6.0 on Ubuntu 20.04

Init host

  1. ssh into VM
  2. sudo su
  3. timedatectl set-timezone Etc/UTC
    1. Set the system timezone to UTC +0
  4. apt update -y && apt upgrade -y && apt install net-tools git wget unzip -y && reboot
    1. Update system and reboot

Install/Setup Redis 5.5.0

  1. apt install redis-server=5:5.0.7-2
  2. systemctl enable redis-server
  3. systemctl start redis-server
  4. netstat -tnlp | grep redis

Install/Setup MySQL 8.0

  1. apt install mysql-server-8.0 python3-mysqldb python3-pip -y
  2. systemctl start mysql
  3. netstat -tnlp | grep mysqld
    1. Check that MySQL is listening on port 3306
  4. secure_installation
    1. Enter “y”
    2. Enter “2” for a strong password
      1. Secure Password Generator
    3. Enter a strong password
    4. Enter “y” to continue with the password provided
    5. Enter “y” to remove anonymous user
    6. Enter “y” to disallow root login remotely
    7. Enter “y” to remove the test database
    8. Enter “y” to reload privileged tables
  5. systemctl enable mysql
  6. Create another secure password
  7. mysql -u root -p
    1. Enter the password from above
  8. CREATE DATABASE fleetdm;
    1. Create FleetDM database
  9. CREATE USER 'fleetdm'@'localhost' IDENTIFIED BY '<Password>';
    1. Create FleetDM MySQL user
  10. GRANT ALL PRIVILEGES ON fleetdm.* TO 'fleetdm'@'localhost';
    1. Provide FleetDM the permissions to interact with the FleetDM database
  11. exit
  12. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/mysql/fleetdm.cnf -O /etc/mysql/conf.d/fleetdm.cnf
    1. Download MySQL settings config
  13. systemctl restart mysql

Install/Setup FleetDM 3.6.0

  1. adduser fleetdm --system --no-create-home
    1. Create a user
  2. groupadd fleetdm
    1. Create a group
  3. usermod -aG fleetdm fleetdm
    1. Add the fleetdm user as a member of the fleetdm group
  4. cd /tmp/ && wget https://github.com/fleetdm/fleet/releases/download/3.6.0/fleet.zip
  5. unzip fleet.zip
  6. mv linux/fleet /usr/local/bin/fleet
    1. Move FleetDM binary to the local bin directory
  7. mkdir /etc/fleetdm
  8. chown root:root -R /etc/fleetdm
    1. Create a config directory for FleetDM
  9. mkdir /var/log/osquery
  10. chown fleetdm:fleetdm -R /var/log/osquery 
    1. Create a directory to record the results of Osquery endpoints
  11. openssl req -newkey rsa:2048 -nodes -keyout /etc/ssl/private/fleetdm.key -x509 -days 365 -out /etc/ssl/certs/fleetdm.crt
    1. Generate self-signed private key and public certificate for FleetDM
  12. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/fleetdm/fleetdm.yml -O /etc/fleetdm/fleetdm.yml
    1. Download FleetDM config
  13. chown root:fleetdm /etc/fleetdm/fleetdm.yml
  14. chmod 640 /etc/fleetdm/fleetdm.yml
  15. sed -i "s/{{ fleetdm_jwt }}/$(openssl rand -base64 32 | tr -cd '[:alnum:]')/g" /etc/fleetdm/fleetdm.yml
    1. Generate a random JWT key and set the value in the FleetDM config
  16. sed -i "s/{{ fleetdm_db_name }}/<FleetDM database name>/g" /etc/fleetdm/fleetdm.yml
    1. Set the database name
  17. sed -i "s/{{ fleetdm_username }}/<FleetDM username>/g" /etc/fleetdm/fleetdm.yml
    1. Set the MySQL username
  18. sed -i "s/{{ fleetdm_password }}/<FleetDM password>/g" /etc/fleetdm/fleetdm.yml
    1. Set the MySQL password
  19. /usr/local/bin/fleet prepare db --config /etc/fleetdm/fleetdm.yml
    1. Init FleetDM database
  20. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/fleetdm/fleetdm-systemd.service -O /etc/systemd/system/fleetdm.service
    1. Download SystemD config for FleetDM
  21. systemctl enable fleetdm
  22. systemctl start fleetdm
  23. netstat -tnlp | grep fleet
    1. Check that MySQL is listening on port 8080

Install NGINX

  1. apt install nginx -y
  2. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/nginx/nginx.conf -O /etc/nginx/nginx.conf
  3. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/nginx/nginx_fleetdm.conf -O /etc/nginx/conf.d/fleetdm.conf
    1. Download NGINX configs
  4. systemctl enable nginx
  5. systemctl restart nginx
  6. netstat -tnlp | grep nginx
    1. Check that MySQL is listening on port 80 and 443

Setup UFW

  1. ufw allow OpenSSH
  2. ufw allow 'Nginx HTTP'
  3. ufw allow 'Nginx HTTPS'
  4. ufw enable

Setup FleetDM via WebGUI

  1. Open a web browser to FleetDM
  2. Setup user
    1. Enter a <username> into Username
    2. Enter a <password> into Password
    3. Enter an <e-mail> for the admin user
    4. Select “Next”
  3. Organization details
    1. Enter organization name
    2. Enter the organization URL
    3. Select “Next”
  4. Set Fleet URL
    1. Accept the default Fleet URL, unless you need to change it
    2. Select “Submit”
  5. Select “Finish”

Deploy Osquery v4.6.0 agent

Osquery.flags vs. Osquery.conf

A common struggle for new Osquery users is knowing when a config option should be set in osquery.flags or osquery.conf. First, I would like to provide this Youtube video which is a recording of an Osquery office hours meeting where I asked this question to the core developers of Osquery and the discussion that followed. @zwass started the discussion by stating that osqueryd --help outputs two sections of config options. The first set of config options must be placed into the osquery.flags file and the second set can be specified in both config files. @teddyreedv stated that the osquery.flags file is meant to define the start-up behavior of Osquery. Therefore, the config options that are specified in osquery.flags are static throughout the Osquery runtime but the config options specified in osquery.conf are dynamic throughout the runtime. For example, in the osqeury.flags file you would specify which user the Osquery agent should run as, which is not something you can change during runtime.

Furthermore, there are additional considerations to understand when connecting Osquery to a fleet manager/TLS endpoint like FleetDM. The first consideration is the fleet manager can not modify the osquery.flags file; however, if a config option can be set in both config files the value specified in osquery.conf will override the value in osquery.flags. What this ultimately means is in this scenario, the server gets the final say in how Osquery operates. For example, you can specify the --max_read option to 1MB in osquery.flags but the server could override this setting by setting this value to 5MB in osquery.conf. To conceptualize this, @zwass made a good comment that the order of operations when loading a config starts with osquery.flags and then osquery.conf is loaded, which overrides any settings specified by osquery.flags

To this point, the discussion has only provided how Osquery operates and did not provide specific guidance on how to construct config files for new users. It’s important to understand how Osquery ingests these config options to create the optimal configs for your environment. My guidance for new users for constructing Osquery configs is to create separate configs that target specific environments (QA vs. prod vs. cloud) and for each target OS (Linux, macOS, and Windows). First, start with constructing the osquery.flags file and keep it as small as possible, which means only specifying the necessary flags to start Osquery in the desired state. For example, if Osquery is connecting to a fleet manager you need to specify the necessary flags such as: --enroll_secret, --tls_server_cert, and --tls_hostname to connect to a fleet manager to for further instructions.

For constructing osquery.conf I still recommend generating a config file for each environment and OS flavor. My reasoning for this is you may want to monitor multiple sets of machines differently. For example, the data you might collect from a set of Windows user endpoints is not the same set of data you would collect from Linux machines in AWS. Additionally, not all Osquery tables are cross-platform so on Windows you may collect the domain information with the ntdomains table, which does not exist for Linux. Conversely, on Linux, you may collect all the configured sudo users with the sudoers table. In addition to targeting the OS, the environment type may also play a factor in deciding how you monitor the machine. In a production environment, you may need to squeeze all the performance you can out of a system to keep your platform fast. Therefore, in a production environment, you may set strict Osquery watchdog settings so Osquery doesn’t impact system performance. Lastly, one of the ways I learned to construct my Osquery.conf is by looking at examples on Github such as this repo: palantir/osquery-configuration.

Manual Osquery install on Ubuntu

  1. SSH into UbuntuVM
  2. export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
  3. sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys $OSQUERY_KEY
  4. sudo add-apt-repository 'deb [arch=amd64] https://pkg.osquery.io/deb deb main'
  5. sudo apt-get update -y
  6. sudo apt-get install osquery -y
  7. openssl s_client -showcerts -connect <FleetDM FQDN>:443 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /etc/osquery/fleetdm.crt
    1. Write the FleetDM public certificate to disk
  8. Open a web browser to FleetDM and Login
    1. Select “Add new host” in the top right
    2. Copy the enrollment secret
  9. echo '<FleetDM enroll secret>' > /etc/osquery/osquery.key
  10. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/osquery/osquery_linux.flags -O /etc/osquery/osquery.flags
    1. Download Osquery.flags config
  11. sed -i 's/{{ fleetdm_fqdn }}/<FleetDM FQDN>/g' /etc/osquery/osquery.flags
  12. sed -i 's/{{ fleetdm_port }}/<FleetDM port - default 443>/g' /etc/osquery/osquery.flags
  13. systemctl restart osqueryd
  14. systemctl status osqueryd
    1. Make sure there are no errors

Manual Osquery install on Windows

  1. Login into Windows
  2. Open Powershell as Administrator
  3. Invoke-WebRequest -Uri https://pkg.osquery.io/windows/osquery-4.6.0.msi -OutFile $ENV:TEMP\osquery-4.6.0.msi -MaximumRedirection 3
    1. Download Osquery
  4. Start-Process $ENV:TEMP\osquery-4.6.0.msi -ArgumentList '/quiet' -Wait
    1. Install Osquery
  5. Open a web browser to FleetDM and Login
  6. Click “Add new host” in the top right
    1. Click “Download” under Enroll secret
    2. Click “Download” under Server Certificate
    3. Click “Download” under Flagfile
  7. cd ~/Downloads
  8. mv .\secret.txt 'C:\Program Files\osquery\secret.txt'
  9. mv .\fleet.pem 'C:\Program Files\osquery\fleet.pem'
  10. rm 'C:\Program Files\osquery\osquery.flags'
  11. mv .\flagfile.txt 'C:\Program Files\osquery\osquery.flags'
  12. Restart-Service -Name osqueryd

Automated Osquery install with Ansible on Windows and Ubuntu

Deploy on Ubuntu

Init playbook for Ubuntu

  1. export FLEETDM_TOKEN=$(curl -k -X POST https://<FleetDM FQDN>:<Port>/api/v1/kolide/login -d '{"Username": "<admin username>", "Password": "<admin password>"}' | jq -r '.token')
    1. Request a FleetDM JWT authentication token
    2. Store the JWT authentication token in an environment variable named FLEETDM_TOKEN
    3. For more information on this process see my blog post here: DEMYSTIFYING THE KOLIDE FLEET API WITH CURL, PYTHON, FLEETCTL, AND ANSIBLE
  2. vim group_vars/fleetdm.yml and set:
    1. fleetdm_fqdn – Set this value to the FQDN of FleetDM
    2. fleetdm_port – Set the port that FleetDM is exposed on – default port 443
  3. vim group_vars/osquery.yml and set:
    1. osquery_version – OPTIONAL – Set the version of Osquery you want to install – default 4.6.1
  4. vim hosts.ini add IP addresses for Linux hosts under [osquery_linux]

Run playbook

  1. ansible-playbook -i hosts.ini deploy_osquery_linux.yml -u <user> -K
    1. Enter password

Deploy on Windows

Init playbook for Windows

  1. export FLEETDM_TOKEN=$(curl -k -X POST https://<FleetDM FQDN>:<Port>/api/v1/kolide/login -d '{"Username": "<admin username>", "Password": "<admin password>"}' | jq -r '.token')
  2. vim group_vars/fleetdm.yml and set:
    1. fleetdm_fqdn – Set this value to the FQDN of FleetDM
    2. fleetdm_port – Set the port that FleetDM is exposed on – default port 443
  3. vim group_vars/osquery.yml and set:
    1. osquery_version – OPTIONAL – Set the version of Osquery you want to install
  4. vim group_vars/windows.yml and set:
    1. ansible_user – Set this to a Windows username
    2. ansible_password – Set this to a Windows password
  5. vim hosts.ini add IP addresses for Linux hosts under [osquery_windows]

Init Windows VM for Ansible

  1. Log into Windows VM
  2. Open Powershell as Administrator
  3. Set-ExecutionPolicy Unrestricted
    1. Enter “y”
  4. powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1'))"

Run playbook

  1. ansible-playbook -i hosts.ini deploy_osquery_windows.yml

Create a Live query

WebGUI

  1. Login into FleetDM
  2. Select “Queries” on the left
  3. Select “Create new query” in the top right
    1. Enter Get processes into the Query title field
    2. Enter SELECT * FROM processes into the SQL field
    3. Select MS Windows for Select targets
    4. Select “Run” and wait

FleetCTL

  1. cd /tmp && wget https://github.com/kolide/fleet/releases/download/3.2.0/fleet.zip
  2. unzip fleet.zip
  3. mv <OS - macOS:darwin, linux:linux>/fleetctl /usr/local/bin/fleetctl
  4. fleetctl config set --address https://<FleetDM FQDN>:<port> --tls-skip-verify
    1. Set the Fleet API address
    2. Only specify --tls-skip-verify, if you have a self-signed certificate
  5. fleet login
    1. Enter Kolide user e-mail
    2. Enter Kolide user password
  6. fleetctl query --query "<Osquery query>" --hosts <Kolide friendly name>

Create a saved query

WebGUI

  1. Login into FleetDM
  2. Select “Queries” on the left
  3. Select “Create a new query”
    1. Enter Netstat into Quer title
    2. Enter SELECT DISTINCT process.name, listening.port, listening.address, process.pid FROM processes AS process JOIN listening_ports AS listening ON process.pid = listening.pid; into SQL
    3. Select All hosts for Select targets
    4. Select Save > Save As New
  4. Select “Queries” on the left

Create a query pack

FleetCTL

  1. wget https://raw.githubusercontent.com/osquery/osquery/master/packs/ossec-rootkit.conf
  2. fleetctl convert -f ossec-rootkit.conf > ossec-rootkit--pack-fleet.yaml
    1. Convert JSON to YAML
  3. fleetctl apply -f ossec-rootkit--pack-fleet.yaml
  4. Login into FleetDM
  5. Select “Queries” on the left

WebGUI

  1. Login into FleetDM
  2. Select “Queries” on the left
  3. Select “Create new pack”
    1. Enter Collect sys info for Query pack title
    2. Select All hosts for Select targets
    3. Select “Save query pack”
  4. Select the drop-down in the top right
  5. Select “Netstat”
    1. Enter settings for query
    2. Select “Save”

FleetDM file carving with FleetCTL

  1. fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path = "C:\Windows\System32\drivers\etc\hosts"'
  2. fleetctl get carves
  3. fleetctl get carve 1 --outfile <outfile name>.tar
  4. tar -xvf <outfile name>.tar

Lessons learned

I am currently reading a book called “Cracking the Coding Interview” and it is a great book. One interesting part of the book is their matrix to describe projects you worked on and the matrix contains the following sections which are: challenges, mistakes/failures, enjoyed, leadership, conflicts, and what would you do differently. I am going to try and use this model at the end of my blog posts to summarize and reflect on the things I learn. I don’t blog to post things that I know, I blog to learn new things and to share the knowledge of my security research.

New skills/knowledge

  • Learned how to utilize the FleetDM API
  • Learned new features of FleetDM

Enjoyed

Challenges

  • Inconsistency of features between docker-compose v2.x and v3.x

References

Leave a Reply

Your email address will not be published. Required fields are marked *