The purpose of this blog post is to provide multiple methods on how to install/setup FleetDM, how to deploy Osquery, and demonstrate how to use features of FleetDM + FleetCTL. This blog post generated an Ansible playbook, Docker-composes for Swarm and non-swarm, Vagrant to create a VM, and manual instructions for installing FleetDM on Ubuntu 20.04. Additionally, there are Ansible playbooks for deploying the Osquery agent on Windows and Ubuntu with manual instructions as well. Lastly, I will end by demonstrating how to use the FleetDM WebGUI and FleetCTL tool to manage FleetDM and interact with your Osquery agents.

Goals

• Setup FleetDM with Docker
• Setup FleetDM with Ansible
• Setup FleetDM with Vagrant
• Setup FleetDM with manual instructions
• Install/Setup Osquery on Windows 10
• Install/Setup Osquery on Ubuntu 20.04
• Install/Setup Osquery on macOS Big
• Create a live query on Fleet with UI and FleetCTL CLI tool
• Create a saved query on Fleet with UI and FleetCTL CLI tool
• Create a query pack on Fleet with UI and FleetCTL CLI tool

Update log

• July 15th 2021 – Updated Docker from Fleet v3.7.1 to v4.0.1
• August 29th 2021 – Updated Docker from Fleet v4.0.1 to v4.2.3
• August 29th 2021 – Added instructions to install/setup Osquery on macOS Big Sur
• September 24th 2021 – Added Vagrant to spin up Fleet on Ubuntu 20.04 and updated Ansible playbook to use TARs
• September 24th 2021 – Updated Docker and Ansible from Fleet v4.2.3 to v4.3.1
• December 18th 2021 – Updated Docker and Ansible from Fleet v4.3.1 to v4.7.0
• March 20th 2022 – Updated Docker and Ansible from Fleet v4.7.0 to v4.11.0

Background

What is Fleet FleetDM?

Fleet is the most widely used open source osquery manager. Deploying osquery with Fleet enables programmable live queries, streaming logs, and effective management of osquery across 50,000+ servers, containers, and laptops. It’s especially useful for talking to multiple devices at the same time.

Important note: Common name match for Osquery cert check

This blog post will assume you have the knowledge and capability to create a DNS A record that points to where FleetDM is being hosted. When the Osquery agent connects to FleetDM it will verify that the common name in the public certificate being served by FleetDM matches the common name specified in the

osquery.flags file: --tls_hostname=<FleetDM FQDN>. Below I have included a screen to demonstrate what I mean. Not only does the common name have to match but the public certificate being served by FleetDM and the local certificate saved on the endpoint must match as well. If this criterion is not met then Osquery will generate the following error:

Request error: certificate verify failed

Request error: certificate verify failed.

Install FleetDM with Docker-compose v2.x

WARNING

The Docker-compose v2.x setup is for development use ONLY. The setup contains hard-coded credentials in configs and environment variables. For a more secure Docker deployment please skip to the next section to use Docker Swarm which implements Docker secrets.

WARNING

1. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout conf/tls/fleet.key -out conf/tls/fleet.crt
   openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout conf/tls/fleet.key -out conf/tls/fleet.crt
   1. Generate TLS private key and public certificate
2. git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
   git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
3. cd FleetDM-Automation
   cd FleetDM-Automation
4. vim .env
   vim .env and set
   1. FLEETDM_VERSION
      FLEETDM_VERSION – Version of FleetDM to use – Only change if you want to use a different version
   2. FLEETDM_MYSQL_USER
      FLEETDM_MYSQL_USER – Name of the MySQL user for FleetDM
   3. FLEETDM_MYSQL_PASSWORD
      FLEETDM_MYSQL_PASSWORD – Password for the MySQL user for FleetDM
   4. FLEETDM_MYSQL_DATABASE
      FLEETDM_MYSQL_DATABASE – Name of the MySQL database for FleetDM
   5. FLEETDM_MYSQL_PASSWORD
      FLEETDM_MYSQL_PASSWORD – Password for the MySQL root user
   6. FLEETDM_JWT_KEY
      FLEETDM_JWT_KEY – The JWT token
      1. Generate a random key with
         openssl rand -base64 32
         openssl rand -base64 32
5. docker-compose build
   docker-compose build
6. docker-compose run --rm fleet fleet prepare db --config /etc/fleet/fleet.yml
   docker-compose run --rm fleet fleet prepare db --config /etc/fleet/fleet.yml
7. docker-compose up -d
   docker-compose up -d

Install FleetDM with Docker-compose v3.x (Swarm)

Unfortunately, Docker-compose version 3.X (DockerSwarm) doesn’t interact with the

.env file the same as v2.X. The trade-off is a more secure deployment of the FleetDM stack because secrets are not hardcoded in configs or stored in environment variables. Below we create Docker secrets that will contain these sensitive secrets to be used by the Docker containers.

Furthermore, any changes such as changing the container version or the MySQL database name in the

.env file will have no effect. These settings need to be changed in the docker-compose-swarn.yml file and the same changes may need to be carried over to the FleetDM config located: conf/docker/fleetdm/fleetdm-swarm.yml. Lastly, another benefit of Docker Swarm is we can have multiple instances (replicas) of FleetDM running for high-availability.

1. git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
   git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
2. cd FleetDM-Automation
   cd FleetDM-Automation
3. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout conf/tls/fleet.key -out conf/tls/fleet.crt
   openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout conf/tls/fleet.key -out conf/tls/fleet.crt
   1. Generate TLS private key and public certificate
4. openssl rand -base64 32 | tr -cd '[:alnum:]' | docker secret create fleetdm-jwt-key -
   openssl rand -base64 32 | tr -cd '[:alnum:]' | docker secret create fleetdm-jwt-key -
   1. Generate a JWT key and create a secret named
      fleetdm-jwt-key
      fleetdm-jwt-key
5. openssl rand -base64 32 | tr -cd '[:alnum:]' | docker secret create mysql-root-password -
   openssl rand -base64 32 | tr -cd '[:alnum:]' | docker secret create mysql-root-password -
   1. Generate random password containing only alphanumeric characters for the MySQL root password
6. openssl rand -base64 32 | tr -cd '[:alnum:]' | docker secret create mysql-fleetdm-password -
   openssl rand -base64 32 | tr -cd '[:alnum:]' | docker secret create mysql-fleetdm-password -
   1. Generate random password containing only alphanumeric characters for the MySQL FleetDM user password
7. docker stack deploy -c docker-compose-swarm.yml fleetdm
   docker stack deploy -c docker-compose-swarm.yml fleetdm
   
8. docker service logs -f fleetdm_fleet
   docker service logs -f fleetdm_fleet
   1. The first time these containers are started they will error out until the MySQL database initializes

Install FleetDM on Ubuntu 20.04 with Ansible

Init playbook

1. git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
   git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
2. cd FleetDM-Automation
   cd FleetDM-Automation
3. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout conf/tls/fleet.key -out conf/tls/fleet.crt
   openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout conf/tls/fleet.key -out conf/tls/fleet.crt
   1. Generate TLS private key and public certificate
4. pip3 install ansible
   pip3 install ansible
5. vim hosts.ini
   vim hosts.ini and add IP address under
   [fleetdm]
   [fleetdm]
6. vim all.yml
   vim all.yml and set:
   1. base_domain
      base_domain – The domain for your network and the base domain of the FQDN
   2. timezone
      timezone – OPTIONAL – Change the default timezone of UTC +0
7. openssl rand -base64 32

   openssl rand -base64 32
8. Copy the output from the command
9. vim fleetdm.yml
   vim fleetdm.yml and set:
   1. fleetdm_jwt

      fleetdm_jwt –  Set this to the random string generated by the OpenSSL command
   2. mysql_root_password

      mysql_root_password – Set the root password for MySQL
   3. mysql_fleetdm_password

      mysql_fleetdm_password –  Set the password for FleetDM MySQL user

Run playbook

1. ansible-playbook -i hosts.ini deploy_fleetdm.yml -u <user> -K
   ansible-playbook -i hosts.ini deploy_fleetdm.yml -u <user> -K
   1. Enter the password for the user

Install FleetDM on Ubuntu 20.04 with Vagrant

The Vagrantfile provided relies on the Ansible playbook above to setup Fleet on the Ubuntu VM.

1. git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
   git clone https://github.com/CptOfEvilMinions/FleetDM-Automation
2. cd FleetDM-Automation
   cd FleetDM-Automation
3. Please follow the instructions outlined in the Ansible section above specifically the “Init playbook” section
4. vagrant up
   vagrant up
   
5. vagrant ssh
   vagrant ssh

Manual install of FleetDM 3.6.0 on Ubuntu 20.04

Init host

1. ssh into VM
2. sudo su
   sudo su
3. timedatectl set-timezone Etc/UTC
   timedatectl set-timezone Etc/UTC
   1. Set the system timezone to UTC +0
4. apt update -y && apt upgrade -y && apt install net-tools git wget unzip -y && reboot
   apt update -y && apt upgrade -y && apt install net-tools git wget unzip -y && reboot
   1. Update system and reboot

Install/Setup Redis 5.5.0

1. apt install redis-server=5:5.0.7-2
   apt install redis-server=5:5.0.7-2
2. systemctl enable redis-server
   systemctl enable redis-server
3. systemctl start redis-server
   systemctl start redis-server
4. netstat -tnlp | grep redis
   netstat -tnlp | grep redis

Install/Setup MySQL 8.0

1. apt install mysql-server-8.0 python3-mysqldb python3-pip -y
   apt install mysql-server-8.0 python3-mysqldb python3-pip -y
2. systemctl start mysql
   systemctl start mysql
3. netstat -tnlp | grep mysqld
   netstat -tnlp | grep mysqld
   1. Check that MySQL is listening on port 3306
4. secure_installation
   secure_installation
   1. Enter “y”
   2. Enter “2” for a strong password
      1. Secure Password Generator
   3. Enter a strong password
   4. Enter “y” to continue with the password provided
   5. Enter “y” to remove anonymous user
   6. Enter “y” to disallow root login remotely
   7. Enter “y” to remove the test database
   8. Enter “y” to reload privileged tables
5. systemctl enable mysql
   systemctl enable mysql
6. Create another secure password
7. mysql -u root -p
   mysql -u root -p
   1. Enter the password from above
8. CREATE DATABASE fleetdm;
   CREATE DATABASE fleetdm;
   1. Create FleetDM database
9. CREATE USER 'fleetdm'@'localhost' IDENTIFIED BY '<Password>';
   CREATE USER 'fleetdm'@'localhost' IDENTIFIED BY '<Password>';
   1. Create FleetDM MySQL user
10. GRANT ALL PRIVILEGES ON fleetdm.* TO 'fleetdm'@'localhost';
    GRANT ALL PRIVILEGES ON fleetdm.* TO 'fleetdm'@'localhost';
    1. Provide FleetDM the permissions to interact with the FleetDM database
11. exit
    exit
12. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/mysql/fleetdm.cnf -O /etc/mysql/conf.d/fleetdm.cnf
    wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/mysql/fleetdm.cnf -O /etc/mysql/conf.d/fleetdm.cnf
    1. Download MySQL settings config
13. systemctl restart mysql
    systemctl restart mysql

Install/Setup FleetDM 3.6.0

1. adduser fleetdm --system --no-create-home
   adduser fleetdm --system --no-create-home
   1. Create a user
2. groupadd fleetdm
   groupadd fleetdm
   1. Create a group
3. usermod -aG fleetdm fleetdm
   usermod -aG fleetdm fleetdm
   1. Add the
      fleetdm
      fleetdm user as a member of the
      fleetdm
      fleetdm group
4. cd /tmp/ && wget https://github.com/fleetdm/fleet/releases/download/3.6.0/fleet.zip
   cd /tmp/ && wget https://github.com/fleetdm/fleet/releases/download/3.6.0/fleet.zip
5. unzip fleet.zip
   unzip fleet.zip
6. mv linux/fleet /usr/local/bin/fleet
   mv linux/fleet /usr/local/bin/fleet
   1. Move FleetDM binary to the local bin directory
7. mkdir /etc/fleetdm
   mkdir /etc/fleetdm
8. chown root:root -R /etc/fleetdm
   chown root:root -R /etc/fleetdm
   1. Create a config directory for FleetDM
9. mkdir /var/log/osquery
   mkdir /var/log/osquery
10. chown fleetdm:fleetdm -R /var/log/osquery
    chown fleetdm:fleetdm -R /var/log/osquery 
    1. Create a directory to record the results of Osquery endpoints
11. openssl req -newkey rsa:2048 -nodes -keyout /etc/ssl/private/fleetdm.key -x509 -days 365 -out /etc/ssl/certs/fleetdm.crt
    openssl req -newkey rsa:2048 -nodes -keyout /etc/ssl/private/fleetdm.key -x509 -days 365 -out /etc/ssl/certs/fleetdm.crt
    1. Generate self-signed private key and public certificate for FleetDM
12. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/fleetdm/fleetdm.yml -O /etc/fleetdm/fleetdm.yml
    wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/fleetdm/fleetdm.yml -O /etc/fleetdm/fleetdm.yml
    1. Download FleetDM config
13. chown root:fleetdm /etc/fleetdm/fleetdm.yml
    chown root:fleetdm /etc/fleetdm/fleetdm.yml
14. chmod 640 /etc/fleetdm/fleetdm.yml
    chmod 640 /etc/fleetdm/fleetdm.yml
15. sed -i "s/{{ fleetdm_jwt }}/$(openssl rand -base64 32 | tr -cd '[:alnum:]')/g" /etc/fleetdm/fleetdm.yml
    sed -i "s/{{ fleetdm_jwt }}/$(openssl rand -base64 32 | tr -cd '[:alnum:]')/g" /etc/fleetdm/fleetdm.yml
    1. Generate a random JWT key and set the value in the FleetDM config
16. sed -i "s/{{ fleetdm_db_name }}/<FleetDM database name>/g" /etc/fleetdm/fleetdm.yml
    sed -i "s/{{ fleetdm_db_name }}/<FleetDM database name>/g" /etc/fleetdm/fleetdm.yml
    1. Set the database name
17. sed -i "s/{{ fleetdm_username }}/<FleetDM username>/g" /etc/fleetdm/fleetdm.yml
    sed -i "s/{{ fleetdm_username }}/<FleetDM username>/g" /etc/fleetdm/fleetdm.yml
    1. Set the MySQL username
18. sed -i "s/{{ fleetdm_password }}/<FleetDM password>/g" /etc/fleetdm/fleetdm.yml
    sed -i "s/{{ fleetdm_password }}/<FleetDM password>/g" /etc/fleetdm/fleetdm.yml
    1. Set the MySQL password
19. /usr/local/bin/fleet prepare db --config /etc/fleetdm/fleetdm.yml
    /usr/local/bin/fleet prepare db --config /etc/fleetdm/fleetdm.yml
    1. Init FleetDM database
20. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/fleetdm/fleetdm-systemd.service -O /etc/systemd/system/fleetdm.service
    wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/fleetdm/fleetdm-systemd.service -O /etc/systemd/system/fleetdm.service
    1. Download SystemD config for FleetDM
21. systemctl enable fleetdm
    systemctl enable fleetdm
22. systemctl start fleetdm
    systemctl start fleetdm
23. netstat -tnlp | grep fleet
    netstat -tnlp | grep fleet
    1. Check that MySQL is listening on port 8080

Install NGINX

1. apt install nginx -y
   apt install nginx -y
2. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/nginx/nginx.conf -O /etc/nginx/nginx.conf
   wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/nginx/nginx.conf -O /etc/nginx/nginx.conf
3. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/nginx/nginx_fleetdm.conf -O /etc/nginx/conf.d/fleetdm.conf
   wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/ansible/nginx/nginx_fleetdm.conf -O /etc/nginx/conf.d/fleetdm.conf
   1. Download NGINX configs
4. systemctl enable nginx
   systemctl enable nginx
5. systemctl restart nginx
   systemctl restart nginx
6. netstat -tnlp | grep nginx
   netstat -tnlp | grep nginx
   1. Check that MySQL is listening on port 80 and 443

Setup UFW

1. ufw allow OpenSSH
   ufw allow OpenSSH
2. ufw allow 'Nginx HTTP'
   ufw allow 'Nginx HTTP'
3. ufw allow 'Nginx HTTPS'
   ufw allow 'Nginx HTTPS'
4. ufw enable
   ufw enable

Setup FleetDM via WebGUI

1. Open a web browser to FleetDM
2. Setup user
   1. Enter a
      <username>
      <username> into Username
   2. Enter a
      <password>
      <password> into Password
   3. Enter an
      <e-mail>
      <e-mail> for the admin user
   4. Select “Next”
3. Organization details
   1. Enter organization name
   2. Enter the organization URL
   3. Select “Next”
4. Set Fleet URL
   1. Accept the default Fleet URL, unless you need to change it
   2. Select “Submit”
5. Select “Finish”
   

Deploy Osquery v4.6.0 agent

Osquery.flags vs. Osquery.conf

A common struggle for new Osquery users is knowing when a config option should be set in

osquery.flags or osquery.conf. First, I would like to provide this Youtube video which is a recording of an Osquery office hours meeting where I asked this question to the core developers of Osquery and the discussion that followed. @zwass started the discussion by stating that osqueryd --help outputs two sections of config options. The first set of config options must be placed into the osquery.flags file and the second set can be specified in both config files. @teddyreedv stated that the osquery.flags file is meant to define the start-up behavior of Osquery. Therefore, the config options that are specified in osquery.flags are static throughout the Osquery runtime but the config options specified in osquery.conf are dynamic throughout the runtime. For example, in the osqeury.flags file you would specify which user the Osquery agent should run as, which is not something you can change during runtime.

Furthermore, there are additional considerations to understand when connecting Osquery to a fleet manager/TLS endpoint like FleetDM. The first consideration is the fleet manager can not modify the

osquery.flags file; however, if a config option can be set in both config files the value specified in osquery.conf will override the value in osquery.flags. What this ultimately means is in this scenario, the server gets the final say in how Osquery operates. For example, you can specify the --max_read option to 1MB in osquery.flags but the server could override this setting by setting this value to 5MB in osquery.conf. To conceptualize this, @zwass made a good comment that the order of operations when loading a config starts with osquery.flags and then osquery.conf is loaded, which overrides any settings specified by osquery.flags

To this point, the discussion has only provided how Osquery operates and did not provide specific guidance on how to construct config files for new users. It’s important to understand how Osquery ingests these config options to create the optimal configs for your environment. My guidance for new users for constructing Osquery configs is to create separate configs that target specific environments (QA vs. prod vs. cloud) and for each target OS (Linux, macOS, and Windows). First, start with constructing the

osquery.flags file and keep it as small as possible, which means only specifying the necessary flags to start Osquery in the desired state. For example, if Osquery is connecting to a fleet manager you need to specify the necessary flags such as: --enroll_secret, --tls_server_cert, and --tls_hostname to connect to a fleet manager to for further instructions.

For constructing

osquery.conf I still recommend generating a config file for each environment and OS flavor. My reasoning for this is you may want to monitor multiple sets of machines differently. For example, the data you might collect from a set of Windows user endpoints is not the same set of data you would collect from Linux machines in AWS. Additionally, not all Osquery tables are cross-platform so on Windows you may collect the domain information with the ntdomains table, which does not exist for Linux. Conversely, on Linux, you may collect all the configured sudo users with the sudoers table. In addition to targeting the OS, the environment type may also play a factor in deciding how you monitor the machine. In a production environment, you may need to squeeze all the performance you can out of a system to keep your platform fast. Therefore, in a production environment, you may set strict Osquery watchdog settings so Osquery doesn’t impact system performance. Lastly, one of the ways I learned to construct my Osquery.conf is by looking at examples on Github such as this repo: palantir/osquery-configuration.

Manual Osquery install on Ubuntu

1. SSH into UbuntuVM
2. export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
   export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
3. sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys $OSQUERY_KEY
   sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys $OSQUERY_KEY
4. sudo add-apt-repository 'deb [arch=amd64] https://pkg.osquery.io/deb deb main'
   sudo add-apt-repository 'deb [arch=amd64] https://pkg.osquery.io/deb deb main'
5. sudo apt-get update -y
   sudo apt-get update -y
6. sudo apt-get install osquery -y
   sudo apt-get install osquery -y
7. openssl s_client -showcerts -connect <FleetDM FQDN>:443 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /etc/osquery/fleetdm.crt
   openssl s_client -showcerts -connect <FleetDM FQDN>:443 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /etc/osquery/fleetdm.crt
   1. Write the FleetDM public certificate to disk
8. Open a web browser to FleetDM and Login
   1. Select “Add new host” in the top right
   2. Copy the enrollment secret
9. echo '<FleetDM enroll secret>' > /etc/osquery/osquery.key
   echo '<FleetDM enroll secret>' > /etc/osquery/osquery.key
10. wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/osquery/osquery_linux.flags -O /etc/osquery/osquery.flags
    wget https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/osquery/osquery_linux.flags -O /etc/osquery/osquery.flags
    1. Download
       Osquery.flags
       Osquery.flags config
11. sed -i 's/{{ fleetdm_fqdn }}/<FleetDM FQDN>/g' /etc/osquery/osquery.flags
    sed -i 's/{{ fleetdm_fqdn }}/<FleetDM FQDN>/g' /etc/osquery/osquery.flags
12. sed -i 's/{{ fleetdm_port }}/<FleetDM port - default 443>/g' /etc/osquery/osquery.flags
    sed -i 's/{{ fleetdm_port }}/<FleetDM port - default 443>/g' /etc/osquery/osquery.flags
13. systemctl restart osqueryd
    systemctl restart osqueryd
14. systemctl status osqueryd
    systemctl status osqueryd
    1. Make sure there are no errors

Manual Osquery install on Windows

1. Login into Windows
2. Open Powershell as Administrator
3. Invoke-WebRequest -Uri https://pkg.osquery.io/windows/osquery-4.6.0.msi -OutFile $ENV:TEMP\osquery-4.6.0.msi -MaximumRedirection 3
   Invoke-WebRequest -Uri https://pkg.osquery.io/windows/osquery-4.6.0.msi -OutFile $ENV:TEMP\osquery-4.6.0.msi -MaximumRedirection 3
   1. Download Osquery
4. Start-Process $ENV:TEMP\osquery-4.6.0.msi -ArgumentList '/quiet' -Wait
   Start-Process $ENV:TEMP\osquery-4.6.0.msi -ArgumentList '/quiet' -Wait
   1. Install Osquery
5. Open a web browser to FleetDM and Login
6. Click “Add new host” in the top right
   1. Click “Download” under Enroll secret
   2. Click “Download” under Server Certificate
   3. Click “Download” under Flagfile
7. cd ~/Downloads
   cd ~/Downloads
8. mv .\secret.txt 'C:\Program Files\osquery\secret.txt'
   mv .\secret.txt 'C:\Program Files\osquery\secret.txt'
9. mv .\fleet.pem 'C:\Program Files\osquery\fleet.pem'
   mv .\fleet.pem 'C:\Program Files\osquery\fleet.pem'
10. rm 'C:\Program Files\osquery\osquery.flags'
    rm 'C:\Program Files\osquery\osquery.flags'
11. mv .\flagfile.txt 'C:\Program Files\osquery\osquery.flags'
    mv .\flagfile.txt 'C:\Program Files\osquery\osquery.flags'
12. Restart-Service -Name osqueryd
    Restart-Service -Name osqueryd

Manual Osquery v4.9.0 install on macOS

1. Login into macOS
2. Open terminal as sudo
3. curl https://pkg.osquery.io/darwin/osquery-4.9.0.pkg --output /tmp/osquery-4.9.0.pkg
   curl https://pkg.osquery.io/darwin/osquery-4.9.0.pkg --output /tmp/osquery-4.9.0.pkg
   1. Download Osquery
4. sudo installer -pkg /tmp/osquery-4.9.0.pkg -target /
   sudo installer -pkg /tmp/osquery-4.9.0.pkg -target /
   1. Install Osquery
5. openssl s_client -showcerts -connect <FleetDM FQDN>:443 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /var/osquery/fleetdm.crt
   openssl s_client -showcerts -connect <FleetDM FQDN>:443 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /var/osquery/fleetdm.crt
   1. Write the FleetDM public certificate to disk
6. Open a web browser to FleetDM and Login
   1. Select “Add new host” in the top right
   2. Copy the enrollment secret
7. echo '<FleetDM enroll secret>' > /var/osquery/osquery.key
   echo '<FleetDM enroll secret>' > /var/osquery/osquery.key
8. curl https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/osquery/osquery_macos.flags --output /var/osquery/osquery.flags
   curl https://raw.githubusercontent.com/CptOfEvilMinions/FleetDM-Automation/main/conf/osquery/osquery_macos.flags --output /var/osquery/osquery.flags
   1. Download
      osquery.flags
      osquery.flags config
9. sed -i '' "s/{{ fleetdm_fqdn }}/<FleetDM FQDN>/g" /var/osquery/osquery.flags
   sed -i '' "s/{{ fleetdm_fqdn }}/<FleetDM FQDN>/g" /var/osquery/osquery.flags
10. sed -i '' "s/{{ fleetdm_port }}/<FleetDM port>/g" /var/osquery/osquery.flags
    sed -i '' "s/{{ fleetdm_port }}/<FleetDM port>/g" /var/osquery/osquery.flags
    1. Set FleetDM FQDN and port for Osquery to report too
11. cp /var/osquery/com.facebook.osqueryd.plist /Library/LaunchDaemons/com.facebook.osqueryd.plist
    cp /var/osquery/com.facebook.osqueryd.plist /Library/LaunchDaemons/com.facebook.osqueryd.plist
    1. Copy LaunchDaemon
12. In the top-right select the Apple icon then “System Preferences”
13. Security & Privacy > Privacy > Full Disk Access
14. Select the the lock in the bottom left
15. Press Shift + Command + G
16. Enter
    /usr/local/bin/osqueryd
    /usr/local/bin/osqueryd
17. Drag and drop
    osqueryd
    osqueryd from Finder to Full Disk Access section
18. Make sure
    osqueryd
    osqueryd is checked
19. Press the lock in the bottom left to lock these settings from being modified
20. sudo launchctl load /Library/LaunchDaemons/com.facebook.osqueryd.plist
    sudo launchctl load /Library/LaunchDaemons/com.facebook.osqueryd.plist
    1. Start LaunchDaemon for Osquery
21. sudo osqueryctl start
    sudo osqueryctl start
22. sudo osqueryctl status
    sudo osqueryctl status

Automated Osquery install with Ansible on Windows and Ubuntu

Deploy on Ubuntu

Init playbook for Ubuntu

1. export FLEETDM_TOKEN=$(curl -k -X POST https://<FleetDM FQDN>:<Port>/api/v1/fleet/login -d '{"Email": "<admin username>", "Password": "<admin password>"}' | jq -r '.token')
   export FLEETDM_TOKEN=$(curl -k -X POST https://<FleetDM FQDN>:<Port>/api/v1/fleet/login -d '{"Email": "<admin username>", "Password": "<admin password>"}' | jq -r '.token')
   1. Request a FleetDM JWT authentication token
   2. Store the JWT authentication token in an environment variable named
      FLEETDM_TOKEN
      FLEETDM_TOKEN
   3. For more information on this process see my blog post here: DEMYSTIFYING THE KOLIDE FLEET API WITH CURL, PYTHON, FLEETCTL, AND ANSIBLE
2. vim group_vars/fleetdm.yml
   vim group_vars/fleetdm.yml and set:
   1. fleetdm_fqdn
      fleetdm_fqdn – Set this value to the FQDN of FleetDM
   2. fleetdm_port
      fleetdm_port – Set the port that FleetDM is exposed on – default port 443
3. vim group_vars/osquery.yml
   vim group_vars/osquery.yml and set:
   1. osquery_version
      osquery_version – OPTIONAL – Set the version of Osquery you want to install – default 4.6.1
4. vim hosts.ini
   vim hosts.ini add IP addresses for Linux hosts under
   [osquery_linux]
   [osquery_linux]

Run playbook

1. ansible-playbook -i hosts.ini deploy_osquery_linux.yml -u <user> -K
   ansible-playbook -i hosts.ini deploy_osquery_linux.yml -u <user> -K
   1. Enter password

Deploy on Windows

Init playbook for Windows

1. export FLEETDM_TOKEN=$(curl -k -X POST https://<FleetDM FQDN>:<Port>/api/v1/fleet/login -d '{"Email": "<admin username>", "Password": "<admin password>"}' | jq -r '.token')
   export FLEETDM_TOKEN=$(curl -k -X POST https://<FleetDM FQDN>:<Port>/api/v1/fleet/login -d '{"Email": "<admin username>", "Password": "<admin password>"}' | jq -r '.token')
2. vim group_vars/fleetdm.yml
   vim group_vars/fleetdm.yml and set:
   1. fleetdm_fqdn
      fleetdm_fqdn – Set this value to the FQDN of FleetDM
   2. fleetdm_port
      fleetdm_port – Set the port that FleetDM is exposed on – default port 443
3. vim group_vars/osquery.yml
   vim group_vars/osquery.yml and set:
   1. osquery_version
      osquery_version – OPTIONAL – Set the version of Osquery you want to install
4. vim group_vars/windows.yml
   vim group_vars/windows.yml and set:
   1. ansible_user
      ansible_user – Set this to a Windows username
   2. ansible_password
      ansible_password – Set this to a Windows password
5. vim hosts.ini
   vim hosts.ini add IP addresses for Linux hosts under
   [osquery_windows]
   [osquery_windows]

Init Windows VM for Ansible

1. Log into Windows VM
2. Open Powershell as Administrator
3. Set-ExecutionPolicy Unrestricted
   Set-ExecutionPolicy Unrestricted
   1. Enter “y”
4. powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1'))"
   powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1'))"

Run playbook

1. ansible-playbook -i hosts.ini deploy_osquery_windows.yml
   ansible-playbook -i hosts.ini deploy_osquery_windows.yml

Create a Live query

WebGUI

1. Login into FleetDM
2. Select “Queries” on the left
3. Select “Create new query” in the top right
   1. Enter
      Get processes
      Get processes into the Query title field
   2. Enter
      SELECT * FROM processes
      SELECT * FROM processes into the SQL field
   3. Select
      MS Windows
      MS Windows for Select targets
   4. Select “Run” and wait

FleetCTL

1. cd /tmp && wget https://github.com/kolide/fleet/releases/download/3.2.0/fleet.zip
   cd /tmp && wget https://github.com/kolide/fleet/releases/download/3.2.0/fleet.zip
2. unzip fleet.zip
   unzip fleet.zip
3. mv <OS - macOS:darwin, linux:linux>/fleetctl /usr/local/bin/fleetctl
   mv <OS - macOS:darwin, linux:linux>/fleetctl /usr/local/bin/fleetctl
4. fleetctl config set --address https://<FleetDM FQDN>:<port> --tls-skip-verify
   fleetctl config set --address https://<FleetDM FQDN>:<port> --tls-skip-verify
   1. Set the Fleet API address
   2. Only specify
      --tls-skip-verify
      --tls-skip-verify, if you have a self-signed certificate
5. fleet login
   fleet login
   1. Enter Fleet user e-mail
   2. Enter Fleet user password
6. fleetctl query --query "<Osquery query>" --hosts <Fleet friendly name>
   fleetctl query --query "<Osquery query>" --hosts <Fleet friendly name>

Create a saved query

WebGUI

1. Login into FleetDM
2. Select “Queries” on the left
3. Select “Create a new query”
   1. Enter
      Netstat
      Netstat into Quer title
   2. Enter
      SELECT DISTINCT process.name, listening.port, listening.address, process.pid FROM processes AS process JOIN listening_ports AS listening ON process.pid = listening.pid;
      SELECT DISTINCT process.name, listening.port, listening.address, process.pid FROM processes AS process JOIN listening_ports AS listening ON process.pid = listening.pid; into SQL
   3. Select
      All hosts
      All hosts for Select targets
   4. Select Save > Save As New
4. Select “Queries” on the left

Create a query pack

FleetCTL

1. wget https://raw.githubusercontent.com/osquery/osquery/master/packs/ossec-rootkit.conf
   wget https://raw.githubusercontent.com/osquery/osquery/master/packs/ossec-rootkit.conf
2. fleetctl convert -f ossec-rootkit.conf > ossec-rootkit--pack-fleet.yaml
   fleetctl convert -f ossec-rootkit.conf > ossec-rootkit--pack-fleet.yaml
   1. Convert JSON to YAML
3. fleetctl apply -f ossec-rootkit--pack-fleet.yaml
   fleetctl apply -f ossec-rootkit--pack-fleet.yaml
4. Login into FleetDM
5. Select “Queries” on the left

WebGUI

1. Login into FleetDM
2. Select “Queries” on the left
3. Select “Create new pack”
   1. Enter
      Collect sys info
      Collect sys info for Query pack title
   2. Select All hosts for Select targets
   3. Select “Save query pack”
4. Select the drop-down in the top right
5. Select “Netstat”
   1. Enter settings for query
   2. Select “Save”

FleetDM file carving with FleetCTL

1. fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path = "C:\Windows\System32\drivers\etc\hosts"'
   fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path = "C:\Windows\System32\drivers\etc\hosts"'
2. fleetctl get carves
   fleetctl get carves
3. fleetctl get carve 1 --outfile <outfile name>.tar
   fleetctl get carve 1 --outfile <outfile name>.tar
4. tar -xvf <outfile name>.tar
   tar -xvf <outfile name>.tar

Lessons learned

I am currently reading a book called “Cracking the Coding Interview” and it is a great book. One interesting part of the book is their matrix to describe projects you worked on and the matrix contains the following sections which are: challenges, mistakes/failures, enjoyed, leadership, conflicts, and what would you do differently. I am going to try and use this model at the end of my blog posts to summarize and reflect on the things I learn. I don’t blog to post things that I know, I blog to learn new things and to share the knowledge of my security research.

New skills/knowledge

• Learned how to utilize the FleetDM API
• Learned new features of FleetDM

Enjoyed

• Committing code upstream to the FleetDM project to support passwords via Docker secrets
• Committing code upstream to the FleetDM project to run the Docker container as a non-root user

Challenges

• Inconsistency of features between docker-compose v2.x and v3.x

References

• How To Set or Change Timezone on Ubuntu 18.04
• Check package version using apt-get/aptitude?
• How To Install and Secure Redis on Ubuntu 18.04
• How To Install MySQL 8.0 on Ubuntu 20.04
• How to Create MySQL Users Accounts and Grant Privileges
• Setting up a UFW Secured Nginx Reverse Proxy with HTTP Authentication and TLS Certificates from Let’s Encrypt
• Generating a self-signed certificate using OpenSSL
• Fleet releases
• Linux: How to Create a Group in Ubuntu Linux Server 14.04 LTS
• How to create a Linux user that cannot log in
• POC: USING KSQL TO ENRICH ZEEK LOGS WITH OSQUERY AND SYSMON DATA
• Using openssl to get the certificate from a server
• Installing Chocolatey
• PowerTip: Use PowerShell to Find the Temp Folder Path
• Out-File
• DEMYSTIFYING THE KOLIDE FLEET API WITH CURL, PYTHON, FLEETCTL, AND ANSIBLE
• Introduction to osquery for Threat Detection and DFIR
• Youtube – osquery Office Hours 2020 12 22
• Osquery docs – Command-line flags
• Github – palantir/osquery-configuration