Mar 01 2020
Leave a comment
By spartan2194 Incident Response, PoC, Threat Detection Engineering, Tools Image

Operation cleanup: Eradicating malware with Osquery and Kolide

This blog post is going to cover an Osquery extension that I engineered with osquery-go to eradicate malware. This extension has the ability to delete files, kill processes, delete directories, and can be used with the builtin YARA table. This blog post will act as documentation for the setup and operation of this Osquery extension.

Background

What is Osquery?

Osquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. With Osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes. SQL tables are implemented via a simple plugin and extensions API.

What is Osquery-go?

In Osquery, SQL tables, configuration retrieval, log handling, etc. are implemented via a robust plugin and extensions API. This project contains Go bindings for creating Osquery extensions in Go. To create an extension, you must create an executable binary that instantiates an ExtensionManagerServer and registers the plugins that you would like to be added to Osquery. You can then have Osquery load the extension in your desired context (ie: in a long-running instance of OsqueryD or during an interactive query session with Osqueryi).

What is YARA?

YARA is a tool aimed at (but not limited to) helping malware researchers to identify and classify malware samples. With YARA you can create descriptions of malware families (or whatever you want to describe) based on textual or binary patterns. Each description, a.k.a rule, consists of a set of strings and a boolean expression which determine its logic.

Install Osquery

  1. Download https://pkg.osquery.io/darwin/osquery-4.1.2.pkg
  2. Install Osquery

Install/Setup go env 

  1. git clone https://github.com/CptOfEvilMinions/osquery-malware-removal
  2. cd osquery-malware-removal
  3. go get

Compile Osquery extension

  1. GOOS=windows GOARCH=amd64 go build -o osquery_malware_removal.exe osquery-malware-removal.go
    1. Compile executable for Windows
  2. GOOS=darwin GOARCH=amd64 go build -o osquery_malware_removal.macho osquery-malware-removal.go
    1. Compile executable for macOS
  3. GOOS=linux GOARCH=amd64 go build -o osquery_malware_removal.elf osquery-malware-removal.go
    1. Compile executable for Linux

Test Osquery extension

Load Osqueri with extension

  1. osqueryi --extension <Path to project>/osquery-malware-removal/osquery-malware-removal
  2. .tables
  3. SELECT * FROM malware_removal;

Delete file

  1. Open a new terminal
  2. touch /tmp/hello0
  3. touch /tmp/hello1
  4. sudo chown root /tmp/hello1
  5. Back to Osqueryi
  6. SELECT * FROM malware_removal WHERE file_path='/tmp/hello0';
  7. SELECT * FROM malware_removal WHERE file_path='/tmp/hello1';

Delete directory

  1. mkdir /tmp/testdir1
    1. touch /tmp/testdir1/hello2
    2. touch /tmp/testdir1/hello3
  2. mkdir /tmp/testdir2
    1. touch /tmp/testdir2/hello4
    2. touch /tmp/testdir2/hello5
    3. sudo chown root /tmp/testdir2/hello5
  3. mkdir /tmp/testdir3
    1. touch /tmp/testdir3/hello6
    2. touch /tmp/testdir3/hello7
    3. sudo chown root /tmp/testdir3
  4. SELECT * FROM malware_removal WHERE dir_path='/tmp/testdir1';
  5. SELECT * FROM malware_removal WHERE dir_path='/tmp/testdir2';
  6. SELECT * FROM malware_removal WHERE dir_path='/tmp/testdir3';
  7. SELECT * FROM malware_removal WHERE dir_path='/tmp/testdir4';

Delete a file with SHA256 hash

  1. touch /tmp/hello8
  2. shasum -a 256 /tmp/hello8
    1. Copy output from command
  3. SELECT * FROM malware_removal WHERE dir_path='/tmp' AND file_hash='<SHA256 hash outout from above>';
  4. SELECT * FROM malware_removal WHERE dir_path='/tmp' AND file_hash='<hash from above minus one character>';

Terminate process

  1. ipython
    1. import os
    2. os.getpid()
    3. Copy output
  2. SELECT * FROM malware_removal WHERE pid=<PID from above>;
  3. SELECT * FROM malware_removal WHERE pid=<PID of ROOT process>;
  4. SELECT * FROM malware_removal WHERE pid=<PID that doesn't exist>;

Testing Osquery extension with YARA

Test Osquery YARA table

The EICAR file is a UNmalicious file that all anti-virus products SHOULD alert on. We are going to download an EICAR file as our “malicious file” but it is benign

  1. sudo mkdir /var/osquery/yara
  2. sudo curl https://raw.githubusercontent.com/airbnb/binaryalert/master/rules/public/eicar.yara -o /var/osquery/yara/eicar.yara
  3. cd ~/Downloads
  4. curl http://www.eicar.org/download/eicar.com -o eicar.com
  5. osqueryi
  6. SELECT * FROM yara WHERE path like '/Users/%/Downloads/%' AND sigfile='/var/osquery/yara/eicar.yara';

JOIN YARA and malware_removal _table

  1. sudo mkdir /var/osquery/Extensions
  2. sudo cp osquery-malware-removal /var/osquery/Extensions/osquery-malware-removal.ext
  3. osqueryi --extension /var/osquery/Extensions/osquery-malware-removal.ext
  4. SELECT y.path, y.matches, y.count, m.status, m.status_bool FROM yara as y JOIN malware_removal as m ON y.path=m.file_path WHERE path like '/Users/%/Downloads/%' AND sigfile='/var/osquery/yara/eicar.yara' AND COUNT > 0;

Install/Setup Kolide + Osquery

Spin up Kolide with Docker

  1. git clone https://github.com/CptOfEvilMinions/Kolide-Docker.git
  2. cd Kolide-Docker
  3. docker-compose run --rm kolide fleet prepare db --config /etc/kolide/kolide.yml
  4. docker-compose up -d
  5. Open a browser to https://<Docker IP addr>:8443
  6. SET USERNAME & PASSWORD
    1. Enter username
    2. Enter password
    3. Enter e-mail
    4. Select “Submit”
  7. SET ORGANIZATION DETAILS
    1. Enter org name
    2. Enter org https://<FQDN>
    3. Select “Submit”
  8. SET KOLIDE WEB ADDRESS
    1. Enter https://<FQDN>
  9. Select “Submit”

 

Install/Setup OsqueryD + Osquery extensions on macOS

  1. Open terminal as sudo
  2. cd /var/osquery
  3. sudo curl https://raw.githubusercontent.com/CptOfEvilMinions/Kolide-Docker/master/conf/osquery/osquery.flags -o /var/osquery/osquery.flags
  4. sudo sed -i '' 's/kolide_fqdn/kolide.hackinglab.local/g' osquery.flags
  5. sudo sed -i '' 's/kolide_port/8443/g' osquery.flags
  6. sudo chown root Extensions
  7. Go to Kolide
  8. Select “Add New Host”
  9. Copy key
  10. Back to terminal
  11. echo '<Kolide key>' | sudo tee /var/osquery/osquery.key
  12. openssl s_client -connect kolide.hackinglab.local:8443 -showcerts </dev/null 2>/dev/null|openssl x509 -outform PEM | sudo tee /var/osquery/certs/certificate.crt
  13. sudo launchctl load /var/osquery/com.facebook.osqueryd.plist
  14. Go back to Kolide

Download EICAR file

  1. curl http://www.eicar.org/download/eicar.com -o eicar.com

Remove malware with Kolide

  1. From the Kolide homepage select the host
  2. SELECT * FROM osquery_extensions
  3. SELECT * FROM yara WHERE path like '/Users/%/Downloads/%' AND sigfile='/var/osquery/yara/eicar.yara';
  4. SELECT y.path, y.matches, y.count, m.status, m.status_bool FROM YARA as y JOIN malware_removal as m ON y.path=m.file_path WHERE path like '/Users/%/Downloads/%' AND sigfile='/var/osquery/yara/eicar.yara' AND COUNT > 0;

Discussion

No role-based access control (RBAC)

I don’t know of any open source fleet control managers that implement role-based access control to restrict which tables users can use. This means if a user has the ability to invoke a query on an Osquery agent they have the ability to invoke deletion of files or termination of processes.

Lessons learned

I am currently reading a book called “Cracking the Coding Interview” and it is a great book. One interesting part of the book is their matrix to describe projects you worked on and the matrix contains the following sections which are: challenges, mistakes/failures, enjoyed, leadership, conflicts, and what you’d do differently. I am going to try and use this model at the end of my blog posts to summarize and reflect on the things I learn. I don’t blog to post things that I know, I blog to learn new things and to share the knowledge of my security research.

Challenges

  • Learning a new programming language which is golang
    • Learning all the new data structures and data structure names, such as a slice is equivalent to a Python list and a map is equivalent to a directory
  • Assuming that writing golang would automatically make the code compatible with all OSes. Unfortunately, that was not the case when I attempted to use the library syscall which is compatible with Linux and Darwin. I had to shift to gopsutils to ensure my terminateProcess function worked on all platforms.

Enjoyed

  • I loved that I got to engineer a new security mechanism for my favorite open source project Osquery
  • Learning how to read the osquery-go struct blueprint to extract values.
  • I frustratedly enjoyed learning how to enforce handling errors in golang. With languages like Python, the error is often abstracted away until it happens but golang enforces you to handle it and this is helping me learn how to handle different error cases.
  • Implemented a successful golang project with a go mod – requirements.txt equivalent in Python
  • Thinking of new edges cases to test my new Osquery extension against such as “What if I try to delete a directory that doesn’t exist?” or “What if I try to delete a hash that doesn’t exist?”. I was actually able to catch some and repair some errors in my code by doing this.
  • Learning how to setup TravisCI + Github to release new builds

What You’d Do Differently

  • I wish I knew how to do golang tests so I didn’t have to keep making files and directories manually to delete while testing
  • The current extensions can delete files and kill processes but it doesn’t stop network connections. In an incident response scenario, you may want the ability to kill a network connection via firewall-based rule. Since Windows, macOS, and Linux all utilize different platforms for firewalls, I didn’t implement that functionality.
  • At the current time of this writing, the extension does not provide the capability to remove malicious Registry entries, Scheduled Tasks, and more on Windows

References

Leave a Reply

Your email address will not be published. Required fields are marked *