For several years I have always wanted to write an Osquery extension to perform memory dumps and analysis. I never got the time to do a deep dive into my idea but since I have been creating some Osquery-go extensions lately, I decided to take a crack at my idea. This blog post will provide a high overview of the architecture of these Osquery extensions for this project, how to generate memory dumps, and how to remotely analyze these memory dumps with Osquery. Follow me with another threat detection engineering experience with Osquery-go.

Background

What is osquery?

Osquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.

What is osquery-go?

In Osquery, SQL tables, configuration retrieval, log handling, etc. are implemented via a robust plugin and extensions API. This project contains Go bindings for creating osquery extensions in Go. To create an extension, you must create an executable binary which instantiates an ExtensionManagerServer and registers the plugins that you would like to be added to osquery. You can then have osquery load the extension in your desired context (ie: in a long running instance of osqueryd or during an interactive query session with osqueryi). For more information about how this process works at a lower level, see the osquery wiki.

What is DumpIt?

The DumpIt utility is used to generate a physical memory dump of Windows machines. It works with both x86 (32-bits) and x64 (64-bits) machines. The raw memory dump is generated in the current directory; only a confirmation question is prompted before starting. Perfect for deploying the executable on USB keys, for quick incident responses needs.

What is ProcDump?

ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike. ProcDump also includes hung window monitoring (using the same definition of a window hang that Windows and Task Manager use), unhandled exception monitoring and can generate dumps based on the values of system performance counters. It also can serve as a general process dump utility that you can embed in other scripts.

What is Volatility?

The Volatility Framework is a completely open collection of tools, implemented in Python under the GNU General Public License, for the extraction of digital artifacts from volatile memory (RAM) samples. The extraction techniques are performed completely independent of the system being investigated but offer visibility into the runtime state of the system. The framework is intended to introduce people to the techniques and complexities associated with extracting digital artifacts from volatile memory samples and provide a platform for further work into this exciting area of research.

Architecture of extensions

To understand how to operate these Osquery tables/extensions you must first understand the construction of the extensions. This section will cover the architecture and design decisions of each Osquery extension.

memory_dump table

To my knowledge, no open-source golang projects exist to perform full memory dumps or even process memory dumps. Therefore, this Osquery-go extension leverages pre-existing tools such as DumpIT to perform full memory captures and ProcDump to perform process memory captures. The first iteration of this extension required the machine to have DumpIt and ProcDump on the machine and in a specific location. This approach causes a lot of headaches to ensure the following were all correct: Osquery is installed, Osquery extension is on the machine, Osquery is configured to use the extension, ProcDump and DumpIt are in the correct locations, and the machine has the required versions of DumpIt and ProcDump. All of these requirements create a logistical NIGHTMARE so I did some Googling and found the following golang project go-bindata.

This golang project allows you to compile static content such as binaries into your golang binary. So, now the logistical nightmare is reduced to Osquery is installed, Osquery extension is on the machine and Osquery is configured to use the extension. When the extension is loaded by Osquery, it unpacks the binaries for DumpIt and ProcDump to disk (See figure below). This means the maintainer of the Osquery extension controls what versions of DumpIt and ProcDump are used. When the binaries are written to disk there is a verification check to ensure the binaries on disk match the binaries being shipped with the Osquery extension. EVERY TIME this Osquery extension is used it does a verification check to ensure the binaries on disk have NOT been modified. If the binaries have been modified the extension will overwrite the binaries with the appropriate binary.

Now, this does mean that the Osquery extension will be making an external call to the binaries on disk to perform the necessary actions. So, in reality, this Osquery extension is just a wrapper around DumpIt and ProcDump to allow the incident responders to create memory dumps remotely. If the incident responder would like the memory dump they can use the Osquery “carve” table to pull the memory dump from the remote machine. However, we do recommend performing a process dump because they are CONSIDERABLY smaller. In the event, an incident responder wants to analyze a full memory dump we included the necessary tooling in the next extension.

memory_analyze table

This extension was designed knowing that incident responders would want to analyze full memory dumps but without the need to upload the memory dump (images) to a remote server. This Osquery-go extension is packed (like above) with Volatility v3 and it leverages the power of this framework to analyze memory images. Volatility version 3 is still in beta but their new redesign will be the model moving forward with how to interact with Volatility and to initiate plugins. This POC decided to use version 3 so this project would be compatible with the future of Volatility. A query for this extension/table must specify the location of a memory dump on disk and a Volatility plugin. The results from Volatility can be returned in the following format such as CSV and JSON.

Freedom to choose your tools

The architecture discussed above also allows you to bring “your owns tools”. Meaning that if you would prefer to use Rekall over Volatility you can do so or use Volatility v2 over Volatility v3. Furthermore, you can do the same with the memory dumping tools. The first iteration of this tool used WinPmem for full memory dumps but it wasn’t compatible with Volatility v3. This architecture allowed me to swap out WinPmem for DumpIt seamlessly.

Compile Osquery extensions

Compile memory_dump table

1. git clone https://github.com/CptOfEvilMinions/osquery-memory-forensics
2. cd osquery-memory-forensics
3. Download ProcDump
4. Rename to procdump.exe
5. Download DumpIt
6. Rename to dumpit.exe
7. Copy binaries to bins/dump as procdump.exe and dumpit.exe

   

8. go get -u github.com/go-bindata/go-bindata/...
9. go install github.com/go-bindata/go-bindata/...
10. go-bindata -o assets/bindata.go -pkg assets bins/...
11. ls -lh assets/dump/bindata.go
12. GOOS=windows go build -o osquery_memory_forensics_dump.exe osquery-memory-forensics-dump.go
13. ls -lh *.exe

Compile memory_analyze table

1. cd osquery-memory-forensics
2. Download volatility V3 ZIP
3. Extract ZIP
4. Copy vol.exe to cd osquery-memory-forensics/bins/analyze as volatility.exe
5. ~/go/bin/go-bindata -o assets/analyze/bindata.go -pkg analyze bins/analyze/...
6. ls -lh assets/dump/bindata.go
7. GOOS=windows go build -o osquery_memory_forensic_analyze.exe cmd/osquery-memory-forensics-analysis/osquery-memory-forensics-analyze.go
8. ls -lh *.exe

Install/Setup Osquery and extensions

Install Osquery

1. Create a Windows 10 VM
2. Download and Install Osquery

Setup Osquery extensions

1. Open Powershell as Administrator
2. cd 'C:\Program Files\osquery'
3. New-Item -Path "C:\Program Files\osquery" -Name "Extensions" -ItemType "directory"
4. Copy compiled extensions above to Windows machine

Test memory_dump extension

1. .\osqueryi.exe --extension .\Extensions\osquery_memory_forensic_dump.exe --allow_unsafe
2. SELECT * FROM memory_dump;
3. SELECT * FROM memory_dump WHERE pid=<PID of explorer.exe>;
4. SELECT * FROM memory_dump WHERE full_memory_dump=1;
5. Browse to C:\forensics\data
6. .quit

Test memory_analyze

1. .mode line
2. SELECT * FROM memory_analyze WHERE file_path="<path to memory dump>" AND plugin="windows.info";
3. SELECT * FROM memory_analyze WHERE file_path="<path to memory dump>" AND plugin="windows.pslist";
4. .quit

Install/Setup Kolide + OsqueryD + Osquery extensions

Spin up Kolide with Docker

1. cd docker-kolide
2. docker-compose run --rm kolide fleet prepare db --config /etc/kolide/kolide.yml
3. docker-compose up
4. Open a browser to http://<Docker IP addr>:8080
5. SET USERNAME & PASSWORD
   1. Enter username
   2. Enter password
   3. Enter e-mail
   4. Select “Submit”
6. SET ORGANIZATION DETAILS
   1. Enter org name
   2. Enter org http://<FQDN>
   3. Select “Submit”
7. SET KOLIDE WEB ADDRESS
   1. Enter https://<FQDN>
   2. Select “Submit”

Install/Setup OsqueryD + Osquery extensions on Windows 10

1. Open Powershell as Administrator
2. cd C:\Program Files\osquery
3. icacls .\Extensions /setowner Administrators /t
4. icacls .\Extensions /grant Administrators:f /t
5. icacls .\Extensions /inheritance:r /t
6. icacls .\Extensions /inheritance:d /t
7. Go to the Kolide hompage
8. Select “Add New Host” in top right
9. Copy key
10. Set-Content -Path 'C:\Program Files\osquery\osquery.key' -Value<Paste key from Kolide here>'
11. [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
12. Invoke-WebRequest -Uri https://raw.githubusercontent.com/CptOfEvilMinions/osquery-memory-forensics/master/conf/osquery/extensions.load -OutFile 'C:\Program Files\osquery\extensions.load'
13. Invoke-WebRequest -Uri https://raw.githubusercontent.com/CptOfEvilMinions/osquery-memory-forensics/master/conf/osquery/osquery.flags -OutFile 'C:\Program Files\osquery\osquery.flags'
15. (Get-Content -path 'C:\Program Files\osquery\osquery.flags' -Raw) -replace "kolide_fqdn","<FQDN of Kolide>" | Set-Content -Path 'C:\Program Files\osquery\osquery.flags'
16. (Get-Content -path 'C:\Program Files\osquery\osquery.flags' -Raw) -replace "kolide_port","<Port being used by Kolide>" | Set-Content -Path 'C:\Program Files\osquery\osquery.flags'
17. Download HTTPS public certificate from Kolde to local machine
18. Start-Service osqueryd
19. Return to Kolide

Initiate memory dumps with Kolide

1. Select an Osquery host from the Kolide homepage
2. SELECT * FROM osquery_extensions
3. Select “Run”
4. SELECT * FROM memory_dump WHERE pid=<PID>
5. SELECT * FROM memory_dump WHERE full_memory_dump =1

Analyze the memory dump with Kolide

1. SELECT * FROM memory_analyze WHERE file_path="<file path from above> plugin="windows.info"
2. SELECT * FROM memory_analyze WHERE file_path="<file path from above> plugin="windows.pslist"

Discussion

Only supports Windows

This POC only supports Windows but the foundation exists to expand the capability to other operating systems such as Linux and macOS. I choose to focus on Windows because a large majority of memory analysis frameworks and tools target Windows, not to mention most malware targets Windows as well. Furthermore, the macOS security SIP feature makes it practically impossible to use traditional memory dumping tools.

Support for Volatility 3.X

I made a design choice to build this project around supporting Volatility version 3. One of the biggest benefits is that this version will auto-detect and download the appropriate OS profile to analyze the memory image. This is a HUGE win because it was a pain to determine the correct profile and if you didn’t have a profile you had to make your own. Not to mention Volatility v3 is on the horizon and v2 will be phased out so this project has built-in support for the future of Volatility.

DumpIt vs. WinPmem

Typically, my go-to tool of choice is WinPmem for full memory dumps but for some reason, the memory images being generated were not working with Volatility. I tried DumpIt and the memory images generated worked with Volatility 2 and 3. I am not sure if Winpmem doesn’t support newer versions of Windows 10 or Volatility can’t read it’s output but nonetheless it was not a viable option. As discussed above, if you want to use WinPmem the architecture of the project allows for that to happen.

Volatility output

In the screenshots above I demonstrated the output of Volatility in CSV format but personally I prefer JSON. The Volatility v3 documentation states it has the ability to output to JSON but during my testing, it kept erroring when I set the output render module to JSON. Unsure of why this was happening I assume it is something they are working on and Volatility v3 is still in beta. I did design the Osquery extension to take in a parameter of output_render which allows the incident responder to select the output render format. If and when Volatility v3 supports JSON this Osquery extension is ready to utilize it.

Alternative tooling

Memory analysis frameworks

• Rekall
• Volatility v2

Memory dumping tools

• glmcdona/Process-Dump
• WinPmem
• KsDumper
• microsoft/ProcDump-for-Linux

Addressing security concerns for memory tools

While memory forensics is a powerful method to analyze a machine, the data stored within memory may contain sensitive information. The risk with these tools is that an adversary could abuse these tools to extract sensitive information from memory such as passwords, keys, browser cookies, and more. The architecture of these extensions allows for the security team to audit the utilization of these memory tools. To audit for malicious invocation of these tools you need to monitor the following criteria: the file path location of the memory tools C:\Users\<user>\AppData\Roaming\byteexec, the parent process should always be osqueryD, and the SHA256 file hash of the binary.

As the security team, you are in control of the applications being used for memory dumping and analysis so you should know the SHA256 hashes of those binaries. The golang byteexec package has a default location of C:\Users\<user>\AppData\Roaming\byteexec to store binaries to disk. You may modify this location but the main point is that knowing the directory path is important for auditing. Lastly, the only process that should be invoking these memory tools is osqueryD. If these memory tools are invoked in any way that does not meet the criteria above, then you should generate an alert.

Lessons learned

I am currently reading a book called “Cracking the Coding Interview” and it is a great book. One interesting part of the book is their matrix to describe projects you worked on and the matrix contains the following sections which are: challenges, mistakes/failures, enjoyed, leadership, conflicts, and what would you’d do differently. I am going to try and use this model at the end of my blog posts to summarize and reflect on the things I learn. I don’t blog to poss things that I know, I blog to learn new things and to share the knowledge of my security research.

Challenges

• Learning a new programming language which is golang
• Reading literature on what appears to be a “religious debate” over the best way to structure a Golang project.
• Understanding that functions starting with a lowercase letter can NOT be exported outside the Golang package. 

Mistakes and failures

The current implementation above was round two of this project. My first implementation relied on the Winpmem and ProcDump binaries to exist on the system for this extension to function. Not only was this implementation not operationally friendly, but it also had security implications and WinPmem was generating images that weren’t capable with Volatility. The security implication is that someone could overwrite those binaries to have their payload executed with Administrative privileges. Personally, I prefer doing things the right way over “it works”, so I did some digging on Google and found “go-bindata” which allows me to compile the needed binaries into this extension.

Enjoyed

• I loved that I got to engineer a new security mechanism for my favorite open source project Osquery
• Enjoyed learning Volatility version 3. The last time I did memory forensics I used Volatility 2.6
• Biggest Golang code base to date for all my open source projects – 561 lines.

What You’d Do Differently

• If Golang projects existed to perform memory dumps I would have preferred to use them. Instead, my project has to rely on existing tooling and having to interact with binaries is not easy. Also if you want to change the functionality of how a binary is invoked you have to re-compile the Osquery extension.
• For some reason, ProcDump kept exiting with an exit code of 4294967294. I have no idea what this error means but it doesn’t seem to affect the output. Instead of digging and fixing the issue, I made an exception for it.
• Storage alternatives for memory tools
  • Encrypting the memory tools on disk so only the Osquery extensions can use those binaries.
  • Instruct golang to make the memory tools on disk owned by SYSTEM so they cannot even be modified by an administrative user.
  • Use a golang project like afero to create a virtual file system in memory

References

• ProcDump v9.0
• Winpmem Documentation
• Github – go-bindata/go-bindata
• Volatility profiles
• Volatility compiling Binaries with Pyinstaller
• EXE file PNG
• Harddrive PNG
• Binary file PNG