Here we go again! This blog post is tangential to my previous blog post on creating an Osquery extension with Python but this time we are using golang. Osquery is my favorite open-source security tool and golang is becoming a popular programming language so fusing them together allows us to engineer tools to detect threats. This blog post will build an Osquery-go extension to calculate the CommunityID of a network connection utilizing the Osquery-polylogyx extension pack to monitor network connections. In blog posts to follow, we will correlate network-based events monitored by Zeek and host-based events generated by Osquery using the CommunityID. So follow me again as your adventure guide on this development journey to make an Osquery extension with osquery-go.

NOTICE

Osquery version 4.20 was released on February 10th 2019 which contains zwass PR to include communityID generation in the Osquery project. This blog post should be viewed as a learning exercise on how to create an Osquery extension with osquery-go.

NOTICE

Background

What is Threat Detection Engineering?

In a nutshell, Threat Detection Engineering is related but not limited to the following high-level activities:

• Drive proactive identification of threats to the environment with the rapid deployment of detection controls.
• Liaise with internal teams and especially data owners, establishing relationships focused on spotting new detection opportunities.
• Make threat intelligence consumable by integrating it with SIEM and other tools that are part of the security arsenal (SOAR, EDR, etc).
• Create custom code to aid the detection of malicious activity via security alerts (rules), reports (dashboards) or simply automation (scripting).
• Establish a continuous quality assurance process, with special attention to internal users/customers feedback (SOC, monitoring teams — in special).

What is Osquery?

Osquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. With Osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.

What is communityID?

CommunityID is a new feature being implemented by networking security applications such as Zeek and Suricata. A CommunityID is a hash of the tuple (destination IP address, source IP address, destination port, source port, protocol)  and this tuple defines a unique connection. For example, let’s say Suricata detects malicious activity and when you examine the details of the alert it will include a unique hash as the value of communityID.

What is Osquery-go?

In osquery, SQL tables, configuration retrieval, log handling, etc. are implemented via a robust plugin and extensions API. This project contains Go bindings for creating osquery extensions in Go. To create an extension, you must create an executable binary which instantiates an ExtensionManagerServer and registers the plugins that you would like to be added to osquery. You can then have osquery load the extension in your desired context (ie: in a long running instance of osqueryd or during an interactive query session with osqueryi).

What is Osquery-polylogyx exetnsions?

PolyLogyx OSQuery Extension (plgx_win_extension.ext.exe) for Windows platform extends the core osquery on Windows by adding real time event collection capabilities to osquery on Windows platform. The capabilities are built using the kernel services library of PolyLogyx. The current release of the extension is a ‘community-only’ release. It is a step in the direction aimed at increasing osquery footprint and adoption on Windows platform. With the extension acting as a proxy into Windows kernel for osquery, the possibilities can be enormous.

Install/Setup Golang environment on macOS

1. curl -o ~/Downloads/golang.pkg https://dl.google.com/go/go1.13.3.darwin-amd64.pkg
2. Open Finder to the Downloads folder
3. Install GOLANG
4. Open terminal
5. export GOROOT=/usr/local/go
6. echo 'export GOROOT=/usr/local/go' >> ~/.zshrc
7. go version

Setup Osquery-go environment

1. git clone https://github.com/CptOfEvilMinions/osquery-go-communityid
2. cd osquery-go-communityid
3. go get github.com/kolide/osquery-go
4. go get github.com/Microsoft/go-winio
   1. Pulls down libraries to generate Windows binaries

Code breakdown

func main()

The main function registers the new table/extension with Osquery. First, if any command-line arguments were specified they will be extracted. Second, the NewExtensionManagerServer function is used to initialize a server object which will be used to register our plugin. The server object creates a communication bridge between Osquery and our extension. Third, the RegisterPlugin function takes in three parameters which are the name of the table, a function containing the table column definitions, and the generate function used to create the results of the table.

func CommunityIDColumns()

This function defines the columns of the table and the variable type of each column.

func CommunityIDTableGenerate()

This function extracts all query supplied values necessary to generate a communityID. Next, the GenerateCommunityID function is called to generate a CommunityID hash of the network connection. Lastly, a map of the query-supplied values and the newly generated CommunityID hash is returned to Osqury to be rendered.

func GenerateCommunityID

Compile new Osquery extension

1. Back to terminal
2. GOOS=windows GOARCH=amd64 go build -o osquery_community_id_ext.exe osquery_community_id_ext.go
   1. Compile for Windows
3. GOOS=darwin GOARCH=amd64 go build -o osquery_community_id.ext osquery_community_id_ext.go
   1. Compile for macOS
4. file osquery_community_id_ext.*

Testing newly compiled Osquery extension

1. Open a new terminal tab
2. osqueryi --nodisable_extensions
3. select value from osquery_flags where name = 'extensions_socket';
   1. Copy output from query
4. Open another terminal tab
5. ./osquery_community_id_ext.macho /Users/thunderwagon/.osquery/shell.em
6. Back to the Osqueryi terminal tab
7. SELECT * FROM community_id WHERE src_ip='1.1.1.1' AND src_port=5858 AND dst_ip='8.8.8.8' AND dst_port=80 AND src_port=5858 and protocol=6;
   
8. .quit

Install/Setup Osquery and Osquery extension on Windows 10

Install Osquery on Windows 10

1. Open a browser and browse to https://pkg.osquery.io/windows/osquery-4.0.2.msi
   1. Polyglogyx – “It has been built and tested with 3.2.6. It also works with 3.3.0 and 4.0.x”
2. Install Osquery

Test extension on Windows 10

1. Open a Powershell instance as Administrator
2. cd 'C:\Program Files\osquery'
3. New-Item -Path "C:\Program Files\osquery" -Name "Extensions" -ItemType "directory"
4. [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
5. Invoke-WebRequest -Uri https://github.com/CptOfEvilMinions/osquery-go-communityid/releases/download/v1.1/osquery_community_id_ext.exe -OutFile 'C:\Program Files\osquery\Extensions\osquery_community_id_ext.exe'
6. .\osqueryi.exe --nodisable_extensions
   
7. SELECT value FROM osquery_flags WHERE name='extensions_socket';
8. Open a Powershell instance
9. cd 'C:\Program Files\osquery\Extensions'
10. .\osquery_community_id_ext.exe <value from queery above>
11. Go back to PowerShell instance running Osqueryi
12. SELECT * FROM community_id WHERE src_ip='1.1.1.1' AND src_port=5858 AND dst_ip='2.2.2.2' AND dst_port=80 AND protcol=6;
13. .quit

CommunityID and Polylogyx extensions on Windows 10

Add Osquery-polylogy extensions

1. DISABLE WINDOWS DEFENDER
2. cd $home\Downloads
3. Invoke-WebRequest -Uri https://github.com/polylogyx/osq-ext-bin/archive/master.zip -OutFile osq-ext-bin.zip -MaximumRedirection 3
4. Expand-Archive .\osq-ext-bin.zip -DestinationPath .
5. cd osq-ext-bin-master
   
6. Copy-Item .\plgx_win_extension.ext.exe -Destination 'C:\Program Files\osquery\Extensions\plgx_win_extension.ext.exe'
7. (Get-Content -path .\osquery.flags -Raw) -replace "ProgramData","Program Files" | Set-Content -Path .\osquery.flags
8. Copy-Item .\osquery.flags -Destination 'C:\Program Files\osquery\osquery.flags'
9. Set-Content -Path 'C:\Program Files\osquery\extensions.load' -Value 'C:\Program Files\osquery\Extensions\plgx_win_extension.ext.exe'

Test Osquery-polylogy and win_socket_events table

1. Stop-Service -Name osqueryd
2. cd 'C:\Program Files\osquery'
3. .\osqueryi.exe --flagfile .\osquery.flags
4. .tables
5. SELECT * FROM win_socket_events;
6. .quit

Joining win_socket_events  and CommunityID  tables

1. .\osqueryi.exe --flagfile .\osquery.flags
2. Open another Powershell instance
3. cd 'C:\Program Files\osquery\Extensions'
4. .\osquery_community_id_ext.exe <value from query above>
5. Go back to Osquery Powershell instance
6. SELECT s.local_address, s.local_port, s.remote_address, s.remote_port, s.protocol, c.community_id, s.pid, s.process_name
FROM win_socket_events as s JOIN community_id as c
ON s.local_address=c.src_ip AND
s.local_port=c.src_port AND
s.remote_address=c.dst_ip AND
s.remote_port=c.dst_port AND
s.protocol=c.protocol
WHERE action='SOCKET_CONNECT';
   
7. .quit

Setup Osquery extension for OsqueryD

1. Open Notepad as Adminsitrator
2. Open C:\Program Files\osquery\extensions.load
3. Paste the following:
   
C:\Program Files\osquery\Extensions\osquery_community_id_ext.exe
C:\Program Files\osquery\Extensions\plgx_win_extension.ext.exe

4. Open a Powershell instance as Administrator
5. cd 'C:\Program Files\osquery'
6. icacls .\Extensions /setowner Administrators /t
7. icacls .\Extensions /grant Administrators:f /t
8. icacls .\Extensions /inheritance:r /t
9. icacls .\Extensions /inheritance:d /t
10. .\osqueryi.exe --flagfile .\osquery.flags
11. SELECT s.local_address, s.local_port, s.remote_address, s.remote_port, s.protocol, c.community_id, s.pid, s.process_name FROM win_socket_events as s JOIN community_id as c ON s.local_address=c.src_ip AND s.local_port=c.src_port AND s.remote_address=c.dst_ip AND s.remote_port=c.dst_port AND s.protocol=c.protocol WHERE action='SOCKET_CONNECT';
12. .quit
13. Invoke-WebRequest -Uri https://raw.githubusercontent.com/CptOfEvilMinions/osquery-go-communityid/master/conf/windows/osquery.conf -OutFile 'C:\Program Files\osquery\osquery.conf'
14. Start-Service -name osqueryd

Install/Setup Osquery and Osquery extension on macOS

Install Osquery on macOS

1. Open a browser and browse to https://pkg.osquery.io/darwin/osquery-4.0.2.pkg
   1. Polyglogyx – “It has been built and tested with 3.2.6. It also works with 3.3.0 and 4.0.x”
2. Install Osquery

Test extension on macOS

1. Open a terminal tab
2. sudo mkdir /var/osquery/Extensions
3. COPY executable to /var/osquery/Extensions
4. osqueryi --nodisable_extensions
5. SELECT value FROM osquery_flags WHERE name = 'extensions_socket';
6. Open another terminal tab
7. cd /var/osquery/Extensions
8. ./osquery_community_id_ext.macho <sokcet path from above>
9. .tables
   1. Make sure community_id exists
10. SELECT * FROM community_id WHERE src_ip='1.1.1.1' AND src_port=5858 AND dst_ip='2.2.2.2' AND dst_port=80 AND protocol=6;
11. .quit

Joining CommunityID and process_open_sockets

1. SELECT DISTINCT p.local_address, p.local_port, p.remote_address, p.remote_port, p.protocol, p.state, c.community_id FROM process_open_sockets as p JOIN community_id as c ON c.src_ip= p.local_address AND c.src_port=p.local_port AND c.dst_ip=p.remote_address AND c.dst_port=p.remote_port AND p.protocol !=0 AND p.remote_port !=0 AND c.protocol=p.protocol;

Setup Osquery extension for OsqueryD

1. Open a terminal tab
2. cd /var/osquery
3. curl
4. curl https://raw.githubusercontent.com/CptOfEvilMinions/osquery-go-communityid/master/conf/macos/osquery.flags -o /var/osquery/osquery.flags
5. curl https://raw.githubusercontent.com/CptOfEvilMinions/osquery-go-communityid/master/conf/macos/osquery.conf -o /var/osquery/osquery.conf
6. curl https://raw.githubusercontent.com/CptOfEvilMinions/osquery-go-communityid/master/conf/macos/extensions.load -o /var/osquery/extensions.load
7. sudo cp /var/osquery/com.facebook.osqueryd.plist /Library/LaunchDaemons/com.facebook.osqueryd.plist
8. sudo launchctl load /Library/LaunchDaemons/com.facebook.osqueryd.plist
9. sudo tail -f /var/log/osquery/osqueryd.results.log

Discussion

Github PR for a communityID table

On Tuesday February 4th 2020, I participated in the Osquery office hours where zwass debuted his new PR to the Osquery project for communityID generation. So the timing of my blog post and this PR are very timely. Nonetheless, this exercise was a good learning experience and I hope this blog post is helpful for developing Osquery extensions with golang.

Cross-platform

As you probably noticed above, osquery-go allows for cross-platform compilation on any platform for any platform. Using golang allows for a simple build pipeline because you could use a golang Docker container to build the executable for any platform.

Dependency on Polylogyx for Windows

This blog post relies on the Osquery-Polylogyx extension pack to monitor Windows socket events because Osquery does not provide this functionality. The Osquery-Polylogyx extension pack is a work of art in my opinion and a pinnacle example of threat detection engineering. However, the project only works with version 3.2.6 and 4.0.2 of Osquery and it requires Windows Defender to be disabled. I will make the assumption that if your environment has an EDR product like Carbon Black it will block this for the same reasons as Windows Defender. However, if you can’t afford an EDR product this will get you as close as possible for free but without the prevention capabilities.

Hardware resource consumption

In the previous blog post, I created an osquery-python extension for Windows that consumed 50% of the CPU utilization when idle. I am happy to report that the osquery-go equivalent only consumes less than 1% of CPU and 10MB at idle.

Lessons learned

I am currently reading a book called “Cracking the Coding Interview” and it is a great book!!! One interesting part of the book is their matrix to describe projects you worked on and the matrix contains the following sections which are: challenges, mistakes/failures, enjoyed, leadership, conflicts, and what would you’d do differently. I am going to try and use this model at the end of my blog posts moving forward to summarize and reflect on things I learned but also the challenges. The purpose of this lesson learned section is to distill the experiences from my security research project that should be actively taken into account in future projects and to share the knowledge of my journey.

Challenges

• The biggest challenge of this blog post was learning a new programming language which was golang.
• Figuring out how to get the Polylogyx extensions working on Windows.
• Expressing the golang code in words
• Reading other people’s golang code to understand the osquery-go capabilities

Mistakes/Failures

• Not implementing TravisCI to automatically build new binaries based on new commits

Enjoyed

• I loved that I got to engineer a new security mechanism for my favorite open source project
• Learning how to read golang structs to extract values from the queryContext variable provided by osquery-go
• I frustratedly enjoyed learning how to enforce handling errors in golang. With languages like Python, the error is often abstracted away until it happens but golang enforces you to handle errors and this is helping me learn how to handle different error scenarios.
• Implemented a successful golang project with a “go mod” – requirements.txt equivalent in Python

References

• Github – kolide/osquery-go
• How To Install Go 1.13 on MacOS
• How To Build Go Executables for Multiple Platforms on Ubuntu 16.04
• Extending Osquery with Go
• Process and socket auditing with osquery
• Github – kolide/launcher – launcher-extension.go 
• Osquery SDK and Extensions