Category Archives: Homelab

Hunting malware and malicious MCPs in memory on Kubernetes with FleetDM + Osquery + YARA

As a threat detection engineer, I’ve always wanted the ability to scan for malware in memory at scale. As an Osquery fanboy, I frequently tinkered with the idea of integrating memory forensics into Osquery but was unsuccessful. Recently, Osquery and Fleet introduced new capabilities to scan memory at scale with YARA rules.  The new yara_process table on Osquery provides the ability to scan process memory with a YARA rule. Around the same time, I had the opportunity to collaborate with the Fleet team to introduced authenticated YARA rule distribution. Seeing these two capabilities come together made me realize that hunting for malware in memory at scale with Osquery was no longer just an idea, but was now a reality.

In true MythBusters fashion, I stopped theorizing and started hacking. I built two detection scenarios in my homelab to see whether these new capabilities could detect threats executing entirely in memory. The first simulates a fictitious company’s “mission-critical” Kubernetes-hosted Damn Vulnerable Web App (DVWA) being compromised by a Sliver implant executing entirely in memory inside a Kubernetes container. The second shifts to one of today’s emerging attack surfaces: a malicious local MCP server that steals credentials from a user’s machine. Along the way, we’ll use Fleet to securely distribute YARA rules, Osquery to enumerate processes and scan memory, and a little SQL wizardry to hunt both threats.

Continue reading

Connecting Kubernetes VMs to Proxmox CEPH using FRR

Over Thanksgiving break, I acquired three new-to-me Dell R720 servers with the intention of creating a high-availability Proxmox cluster using CEPH as shared storage. Like many starting this new journey, I found thousands of blog posts and YouTube videos on how to set this up. However, when it comes time to connect VMs running on Proxmox such as Kubernetes nodes to the Ceph cluster, I was unable to find a working solution. The tenacious engineer in me saw this as a challenge, and after A LOT of Googling and late nights, I was finally able to succeed. Below is a proof of concept for how to configure MicroK8s with network access to CEPH via FRR + OSPF.

Continue reading