System Administration | HoldMyBeer

Category Archives: System Administration

Leave a comment

Homelab, Kubernetes, System Administration

Connecting Kubernetes VMs to Proxmox CEPH using FRR

Over Thanksgiving break, I acquired three new-to-me Dell R720 servers with the intention of creating a high-availability Proxmox cluster using CEPH as shared storage. Like many starting this new journey, I found thousands of blog posts and YouTube videos on how to set this up. However, when it comes time to connect VMs running on Proxmox such as Kubernetes nodes to the Ceph cluster, I was unable to find a working solution. The tenacious engineer in me saw this as a challenge, and after A LOT of Googling and late nights, I was finally able to succeed. Below is a proof of concept for how to configure MicroK8s with network access to CEPH via FRR + OSPF.

Continue reading →

Leave a comment

PoC, System Administration, Tools

Spark, JupyterHub, Minio, and Helm on Kubernetes

At work we recently got Databricks which utilizes open source technologies under the hood. This got me thinking whether I could create a Databricks equivalent with open source software in my homelab. Over the Thanksgiving holiday week, I started playing around with deploying JupyterHub, Minio, and Spark on Kubernetes with Helm. I was able to get a working proof of concept (PoC) that would allow me to read raw log data from Minio using Spark jobs initiated by Python Jupyter notebook to ingest those events into a Spark schema, write that data as a Delta table, and then query said Delta table using a Jupyter notebook.

Continue reading →

Leave a comment

DevOps, Network Security, System Administration, Tools

Connecting to my homelab remotely with Hashicorp Boundary v0.2.0 and Auth0

The purpose of this blog post is to provide multiple methods on how to install/setup Hashicorp Boundary. This blog post generated an Ansible playbook, Docker-compose for Swarm, and manual instructions for installing Boundary on Ubuntu 20.04. In addition, this blog post will demonstrate how to setup Auth0 OIDC authentication for single-sing on. Lastly, I will end this blog post with connecting to a remote machine in my homelab via SSH using the Boundary Desktop and CLI client.

Continue reading →

Leave a comment

Incident Response, IR Tales, Logging, System Administration, Tools

IR Tales: The Quest for the Holy SIEM: Splunk + Sysmon + Osquery + Zeek

This blog post is the season finale in a series to demonstrate how to install and setup common SIEM platforms. The ultimate goal of each blog post is to empower the reader to choose their own adventure by selecting the best SIEM based on their goals or requirements. Each blog post in the series will provide Docker-compose v2, Docker-compose for Swarm, Ansible, Vagrant, and manual instructions to allow the reader to setup each platform with the deployment method of their choosing. In addition to setting up Splunk, I will cover fundamental Splunk concepts such as the Common Information Model (CIM). Lastly, I will provide step-by-step instructions to install Sysmon + Splunk Universal Forwarder on Windows, Osquery + Splunk Universal Forwarder on Ubuntu, and Zeek + Filebeat to ship logs to Splunk.

Continue reading →

Leave a comment

Logging, Network Security, System Administration, Tools

Implementing Logstash and Filebeat with mutual TLS (mTLS)

Do you know if your Filebeat client is connecting to a rogue Logstash server? Do you know if your Logstash server is accepting random logs from random devices? If you have answered “I don’t know” to either of these questions then this blog post is for you. The purpose of this blog post is to provide instructions on how to setup Logstash and Filebeat with mutual TLS (mTLS). The step-by-step instructions in this post, will demonstrate how to create the certificate chain of trust using Vault. Lastly, I will cover the Python script I created to automate constructing this logging certificate chain of trust.

Continue reading →

Leave a comment

IR Tales, Logging, System Administration, Tools

IR Tales: The Quest for the Holy SIEM: Graylog + AuditD + Osquery

This blog post is the second in a series to demonstrate how to install and setup common SIEM platforms. The ultimate goal of each blog post is to empower the reader to choose their own adventure by selecting the best SIEM based on their goals or requirements. Each blog post in the series will provide Docker-compose v2, Docker-compose for Swarm, Ansible, Vagrant, and manual instructions to allow the reader to setup each platform with the deployment method of their choosing. This blog post will also cover how to setup the Graylog with Elasticsearch and Mongo. In addition to setting up the Graylog I will provide instructions to install Osquery + Filebeat on Windows and AuditD + Auditbeat on Ubuntu to ship logs to Elastic.

Continue reading →

Leave a comment

DevOps, System Administration, Tools

Getting started with Hashicorp Vault v1.6.1

The purpose of this blog post is to provide multiple methods on how to install/setup Vault. This blog post generated an Ansible playbook, Docker-composes for Swarm and non-swarm, and manual instructions for installing Vault on Ubuntu 20.04. Additionally, over the past couple of months, I have been learning Vault and demonstrating different ways to incorporate Vault. This blog post will be a condensed version of the content in those blog posts and a jumping off point to those blog posts as well.

Continue reading →

Leave a comment

IR Tales, Logging, System Administration, Tools

IR Tales: The Quest for the Holy SIEM: Elastic stack + Sysmon + Osquery

This blog post is the first in a series to demonstrate how to install and setup common SIEM platforms. The ultimate goal of each blog post is to empower the reader to choose their own adventure by selecting the best SIEM based on their goals or requirements. Each blog post in the series will provide Docker-compose v2, Docker-compose for Swarm, Ansible, Vagrant, and manual instructions to allow the reader to setup each platform with the deployment method of their choosing. This blog post will cover how to setup the Elastic stack formerly known as ELK. In addition to setting up the Elastic stack I will provide instructions to install Sysmon + Winlogbeat on Windows and Osquery + Filebeat on Ubuntu to ship logs to Elastic.

Continue reading →

Incident Response, Logging, System Administration, Tools

Getting started with FleetDM

The purpose of this blog post is to provide multiple methods on how to install/setup FleetDM, how to deploy Osquery, and demonstrate how to use features of FleetDM + FleetCTL. This blog post generated an Ansible playbook, Docker-composes for Swarm and non-swarm, Vagrant to create a VM, and manual instructions for installing FleetDM on Ubuntu 20.04. Additionally, there are Ansible playbooks for deploying the Osquery agent on Windows and Ubuntu with manual instructions as well. Lastly, I will end by demonstrating how to use the FleetDM WebGUI and FleetCTL tool to manage FleetDM and interact with your Osquery agents.

Continue reading →

Leave a comment

DevOps, System Administration, Tools

Vault: Connecting entities, auth backends, groups, and policies OH MY

While working on my osquery-file-carve-server project I determined my application needed authentication. However, I didn’t want to pigeon hole my application to a single platform/service for authentication. After some research, I decided to implement support for Vault into my application because it provides the ability for users to authenticate using various methods. However, during my research, I had a hard time understanding how the various Vault components connected to create this functionality.

This blog post will provide an understanding of the Vault components used to implement this functionality. In addition, it will demonstrate the relationship between the various Vault components: authentication backends, entities, groups, and policies. The final result of combining these Vault components is a system that can authenticate a single user using different authentication services.

Continue reading →

Leave a comment

Incident Response, Logging, PoC, System Administration, Tools

Setting up Kolide and Osquery with client certificates for mutual TLS (mTLS)

Do you know if your Osquery client is connecting to the right server? Do you know if your Kolide server is accepting requests from rogue devices? If you have answered “I don’t know” to either of these questions then this blog post is for you. This blog post will be demonstrating how to setup Kolide + Osquery with mutual TLS (mTLS). Mutual TLS is a mechanism that can be implemented to verify the identity of the server and clients. Continue reading →

PoC, System Administration, Tools

Install/Setup Vault for PKI + NGINX + Docker – Becoming your own CA

Hashicorp Vault (Vault) is an open-source tool for managing secrets. This blog post will demonstrate how to use Vault to generate a root CA for trusted TLS communication and how to generate client certificates for mutual TLS communication. Not only does this blog post contain a high-level overview of Vault, it includes working infrastructure-as-code and step-by-step tutorial.

Continue reading →

Leave a comment

DevOps, System Administration, Tools

My Homelab Docker setup

Just like my latest post on my logging pipeline, people want to know more about my Docker set up to learn from or replicate. This blog post is my attempt to share my Docker set up as a framework for newcomers. The hope is that the explanation of the architecture, design decisions, working infrastructure-as-code, and the knowledge I accumulated over the years will be beneficial to the community.

Continue reading →

Leave a comment

DevOps, Incident Response, Logging, Network Security, PoC, System Administration, Threat Detection Engineering, Tools

Reducing your alert fatigue with AskJeevesSecBot

In incident response, there is a disconnect between a security alert being generated and a user’s confirmation of the security alert. For example, generating an alert every time a user runs “curl” on a production system would generate a bunch of false positives that can lead to what is called “alert fatigue”. But if we extend our incident response capabilities to include the user as part of the triage process we could reduce the number of alerts. This blog post is going to demonstrate AskJeevesSecBot which is an open-source proof of concept (POC) of how to integrate Slack and user responses into your security pipeline, specifically during the triage phase of the incident response process. In addition to a PoC, this blog post will also provide a deep dive into the architecture of this project, design decisions, and lessons learned as an evolving threat detection engineer.

Continue reading →

Incident Response, Logging, PoC, System Administration, Tools

My logging pipeline: Splunk, Logstash, and Kafka

Over the years I have built several logging pipelines within my homelab and each used different technologies and methodologies but now I have finally built a pipeline that suites my needs. When I tell people about my pipeline they usually ask if I have a blog post on it because they want to know more or replicate it. This blog post is my attempt to share my logging pipeline as a framework for newcomers. The hope is that the explanation of the architecture, design decisions, working infrastructure-as-code, and the knowledge I accumulated over the years will be beneficial to the community. Continue reading →

System Administration, Tools

My journey for upgrading Proxmox VE 5.4 to 6.0

Most guides on the internet show you how to upgrade Proxmox 5.4 to 6.0 via the built-in mechanism. However, for major version upgrades, I prefer to do an installation from scratch, NOT applying the update via apt-get upgrade. This method of upgrading allows me to clean up any crud that has accumulated over the years. Lastly, this guide will cover how to backup your VMS before upgrading to Proxmox 6.0. Continue reading →

System Administration, Tools

Install/Setup Kolide Fleet + Graylog + OSQuery with Windows and Linux deployment

In this blog post we will be installing, setting up, and utilizing Kolide Fleet as our OSQuery fleet manager. As stated by Kolide, ” Fleet is a state of the art host monitoring platform tailored for security experts. Leveraging Facebook’s battle-tested OSQuery project, Fleet delivers fast answers to big questions.” In future blog posts I plan on using this tool for incident response and threat hunting scenarios.

Continue reading →

Red Teaming, System Administration, Threat Hunting, Tools

Part 1: Intro to Threat Hunting with Powershell Empire, Windows event logs, and Graylog

One of the biggest trends in infosec, besides the word cyber, is threat hunting. First, I want to start by defining threat hunting as the action of “investigation without cause” and this concept is nothing new. It’s been around for years but we didn’t have a catchy marketing term associated with it. In this post, I will breakdown the Sqrrl threat hunting model, Powershell Empire for adversary activity, and instructions on setting up Graylog for log aggregation and a search platform to perform threat hunting. Finally, I would like to point out all Ansible playbooks used in this post are publicly accessible on my Github page in a repo called “AgileFalcon“.

Continue reading →

System Administration, Tools

Setup/Configure OPNsense router in AWS VPC

For the longest time I was under the assumption a router in an AWS VPC could only have one NIC. Writing IPtables/firewall rules for one interface can be tedious!!! However, I have discovered how to setup OPSense in an AWS VPC with multiple NICs. That is right, you will have a WAN NIC(public subnet) and a NIC for each private subnet.

Continue reading →

Incident Response, System Administration, Tools

Install/Setup Wazuh 2.0, ELK 5.0, and client deployment

Visualize, analyze and search your host IDS alerts. Elastic Stack is the combination of three popular Open Source projects for log management, known as Elasticsearch, Logstash and Kibana(ELK). Together they provide a real-time and user-friendly console for your OSSEC alerts. OSSEC Wazuh integration with Elastic Stack comes with out-of-the-box dashboards for PCI DSS compliance and CIS benchmarks. You can do forensic and historical analysis of OSSEC alerts and store your data for several years, in a reliable and scalable platform. This post is updating a pervious post of mine using Wazuh 1.0 and version 2.0 of the ELK stack. This post will contain a general setup and configuration for a central logging server.

Continue reading →