This blog post is going to cover how to ingest OSquery logs with Rsyslog v8. Most setups I have come across have Rsyslog ingesting the logs from disk, but this setup will ingest logs via the system journal. OSquery supports writing logs to disk and to the system journal. This post also contains a setup via Ansible and a manual walkthrough. Lastly, explanations of Rsyslog and OSquery configs.

Goals

Setup OSquery logging
Ship logs with Rsyslogclient
- Collect logs via SYSLOG pipeline
Ingest logs with Rsyslog server
Deploy with a simple logging infrastructure with Ansible
Learn about Rsyslog configs
Learn about OSquery configs

Background

OSquery

OSquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.

Rsyslog

Rsyslog is a rocket-fast system for log processing. It offers high-performance, great security features and a modular design. While it started as a regular syslogd, rsyslog has evolved into a kind of swiss army knife of logging, being able to accept inputs from a wide variety of sources, transform them, and output to the results to diverse destinations.

Install/Setup Rsyslog v8 on Ubuntu 18.04

Manual install

Install/Setup Rsyslog v8

sudo su
sudo add-apt-repository ppa:adiscon/v8-stable
1. Press enter
sudo apt-get update -y
sudo apt-get install rsyslog rsyslog-relp rsyslog-mmjsonparse -y
wget https://raw.githubusercontent.com/CptOfEvilMinions/BlogProjects/master/osquery_rsyslog/conf/rsyslog_server/rsyslog.conf -O /etc/rsyslog.conf
1. sed -i 's#{{ rsyslog_output_module }}#imrelp#g' /etc/rsyslog.conf
2. sed -i 's#{{ rsyslog_input_module }}#imrelp#g' /etc/rsyslog.conf
3. sed -i 's#{{ rsyslog_port }}#1514#g' /etc/rsyslog.conf
4. sed -i 's#{{ rsyslog_tls }}#on#g' /etc/rsyslog.conf
wget https://raw.githubusercontent.com/CptOfEvilMinions/BlogProjects/master/osquery_rsyslog/conf/rsyslog_server/50-default.conf -O /etc/rsyslog.d/50-default.conf
wget https://raw.githubusercontent.com/CptOfEvilMinions/BlogProjects/master/osquery_rsyslog/conf/rsyslog_server/60-osquery-to-disk.conf -O /etc/rsyslog.d/60-osquery-to-disk.conf
chmod 400 /etc/rsyslog.d/*.conf
mkdir /var/log/rsyslog
chown syslog:adm /var/log/rsyslog
systemctl start rsyslog && systemctl enable rsyslog

UFW firewall

ufw enable
1. Enter “Y”, if asked about disrupting existing ssh connections
ufw allow 22/tcp
ufw allow 1514/tcp
ufw reload

Ansible deployment

git clone https://github.com/CptOfEvilMinions/BlogProjects.git
cd BlogProjects/osquery_rsyslog
vim hosts.ini and set IP address under rsyslog-server
mv group_vars/all.yml.example group_vars/all.yml and set:
1. slack_token – OPTIONAL for Slack notifications
2. slack_channel – OPTIONAL for Slack notifications
mv group_vars/logging.yml.example group_vars/logging.yml
1. rsyslog_host – FQDN/IP address of Rsyslog server
2. rsyslog_port – Port to ingest logs
3. rsyslog_input_module – Module for Rsyslog to ingest logs
  1. Default set to imrelp
4. rsyslog_output_module – Module for Rsyslog to send logs
  1. Default set to omrelp
  2. If using RELP for ingesting/sending logs you can enable TLS with rsyslog_tls
ansible-playbook -i hosts.ini deploy_rsyslog_server.yml -u <username> -K

Install/Setup OSquery + RSYSLOG client

Manual install

Install/Setup OSQuery on Ubuntu 18.04

sudo su
export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys $OSQUERY_KEY
sudo add-apt-repository 'deb [arch=amd64] https://pkg.osquery.io/deb deb main'
sudo apt-get update -y
sudo apt-get install osquery -y
wget https://raw.githubusercontent.com/CptOfEvilMinions/BlogProjects/master/osquery_rsyslog/conf/osquery/osquery.conf -O /etc/osquery/osquery.conf
1. sed -i 's#{{ osquery_logger_path }}#/var/log/osquery#g' /etc/osquery/osquery.conf
2. sed -i 's#{{ osquery_pidfile }}#/var/osquery/osquery.pidfile#g' /etc/osquery/osquery.conf
3. sed -i 's#{{ osquery_database_path }}#/var/osquery/osquery.db#g' /etc/osquery/osquery.conf
4. sed -i 's#{{ osquery_packs }}#/usr/share/osquery/packs#g' /etc/osquery/osquery.conf
systemctl start osqueryd && systemctl enable osqueryd

Install/Setup Rsyslog v8

sudo add-apt-repository ppa:adiscon/v8-stable
sudo apt-get update -y
sudo apt-get install rsyslog rsyslog-relp rsyslog-mmjsonparse -y
wget https://raw.githubusercontent.com/CptOfEvilMinions/BlogProjects/master/osquery_rsyslog/conf/rsyslog_client/rsyslog.conf -O /etc/rsyslog.conf
1. sed -i 's#{{ rsyslog_output_module }}#omrelp#g' /etc/rsyslog.conf
wget https://raw.githubusercontent.com/CptOfEvilMinions/BlogProjects/master/osquery_rsyslog/conf/rsyslog_server/50-default.conf -O /etc/rsyslog.d/50-default.conf
wget https://raw.githubusercontent.com/CptOfEvilMinions/BlogProjects/master/osquery_rsyslog/conf/rsyslog_client/60-osquery.conf -O/etc/rsyslog.d/60-osquery
systemctl start rsyslog && systemctl enable rsyslog

Ansible deployment

vim hosts.ini and set IP address under rsyslog-client
ansible-playbook -i hosts.ini deploy_rsyslog_osquery_client.yml -u <username> -K

Verify setup

Rsyslog server

rsyslogd -N 1
netstat -tnlp – Rsyslog should be running on port 1514
cd /var/log/rsyslog && ls – Directory should have directories of hostnames. This means RSYSLOG is receiving logs and storing them to disk
cat /var/log/rsyslog/<ubuntu>/osquery/<year>/<month>/<date>/osquery.log
tcpdump port 1514

Rsyslog + OSquery client

rsyslogd -N 1
systemctl status osqueryd
systemctl status rsyslog
cat /var/log/osquery/osqueryd.results.log
tcpdump port 1514

Let’s talk configs

Client configs

rsyslog.conf – client

#######################################################################
#
# RSYSLOG configuration file
# The settings in this file are global for all of RSYSLOG
#
#######################################################################
################################### MODULES ###########################
#
# Like in Python or any other programming language, you need to import 
# modules for functionality. The following modules are loaded globally 
# and can be used in any RSYSLOG config. Modules may also be loaded on 
# a per config basis.
#
# - IMjournal: Reads logs from the journal system
# - IMfile: Module to read files from disk. The "PollingInterval" 
#   setting specifies how often files are to be polled for new data.
# - OMrelp: Module used to send logs to a remote server using the RELP 
#   protocol
# - MMjsonprase: This module is used to parse JSON logs
#
#######################################################################
module(load="imjournal") # provides access to the systemd journal
module(load="imfile" PollingInterval="10")
module(load="omrelp")
module(load="mmjsonparse")


################################ GLOBAL DIRECTIVES ####################
#
# This section should include settings/variables that will be global 
# for all of RSYSLOG.
#
# Privileges used to run RSYSLOG
#   - PrivDropToUser
#   - PrivDropToGroup
#
# These privileges may restrict RSYSLOG from reading certain logs. For 
# example, OSquery writes a log to disk with user/group permissions of 
# root and ONLY root can read it. This means RSYSLOG can not read those
# logs with the permissions above. This is why we are reading the logs
# from journal :).
#
#  - ActionFileDefaultTemplate: The default template for how RSYSLOG 
#    will format messages when writing to disk or sending logs to a 
#    remote server.
# 
#  - Max Message Size: Maximum message size for a log. The server MUST 
#    support a value greater than or equal to this variable. This 
#    setting is good for application crashes with long tracelogs. 
#
#  - IMJournalStateFile: Specifies where the state file for persisting 
#    journal state is located.
#
#######################################################################
$PrivDropToUser syslog
$PrivDropToGroup adm

# Where to place auxiliary files
$WorkDirectory /var/spool/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Max message size
$MaxMessageSize 64k


# File to store the position in the journal
$IMJournalStateFile imjournal.state


######################## IncludeConfigs ###############################
# This directive will load all the confings from the config specified
#######################################################################
$IncludeConfig /etc/rsyslog.d/*.conf

#######################################################################
# end of the forwarding rule #
#######################################################################

50-default.conf

#######################################################################
#
# This is an RSYSLOG config that is placed in /etc/rsyslog.d and loaded
# by the $IncludeConfig directive in rsyslog.conf.
#
# This file contians RSYSLOG rules for system logs. The system RSYSLOG 
# is running on is still generating logs. For this setup, we want to 
# send OSQuery logs to a remote sever BUT I still want the system logs 
# to be recorded to the disk. For example, I still want authentication 
# logs to be recorded in /var/log/auth.log
#
#######################################################################
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
kern.*                                         /dev/log/kern.log

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none       /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                     /var/log/auth.log

# Log all the mail messages in one place.
mail.*                                         -/var/log/maillog


# Log cron stuff
cron.*                                         /var/log/cron

# Everybody gets emergency messages
*.emerg                                        :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                 /var/log/spooler

# Save boot messages also to boot.log
local7.*                                       /var/log/boot.log

60-osquery.conf

#######################################################################
#
# This is an RSYSLOG config that placed in /etc/rsyslog.d and loaded 
# by the $IncludeConfig directive in rsyslog.conf.
#
# The if statement is evaluated first. The if statement is looking at 
# JOURNAL logs(module(load="imjournal")) contatining a program name 
# of "osqueryd" and logs that DO NOT contain ".cpp". If the logs 
# message meets these requirements, the action block is executed. The
# action block is telling RSYSLOG to send it to the remote 
# server(192.168.1.129), using port 1514, using TLS, and using the 
# TraditionalForwardFormat template.
#
# The template starts with a name directive to identify itself, like
# a function name. Next, we are stating the message format will be 
# defined as a string. Finally, the string directive is defining how 
# the log message will be sent to the server. 
#
# Anyhting wrapped in "%" is an RSYSLOG variable. The first variable 
# is "<%PRI%>", which tells RSYSLOG to ship the logs severity and 
# facility. The facility indicates where the message originated from 
# (e.g. kernel, mail subsystem) while the severity provides a glimpse 
# of how important the message might be (e.g. error or informational. 
# The second variable is %TIMESTAMP%, which tells RSYSLOG to include 
# the current time the log was ingested by RSYSLOG - NOT THE TIME 
# generated by the original log creator.
#
# The third variable is %HOSTNAME%, which includes the hostname of the 
# system RSYSLOG is running on. This is helpful when trying to track 
# down which system "generated" or rather which system sent the log. 
# The fourth variable "osquery", usually has "%syslogtag% but 
# I couldn't find a good way to set the syslog tag. The last variable 
# is USUALLY  the %msg% variable which is the actual log. The 
# "msg:::sp-if-no-1st-sp" directive is used for formatting and more 
# information can be found here on it:
#
# https://www.rsyslog.com/doc/v8-stable/configuration/property_replacer.html
# 
# After the action block there is another "action" which is "stop". 
# The /etc/rsyslog.d directory may contain lots of configs and each 
# config will evaluate each log. However, this config is the ONLY 
# config evaluating OSquery logs. Therefore, we can tell RSYSLOG 
# to move on to the next log after this config has evaluated a log 
# containing "osqueryd".
#
#######################################################################
template (
    name="TraditionalForwardFormat"
    type="string"
    string="<%PRI%>%TIMESTAMP% %HOSTNAME% osquery%msg:::sp-if-no-1st-sp%%msg%\n"
)

if ($programname == "osqueryd") and not ($msg contains ".cpp:") then {
    action(
        type="omrelp"
        Target="192.168.1.129"
        Port="1514"
        tls="on"
        Template="TraditionalForwardFormat"
    )
    stop
}

osquery.conf

#######################################################################
#
# OSquery.conf
#
#######################################################################
#######################################################################
#
# OSquery flags/options
#
# - config_plugin - Define where the OSquery config comes from. This 
#   config says the config comes from disk(/etc/osquery/osquery.conf). 
#   This option can be set to TLS and a config can come from a server.
#
# - logger_plugin - Define what OSquery should do with the logs. This
#   config says, OSquery should send the logs to disk(/var/log/osquery)
#   and send logs to the SYSLOG journal.
#
# - schedule_default_interval - The default interval to run queries
#
# - host_identifier - The host identifier for the logs. This option
#   can be set to the hostname OR the hosts UUID.
#
####################################################################### 
{
  "options": {
    "config_plugin": "filesystem",
    "logger_plugin": "filesystem,syslog",
    "logger_path": "{{ osquery_logger_path }}",
    "logger_snapshot_event_type": "true",
    "disable_logging": "false",
    "log_result_events": "true",
    "schedule_splay_percent": "10",
    "pidfile": "{{ osquery_pidfile }}",
    "events_expiry": "3600",
    "database_path": "{{ osquery_database_path }}",
    "verbose": "false",
    "worker_threads": "2",
    "enable_monitor": "true",
    "disable_events": "false",
    "disable_audit": "false",
    "audit_allow_config": "true",
    "audit_allow_sockets": "true",
    "host_identifier": "hostname",
    "schedule_default_interval": "3600", 
    "enable_syslog": "false"
  },

#######################################################################
# 
# Define the platform OSquery is running on
# 
#######################################################################

  "platform": "linux",

#######################################################################
# 
# OSquery queries to run
# 
#######################################################################

  "schedule": {
    "process_events":{
      "query": "SELECT auid, cmdline, ctime, cwd, egid, euid, gid, parent, path, pid, time, uid FROM process_events WHERE path NOT IN ('/bin/sed', '/usr/bin/tr', '/bin/gawk', '/bin/date', '/bin/mktemp', '/usr/bin/dirname', '/usr/bin/head', '/usr/bin/jq', '/bin/cut', '/bin/uname', '/bin/basename') and cmdline NOT LIKE '%_key%' AND cmdline NOT LIKE '%secret%';",
      "interval": 10
    },
    "socket_events":{
      "query": "SELECT action, auid, family, local_address, local_port, path, pid, remote_address, remote_port, success, time FROM socket_events WHERE success=1 AND path NOT IN ('/usr/bin/hostname') AND remote_address NOT IN ('127.0.0.1', '169.254.169.254', '', '0000:0000:0000:0000:0000:0000:0000:0001', '::1', '0000:0000:0000:0000:0000:ffff:7f00:0001', 'unknown', '0.0.0.0', '0000:0000:0000:0000:0000:0000:0000:0000');",
      "interval": 10
    },
    "disk_space": {
      "query": "select path, round((blocks_available * blocks_size *10e-10),2) as gigs_free, round((blocks_free*1.0/blocks * 100),2) as percent_free from mounts where path='/';",
      "interval": 300
    },
    "python_packages": {
      "query": "SELECT * FROM python_packages;",
      "interval": 300
    }
  },
#######################################################################
# 
# File intergirty monitoring
# 
# file_paths:
#  - configuration: A list of directories to monitor for config 
#    changes, such as /etc/passwd
#
#  - bianries: A list of directories to monitor for binary changes
# 
#######################################################################
  "file_paths": {
    "configuration": [
      "/etc/passwd",
      "/etc/shadow",
      "/etc/ld.so.conf",
      "/etc/ld.so.conf.d/%%",
      "/etc/pam.d/%%",
      "/etc/resolv.conf",
      "/etc/rc%/%%",
      "/etc/my.cnf",
      "/etc/hosts",
      "/etc/hostname",
      "/etc/fstab",
      "/etc/crontab",
      "/etc/cron%/%%",
      "/etc/init/%%",
      "/etc/rsyslog.conf",
      "/etc/ssh",
      "/etc/bash.bashrc",
      "/etc/environment",
      "/etc/fail2ban/%%",
      "/etc/logrotate.d/%%",
      "/etc/osquery/%%",
      "/etc/systemd/%%",
      "/etc/ufw/%%"
    ],
    "binaries": [
      "/usr/bin/%%",
      "/usr/sbin/%%",
      "/bin/%%",
      "/sbin/%%",
      "/usr/local/bin/%%",
      "/usr/local/sbin/%%"
    ]
  }, 
#######################################################################
# 
# OSquery packs - A collection of queries for a specific assestment.
# For example, there is a pack to detect rootkits or a pack to detect
# IT compliance.
# 
#######################################################################
  "packs": {
    "osquery-monitoring": "{{ osquery_packs }}/osquery-monitoring.conf",
    "incident-response": "{{ osquery_packs }}/incident-response.conf",
    "it-compliance": "{{ osquery_packs }}/it-compliance.conf",
    "vuln-management": "{{ osquery_packs }}/vuln-management.conf",
    "ossec-rootkit": "{{ osquery_packs }}/ossec-rootkit.conf"
  }
}

Server configs

rsyslog.conf – server

#######################################################################
#
# The settings in this file are global for all of RSYSLOG 
#
#######################################################################

################################### MODULES ###########################
#
# Like in Python or any other programming language, you need to import 
# modules for functionality. The following modules are loaded globally 
# and can be used in any RSYSLOG config. Modules may also be loaded on 
# a per config basis. 
# 
# - IMjournal: Reads logs from the journal system 
# - MMjsonprase: This module is used to parse JSON logs
#
######################################################################
module(load="imjournal") # provides access to the systemd journal
module(load="mmjsonparse")

################################### Inputs ############################
#
# This section defines all the inputs to ingest logs from remote 
# clients.
#
# - imrelp: This moduel allows you to ingest logs from remote clients 
#   using the RELP protocol
# - input:
#   - type: Defines the module to use for this input
#   - port: Port to ingest logs on
#   - tls: Enable TLS on the logs
#
#######################################################################
module(load="imrelp")
input(
    type="imrelp"
    port="1514"
    tls="on"
)

################################## GLOBAL DIRECTIVES ##################
#
# This section should include settings/variables that will be global 
# for all of RSYSLOG. 
# 
# Privileges used to run RSYSLOG 
# - PrivDropToUser 
# - PrivDropToGroup 
# 
# These privileges may restrict RSYSLOG from writing certain locations.
# By default, RSYSLOG can not write logs to /var/log because only root
# can. Above we created the /var/log/rsyslog directory and then we set
# permissions on the directoru
# 
# - ActionFileDefaultTemplate: The default template for how RSYSLOG 
#   will format messages when writing to disk. 
# 
# - WorkDirectory: location for spool files
#
# - Max Message Size: Maximum message size for a log that can be 
#   ingested.
# 
# - IMJournalStateFile: Specifies where the state file for persisting 
#   journal state is located. 
#
#######################################################################
$PrivDropToUser syslog
$PrivDropToGroup adm

# Where to place auxiliary files
$WorkDirectory /var/spool/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Max message size
$MaxMessageSize 64k


# File to store the position in the journal
$IMJournalStateFile imjournal.state

############################# IncludeConfigs ##########################
# This directive will load all the confings from the config specified #######################################################################
$IncludeConfig /etc/rsyslog.d/*.conf

#######################################################################
# end of the forwarding rule #
#######################################################################

60-osquery-to-disk.conf

#######################################################################
# 
# This is an RSYSLOG config that placed in /etc/rsyslog.d and loaded 
# by the $IncludeConfig directive in rsyslog.conf.
#
# The if statement is evaluated first. The if statement is logging for
# all logs that have a program name of "osquery". If a log matches this
# condition RSYSLOG calls the "osquery-output" "function"/ruleset. This
# ruleset contains an action block to write files to disk. The action
# type is "omfile" to write files to disk, to a dynamic filepath
# (dynafile), with the log template of "osquery-temp".
#
# The osquery-dyn template will save logs to filepath defined below. 
# However, the filepath will change based on the hostnmae the log 
# originates from, the current year, the current month, the current
# day. This means OSquery logs will be saved in a directory by 
# hostname, by year, by month, and by day. This provides granularity 
# of where to look for a log based on a hostname and timeframe.
#
# The osquery-temp tempalte defines how the log will be written to 
# disk. The only part of log we care about is the %msg% and strip 
# other attributes like <%PRI%>, %timestamp%, %hostname%, and etc. 
# This will only write the logs generated by OSquery which is in 
# JSON foramt.
#
#######################################################################
template(
    name="osquery-dyn"
    type="string"
    string="/var/log/rsyslog/%HOSTNAME%/%programname%/%$YEAR%/%$MONTH%/%$DAY%/%programname%.log"
)

template (
    name="osquery-temp"
    type="string"
    string="%msg%\n"
)

##### Ruleset #####
ruleset (name="osquery-output") {
    action(
        type="omfile"
        dynafile="osquery-dyn"
        template="osquery-temp"
    )
}

##### Output #####
if $programname == 'osquery' then {
    call osquery-output
    stop
}

HoldMyBeer

Cause every great story starts with "Hold my beer"

Logging OSquery with Rsyslog v8 – Love at first sight

Goals

Background

OSquery

Rsyslog

Install/Setup Rsyslog v8 on Ubuntu 18.04

Manual install

Install/Setup Rsyslog v8

UFW firewall

Ansible deployment

Install/Setup OSquery + RSYSLOG client

Manual install

Install/Setup OSQuery on Ubuntu 18.04

Install/Setup Rsyslog v8

Ansible deployment

Verify setup

Rsyslog server

Rsyslog + OSquery client

Let’s talk configs

Client configs

rsyslog.conf – client

50-default.conf

60-osquery.conf

osquery.conf

Server configs

rsyslog.conf – server

60-osquery-to-disk.conf

Resources/Sources

Leave a Reply Cancel reply