Graylog has released version 3 with new features and major changes. This blog post will explain how to setup up Graylog version 3 on an Ubuntu server. Once Graylog is running, we will explore setting up logging clients, logging inputs, data extractors, threat intel pipelines, Slack alerts, dashboards and more.

Intro

This blog post is how to setup up Graylog version 3 on an Ubuntu server 18.04 with your choice of a manual install, Ansible, or Docker. Once Graylog is running, I have instructions on shipping NGINX logs with Rsyslog and Zeek/BRO logs in JSON format with Filebeat. Once the logs are ingested, we will create logging inputs, data extractors, pipelines for threat intelligence, Slack alerts, and a dashboard to view Zeel logs.

Goals

Install and setup Graylog v3
- Manual install
- Ansible
- Docker
Learn how to ship logs to Graylog with Rsyslog and Filebeat
- Shipping Zeek/Bro logs in JSON format with Filebeat
- Shipping NGINX logs with Rsyslog
Setting up a central logging server
Graylog features
- pipelines
- content packs
- logging input
- data extractors
- alerts
- dashboards

Install/Setup Graylog 3 on Ubuntu 18.04

Manual install/setup

Install things

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen -y

Install/Setup Mongo

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
sudo apt-get update -y
sudo apt-get install -y mongodb-org
sudo systemctl daemon-reload
sudo systemctl enable mongod.service
sudo systemctl restart mongod.service

Install/Setup Elasticsearch

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update && sudo apt-get install elasticsearch-oss
sed -i 's/#cluster.name: my-application/cluster.name: graylog/g' /etc/elasticsearch/elasticsearch.yml
echo "action.auto_create_index: false" >> /etc/elasticsearch/elasticsearch.yml
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl restart elasticsearch.service

Install/Setup Graylog

wget https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.deb
sudo dpkg -i graylog-3.0-repository_latest.deb
sudo apt-get update && sudo apt-get install graylog-server
echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1
1. Copy the output
sed -i 's/root_password_sha2 =/root_password_sha2 = <Output from above>/g /etc/graylog/server/server.conf
pwgen -N 1 -s 96
sed -i 's/password_secret =/password_secret = <Output from above>/g' /etc/graylog/server/server.conf
sudo systemctl daemon-reload
sudo systemctl enable graylog-server.service
sudo systemctl start graylog-server.service

Install/Setup NGINX + HTTPS

sudo apt-get install nginx -y
mkdir /etc/nginx/ssl
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/graylog.key -out /etc/nginx/ssl/graylog.crt
1. Enter country
2. Enter state
3. Enter city
4. Enter organization name
5. Enter organization unit name
6. Enter common name
openssl dhparam -out /etc/nginx/ssl/dhparam.pem 4096

cat > /etc/nginx/nginx.conf << 'EOF'
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
  worker_connections 768;
  # multi_accept on;
}

http {

  ##
  # Basic Settings
  ##

  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;
  # server_tokens off;

  # server_names_hash_bucket_size 64;
  # server_name_in_redirect off;

  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  ##
  # SSL Settings
  ##
  ssl_protocols TLSv1.1 TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
  ssl_ecdh_curve secp384r1;
  ssl_dhparam /etc/nginx/ssl/dhparam.pem;


  ##
  # Logging Settings
  ##

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  ##
  # Gzip Settings
  ##

  gzip on;

  # gzip_vary on;
  # gzip_proxied any;
  # gzip_comp_level 6;
  # gzip_buffers 16 8k;
  # gzip_http_version 1.1;
  # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

  ##
  # Virtual Host Configs
  ##
  include /etc/nginx/conf.d/*.conf;
}
EOF

cat > /etc/nginx/conf.d/graylog.conf << 'EOF'
server {
    listen 80 default_server;
    server_name _;
    return 301 https://$host$request_uri;
}

server
{
    listen      443 ssl spdy;
    server_name graylog.hackinglab.local;

    # <- your SSL Settings here!
    ssl_certificate /etc/nginx/ssl/graylog.crt;
    ssl_certificate_key /etc/nginx/ssl/graylog.key;

    location /
    {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Graylog-Server-URL https://$server_name/;
      proxy_pass       http://127.0.0.1:9000;
    }
}
EOF

sudo systemctl daemon-reload
sudo systemctl enable nginx.service
sudo systemctl start nginx.service

Install/Setup UFW

sudo ufw enable
sudo ufw default deny incoming
sudo ufw allow ssh
1. Allow SSH
sudo ufw allow 80/tcp
1. Allow HTTP for NGINX
sudo ufw allow 443/tcp
1. Allow HTTPS for NGINX
sudo ufw allow 1514/tcp
1. Syslog via TCP
sudo ufw allow 1515/udp
1. Syslog via UDP
sudo ufw allow 5044/tcp
1. Filebeat
sudo ufw reload
sudo ufw status

Prod setup – Ansible install/setup

git clone https://github.com/CptOfEvilMinions/BlogProjects.git
cd BlogProjects/Graylogv3
vim hosts.ini and set:
1. ansible_host – Set to the IP addr for Graylog
mv group_vars/all.yml.example group_vars/all.yml
1. hostname – Set hostname for the new graylog box
2. base_domain – Set the domain for graylog
3. timezone – Set country/state for NTP(time)
4. cert stuff for OpenSSL cert – cert_*
ansible-playbook -i host.ini deploy_graylog.yml -u <user> -K

Dev setup – Docker install/Setup

docker-compose up -d

Create Graylog inputs via webGUI

Create inputs

Browse to https://<Graylog FQDN/IP addr> and login
1. Username: admin
2. Password: <root_password_sha2 input, mine was Password123!>
Select “System” then “Inputs”

Create Beats input

Select “Beats” from the drop down menu
Select “Launch new input”
1. Select “<Graylog node>” from drop down menu for “Node”
2. Enter “Beats input” for Title
3. Enter “5044” for Port
4. UNCHECK “Do not add Beats type as prefix(optional)” at the bottom
Select “save”

Create Syslog TCP input

Select “Syslog TCP” from the drop down menu
Select “Launch new input”
1. Select “<Graylog node>” from drop down menu for “Node”
2. Enter “Syslog TCP” for Title
3. Enter “1514” for Port
Select “save”

Create Syslog UDP input

Select “Syslog UDP” from the drop down menu
Select “Launch new input”
1. Select “<Graylog node>” from drop down menu for “Node”
2. Enter “Syslog UDP” for Title
3. Enter “1515” for Port
Select “save”

Install/Setup logging shippers on Ubuntu 18.04

Install/Setup Filebeat+ Zeek JSON logs

vim hosts.ini and set:
1. ansible_host – Set to the IP addr for Filebeat-agents
ansible-playbook -i host.ini deploy_filebeat_bro.yml -u <user> -K

Install/Setup Rsyslog + NGINX

vim hosts.ini and set:
1. ansible_host – Set to the IP addr for Rsyslog-agents
ansible-playbook -i host.ini deploy_rsyslog_nginx.yml -u <user> -K

Setup/Configure Graylog

Create alternative admin user

Select “System” and then “Authentication”
Select “Add a new user” in the top right
1. Enter a username
2. Enter a name
3. Enter the user’s e-mail
4. Enter a password
5. Select a role for the user
  1. The Reader role grants basic access to the system and will be enabled.
  2. The Admin role grants access to everything in Graylog.
6. Leave timeout as default
7. Select a timezone for the user
8. Select “Create user”

Create extractor for NGINX access.log

The point of this section is to show the difference between logs shipped in SYSLOG format vs. logs shipped in JSON format. In JSON format, everything is a key:value pair so it’s easy for Graylog to ingest. SYSLOG has a message field that contains the actual content but is not parsed by Graylog. This section provides a quick tutorial on how to extract fields from the message field in SYSLOG.

Select “Search” at the top and search for “NGINX”
You should see a result like the screenshot below. If so all the information we need is in the message and not in key:value format.
Select the drop-down arrow from within the message field.
Go to “Create extractor field for message” > “Split & Index”
1. Extract hostname
  1. Enter ” “(a space) into “Split by”
  2. Enter “1” for “Target index”
  3. Select “try”
  4. If successful at extracting the hostname from the log message above, continue
  5. Enter “hostname” for “Store as field”
  6. Enter “NGINX access logs extractor” for “Extractor title”
  7. Select “Lowercase” from the drop-down menu for “Add converter”
  8. Select “Add”
  9. Return to the top of the page
2. Extract URL
  1. Enter ” “(a space) into “Split by”
  2. Enter “13” for “Target index”
  3. Select “try”
  4. If successful at extracting the URL from the log message above, continue
  5. Enter “URL” for “Store as field”
  6. Select “Uppercase” from the drop-down menu for “Add converter”
“Create extractor”

Create a stream for Zeek

Zeek log stream

Select “Streams” at the top
Select “Create stream”
1. Enter “Zeek stream” for the name
2. Enter “Zeek network logs” for the description
3. Select “Default index set” for Index set
4. Select “Save”
Select “Manage rules” for the “Zeek” stream
1. Select “Add stream rule”
  1. Enter “source” for Fie;d
  2. Select “contain” for Type
  3. Enter “/usr/local/bro/logs/current/” for “Value”
  4. Select “Save”
2. Select “I’m done” in the bottom left
Select “Start stream” for the “Zeek” stream

Create a dashboard for Zeek

Select “Dashboards” at the top
Select “Create dashboard”
1. Enter “Zeek” for name
2. Enter “Zeek network logs” for the description
3. Select Save
Select “Streams” then “Zeek”
Select “query” under “fields” on the left side
Expand “query” and select “Quick values”
Scroll to the top of the page
Select “Customize” then “Configuration”
1. Enter “10” for “Number of top/bottom values”
2. Select “Update”
Select “Add to dashboard” then “Zeek”
1. Enter “Top 10 most queries domains” for Title
2. Select “Create”
DEselect ” query” under “fields” on the left side
Select “ja3” under “fields” on the left side
1. Expand “query” and select “Quick values”
Scroll to the top of the page
Select “Customize” then “Configuration”
1. Enter “20” for “Number of top/bottom values”
2. Select “Update”
Select “Add to dashboard” then “Zeek”
1. Enter “Top 20 JA3 hashes” for Title
2. Select “Create
Select “Dashboards” at the top
Select “Zeek”

Create a threat intelligence pipeline for TOR exit nodes

Change the message processor configuration

Select “System” then “Configuration”
Select “Update” under the “Message Processors Configuration” section
1. Move “Message Filter Chain” above “Pipeline processor”
  1. From
  2. To
2. Select Save

Enable plugins

Select “System” at the top then “Configuration”
Scroll down to the “plugin” section
Select “Configure” under “Threat Intelligence Lookup Configuration”
1. Select “Allow Tor exit node lookups?” and “Allow Abuse.ch Ransomware tracker lookups?”
2. Select “Save”

Install a content pack

Select “System” at the top then “Content Packs”
Select “Install” for “Tor Exit Node List – Threat Intel Plugin”
1. Select “Install”

Create a pipeline

Select “System” at the top then “Pipelines”
Select “Add new pipeline”
1. Enter “threat intel pipeline” for title
2. Enter “threat intel pipeline” for the description
3. Select Save
Select “Edit connections”
1. Select “Zeek stream” for streams
2. Select “Save”

Select “Manage rules” at the top

Select “Create rule”

Enter “Threat intel domain lookup” for the description”

rule "TOR"
when
  has_field("id_resp_h")
then
  let intel = tor_lookup(to_string($message.id_resp_h));
  set_field("tor_node", intel.threat_indicated);
end

Select Save

Select “Manage pipelines” at the top
Select “threat intel pipeline”
Select “Edit” under “stage 0”
1. Enter “0” for stage
2. Select “At least one of the rules on this stage matches the message” for “Continue processing on next stage when”
3. Add “tor” rule
4. Select Save

Testing threat intel pipeline

For this example, I am going to assume you enabled the TOR threat intel plugin in the previous section.

Browse to TOR exit node list
Select one IP address from the list
SSH into a box on the network being monitored by Zeek
curl http://<IP address of TOR exit node>
Go back to Graylog and select “Search” at the top
Enter “<IP address of TOR exit node>”
Select the first entry and expand it
1. Look for a field with a key name that contains “tor_node”

Create Slack alerts

Install Slack plugin for alerts

SSH into Graylog
cd /tmp
wget https://github.com/graylog-labs/graylog-plugin-slack/releases/download/3.1.0/graylog-plugin-slack-3.1.0.jar
mv graylog-plugin-slack-3.1.0.jar /usr/share/graylog-server/plugin/
systemctl restart graylog-server

Creating Slack webhook

Log into a Slack administrator account
Select “Administration” then “Manage Apps”
Select “Custom integrations” on the left
Select “Add configuration” under webhook
1. Select a channel for the webhook
2. Select “Add Incoming Webhook integration”
Copy the Webhook URL in red

Create Slack notifier

Select “Alerts” at the top
Select “Notifications”
Select “Add new notification”
1. Select “Zeek stream” for stream
2. Select “Slack Alarm callback” for the notification type
3. Select “Add alert notification”
  1. Enter “TOR node” for “title”
  2. Enter “<Slack webhook>” for “Webhook URL”
  3. Enter “#<channel selected for Webhook>”
  4. Select “Save”

Create an alert condition

Select “Add new conditions”
1. Select “Zeek stream” for “Alert on stream”
2. Select “Field Content Alert Condition”
3. Select “Add Alert Condition”
  1. Enter “TOR node detected” for title
  2. Enter “tor_node” for field
  3. Enter “true” for value
  4. Select “Save”

Testing alert