Install/Setup Graylog 3 on Ubuntu 18.04 – Zeeks logs + threat intel pipeline

 

Graylog has released version 3 with new features and major changes. This blog post will explain how to setup up Graylog version 3 on an Ubuntu server. Once Graylog is running, we will explore setting up logging clients, logging inputs, data extractors, threat intel pipelines, Slack alerts, dashboards and more.

Intro

This blog post is how to setup up Graylog version 3 on an Ubuntu server 18.04 with your choice of a manual install, Ansible, or Docker. Once Graylog is running, I have instructions on shipping NGINX logs with Rsyslog and Zeek/BRO logs in JSON format with Filebeat. Once the logs are ingested, we will create logging inputs, data extractors,  pipelines for threat intelligence, Slack alerts, and a dashboard to view Zeel logs.

Goals

  • Install and setup Graylog v3
    • Manual install
    • Ansible
    • Docker
  • Learn how to ship logs to Graylog with Rsyslog and Filebeat
    • Shipping Zeek/Bro logs in JSON format with Filebeat
    • Shipping NGINX logs with Rsyslog
  • Setting up a central logging server
  • Graylog features
    • pipelines
    • content packs
    • logging input
    • data extractors
    • alerts
    • dashboards

Install/Setup Graylog 3 on Ubuntu 18.04

Manual install/setup

Install things

  1. sudo apt-get update && sudo apt-get upgrade
  2. sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen -y

Install/Setup Mongo

  1. sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
  2. echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
  3. sudo apt-get update -y
  4. sudo apt-get install -y mongodb-org
  5. sudo systemctl daemon-reload
  6. sudo systemctl enable mongod.service
  7. sudo systemctl restart mongod.service

Install/Setup Elasticsearch

  1. wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
  2. echo "deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
  3. sudo apt-get update && sudo apt-get install elasticsearch-oss
  4. sed -i 's/#cluster.name: my-application/cluster.name: graylog/g' /etc/elasticsearch/elasticsearch.yml
  5. echo "action.auto_create_index: false" >> /etc/elasticsearch/elasticsearch.yml
  6. sudo systemctl daemon-reload
  7. sudo systemctl enable elasticsearch.service
  8. sudo systemctl restart elasticsearch.service

Install/Setup Graylog

  1. wget https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.deb
  2. sudo dpkg -i graylog-3.0-repository_latest.deb
  3. sudo apt-get update && sudo apt-get install graylog-server
  4. echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1
    1. Copy the output
  5. sed -i 's/root_password_sha2 =/root_password_sha2 = <Output from above>/g /etc/graylog/server/server.conf
  6. pwgen -N 1 -s 96
  7. sed -i 's/password_secret =/password_secret = <Output from above>/g' /etc/graylog/server/server.conf
  8. sudo systemctl daemon-reload
  9. sudo systemctl enable graylog-server.service
  10. sudo systemctl start graylog-server.service

Install/Setup NGINX + HTTPS

  1. sudo apt-get install nginx -y
  2. mkdir /etc/nginx/ssl
  3. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/graylog.key -out /etc/nginx/ssl/graylog.crt
    1. Enter country
    2. Enter state
    3. Enter city
    4. Enter organization name
    5. Enter organization unit name
    6. Enter common name
  4. openssl dhparam -out /etc/nginx/ssl/dhparam.pem 4096
  5. cat > /etc/nginx/nginx.conf << 'EOF'
    user www-data;
    worker_processes auto;
    pid /run/nginx.pid;
    include /etc/nginx/modules-enabled/*.conf;
    
    events {
      worker_connections 768;
      # multi_accept on;
    }
    
    http {
    
      ##
      # Basic Settings
      ##
    
      sendfile on;
      tcp_nopush on;
      tcp_nodelay on;
      keepalive_timeout 65;
      types_hash_max_size 2048;
      # server_tokens off;
    
      # server_names_hash_bucket_size 64;
      # server_name_in_redirect off;
    
      include /etc/nginx/mime.types;
      default_type application/octet-stream;
    
      ##
      # SSL Settings
      ##
      ssl_protocols TLSv1.1 TLSv1.2;
      ssl_prefer_server_ciphers on;
      ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
      ssl_ecdh_curve secp384r1;
      ssl_dhparam /etc/nginx/ssl/dhparam.pem;
    
    
      ##
      # Logging Settings
      ##
    
      access_log /var/log/nginx/access.log;
      error_log /var/log/nginx/error.log;
    
      ##
      # Gzip Settings
      ##
    
      gzip on;
    
      # gzip_vary on;
      # gzip_proxied any;
      # gzip_comp_level 6;
      # gzip_buffers 16 8k;
      # gzip_http_version 1.1;
      # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
    
      ##
      # Virtual Host Configs
      ##
      include /etc/nginx/conf.d/*.conf;
    }
    EOF
  6. cat > /etc/nginx/conf.d/graylog.conf << 'EOF'
    server {
        listen 80 default_server;
        server_name _;
        return 301 https://$host$request_uri;
    }
    
    server
    {
        listen      443 ssl spdy;
        server_name graylog.hackinglab.local;
    
        # <- your SSL Settings here!
        ssl_certificate /etc/nginx/ssl/graylog.crt;
        ssl_certificate_key /etc/nginx/ssl/graylog.key;
    
        location /
        {
          proxy_set_header Host $http_host;
          proxy_set_header X-Forwarded-Host $host;
          proxy_set_header X-Forwarded-Server $host;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Graylog-Server-URL https://$server_name/;
          proxy_pass       http://127.0.0.1:9000;
        }
    }
    EOF
  7. sudo systemctl daemon-reload
  8. sudo systemctl enable nginx.service
  9. sudo systemctl start nginx.service

Install/Setup UFW

  1. sudo ufw enable
  2. sudo ufw default deny incoming
  3. sudo ufw allow ssh
    1. Allow SSH
  4. sudo ufw allow 80/tcp
    1. Allow HTTP for NGINX
  5. sudo ufw allow 443/tcp
    1. Allow HTTPS for NGINX
  6. sudo ufw allow 1514/tcp
    1. Syslog via TCP
  7. sudo ufw allow 1515/udp
    1. Syslog via UDP
  8. sudo ufw allow 5044/tcp
    1. Filebeat
  9. sudo ufw reload
  10. sudo ufw status

Prod setup – Ansible install/setup

  1. git clone https://github.com/CptOfEvilMinions/BlogProjects.git
  2. cd BlogProjects/Graylogv3
  3. vim hosts.ini and set:
    1. ansible_host – Set to the IP addr for Graylog
  4. mv group_vars/all.yml.example group_vars/all.yml
    1. hostname – Set hostname for the new graylog box
    2. base_domain – Set the domain for graylog
    3. timezone – Set country/state for NTP(time)
    4. cert stuff for OpenSSL cert – cert_*
  5. ansible-playbook -i host.ini deploy_graylog.yml -u <user> -K

Dev setup – Docker install/Setup

  1. docker-compose up -d

Create Graylog inputs via webGUI

Create inputs

  1. Browse to https://<Graylog FQDN/IP addr> and login
    1. Username: admin
    2. Password: <root_password_sha2  input, mine was Password123!>
  2. Select “System” then “Inputs”

Create Beats input

  1. Select “Beats” from the drop down menu
  2. Select “Launch new input”
    1. Select “<Graylog node>” from drop down menu for “Node”
    2. Enter “Beats input” for Title
    3. Enter “5044” for Port
    4. UNCHECK “Do not add Beats type as prefix(optional)” at the bottom
  3. Select “save”

Create Syslog TCP input

  1. Select “Syslog TCP” from the drop down menu
  2. Select “Launch new input”
    1. Select “<Graylog node>” from drop down menu for “Node”
    2. Enter “Syslog TCP” for Title
    3. Enter “1514” for Port
  3. Select “save”

Create Syslog UDP input

  1. Select “Syslog UDP” from the drop down menu
  2. Select “Launch new input”
    1. Select “<Graylog node>” from drop down menu for “Node”
    2. Enter “Syslog UDP” for Title
    3. Enter “1515” for Port
  3. Select “save”

Install/Setup logging shippers on Ubuntu 18.04

Install/Setup Filebeat+ Zeek JSON logs

  1. vim hosts.ini and set:
    1. ansible_host – Set to the IP addr for Filebeat-agents
  2. ansible-playbook -i host.ini deploy_filebeat_bro.yml -u <user> -K

Install/Setup Rsyslog + NGINX

  1. vim hosts.ini and set:
    1. ansible_host – Set to the IP addr for Rsyslog-agents
  2. ansible-playbook -i host.ini deploy_rsyslog_nginx.yml -u <user> -K

Setup/Configure Graylog

Create alternative admin user

  1. Select “System” and then “Authentication”
  2. Select “Add a new user” in the top right
    1. Enter a username
    2. Enter a name
    3. Enter the user’s e-mail
    4. Enter a password
    5. Select a role for the user
      1. The Reader role grants basic access to the system and will be enabled.
      2. The Admin role grants access to everything in Graylog.
    6. Leave timeout as default
    7. Select a timezone for the user
    8. Select “Create user”

Create extractor for NGINX access.log

The point of this section is to show the difference between logs shipped in SYSLOG format vs. logs shipped in JSON format. In JSON format, everything is a key:value pair so it’s easy for Graylog to ingest. SYSLOG has a message field that contains the actual content but is not parsed by Graylog. This section provides a quick tutorial on how to extract fields from the message field in SYSLOG.

  1. Select “Search” at the top and search for “NGINX”
  2. You should see a result like the screenshot below. If so all the information we need is in the message and not in key:value format.
  3. Select the drop-down arrow from within the message field.
  4. Go to “Create extractor field for message” > “Split & Index”
    1. Extract hostname
      1. Enter ” “(a space) into “Split by”
      2. Enter “1” for “Target index”
      3. Select “try”
      4. If successful at extracting the hostname from the log message above, continue
      5. Enter “hostname” for “Store as field”
      6. Enter “NGINX access logs extractor” for “Extractor title”
      7. Select “Lowercase” from the drop-down menu for “Add converter”
      8. Select “Add”
      9. Return to the top of the page
    2. Extract URL
      1. Enter ” “(a space) into “Split by”
      2. Enter “13” for “Target index”
      3. Select “try”
      4. If successful at extracting the URL from the log message above, continue
      5. Enter “URL” for “Store as field”
      6. Select “Uppercase” from the drop-down menu for “Add converter”
  5. “Create extractor”

Create a stream for Zeek

Zeek log stream

  1. Select “Streams” at the top
  2. Select “Create stream”
    1. Enter “Zeek stream” for the name
    2. Enter “Zeek network logs” for the description
    3. Select “Default index set” for Index set
    4. Select “Save”
  3. Select “Manage rules” for the “Zeek” stream
    1. Select “Add  stream rule”
      1. Enter “source” for Fie;d
      2. Select “contain” for Type
      3. Enter “/usr/local/bro/logs/current/” for “Value”
      4. Select “Save”
    2. Select “I’m done” in the bottom left
  4. Select “Start stream” for the “Zeek” stream

Create a dashboard for Zeek

  1. Select “Dashboards” at the top
  2. Select “Create dashboard”
    1. Enter “Zeek” for name
    2. Enter “Zeek network logs” for the description
    3. Select Save
  3. Select “Streams” then “Zeek”
  4. Select “query” under “fields” on the left side
  5. Expand “query” and select “Quick values”
  6. Scroll to the top of the page
  7. Select “Customize” then “Configuration”
    1. Enter “10” for “Number of top/bottom values”
    2. Select “Update”
  8. Select “Add to dashboard” then “Zeek”
    1. Enter “Top 10 most queries domains” for Title
    2. Select “Create”
  9. DEselect ” query” under “fields” on the left side
  10. Select “ja3” under “fields” on the left side
    1. Expand “query” and select “Quick values”
  11. Scroll to the top of the page
  12. Select “Customize” then “Configuration”
    1. Enter “20” for “Number of top/bottom values”
    2. Select “Update”
  13. Select “Add to dashboard” then “Zeek”
    1. Enter “Top 20 JA3 hashes” for Title
    2. Select “Create
  14. Select “Dashboards” at the top
  15. Select “Zeek”

Create a threat intelligence pipeline for TOR exit nodes

Change the message processor configuration

  1. Select “System” then “Configuration”
  2. Select “Update” under the “Message Processors Configuration” section
    1. Move “Message Filter Chain” above “Pipeline processor”
      1. From
      2. To
    2. Select Save

Enable plugins

  1. Select “System” at the top then “Configuration”
  2. Scroll down to the “plugin” section
  3. Select “Configure” under “Threat Intelligence Lookup Configuration”
    1. Select “Allow Tor exit node lookups?” and “Allow Abuse.ch Ransomware tracker lookups?”
    2. Select “Save”

Install a content pack

  1. Select “System” at the top then “Content Packs”
  2. Select “Install” for “Tor Exit Node List – Threat Intel Plugin”
    1. Select “Install”

Create a pipeline

  1. Select “System” at the top then “Pipelines”
  2. Select “Add new pipeline”
    1. Enter “threat intel pipeline” for title
    2. Enter “threat intel pipeline” for the description
    3. Select Save
  3. Select “Edit connections”
    1. Select “Zeek stream” for streams
    2. Select “Save”
  4. Select “Manage rules” at the top
    1. Select “Create rule”
      1. Enter “Threat intel domain lookup” for the description”
      2. rule "TOR"
        when
          has_field("id_resp_h")
        then
          let intel = tor_lookup(to_string($message.id_resp_h));
          set_field("tor_node", intel.threat_indicated);
        end
      3. Select Save
  5. Select “Manage pipelines” at the top
  6. Select “threat intel pipeline”
  7. Select “Edit” under “stage 0”
    1. Enter “0” for stage
    2. Select “At least one of the rules on this stage matches the message” for “Continue processing on next stage when”
    3. Add “tor” rule
    4. Select Save

Testing threat intel pipeline

For this example, I am going to assume you enabled the TOR threat intel plugin in the previous section.

  1. Browse to TOR exit node list
  2. Select one IP address from the list
  3. SSH into a box on the network being monitored by Zeek
  4. curl http://<IP address of TOR exit node>
  5. Go back to Graylog and select “Search” at the top
  6. Enter “<IP address of TOR exit node>”
  7. Select the first entry and expand it
    1. Look for a field with a key name that contains “tor_node”

Create Slack alerts

Install Slack plugin for alerts

  1. SSH into Graylog
  2. cd /tmp
  3. wget https://github.com/graylog-labs/graylog-plugin-slack/releases/download/3.1.0/graylog-plugin-slack-3.1.0.jar
  4. mv graylog-plugin-slack-3.1.0.jar /usr/share/graylog-server/plugin/
  5. systemctl restart graylog-server

Creating Slack webhook

  1. Log into a Slack administrator account
  2. Select “Administration” then “Manage Apps”
  3. Select “Custom integrations” on the left
  4. Select “Add configuration” under webhook
    1. Select a channel for the webhook
    2. Select “Add Incoming Webhook integration”
  5. Copy the Webhook URL in red

Create Slack notifier

  1. Select “Alerts” at the top
  2. Select “Notifications”
  3. Select “Add new notification”
    1. Select “Zeek stream” for stream
    2. Select “Slack Alarm callback” for the notification type
    3. Select “Add alert notification”
      1. Enter “TOR node” for “title”
      2. Enter “<Slack webhook>” for “Webhook URL”
      3. Enter “#<channel selected for Webhook>”
      4. Select “Save”

Create an alert condition

  1. Select “Add new conditions”
    1. Select “Zeek stream” for “Alert on stream”
    2. Select “Field Content Alert Condition”
    3. Select “Add Alert Condition”
      1. Enter “TOR node detected” for title
      2. Enter “tor_node” for field
      3. Enter “true” for value
      4. Select “Save”

Testing alert

For this example, I am going to assume you enabled the TOR threat intel plugin in the previous section.

  1. Browse to TOR exit node list
  2. Select one IP address from the list
  3. SSH into a box on the network being monitored by Zeek
  4. curl http://<IP address of TOR exit node>

Resources/Sources

Leave a Reply

Your email address will not be published. Required fields are marked *