Apr 03 2019
1 Comment
By admin Logging, Network Security, Tools

Part 1: Install/Setup Zeek + pf_ring on Ubuntu 18.04 on Proxmox 5.3 + openVswitch

 

Monitoring your home network can be challenging without enterprise-grade equipment. Although monitoring your home network can prove to be difficult, Proxmox and Zeek provide the perfect solution to monitor your home network. This blog post will cover how to setup Zeek+PF_Ring to monitor network traffic on Proxmox.

Goals

  • Setup a NON-standalone Zeek
    • manager
    • proxy
    • worker – multiple processes
  • PF_RING to capture network traffic
  • Proxmox + OpenVswitch
    • OpenVswitch to allow traffic monitoring

Background

What is Proxmox?

Proxmox VE is a completely open-source platform for enterprise virtualization. With the built-in web interface, you can easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.

My Proxmox box

My Proxmox box(hypervisor) is a custom build with server grade hardware:

  • CPU: Intel Xeon quad-core
  • Memory: 64Gbs of DDR4 ECC
  • SSD: Kingston 120GB – Proxmox install
  • HDD: 2x 4TB Western Digital enterprise drives
    •  ZFS “RAID 1” for 4TB of storage
  • NIC: Intel PRO/1000 VT
    • Has 4 ports

What is Zeek?

Zeek is a passive, open-source network traffic analyzer. It is primarily a security monitor that inspects all traffic on a link in depth for signs of suspicious activity. More generally, however, Bro supports a wide range of traffic analysis tasks even outside of the security domain, including performance measurements and helping with troubleshooting.

What is pf-ring?

PF_RING is a high-speed packet capture library that turns a commodity PC into an efficient and cheap network measurement box suitable for both packet and active traffic analysis and manipulation. Moreover, PF_RING opens totally new markets as it enables the creation of efficient application such as traffic balancers or packet filters in a matter of lines of codes

Network diagram

The diagram below displays Zeek running on a separate box but, in reality, it is actually running on Proxmox. This was done intentionally to show all the physical cabling for my home network.

Proxmox

Install/Setup openVswitch for mirror ports on Proxmox 5.3

ASSUMPTIONS

This blog post ASSUMES your Proxmox box has TWO PHYSICAL ethernet ports. One interface is dedicated for Proxmox maintenance and VM traffic. The second interface is dedicated to being a mirror port for Zeek.

Install/Setup OpenVswitch

  1. SSH into Proxmox
  2. apt-get update -y && apt-get install openVswitch-switch -y

Add mirror port

  1. Browse to https://<IP addr of proxmox>: is 8006 and login
  2. Expand “datacenter” in the left and select the Proxmox node you want to run Zeek on
  3. Expand “System” then “network”
  4. Select “Create” at the top then select “OVS bridge”
    1. Leave the name as the default
      1. Note this name for the next section
    2. Check “Autostart”
    3. Enter “<interface>” into “Bridge ports”
    4. Enter “mirror port” into “Comments”
  5. Select “Create”

Create Ubuntu 18.04 VM for Zeek

For my homelab, I have VM templates created that I clone from. At this point, I assume you have a VM created or have created a clone.

  1. Select the VM
  2. Select “Hardware”
    1. Set CPU to have 4 cores
    2. Set memory to 4GB
  3. Select “Add” then “Network device”
    1. Select “vmbrX” – mirror port
    2. Leave the VLAN blank
    3. Select “Intel E1000” for Model
    4. Select “Add”
  4. Start VM

Add mirror port on boot

  1. SSH into Proxmox
  2. cd /srv
  3. ip a | grep tap | grep ovs-system
    1. Note the tap<VM ID>i<X>
  4. cat > /srv/mirror_port.sh << 'EOF'
#!/bin/dash

MIRRORPORTLOG=/srv/mirror_port.log

date >> $MIRRORPORTLOG

echo "####################" >> $MIRRORPORTLOG

echo "Clearing any existing mirror..." >> $MIRRORPORTLOG

ovs-vsctl clear bridge <OVS bridge interface name - vmbrX> mirrors

echo "Creating mirror on <OVS bridge interface name - vmbrX> for Zeek" >> $MIRRORPORTLOG

ovs-vsctl -- --id=@p get port <tap interface> \
-- --id=@m create mirror name=span1 select-all=true output-port=@p \
-- set bridge <OVS bridge interface name - vmbrX> mirrors=@m >> $MIRRORPORTLOG

echo "Showing existing mirrors" >> $MIRRORPORTLOG

ovs-vsctl list Mirror >> $MIRRORPORTLOG

echo "####################" >> $MIRRORPORTLOG
EOF
  5. chmod 700 /srv/mirror_port.sh
  6. ./mirror_port
  7. echo "@reboot /srv/mirror_port.sh" >> /var/spool/cron/crontabs/root

Install/Setup Zeek + pf-ring from source

My network

As you can see in the diagram above, my home network has multiple VLANs(mini-networks) all connected to the switch. For Zeek to monitor all the traffic in my network, I created a mirror port on the switch. Therefore, the switch is making a copy of each packet that flows through it to the mirror port. This allows Zeek to monitor all the traffic on my home network.

Manual install

Update machine

  1. sudo su
  2. apt-get update -y
  3. apt-get upgrade -y
  4. apt-get dist-upgrade -y
  5. reboot
  6. sudo apt-mark hold linux-image-generic linux-headers-generic
    1. DISABLING kernel updates
    2. Because we compiled PFRing in this kernel, any kernel builds may cause the PFRing module to fail to load.  You will need to recompile PFRing if you update your kernel after compiling.

Enable monitoring interface on boot

  1. ip link set <interface> up
  2. echo "@reboot ip link set <interface> up" >> /var/spool/cron/crontabs/root

Install/Setup pf-ring from source

  1. apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libgeoip-dev build-essential libelf-dev -y
  2. cd /opt
  3. git clone https://github.com/ntop/PF_RING.git
  4. cd PF_RING/kernel
  5. make
  6. insmod ./pf_ring.ko
  7. cd ../userland
  8. make
  9. cd lib
  10. ./configure –prefix=/opt/PF_RING
  11. make install
  12. cd ../libpcap
  13. ./configure --prefix=/opt/PF_RING/
  14. make install
  15. cd ../tcpdump-*
  16. ./configure --prefix=/opt/PF_RING/
  17. make install
  18. cd ../../kernel
  19. make
  20. make install
  21. echo "pf_ring" >> /etc/modules
    1. Load pf_ring at boot
  22. reboot
  23. lsmod | grep pf_ring

Install/Setup Zeek from source

  1. sudo su
  2. apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev git -y
  3. cd /tmp
  4. git clone --recursive https://github.com/zeek/zeek
  5. cd zeek
  6. ./configure --with-pcap=/opt/PF_RING --prefix=/opt/bro/
  7. make
  8. make install
  9. echo "$PATH:/opt/bro/bin" >/etc/environment
  10. export PATH=/opt/bro/bin:$PATH

Setup/Configure Zeek

  1. cat > /opt/bro/etc/node.cfg << 'EOF'
# Example BroControl node configuration.
#
# This example has a standalone node ready to go except for possibly changing
# the sniffing interface.


## Below is an example clustered configuration. 
[manager]
type=manager
host=localhost
#
[proxy-1]
type=proxy
host=localhost
#
[worker-1]
type=worker
host=localhost
interface=ens18
lb_method=pf_ring 
lb_procs=5
EOF
  2. vim /opt/bro/etc/network.cfg
    1. Add networks in CIDR notation that Zeek will be monitoring
    2. # List of local networks in CIDR notation, optionally followed by a
# descriptive tag.
# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes.

10.0.0.0/8
172.16.0.0/16
192.168.1.0/24
    3. Save and exit

Start Zeek

  1.  /opt/bro/bin/broctl
    1. install
    2. deploy
    3. status

Ansible

host.ini

  1. git clone https://github.com/CptOfEvilMinions/BlogProjects.git
  2. cd BlogProjects/zeek_pfring
  3. vim hosts.ini and set:
    1. ansible_host for zeek01
  4. Save and exit

group_vars/all.yml

  1. vim group_vars/all.yml
    1. General
      1. timezone – Set timezone for Zeek machine
    2. Zeek
      1. zeek_hostname – Set hostname of Zeek machine
      2. zeek_interface – Set interface for Zeek to monitor
      3. zeek_mail_to – E-mail to send Zeek alerts
      4. zeek_geoip – Enable/Disable GeoIP tagging
      5. zeek_file_extraction – Enable/Disable file extraction
      6. zeek_stats – Enable/Disable Zeek stats
      7. zeek_custom_scripts -Enable/Disable custom scripts in conf/Zeek/scripts
    3. Slack
      1. slack_token – OPTIONAL – Ansible notications
      2. slack_channel – OPTIONAL – Ansible notications
  2. Save and exit

Zeek configs

  1. mv conf/zeek/networks.cfg.example conf/zeek/networks.cfg
  2. vim conf/zeek/networks.cfg
    1. Add networks in CIDR notation that Zeek will be monitoring 
      # List of local networks in CIDR notation, optionally followed by a# descriptive tag.
# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes.
10.0.0.0/8
172.16.0.0/16
192.168.1.0/24
    2. Save and exit

Deploy

  1. ansible-playbook -i hosts.ini deploy_zeek_pf_ring.yml -u <user> -K

Verify Zeek is running

Running processes

  1. /opt/bro/bin/broctl status

Logs

  1. ls /opt/bro/logs/current
  2. tail -f  /opt/bro/logs/curren/conn.log

Debugging

  1. /opt/bro/bin/broctl diag
  2. /opt/bro/bin/broctl check

Credit

Black Hills Infosec

Shoutout to Black Hills Infosec on the tutorial to set up Bro(now Zeek) on ESXi. The idea for this blog post came from this setup but I modified it for Proxmox.

vext.info

Shoutout to vext.info on the tutorial on how to setup a mirror port on Proxmox with OpenVswitch.

Resources/Sources

One thought on “Part 1: Install/Setup Zeek + pf_ring on Ubuntu 18.04 on Proxmox 5.3 + openVswitch”

  1. Corbin says:
    July 19, 2019 at 10:29 am

    Really good tutorial, thank you for this!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *