Install/Setup Doorman + OSQuery on Windows, Mac OSX, and Linux deployment

In this post I am going to explore the tool OSquery. OSquery allows you to easily ask questions about your Linux, Windows, and macOS infrastructure. Whether your goal is intrusion detection, infrastructure reliability, or compliance, OSquery gives you the ability to empower and inform a broad set of organizations within your company. It is a tool that is used by system administrators, incident responders, and ole mighty threat hunters. However, in this post I will not be posting how to use OSquery for threat hunting. I hope to utilize the tool in my environment and write a later post :).

Doorman/OSquery components and terms

Terms

  • Node– A single machine
  • Fleet – All the machines controlled and owned by an enterprise
  • Queries – A query runs a set of tasks on fleet of machines on a specified interval
  • Distributed – An on the fly query
  • Packs – OSquery query packs are groups of queries to be added to the OSquery schedule

Install/Setup Doorman Remote API server

Install/Setup Doorman on CentOS 7 64-bit with Docker

Install/Setup NTPd on Centos

  1. yum install ntp ntpdate ntp-doc -y
  2. systemctl enable ntpd
  3. systemctl start ntpd
  4. ntpdate pool.ntp.org || true

Install/Setup Postgres and Redis

Install/Setup Postgres database

  1. yum update -y && yum upgrade -y
  2. rpm -Uvh https://yum.postgresql.org/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm
  3. yum install postgresql96-server postgresql96 -y
    1. MUST install Postgresql 9.4 or later
  4. /usr/pgsql-9.6/bin/postgresql96-setup initdb
  5. sed -i 's#host all all 127.0.0.1/32 ident#host all all 127.0.0.1/32 md5#'g /var/lib/pgsql/9.6/data/pg_hba.conf
  6. systemctl enable postgresql-9.6.service
  7. systemctl start postgresql-9.6.service
  8. su – postgres
  9.  psql
    1. CREATE ROLE doorman WITH LOGIN PASSWORD ‘<password>’;
      1. password can NOT contain “@” or “#”
    2. CREATE DATABASE doorman;
    3. ALTER DATABASE doorman OWNER TO doorman;
    4. GRANT ALL PRIVILEGES ON DATABASE doorman TO doorman;
    5. \q
  10. exit
  11. psql -U doorman -h 127.0.0.1 -d doorman -W
    1. Test to make sure you can connect as doorman user on postgres
  12. useradd doorman

Install/Setup Redis

  1. yum install redis -y
  2. systemctl enable redis
  3. systemctl start redis

Install/Setup Doorman

  1. yum install python-pip python-devel libffi-devel gcc postgresql-devel npm -y
    1. For Centos 7.3: rpm -ivh https://kojipkgs.fedoraproject.org//packages/http-parser/2.7.1/3.el7/x86_64/http-parser-2.7.1-3.el7.x86_64.rpm && yum -y install nodejs
  2. pip install –upgrade pip
  3. cd /opt
  4. git clone https://github.com/mwielgoszewski/doorman.git
  5. cd doorman
  6. pip install virtualenv
  7. virtualenv env
  8. source env/bin/activate
  9. pip install -r requirements.txt
  10. chown doorman:doorman -R /opt/doorman
  11. vim doorman/settings.py
    1. scroll to “class ProdConfig(Config):”
      SQLALCHEMY_DATABASE_URI = 'postgresql://doorman:<doorman password>@127.0.0.1:5432/doorman'
      DOORMAN_ENROLL_SECRET = ['<randomly generated secret key>']
      BROKER_URL = 'redis://localhost:6379/0'
      CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
  12. mkdir /var/log/doorman
  13. chown doorman:doorman -R /var/log/doorman
  14. export DOORMAN_ENV=prod
    1. Set this variable in /etc/profile to be permanent
  15. su – doorman -c “cd /opt/doorman; source env/bin/activate; python manage.py db upgrade”
  16. npm install bower -g
  17. bower install
  18. npm install -g less

Install/Setup Nginx + WSGI/Flask + OpenSSL

Install/Setup Nginx and OpennSSL

  1. yum install nginx -y
  2. mkdir /etc/nginx/ssl
  3. openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/private.key -out /etc/nginx/ssl/certificate.crt
  4. sed -i -e ‘38,87d’ /etc/nginx/nginx.conf
  5. cat > /etc/nginx/conf.d/osquery.conf <<\EOF
    server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    return 301 https://$host$request_uri;
    }server {
    listen 443 ssl;
    server_name _;ssl_certificate /etc/nginx/ssl/certificate.crt;
    ssl_certificate_key /etc/nginx/ssl/private.key;ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!MD5;location / {
    include uwsgi_params;
    uwsgi_pass unix:/opt/doorman/doorman.sock;
    }
    }
    EOF

Install/Setup WSGI and Flask

  1. cd /opt/doorman
  2. pip install uwsgi flask
  3. cat > doorman.ini << EOF
    [uwsgi]
    master = true
    processes = 5
    home = env
    wsgi-file = manage.py
    callable = app
    socket = doorman.sock
    chmod-socket = 660
    vacuum = true
    die-on-term = true
    smart-attach-daemon = /opt/doorman/celery.pid celery worker -A doorman.worker:celery --pidfile=/opt/doorman/celery.pid
    env = DOORMAN_ENV=prod
    EOF
  4. cat > /etc/systemd/system/doorman.service << EOF
    [Unit]
    Description=uWSGI instance to serve Doorman
    After=network.target[Service]
    User=doorman
    Group=nginx
    WorkingDirectory=/opt/doorman
    Environment="PATH=/opt/doorman/env/bin:/usr/bin"
    ExecStart=/opt/doorman/env/bin/uwsgi --ini doorman.ini[Install]
    WantedBy=multi-user.target
    EOF
  5. systemctl enable doorman
  6. systemctl start doorman
  7. systemctl enable nginx
  8. systemctl start nginx
  9. setsebool httpd_can_network_connect 1 -P

Install/Setup FirewallD

  1. yum install firewalld -y
  2. systemctl start firewalld
  3. systemctl enable firewalld
  4. firewall-cmd –zone=public –permanent –add-service=http
  5. firewall-cmd –zone=public –permanent –add-service=https
  6. firewall-cmd –zone=public –permanent –add-service=ssh
  7. firewall-cmd –reload

Accessing Doorman

  1. Browse to “https://<IP addr of doorman>/manage

Install/Setup OSQuery on Linux, Mac, and Windows

Install/Setup OSQuery on CentOS 7 Server 64-bit

  1. yum update -y && yum upgrade -y
  2. yum install yum-utils -y
  3. curl https://s3.amazonaws.com/osquery-packages/rpm/RPM-GPG-KEY-osquery | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-osquery
  4. yum install install yum-utils -y
  5. yum-config-manager –add-repo https://s3.amazonaws.com/osquery-packages/rpm/osquery-s3-rpm.repo
  6. yum-config-manager –enable osquery-s3-rpm
  7. yum install osquery -y
  8. openssl s_client -showcerts -connect <doorman IP addr>:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >/etc/osquery/certificate.crt
  9. cat > /etc/osquery/osquery.flags << 'EOF'
    --host_identifier=uuid
    --config_plugin=tls
    --config_tls_endpoint=/config
    --config_tls_refresh=10
    --config_tls_max_attempts=3
    --enroll_tls_endpoint=/enroll
    --enroll_secret_path=/etc/osquery/osquery.key
    --disable_distributed=false
    --distributed_plugin=tls
    --distributed_interval=10
    --distributed_tls_max_attempts=3
    --distributed_tls_read_endpoint=/distributed/read
    --distributed_tls_write_endpoint=/distributed/write
    --logger_plugin=tls
    --logger_tls_endpoint=/log
    --logger_tls_period=5
    --tls_hostname=<doorman IP addr>:443
    --tls_server_certs=/etc/osquery/certificate.crt
    --log_result_events=false
    --pack_delimiter=/
    --utc
    --verbose
    EOF
  10. cat > /etc/osquery/osquery.key << 'EOF'
    <randomly generated secret key for Doorman>
    EOF
  11. systemctl enable osqueryd
  12. systemctl start osqueryd
  13. Browse to “https://<doorman IP addr>:443/manage/nodes” to confirm node was added

Install/Setup OSQuery on Ubuntu 16.04 64-bit

  1. sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
  2. sudo add-apt-repository “deb [arch=amd64] https://osquery-packages.s3.amazonaws.com/deb deb main”
  3. sudo apt-get update -y
  4. sudo apt-get install osquery -y
  5. openssl s_client -showcerts -connect <doorman IP addr>:443 </dev/null 2>/dev/null|openssl x509 -outform PEM | sudo tee /etc/osquery/certificate.crt
  6. cat << EOF | sudo tee/etc/osquery/osquery.flags
    --host_identifier=uuid
    --config_plugin=tls
    --config_tls_endpoint=/config
    --config_tls_refresh=10
    --config_tls_max_attempts=3
    --enroll_tls_endpoint=/enroll
    --enroll_secret_path=/etc/osquery/osquery.key
    --disable_distributed=false
    --distributed_plugin=tls
    --distributed_interval=10
    --distributed_tls_max_attempts=3
    --distributed_tls_read_endpoint=/distributed/read
    --distributed_tls_write_endpoint=/distributed/write
    --logger_plugin=tls
    --logger_tls_endpoint=/log
    --logger_tls_period=5
    --tls_hostname=<doorman IP addr>:443
    --tls_server_certs=/etc/osquery/certificate.crt
    --log_result_events=false
    --pack_delimiter=/
    --utc
    --verbose
    EOF
  7. cat << EOF | sudo tee /etc/osquery/osquery.key
    <randomly generated secret key for Doorman>
    EOF
  8. sudo systemctl enable osqueryd
  9. sudo systemctl start osqueryd

Install/Setup OSQuery on Mac OSX

  1. brew update
  2. brew install osquery
  3. openssl s_client -showcerts -connect <doorman IP addr>:443 </dev/null 2>/dev/null|openssl x509 -outform PEM | sudo tee /var/osquery/certificate.crt
  4. rm -rf /var/osquery/osquery.example.conf
  5. cat << EOF | sudo tee /var/osquery/osquery.flags
    --host_identifier=uuid
    --config_plugin=tls
    --config_tls_endpoint=/config
    --config_tls_refresh=10
    --config_tls_max_attempts=3
    --enroll_tls_endpoint=/enroll
    --enroll_secret_path=/var/osquery/osquery.key
    --disable_distributed=false
    --distributed_plugin=tls
    --distributed_interval=10
    --distributed_tls_max_attempts=3
    --distributed_tls_read_endpoint=/distributed/read
    --distributed_tls_write_endpoint=/distributed/write
    --logger_plugin=tls
    --logger_tls_endpoint=/log
    --logger_tls_period=5
    --tls_hostname=<doorman IP addr>:443
    --tls_server_certs=/var/osquery/certificate.crt
    --log_result_events=false
    --pack_delimiter=/
    --utc
    --verbose
    EOF
  6. cat << EOF | sudo tee /var/osquery/osquery.key
    <randomly generated secret key for Doorman>
    EOF
  7. sudo cp /var/osquery/com.facebook.osqueryd.plist /Library/LaunchDaemons/
  8. sudo launchctl load /Library/LaunchDaemons/com.facebook.osqueryd.plist
  9. sudo launchctl start /Library/LaunchDaemons/com.facebook.osqueryd.plist

Install/Setup OSQuery on Windows

Install/Setup Choclately the package manager for Windows

  1. Open Powershell as an Administrator
  2. Set-ExecutionPolicy RemoteSigned
  3. Copy certificate.pem from Doorman server to Windows
  4. Copy and Paste: iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
  5. Close Powershell and re-open as Administrator
  6. choco install osquery --params='/InstallService'
    1. This will install OSquery as a Windows service
  7. $config = "
    --host_identifier=uuid`r`n
    --config_plugin=tls`r`n
    --config_tls_endpoint=/config`r`n
    --config_tls_refresh=10`r`n
    --config_tls_max_attempts=3`r`n
    --enroll_tls_endpoint=/enroll`r`n
    --enroll_secret_path=C:\ProgramData\osquery\osquery.key`r`n
    --disable_distributed=false`r`n
    --distributed_plugin=tls`r`n
    --distributed_interval=10`r`n
    --distributed_tls_max_attempts=3`r`n
    --distributed_tls_read_endpoint=/distributed/read`r`n
    --distributed_tls_write_endpoint=/distributed/write`r`n
    --logger_plugin=tls`r`n
    --logger_tls_endpoint=/log`r`n
    --logger_tls_period=5`r`n
    --tls_hostname=<doorman IP addr>:443`r`n
    --tls_server_certs=C:\ProgramData\osquery\certificate.crt`r`n
    --log_result_events=false`r`n
    --pack_delimiter=/`r`n
    --utc`r`n
    --verbose`r`n"
  8. $config | Out-File -FilePath C:\Program Data\osquery\osquery.flags
  9. Start-service osqueryd
  10. Get-Service | Where-Object {$_.name -eq “osqueryd”}

Setup/Configure/Secure Doorman

Setup LDAP/local user authentication

Local authentication

  1. cd /opt/doorman
  2. vim doorman/settings.py
    1. Add DOORMAN_AUTH_METHOD = 'doorman' to “Class ProcConfig():” section
    2. save, exit
  3. systemctl restart doorman
  4. python manage adduser –email [email protected] test
    1. Enter password for user
  5. Browse to “http://<DOORMAN IP addr>/manage”
  6. Enter login credentials from above and select “Login”

LDAP authentication

  1. cd /opt/doorman
  2. vim doorman/settings.py
    1. Add DOORMAN_AUTH_METHOD = 'ldap' to “Class ProcConfig():” section
    2. Then scroll up to the LDAP section and set your settings. The settings below are a basic setup for Freeipa

      LDAP_HOST = '<hostname of FreeIPA>'
      LDAP_PORT = 636
      LDAP_USE_SSL = True
      LDAP_BASE_DN = 'cn=users,cn=accounts,dc=<domain, example>,dc=<tld, com>'
    3. save, exit
  3. systemctl restart doorman
  4. Browse to “http://<DOORMAN IP addr>/manage”
  5. Enter LDAP credentials and select “Login”

Add OSQuery packs

Setup new pack

  1. Browse to “https://github.com/facebook/osquery/tree/master/packs”
  2. For our example we will install the “hardware-monitoring.conf” pack
  3. Download the hardware-monitoring.conf
  4. Login into Doorman and select “Packs” at the top
  5. Select “Choose file” and select the pack on disk
  6. Select “Update Query Pack”

 

Doorman scanning

Distributed scans

  1. Login into Doorman and select “Add” then “Distributed”
  2. Enter “SELECT uid, name FROM listening_ports l, processes p WHERE l.pid=p.pid; into Query
    1. For more information about queries look here
  3. Select specific nodes in the node section
  4. Select specific tags to scan a set of nodes with a particular tag
    1. If you select nothing from above it will scan everything
  5. Select “Add distributed query”

Interval scans

  1. Select “Add” then “Query”
  2. Enter “Get all listening ports” for name
  3. Enter “select * from listening_ports”
  4. Enter “3600” for interval
    1. The interval is in seconds
  5. Select “All” for platforms
  6. Select a hardware pack to run but for this scan we will not
  7. Select specific tags to scan a set of nodes with a particular tag
  8. Select “Add query”

 

Resources/Sources

6 thoughts on “Install/Setup Doorman + OSQuery on Windows, Mac OSX, and Linux deployment

  1. Ralph Lawrence says:

    I am greatly appreciated of your work. I would like to ask a question, can doorman be used only only on linux. I would love to use it on a windows a machine.

  2. Amit Srivastav says:

    this is really a great documentation. Thanks

  3. Vinay Sannareddy says:

    I’m facing issue when trying to to start doorman service.
    doorman.service lacks both Execstart= and Execstop= setting.
    Someone please suggest what needs to be done or any changes to dorman.service file.

  4. Mabel says:

    An impressive share! I have just forwarded this onto a colleague who was conducting a little homework on this.
    And he actually ordered me lunch due to the fact that I stumbled upon it
    for him… lol. So allow me to reword this…. Thanks for the meal!!
    But yeah, thanx for spending some time to discuss this
    subject here on your web site.

Leave a Reply

Your email address will not be published. Required fields are marked *