Jul 24 2024
Leave a comment
By spartan2194 Incident Response, Logging, Network Security, Tools, WebSec

Making Damn Vulnerable Web Application (DVWA) almost unhackable with Cilium and Tetragon

In this universe of ever-changing landscapes and unlimited hackers, defenders are searching for a hero to help defend the Kubernetes universe. To paraphrase Nick Fury, we need to assemble a group of remarkable tools who can work together to fight battles the vulnerable apps can’t. In this blog post, we are going to assemble our own Avengers team using Cilium and Tetragon to defend the Damn Vulnerable Web Application (DVWA) against the unearthly invaders, rendering it almost unhackable. Tetragon + Cilium will provide process, file, HTTP, and network-based defenses to thwart the known evil OWASP’s top 10. DVWA is a web app that was intentionally designed to be vulnerable to OWASP’s top 10 as a training resource. Lastly, I will end with a director’s commentary on my opinions to integrate Cilium + Tetragon in an enterprise and some ideas to close the gap between developers and security.

DISCLAIMERS

The information contained in this blog post is for educational purposes ONLY! HoldMyBeerSecurity.com/HoldMyBeer.xyz and its authors DO NOT hold any responsibility for any misuse or damage of the information provided in blog posts, discussions, activities, or exercises. 
The mitigations implemented in this blog post should not be used as substitutes for actually fixing the root cause of the security issue. They are merely demonstrations of Cilium and Tetragon capabilities.
The red team exercise performed in this blog post occurred in a home lab network. Therefore, the malicious activity performed was on machines that I own and have permission to perform these actions.

DISCLAIMERS

Goals

  • Install/Setup k3s
  • Install Cilium on k3s
  • Install Tetragon on k3s
  • Deploy DVWA on k3s
  • Apply Cilium and Tetragon policies to mitigate DVWA vulnerabilities

Background

What is eBPF?

eBPF is a revolutionary technology with origins in the Linux kernel that can run sandboxed programs in a privileged context such as the operating system kernel. It is used to safely and efficiently extend the capabilities of the kernel without requiring to change kernel source code or load kernel modules.

Historically, the operating system has always been an ideal place to implement observability, security, and networking functionality due to the kernel’s privileged ability to oversee and control the entire system. At the same time, an operating system kernel is hard to evolve due to its central role and high requirement towards stability and security. The rate of innovation at the operating system level has thus traditionally been lower compared to functionality implemented outside of the operating system. Below is a list of resources I recommend to learn more about eBPF.

What is DVWA?

Damn Vulnerable Web Application (DVWA) is a PHP/MySQL web application that is damn vulnerable. Its main goal is to be an aid for security professionals to test their skills and tools in a legal environment, help web developers better understand the processes of securing web applications and to aid both students & teachers to learn about web application security in a controlled classroom environment.

The aim of DVWA is to practice some of the most common web vulnerabilities, with various levels of difficulty, with a simple straightforward interface. Please note, there are both documented and undocumented vulnerabilities with this software. This is intentional. You are encouraged to try and discover as many issues as possible.

What is Kubernetes?

Kubernetes (k8s) is an open-source container-orchestration system for automating application deployment, scaling, and management. It was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation. It aims to provide a “platform for automating deployment, scaling, and operations of application containers across clusters of hosts”. It works with a range of container tools, including Docker. Many cloud services offer a Kubernetes-based platform or infrastructure as a service (PaaS or IaaS) on which Kubernetes can be deployed as a platform-providing service. Many vendors also provide their own branded Kubernetes distributions.

What is Cilium?

Cilium is an open source, cloud native solution for providing, securing, and observing network connectivity between workloads, fueled by the revolutionary Kernel technology eBPF

What is Teragon?

Tetragon is a flexible Kubernetes-aware security observability and runtime enforcement tool that applies policy and filtering directly with eBPF, allowing for reduced observation overhead, tracking of any process, and real-time enforcement of policies.

Install/Setup k3s, Cilium, Tetragon, Kubectl

Install/Setup k3s

  1. ssh to Ubuntu VM
  2. sudo su
  3. Curl automated script to install k3s: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--flannel-backend=none --disable-network-policy' sh -
    1. By default K3s install includes flannel as the CNI and enables certain network policies, per the documentation we need to disable those
  4. Copy /etc/rancher/k3s/k3s.yaml to local machine to access Kubernetes (k8s) via kubectl
  5. Allow access to k8s API via UFW:ufw allow 6443/tcp
  6. Allow k8s network communication: ufw allow from 10.42.0.0/16
  7. Allow k8s network communication: ufw allow from 10.43.0.0/16
  8. exit

Setup kubectl

  1. Install Kubectl on local machine
  2. Pull kubectl config from k3s: scp <username>@<k3s IP addr>:/etc/rancher/k3s/k3s.yaml ~/.kube/config
  3. Replace localhost IP addr: cat /etc/rancher/k3s/k3s.yaml | sed 's#127.0.0.1#<k3s IP addr>#g'
  4. Verify config is working: kubectl get pods -n kube-system

Install/Setup MetalLB

  1. git clone https://github.com/CptOfEvilMinions/BlogProjects
  2. cd BlogProjects/k8s-dvwa-cilium
  3. Install MetalLB: kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.10/config/manifests/metallb-native.yaml
  4. vim conf/k3s/metallb-ip-pool.yaml and set IP range

    ---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default
  namespace: metallb-system
spec:
  addresses:
  - <start IP addr>-<end IP addr>
  autoAssign: true
  5. kubectl apply -f conf/k3s/metallb-ip-pool.yaml

Install/Setup Cilium

  1. Install Cilium via Helm: cilium install --version 1.15.5 --set=ipam.operator.clusterPoolIPv4PodCIDRList="10.42.0.0/16"
  2. Install Cilium CLI on your local machine
  3. Enable Cilium Hubble: cilium hubble enable --ui
  4. Wait until everything turns green: cilium status --wait

Install/Setup Tetragon on k3s

  1. Install Tetragon with Helm: helm install tetragon cilium/tetragon -n kube-system
  2. Install Tetragon CLI
  3. Create a tunnel to tetragon service: kubectl port-forward -n kube-system ds/tetragon 54321:54321
  4. Get status: tetra status

Deploy DVWA to k3s with Helm

  1. cd ../k8s-dvwa
  2. Create namespace: kubectl create ns dvwa
  3. Generate passwords for MySQL: kubectl create secret generic -n dvwa mysql --from-literal=mysql-root-password=$(openssl rand -hex 20) --from-literal=mysql-replication-password=$(openssl rand -hex 20) --from-literal=mysql-password=$(openssl rand -hex 20)
  4. Deploy DVWA: helm install dvwa . -n dvwa -f values.yaml
  5. Create k8s load balancer: kubectl apply -f conf/dvwa/load-balancer.yaml
  6. Wait two minutes for MySQL to start and initialize
  7. Get IP addr of k8s load balancer: kubectl get svc -n dvwa dvwa-load-balancer-external-access
  8. Open browser to http://<DVWA IP addr>/login.php
  9. Login
    1. Username: admin
    2. Password: password
  10. Select “Create / Reset Database”
  11. Re-login

Command injection

The optimal solution would be to have this capability rewritten using a PHP library to perform this action. However, a quick Google search didn’t yield any libraries to support this functionality without calling the ping command. Assuming the business wants to keep the capability as is, we can use Tetragon and Cilium to mitigate this vulnerability. Tetragon provides the ability to monitor and block processes, and Cilium provides the ability to monitor and block network connections. We will also assume for this exercise that the web app is used internally by engineers to test if an internal or external machine is alive via an ICMP ping request.

Cilium

---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: dvwa-allow-external-ingress
  namespace: dvwa
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/instance: dvwa
      app.kubernetes.io/name: dvwa
  ingress:
    - fromEntities:
        - world
      ...
      ...
      ...
   egress:
    # Allow ICMP egress
    # https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
    # ICMP type 8 is Echo
    - icmps:
      - fields:
        - type: 8
          family: IPv4
        - type: 8
          family: IPv6

To block this malicious activity from the network perspective we are going to use a CiliumNetworkPolicy to restrict egress from the DVWA web app. Cilium based network policies will default to blocking if a rule is not specified. Our policy has a rule to specifically allow outbound (egress) ICMP requests of type 8 which are PING echo requests to ask if a host is alive. This will prevent the following attacker vector but not limited to: CURLing a random HTTP endpoint for a malicious script to execute or write to disk.

  1. Browse to http://<DVWA IP addr>/vulnerabilities/exec/#
  2. Enter 8.8.8.8;curl http://ifconfig.me into “Enter an IP address”
  3. Select “Submit”
  4. kubectl apply -f conf/tetragon/networkpolicy.yaml
  5. Enter 8.8.8.8;curl http://ifconfig.me into “Enter an IP address”
  6. Select “Submit”

  7. Open a new terminal tab
  8. Open Hubble via command line: cilium hubble ui
  9. Select the dvwa namespace

 Validating Cilium policy

  1. kubectl exec --stdin=true --tty=true -n dvwa svc/dvwa -- bash
  2. ping 8.8.8.8
  3. curl http://ifconfig.me

Tetragon

---
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: "command-line-injection"
  namespace: "dvwa"
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/instance: dvwa
      app.kubernetes.io/name: dvwa
  kprobes:
  - call: "sys_execve"
    syscall: true
    return: true
    args:
    - index: 0
      type: "string" # file path
    returnArg:
      index: 0
      type: "int"
    returnArgAction: "Post"
    selectors:
    - matchPIDs:
      - operator: In
        followForks: true
        isNamespacePID: true
        values:
          - 1 # Apache root 
      matchArgs:      
      - index: 0
        operator: "NotEqual"
        values:
          - "/usr/bin/ping"
          - "/bin/sh"
      matchActions:
      - action: Override
        argError: -1
      - action: Post

To block the actual vulnerability, which is command line injection, we are going to use a Tetragon TracingPolicy with pod label selectors set to the DVWA pod. The tracing policy above will monitor all syscalls using the execve method. The execve method can take in several arguments, but we are going to monitor the file argument, which contains a file path to an executable. Additionally, we specify that we want to override the return argument by using returnArg. If execve is called successfully, it will simply return an int set to zero for success. To prevent the syscall, we are going to override the return value by returning a non-zero value if the syscall matches our criteria. Our criteria is to monitor all execve syscalls performed by the PID of 1, which in our case is Apache and any child processes spawned by it (followForks). If Apache does make an execve syscall but the first argument doesn’t match anything in the following list ["/usr/bin/ping", "/bin/sh"] it will return a non-zero. The non-zero return will indicate to the application making the call that the operation was unsuccessful, thus preventing command line injection.

  1. Browse to http://<DVWA IP addr>/vulnerabilities/exec/#
  2. Enter 8.8.8.8;whoami into “Enter an IP address”
  3. Select “Submit”
  4. kubectl apply -f conf/tetragon/command-line-injection.yaml
  5. Enter 8.8.8.8;whoami into “Enter an IP address”
  6. Select “Submit”

 Validating Tetragon policy

We are going to prove that Tetragon is actually intercepting the syscall to prevent execution and not just returning an error. The test below uses the command cp, which is not on the approved list. After the command has been submitted, we are going to look for the existence of /tmp/passwd.

  1. Browse to http://<DVWA IP addr>/vulnerabilities/exec/#
  2. Enter 8.8.8.8;cp /etc/passwd /tmp/passwd into “Enter an IP address”
  3. kubectl exec --stdin=true --tty=true -n dvwa svc/dvwa -- bash

File Inclusion

Tetragon

---
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: "block-non-var-www-file-access"
  namespace: "dvwa"
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/instance: dvwa
      app.kubernetes.io/name: dvwa
  kprobes:
  - call: "security_file_open"
    syscall: false
    return: true
    args:
    - index: 0
      type: "file" # (struct file *) used for getting the path
    returnArg:
      index: 0
      type: "int"
    returnArgAction: "Post"
    selectors:
    - matchPIDs:
      - operator: In
        followForks: true
        isNamespacePID: true
        values:
          - 1 # Apache root 
    # - matchBinaries:
    #   - operator: "In"
    #     values:
    #     - "/usr/sbin/apache2"
      matchArgs:
      - index: 0
        operator: "NotPrefix"
        values:
        - "/var/www/html/"
        - "/tmp/sess_"
      matchActions:
      - action: Override
        argError: -2
      - action: Post

The DVWA web app file inclusion vulnerability allows the web app to read any file on disk, including files such as /etc/passwd . The web app only needs to read files located in /var/www/html/ and sessions stored at /tmp/sess_* for normal operations. Therefore, we will use a Tetragon TracingPolicyNamespaced to monitor security_file_permission calls. Per the Tetragon documentation, “hook into the security_file_permission function which is called on every file access.” The policy we will apply will allow PID 1 (Apache) to access files located in /var/www/html/ or /tmp/sess_*. If a file open request is made by Apache for a file outside of these locations, Tetragon will override the return arg for security_file_permission by returning a -2, which means there was error opening the file.

  1. Make the following GET request: http://<DVWA IP addr>/vulnerabilities/fi/?page=../../../../../../etc/passwd
  2. kubectl apply -f conf/tetragon/file-inclusion-blocking.yaml
  3. Make the following GET request: http://<DVWA IP addr>/vulnerabilities/fi/?page=../../../../../../etc/passwd 

Cilium

As we can see in the screenshots above the Tetragon policy works as expected at blocking Apache from reading /etc/passwd. While the above will prevent the web vulnerability in the web app, we can also go one step further with a control-in-depth approach. First, revert the Tetragon policy and apply a CiliumNetworkPolicy to enforce HTTP-based rules on requests made via the Cilium load balancer. The HTTP rule below only allows GET requests matching the URL path regex pattern (Regex101 screenshot). As a side note, I have generated Regex101 links for each regex created for the HTTP rules. Regex101 is a great tool to build regexes and will help you take these example rules to implement in your own environment.

########## File inclusion ##########
# Good: /vulnerabilities/fi/?page=file1.php
# Bad: /vulnerabilities/fi/?page=../../../../../../etc/passwd
# Bad: /vulnerabilities/fi/?page=/etc/passwd
# Regex101: https://regex101.com/r/aM1H9R/1
- method: "GET"
  path: /vulnerabilities\/fi\/\?page=\w+\.php$
  1. Delete the Tetragon policy: kubectl delete -f conf/tetragon/file-inclusion-blocking.yaml
  2. Apply network policy: kubectl apply -f conf/cilium/dvwa-networkpolicy.yaml
  3. Make the following GET request: http://<DVWA IP addr>/vulnerabilities/fi/?page=../../../../../../etc/passwd

File upload

The DVWA web app file upload feature allows users to upload files to this web app. This capability is not inherently a vulnerability, but it can allow other vulnerabilities to be exploited. For example, an attacker could upload a PHP web shell or use the upload feature to host malicious code. Hosting malicious code allows for further attacks such as bypassing CSRF and CSP controls. We can use Tetragon to prevent file upload based on file attributes and Cilium can enforce specific file types. Below, I am going to demonstrate the Tetragon and Cilium policies but also demonstrate how all the policies applied to this point neuter a common web shell.

Tetragon

---
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: "block-uploading-files-with-php-extension"
  namespace: "dvwa"
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/instance: dvwa
      app.kubernetes.io/name: dvwa
  kprobes:
  - call: "do_renameat2"
    syscall: false
    return: true
    args:
    - index: 0
      type: int
    - index: 1
      type: "filename" # old path
    - index: 2
      type: int
    - index: 3
      type: "filename" # new path
    returnArg:
      index: 0
      type: "int"
    returnArgAction: "Post"
    selectors:
    - matchPIDs:
      - operator: In
        followForks: true
        isNamespacePID: true
        values:
        - 1
    - matchArgs:
      - index: 1
        operator: "Prefix"
        values:
        - "../../hackable/uploads"
      - index: 1
        operator: "Postfix"
        values:
          - ".php"
    - matchActions:
      - action: Override
        argError: -1
      - action: Post

It should be noted that this tracing policy can easily be bypassed with ease. The intention is to demonstrate the capabilities of Tetragon in this blog post. Additionally, this technique could be used in an incident scenario as a temporary patch until better controls can be implemented. When an upload is performed via this web app, the contents of the upload are written to /tmp and then moved to the appropriate location. It should be noted that really a move is actually just renaming the file. The rename syscall takes in two parameters which are the old/current file path and the new filename/new file path. Additionally, the reason we are specifying a relative path for the new is because the context of the web app is being executed from the following directory location: /var/www/html/vulnerabilities/upload. Thus, the app provides a relative path to move the file to and the path is not resolved.

The policy above will monitor all rename syscalls performed by Apache (PID 1). If the new file location is the relative path for file uploads and the new file name ends with .php, we should return an error code. This will terminate the rename/move call to stop the temporary file from being placed in the proper location for file uploads. While this isn’t perfect because it still wastes disk space, this was a fun journey of tracking down how the web app works.

  1. Browse to http://<DVWA IP addr>/vulnerabilities/uploads
  2. Select a PHP file
  3. Select “Upload”
  4. kubectl apply -f conf/tetragon/file-uploads.yaml

Cilium

At the Cilium level we can enforce that the file uploads set the Content-Type header to an approved file type. Since this field is controlled by the user,  it can be modified to an approved value.

########## File upload ##########
# Allow GET requests to the webpage
- method: GET
  path: "/vulnerabilities/upload$"
# Allow JPEG uploads
- method: POST
  path: "/vulnerabilities/upload$"
  headers:
    - 'Content-Type: image/jpeg'
# Allow PNG uploads
- method: POST
  path: "/vulnerabilities/upload$"
  headers:
    - 'Content-Type: image/png'
  1. Browse to http://<DVWA IP addr>/vulnerabilities/uploads
  2. Select a PHP file
  3. Select “Upload”

PHP web shell

In this section, we are going to demonstrate using web shells as an attack vector. One type of web shells is some malicious PHP code that will create a reverse shell back to an attacker-owned machine. We can mitigate this attack path using the CiliumNetworkPolicy from the “Command injection” section above to restrict egress from the DVWA web app. Another form of a web shell is one that allows an attacker to run malicious actions using a web shell like C99Shell-PHP8. Again, if we apply the Tetragon TracingPolicy from the “Command injection” section we can prevent malicious commands from being executed. If we apply the Tetragon TracingPolicy to monitor security_file_permission from the “File inclusion” section we can prevent the web shell from reading files outside the defined scope. Unfortunately, one downfall is an attacker could still access uploads from other users because it’s within the allowed scope. The goal here was to demonstrate that a small amount of well defined policies can prevent a multitude of vulnerabilities and attack vectors. Unfortunately, we were not able to prevent the use of the C99 webshell but it is essentially neutered.

  1. Upload a web shell to the web app
  2. Browse to http://<DVWA IP addr>/hackable/uploads/c99.php
  3. If we enter whoami into Command execute section
  4. Select “Execute”
  5. Select the home icon in top left
  6. kubectl apply -f conf/tetragon/command-line-injection.yaml
  7. RE-run the same command
  8. Select the home icon in top left
  9. Use the folder browser to access /etc/passwd with the text editor
  10. Select the home icon in top left
  11. kubectl apply -f conf/tetragon/file-inclusion-blocking.yaml
  12. Unable to browse to /etc/

SQL injection

This feature of the web app allows a user to look up other users by a user ID (uid). When a user submits the UID, it is added to the URL as a parameter that is then passed to the MySQL engine. Therefore, we can implement a Cilium Network Policy that enforces a regex and anything that doesn’t match it will be denied. This same approach can be applied to the SQL injection (blind) example as well.

########## SQL injection ##########
- method: "GET"
  path: /vulnerabilities/(sqli|sqli_blind)/\?id=\d+&Submit=Submit(#)?$

It should be noted that I spent an extensive amount of time trying to determine if there was a syscall I could hook to prevent SQL injection within the kernel. The only information I could find was using user statically defined tracing (USDT) to track MySQL queries. However, I think this analysis is generated AFTER the query has been executed because it generates stats such as how long each query takes to execute. Therefore, it’s likely this method can’t be used to prevent SQL injection before the query is executed. I also found that uprobe may be able to could hook a function call within an application but there wasn’t enough information to implement this.

  1. Make the following GET request: http://<DVWA IP addr>/vulnerabilities/sqli/?id=1&Submit=Submit#
  2. Enter 1' OR '1'='1'# to text box and hit “Submit”
  3. Copy the URL which encoded the payload
  4. Apply network policy: kubectl apply -f conf/cilium/dvwa-networkpolicy.yaml
  5. Make the following GET request: http://<DVWA IP addr>/vulnerabilities/sqli/?id=1'+OR+'1'%3D'1'%23&Submit=Submit#

XSS

Reflected

This feature of the web app allows the user to enter their name and have it be reflected back to them. The name submitted is appended to the URL as a parameter. For our use case, we are going to assume that a valid name would be someone’s first name that ONLY contains alphabetic characters. Therefore, we will use a Cilium Network policy to define a valid regex for the URL to prevent this vulnerability. The policy allows HTTP GET requests and the URL parameter name must specify a word (alphabetic characters).

########## XSS ##########
# Good: /vulnerabilities/xss_r/?name=bob#
# Bad: /vulnerabilities/xss_r/?name=<script>alert("xss")<%2Fscript>#
# Regex101: https://regex101.com/r/OT3gFy/1
- method: "GET"
  path: /vulnerabilities\/xss_r\/\?name=\w+(#)?$
  1. Browse to http://<DVWA IP addr>/vulnerabilities/xss_r/
  2. Enter <script>alert("xss")</script> into the name input
  3. Select “Submit”
  4. Copy the encoded URL
  5. kubectl apply -f conf/cilium/networkpolicy.yaml
  6. RE-run the above

DOM

This feature of the web app allows the user to select a preferred language. The language selected is submitted by appending the selection to the URL as a parameter. For our use case, we are going to assume that a valid language would be a single word containing only alphabetic characters. Therefore, we will use a Cilium Network policy to define a valid regex for the URL to prevent this vulnerability. The policy allows HTTP GET requests and the URL parameter default must specify a word (alphabet characters).

# Good: /vulnerabilities/xss_d/?default=English
# Bad: /vulnerabilities/xss_d/?default=<script>alert("xss")<%2Fscript>#
# Regex101: https://regex101.com/r/TEVesk/1
- method: "GET"
  path: /vulnerabilities\/xss_d\/\?default=\w+(#)?$
  1. Browse to http://<DVWA IP addr>/vulnerabilities/xss_d/
  2. Select a language from the drop down menu and select “Submit”
  3. Copy the URL
  4. Make the following GET request: http://<DVWA IP addr>/vulnerabilities/xss_d/?default=<script>alert(1)</script>
  5. kubectl apply -f conf/cilium/networkpolicy.yaml
  6. RE-run the above, we can see that we get access denied because the URL doesn’t match regex.

Stored

This feature of the web app attempts to simulate a guestbook. Like a guestbook, the web app allows the user to digitally sign by providing their name and a custom message as input. When this information is submitted, it’s stored by the application to keep a record of guests. The app reflects the user’s payload back to the user upon submission. A more typical example is a user web forum that allows users to have conversations by submitting comments. If the web app is not sanitizing input from users and is not wrapping input from user’s when rendering it can lead to exploitation. Since the malicious payload is submitted via the HTTP body, this is not something Cilium can inspect and thus we can’t prevent it.

Open HTTP Redirect

An open HTTP redirect attack occurs when a web application allows an external user to specify a URL as a parameter in a redirect link. This redirect link can be used to transfer the user to the malicious link. The term “open” in this context means that the redirection is not restricted to a specific domain or a set of trusted destinations. Below we specify the re-direct URL to https://github.com which the browser happily redirects too. To prevent a malicious redirect we are going to use a CiliumNetworkPolicy to restrict valid URLs. DVWA provides redirects to itself but we are going to expand this capability to allow redirects to auth portals like Okta.

# Allow DNS
- toEndpoints:
  - matchLabels:
      "k8s:io.kubernetes.pod.namespace": kube-system
      "k8s:k8s-app": kube-dns
  toPorts:
    - ports:
        - port: "53"
          protocol: UDP
      rules:
        dns:
            - matchPattern: '*.dvwa.svc.cluster.local'
  1. Make the following GET request: http://<DVWA IP addr>/vulnerabilities/open_redirect/source/low.php?redirect=https://github.com?id=1
  2. kubectl apply -f conf/cilium/networkpolicy.yaml
  3. Make the following GET request: http://<DVWA IP addr>/vulnerabilities/open_redirect/source/low.php?redirect=https://github.com?id=1
  4. Make the following GET request: http://<DVWA IP addr>/vulnerabilities/open_redirect/source/low.php?redirect=https://dev-123456.okta.com?id=1

Brute forcing

The DVWA web app is susceptible to a brute force attack to enumerate valid username and password combinations. To combat brute force attacks, one mitigation is to implement rate limiting on API endpoints such as the /login endpoint. For example, if a user has ten consecutive failed logins (401 Unauthorized) within a ten minute period then, that IP address should be temporarily blocked for an hour. While this won’t eliminate the brute force vulnerability entirely, it significantly increases the amount of time needed to perform this attack. Under the hood, Cilium uses Envoy to implement the load balancing and HTTP rule capabilities. Cilium also supports a custom Envoy config that uses Envoy extensions, such as the built-in rate limiting extension. Configuring Envoy to perform rate limiting is outside the scope of this blog post, but it is doable. Below I have listed some resources if you want to implement this.

“The ALMOST”

We finally arrive at the vulnerabilities that we can’t prevent in this web app with Cilium or Tetragon: CSRF, Weak Session IDs, CSP, Javascript, and Authorisation Bypass. These specific vulnerabilities can only be remediated by fixing the logic of the web app.

Closing the gap between security and developers

This blog post demonstrates the power of Cilium and Tetragon to protect an enterprise if an application contains a vulnerability. However, there is still a gap between security and developers. To implement these controls, developers would need to learn Cilium and Tetragon to create policies. Additionally, every time the application is updated, changes will need to be made to ensure that the changes are not blocked.

In my opinion, Cilium and Tetragon policy generation could be streamlined by creating a tool like aa-easyprof. AppArmor comes with a command line tool called aa-easyprof to generate profiles based on actions by an application. Cilium and Tetragon could use the observability data recorded by Hubble to generate profiles for developers. In theory, a developer could create or update an application and then push the changes to QA. Allow the application some time to bake. Once the bake period is complete, a profile is generated based on the events in Hubble for a human to review. Lastly, a tool that could ingest an API spec such as OpenAPIv3 and generate a Cilium network policy with strict HTTP rules. As seen above, having strict HTTP rules can prevent a vulnerability from being exploited.

Lessons learned

Web application security 101

I am a huge proponent if you don’t use your knowledge to keep it fresh, ya lose it. My current role nor my personal projects have been focused on web app security, so it was really nice to do a project to refresh all this knowledge for this problem space.

TracingPolicy vs. TracingPolicyNamespaced

The default documentation for policies is kind: TracingPolicy which applies the policy to the ENTIRE HOST. For example when testing, I wrote a policy to prevent access to /etc/passwd, not knowing it would block ALL requests including requests from the bare metal host. In one swift policy, I borked my entire test k8s instance because the bare-metal host needed access to this file to operate. So while I did secure my web app…. I also made the host unusable. My takeaway from this is that while Tetragon was created to target Kubernetes and containers, eBPF works at the kernel level. Thus a poorly written blocking policy will affect everything running on that kernel. Unless you need to prevent an action fleet-wide, I would limit the blast radius to specific pod namespaces.

Furthermore, even when using a TracingPolicyNamespaced policy havoc can still ensue. This policy will affect ALL pods in the namespace. For example, when creating the command line injection policy, I put a block in place to prevent access to all binaries except ping by Apache. Consequently, the MySQL pod was not able to start itself. In addition to using the right kind, I also recommend supplying podSelectors with the appropriate labels to target specific pods.

Sigkill vs. blocking

Tetragon provides two methods to prevent a malicious action from being executed which are blocking and sigkill. With blocking we force Tetragon to override the return with an error typically by returning a non-zero int. Sigkill will terminate the process that is attempting to perform the malicious action. It should be noted that sigkill is the nuclear option. Imagine you have 10 users on your web app and one of those user’s is a bug bounty hunter; they could essentially kill the web app with their testing. Additionally, a malicious actor could also continuously trigger a policy with a malicious request that would kill the application, thus making it unreachable. Lastly, I recommend using the blocking action as the primary method to block a malicious action and only use sigkill when appropriate.

References

Leave a Reply

Your email address will not be published. Required fields are marked *