How to setup Falco in Kubernetes Cluster to Detect Deep Kernel activities

Here in this article we will see how we can setup the kubernetes node for analysing and alerting on the runtime activities related to Kernel API calls. We will be installing Falco on the Kubernetes node and launch a set up pods carrying out unusually kernel activity and validate that these activities are getting logged in the system logs for further review and alerting purpose.

Table of Contents

Test Environment

Fedora 36 server
Kubernetes Cluster (1 master + 1 worker node)

What is Falco

Falco is an open source runtime security tool originally built by Sysdiag. It is basically used to detect and alert on system call related to Kernel API. It has a set of default rules that check the kernel for unusual behaviour such privileged container escalation, writing to sensitive files on the file system, unusual network connections and more.

Procedure

Step1: Install Falco on Worker Node

Here in this step we are going to setup the Falco runtime security tool RPM package repository to install it on Fedora OS node.

[admin@kubenode ~]$ sudo rpm --import https://falco.org/repo/falcosecurity-3672BA8F.asc
[admin@kubenode ~]$ sudo curl -s -o /etc/yum.repos.d/falcosecurity.repo https://falco.org/repo/falcosecurity-rpm.repo
[admin@kubenode ~]$ cat /etc/yum.repos.d/falcosecurity.repo
# falcosecurity-rpm - packages by falcosecurity
[falcosecurity-rpm]
name=falcosecurity-rpm
baseurl=https://download.falco.org/packages/rpm
gpgcheck=1
gpgkey=https://falco.org/repo/falcosecurity-3672BA8F.asc
repo_gpgcheck=0
enabled=1

Once the RPM package repository setup is completed, we can install Falco tool as shown below.

[admin@kubenode ~]$ sudo dnf install falco

Step2: Start Falco service

Let’s now start up the Falco systemd service as shown below.

[admin@kubenode ~]$ sudo systemctl start falco.service
 
[admin@kubenode ~]$ sudo systemctl status falco.service 
[sudo] password for admin: 
● falco.service - Falco: Container Native Runtime Security
     Loaded: loaded (/usr/lib/systemd/system/falco.service; enabled; vendor preset: disabled)
     Active: active (running) since Sat 2022-12-03 15:11:02 IST; 54s ago
       Docs: https://falco.org/docs/
   Main PID: 856 (falco)
      Tasks: 13 (limit: 4685)
     Memory: 32.4M
        CPU: 9.022s
     CGroup: /system.slice/falco.service
             └─ 856 /usr/bin/falco --pidfile=/var/run/falco.pid

Step3: Verify Falco Configuration

Falco default configuration file is installed at the following location. This file basically contains details about the type of output setup and plugins that are active. It also contains the locations of all the falco rules file that are used by the falco rule engine for analysing kernel activity violations.

[admin@kubenode falco]$ ls -ltr /etc/falco/falco.yaml 
-rw-r--r--. 1 root root 15594 Nov 16 11:54 /etc/falco/falco.yaml
 
[admin@kubenode falco]$ grep -A 2 "syslog_output" /etc/falco/falco.yaml 
syslog_output:
  enabled: true
 
[admin@kubenode falco]$ grep -A 3 "rules_file" /etc/falco/falco.yaml 
# The name "rules_file" is only for backwards compatibility.
# If the entry is a file, it will be read directly. If the entry is a directory,
# every file in that directory will be read, in alphabetical order.
#
--
rules_file:
  - /etc/falco/falco_rules.yaml
  - /etc/falco/falco_rules.local.yaml
  - /etc/falco/rules.d

Step4: Verify Default Falco Rules

As shown in previous step, Falco rules are defined the same location as the Falco configuration files. The rules files that are by default included are as shown below and any custom rules file under rules.d folder.

[admin@kubenode falco]$ ls -ltr falco_rules.*
-rw-r--r--. 1 root root 140728 Oct 19 19:01 falco_rules.yaml
-rw-r--r--. 1 root root   3100 Nov 24 11:58 falco_rules.local.yaml
 
[admin@kubenode falco]$ ls -ltr rules.d/

Let’s try to look at one of the default rule as shown below.

[admin@kubenode falco]$ grep -A 5 "Write below etc" falco_rules.yaml 
- rule: Write below etc
  desc: an attempt to write to any file below /etc
  condition: write_etc_common
  output: "File below /etc opened for writing (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline pid=%proc.pid parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)"
  priority: ERROR
  tags: [filesystem, mitre_persistence]

Step5: Update the Default Falco Rule

This above rule basically provides a error message in the sysout logs when ever there is a write activity happening under the /etc configuration directory. Let’s try to update the default rules output logging message as shown below. We are going to copy the following rule section into “falco_rules.local.yaml” as shown below and update the output message.

[admin@kubenode falco]$ cat falco_rules.local.yaml 
...
- rule: Write below etc
  desc: an attempt to write to any file below /etc
  condition: write_etc_common
  output: "SUSPECIOUS FILE UNDER /etc WRITTEN!! (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline pid=%proc.pid parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)"
  priority: ERROR
  tags: [filesystem, mitre_persistence]
...

Once the rules file is updated, we need to make sure that the falco systemd service is restarted. Let’s restart the service now.

[admin@kubenode falco]$ sudo systemctl restart falco.service

Step6: Create a Pod that writes to /etc directory

[admin@kubemaster ~]$ alias k='kubectl'
[admin@kubemaster ~]$ export do='-o yaml --dry-run=client'
[admin@kubemaster ~]$ kubectl create ns falcotest

[admin@kubemaster ~]$ cat touchetc.yml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: touchetc
  name: touchetc
  namespace: falcotest
spec:
  containers:
  - command:
    - sh
    - -c
    - touch /etc/trojan.exe; sleep 1d
    image: busybox
    name: touchetc
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

[admin@kubemaster ~]$ k create -f touchetc.yml 
pod/nginx created

Step7: Validate the Sysout logs

Now that we have launched a container which write to /etc configuration directory, ideally this activity should be be logged in our sysout logs with the updated output message as shown below. Here we are going to check this logging using two different methods. They are using the falco status and journalctl.

[root@kubenode falco]# systemctl status falco.service 
● falco.service - Falco: Container Native Runtime Security
     Loaded: loaded (/usr/lib/systemd/system/falco.service; enabled; vendor preset: disabled)
     Active: active (running) since Wed 2022-12-07 12:09:34 IST; 2min 17s ago
       Docs: https://falco.org/docs/
   Main PID: 7857 (falco)
      Tasks: 12 (limit: 4685)
     Memory: 29.3M
        CPU: 3.615s
     CGroup: /system.slice/falco.service
             └─ 7857 /usr/bin/falco --pidfile=/var/run/falco.pid
 
Dec 07 12:09:35 kubenode.stack.com falco[7857]: Falco initialized with configuration file: /etc/falco/falco.yaml
Dec 07 12:09:35 kubenode.stack.com falco[7857]: Loading rules from file /etc/falco/falco_rules.yaml
Dec 07 12:09:35 kubenode.stack.com falco[7857]: Loading rules from file /etc/falco/falco_rules.local.yaml
Dec 07 12:09:35 kubenode.stack.com falco[7857]: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Dec 07 12:09:35 kubenode.stack.com falco[7857]: Starting health webserver with threadiness 8, listening on port 8765
Dec 07 12:09:35 kubenode.stack.com falco[7857]: Enabled event sources: syscall
Dec 07 12:09:35 kubenode.stack.com falco[7857]: Opening capture with Kernel module
Dec 07 12:11:11 kubenode.stack.com falco[7857]: 12:11:11.110045673: Error File below / or /root opened for writing (user=root user_loginuid=1000 command=less pid=7960 parent=systemctl file=>
Dec 07 12:11:16 kubenode.stack.com falco[7857]: 12:11:16.582514276: Error File below / or /root opened for writing (user=root user_loginuid=1000 command=less pid=7962 parent=systemctl file=>
Dec 07 12:11:48 kubenode.stack.com falco[7857]: 12:11:48.117445145: Error SUSPECIOUS FILE UNDER /etc WRITTEN!! (user=<NA> user_loginuid=-1 command=touch /etc/trojan.exe pid=8081 parent=sh p>

[root@kubenode falco]# journalctl -f -n -u falco | grep trojan
Dec 07 12:11:48 kubenode.stack.com falco[7857]: 12:11:48.117445145: Error SUSPECIOUS FILE UNDER /etc WRITTEN!! (user=&lt;NA> user_loginuid=-1 command=touch /etc/trojan.exe pid=8081 parent=sh pcmdline=sh -c touch /etc/trojan.exe; sleep 1d file=/etc/trojan.exe program=touch gparent=&lt;NA> ggparent=&lt;NA> gggparent=&lt;NA> container_id=3cfe86f47e41 image=&lt;NA>)

You can also further update the logging output with specific fields to be written. For a complete list of fields that are supported by Falco refer Falco Supported Fields.

Hope you enjoyed reading this article. Thank you..

How to setup Falco in Kubernetes Cluster to Detect Deep Kernel activities