Behavioral analytics is the process of observing what is going on in a system and identifying abnormal and potentially malicious events.
If you are adopting containers and cloud, you take advantage of a consistent, portable software environment that allows applications to run and scale easily anywhere.When it comes to security, this is a whole new world with new rules. Containers are also black boxes, often invisible to traditional host-based tools.
There may be actions that are not risky most of the time in the cluster. However, this does not mean that a malicious attack will not occur.If we run our production applications in cluster environments such as Kubernetes, we need to be aware of any abnormal activity that may contain threats. In addition, it is important to record threatening activities for later analysis of the attack and taking measures.
One of the widely used tools for behavioral analytics is Falco, developed as open source by Sysdig.
Falco is exceptional at detecting unexpected application behavior and alerting on threats at runtime, by analyzing kernel system calls to provide deep visibility into container, host, and cluster activity.It also taps into other data sources, such as Kubernetes API audit events.
So, what are unexpected behaviors? Examples of unexpected behaviors are as follows:
Unexpected spawned process
The processes that our application running inside a container will run are mostly known. For example, if we containerize our Java application, we expect a Java process to run normally. However, opening bash inside the container and sending a request to a server with the curl command, unlike known processes, is abnormal behavior.
Unexpected file access
There may be some files that our application needs to access to operate. We know which files need to be accessed, such as application properties files, reading static files. However, reading and writing requests to directories such as the root directory or /usr/bin that the application would not normally access is unexpected behavior.
Unexpected cryptojacking process
When creating container images, a base image prepared by someone else is mostly used. There may be libraries and executable files that we do not know in the base image used. Different versions of crypto mining attacks have been encountered by many institutions or individuals before. To ensure the security of our application and prevent resources from being consumed by malicious people, we need to be aware of such malicious processes and take action when they are running.
Falco works by looking at file changes, network activity, the process table, and other data for suspicious behavior and then logs the event. Falco needs to be installed on all worker nodes of the cluster.
You can install Falco on an Ubuntu server as follows.
curl -s https://falco.org/repo/falcosecurity-packages.asc | apt-key add -
echo "deb https://download.falco.org/packages/deb stable main" | tee -a /etc/apt/sources.list.d/falcosecurity.list
apt-get update -y
apt install -y dkms make linux-headers-$(uname -r)
apt install -y clang llvm
apt install -y dialog
apt-get install -y falcoWhen you install the Dialog package, the interface will appear during installation.
In this step, we need to select the Falco driver. This data source is collected using either a kernel module or an eBPF probe, which we call drivers. We choose Kmod from among four options.
If we want automatic ruleset updates in the next step, we should say yes. It is recommended by Falco to update rulesets automatically by default.
After the installation is complete, you can use Falco. Falco needs a ruleset to analyze and scan workloads. You can use the default rulesets prepared by Falco or create your own ruleset according to your needs.
For example, if we want to be notified when a shell is executed inside the container, we can write a ruleset like the following.
vi demo_rule.yaml
- rule: shell_in_container
desc: notice shell activity within a container
condition: >
evt.type = execve and
evt.dir = < and
container.id != host and
(proc.name = bash or
proc.name = ksh)
output: >
shell in a container
(user=%user.name container_id=%container.id container_name=%container.name
shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
priority: WARNINGYou can review the falco_rules.yaml file for a more comprehensive ruleset.
After defining the sample rule, you can get a report on which containers run shell processes by running the following command.
When running the falco command, the -r parameter is used to specify the file where the rule is written. The -M parameter specifies the time for which the analysis will be run. In our scenario, we want to run it for 60 seconds.
sudo falco -r demo_rule.yaml -M 45 > falco-report.log
If you run an nginx pod in the cluster for testing purposes and open a bash shell, it will log the event log to the log file.
Let’s bring up a simple nginx pod in the cluster and log in to the pod with bash when the pod is up and running.
kubectl run nginx — image=nginx
kubectl exec -it nginx bash
When you log in to the container with bash, it will write a log entry to the log file as follows.
Warning shell in a container (user=root container_id=8e93d7f8f7aa container_name=nginx shell=bash parent=runc cmdline=bash)
Events detected: 1
Rule counts by severity:
WARNING: 1
Triggered rules by rule name:
shell_in_container: 1Falco operates by analyzing kernel system calls to offer deep insights into container, host, and cluster activities, including unexpected behaviors like spawned processes, file access anomalies, and cryptojacking attempts. Its ability to tap into various data sources, such as Kubernetes API audit events, makes it a comprehensive solution for threat detection in Kubernetes environments.