Container information missing in the alerts generated by the "Privileged Shell Spawned in Container" rule. #3345

dmeetreey · 2024-09-26T14:31:49Z

Hello,

My team and I have recently encountered an issue where alerts triggered by the "Privileged Shell Spawned in Container" rule are missing container-related details.

Example of Alert Output:

Critical Privileged Shell Spawned in Container (user.uid=0 proc.cmdline=sh proc.name=sh proc.pname=<NA> gparent=<NA> ggparent=<NA> gggparent=<NA> proc.pcmdline=<NA> user.name=root user.loginuid=-1 proc.args= container.name=<NA> evt.type=execve evt.res=SUCCESS proc.pid=2121961 proc.cwd= proc.ppid=2121895 proc.sid=1 proc.exepath=/bin/sh user.loginname=<NA> group.gid=0 group.name=root, container.image=<NA>)
container_id=827accfc01b7 
container_image=<NA> 
container_image_tag=<NA> 
container_name=<NA> 
k8s_ns=<NA> 
k8s_pod_name=<NA>

Rule Configuration:

- rule: Privileged shell spawned inside container
  enabled: true
  desc: This rule detects the creation of a shell as root for interaction within a container. If this rule fires, it may be an indication of compromise.
  condition: spawned_process and container and shell_procs and user.uid = 0 and (proc.args = "" or proc.args startswith "-i") and not container_entrypoint and not user_shell_container_exclusions and not user_expected_terminal_shell_in_container_conditions and proc.tty = 0
  exceptions:
  [REDACTED]
  output: Privileged Shell Spawned in Container (user.uid=%user.uid proc.cmdline=%proc.cmdline proc.name=%proc.name proc.pname=%proc.pname gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] proc.pcmdline=%proc.pcmdline user.name=%user.name user.loginuid=%user.loginuid proc.args=%proc.args container.name=%container.name evt.type=%evt.type evt.res=%evt.res proc.pid=%proc.pid proc.cwd=%proc.cwd proc.ppid=%proc.ppid proc.sid=%proc.sid proc.exepath=%proc.exepath user.loginname=%user.loginname group.gid=%group.gid group.name=%group.name, container.image=%container.image.repository)
  priority: CRITICAL
  tags: [container, shell, mitre_container_administration_command, mitre_execution, CIS]

Environment:

Falco version: 0.38.2
Driver: Modern eBPF driver

Problem:
The missing container information makes it difficult to troubleshoot and correlate alerts with the actual containers.

Expected Behavior:
We expect the alert output to include detailed container information such as container name, image, image tag, Kubernetes namespace, and pod name.

Additional Context:
It’s critical for us to have complete information in alerts for effective incident response and troubleshooting.

Request:
Could you please provide guidance on how to resolve this issue or if it’s a known issue of Falco? Any suggestions or workarounds would be greatly appreciated.

Thank you for your support!

The text was updated successfully, but these errors were encountered:

dmeetreey · 2024-09-26T14:34:33Z

/kind bug

UgniusV · 2024-10-15T13:21:43Z

Hey @poiana any updates for this?

leogr · 2024-10-16T07:32:32Z

Hey @dmeetreey

Can you share more details about the running environment (OS, Kernel version, etc..) and the setup (installation method, config, etc)?

Thanks

doublez13 · 2024-10-17T18:56:36Z

I'm also seeing fields missing from certain alerts. Some have them populated while others don't.

For instance, popping a shell into a terminal triggers the following alert, but the fields are usually null (but not always).

{
  "hostname": "REDACTED",
  "output": "18:44:49.495227176: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=www-data user_uid=33 user_loginuid=-1 process=bash proc_exepath=/usr/bin/bash parent=systemd command=bash terminal=34816 exe_flags=EXE_LOWER_LAYER container_id=d4758580aec2 container_image=<NA> container_image_tag=<NA> container_name=<NA> k8s_ns=<NA> k8s_pod_name=<NA>)",
  "output_fields": {
    "container.id": "d4758580aec2",
    "container.image.repository": null,
    "container.image.tag": null,
    "container.name": null,
    "evt.arg.flags": "EXE_LOWER_LAYER",
    "evt.time": 1729190689495227176,
    "evt.type": "execve",
    "k8s.ns.name": null,
    "k8s.pod.name": null,
    "proc.cmdline": "bash",
    "proc.exepath": "/usr/bin/bash",
    "proc.name": "bash",
    "proc.pname": "systemd",
    "proc.tty": 34816,
    "user.loginuid": -1,
    "user.name": "www-data",
    "user.uid": 33
  },
  "priority": "Notice",
  "rule": "Terminal shell in container",
  "source": "syscall",
  "tags": [
    "T1059",
    "container",
    "maturity_stable",
    "mitre_execution",
    "shell"
  ],
  "time": "2024-10-17T18:44:49.495227176Z"
}

I've got falco deployed via helm with the following settings...

tty: true
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true

collectors:
  enabled: true
  containerd:
    enabled: true
    socket: /run/containerd/containerd.sock
  docker:
    enabled: false
  crio:
    enabled: false

#As mentioned in the docs, I've disabled async, but am still seeing null values
extra:
  args:
    - '--disable-cri-async'

#I've got a few custom rules not included

Falco version: 0.39.1 (x86_64)
System info: Linux version 6.1.0-25-amd64 ([email protected]) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.106-3 (2024-08-26)

UgniusV · 2024-10-18T09:54:45Z

Can you share more details about the running environment (OS, Kernel version, etc..) and the setup (installation method, config, etc)?

@leogr

sure:

OS Info

root@falcosecurity-falco-sandbox1-lpjn6:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Kernel version:

root@falcosecurity-falco-sandbox1-lpjn6:/# uname -a
Linux falcosecurity-falco-sandbox1-lpjn6 6.1.94-1.el8.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 20 08:31:35 UTC 2024 x86_64 GNU/Linux

config:

base_syscalls:
  custom_set: []
  repair: false
buffered_outputs: true
config_files:
- /etc/falco/config.d
engine:
  kind: modern_ebpf
  modern_ebpf:
    buf_size_preset: 8
    cpus_for_each_buffer: 2
    drop_failed_exit: false
falco_libs:
  thread_table_size: 262144
file_output:
  enabled: false
  filename: ./events.txt
  keep_alive: false
grpc:
  bind_address: unix:///run/falco/falco.sock
  enabled: true
  threadiness: 0
grpc_output:
  enabled: true
http_output:
  ca_bundle: ""
  ca_cert: ""
  ca_path: /etc/falco/certs/
  client_cert: /etc/falco/certs/client/client.crt
  client_key: /etc/falco/certs/client/client.key
  compress_uploads: false
  echo: false
  enabled: true
  insecure: false
  keep_alive: false
  mtls: false
  url: http://falcosecurity-falcosidekick:80/
  user_agent: falcosecurity/falco
json_include_output_property: true
json_include_tags_property: true
json_output: true
libs_logger:
  enabled: false
  severity: debug
load_plugins:
- k8saudit
- json
log_level: notice
log_stderr: true
log_syslog: true
metrics:
  convert_memory_to_mb: true
  enabled: true
  include_empty_values: false
  interval: 1h
  kernel_event_counters_enabled: true
  libbpf_stats_enabled: true
  output_rule: true
  resource_utilization_enabled: true
  rules_counters_enabled: true
  state_counters_enabled: true
output_timeout: 2000
outputs:
  max_burst: 1000
  rate: 0
outputs_queue:
  capacity: 0
plugins:
- init_config:
    maxEventSize: "125829120"
    webhookMaxBatchSize: "125829120"
  library_path: libk8saudit.so
  name: k8saudit
  open_params: http://:9765/k8s-audit
- init_config: ""
  library_path: libjson.so
  name: json
priority: debug
program_output:
  enabled: false
  keep_alive: false
  program: 'jq ''{text: .output}'' | curl -d @- -X POST https://hooks.slack.com/services/XXX'
rule_matching: first
rules_files:
- /etc/falco/common-rules.yaml
stdout_output:
  enabled: true
syscall_event_drops:
  actions:
  - log
  - alert
  max_burst: 1
  rate: 0.03333
  simulate_drops: false
  threshold: 0.1
syscall_event_timeouts:
  max_consecutives: 1000
syslog_output:
  enabled: true
time_format_iso_8601: false
watch_config_files: true
webserver:
  enabled: true
  k8s_healthz_endpoint: /healthz
  listen_port: 8765
  prometheus_metrics_enabled: true
  ssl_certificate: /etc/falco/falco.pem
  ssl_enabled: false

Installation method: modern eBPF

leogr · 2024-10-18T14:55:36Z

These are obvious questions, but they may still help:

@doublez13 have you checked the socket filepath exits?

collectors:
  enabled: true
  containerd:
    enabled: true
    socket: /run/containerd/containerd.sock

@alacuku the unixsocket may exist, still may not be accessible for some reason since we are using leastPrivileged: true? 🤔

driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true

@doublez13 it would be helpful to know if leastPrivileged: false solved the issue. If you can try, let us know 🙏

leogr · 2024-10-18T14:59:38Z

@UgniusV

have you used Helm? Or have you deployed Falco directly on a host using the package (deb/rpm/tar) installation?
I can't see the container_engines in your configuration. So, it looks like Falco isn't configured to collect container metadata.

doublez13 · 2024-10-18T16:37:07Z

@doublez13 have you checked the socket filepath exits?

The socket file on the host is at /run/containerd/containerd.sock, which then appears to get mounted in the falco container at /host/run/containerd/containerd.sock. Let me know If I need to change the helm value.

@doublez13 it would be helpful to know if leastPrivileged: false solved the issue. If you can try, let us know 🙏

Ah setting leastPrivileged: false got all the pod information!
I've reverted the setting to leastPrivileged: true and confirmed that the values are null again, so that setting seems to be the culprit.

leogr · 2024-10-21T10:07:08Z

leastPrivileged: true

Thank you so much! I hope we are close to finding a solution.

@doublez13
May I ask you for another try?
I suspect that we need another capability to make Falco read the container socket properly.

The missing capability is CAP_DAC_READ_SEARCH. You can add it using the containerSecurityContext option in the helm chart.
So, if possible for you, give it a try with:

containerSecurityContext:
  securityContext:
    capabilities:
      add:
        - BPF
        - SYS_RESOURCE
        - PERFMON
        - SYS_PTRACE
        - CAP_DAC_READ_SEARCH
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true

Thanks

dmeetreey · 2024-10-21T12:11:31Z

Hi, @leogr!

@UgniusV

have you used Helm? Or have you deployed Falco directly on a host using the package (deb/rpm/tar) installation?

I can't see the container_engines in your configuration. So, it looks like Falco isn't configured to collect container metadata.

Yes, we are using Helm.
Updated the config with container_engines, will check if this solves the issue!

leogr · 2024-10-21T13:39:50Z

Hey @dmeetreey

Thank you!

Just one doubt:

Yes, we are using Helm.

Updated the config with container_engines, will check if this solves the issue!

The helm chart should correctly set the container_engines configuration when collectors are enabled. So I don't understand why you must manually update container_engines. Am I missing something? 🤔

doublez13 · 2024-10-21T14:46:44Z

@leogr

The missing capability is CAP_DAC_READ_SEARCH. You can add it using the containerSecurityContext option in the helm chart. So, if possible for you, give it a try with:

Unfortunately I'm not seeing any difference between the following two configs.

containerSecurityContext:
  capabilities:
    add:
      - BPF
      - SYS_RESOURCE
      - PERFMON
      - SYS_PTRACE
#      - CAP_DAC_READ_SEARCH

tty: true
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true

containerSecurityContext:
  capabilities:
    add:
      - BPF
      - SYS_RESOURCE
      - PERFMON
      - SYS_PTRACE
      - CAP_DAC_READ_SEARCH

tty: true
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true

Both have the container fields displayed as follows: (container_image=<NA> container_image_tag=<NA> container_name=<NA> k8s_ns=<NA> k8s_pod_name=<NA>)

dmeetreey · 2024-10-21T15:17:17Z

Hey @dmeetreey

Thank you!

Just one doubt:

Yes, we are using Helm.

Updated the config with container_engines, will check if this solves the issue!

The helm chart should correctly set the container_engines configuration when collectors are enabled. So I don't understand why you must manually update container_engines. Am I missing something? 🤔

@leogr
I guess I misunderstood what you meant.
If I understand correctly now, it is enough to set necessary collectors to true instead of container_engines to start logging the container metadata?
Couldn't find it in the docs, so just making sure.

leogr · 2024-10-21T15:47:02Z

If I understand correctly now, it is enough to set necessary collectors to true instead of container_engines to start logging the container metadata?

If you are deploying Falco via Helm, you should only use collectors setting in the helm values. The chart will configure Falco's config for you. On the other hand, if you're not using Helm, you should configure container_engines by yourself.

So, if you haven't enabled collectors in the Helm chart, please try to enable them (note that they are enabled by default; also, please check that the container unix socket path matches your environment).

If you have collectors enabled already, I would have expected to see the container configuration translated to the container_engines setting in Falco config.

I'm unsure what is happening in your case since I did not see container_engines in Falco's config you posted above.

Hoping my explanation makes sense now 😇

leogr · 2024-10-22T09:03:21Z

@leogr

The missing capability is CAP_DAC_READ_SEARCH. You can add it using the containerSecurityContext option in the helm chart. So, if possible for you, give it a try with:

Unfortunately I'm not seeing any difference between the following two configs.
containerSecurityContext:
  capabilities:
    add:
      - BPF
      - SYS_RESOURCE
      - PERFMON
      - SYS_PTRACE
#      - CAP_DAC_READ_SEARCH

tty: true
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true
containerSecurityContext:
  capabilities:
    add:
      - BPF
      - SYS_RESOURCE
      - PERFMON
      - SYS_PTRACE
      - CAP_DAC_READ_SEARCH

tty: true
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true
Both have the container fields displayed as follows: (container_image=<NA> container_image_tag=<NA> container_name=<NA> k8s_ns=<NA> k8s_pod_name=<NA>)

@doublez13

Is AppArmor enabled on your system? If yes, this should be the only possible cause. If this is the case, we don't have a solution at this time, so I recommend you use leastPrivileged: false for now.

To address this in the long term, I will open a specific issue for the least privileged issue with the Helm chart (or feel free to open it by you if you want).

doublez13 · 2024-10-24T15:24:04Z

Is AppArmor enabled on your system? If yes, this should be the only possible cause. If this is the case, we don't have a solution at this time, so I recommend you use leastPrivileged: false for now.

To address this in the long term, I will open a specific issue for the least privileged issue with the Helm chart (or feel free to open it by you if you want).

@leogr

AppArmor seems to be the cause!
If I run the pod with least leastPrivileged: true and manually set the apparmor profile to unconfined the pod fields are filled.
Thanks for helping to track this down!

leogr · 2024-10-28T14:25:11Z

Is AppArmor enabled on your system? If yes, this should be the only possible cause. If this is the case, we don't have a solution at this time, so I recommend you use leastPrivileged: false for now.
To address this in the long term, I will open a specific issue for the least privileged issue with the Helm chart (or feel free to open it by you if you want).

@leogr

AppArmor seems to be the cause! If I run the pod with least leastPrivileged: true and manually set the apparmor profile to unconfined the pod fields are filled. Thanks for helping to track this down!

Good to know, thank you!

I guess we need both to update the documentation and find a way to force AppArmor profile to unconfined from the pod manifest (not sure if possible).
Adding this to my todo list 😇
/assign

leogr · 2024-10-28T14:28:05Z

Just noticed @doublez13 was already working on the chart, awesome! falcosecurity/charts#769

poiana added the needs-kind label Sep 26, 2024

poiana added kind/bug and removed needs-kind labels Sep 26, 2024

doublez13 mentioned this issue Oct 24, 2024

falco: when leastPrivileged is true, set the apparmor profile to … falcosecurity/charts#769

Merged

3 tasks

poiana assigned leogr Oct 28, 2024

RomanenkoDenys mentioned this issue Nov 13, 2024

[runtime-audit-engine] fix k8s labels collection from containers in syscall event source. deckhouse/deckhouse#10639

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Container information missing in the alerts generated by the "Privileged Shell Spawned in Container" rule. #3345

Container information missing in the alerts generated by the "Privileged Shell Spawned in Container" rule. #3345

dmeetreey commented Sep 26, 2024 •

edited

Loading

dmeetreey commented Sep 26, 2024

UgniusV commented Oct 15, 2024

leogr commented Oct 16, 2024

doublez13 commented Oct 17, 2024 •

edited

Loading

UgniusV commented Oct 18, 2024

leogr commented Oct 18, 2024

leogr commented Oct 18, 2024

doublez13 commented Oct 18, 2024 •

edited

Loading

leogr commented Oct 21, 2024

dmeetreey commented Oct 21, 2024 •

edited

Loading

leogr commented Oct 21, 2024 •

edited

Loading

doublez13 commented Oct 21, 2024

dmeetreey commented Oct 21, 2024 •

edited

Loading

leogr commented Oct 21, 2024

leogr commented Oct 22, 2024

doublez13 commented Oct 24, 2024

leogr commented Oct 28, 2024

leogr commented Oct 28, 2024

Container information missing in the alerts generated by the "Privileged Shell Spawned in Container" rule. #3345

Container information missing in the alerts generated by the "Privileged Shell Spawned in Container" rule. #3345

Comments

dmeetreey commented Sep 26, 2024 • edited Loading

dmeetreey commented Sep 26, 2024

UgniusV commented Oct 15, 2024

leogr commented Oct 16, 2024

doublez13 commented Oct 17, 2024 • edited Loading

UgniusV commented Oct 18, 2024

leogr commented Oct 18, 2024

leogr commented Oct 18, 2024

doublez13 commented Oct 18, 2024 • edited Loading

leogr commented Oct 21, 2024

dmeetreey commented Oct 21, 2024 • edited Loading

leogr commented Oct 21, 2024 • edited Loading

doublez13 commented Oct 21, 2024

dmeetreey commented Oct 21, 2024 • edited Loading

leogr commented Oct 21, 2024

leogr commented Oct 22, 2024

doublez13 commented Oct 24, 2024

leogr commented Oct 28, 2024

leogr commented Oct 28, 2024

dmeetreey commented Sep 26, 2024 •

edited

Loading

doublez13 commented Oct 17, 2024 •

edited

Loading

doublez13 commented Oct 18, 2024 •

edited

Loading

dmeetreey commented Oct 21, 2024 •

edited

Loading

leogr commented Oct 21, 2024 •

edited

Loading

dmeetreey commented Oct 21, 2024 •

edited

Loading