Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Got "couldn't init manager error" when launching ecapture in docker container. #403

Closed
pfmiles opened this issue Oct 13, 2023 · 5 comments
Closed

Got "couldn't init manager error" when launching ecapture in docker container. #403

pfmiles opened this issue Oct 13, 2023 · 5 comments
Labels
question Further information is requested

Comments

@pfmiles
Copy link

pfmiles commented Oct 13, 2023

Trying to run ecapture from cmd line, in a Docker container, as root. It emits error:

root@test-xhunterbase-56b679589-w7nmz:/tmp/ecapture-v0.6.3-linux-x86_64# ./ecapture --libssl="/usr/lib/x86_64-linux-gnu/libssl.so.1.1" --gnutls="/usr/lib/x86_64-linux-gnu/libgnutls.so.30" --nspr="/usr/lib/x86_64-linux-gnu/libnspr4.so" tls
2023/10/13 14:08:29 Your environment is a container. We will not detect the BTF config.
tls_2023/10/13 14:08:29 ECAPTURE :: ecapture Version : linux_x86_64:0.6.3-20230927-f0cfbdf:5.15.0-1046-azure
tls_2023/10/13 14:08:29 ECAPTURE :: Pid Info : 29690
tls_2023/10/13 14:08:29 ECAPTURE :: Kernel Info : 4.19.24
tls_2023/10/13 14:08:29 EBPFProbeOPENSSL        module initialization
tls_2023/10/13 14:08:29 EBPFProbeOPENSSL        master key keylogger: ecapture_masterkey.log
tls_2023/10/13 14:08:29 ECAPTURE ::     Module.Run()
tls_2023/10/13 14:08:29 EBPFProbeOPENSSL        UPROBE MODEL
tls_2023/10/13 14:08:29 EBPFProbeOPENSSL        origin version:OpenSSL 1.1.1n, as key:openssl 1.1.1n
tls_2023/10/13 14:08:29 EBPFProbeOPENSSL        HOOK type:2, binrayPath:/usr/lib/x86_64-linux-gnu/libssl.so.1.1
tls_2023/10/13 14:08:29 EBPFProbeOPENSSL        libPthread:/lib/x86_64-linux-gnu/libpthread.so.0
tls_2023/10/13 14:08:29 EBPFProbeOPENSSL        Hook masterKey function:SSL_write
tls_2023/10/13 14:08:29 EBPFProbeOPENSSL        Your kernel version is less than 5.2, the following parameters will be ignored:[target_pid, target_uid, target_port]
tls_2023/10/13 14:08:29 EBPFProbeOPENSSL        BPF bytecode filename:user/bytecode/openssl_1_1_1j_kern_less52.o
tls_2023/10/13 14:08:29 EBPFProbeOPENSSL        module run failed, [skip it]. error:couldn't init manager error:operation not permitted , couldn't adjust RLIMIT_MEMLOCK
tls_2023/10/13 14:08:29 EBPFProbeGNUTLS module initialization
tls_2023/10/13 14:08:29 ECAPTURE ::     Module.Run()
tls_2023/10/13 14:08:29 EBPFProbeGNUTLS BPF bytecode filename:user/bytecode/gnutls_kern_less52.o
tls_2023/10/13 14:08:29 EBPFProbeGNUTLS HOOK type:2, binrayPath:/usr/lib/x86_64-linux-gnu/libgnutls.so.30
tls_2023/10/13 14:08:29 EBPFProbeGNUTLS module run failed, [skip it]. error:couldn't init manager error:operation not permitted , couldn't adjust RLIMIT_MEMLOCK
tls_2023/10/13 14:08:29 EBPFProbeNSPR   module initialization
tls_2023/10/13 14:08:29 ECAPTURE ::     Module.Run()
tls_2023/10/13 14:08:29 EBPFProbeNSPR   BPF bytecode filename:user/bytecode/nspr_kern_less52.o
tls_2023/10/13 14:08:29 EBPFProbeNSPR   HOOK type:2, binrayPath:/usr/lib/x86_64-linux-gnu/libnspr4.so
tls_2023/10/13 14:08:29 EBPFProbeNSPR   module run failed, [skip it]. error:couldn't init manager error:operation not permitted , couldn't adjust RLIMIT_MEMLOCK 
tls_2023/10/13 14:08:29 ECAPTURE ::     No runnable modules, Exit(1)

It seems all of three possible tls hooking modules failed to run. And all of those point to a same error: couldn't init manager error:operation not permitted , couldn't adjust RLIMIT_MEMLOCK

To Reproduce
Steps to reproduce the behavior:
Just download ecapture binary(amd64 version), then unzip it and run the command in a bash repl.

Expected behavior

Screenshots

Linux Server/Android (please complete the following information):

  • Env:
  • KUBERNETES_SERVICE_PORT_HTTPS=443
    KUBERNETES_SERVICE_PORT=6443
    HOSTNAME=test-xhunterbase-56b679589-w7nmz
    LANGUAGE=en_US.UTF-8
    TEST_XHUNTERBASE_SVC_PORT_22_TCP=tcp://172.21.10.6:22
    JAVA_HOME=/opt/jdk
    PWD=/tmp/ecapture-v0.6.3-linux-x86_64
    LS_OPTIONS=--color=auto
    TEST_XHUNTERBASE_SVC_PORT_22_TCP_PORT=22
    HOME=/root
    LANG=en_US.UTF-8
    KUBERNETES_PORT_443_TCP=tcp://172.21.0.1:443
    TEST_XHUNTERBASE_SVC_PORT_10000_TCP_PORT=10000
    TEST_XHUNTERBASE_SVC_PORT_10000_TCP=tcp://172.21.10.6:10000
    TEST_XHUNTERBASE_SVC_PORT_10000_TCP_ADDR=172.21.10.6
    TEST_XHUNTERBASE_SVC_PORT_10000_TCP_PROTO=tcp
    TERM=xterm-256color
    TEST_XHUNTERBASE_SVC_PORT_22_TCP_ADDR=172.21.10.6
    aliyun_log_crd_user_defined_id=k8s-group-c3e4daee1064f4158891f67a0e996ef7a
    SHLVL=2
    TEST_XHUNTERBASE_SVC_PORT=tcp://172.21.10.6:22
    KUBERNETES_PORT_443_TCP_PROTO=tcp
    KUBERNETES_PORT_443_TCP_ADDR=172.21.0.1
    TEST_XHUNTERBASE_SVC_PORT_22_TCP_PROTO=tcp
    TEST_XHUNTERBASE_SVC_SERVICE_PORT_TEST_XHUNTERBASE_SVC_22_22=22
    TEST_XHUNTERBASE_SVC_SERVICE_PORT_TEST_XHUNTERBASE_SVC_10000_10000=10000
    TEST_XHUNTERBASE_SVC_SERVICE_PORT=22
    KUBERNETES_SERVICE_HOST=172.26.19.176
    LC_ALL=en_US.UTF-8
    KUBERNETES_PORT=tcp://172.21.0.1:443
    KUBERNETES_PORT_443_TCP_PORT=443
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/jdk/bin
    TEST_XHUNTERBASE_SVC_SERVICE_HOST=172.21.10.6
    OLDPWD=/tmp
    _=/usr/bin/env
  • OS: Debian 11 "bullseye"
  • Arch: amd64
  • Kernel Version: 4.19.24-7.34.cbp.al7.x86_64
  • Version: v0.6.3-linux-x86_64

Additional context
I'm running a debian 11 linux image in a docker container.
@cfc4n
Copy link
Member

cfc4n commented Oct 13, 2023

what's your docker launched command?

@cfc4n
Copy link
Member

cfc4n commented Oct 14, 2023 •
edited
Loading

Have you granted SYS_ADMIN permissions to Docker?

e.g. : docker run --cap-add=SYS_ADMIN

If you encounter any other issues while using eCapture in a container, you can refer to #65.

@cfc4n cfc4n added the question Further information is requested label Oct 14, 2023
@pfmiles
Copy link
Author

pfmiles commented Oct 16, 2023

Have you granted SYS_ADMIN permissions to Docker?

e.g. : docker run --cap-add=SYS_ADMIN

If you encounter any other issues while using eCapture in a container, you can refer to #65.

Thanks for the suggestion. I did not add the --cap-add=SYS_ADMIN option when executing docker run. I will try adding this option as soon as possible to see if it works. However, it is a bit troublesome because I am not running docker in the local command line, but on the Alibaba Cloud. Therefore, I may need to wrestle with its web console for a while. I will try it out and provide the results as soon as possible.

@pfmiles
Copy link
Author

pfmiles commented Oct 19, 2023

Sorry for the late reply.
I couldn't find a place in the Aliyun web console where I can add the --cap-add=SYS_ADMIN or --privileged options.
So I tried using a Kubernetes client SDK programmatically to add these options, but I received an error message saying that the managed serverless service does not support adding these options:
1
2

Then I ran the same x64-based image on my own Macbook Pro (M1) laptop with Rosetta2 for automatic instruction translation, with the --privileged option:

docker run --hostname=ebd37a374017 --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -p 6022:22 --label='version=0.4.0' --runtime=runc -d -it --privileged xhunterbase:0.4.0

The output was as follows:

admin@ebd37a374017:/tmp/ecapture-v0.6.4-linux-x86_64$ sudo ./ecapture --libssl="/usr/lib/x86_64-linux-gnu/libssl.so.1.1" --gnutls="/usr/lib/x86_64-linux-gnu/libgnutls.so.30" --nspr="/usr/lib/x86_64-linux-gnu/libnspr4.so" tls
2023/10/19 18:05:44 Your environment is a container. We will not detect the BTF config.
tls_2023/10/19 18:05:44 ECAPTURE :: ecapture Version : linux_x86_64:0.6.4-20231015-f50129f:5.15.0-1047-azure
tls_2023/10/19 18:05:44 ECAPTURE :: Pid Info : 223
tls_2023/10/19 18:05:44 ECAPTURE :: Kernel Info : 5.15.49
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	module initialization
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	master key keylogger: ecapture_masterkey.log
tls_2023/10/19 18:05:44 ECAPTURE ::	Module.Run()
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	UPROBE MODEL
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	origin version:OpenSSL 1.1.1w, as key:openssl 1.1.1w
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	OpenSSL/BoringSSL version not found from shared library file, used default version:linux_default_1_1_1
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	HOOK type:2, binrayPath:/usr/lib/x86_64-linux-gnu/libssl.so.1.1
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	libPthread:/lib/x86_64-linux-gnu/libpthread.so.0
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	Hook masterKey function:SSL_write
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	target all process. 
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	target all users. 
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	BPF bytecode filename:user/bytecode/openssl_1_1_1j_kern.o
tls_2023/10/19 18:05:44 EBPFProbeOPENSSL	module run failed, [skip it]. error:couldn't init manager error:map connect_events: map create: function not implemented (without BTF k/v) , couldn't load eBPF programs, cs:&{map[.rodata:Array(keySize=4, valueSize=36, maxEntries=1, flags=128) active_ssl_read_args_map:Hash(keySize=8, valueSize=16, maxEntries=1024, flags=0) active_ssl_write_args_map:Hash(keySize=8, valueSize=16, maxEntries=1024, flags=0) bpf_context:LRUHash(keySize=8, valueSize=408, maxEntries=2048, flags=0) bpf_context_gen:Array(keySize=4, valueSize=408, maxEntries=1, flags=0) connect_events:PerfEventArray(keySize=4, valueSize=4, maxEntries=5, flags=0) data_buffer_heap:PerCPUArray(keySize=4, valueSize=4152, maxEntries=1, flags=0) mastersecret_events:PerfEventArray(keySize=4, valueSize=4, maxEntries=5, flags=0) network_map:LRUHash(keySize=20, valueSize=20, maxEntries=10240, flags=0) skb_data_buffer_heap:PerCPUArray(keySize=4, valueSize=40, maxEntries=1, flags=0) skb_events:PerfEventArray(keySize=4, valueSize=4, maxEntries=5, flags=0) ssl_st_fd:Hash(keySize=8, valueSize=8, maxEntries=10240, flags=0) tls_events:PerfEventArray(keySize=4, valueSize=4, maxEntries=5, flags=0)] map[egress_cls_func:0xc0001e6c60 ingress_cls_func:0xc0001e6bd0 probe_SSL_set_fd:0xc0001e6ab0 probe_connect:0xc0001e6e10 probe_entry_SSL_read:0xc0001e6d80 probe_entry_SSL_write:0xc0001e6900 probe_ret_SSL_read:0xc0001e6a20 probe_ret_SSL_write:0xc0001e6990 probe_ssl_master_key:0xc0001e6b40 tcp_sendmsg:0xc0001e6cf0] 0xc000264180 LittleEndian}
tls_2023/10/19 18:05:44 EBPFProbeGNUTLS	module initialization
tls_2023/10/19 18:05:44 ECAPTURE ::	Module.Run()
tls_2023/10/19 18:05:44 EBPFProbeGNUTLS	BPF bytecode filename:user/bytecode/gnutls_kern.o
tls_2023/10/19 18:05:44 EBPFProbeGNUTLS	HOOK type:2, binrayPath:/usr/lib/x86_64-linux-gnu/libgnutls.so.30
tls_2023/10/19 18:05:44 EBPFProbeGNUTLS	target all process. 
tls_2023/10/19 18:05:44 EBPFProbeGNUTLS	module run failed, [skip it]. error:couldn't init manager error:map active_ssl_write_args_map: load BTF: detect support for Map BTF (Var/Datasec): detect support for BTF: function not implemented , couldn't load eBPF programs, cs:&{map[.rodata:Array(keySize=4, valueSize=32, maxEntries=1, flags=128) active_ssl_read_args_map:Hash(keySize=8, valueSize=8, maxEntries=1024, flags=0) active_ssl_write_args_map:Hash(keySize=8, valueSize=8, maxEntries=1024, flags=0) data_buffer_heap:PerCPUArray(keySize=4, valueSize=4144, maxEntries=1, flags=0) gnutls_events:PerfEventArray(keySize=4, valueSize=4, maxEntries=5, flags=0)] map[probe_entry_SSL_read:0xc0001e7440 probe_entry_SSL_write:0xc0001e7320 probe_ret_SSL_read:0xc0001e74d0 probe_ret_SSL_write:0xc0001e73b0] 0xc0000ca880 LittleEndian}
tls_2023/10/19 18:05:44 EBPFProbeNSPR	module initialization
tls_2023/10/19 18:05:44 ECAPTURE ::	Module.Run()
tls_2023/10/19 18:05:44 EBPFProbeNSPR	BPF bytecode filename:user/bytecode/nspr_kern.o
tls_2023/10/19 18:05:44 EBPFProbeNSPR	HOOK type:2, binrayPath:/usr/lib/x86_64-linux-gnu/libnspr4.so
tls_2023/10/19 18:05:44 EBPFProbeNSPR	target all process. 
tls_2023/10/19 18:05:44 EBPFProbeNSPR	module run failed, [skip it]. error:couldn't init manager error:map active_ssl_write_args_map: load BTF: detect support for Map BTF (Var/Datasec): detect support for BTF: function not implemented , couldn't load eBPF programs, cs:&{map[.rodata:Array(keySize=4, valueSize=32, maxEntries=1, flags=128) active_ssl_read_args_map:Hash(keySize=8, valueSize=8, maxEntries=1024, flags=0) active_ssl_write_args_map:Hash(keySize=8, valueSize=8, maxEntries=1024, flags=0) data_buffer_heap:PerCPUArray(keySize=4, valueSize=4144, maxEntries=1, flags=0) nspr_events:PerfEventArray(keySize=4, valueSize=4, maxEntries=5, flags=0)] map[probe_entry_SSL_read:0xc0001e7cb0 probe_entry_SSL_write:0xc0001e7b90 probe_ret_SSL_read:0xc0001e7b00 probe_ret_SSL_write:0xc0001e7c20] 0xc0000cac80 LittleEndian} 
tls_2023/10/19 18:05:44 ECAPTURE :: 	No runnable modules, Exit(1)

I can notice that the error message has changed, but it still failed to start, indicating "...function not implemented...". I suspect that the eBPF feature is not fully supported by the macOS kernel on my Macbook Pro.

Since the serverless service I mainly use on the cloud does not allow me to add admin privileges to the Docker container instances, I may not use ecapture in this scenario for now.
I also do not currently have any Linux servers with a kernel version greater than 4.18, so I haven't done any further testing.

If I ever need to do something similar on a VPS machine that I can manage myself, I will follow the instructions in issue:#65 and try using ecapture, which should work I guess.

Thanks anyway!

@cfc4n
Copy link
Member

cfc4n commented Oct 21, 2023

The privileged parameter will introduce significant security issues, endangering the host system. Some public cloud vendors disable this.

The host kernel in macOS differs from Linux and does not support eBPF features.

For security issues arising from granting SYS_ADMIN permissions to containers, you can refer to this video: eBPF容器逃逸与隐藏账号rootkit

@cfc4n cfc4n closed this as completed Oct 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pfmiles @cfc4n