Weird Error in Logs Preventing External Clients from Connecting to Internal Sites

[charlestephen]

So just as a baseline:

• I'm running the container using Podman with a macvlan network mapped to 10.88.88.2, which is a DMZ zone on my home network configuration. My router passes all HTTP/S traffic (among other forwarded ports) this address and I have network firewall rules which prevent the DMZ zone from communicating with others within my home network unless the communication is initiated by a privileged user from outside of the DMZ zone.
• I further isolate the container from the host with host and Podman firewall rules preventing communication between the network interfaces and by dropping container/Linux capabilities to reduce security exposure (Podman requires a root user when using the macvlan or host network, thus the dropping of unnecessary capabilities to reduce exposure).
• I have an additional internal Podman container network attached to both this container and the backend containers for which it's routing traffic. I map the backend containers using their IPs from this network.

Since your most recent update, I have been getting lots of errors like the following:

/usr/local/nginx/lib/lua/crowdsec.lua:293: in function </usr/local/nginx/lib/lua/crowdsec.lua:240>, context: ngx.timer, client: 192.168.2.3, server: 0.0.0.0:81

2024/11/03 15:54:18 [error] 1218#1218: *292 connect() failed (113: Host is unreachable), context: ngx.timer, client: 52.22.236.30, server: 0.0.0.0:443

2024/11/03 15:54:18 [error] 1218#1218: *292 lua entry thread aborted: runtime error: /usr/local/nginx/lib/lua/crowdsec.lua:293: request failed: host is unreachable

stack traceback:

coroutine 0:

        [C]: in function 'error'

        /usr/local/nginx/lib/lua/crowdsec.lua:293: in function </usr/local/nginx/lib/lua/crowdsec.lua:240>, context: ngx.timer, client: 52.22.236.30, server: 0.0.0.0:443

2024/11/03 15:54:19 [error] 1451#1451: *294 connect() failed (113: Host is unreachable), context: ngx.timer, client: 192.168.2.3, server: 0.0.0.0:81

2024/11/03 15:54:19 [error] 1451#1451: *294 lua entry thread aborted: runtime error: /usr/local/nginx/lib/lua/crowdsec.lua:293: request failed: host is unreachable

stack traceback:

coroutine 0:

For external users and services, such as my uptime monitor, this results in failed connections (as in the first failure) but for internal users, it does not, despite the error. What's weird is that previous versions worked AND even as of last night, this version was working fine with no error caused by a 0.0.0.0 host. I can't make sense of it. Any thoughts?

[Zoey2936]

your crowdse config file?

[charlestephen]

Sure. For reference, I've changed the secret keys. Also, the internal Podman network is 10.99.99.0/24 with the host being at 10.99.99.1, thus the Crowdsec and Appsec API being there since I'm using Crowdsec from the Debian host.

ENABLED=true
API_URL=http://10.99.99.1:8080
API_KEY=API_KEY_IS_SECRET
CACHE_EXPIRATION=3600
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=captcha
REQUEST_TIMEOUT=2500
UPDATE_FREQUENCY=10
# By default internal requests are ignored, such as any path affected by rewrite rule.
# set ENABLE_INTERNAL=true to allow checking on these internal requests.
ENABLE_INTERNAL=true
# live or stream
MODE=stream
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
BAN_TEMPLATE_PATH=/data/etc/crowdsec/ban.html
REDIRECT_LOCATION=
RET_CODE=403
#those apply for "captcha" action
#valid providers are recaptcha, hcaptcha, turnstile
CAPTCHA_PROVIDER=recaptcha
# Captcha Secret Key
SECRET_KEY=SECRET_KEY_IS_SECRET
# Captcha Site key
SITE_KEY=SITE_KEY_IS_SECRET
CAPTCHA_TEMPLATE_PATH=/data/etc/crowdsec/captcha.html
CAPTCHA_EXPIRATION=3600

APPSEC_URL=http://10.99.99.1:7422
APPSEC_FAILURE_ACTION=passthrough
APPSEC_CONNECT_TIMEOUT=30000
APPSEC_SEND_TIMEOUT=30000
APPSEC_PROCESS_TIMEOUT=30000
ALWAYS_SEND_TO_APPSEC=true
SSL_VERIFY=true

[Zoey2936]

and npmplus can access http://10.99.99.1:8080?

[charlestephen]

So I've registered the npmplus bouncers (I'm running two distinct instances of NPMPlus) using cscli bouncers add npmplus-uid but they no longer show up with their version or any details about them on Crowdsec and when I just ran the cscli bouncers list it shows both bouncers but no further details.

This is obviously complicated because of my network configuration, but I had it working before. I will try some troubleshooting.

[charlestephen]

Figured it out. I am running two instances of NPMPlus that were both trying to auth against the same endpoint. Crowdsec doesn't like this and kept failing and restarting quietly (I had to dig into the logs rather deeply to find the issue). You should make a note in your docs that multiple instances of NPMPlus require multiple Crowdsec endpoints to work properly

[Zoey2936]

you need at least different api keys and the readme mentions at no place that crowdsec api keys can be reused

[charlestephen]

I had different apis. There needs to be two different CrowdSec listening endpoints as well. That isn’t necessarily clear from your docs.

[Zoey2936]

no it is no needed, I have four NPMplus all connecting to the same crowdsec api with four different api keys

[charlestephen]

Ah, maybe it's because I am using the Crowdsec host install? Who knows. It wouldn't let me use different API keys on it, in any event.

[Zoey2936]

maybe ask at the crowdsec repo?