Segmentation fault (core dumped) #1140

jpvdw86 · 2024-11-06T08:43:38Z

What happened?

We are running FrankenPHP (non-static variant) in Docker on our Kubernetes platform. This is in worker mode with a Symfony application. The application seems to work well; however, as traffic increases, we regularly encounter segmentation fault errors.

We have already rolled back from the PHP 8.3 variant with Gmagick because we were getting errors during some crons with the message, "Disconnected 'Magick: abort due to signal 11 (SIGSEGV) Segmentation Fault' ... Aborted (core dumped)."

We have therefore reverted to PHP 8.2 with Imagick. The "Magick abort" error is gone, but we still sporadically experience segmentation faults.

In the documentation, I read about the OpenSSL bug, but this only applies to the static build, correct? Additionally, we are following the recommendation to use the provided Docker image.

Base dockerfile

FROM dunglas/frankenphp:php8.3-bookworm

ARG TARGETPLATFORM
ARG TARGETARCH
ENV COMPOSER_ALLOW_SUPERUSER=1
ENV PHP_INI_SCAN_DIR=":$PHP_INI_DIR/app.conf.d"

WORKDIR /var/www/html

RUN apt-get update && apt-get install -y -f \
    build-essential \
    libgraphicsmagick1-dev \
    python3 \
    python3-pip \
    curl \
    nodejs \
    wget \
    git \
    npm \
    librabbitmq-dev \
    autoconf \
    ca-certificates \
    fontconfig \
    libc6 \
    libfreetype6 \
    libjpeg62-turbo \
    libpng16-16 \
    libssl3 \
    libstdc++6 \
    libx11-6 \
    libxcb1 \
    libxext6 \
    libxrender1 \
    xfonts-75dpi \
    xfonts-base \
    cron \
    zlib1g && \
    npm install -g node-gyp

RUN apt-get install -y -f procps

RUN curl -o- -L https://yarnpkg.com/install.sh | bash && \
    export PATH="$HOME/.yarn/bin:$HOME/.config/yarn/global/node_modules/.bin:$PATH"

RUN wget https://github.com/wkhtmltopdf/packaging/releases/download/0.12.6.1-3/wkhtmltox_0.12.6.1-3.bookworm_${TARGETARCH}.deb && \
    dpkg -i wkhtmltox_0.12.6.1-3.bookworm_${TARGETARCH}.deb || apt-get install -f -y;

RUN set -eux; \
    install-php-extensions \
    opcache \
    gmagick \
    @composer \
    amqp \
    bcmath \
    exif \
    gd \
    apcu \
    pdo_mysql \
    ftp \
    soap \
    intl \
    csv \
    zip \
    redis \
    ssh2 \
    yaml;

COPY Caddyfile /etc/caddy/Caddyfile
COPY conf.d/10-app.ini $PHP_INI_DIR/app.conf.d/

RUN apt-get autoremove -y \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

CMD [ "frankenphp", "run", "--config", "/etc/caddy/Caddyfile" ]

Build Type

Docker (Debian Bookworm)

Worker Mode

Yes

Operating System

GNU/Linux

CPU Architecture

x86_64

PHP configuration

Relevant log output

################ Start FrankenPHP ################ 
{"level":"info","ts":1730881115.6973963,"msg":"using config from file","file":"/var/www/html/.docker/frankenphp/Caddyfile"}
{"level":"info","ts":1730881115.6984708,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1730881115.6984835,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/var/www/html/.docker/frankenphp/Caddyfile","line":15}
{"level":"info","ts":1730881115.699203,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"warn","ts":1730881115.6993783,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
{"level":"info","ts":1730881115.6994014,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000935480"}
{"level":"info","ts":1730881115.6997063,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1730881115.699732,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1730881115.7017167,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
{"level":"info","ts":1730881115.7018588,"logger":"tls","msg":"finished cleaning storage units"}
<br />
<b>Warning</b>:  Can't preload unlinked class App\UX\DataTable\Adapter\Doctrine\ORM\QueryBuilderProcessorInterface@anonymous: Unknown interface App\UX\DataTable\Adapter\Doctrine\ORM\QueryBuilderProcessorInterface in <b>/var/www/html/src/UX/DataTable/Adapter/Doctrine/ORMAdapter.php</b> on line <b>238</b><br />
{"level":"info","ts":1730881124.2756066,"msg":"FrankenPHP started 🐘","php_version":"8.2.25","num_threads":26}
{"level":"info","ts":1730881124.2758808,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1730881124.2758927,"msg":"serving initial configuration"}
<some requests......>
Segmentation fault (core dumped)

Caddyfile

{
	frankenphp {
		num_threads: 6
		worker {
			file /var/www/html/public/index.php
			num 25
			env APP_RUNTIME Runtime\FrankenPhpSymfony\Runtime
		}
	}
}

:85 {
	metrics
	log {
    	output stderr
		level WARN
		format console
	}
}

:80 {
	log {
		output stderr
		level DEBUG
		format console
	}

	root * /var/www/html/public
	encode zstd br gzip

	vulcain

	header {
		-Server
		-X-Powered-By
		?Permissions-Policy "browsing-topics=()"
		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
	}

    header /bundles/* {
        Cache-Control "max-age=86400, private"
    }

    header /build/* {
        Cache-Control "max-age=86400, private"
    }

	php_server
}

EDIT: Another error appears, maybe related ?

github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1({0x7f33ac7b0810, 0xc00033e3c0}, 0xc000121560)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:268 +0x244 fp=0xc000e86df0 sp=0xc000e86d78 pc=0x1342c24
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2325640?, {0x7f33ac7b0810?, 0xc00033e3c0?}, 0xe?)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x29 fp=0xc000e86e18 sp=0xc000e86df0 pc=0x1326289
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1({0x7f33ac7b0810, 0xc00033e3c0}, 0xc000121560)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:268 +0x244 fp=0xc000e86e90 sp=0xc000e86e18 pc=0x1342c24
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc000e86f20?, {0x7f33ac7b0810?, 0xc00033e3c0?}, 0x4?)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x29 fp=0xc000e86eb8 sp=0xc000e86e90 pc=0x1326289
github.com/caddyserver/caddy/v2/modules/caddyhttp.VarsMiddleware.ServeHTTP(0xc00015d8c0, {0x7f33ac7b0810, 0xc00033e3c0}, 0xc000121560, {0x230f9e0, 0xc0001309e0})
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/vars.go:67 +0x2c2 fp=0xc000e86f90 sp=0xc000e86eb8 pc=0x134bd22
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*VarsMiddleware).ServeHTTP(0xc00015c600?, {0x7f33ac7b0810?, 0xc00033e3c0?}, 0xc00015c7e0?, {0x230f9e0?, 0xc0001309e0?})
	<autogenerated>:1 +0x38 fp=0xc000e86fd0 sp=0xc000e86f90 pc=0x1355538
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x7f33ac7b0810, 0xc00033e3c0}, 0xc000121560)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:333 +0xd2 fp=0xc000e87030 sp=0xc000e86fd0 pc=0x1343032
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x230f9e0?, {0x7f33ac7b0810?, 0xc00033e3c0?}, 0xc000121560?)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x29 fp=0xc000e87058 sp=0xc000e87030 pc=0x1326289
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1({0x7f33ac7b0810, 0xc00033e3c0}, 0xc000121560)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:300 +0x325 fp=0xc000e870d0 sp=0xc000e87058 pc=0x1342d05
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x7f33ac7e3018?, {0x7f33ac7b0810?, 0xc00033e3c0?}, 0x18?)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x29 fp=0xc000e870f8 sp=0xc000e870d0 pc=0x1326289
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).enforcementHandler(0x415ce5?, {0x7f33ac7b0810?, 0xc00033e3c0?}, 0x1?, {0x230f9e0?, 0xc000130a00?})
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/server.go:447 +0x24b fp=0xc000e871d0 sp=0xc000e870f8 pc=0x13455ab
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Provision.(*Server).wrapPrimaryRoute.func1({0x7f33ac7b0810?, 0xc00033e3c0?}, 0x4e23af?)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/server.go:423 +0x35 fp=0xc000e87210 sp=0xc000e871d0 pc=0x131c7f5
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc0002e4870?, {0x7f33ac7b0810?, 0xc00033e3c0?}, 0xc000121560?)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x29 fp=0xc000e87238 sp=0xc000e87210 pc=0x1326289
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000877208, {0x2320da0, 0xc0008120e0}, 0xc000121200)
	/root/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/server.go:353 +0xbbc fp=0xc000e87b60 sp=0xc000e87238 pc=0x134429c
net/http.serverHandler.ServeHTTP({0x231a1b8?}, {0x2320da0?, 0xc0008120e0?}, 0x6?)
	/usr/local/go/src/net/http/server.go:3142 +0x8e fp=0xc000e87b90 sp=0xc000e87b60 pc=0x7c38ae
net/http.(*conn).serve(0xc000989c20, {0x2325640, 0xc000490b70})
	/usr/local/go/src/net/http/server.go:2044 +0x5e8 fp=0xc000e87fb8 sp=0xc000e87b90 pc=0x7beb88
net/http.(*Server).Serve.gowrap3()
	/usr/local/go/src/net/http/server.go:3290 +0x28 fp=0xc000e87fe0 sp=0xc000e87fb8 pc=0x7c40c8
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000e87fe8 sp=0xc000e87fe0 pc=0x47d9a1
created by net/http.(*Server).Serve in goroutine 24
	/usr/local/go/src/net/http/server.go:3290 +0x4b4

The text was updated successfully, but these errors were encountered:

jpvdw86 · 2024-11-07T08:32:08Z

Small update. We just disabled worker mode and now its working without any segmentation faults

dunglas · 2024-11-07T10:01:38Z

Would you manage to gather a stack trace? This is the only way we can get an idea of what's going on :/

Alliballibaba2 · 2024-11-07T16:19:34Z

Could you also try with worker mode with op_cache disabled?

jpvdw86 · 2024-11-20T19:23:43Z

In certain situations, we experienced significant issues with gmagick. As a result, we reverted to PHP 8.2 with imagick, which resolved some of the problems.

However, segmentation faults still occur frequently. To address this, we disabled the opcache preload function. It seemed to reduce the errors somewhat, but this observation is based purely on intuition. We have not fully disabled opcache to test further. @Alliballibaba2

@dunglas Unfortunately, I was unable to generate a stack trace or kernel dumps. We rolled back the environment to a version without FrankenPHP. On the test environment, I couldn't reproduce the issue, likely due to insufficient load to simulate the real-world scenario.

We also performed a load test with K6, but everything ran smoothly during testing. The issue arises only under real-time conditions, where large amounts of data and forms are processed. This is challenging to replicate in a controlled test environment. The application in question is an internal tool used to streamline business processes and is protected behind SSO authentication.

jpvdw86 added the bug Something isn't working label Nov 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault (core dumped) #1140

Segmentation fault (core dumped) #1140

jpvdw86 commented Nov 6, 2024 •

edited

Loading

jpvdw86 commented Nov 7, 2024

dunglas commented Nov 7, 2024

Alliballibaba2 commented Nov 7, 2024

jpvdw86 commented Nov 20, 2024

Segmentation fault (core dumped) #1140

Segmentation fault (core dumped) #1140

Comments

jpvdw86 commented Nov 6, 2024 • edited Loading

What happened?

Build Type

Worker Mode

Operating System

CPU Architecture

PHP configuration

Relevant log output

EDIT: Another error appears, maybe related ?

jpvdw86 commented Nov 7, 2024

dunglas commented Nov 7, 2024

Alliballibaba2 commented Nov 7, 2024

jpvdw86 commented Nov 20, 2024

jpvdw86 commented Nov 6, 2024 •

edited

Loading