frankenphp_worker_php_ready_worker is skewing to negative values with worker restarts

[LauJosefsen]

What happened?

Hi.

We have excitingly been trying out the new 1.3.0 release. We immediately setup an alert on the frankenphp_worker_php_ready_worker metric on our environments.

We however observed that this metric has a decreasing pattern to it, that is not equal to the relationship of total_workers - busy_workers.

Look at this plot of the metric of a somewhat busy container (20RPS ish) (each line is a seperate container)

I was very worried of this metric, however it has now become a negative value, indicating that is keep decreasing even below 0. In our case we may see the ready_worker metric report -2, while the busy worker metric is 4 and total workers is 60. So in this case I assume the correct value is 56? Or close to 56.

It looks to be somewhat correlated to worker restarts, looking at this graph:

The service is responding as normal even with negative ready workers

Build Type

Docker (Debian Bookworm)

Worker Mode

Yes

Operating System

GNU/Linux

CPU Architecture

x86_64

PHP configuration

Not relevant

Relevant log output

No response

[AlliBalliBaba]

Yeah it looks like ReadyWorker is only called once, while StopWorker is called every time a worker restarts, skewing the metric.
@withinboredom after #1137 this could maybe also be refactored to pull metrics instead of pushing them, since we can just loop over all workers and the threads assigned to them.

[withinboredom]

This is possibly a merge-conflict resolution error on my part, but at a glance, it appears to be working as designed.

It is stopping workers, then it calls initWorkers, which should eventually call startNewWorkerThread, which should call metrics.StartWorker(worker.fileName), which should increment the worker count back up.

I think something is missing here.

this could maybe also be refactored to pull metrics instead of pushing them

I don't know how I would feel about a God System reaching into things just to extract metrics.

[withinboredom]

I think something is missing here.

Aha!

frankenphp_worker_php_ready_worker, per the documentation:

The number of workers that have called frankenphp_handle_request at least once.

This should be a static number once started. It doesn't (usually) increase. It is indicative of a successful startup, not part of request handling itself.

[withinboredom]

ah, yeah. The metric changed meaning during the merge conflict. I'll fix it.