TaskList metrics / Scaling Workers

[tfcace]

Hi all!

Is there a metric to show the size of task lists?
The motivation is - I'm trying to see if there are any metrics that can serve as driver to scaling the number of Workers.

So if the size of the tasks lists (or other metric) goes up, we should scale the number of Workers handling that task list.

Many thanks!
Assaf

[longquanzheng]

There is no metrics of backlog size.

The metrics that you should look at:

Client side metrics

All those metrics are tagged by domain and WorkflowType.

• scheduled-to-start-latency metrics for activity/decision. There could be only two reasons that this is too high: 1) either tasklist is overloaded that you need to split the tasklist(by using multiple tasklists or enabling scalable tasklist feature), see discussion 2) worker doesn't have enough capacity to poll the tasks from tasklist, so you need to add more worker like you said to scaling the workers.
  • Go client activity: https://github.com/uber-go/cadence-client/blob/v0.18.2/internal/common/metrics/constants.go#L61
  • Go client decision: https://github.com/uber-go/cadence-client/blob/v0.18.2/internal/common/metrics/constants.go#L46
  • Java client activity: hhttps://github.com/uber/cadence-java-client/blob/v3.3.0/src/main/java/com/uber/cadence/internal/metrics/MetricsType.java#L85
  • Java client decision: Go client activity: https://github.com/uber-go/cadence-client/blob/v0.18.2/internal/common/metrics/constants.go#L61
• poll-no-task counter metrics indicate that the worker is idle for the long poll(default to 2 mins). Having this counter being non-zero means your worker is enough
  • you can find the according latency in https://github.com/uber-go/cadence-client/blob/v0.18.2/internal/common/metrics/constants.go and https://github.com/uber/cadence-java-client/blob/v3.3.0/src/main/java/com/uber/cadence/internal/metrics/MetricsType.java
• similarly, poll-latency can be used to know if worker is idle. Because long-poll timeout is 2 minutes, you can look at P50/P99/Max of the latency. If P99 or Max of the metrics is always 2 minutes then worker should be idle(having extra capacity).

Server side metrics

The above client metrics are usually enough for the purpose of this question.

Usually server metrics only being used by cluster admin, or for further investigation.

There are lots of per_tasklist metrics on server side emitted: https://github.com/uber/cadence/blob/v0.22.0/common/metrics/defs.go#L2194

Note that sticky tasklist are temporarily created per worker lifecycle so could cause high cardinality issue. So they will always use the same word __sticky__ as replacement.
All the metrics are tagged by domain.

[zhangkyou]

My scheduled-to-start-latency metrics both activity and decision, most points less than 100ms(P99),but also had some points big than 40s(P99).
The poll-latency metrics always 60s(P99).
The poll-no-task counter metrics mostly are non-zero,but also had some zero points last 2-3 minutes,the metrics curve looks like some gap(2-3mintues) with zero value.
What does these metrics indicate?

[longquanzheng]

So they mean what I said in the above😐

100ms of scheduled to start latency is pretty good number.
Does the intermittent high scheduled to start latency (40s) correlate with Cadence cluster or your workers being down, or the tasklist is overloaded? If that bothers, you need to find at what caused the high latency by looking at other metrics as well

[tfcace]

Thanks a lot @longquanzheng
To improve my understanding, tasklists are split into activity (scheduled activities), and decisions (workflow flow control)? and they share the same name?

Some followups - can those metrics be observed on a tasklist by tasklist breakdown? Otherwise it may still be hard to know which workers to scale.
Also, is there any correlation between the scheduled-to-start-latency metric and the SCHEDULED_TO_START timeout? Can get a sense also by the number of those timeouts if I have them?

[longquanzheng]

tasklists are split into activity (scheduled activities), and decisions (workflow flow control)? and they share the same name?

They share the same name but are treated as different tasklists internally. Using the same name is because by default workflow will schedule activities using the same tasklist name(when not providing a tasklist). Internally, a tasklist uses "tasklistName+tasklistType" as unique identifier.

can those metrics be observed on a tasklist by tasklist breakdown? Otherwise it may still be hard to know which workers to scale.

Those Go/Java client metrics are tagged by domain+workflowType so you should be able to used them.
In addition, the server is emitting per_tasklist metrics on matching service. I just updated my answer in above to include server metrics.

any correlation between the scheduled-to-start-latency metric and the SCHEDULED_TO_START timeout

Yes. However, the only SCHEDULED_TO_START timeout you can set is for activity. Decision can allow infinite scheduled to start timeout