edits to CPU Performance docs

machines/cpu-performance.html.md

@@ -7,27 +7,31 @@ nav: machines
 We offer two kinds of virtual CPUs for Machines: `shared` and `performance`. Both run on the same physical hardware, have the same clock speed, etc... The difference is how much time they are allowed to spend running your applications.
 
 
-CPU Type      | Period<sup>1</sup> | Baseline Quota<sup>1</sup> | Max Quota Balance<sup>1</sup>
---------      | ------ | -------------- | -----------------
-`shared`      | 80ms   | 5ms (1/16th)   | 500s
-`performance` | 80ms   | 50ms (10/16th) | 5000s
+| CPU Type      | Period<sup>1</sup> | Baseline Quota<sup>1</sup> | Max Burst Balance<sup>1</sup> |
+|---------------|--------------------|----------------------------|-------------------------------|
+| `shared`      | 80ms               | 5ms (1/16)                 | 500s                          |
+| `performance` | 80ms               | 50ms (10/16)               | 5000s                         |
 
-We enforce limits using the [Linux `cpu.cfs_quota_us` cgroup](https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.rst). For each 80ms period of time, we instruct the Linux scheduler to run `shared` vCPUs for no more than 5ms and `performance` vCPUs no more than 50ms. If your application is working hard and reaches its quota, its vCPUs will be suspended for the remainder of the 80ms period.
+We enforce limits through the [Linux scheduler's CPU bandwidth control](https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.rst) by adjusting the `cpu.cfs_quota_us` setting on each machine cgroup. For each 80ms period of time, we set a quota of 5ms for each `shared` vCPU or 50ms for each `performance` vCPU. If your machine cgroup's CPU usage reaches its quota, its tasks will be 'throttled' (not scheduled to run again) for the remainder of the 80ms period.
 
 Quotas are shared between a machine's vCPUs. For example, a `shared-cpu-2x` machine is allowed to run for 10ms per 80ms period, regardless of which vCPU is using that time.
 
-<sup>1</sup> We might change these specific numbers if we feel like it.
+<sup>1</sup> We might change these specific numbers over time.
 
 ## Bursting
 
-APIs and human-facing web applications are sensitive to latency and a 75ms pause in program execution is often unacceptable. These same types of applications often work hard in small bursts and remain idle much of the time. To avoid unfairly suspending the execution of vCPUs in these applications, we allow a balance of unused vCPU time to be accrued. The application is then allowed to spend its balance in bursts. When bursting, the vCPU is allowed to run at up to 100%. When the balance is depleted, the vCPU is limited to running at its baseline quota.
+APIs and human-facing web applications are sensitive to latency and a 75ms pause in program execution is often unacceptable. These same types of applications often work hard in small bursts and remain idle much of the time. To avoid unfairly throttling the execution of vCPUs in these applications, we allow a balance of unused CPU time to be accrued. The application is then allowed to spend its balance in bursts. When bursting, the vCPU is allowed to run at up to 100%. When the balance is depleted, the vCPU is limited to running at its baseline quota.
+
+Because the burst balance is only used for utilization above the baseline quota, the amount of time a machine can _actually_ run at 100% will be longer than the accumulated balance. For example, a `shared` CPU with a burst balance of 500 seconds can burst for `500/(1-.0625)` = 533 seconds. This adjustment is already applied to the burst balance displayed in the dashboard panel.
 
 ## Monitoring
 
-The easiest way to see your CPU utilization, baseline quota, and throttling is on your app's [Managed Grafana](/docs/monitoring/metrics/#managed-grafana) `Fly Instance` dashboard.
+We publish a number of [Instance Load and CPU](/docs/monitoring/metrics/#instance-load-and-cpu) platform metrics you can use to monitor quota and throttling behavior of your Machines. The easiest way to visualize these metrics is on the [CPU Quota Balance and Throttling](https://fly-metrics.net/d/fly-instance/fly-instance?viewPanel=69) panel, on the [Fly Instance](https://fly-metrics.net/d/fly-instance/fly-instance) dashboard in [Managed Grafana](/docs/monitoring/metrics/#managed-grafana):
 
 ![chart showing CPU utilization, steal, baseline, and throttling](/docs/images/cpu-quota.webp)
 
-Here, we can see a machine that was running well bellow it's baseline quota. It had accumulated a 50s/vCPU runtime balance. Then, during a burst of activity, CPU utilization exceeded the baseline quota, causing the balance to drain. When the balance reached 0, the machine was briefly throttled. When CPU utilization went down, throttling was disabled and the balance accumulated again.
+On the dashboard, utilization (0-100%) is displayed as a per-vCPU average, divided by the total number of vCPUs in the instance. (There is also a Per-CPU Utilization panel for a detailed breakdown of utilization across each individual vCPU, but because quotas are shared between a machine's vCPUs only the average matters to the enforced limits.)
+
+Here, we can see a `performance` machine whose `utilization` was running well below its `baseline` quota of 65%, and had accumulated a 50-second burst `balance`. Then, during a burst of activity, CPU utilization exceeded the baseline quota, causing the balance to drain. When the balance reached 0, the machine was briefly `throttle`d. When CPU utilization went down, throttling ended and the balance accumulated again.
 
-A related and somewhat misleading metric is CPU steal. You can see this under the `mode=steal` label in the `fly_instance_cpu` metric. Steal is the amount of time your vCPUs are wanting to run, but our scheduler isn't allowing them to. This can happen due to throttling when your machine exceeds its quota, but it can also be a sign that other machines on the same host are competing for resources. We publish a separate `fly_instance_cpu_throttle` that only includes time your vCPUs were throttled for exceeding quota.
+A related and somewhat misleading metric is CPU steal. You can see this under the `mode=steal` label in the `fly_instance_cpu` metric. Steal is the amount of time your vCPUs are waiting to run, but the scheduler isn't allowing them to. This can happen when your machine exceeds its quota and is throttled, but it can also be a sign that other machines on the same host are competing for resources. In our dashboard panels, we visualize `steal` separately from other CPU utilization since it doesn't consume any CPU quota. We publish a separate `fly_instance_cpu_throttle` metric that only includes time your vCPUs were throttled for exceeding quota.

monitoring/metrics.html.md

@@ -217,13 +217,17 @@ fly_instance_memory_vmalloc_chunk
 and counts the amount of time each CPU (`cpu_id`) has spent performing different kinds of work (`mode`, which may be one of `user`, `nice`, `system`, `idle`, `iowait`, `irq`, `softirq`, `steal`, `guest`, `guest_nice`).
 The time unit is 'clock ticks' of centiseconds (0.01 seconds).
 
-- `cpu_baseline` is the baseline quota for CPU usage across all machine vCPUs. Learn more [here](/docs/machines/cpu-performance).
-
-- `cpu_balance` the the accrued balance of unused baseline CPU quota across all machine vCPUs. Learn more [here](/docs/machines/cpu-performance/).
+The following CPU metrics are related to [CPU Performance](/docs/machines/cpu-performance):
+- `cpu_baseline` is the baseline quota in number of CPUs, calculated from the CPU type and number of vCPUs.
+- `cpu_balance` is the accrued CPU burst balance in clock ticks (centiseconds).
+- `cpu_throttle` is derived from the `throttled_time` field of the cgroup [`cpu.stat`](https://docs.kernel.org/scheduler/sched-bwc.html#statistics), and counts the amount of time the CPU was throttled after exhausting its quota, in 'clock ticks' (centiseconds).
 
 ```
 fly_instance_load_average{minutes}
 fly_instance_cpu{cpu_id, mode} (Counter, centiseconds)
+fly_instance_cpu_baseline (CPUs)
+fly_instance_cpu_balance (centiseconds)
+fly_instance_cpu_throttle (Counter, centiseconds)
 ```
 #### Instance Disks - `fly_instance_disk_`