Skip to content

Commit

Permalink
Fix all dashboard bugs
Browse files Browse the repository at this point in the history
  • Loading branch information
@vishiy
vishiy committed Aug 25, 2022
1 parent 9b5eeeb commit ccdb0c2
Show file tree
Hide file tree
Showing 9 changed files with 20 additions and 18 deletions.
Binary file modified GeneratedMonitoringArtifacts/Default dashboards & metrics & rr lists.docx
View file
Binary file not shown.
2 changes: 2 additions & 0 deletions GeneratedMonitoringArtifacts/Default/default_dashboards_recrules_alerts.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
|**6**|Kubernetes / Kubelet|Kubelet.json|<p>1. kubelet\_node\_name</p><p>2. kubelet\_running\_pods</p><p>3. kubelet\_running\_pod\_count</p><p>4. kubelet\_running\_containers</p><p>5. kubelet\_running\_container\_count</p><p>6. volume\_manager\_total\_volumes</p><p>7. kubelet\_node\_config\_error</p><p>8. kubelet\_runtime\_operations\_total</p><p>9. kubelet\_runtime\_operations\_errors\_total</p><p>10. kubelet\_runtime\_operations\_duration\_seconds\_bucket</p><p>11. kubelet\_pod\_start\_duration\_seconds\_count</p><p>12. kubelet\_pod\_worker\_duration\_seconds\_count</p><p>13. kubelet\_pod\_worker\_duration\_seconds\_bucket</p><p>14. storage\_operation\_duration\_seconds\_count</p><p>15. storage\_operation\_errors\_total</p><p>16. kubelet\_cgroup\_manager\_duration\_seconds\_count</p><p>17. kubelet\_pleg\_relist\_duration\_seconds\_count</p><p>18. kubelet\_pleg\_relist\_interval\_seconds\_bucket</p><p>19. kubelet\_pleg\_relist\_duration\_seconds\_bucket</p><p>20. rest\_client\_requests\_total</p><p>21. rest\_client\_request\_duration\_seconds\_bucket</p><p>22. process\_resident\_memory\_bytes</p><p>23. process\_cpu\_seconds\_total</p><p>24. go\_goroutines</p><p></p>|||
|**7** |Node Exporter / USE Method / Node|Node-rsrc-use.json|<p>1. node\_filesystem\_avail\_bytes</p><p>2. node\_filesystem\_size\_bytes</p><p>3. node\_time\_seconds</p><p>4. node\_exporter\_build\_info</p>|<p>1. instance:node\_cpu\_utilisation:rate5m</p><p>2. instance:node\_load1\_per\_cpu:ratio</p><p>3. instance:node\_memory\_utilisation:ratio</p><p>4. instance:node\_vmstat\_pgmajfault:rate5m</p><p>5. instance:node\_network\_receive\_bytes\_excluding\_lo:rate5m</p><p>6. instance:node\_network\_transmit\_bytes\_excluding\_lo:rate5m</p><p>7. instance:node\_network\_receive\_drop\_excluding\_lo:rate5m</p><p>8. instance:node\_network\_transmit\_drop\_excluding\_lo:rate5m</p><p>9. instance\_device:node\_disk\_io\_time\_seconds:rate5m</p><p>10. instance\_device:node\_disk\_io\_time\_weighted\_seconds:rate5m</p><p>11. instance:node\_num\_cpu:sum</p><p></p>|<p>5. node\_cpu\_seconds\_total</p><p>6. node\_load1</p><p>7. node\_memory\_MemAvailable\_bytes</p><p>8. node\_memory\_Buffers\_bytes</p><p>9. node\_memory\_Cached\_bytes</p><p>10. node\_memory\_MemFree\_bytes</p><p>11. node\_memory\_Slab\_bytes</p><p>12. node\_memory\_MemTotal\_bytes</p><p>13. node\_vmstat\_pgmajfault</p><p>14. node\_network\_receive\_bytes\_total</p><p>15. node\_network\_transmit\_bytes\_total</p><p>16. node\_network\_receive\_drop\_total</p><p>17. node\_network\_transmit\_drop\_total</p><p>18. node\_disk\_io\_time\_seconds\_total</p><p>19. node\_disk\_io\_time\_weighted\_seconds\_total</p><p></p>|
|**8**|Node Exporter / Nodes|Nodes.json|<p>1. node\_cpu\_seconds\_total</p><p>2. node\_load1</p><p>3. node\_load5</p><p>4. node\_load15</p><p>5. node\_memory\_MemTotal\_bytes</p><p>6. node\_memory\_MemFree\_bytes</p><p>7. node\_memory\_Buffers\_bytes</p><p>8. node\_memory\_Cached\_bytes</p><p>9. node\_memory\_MemAvailable\_bytes</p><p>10. node\_disk\_read\_bytes\_total</p><p>11. node\_disk\_written\_bytes\_total</p><p>12. node\_disk\_io\_time\_seconds\_total</p><p>13. node\_filesystem\_size\_bytes</p><p>14. node\_filesystem\_avail\_bytes</p><p>15. node\_network\_receive\_bytes\_total</p><p>16. node\_network\_transmit\_bytes\_total</p><p>17. node\_uname\_info</p><p></p>|||
|**9**|Kubernetes / Compute Resources / Workload|k8s-resources-workload.json|<p>1. kube\_pod\_container\_resource\_requests</p><p>2. kube\_pod\_container\_resource\_limits</p><p>3. container\_memory\_working\_set\_bytes</p><p>4. container\_network\_receive\_bytes\_total</p><p>5. container\_network\_transmit\_bytes\_total</p><p>6. container\_network\_receive\_packets\_total</p><p>7. container\_network\_transmit\_packets\_total</p><p>8. container\_network\_receive\_packets\_dropped\_total</p><p>9. container\_network\_transmit\_packets\_dropped\_total</p><p>10. kube\_namespace\_status\_phase</p>|<p>1. node\_namespace\_pod\_container:container\_cpu\_usage\_seconds\_total:sum\_irate</p><p>2. namespace\_workload\_pod:kube\_pod\_owner:relabel</p>|<p>11. container\_cpu\_usage\_seconds\_total</p><p>12. kube\_pod\_info</p><p>13. kube\_pod\_owner</p><p>14. kube\_replicaset\_owner</p>|



Expand All @@ -33,6 +34,7 @@ Default dashboards & their keep lists
||Kubelet.json|Kubernetes / Kubelet|<p>`kubelet = kubelet\_node\_name|kubelet\_running\_pods|kubelet\_running\_pod\_count|kubelet\_running\_sum\_containers|kubelet\_running\_container\_count|volume\_manager\_total\_volumes|kubelet\_node\_config\_error|kubelet\_runtime\_operations\_total|kubelet\_runtime\_operations\_errors\_total|kubelet\_runtime\_operations\_duration\_seconds\_bucket|kubelet\_runtime\_operations\_duration\_seconds\_sum| kubelet\_runtime\_operations\_duration\_seconds\_count|kubelet\_pod\_start\_duration\_seconds\_bucket|kubelet\_pod\_start\_duration\_seconds\_sum|kubelet\_pod\_start\_duration\_seconds\_count|kubelet\_pod\_worker\_duration\_seconds\_bucket|kubelet\_pod\_worker\_duration\_seconds\_sum|kubelet\_pod\_worker\_duration\_seconds\_count|storage\_operation\_duration\_seconds\_bucket|storage\_operation\_duration\_seconds\_sum|storage\_operation\_duration\_seconds\_count|storage\_operation\_errors\_total|kubelet\_cgroup\_manager\_duration\_seconds\_bucket|kubelet\_cgroup\_manager\_duration\_seconds\_sum|kubelet\_cgroup\_manager\_duration\_seconds\_count|kubelet\_pleg\_relist\_interval\_seconds\_bucket|kubelet\_pleg\_relist\_interval\_seconds\_count|kubelet\_pleg\_relist\_interval\_seconds\_sum|kubelet\_pleg\_relist\_duration\_seconds\_bucket|kubelet\_pleg\_relist\_duration\_seconds\_count|kubelet\_pleg\_relist\_duration\_seconds\_sum|rest\_client\_requests\_total|rest\_client\_request\_duration\_seconds\_bucket|rest\_client\_request\_duration\_seconds\_sum|rest\_client\_request\_duration\_seconds\_count|process\_resident\_memory\_bytes|process\_cpu\_seconds\_total|go\_goroutines`</p><p></p>|
||<p>Node-rsrc-use.json</p><p></p>|Node Exporter / USE Method / Node|<p>`nodeexporter = node\_filesystem\_avail\_bytes|node\_filesystem\_size\_bytes|node\_time\_seconds|node\_exporter\_build\_info|node\_cpu\_seconds\_total|node\_load1|node\_memory\_MemAvailable\_bytes|node\_memory\_Buffers\_bytes|node\_memory\_Cached\_bytes|node\_memory\_MemFree\_bytes|node\_memory\_Slab\_bytes|node\_memory\_MemTotal\_bytes|node\_vmstat\_pgmajfault|node\_network\_receive\_bytes\_total|node\_network\_transmit\_bytes\_total|node\_network\_receive\_drop\_total|node\_network\_transmit\_drop\_total|node\_disk\_io\_time\_seconds\_total|node\_disk\_io\_time\_weighted\_seconds\_total`</p><p></p>|
||Nodes.json|Node Exporter / Nodes|<p>`nodeexporter = node\_cpu\_seconds\_total|node\_load1|node\_load5|node\_load15|node\_memory\_MemTotal\_bytes|node\_memory\_MemFree\_bytes|node\_memory\_Buffers\_bytes|node\_memory\_Cached\_bytes|node\_memory\_MemAvailable\_bytes|node\_disk\_read\_bytes\_total|node\_disk\_written\_bytes\_total|node\_disk\_io\_time\_seconds\_total|node\_filesystem\_size\_bytes|node\_filesystem\_avail\_bytes|node\_network\_receive\_bytes\_total|node\_network\_transmit\_bytes\_total|node\_uname\_info`</p><p></p>|
||k8s-resources-workload.json|Kubernetes / Compute Resources / Workload|<p>`cadvisor = container\_cpu\_usage\_seconds\_total|container\_memory\_working\_set\_bytes|container\_network\_receive\_bytes\_total|container\_network\_transmit\_bytes\_total|container\_network\_receive\_packets\_total|container\_network\_transmit\_packets\_total|container\_network\_receive\_packets\_dropped\_total|container\_network\_transmit\_packets\_dropped\_total</p><p>kubestate = kube\_pod\_container\_resource\_requests|kube\_pod\_container\_resource\_limits|kube\_namespace\_status\_phase|kube\_pod\_info|kube\_pod\_owner|kube\_replicaset\_owner`</p><p></p>|


**Default alerting rules and their metrics :**
Expand Down
10 changes: 5 additions & 5 deletions mixins/kubernetes/dashboards/resources/cluster.libsonnet
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,19 +20,19 @@ local template = grafana.template;
local tableStyles = {
namespace: {
alias: 'Namespace',
link: '%(prefix)s/d/%(uid)s/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-namespace.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-namespace.json'] },
linkTooltip: 'Drill down to pods',
},
'Value #A': {
alias: 'Pods',
linkTooltip: 'Drill down to pods',
link: '%(prefix)s/d/%(uid)s/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-namespace.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-namespace.json'] },
decimals: 0,
},
'Value #B': {
alias: 'Workloads',
linkTooltip: 'Drill down to workloads',
link: '%(prefix)s/d/%(uid)s/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-workloads-namespace.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-workloads-namespace.json'] },
decimals: 0,
},
};
Expand All @@ -55,7 +55,7 @@ local template = grafana.template;
local networkTableStyles = {
namespace: {
alias: 'Namespace',
link: '%(prefix)s/d/%(uid)s/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-namespace.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-namespace.json'] },
linkTooltip: 'Drill down to pods',
},
'Value #A': {
Expand Down Expand Up @@ -96,7 +96,7 @@ local template = grafana.template;
local storageIOTableStyles = {
namespace: {
alias: 'Namespace',
link: '%(prefix)s/d/%(uid)s/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-namespace.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-namespace.json'] },
linkTooltip: 'Drill down to pods',
},
'Value #A': {
Expand Down
2 changes: 1 addition & 1 deletion mixins/kubernetes/dashboards/resources/multi-cluster.libsonnet
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ local g = import 'github.com/grafana/jsonnet-libs/grafana-builder/grafana.libson
local tableStyles = {
[$._config.clusterLabel]: {
alias: 'Cluster',
link: '%(prefix)s/d/%(uid)s/k8s-resources-cluster?var-datasource=$datasource&var-cluster=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-cluster.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-cluster?var-datasource=$datasource&var-cluster=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-cluster.json'] },
},
};

Expand Down
6 changes: 3 additions & 3 deletions mixins/kubernetes/dashboards/resources/namespace.libsonnet
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ local template = grafana.template;
local tableStyles = {
pod: {
alias: 'Pod',
link: '%(prefix)s/d/%(uid)s/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-pod.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-pod.json'] },
},
};

Expand All @@ -48,7 +48,7 @@ local template = grafana.template;
local networkTableStyles = {
pod: {
alias: 'Pod',
link: '%(prefix)s/d/%(uid)s/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-pod.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-pod.json'] },
linkTooltip: 'Drill down to pods',
},
'Value #A': {
Expand Down Expand Up @@ -97,7 +97,7 @@ local template = grafana.template;
local storageIOTableStyles = {
pod: {
alias: 'Pod',
link: '%(prefix)s/d/%(uid)s/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-pod.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-pod.json'] },
linkTooltip: 'Drill down to pods',
},
'Value #A': {
Expand Down
4 changes: 2 additions & 2 deletions mixins/kubernetes/dashboards/resources/workload-namespace.libsonnet
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ local template = grafana.template;
local tableStyles = {
workload: {
alias: 'Workload',
link: '%(prefix)s/d/%(uid)s/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-workload.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-workload.json'] },
},
workload_type: {
alias: 'Workload Type',
Expand Down Expand Up @@ -95,7 +95,7 @@ local template = grafana.template;
local networkTableStyles = {
workload: {
alias: 'Workload',
link: '%(prefix)s/d/%(uid)s/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$type' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-workload.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$type' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-workload.json'] },
linkTooltip: 'Drill down to pods',
},
workload_type: {
Expand Down
4 changes: 2 additions & 2 deletions mixins/kubernetes/dashboards/resources/workload.libsonnet
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ local template = grafana.template;
local tableStyles = {
pod: {
alias: 'Pod',
link: '%(prefix)s/d/%(uid)s/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-pod.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-pod.json'] },
},
};

Expand Down Expand Up @@ -96,7 +96,7 @@ local template = grafana.template;
local networkTableStyles = {
pod: {
alias: 'Pod',
link: '%(prefix)s/d/%(uid)s/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-pod.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-pod.json'] },
},
'Value #A': {
alias: 'Current Receive Bandwidth',
Expand Down
6 changes: 3 additions & 3 deletions mixins/kubernetes/dashboards/windows.libsonnet
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ local g = import 'github.com/grafana/jsonnet-libs/grafana-builder/grafana.libson
local tableStyles = {
namespace: {
alias: 'Namespace',
link: '%(prefix)s/d/%(uid)s/k8s-resources-windows-namespace?var-datasource=$datasource&var-namespace=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-windows-namespace.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-windows-namespace?var-datasource=$datasource&var-namespace=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-windows-namespace.json'] },
},
};

Expand Down Expand Up @@ -140,7 +140,7 @@ local g = import 'github.com/grafana/jsonnet-libs/grafana-builder/grafana.libson
local tableStyles = {
pod: {
alias: 'Pod',
link: '%(prefix)s/d/%(uid)s/k8s-resources-windows-pod?var-datasource=$datasource&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-resources-windows-pod.json') },
link: '%(prefix)s/d/%(uid)s/k8s-resources-windows-pod?var-datasource=$datasource&var-namespace=$namespace&var-pod=$__cell' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-resources-windows-pod.json'] },
},
};

Expand Down Expand Up @@ -380,7 +380,7 @@ local g = import 'github.com/grafana/jsonnet-libs/grafana-builder/grafana.libson
),

'k8s-windows-cluster-rsrc-use.json':
local legendLink = '%(prefix)s/d/%(uid)s/k8s-windows-node-rsrc-use' % { prefix: $._config.grafanaK8s.linkPrefix, uid: std.md5('k8s-windows-node-rsrc-use.json') };
local legendLink = '%(prefix)s/d/%(uid)s/k8s-windows-node-rsrc-use' % { prefix: $._config.grafanaK8s.linkPrefix, uid: $._config.grafanaDashboardIDs['k8s-windows-node-rsrc-use.json'] };

dashboard.new(
'%(dashboardNamePrefix)sUSE Method / Cluster(Windows)' % $._config.grafanaK8s,
Expand Down
Loading

0 comments on commit ccdb0c2

Please sign in to comment.