Add metrics output

[jspaleta]

• Make search configuration optional, allows generating metrics for all processes
• Convert alert output summary text into prom comment strings
• Add procstat metric family output for Sumo Logic dashboard compatibility
• Add processes metric family output for Sumo Logic dashboard compatibility

[calebhailey]

I think we might need to bump the version in go.mod to 1.17... otherwise I was getting a compile error on UnixMilli

$ go build                                                                                                                      
# github.com/sensu/sensu-processes-check
./metrics.go:24:21: time.Now().UnixMilli undefined (type time.Time has no field or method UnixMilli)
$ go version
go version go1.16.3 linux/amd64

[calebhailey]

I think we should drop the hard-coded host.name tag. I know it's there for dashboard compat, but we should use output_metric_tags for that. We should try to keep any built-in tags as minimal as possible.

Same for the search_string tag. Although you couldn't add this tag via output_metric_tags when searching for multiple processes, it seems likely that it might often be the same as process.executable.name.

[calebhailey]

Encountered an error when configured with output_metrics_format: prometheus_text:

{
  "check": "process-monitoring",
  "component": "agent",
  "error": "unable to extract metric from check output",
  "level": "error",
  "msg": "text format parsing error in line 3: expected '=' after label name, found '.'",
  "namespace": "default",
  "time": "2022-05-12T01:07:02Z"
}

From the Prometheus docs:

Label names may contain ASCII letters, numbers, as well as underscores. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*. Label names beginning with __ are reserved for internal use.

We'll need to change the following labels:

• host.name: let's remove this (this or an alternative may be added via output_metric_tags)
• process.executable.name: rename to process_executable_name or process_name or name?
• process.executable.pid: rename to process_executable_pid or process_pid or pid?