Create JSON decoder

[papigers]

This is a feature request to add a JSON decoder to allow exporting metrics to other data sources that expect a more structured way of organizing the data - like fluent output plugins.
(My specific use case is to support ingesting metrics into the new fluent-bit output plugin I wrote, but I thought it would be better to "enhance" it globally, here).

The format I suggest is inspired by the format of prom2json:

[
  {
      "name": "<name>",
      "description": "<description>",
      "type": "<type>",
      "labels": {
         "<label1>": "<label1_value>",
         "<label2>": "<label2_value>",
         ...
         "<labeln>": "<labeln_value>",
      },
     "timestamp":  "<timestamp>"
      "value": "67542.292",
      /* histogram only */
      "buckets": {
        "bucket1": "67542.292",
        "bucket2": "23902.678",
        ...
        "bucketn": "6865.718"
      },
      /* summary only */
      "quantiles": {
        "qunatile1": "67542.292",
        "qunatile2": "23902.678",
        ...
        "qunatilen": "6865.718"
      },
      "count": "743",
      "sum": "6936936.447000001"
  },
  ...
]

Notice that with this format, every metric sample would be a JSON record.
An alternative way, that is less optimal for my use case, but may be better for others, is to group samples by the metric (similar to how the metrics are represented in msgpack, and in prom2json)

I'd be happy to help and submit a PR, once you approve the specification.

[edsiper]

thanks for raising this topic. I am wondering what kind of other considerations we should have when talking about a new JSON schema for metrics.. maybe an expanded version of our msgpack format ?

are we aware of another apps that ships metrics in JSON by using a specific format ?

cc: @leonardo-albertovich

[leonardo-albertovich]

We should check OTLP since they added JSON as an alternative format for logs they might have done the same thing for metrics and in that case I think we'd probably want to make ours compatible with the standard.

[leonardo-albertovich]

I think we should use this and this to define our schema.

There is at least one node package to do that unless I misundertood the description (proto3-json-serializer)

[edsiper]

I am still thinking, do we need a JSON schema ?, I mean, if some client app wants to compose a compatible metrics format with CMetrics, just use cmetrics directly in C or the Golang bindings, encode as msgpack and ship it over the network

[papigers]

@leonardo-albertovich does that mean we will have to depend on the OLTP proto files here? (don't know if that's even an issue)

Found this which might be useful - https://github.com/Sannis/protobuf2json-c/blob/master/src/protobuf2json.c (last commit was in 2016 so not sure if it should actually be used, but might give an idea to approach this).

@edsiper My main motivation for this issue was to output metrics from FluentBit to a data source that doesn't support any of the existing representations. I think that JSON schema will allow metrics to be exposed & integrated to a wider selection of data sources.

[edsiper]

@papigers ah, you mean something like "metrics to logs" conversion, if that is the case, yeah makes sense. (just I would like to avoid a JSON -> CMetrics conversion for now)

[edsiper]

now we have a readable "style" in msgpack format, we don't want to add a JSON writer here in cmetrics but the functionality is ready from Fluent Bit side.

Closing as it won't be implemented here unless a specific json decoder for certain backend is added.