NC | NSFS | Wrap with try-catch Prometheus Reporting start_server

[shirady]

Explain the changes

1. The function clusterMetrics might throw an error (with an error message Operation timed out.), in case it fails, there is no try-catch block, and will end with uncaughtException. To avoid that we wrap it with try-catch block, log an error message and return from the function (we can't proceed with undefined metrics variable). Also added a json response with the details of the error in case it happens.
2. I noticed that the call prom_reporting.start_server(metrics_port, true); in fork_utils.js did not have await and added it - had to change the function signature of start_workers to have async and update its JSDoc @returns part, and also separate the call and the condition in endpoint.js.
3. Add a comment in the endpoint main for developers regarding implementation (running on the main process, running with multiple forks).
4. Change the handing under the function gather_metrics so we will print the json error object that we returned from the server.
5. Change the printing of "_create_nsfs_report: nsfs_report" so we can see the object (we used to see: "core.server.analytic_services.prometheus_reporting:: _create_nsfs_report: nsfs_report [object Object]").
6. Rename a function (we had a minor typo) from nsfs_io_state_handler to nsfs_io_stats_handler.
7. Change the order inside the function start_server in prometheus_reporting.js and set the if (req.url === '/metrics/nsfs_stats') { because it doesn't use metrics.

Issues:

1. As a part of the investigation issue DFBUGS-907 in version 5.15 we saw that the noobaa service exited and restarted with a PANIC error:

[ERROR] CONSOLE:: PANIC: process uncaughtException Error: Operation timed out. at Timeout._onTimeout
(/usr/local/noobaa-core/node_modules/prom-client/lib/cluster.js:59:18) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7)

The node is configured with forks. the error comes from cluster.js line 59: const err = new Error('Operation timed out.'); which is part of function clusterMetrics. clusterMetrics is called only in prometheus_reporting.js in start_server if the condition fork_enabled is true. prom_reporting.start_server is called from the endpoint with fork_enabled false (so it's not relevant), and in the fork_utils.js with fork_enabled true.

GAP - I'm not sure in which cases a node might face this PANIC error (before this suggested change) - this was the first time I saw that I already run long tests on GPFS machine with forks... we have opened an issue in prom-client (see comment).

Testing Instructions:

To make sure that we didn't harm the metrics in NC please run:

1. sudo node src/cmd/nsfs --debug 5 --forks 2 start the NSFS server with multiple forks.
2. sudo node src/cmd/manage_nsfs diagnose metrics see that you get an output (includes 'MetricsStatus', see example here).
3. The error was due to timeout between the forks, to see the thrown error you can do one of the options:
   (1) (quick) we can add an error Throw new Error (SDSD error) after metrics = await aggregatorRegistry.clusterMetrics(); and restart the server (ctrl + c and rerun it).
   (2) (demonstrates the timeout) add a sync wait in the code that would create a delay longer than 5 milliseconds: in fork_utils under nsfs_io_stats_handler add this part:

    const ms = 6000;
    const start = Date.now();
    let now = start;
    while (now - start < ms) {
        now = Date.now();
    }

and restart the server (ctrl + c and rerun it).
4. Fetch the response with sudo node src/cmd/manage_nsfs diagnose metricsand also from another tab withcurl -s http://127.0.0.1:7004/metrics/nsfs_stats | jq .`, expect to see:

{
"error": "Internal server error - timeout",
"message": "Looks like the server is taking a long time to respond (Could not get the metrics)"
}

• Doc added/updated
• Tests added