feat: metrics dumps & extended metrics #2519

Arqu · 2024-07-17T21:44:30Z

Description

Extends several pieces:

introduces a "metric dumper" which is just a way of saying you can sample the internal metrics and write them to a CSV file (which should come in handy for 3 pieces that should come down the line; 1) CI using these to validate behavior, 2) debug dumps from 3rd parties, 3) local debugging)
along with the dumper the doctor plot has been extended to be able to read those dumps and also just generally improved some rough edges so it's less error prone.
node counts are here for relays. You now have a derived metric which simply counts unique daily node connections.

Sample usage for the metrics dumper:
cargo run --bin iroh --all-features -- --metrics-dump-path test.metrics.csv start

Sample of the plotter:
cargo run --bin iroh --all-features -- doctor plot --timeframe 30 --interval 10 --file metrics.dump.csv magicsock_actor_tick_main_total,magicsock_actor_tick_msg_total,magicsock_actor_tick_endpoint_heartbeat_total,magicsock_actor_tick_endpoints_update_receiver_total,magicsock_actor_tick_re_stun_total

Breaking Changes

Notes & open questions

This is merging into #2464 as part of the larger metrics refactor.

Change checklist

Self-review.
Documentation updates following the style guide, if relevant.
Tests if relevant.
All breaking changes documented.

dignifiedquire · 2024-07-18T09:49:21Z

ci is very sad

iroh-metrics/src/service.rs

iroh-net/src/relay/server.rs

dignifiedquire · 2024-07-18T09:55:32Z

iroh-net/src/relay/server.rs

+        match new_conn {
+            true => 1,
+            false => 0,
+        }


new_conn as u64

I generally dislike that as I never feel I can trust that implicit conversion from bool to int in regards to compiler/platforms.

flub · 2024-07-18T12:27:48Z

iroh-cli/src/commands.rs

@@ -41,6 +41,10 @@ pub(crate) struct Cli {
    /// Port to serve metrics on. Disabled by default.
    #[clap(long)]
    pub(crate) metrics_port: Option<MetricsPort>,
+
+    /// If set, metrics will be dumped in CSV format to the specified path at regular intervals (100ms).


isn't 100ms very frequent?

Yes, it's intentional for us to have very high resolution in this setup, you can use the regular scripts and scrape to grafana if you just want general stats over time. This is meant as a high res debug tool.

iroh-cli/src/commands/start.rs

iroh-metrics/src/service.rs

dignifiedquire · 2024-07-22T10:50:34Z

iroh-metrics/src/service.rs

+    if let Err(e) = m {
+        error!("Failed to encode metrics: {e:#}");
+    } else {
+        let m = m.unwrap();


use match statement to avoid unwrap?

## Description Extends several pieces: - introduces a "metric dumper" which is just a way of saying you can sample the internal metrics and write them to a CSV file (which should come in handy for 3 pieces that should come down the line; 1) CI using these to validate behavior, 2) debug dumps from 3rd parties, 3) local debugging) - along with the dumper the `doctor plot` has been extended to be able to read those dumps and also just generally improved some rough edges so it's less error prone. - node counts are here for relays. You now have a derived metric which simply counts unique daily node connections. Sample usage for the metrics dumper: `cargo run --bin iroh --all-features -- --metrics-dump-path test.metrics.csv start` Sample of the plotter: `cargo run --bin iroh --all-features -- doctor plot --timeframe 30 --interval 10 --file metrics.dump.csv magicsock_actor_tick_main_total,magicsock_actor_tick_msg_total,magicsock_actor_tick_endpoint_heartbeat_total,magicsock_actor_tick_endpoints_update_receiver_total,magicsock_actor_tick_re_stun_total` ## Breaking Changes  ## Notes & open questions This is merging into #2464 as part of the larger metrics refactor. ## Change checklist - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented.

Arqu requested review from flub and dignifiedquire July 17, 2024 21:44

Arqu self-assigned this Jul 17, 2024

Arqu added the metrics extracting quantified mesurements from iroh label Jul 17, 2024