Aten to monitor the current node

[Zabrane]

Describe the bug

Hi All

I'm trying to use aten to monitor my Erlang cluster nodes.
The first thing I did is to try monitoring the node itlself:

Reproduction steps

$ rebar3 shell
[...]                                                                                                                                                                                                                                
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling wa8_rest
===> Compiling wa8
===> Compiling wa8_perf
Erlang/OTP 26 [erts-14.0.2] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:64] [jit:ns] [dtrace]

Eshell V14.0.2 (press Ctrl+G to abort, type help(). for help)
===> Booted woo
===> Booted sasl
===> Booted aten
(woo@127.0.0.1)1> node().
'[email protected]'
(woo@127.0.0.1)2> aten:register(node()).
ok
(woo@127.0.0.1)3> flush().
Shell got {node_event,'[email protected]',down}

Strange that I'm getting a down event, isn't it?

Expected behavior

I should get an up event in this case as the node is up and running.

Shell got {node_event,'[email protected]',up}

Additional context

No response

[lukebakken]

Please see these issues for an explanation of this behavior:

• detector: emit a "node up" event for newly registered connected nodes #22
• Consider emitting "node up" events for connected ("known") newly registered nodes #23

[Zabrane]

@lukebakken Thanks for these links.

I've read them all and also read the related paper. Even if I understand the heartbeats logic, and the time of the last received one, I think that aten should always return an up message when monitoring the node it's currently running on.
Consistency is better than following the paper which doesn't handle or even mention this corner case.

The fix should be fairly simple:

case Node == node() of
  true -> Pid ! {node_event, Node, up};
  _ -> Pid ! {node_event, Node, Evt};

Another idea is to discard registering the current node.

One more question if you don't mind: when receiving a down message, should I register the Node again with aten:register(Node). to keep monitoring it?

Many thanks

[Zabrane]

@lukebakken I also noticed in my simulation (2 nodes connected and monitored by aten) that if one node is turned off, aten sends 2 down events:

%% first node: foo@zog
(foo@zog)1> application:ensure_all_started(aten).
{ok,[sasl,crypto,aten]}
(foo@zog)2> net_adm:ping('bar@zog').
pong
(foo@zog)3> aten:register('bar@zog').
ok
(foo@zog)4> flush().
Shell got {node_event,bar@zog,up}
(foo@zog)5> flush(). <------ got 2 down events instead of 1
Shell got {node_event,bar@zog,down}
Shell got {node_event,bar@zog,down}

%% second node: bar@zog
(bar@zog)1> application:ensure_all_started(aten).
{ok,[sasl,crypto,aten]}
(bar@zog)2> net_adm:ping('foo@zog').
pong
(bar@zog)3> aten:register('foo@zog').
ok
(bar@zog)4> flush().
Shell got {node_event,foo@zog,up}
(bar@zog)5> halt(). <------ disconnected on purpose

Any reason for the 2 down events instead of 1?

I'm running Erlang 26.0.2 on macOS Sonoma but I can also test elsewhere if you'd like (Ubuntu 22.4 LTS with Erlang 26.2).

[lukebakken]

@kjnilsson may have more input here.

[kjnilsson]

It does look like this can happen when a node is terminated. I think aten detects it first as slow then as gone - hence the two notifications. I can take a look tomorrow to see if there is anything we can do to avoid it or if it just a "better safe than sorry" scenario.

[kjnilsson]

Also and adding code just to special case monitoring yourself doesn't really make sense to me. I can add a guard to disallow the current node being passed to aten:register/1 one however.

[Zabrane]

Also and adding code just to special case monitoring yourself doesn't really make sense to me. I can add a guard to disallow the current node being passed to aten:register/1 one however.

@kjnilsson this would be great. Many thanks

[Zabrane]

It does look like this can happen when a node is terminated. I think aten detects it first as slow then as gone - hence the two notifications. I can take a look tomorrow to see if there is anything we can do to avoid it or if it just a "better safe than sorry" scenario.

@kjnilsson Exactly. Receiving the same events twice means that aten's users have to handle this case carefully by adding an extra logic. Please let me know whatever you come up with so I can give a go. Thanks

[michaelklishin]

@Zabrane aten's algorithm is based on latency distribution. Monitoring local pids using aten makes no sense.

[Zabrane]

@michaelklishin what's non-sense it to leave such a behavior instead of:

1. make it feel good to the end user
2. simply prevent it as discussed above with @kjnilsson:

> net_kernel:connect_node(node()).
ignored

[lukebakken]

what's non-sense it to leave such a behavior

Many times there is good reason to leave "odd" behavior in place, one of which is that changes to code often introduce subtle bugs.

@kjnilsson has opened the following PR for you to test, which would be appreciated: #37