[Discussion] What to put in the status server ?

[hzeller]

In order to have a side-channel to communicate with the machine, a status server has been added recently, but at this point it is merely an experimental stub.

There can be multiple types of status services, one could be a fully fledged webserver that allows to interact with the machine, but having the simplicity of a very simple request/response socket service would allow for very easy interaction with some UI (which can be an LCD on the machine or something else).

Right now, the interaction is super simple: a single letter in, some JSON formatted variables back

p
{"x_axis":100.000, "y_axis":122.000, "z_axis":0.000, "note":"experimental"}

The request should probably be some json request, to be able to convey parameters easily. Or we go with query parameters similar to HTTP GET requests (?q=some_query). Output as JSON is probably an ok output though somewhat harder to parse than a token-delimited format; it is supported by many languages out of the box and allows flexible extension of what we send.

Anyway, let's discuss what we actually want and what should be provided, so that we then can design what would be a good request/response model. I think it should allow to have extensive support to read the current status of the machine. And limited ways to change parameters without messing with the current running GCode too much:

• Query options: dump positions and variables, somewhat related to this issue. In case of a 3D printer, would also output the temperatures etc.
• Limited capability to modify the current machine that does not mess with positioning, e.g. changing speed factors or acceleration parameters.
• Issuing pause and emergency stops
• Act as an input channel for other machine pauses (e.g. tool change etc.)
• Subscribing to variable changes (added in a comment below by @holla2040)

What else ? Hartley, you have added some functionality to your local status server, what did you add ?

[holla2040]

You may want to also think about direct streaming of subscribed variables, where you subscribe to a variable with delta variance and timeout. Delta variances means that if a variable changes beyond the delta, the server sends an update message. Timeout happens when subscribed variables haven't changed in a while but you want periodic update messages. This is a fairly common approach in the IoT space, eliminating high rate polling for dashboards. The simplest text protocol for streaming is stomp, then mqtt. If you really want to be cutting edge, look into gRPC, which has language agnostic serial data marshalling. Then there's always just REST, but then your back to hammering the server with polling. Let me know what you think.

[holla2040]

One more thing this evening. One reason grbl has been so successful is that's extremely simple (primarily limited by mega328p). I suggest you keep beagleg as similarly as simple as possible. Move webserver, websockets, rpc, dashboards etc away from beagleg but provide a high-speed interconnect between 2 components.I'm sure you already are going down this route.

Since running on a single board computer, you might want to skip all this inter-process communication, marshalling, etc and just do shared memory, where beagleg would expose its variable table as a read-only memory block. I'll think about this a little more. Here's a really sample of C->py shm
https://github.com/martinohanlon/c_python_ipc

[hzeller]

Yes, simplicity is the goal. Here, we should find all the relevant features that would be needed for such an interface (e.g. the idea of subscribing to changes that you described above).The actual implementation should not be part of the core (which rather then provides the programmatic interfaces needed), but the result of plug-ins or IPC-separated processes.

Directly shared memory can be a problem with thread-safety, but some super-simple, low-overhead serialization is probably what I would go for, e.g. Protocol Buffers or even Cap'n Proto (gRPC is based on protocol buffers.)

Anyway, what more features can you think of, paging @lromor @bigguiness

[lromor]

Hi,
it would e nice to add the "gcode instruction currently executed" in the stream.
In this way it will possible to recap from a certain instruction after a stop (since mapping back the last position to the gcode line can have multiple values)

We could provide all these servers as plugins easily loaded in the event server, and support different plugins for the different tastes (ie grpc, bare udp socket, tcp sockets, bare contacts, etc..)

[holla2040]

I think a catch all structure would be great

coordinates of axises configured in config file (not just x,y, and z)
machine coordinates of axises configured in config file (not just x,y, and z)

these taken from grbl's parser state command, beagleg probably has different gcodes
Motion Mode G0, G1, G2, G3, G38.2, G38.3, G38.4, G38.5, G80
Coordinate System Select G54, G55, G56, G57, G58, G59
Plane Select G17, G18, G19
Distance Mode G90, G91
Arc IJK Distance Mode G91.1
Feed Rate Mode G93, G94
Units Mode G20, G21
Cutter Radius Compensation G40
Tool Length Offset G43.1, G49
Program Mode M0, M1, M2, M30
Spindle State M3, M4, M5
Coolant State M7, M8, M9

current selected tool
current feed rate
current spindle speed

soft limit status
hard limit status

machine state (grbl states here Idle, Run, Hold, Jog, Alarm, Door, Check, Home, Sleep)

anything I missed? I'm sure there is.

I think setting a variable to broadcast timeout for this status in milliseconds. 100 would be 10 updates per seconds.

Of course, this catch all status could be broken up into multiple status forms, each with its own timeout. So things like modes aren't sent at a high refresh rate. setting variable timeout to 0 disables broadcast. each status group has a single char request like your current 'p' char.

I also agree, shared memory is not a good approach. leaning towards streaming protobuf or json, prefer json even with 2x-ish byte overhead because debugging simplicity.

Hope this makes sense.

[holla2040]

here's rapidjson impl, status.cpp

#include "rapidjson/document.h"     // rapidjson's DOM-style API
#include "rapidjson/prettywriter.h" // for stringify JSON
#include <cstdio>

using namespace rapidjson;
using namespace std;

int main(int, char*[]) {
    rapidjson::Document status;
    rapidjson::Document::AllocatorType& allocator = status.GetAllocator();

    status.SetObject();

    status.AddMember("x",1.11111,allocator);
    status.AddMember("y",2.22222,allocator);
    status.AddMember("z",3.33333,allocator);
    status.AddMember("a",4.44444,allocator);

    status.AddMember("mx",10.1010,allocator);
    status.AddMember("my",20.2020,allocator);
    status.AddMember("mz",30.3030,allocator);
    status.AddMember("ma",40.4040,allocator);

    status.AddMember("motion","G1",allocator);
    status.AddMember("coordinate","G55",allocator);
    status.AddMember("plane","G17",allocator);
    status.AddMember("distance","G90",allocator);
    status.AddMember("feedrate","G93",allocator);
    status.AddMember("units","G21",allocator);
    status.AddMember("program","M1",allocator);
    status.AddMember("spindle","M3",allocator);
    status.AddMember("coolant","M8",allocator);

    status.AddMember("tool","T1",allocator);
    status.AddMember("feed","F100.0",allocator);
    status.AddMember("spindlespeed","1000",allocator);

    
    status.AddMember("softlimit","x",allocator);
    status.AddMember("hardlimit","",allocator);

    status.AddMember("state","run",allocator);

    StringBuffer sb;
    PrettyWriter<StringBuffer> writer(sb);
    status.Accept(writer);    // Accept() traverses the DOM and generates Handler events.
    puts(sb.GetString());

    return 0;
}

root@bb[580]: g++ status.cpp -o status
root@bb[581]: time ./status
{
"x": 1.11111,
"y": 2.22222,
"z": 3.33333,
"a": 4.44444,
"mx": 10.101,
"my": 20.202,
"mz": 30.303,
"ma": 40.404,
"motion": "G1",
"coordinate": "G55",
"plane": "G17",
"distance": "G90",
"feedrate": "G93",
"units": "G21",
"program": "M1",
"spindle": "M3",
"coolant": "M8",
"tool": "T1",
"feed": "F100.0",
"spindlespeed": "1000",
"softlimit": "x",
"hardlimit": "",
"state": "run"
}

real 0m0.021s
user 0m0.008s
sys 0m0.008s

for json generation, using rapidjson or other json lib might be overkill where sprintf is simple. beagleg isn't accepting json just sending it.

[holla2040]

cout version is slower (doing some extra divides)

root@bb[650]: cat status_cout.cpp

#include <iostream>
#include <iomanip>

int main() {


    std::cout << std::setprecision(9);

    std::cout << "{" << std::endl;

    std::cout << "  \"x\":"               << 1/11.0 << "," << std::endl;
    std::cout << "  \"y\":"               << 2/11.0 << "," << std::endl;
    std::cout << "  \"z\":"               << 3/11.0 << "," << std::endl;
    std::cout << "  \"a\":"               << 4/11.0 << "," << std::endl;

    std::cout << "  \"mx\":"              << 100/11.0 << "," << std::endl;
    std::cout << "  \"my\":"              << 200/11.0 << "," << std::endl;
    std::cout << "  \"mz\":"              << 300/11.0 << "," << std::endl;
    std::cout << "  \"ma\":"              << 400/11.0 << "," << std::endl;

    std::cout << "  \"motion\":"          << "\"G1\"" << "," << std::endl;
    std::cout << "  \"coordinate\":"      << "\"G55\"" << "," << std::endl;
    std::cout << "  \"plane\":"           << "\"G17\"" << "," << std::endl;
    std::cout << "  \"distance\":"        << "\"G90\"" << "," << std::endl;
    std::cout << "  \"feedrate\":"        << "\"G93\"" << "," << std::endl;
    std::cout << "  \"units\":"           << "\"G2\"" << "," << std::endl;
    std::cout << "  \"program\":"         << "\"M1\"" << "," << std::endl;
    std::cout << "  \"spindle\":"         << "\"M3\"" << "," << std::endl;
    std::cout << "  \"coolant\":"         << "\"M8\"" << "," << std::endl;

    std::cout << "  \"tool\":"            << "\"T1\"" << "," << std::endl;
    std::cout << "  \"feed\":"            << "\"F100.0\"" << "," << std::endl;
    std::cout << "  \"spindlespeed\":"    << 1000 << "," << std::endl;

    
    std::cout << "  \"softlimit\":"       << "\"x\"" << "," << std::endl;
    std::cout << "  \"hardlimit\":"       << "\"\"" << "," << std::endl;

    std::cout << "  \"state\":"           << "\"run\"" << std::endl;

    std::cout << "}" << std::endl;


    return 0;
}

root@bb[651]: g++ status_cout.cpp -o status_cout
root@bb[652]: time ./status_cout
{
"x":0.0909090909,
"y":0.181818182,
"z":0.272727273,
"a":0.363636364,
"mx":9.09090909,
"my":18.1818182,
"mz":27.2727273,
"ma":36.3636364,
"motion":"G1",
"coordinate":"G55",
"plane":"G17",
"distance":"G90",
"feedrate":"G93",
"units":"G2",
"program":"M1",
"spindle":"M3",
"coolant":"M8",
"tool":"T1",
"feed":"F100.0",
"spindlespeed":1000,
"softlimit":"x",
"hardlimit":"",
"state":"run"
}

real 0m0.026s
user 0m0.008s
sys 0m0.004s

[hzeller]

I like the relatively simple structure of having everything in one rcord that is sent regularly.
We can have different timeouts or also have a mode in which we only send the full state once on connect, and then only differences; so e.g. if only X changed it would be {"X":"12.3456"}; configurable so that it maybe sends a full status every n-th time or so (a 'keyframe').

Thanks for your experiments. Looks like the overhead for cout is not soo much; if we can get some result with on-board means for relatively low overhead we should avoid external dependencies; so cout or printf() with a large formatting string are probably ok initially.

[hzeller]

The 'only send something when values change'-mode would be in particular interesting for machine gcode variables - there are a couple of thousand of them so only sending changes is very useful :).
We can have a subsection

  "variables" : {
    "#5220" : 1.0000
   "#text_variable" : 42.0000
}

That subsection contains all variables that are set at first connect, but then only variables that changed since last time.

We might send changes immediately when they occur, but rate-limit and don't send more than e.g. 10/second.

[hzeller]

Network stack: I am in favor of just a simple socket , but http or websockets might be more useful in certain context (BUT, we'd add more dependencies on external libraries). So maybe simple socket, but then people can add translation proxies as they wish ?

[holla2040]

onChange, rating limiting, variables, keyframes over simple socket, all this is great and couldn't be simpler.

[holla2040]

here's the last timing, printf

#include <cstdio>

int main() {

    printf("{\n  "
            "\"x\":%.6f,"
            "\"y\":%.6f,"
            "\"z\":%.6f,"
            "\"a\":%.6f,\n"
            "  \"mx\":%.6f,"
            "\"my\":%.6f,"
            "\"mz\":%.6f,"
            "\"ma\":%.6f,\n"
            "  \"motion\":\"%s\","
            "\"coordinate\":\"%s\","
            "\"plane\":\"%s\","
            "\"distance\":\"%s\","
            "\"feedrate\":\"%s\","
            "\"units\":\"%s\","
            "\"program\":\"%s\","
            "\"spindle\":\"%s\","
            "\"coolant\":\"%s\",\n"
            "  \"tool\":\"%s\","
            "\"feed\":\"%s\","
            "\"spindlespeed\":%d,"
            "\"softlimit\":\"%s\","
            "\"hardlimit\":\"%s\","
            "\"states\":\"%s\"\n}\n",
            1/11.0,2/11.0,3/11.0,4/11.0,
            100/11.0,200/11.0,300/11.0,400/11.0,
            "G1","G55","G17","G90","G93","G2","M1","M3","M8",
            "T1","F100.0",1000,"x","","run"
    );

    return 0;
}

root@bb[535]: g++ status_printf.cpp -o status_printf
root@bb[536]: time ./status_printf
{
"x":0.090909,"y":0.181818,"z":0.272727,"a":0.363636,
"mx":9.090909,"my":18.181818,"mz":27.272727,"ma":36.363636,
"motion":"G1","coordinate":"G55","plane":"G17","distance":"G90","feedrate":"G93","units":"G2","program":"M1","spindle":"M3","coolant":"M8",
"tool":"T1","feed":"F100.0","spindlespeed":1000,"softlimit":"x","hardlimit":"","state":"run"
}

real 0m0.019s
user 0m0.008s
sys 0m0.008s