Gui freezes with large G-Code files

[koppi]

Gui freezes, needs to be fixed with non blocking QProcess and asynchronous event handing.

See: https://github.com/koppi/gcoder/blob/master/mainwin.cpp#L121-L123

[koppi]

With d47818d you can run:

tests/ngc-urandom.sh [N]

where N is the number of random G01 moves to create, see screenshot:

[ArcEye]

The QGCodeEditor uses the setPlainText() function, which is very inefficient for big files.

What I did in the QtMDI app was to just load xxK of the file initially and whenever the running line got anywhere near the end of the display, load another chunk.
So long as the user has the bit they are interested in displayed, they have no idea whether it is all in the viewer or not.

As there is no running process moving through the file, the trigger could just be whenever the scroll gets near the end. If the user just looks at the image and not the code, it would never read any more in.

With that done it will be easier to see how much of the delay is down to interpretation and plot display.

A precursor is probably to move away from piping of stdout and to load selected files.

If experimenting 'on-the-fly' in the bash terminal, then write that output to a file and read it from there, so that its size is known and a smaller chunk only can be read in. If it is small it will all be read in anyway

What do you think?

[koppi]

What I want is a non-blocking pipeline:

shell script | gcodeviewer | rs274 | opengl gui

In the current Code-base, QGCodeViewer emits a textChanged() signal, which triggers the g2m rs274 wrapper to start interpreting. I want to make rs274 interpret a line as soon as it gets the first line of G-Code from QGCodeViewer. Does this make sense?

[ArcEye]

I think what you need to do is fork the datastream, not pipe it.

The bottleneck is QTextEdit, which is notoriously slow. My adding line formatting to the equation, makes it even slower.

If you stream to a file and choose a different trigger, the editor and rs274 can access the data at the same time.

There is a wider issue about whether the bash interface is going to be accessible to 'non gurus' .

It is a very quick prototyping tool for writing engraving files, but is the user going to be expected to know that cat ~/machinekit/ngc/myfile.ngc in the bash terminal is how to render a plot for that file?

[koppi]

In QGCoder/libqgcodeeditor@f9827ee I just added a pipe example to QGcodeEditor. It may be helpful to measure the performance of the Qt Gui stuff, and perhaps we can find a more performant solution to display G-Code in a QWidget. Demo of the pipe example:

https://www.youtube.com/watch?v=GN9InCCZRPc

Measure runtime of the QGCodeEditor widget with a 1000 lines G-Code file:

$ time -p bash -c 'cat <(ngc-urandom.sh 1000) | ./pipe '
real 2.94
user 2.72
sys 0.10

where ngc-urandom.sh is:

#!/usr/bin/env bash
N=$1; echo "F100"; cat /dev/urandom | hexdump -v -e '/1 "%u\n"' | paste - - -  | awk '{ print "G1 X"$1" Y"$2" Z"$3 }'|head -n $N; echo "M30"

As for the general direction of the gcoder project, I don't know where this all will lead to. I do not want to make gcoder user-friendly, because this would attract hundrets of new people and will cause all kinds of new issues, I'm not prepared for / don't want to spend my time on (– except, if it gets me paid, of course!).

I share your view and see gcoder more or less as a quick G-Code preview / CAM prototyping solution to try out different CAM software packages that are able to run from the command-line.

Here's my latest WIP screenshot 😄 of gcoder with:

• https://github.com/koppi/involute-gear-generator |
• https://github.com/cammill/cammill

[ArcEye]

As for the general direction of the gcoder project, I don't know where this all will lead to. I do not want to make gcoder user-friendly, because this would attract hundrets of new people and will cause all kinds of new issues, I'm not prepared for / don't want to spend my time on (– except, if it gets me paid, of course!).

I can fully appreciate that point, it is what has prevented me releasing my Qt GUI, libraries, QtVCP system etc.
I was stuck supporting an CD / DVD writing application a long time back and it was a chore.
Thankfully BSD adopted it and I gratefully passed ownership of it to them;)

Now I know the direction you are looking in, I can play with more efficient piping and rendering methods.

QTextEdit is good in a lot of respects, but does an update at each line appended, which really drags with big files, hence my only displaying a section of it.

The other alternative is load an initial section, then load the rest in the background after the interpreter has finished and the image is rendered.

[ArcEye]

I am currently writing in a bit more to the UI.
Menu items for File Open and File Save As,
a settings dialog which pops up and gathers your paths for rs274 and tool table if not already in your .conf file and allows you to change them later, so no more hardcoding.

The File Open introduces the concept of partial loading of large files, which I need to fully work and then
can apply myself to the forking of input in a way that does not detract from the on-the-fly, command line interface, but speeds everything up.

Just a heads up so we don't duplicate.

[ArcEye]

Another item whilst it occurs, so I don't forget.

Currently if you click close [X] on any of the docked windows, there is no way to bring that window back, save
deleting the .conf file to get the default view again.

I will add the necessary in the View menu to hide or show the command and stderr windows

[koppi]

You can right-click and there's a popup with the dock widget list:

[koppi]

If you're at GUI code: can you merge the Actions: Fullscreen and Normal into a checkable Action similar to the code of the AutoZoomAction? Shortcut for toggle fullscreen should be F11 (– like it is in many web-browsers).

[ArcEye]

You can right-click and there's a popup with the dock widget lists:

So you can , as long as you are on the header bar and not the QTextEdit widget itself.

If your're at GUI code: can you merge the Actions: Fullscreen and Normal into a checkable Action like to AutoZoomAction?

Yes, I'll look at it later.

[koppi]

You can borrow some gui / preferences source code from my other project if you like:

https://github.com/bullet-physics-playground/bpp/blob/master/src/prefs.cpp
https://github.com/bullet-physics-playground/bpp/blob/master/src/gui.cpp

Or write it from scratch..

[ArcEye]

Got somewhere now with big file loads, through splitting the load process and only reading first chunk into the QGCodeEditor widget.

The viewer gets its own copy of the file in /tmp/gcode.ngc and renders that completely
The QGCodeEditor reads the original file and only loads the first 3K, with 1K chunk being read in every time the cursor position gets within 100 lines of the number of lines in the editor.
(need to change this if possible, to have it change if that position is viewable, rather than highlighted)

Load time for 10,000 line random gcode down to +/- 5 secs for viewer and editor.

Lots of tidying to do in the next couple of days. 😃

[koppi]

I just played with the bash pipe "involute-gear-generator | cammill | grecode", see:

• involute-gear-generator – Involute gear generator, NodeJS command line utility
• cammill – 2D CAM tool for Linux, BSD, Windows, Mac
• grecode – A program to manipuate G-Code

[koppi]

Here's another one with Maker.js (– the skateboard example):

[ArcEye]

Very Good :)
Certainly got a lot of potential for prototyping from command line tools.

I have made my own random gcode file generator for testing. ($1 is number of lines you want)

#!/bin/bash

echo "G80 G90 G94 G21 G17 G40 G8" > gen.ngc
echo "S500 F200" >> gen.ngc

TARGET=$1
COUNTER=0
while [  $COUNTER -lt $TARGET ]; do
    x=$(( $RANDOM % 100 ))
    y=$(( $RANDOM % 100 ))
    z=$(( $RANDOM % 100 ))
    echo "G01 X$x Y$y Z$z" >> gen.ngc
    let COUNTER=COUNTER+1 
done

echo "G40 G49 G54 G80 G90 G97" >> gen.ngc
echo "M2" >> gen.ngc

A lot of the big files I tried are from 3D printing.
They have too many user M codes and other things in them, those halt the interpreter.

[ArcEye]

Inspiration has struck, I think.

The large file problem occurs with generated stuff piped from stdin too, but the method of dealing with it would have to be different from that of loading from file.

The place therefore to deal with the issue and cache whatever exceeds the size desired to be displayed initially, is in the editor widget itself.

I have already reimplemented QPlainTexteditor::appendPlainText() to add the line highlight formatting.
If the code to limit the amount read in and cache the excess is there, appendPlainText() can just be called by the user, the parent window.
The user can be oblivious of any background methods to optimize rendering speed and add code whenever the viewpoint nears the end of what has been displayed thus far.

So back to QGCodeEditor.

[koppi]

In QGCoder/libqgcodeeditor@a338b25 @ArcEye fixed QGCodeEditor to deal with large files.

Next step in making the gcoder GUI reponsive is to let the QProcesses run detached from the Qt GUI thread, and also make them interruptible with the Qt signal/slot mechanism.

[ArcEye]

I deliberately kept away from running a new thread and optimised as far as possible on the existing one.

It would undoubtedly benefit from running the plot and file display completely separately, so long a synchronised.