ImGui full integration

[d3cod3]

• Menus
• Windows
• Canvas nodes
• Duplicate/Delete object selection
• Add objects selection group displacement
• Add sub-patch navigation tree with current patch view switch

[d3cod3]

At this point in time
Mosaic 7514384f18cd87ea9073cf8595fabc26ca3b26e7
ofxVisualProgramming 719b3d48f5c1b3411ec41b0bc8bd811d858404e3
we have:

• good mouse interaction with nodes
• dragging by header only
• wiring not rendering ok after moving nodes
• pins labels easily overlapping with other elements
• pins vertical positions not overlapping with old pins system
• still rendering old ofxDatGui nodes (to remove)

[d3cod3]

Just detected a problem with activating DSP, it must be something related with the instantiation of the audio device object, probably because of his "different" nature (system object), but need further analysis.

it's crashing here:

PatchObject::drawImGuiNode(ImGuiEx::NodeCanvas&) + 944

can be because this object is inited with zero inlets/outlets, then in the setup connect with the audio config and dynamically add the required inlets/outlets?

if i avoid calling the drawImGuiNode method just for this object (for testing) it doesn't crash until i mouse over the object.

Do you have some suggestions?

[d3cod3]

Yes, it is related with dynamic inlets/outlet creation!

Using other objects with this kind of feature (osc receiving/sending, shader object, etc...) crashes Mosaic!

[Daandelange]

I got the same crash, looks like a call to if(!inited)return; in drawImGui() is missing in drawImGuiNode, or something related. The crashing line is inletsPositionOF[i] = ofVec2f( ... );, probably because the vector has not been re-scaled since PatchObject::PatchObject().

[d3cod3]

Yes, i see, my mistake, i added those vector in the last changes while porting links rendering to imgui.

easy to fix!

[d3cod3]

Added:

• audio signal plotter
• Var plotter
• Bang

[d3cod3]

I'm checking imgui extensions and i found this:

https://github.com/AirGuanZ/imgui-filebrowser

Do you think it would be good idea to integrate file dialogs in imgui too?

Right now i'm using the tinyfiledialogs libs wrappes in an addon:

https://github.com/d3cod3/ofxThreadedFileDialog

that works a lot smoother than the standard OF file dialogs, plus it's non-blocking, so the patch is still running while opening/saving some file from/to disk.

Leaving aside the design (i would like it more to have imgui themed file dialogs windows), what is your opinion about that?

I think we need to consider two main points:

1. Custom file dialog windows can be a little confusing for beginner users, as do not "look the same as always"

2. It will increase Mosaic performance/usability in general?

[d3cod3]

Having some compile issues with the previous filebrowser repo

testing this: https://github.com/gallickgunner/ImGui-Addons

[Daandelange]

[first ImGuiEx link] :: "C++17 is required" ==> I recently read that OF doesn't compile out of the box with C++17 (C++14 yes). Maybe this has changed, just noticing.

But it's a good idea to do file dialogs only with ImGui, yes. Although current implementation is fine too.
Anyways we gotta keep saving native in ofxVP (and threaded??). [ "saving" as file writing, not as path picking ]

Now, personally, I both like and dislike native system dialogs.
 - Bad: They are blocking popups that integrate badly with custom UIs, leaving the app inaccessible. Here we could have a great gain in interface logic, having the file browser integrated in the UI. (I already imagine dragging image from file panel to the canvas, and let it spawn as imageObjectA or imageObjectB)
 - Good: I like the default file browser where I have my shortcuts, drag'n'drop support, and other habits. So that would be a minus, except that we can make it better too. I've had a few great experiences with custom UIs.

So, yes and no. I'd say yes, specially batch file-dialog usage can become faster, with a better integration. And we can always provide an optional native file dialog fallback as an ImGui Button.
This said, the ImGui extension must be robust and smooth.

Note: upcoming ImGui versions will have native support for file drag/drop; and there's already something official working. Can also help to make a choice.
+1 for the design upgrade too with ImGui.

Another file browser : ImGuiAddons / imguifilesystem (web live demo (filesystem section) )

This ImGuiEx looks great too : https://bitbucket.org/wolfpld/tracy/src/master/ (ofxTimeMesurements is fine too)

[d3cod3]

I'm using the code from https://github.com/gallickgunner/ImGui-Addons

I needed to update imgui to 1.76 (pushed commit on my ofxImGui fork), and it seems to work real smooth, and this is the temp testing code:

// TESTING ImGui FileDialog
    if(ImGui::Begin("dummy window"))
    {
        bool open = false, save = false;
        // open file dialog when user clicks this button
        if(ImGui::Button("open file dialog"))
            open = true;
        if(ImGui::Button("save file dialog"))
            save = true;

        //Remember the name to ImGui::OpenPopup() and showFileDialog() must be same...
        if(open)
            ImGui::OpenPopup("Open File");
        if(save)
            ImGui::OpenPopup("Save File");

        if( imguiFileDialog.showFileDialog("Open File", imgui_addons::ImGuiFileBrowser::DialogMode::OPEN, ImVec2(700, 310)) )
        {
            std::cout << imguiFileDialog.selected_fn << std::endl;      // The name of the selected file or directory in case of Select Directory dialog mode
            std::cout << imguiFileDialog.selected_path << std::endl;    // The absolute path to the selected file
        }
        if( imguiFileDialog.showFileDialog("Save File", imgui_addons::ImGuiFileBrowser::DialogMode::SAVE, ImVec2(700, 310)) )
        {
            std::cout << imguiFileDialog.selected_fn << std::endl;      // The name of the selected file or directory in case of Select Directory dialog mode
            std::cout << imguiFileDialog.selected_path << std::endl;    // The absolute path to the selected file
            std::cout << imguiFileDialog.ext << std::endl;              // Access ext separately (For SAVE mode)
            //Do writing of files based on extension here
        }
    }
    ImGui::End();

The source code seams neat, and there are three modes implemented, OPEN DIRECTORY, OPEN FILE and SAVE FILE.

With that we can substitute the actual implementation without problems, then we'll be able to add drag&drop functionalities.
Right now it is already implemented an object spawning mechanism when dropping specific filetypes ( video files, script files, audio files, image files, etc... ) inside the canvas ( using OF dragEvent ) so it will not be difficult to do the port.

I think it seems a good codebase to include in ofxVP, i'm going to test it a little deeper.

[d3cod3]

I'm still working on specific objects graphics/interface with imgui, porting all the OF stuff i made before, file dialogs are completely ported, textures are working in some cases (images YES, video NO, it broke after updating imgui to 1.76), variable plotting and audio signal plotting are working fine, and right now i'm with area constrained draggable points editor for Audio Envelope editing for sound objects.

[d3cod3]

Checking this for advanced data plot:

https://github.com/epezent/implot

maybe for some new objects? we'll see

[Daandelange]

Yes, looks useful. There's a lot of interesting ImGuiExtensions, good idea to start listing some.

[d3cod3]

Just integrated a profiler, so less future unmantained ofxaddons

modified from https://github.com/Raikiri/LegitProfiler

[d3cod3]

[d3cod3]

Some objects are not finished yet, but is starting to look serious

[d3cod3]

[d3cod3]

About the textures drawing between OF and ImGui, i had it solved from some time now, but testing some bigger patches i've found ( and already solved ) a problem:

The core of the problem is the same one related with the plugin system #16 , the openGL limitation to share GPU stuff in the same thread only.

Basically we have:

1. the OF openGL thread
2. the ImGui openGL thread
3. ( plugin system ) each plugin loaded in runtime will have his personal openGL thread

So, there is a way of share data between openGL threads, and it is using pixels data ( meaning, leave the GPU and passing through the CPU ) GPU (thread 1) --> texture to pixels --> CPU --> pixels to texture --> GPU (thread 2)

Reference: https://openframeworks.cc/ofBook/chapters/threads.html

Another way is using PBO, i absolutely have no experience with that, but is never late for learning, so i found this:

http://www.songho.ca/opengl/gl_pbo.html#unpack

with time i'll check it and try to implement it!

In the meantime, having all this bridging via pixels: OF --> texture to pixels --> CPU --> pixel to texture --> ImGui, was costing a lot in computation, and with dense patches, especially the patches with DSP stuff, it was causing problems and sound glitches

so

i temporarily fixed it with a trick, for all the texture visualizing objects i draw the background of the ImGui node transparent:

// Node BG fill
        if(curNodeData.zoomName < ImGuiExNodeZoom_Small || !isTextureNode){
            nodeDrawList->AddRectFilled( curNodeData.outerContentBox.Min, curNodeData.outerContentBox.Max, ImGui::GetColorU32(ImGuiCol_FrameBg, 999.f));
        }else{
            nodeDrawList->AddRectFilled( curNodeData.outerContentBox.Min, curNodeData.outerContentBox.Max, IM_COL32(0,0,0,0));
            nodeDrawList->AddRectFilled( curNodeData.outerContentBox.Min, ImVec2(curNodeData.outerContentBox.Min.x+curNodeData.leftPins.region.GetSize().x,curNodeData.outerContentBox.Max.y), ImGui::GetColorU32(ImGuiCol_FrameBg, 999.f));
            nodeDrawList->AddRectFilled( curNodeData.rightPins.region.Min, curNodeData.outerContentBox.Max, ImGui::GetColorU32(ImGuiCol_FrameBg, 999.f));
        }

and then i draw from OF the texture directly in the correct node position:

inline void drawNodeOFTexture(ofTexture &tex, float &px, float &py, float &w, float &h, float originX, float originY, float scaledW, float scaledH, float zoom, float footerH){

    if(tex.isAllocated()){
        if(tex.getWidth()/tex.getHeight() >= scaledW/scaledH){
            if(tex.getWidth() > tex.getHeight()){   // horizontal texture
                w           = scaledW;
                h           = (scaledW/tex.getWidth())*tex.getHeight();
                px          = 0;
                py          = (scaledH-h)/2.0f;
            }else{ // vertical texture
                w           = (tex.getWidth()*scaledH)/tex.getHeight();
                h           = scaledH;
                px          = (scaledW-w)/2.0f;
                py          = 0;
            }
        }else{ // always considered vertical texture
            w               = (tex.getWidth()*scaledH)/tex.getHeight();
            h               = scaledH;
            px              = (scaledW-w)/2.0f;
            py              = 0;
        }

        if(scaledW*zoom >= 90.0f){
            // background
            ofSetColor(34,34,34);
            ofDrawRectangle(originX,originY,scaledW-2,scaledH+(footerH/zoom));
            // texture
            ofSetColor(255);
            tex.draw(px+originX,py+originY,w-2,h);
        }
    }else{
        if(scaledW*zoom >= 90.0f){
            // background
            ofSetColor(34,34,34);
            ofDrawRectangle(originX,originY,scaledW-2,scaledH+(footerH/zoom));
        }
    }

}

with that, Mosaic performance is back to the previous version, a lot better

Do you have experience with PBO? We can try to solve this later, right now is a hacky solution, but is working perfectly, with one quirk, over texture objects, cables appear in front!

No sweat, i mean, better this than loosing performance at 4 texture objects running in the same patch!

[Daandelange]

I have looked into PBOs some time ago; don't know it well enough. Sounds like an accelerated version of pixels2texture.
I guess OF's built-in pixels2texture methods use the same feature ?

Anyways, it's really not a good idea to do useless pixel2texture (or vice-versa) operations.
Only do them when needed (image decoding, image acquisition, ...).
Also, using big resolutions, these operations become exponentially slow.
Is there no way of sharing GL contexts with plugins ?

What about this ? I see glRenderer is not yet used... Is this approach not working with plugins ?

void nodeObject::drawObjectContent(ofxFontStash *font, shared_ptr<ofBaseGLRenderer>& glRenderer){
	glRenderer->begin();
	ofPushStyle();
	ofSetColor(255,0,0);
	ofNoFill();
	ofDrawRectangle(100,100,100,100);
	glRenderer->end();
}

Another (dummy?) question : Is it possible to load plugins on the GL thread so they don't interfere with it ?

[d3cod3]

Yes, glRenderer is there exactly for the plugins, for basic drawing routines as the ofDrawRectangle , or font drawing, or everything that is computed by CPU, but for textures and shaders are not working (GPU), the only way right now is bridging through pixels ( really expensive )

Is not crashing or anything, just the texture loose the reference and remain black, or with artifacts from last memory region access.

Of built in texture to pixels didn't use PBO, is a heavy load CPU routine, but there is this, from an addon and included in ofxVisualProgramming:

https://github.com/d3cod3/ofxVisualProgramming/blob/refactor-params-and-gui/src/objects/video/ofxFastFboReader.h

I've tested it as it is, but is not working in this particular case yet ( access from two threads )

And no, for the plugin structure as it is, i didn't see the option of loading it inside the openGL thread, but i'll search the web for some info

[Daandelange]

If the plugin system brings a big performance loss (forcing useless GPU<-->CPU operations), is it worth having plugins ?
Let's close this parenthesis and keep plugin talk in #30 .

So here, it's more about the issue of drawing textures from ImGui for previewing purposes, if I understand correctly.
The original approach needed CPU<-->GPU operations, so you found a temporary transparency hack, which should better be done in another way ?

[d3cod3]

This problem is only related to "texture" nodes, basically every objects that works with image in some way, so all the other type of objects do not have problems, hence is still worth having a plugin system, even is "texture" type objects will be discarded.

Probably this problem will appear in the future surface nodes #24 , but we'll see when we get there

The hack with transparency is working great, performance is back at good levels, so for now it doesn't worry me, but i'm going to check this http://www.songho.ca/opengl/gl_pbo.html#unpack, to see if i find a way of optimize the code

and passing from this

to this

and end up implementig this

And that could be even the solution for the plugin problem ( not 100% sure, but... )

[Daandelange]

Yes, the hack is fine in terms of performance. Eventually, if we come back on this, there seems to be ways to do it by passing a function pointer callback (to drawNodeOFTexture from ImGui?) : imgui/issues/1850

[d3cod3]

I like the option of function pointer callback, i'll give it a try

[d3cod3]

function pointer callback not working! same problem, the texture id is not being shared correctly so texture will not appear

[d3cod3]

Removing the subpatches/abstraction idea, i tested something and i didn't like the multiple patches windows that will end appearing inevitably, it will lead to visual confusion due to the highly graphics nature of Mosaic ( while it works fine in PD due to the really minimal graphical style ). So leaving this option, for now at least.

[Daandelange]

Instead of opening new windows we could "zoom into" objects to view their sub-patches. Manually mouse-zooming and/or a trigger button to zoom an object full screen.
Not sure this will allow infinite sub-patches and it might require some important changes to imguiNodeCanvas.

Another solution that could be great is to replace the root canvas (gui) by the child canvas, together with a sub-patch-navigation-bar like [ Root < Subpatch < SubSubPatch ].

[d3cod3]

YES!!! great idea the navigation-bar one! Substitute the root canvas to the current one it can works perfectly.

amazing, i'll add it to the todo list!

[d3cod3]

Ok then, maybe we wait for your merge on jvcleave, and then after testing we merge the ofximgui-next changes to Mosaic master, feels right?

[Daandelange]

Could make the switch easier for others, yes, I just hope it won't take too long.

[d3cod3]

there is no rush

[d3cod3]

One question, i'm trying again to draw textures from imgui and not OF, as it is now, i have checked in your repo advanced example how to load and draw one, but it seems that in order to have the right texture ID it needs to be loaded in the ImGui gui?

doing just this:

ImGui::Image((ImTextureID)(uintptr_t)static_cast<ofTexture *>(_outletParams[0])->getTextureData().textureID, ImVec2(scaledObjW, scaledObjH));

will result into this:

can you point me in some direction?

[Daandelange]

It looks like the texture ID is overwritten by the fontatlas texture. Is your texture correct if you draw it in OF (in a liverpatcher...) ? I guess you already load the texture after loading the custom fonts ?

[d3cod3]

Yes, the texture is loaded with the object, in this case the video viewer, so always after the custom fonts ( loaded in the Mosaic setup )

And yes, drawing the texture in OF works perfect ( as is working now in all the image objects )

[Daandelange]

I'll have a look later, I think currently ofxImGui needs to own the textures, so it's probably grabbing the texID in GLFW "namespace", not the OF GPU "namespace". What happens when there's a 2nd OF textureID ?

[Daandelange]

Ok, found the issue. When loading textures trough ofxImGui, it disables ARBTex, copies the texture internally (taking ownership of it). Ownership is not particularly needed, you can render your own texture, but the TexID needs to remain allocated until the end of the frame (no problem here).
ImGui_Impl_OpenGL can only render GL_TEXTURE_RECTANGLE (ArbTex enabled), not GL_TEXTURE_2D (ArbTex disabled).
Still I have no idea why the fontatlas appears when Arb is enabled, it renders more or less correctly.

// This should fix it; if Mosaic needs them enabled, we might need to provide a (smaller) duplicate one.
ofDisableArbTex();
MyTexture.allocate(100,100, GL_RGBA);
ofEnableArbTex();

Does ImGui::Image(...) work with your ofxImGui branch ?

[d3cod3]

Fantastic!!! I'm going to test it right now, thanks!

ImGui::Image() worked with my branch, yes

[Daandelange]

Ok, I guess the legacy branch uses OF as renderer backend... otherwise I might have stripped something needed. Checking.

Better: (also possible with texture instead of fbo)

ofFboSettings settings;
settings.textureTarget = GL_TEXTURE_2D; // <--- the trick
settings.width = 100;
settings.height = 100;
settings.internalformat = GL_RGBA;
ofFbo fbo;
fbo.allocate(settings);
// Later...
ImGui::Image( GetImTextureID(fbo), {fbo.getWidth(), fbo.getHeight()} );

[d3cod3]

I'm still testing it, the image now is working fine, but i'm seeing some jittering on texture objects ( still image objects works completely fine ), the video framerate is jumpy, so i'll dig a little deeper, i'll let you know how is going

[Daandelange]

With the latest changes that I pushed, single clicks don't work correctly on the ImGui node objects, no idea why.

Edit: ImGui has done some changes in the IO API, introducing some kind of priority routing that lets widgets catch exclusive mouse access... I fear that this breaks the nodal implementation as we're calling multiple timesIsMouseClicked() (and similar funcs) on the same widget.
The best info I could find (not yet documented) : https://github.com/ocornut/imgui/issues/3370#issuecomment-1307763731

Example:

// This works (standalone ofxImGui)
ImGui::Button("#");
if(ImGui::IsItemActivated()){
    ImGui::SameLine();ImGui::Text("Activated!");
}

While here the condition is never triggered:
https://github.com/d3cod3/ofxVisualProgramming/blob/135b2b4d7c34ade4e00bb6d3507396c5496eb92f/src/core/imgui_node_canvas.cpp#L521-L524

Edit2: Mosaic-Engine-Tester also works correctly, so again, I suspect the new event ownership API to break some functions where Mosaic adds custom functionality to the node canvas...

[d3cod3]

Hey, i have been afk, i'm checking right now the issue, yes, it stops working and it seems as you say it, i'm going to take a deep dive in the imgui_node_canvas file and come back with a solution... or not...

Edit1: i've managed to solve some basic stuff, click on nodes menus and dragging from the header, removing IsItemActivated() and using combinations of IsItemHovered() with IsMouseDown()and others, messy but working fine.
The big problem is NOTHING on nodes internal menu is working, nor on the nodes with external interfaces ( knobs, buttons, etc... ) so for now, not so good...

Edit2: BeginDragDropSource is failing to trigger too, so no cables either for now

[Daandelange]

Yes, the nodes are a big mess and I don't understand what's wrong. Some of the broken items can be (semi-)triggered using a double-click.
I just tried with ImGui 1.89.3 which doesn't have the new event ownership API and it doesn't work either, so it's probably something else. (Tip: you can now do this by uncommenting the right version in ofxImGui/libs/UpdateImGui.sh and running it).
The weirdest thing is that the node canvas works in the engine tester, so it's probably an obscure conflict with ofxVP or Mosaic code sending ImGui commands.

[d3cod3]

It seems like this line ( in ofxVisualProgramming setup ): nodeCanvas.setContext(ImGui::GetCurrentContext()); is not linking to the context, maybe ImGui::GetCurrentContext() is working differently from before?

[Daandelange]

That line should do absolutely nothing: it sets the current context to the current context.

I just tested ofxVP/example-ImGui and everything works fine (but I had to adapt some lines, continuing later), so it's definitely related to Mosaic code.
Maybe it's related to the way we setup ofxImGui within ofxVp, or maybe rather how we setup ofxVp within Mosaic.

[Daandelange]

With this, ofxVP compiles and works, but not yet Mosaic ... b543a77

[d3cod3]

Ok, but there has to be something because the Mosaic theme was not loading with nodeCanvas.setContext(ImGui::GetCurrentContext());, then i changed it to nodeCanvas.setContext(ofxVPGui->getContext()); adding a context getter inside Gui.h, and the theme is now loading properly.

[Daandelange]

Ok, I found it, we're mixing the new master/slave shared-context setup with a custom shared gui instance, confusing ofxImGui (see warning messages about incorrect orchestration of begin/end calls). So we need to clean up some old code.
As a quick fix for now: Mosaic works with mainMenu.end(); commented/removed in ofApp::drawImGuiInterface().

Edit: As for the theme, it can to be set like you did, or ideally set the theme and fonts before calling gui->setup(). (preventing the font atlas to be build twice), ofxImGui/example-fonts shows how to do this.

[d3cod3]

YES!!!! b543a77 solved it!!!

Now it's working fine!

Edit1: I'm going to check the example-fonts and fix the theme stuff properly

[d3cod3]

Ok, i've checked Mosaic/ofxVisualProgramming theme and font loading, it is the same as the example-fonts, then i found the error, it was a simple variable name mistake in Gui.cpp, i'll leave the patch here:
fix_theme_loading.patch

Now it's working everything fine

Just uploaded last changes on Mosaic and ofxVisualProgramming, compiling and running fine on macOS, next station, try it on linux...

[Daandelange]

Merged, thanks !
Great that it's working again, I was starting to worry :)

For fixing this correctly, we'd need to remove the vp::setup(&gui_instance) argument and let ofxVP have its own gui instance, which is used internally by ofxImGui for slave/master counting and some kind of garbage collector meccanism for the singleton instances. On the ofxImGui side I'll try to disable making references of the gui handle, to prevent mistakes.

[Daandelange]

I pushed some commits to fix this properly, see note about breaking changes.
(Sorry, the ofxVP one didn't get signed :( )

Edit: I'm compiling Mosaic with the following defines, You don't need to set them but I think it's the best configuration for Mosaic.

defs = defs.concat(['OFXIMGUI_GLFW_EVENTS_REPLACE_OF_CALLBACKS=1']);  // Replace GFLW callbacks (Default value)
defs = defs.concat(['OFXIMGUI_GLFW_FIX_MULTICONTEXT_PRIMARY_VP=0']); // Disable multicontext tweaks
defs = defs.concat(['OFXIMGUI_GLFW_FIX_MULTICONTEXT_SECONDARY_VP=0']); // Disable multicontext tweaks (bis)

[d3cod3]

OK, i'm compiling with the last commits, and your suggested defs, i'll test it on linux and windows too, and i'll come back with some results

[d3cod3]

A small hotfix for linux compiling

linux_compiling_hotfix.patch

[Daandelange]

Thanks, merged ;)