Sending big data between two windows

[LDubya]

Hello,

We use meteor-desktop completely locally -- no remote server. Data is stored locally, in IndexedDB, cookies, etc.

What's the best way to send >10MB of data over the wire between two local windows (same app instance, just two browserWindows)?

We need to open up a hidden background window and send data to it for printing.

Although we are able to store large amounts of data inside of IndexedDB, if we open up a second browserWindow, the new browserWindow does not have access to the IndexedDB databases that were set by the first window. Sometimes we can see that the second browser window shows the data using the Dev Tools, but that data on the second window is not accessible via Dexie or Ground:db / localForage or other APIs for accessing IndexedDB data (although the data is accessible via the first window, where the data was set). How can we get a new BrowserWindow to access the IndexedDB data that was set in the first BrowserWindow, via the JS API? That could be one way.

In addition, the second browser window doesn't have access to the "Desktop" object, and therefore we can't send the data via IPC. (How can we gain access to the Desktop object on a second browserWindow to listen for IPC events? This could be one way, but Desktop returns as undefined.)

The data is often too big for localstorage. Sometimes this data can be >10MB and can't easily be split into smaller parts. For example, we may need to send a single image over the wire that's >10MB. We need something reliable for larger amounts of data than what localStorage will allow for a single entry (big blobs?).

This would also make it too big for Session storage and cookies.

The data is also too big to be sent via a GET request as a stringified query parameter.

Can't send via POST since this app is purely client-side and no server, but POST also has a size limit that is <<10MB.

What's the best way to send >10MB of data between two local windows using Meteor-Desktop?

Localstorage works, but there's the size limit. Are we going to have to stringify the data and then split the strings into <10MB chunks for storage, and then retrieve and merge them all together before parsing, or is there an easier way? The data is already in IndexedDB in the first BrowserWindow. If the second / background BrowserWindow could access that IndexedDB data then there wouldn't even need to be a need to send anything over the wire besides a pointer.

[wojtkowiak]

Hey,

just quick tips:

• when you are spawning a new BrowserWindow you can try to load Desktop into it with

windowSettings.webPreferences.nodeIntegration = false; 
windowSettings.webPreferences.preload = path.resolve(path.join('..', 'app', 'preload.js'));

In the path I'm assuming the code is in .desktop/desktop.js.

• other way is to register your own protocol https://www.electronjs.org/docs/api/protocol and push the data through it from one window and query it from another window

[wojtkowiak]

this might be a dead end but take a look at https://www.electronjs.org/docs/api/session
not sure how this is handled but you can try to take the session from the main window and pass it as windowSettings.webPreferences.session to the new one

[LDubya]

hi @wojtkowiak, thanks.

Looks like preload.js is inside of desktop-build for us.

> find . -name "preload.js"
./.meteor/desktop-build/app/preload.js
./node_modules/meteor-desktop/skeleton/preload.js

Switching the preload to be path.resolve(path.join('..', 'desktop-build', 'app', 'preload.js')) seems to make Desktop discoverable in the second window.

I'll report back how this ends up working. I believe we should be able to send large amounts of data via IPC, but only one way to find out.

[LDubya]

So far so good, able to send an 11.630833625793457 MB string over IPC, pretty much instantaneously.

[LDubya]

Hi @wojtkowiak, looks like we ran into a problem. How would one get the preload.js to load when it’s packaged inside of the app.asar?

When running the build script to package and sign the app as a .app / .pkg, that preload file is compiled inside the app.asar and no longer exists as an independent file that can be added to the preload of a new browserWindow.

webPreferences.preload wants an absolute filepath, but using the code above in production resolves the absolute path to be inside of the user directory (users/User/Library/containers/com.app.bundleID/desktop-build/app/preload.js). This is strange because this isn’t where the app.App is located — we’d expect the absolute path to resolve to be in the Applications directory where the app is installed, but looks like maybe it’s resolving to where the app data is kept?

Maybe we can just copy and paste the raw JS code somewhere inside the app’s source instead of referring to an external file?

[wojtkowiak]

Hey,

path.resolve(path.join('..', 'desktop-build', 'app', 'preload.js'))

I can not check it now but this path is weird because I believe your cwd is desktop-build - can you check if this also works path.join(__dirname, 'app', 'preload.js')?

When it comes to production build you should be able to change it to proper one when packaged. You are getting the ref to skeleton app in the desktop.js constructor (or any other module)

constructor({
        log, skeletonApp, appSettings, eventsBus, modules, Module
    })

try using it to change the path when in production build skeletonApp.isProduction() ? 'app.asar' : 'app'

[LDubya]

Using path.join(__dirname, 'app', 'preload.js') leads the development app ($npm run build) to try to load inside of the .asar file. It tries to find the preload.js in the following path: /path/to/app/.meteor/desktop-build/desktop.asar/app/preload.js

switching the path to be path.join(__dirname, "../", 'app', 'preload.js') works in development.

In the production build, path.join(__dirname, 'app', 'preload.js') tries to find the preload.js inside of the .asar file: /path/to/App.app/Contents/Resources/app/desktop.asar/app/preload.js, which of course can't be resolved (electron-userland/electron-builder#751).

In the production build, path.join(__dirname, "../", 'app', 'preload.js') (which works in development) tries to find the preload.js in the following path: /path/to/App.app/Contents/Resources/app/app/preload.js, which is probably a little better since it's not trying to search inside .asar, but the preload.js would need to be added somewhere in that App.app/Contents path during building. By default everything is just packaged inside app.asar. So would probably need to package the JS file separately, outside of the .asar, and then try to reference it that way, using your skeletonApp.isProduction() to give a different preload path on development vs packaged.

[LDubya]

K, for anyone running into this in the future:

Configure the builder to make the preload.js file available outside of the packaged .asar: (here it will be placed in App.app/Contents/Resources/extraResources)

settings.json:

"builderOptions": {
        "extraResources": [
            {
                "from": "./node_modules/meteor-desktop/skeleton/",
                "to": "extraResources",
                "filter": [
                  "**/preload.js"
                ]
            }
        ]
}

Then in desktop.js for the new browserWindow:

let preloadPath;
if(skeletonApp.isProduction()){ 
    // production preload.js path
    preloadPath = path.resolve(path.join(process.resourcesPath, 'extraResources', 'preload.js'));
}
else{ 
    // dev preload.js path 
    // depending on your setup you may have to set this to a different location
    preloadPath = path.resolve(path.join(__dirname, "../", 'app', 'preload.js'));
}

Then both the dev and production environments should be able to get the right preload.js path.

Thanks for the help @wojtkowiak