Lua load external array data

[blancoberg]

Describe the solution you'd like:
Ability to load files that contains array data in lua modulator.

When the patch prelude is implemented, it will be possible to initiate larger arrays upon patch load instead on each note init, which is great.
However for really large arrays, it would be much more practical to be able to load those values from an external file.

The function should only be able to load a specific type of format and that file needs to be located inside the same folder/subfolder of the patch. The function would also only be accessible via patch prelude.

state.values = state.loadTable("values.txt")

An alternative to external files could be to actually import the files into the patch itself. Maybe have a list in patch prelude with all the available files that you can access with some function.

state.values = state.getTable(tableId)

[baconpaul]

I really really don’t want to break the sandbox which keeps the os features turned off. Allowing filesystem access requires real care and thought to keep fxp safe

maybe a custom c function which loads a csv file with a surge controlled opt in and does the io in c is the way to go

[mkruselj]

This is a possible attack vector, I think I wouldn't want to allow for this.

[blancoberg]

Ok, how about the other suggestion?
you import the files manually and they are added to a list of files in patch prelude, that you can access with an id

[mkruselj]

Feels like a lot of work when you could just inline the data that you need...

If you're worried about things not looking nice with big tables, perhaps section folding in the code editors is what we need instead...

[baconpaul]

Well I think exposing the io package would be the vector

we already have a way to bind c++ functions so I was kinda suggesting we write a custom c++ function which

1: checks a user pref
2: if false which is default sends a lua error saying you tried to load a csv file with lua and you have this off so we returned an empty table
2: if true then looks at the file name etc and loads the csv in c++ etc

the other thing is this makes patches non portable though. So one other thought is we add a data field to the formula modulator which is a third tab in the editor where you can paste in a file store it in the patch etc

[baconpaul]

Which is basically your other suggestion yeah with a tabbed ui

[baconpaul]

I think adding a data tab to the editor is the answer we want for a couple of reasons

1. no attack vector
2. No messy ui
3. Easy to stream
4. We can evaluate it once as opposed to once per run and cache it so parse time isn’t an issue
5. We can evaluate it into a table with a well defined name like “state.data”
6. You can put whatever you want in it as long as it evaluates to a lua tanle

seems to meet all the implied constraints here

[blancoberg]

yes that def sounds like the best solution

[mkruselj]

Mmmm I'm not sure if this needs to be a whole new tab. It could perhaps just be a dropdown menu which has an option to import the file and subsequently shows the array names created from those files (perhaps with the path to OG file in parens), and clicking those entries with array names would remove that array. Or actually, use the custom menu entry we have for modulation assignments here, the pencil for renaming the table and red X to delete.

Seems like a more consistent UX?

[blancoberg]

[baconpaul]

What does the pencil do? But sure I’m fine on any of these ux variants which let you import locally but don’t expose lua io

[mkruselj]

Pencil would rename the array.

[blancoberg]

Pencil would rename the array.

maybe its simpler to just name them after the filename. That way it will be easier to update the file as it will just be overwritten on import if it already exists

[mkruselj]

That would of course be the default behavior, but it's not a bad option to have after the fact either, IMO.

[baconpaul]

Right

So I was thinking what if you want a data segment which isn't a file? That's kinda why i was thinking you would have [Code | Prelude | Data] at the bottom and data would just be lua code with the data file in it. Then any lua expressible data structure can be part of the user data prelude.