Iterating Modulators w/ panel:getModulatorByIndex(i): Any Benefits of Dedicated Vars vs Lookup Hashtable?

[unityconstruct]

I have incoming dumps deciphered and mapped to tables so looking at iteration options for efficient bi-directional assignment to/from the dumps.

Parsed Ctrlr.org and there are several variations available that are technical solutions, but understand that panel:getModulatorByName() is a costly way to go about it, with atom noting that foraging by ModName results in iteratively running down the ModList each time so currenty looking at two other methods using:

panel:getNumModulators() |  panel:getModulatorByIndex(i)

• assuming the reason this is faster is that it iterates KEYS in order, not VALUES by searching
• I'm curious if there's any benefit to assigning each modulator to a dedicated variable
  • this only makes sense if by this method lua passes references instead of creating copies
  • if vars are simply pointers, then by assigning them upfront would significanly reduce overhead of parsing Ctrlr's master list.
  • var count not an issue as the code can be generated in Excel with formulae & I already have all Device Params enumerated from deconstructing the sysex spec.

custom_var1 = panel:getModulatorByIndex(i)
custom_var2 = panel:getModulatorByIndex(i+1)
..
custom_var399 = panel:getModulatorByIndex(399)

any thoughts/comments appreciated

---

Ctrlr.org examples

'''
Well linear programming (instead of using loops) is always faster, but loops are just easier to use. I spent some time on this some time ago and tried to address the performance issue by adding hash tables with modulator names, but that wasn’t very helpful.

What you have to keep in mind, that when you do getModulatorByName() Ctrlr iterates through the entire list of modulators and tries to find the one with the name provided, if the modulator is at the end of the list then it might take a while.
'''

mapping mods by index

• One way to go through all the modulators in the panel quickly:

    n = panel:getNumModulators()

    for i=0,n-1 do
        mod = panel:getModulatorByIndex(i)
    end

You can use it to quickly send patches for users, to change properties on the panel to all modulators/components (i’ll be doing a patch manager with a web repository so that i can distribute simple patches quickly, like the bug with the mouse cursor beeing None should be fixed by a small snip of code like that)

dnaldoog extended version with L-wrapped getName

• panel:getModulatorByIndex(i)

myGenerateModNames = function()
    n = panel:getNumModulators()

    for i=0,n-1 do
        mod = panel:getModulatorByIndex(i)
        local Name= L(mod:getName())
        print (i,Name)
    end
end -- function

print=function(…)
    local output=””
    for i,v in ipairs(arg) do
        v=string.format(“%s”,v)
        output = output..” “..v
    end
    console(String(output))
end --function

myGenerateModNames()

Generic for-loop

• panel:getModulatorByName(modName)

function updateModulators(midi)
    local t={dial1=8,dial2=10,dial3=12}
    for k,v in pairs (t) do
        local b1=m:getData():getByte(v)
        local b2=m:getData():getByte(v+1)
        local value=combine4n(b2,b1)
        panel:getModulatorByName(k):getComponent():setValue(value,true)
    end
end

• panel:getModulatorWithProperty(“vstIndex”,ID)

for d = 0, "your max"
    e = panel:getModulatorWithProperty(“vstIndex”,d)
    f = e:getProperty("modulatorCustomIndex")
        for i=0,56 do
            if f == nrpnbase+i then
                j = 128 * programData:getByte((i*2)+10) +programData:getByte((i*2)+11)
                --console(toString(i))
                --console(toString(j ))
                e:setModulatorValue(j, false, true, false)
            end
        end
end

Custom Property

• panel:setProperty(propName,propValue)

panel:setProperty ("myVeryCustomProperty", "my even more custom piece of data")

• You can set any property that’s not a property defined in ctrlr and it will be saved with the panel (or a modulator)
• it will be saved with the panel and you can re-read it, just remember it’s only strings/ints/bools so convert your memory blocks to a hex string (i forgot to add base64 encoding but hexString should be enough)

[dnaldoog]

If you create a lua table with keys with values pointing to an object like panel:getModulator() or panel:getComponent() then I am pretty sure those are references and won't change during runtime.

I use metatables to achieve this. This way you don't end up with a slew of variable names in the global space _G. I wrote this before goodweather pointed out that you can run lua functions from the console (I never knew that until recently) thus updating an init script that might contain lua variables for modulators ― but even so it's a convenient way to write code and you can use any variable name using lua's [""] table key syntax.

There is probably a performance hit when a variable pointing to a panel object needs to be assigned a lua table key using the metatable, but I doubt it's a big deal.

I can't think of any benefit to using serialised code when updating modulators from a sysex dump and collating data for sending in order. I think the hash/key way makes more sense.

tab = { -- list of all modulators in a sysex dump
    lfoDelay = {pos = 3},
    lfoDelay2 = {pos = 4},
    ["my cutoff"] = {pos = 7},
    ["waveform MSB"] = {pos =8},
    ["waveform LSB"] = {pos = 9}
}

--incoming sysex as midi:getData():toHexString(1)

--(`F0 00 00 03 7F 00 00 63 01 00 F7`)

--assign incoming dump:

for k, arr in pairs(tab) do
    panel:getModulatorByName(k):setModulatorValue(midi:getData():getByte(arr.pos), false, true, false) 
    -- maybe false,false,false ??
end
local msb = panel:getModulatorByName(tab["waveform MSB"]):getModulatorValue() -- this could be a hidden control
local lsb = panel:getModulatorByName(tab["waveform LSB"]):getModulatorValue() -- this is visible
local waveform = (msb * 128) + lsb -- often there are more than 128 waveforms
panel:getModulatorByName(tab["waveform LSB"]):getModulatorValue(waveform, false, true, false)

---

-- Sending Midi
local mess = MemoryBlock("F0 00 00 00 00 00 00 00 F7")
for k, v in pairs(tab) do
    mess:setByte(v.pos, panel:getModulatorByName(k):getModulatorValue())
end
mess:setByte(tab["waveform LSB"].pos, panel:getModulatorByName(tab["waveform LSB"]):getModulatorValue() - 128) 
-- quick way to create 14 bit value

pairs will rarely loop in the same order as the table is written

N.B. In my metatable system linked above I would substitute _m for panel:getModulatorByName in all this code!
So by the time you reaccess say panel:getModulatorByName(tab["waveform LSB"]) (using _m["waveform LSB"]) the variable (table key reference to panel:getModulatorByName(tab["waveform LSB"])) has already been created and panel:getModulatorByName(tab["waveform LSB"]) is not directly accessed again. The object is now pointed to by _m["waveform LSB"] and panel:getModulatorByName(tab.lfoDelay) is now _m.lfoDelay.

Something like that! 😎

[unityconstruct]

@dobo365 @dnaldoog THANKS! for responses... digesting it currently.

metatables

• That metatables link is sweet and def adds to the solutions toolbox.
• I went the OO approach out of habit, so using metat's exclusively... looks like I can do the same with a MODS table. While Global space is already cluttered with Ctrlr making use of global funcs, my motivation was to have everything available using 'dot-syntax'
• I created a DM(DeviceManager) metatable which (syntactically) houses everything but the Methods wired into Ctrlr ( and these simply call into DM & pass mod/comp/value ).
• in VSCode, this allows intellisense to bring up options as I type and helped partition the surprisingly large function library I didn't expect to need. I significantly underestimated the code-behind that goes into these panels... it gets deep quickly once you go behind simple CC linking of knobs.. I've refactored everything no less than 5 times as classes were getting 3000 lines long 😎 and as I figured out how to effectively leverage lua's ecosystem... which apparently allows you to do things 15 diff ways.

I appreciate all posts you gents have on Ctrlr.org and by using site: in the browser I normally can get right to what is needed
site:ctrlr.org timer:setCallback

init():

I'm having some classes that started failing on bootup, requiring another compile all to seat things properly and while I'm using init() when running back-end tests in VSCODE, didn't know this was operable in Ctrlr. Was JUST fiddling with file order in the LUA editor tree & coming up short.. Will look at throwing init() in there to control loading sequence.

enums

• Also mulling over creating an enum working with mods.. with the advantage of having the IDE present valid options
  
• Going to look at either creating enum for _m KEYS or (if mod names are normalized) just having the mods be the enum members
  

Thanks again for the mentoring!

-- enum declaration
        ---@enum BankIndex
        BankIndex = {
            Bank0 = 0,
            Bank1 = 1,
            Bank2 = 2,
            Bank3 = 3
        },
- implementation
        -- ---@type ROMID
        -- mm_select_preset_romid = GET.Enum.ROMID.USER,
        ---@type ROMID
        cc_romid=GET.Enum.ROMID.USER,
        ---@type BankIndex
        cc_bankindex = GET.Enum.BankIndex.Bank0,

• DeviceManager MasterClass/Object

--- @class DeviceManager
DeviceManager = {}


---@return DeviceManager
function DeviceManager:new()
    setmetatable({},self)
    self.__index = DeviceManager
    --[[ CONFIG ]]
    self.MIDI_SEND_ENABLED = false
    self.CONSOLE_ENABLED = true
    self.CONSOLE_VERBOSE = true
    --[[ CLASSES ]]
    self.RX = ReceiverController:new()
    self.DEVCTL = DeviceController:new()
    self.RXSysex = SysexReceiverHandler:new()
    self.RXMidi = MidiReceiverHandler:new()
    self.TX = SenderHandler:new()
    self.PARAM = ParameterHandler:new()
    self.DUMP = DumpRequestHandler:new()
    self.PRESET = PresetHandler:new()
    self.MODS = {}
    self.UI = UiHandler:new()
    --[[ STATE ]]
    self.DUMPS = {
        PrestDumpHeaderResponse = "",
        HardwareConfigDump = {},
        SetupDump = {},
        PresetDump = {},
        PresetDumpHeader = {},
    }
    self.STATE = {
        ---@type string
        syx_deviceid = "00",
        ---@type integer
        midi_chan = 1,
        ---@type integer
        master_vol = 80,
        ---@type integer
        midi_delay = 60,
   ...
  }
  --[[ CONFIG ]]
  
  --- MIDI_SEND_ENABLED State Change
  --- @param value number|integer|boolean
  function DeviceManager:MidiSendEnable(value)
      DM.MIDI_SEND_ENABLED = DU.strToBool(value)
      p(string.format("MIDI_SEND_ENABLED :[%s]",tostring(DM.MIDI_SEND_ENABLED)))
  end
  ...
end

• dot-syntax implementation

    DM.STATE.dump_preset_id = presetNum
    DM.STATE.dump_rom_id = DM.STATE.cc_romid
    DM.DUMP:SendPresetDumpRequest(DM.STATE.dump_preset_id, DM.STATE.dump_rom_id)

[dobo365]

On my side I just use panel:getModulatorByName() at panel initialization to assign a variable to each modulator I'm using.
From there, I use the variables and there is direct access to each modulator
So, easy and no need of something fancier.
I use panel:getModulatorByIndex(i) in a loop only for Muting / Unmuting all dynamic mods. But there, no need to go specific.

[dnaldoog]

I think goodweather’s system is indeed more straightforward.

The system I use has one advantage that I can see. Not having to invent legal lua variable names for controls listed in a manual. For example, in the Yamaha TX81Z there is a parameter Operator 4-1 On/Off (bits 1-3). With my system I can name a modulator just that and access it with _c(“Operator 4-1 On/Off (bits 1-3)”):getProperty() etc. if I need to access that control in another function I don’t need to look up the legal lua variable name in an init script or whatever.

Of course, if the modulator is named something legal like operator41 then the lua variable is easy to access, just do operator41:getComponent():getProperty() but I find my way easier when cross referencing back to the manual especially with deep and complex machines.

It’s not a big deal I guess- just another way of avoiding that potential bottleneck of looping through the panel by name to find an object in a loop.

For me it’s a question of workflow rather than any big gains in runtime efficiency.

[unityconstruct]

RE: init node

@dnaldoog

• saw reference to init() in that metatables link you provided
• I just added method 'init' in Ctrlr Lua Editor, added my globals instantiation, compiled, shut evertyhing down & brought back up and it resolved my class loading issues, but unsure of the mechanics of it all:
  • there's no init.lua file (only exists in the XML repo)
  • no init() function
  • no init{} table

Assuming that Ctrlr has special hook to run the init node first if it exists?

• it fixed class loading issues that weren't resolved even when placing these calls outside function() in any other file/node regardless of location in the node tree.

console("initializing")
console("initializing: DataUtils")
DU = DataUtils:new()
console("initializing: DataUtils Done")
console("initializing: GlobalFuncs")
FUNC = GlobalFuncs:new()
console("initializing: GlobalFuncs Done")
console("initializing: DeviceManager")
DM = DeviceManager:new()
console("initializing: DeviceManager Done")
console("initialized")

[unityconstruct]

@dnaldoog @dobo365 thanks for info...! basic proof-of-concept working

• already had paramId suffix on mod names... my dump table uses paramIds for the keys, so just extracted modname suffix and went for it.
• VoiceMod table is an enum so can access via dot-syntax or the VoiceMods[""] accessor
• might go another route for non-param/data things like LCDs and App Controls.

THANKS again for the insights

-- Dump Contract for Sysex Parsing

self.Contract = {
            ---@type table<integer, table<integer,integer>>
            PresetDump = {
                [899]   = {1  ,1  },
                [900]   = {2  ,2  },
                [901]   = {3  ,3  },
                [902]   = {4  ,4  },
   ...
}

--- caller test code:
local mods = Mods:new()
what(mods.VoiceMods.kb_layer1_ctune_14251)
Mods:VoiceModsUpdate()


---@class Mods
Mods = {}
---Mods
---@param o? any
---@return Mods
function Mods:new(o)
    o = o or {}
    setmetatable({},self)
    self.__index = self
    self.VoiceMods = Mods:VoiceModsGet()
    return self
end

function Mods:VoiceModsGet()
    --[[ PREST LAYER PARAMS]]
    ---@enum VoiceMods
    local VoiceMods = {
        ---@type CtrlrSlider
        kb_layer1_ctune_14251 = panel:getModulator("kb_layer1_ctune_14251"),
        ---@type CtrlrSlider
        kb_layer1_ftune_14261 = panel:getModulator("kb_layer1_ftune_14261"),
    }
    return VoiceMods
end

function Mods:VoiceModsUpdate()
    p("VoiceModsUpdate: Updates: STARTED")
    for k,v in pairs(self.VoiceMods) do
        if(DU:isNotNull(k) and DU:isNotNull(v)) then
            p(string.format("Updating mod:[%s] from dumptable",tostring(k)))
            Mods:GetValue(v)
        end
    end
    p("VoiceModsUpdate: Updates: COMPLETED")
end

---Get ParamId Suffix from name. searches for LAST OCCURENCE of delimeter and retreives any characters after
--- for mods fetch name with: [L(mod:getName())]
--- for comps fetch anme with [L(comp:getOwner():getName())]
---@param name string: component
---@param delimeter string?: delimeter to look for
---@return string: paramId use to lookup values in the ParamId2ValueMapTable
function Mods:GetParamIdFromName(name,delimeter)
    delimeter = delimeter or "_"
    local suffix = ""
    if(DU:isNullOrEmpty(name)) then
        p("GetParamIdFromName:ERROR: provided name is nil or Empty")
    else
       suffix =  DU.getSuffixByText(name,delimeter)
       if(DU:isNull(suffix)) then
            p(string.format("GetParamIdFromName:ERROR: Suffix not found for name[%s]",tostring(name)))
            return ""
       else
            p(string.format("GetParamIdFromName:name:[%s] id:[%s]",tostring(name),tostring(suffix)))
       end
    end
    return suffix
end

---Update Modulator Value from device ParamId2ValueMapTable
---@param id any
function Mods:GetParamValueFromDump(id)
    DM.DUMPS.PresetDump:GetDumpValue_DeNibbledIntById(id)
end

---Get modulator value
---@param mod CtrlrModulator
function Mods:GetValue(mod)
    if(DU:isNull(mod)) then p("GetValue:ERROR: Provided mod is nil. Aborting value update") return end
    local name = DM.UI.COMP:GetModName(mod)  -- Mods.VoiceMods.cbVoice1_Filter:getName()
    if(DU:isNull(name)) then p("GetValue:ERROR: Failed to fetch mod name. Aborting value update") return end
    local paramId = Mods:GetParamIdFromName(name,"_")
    if(DU:isNull(paramId)) then p(string.format("GetValue:ERROR: Failed to extract paramId from mod name[%s]. Aborting value update",tostring(name))) return end
    local value = DM.DUMPS.PresetDump:GetDumpValue_DeNibbledIntById(tonumber(paramId))
    if(DU:isNull(value)) then p(string.format("GetValue:ERROR: Failed to fetch dump value for name:[%s]  paramId[%s]. Aborting value update",tostring(name),tostring(paramId))) return end
    -- mod:setValue()
    mod:setValue(value,false)
    if(tostring(mod:getValue()) == tostring(value)) then
        p(string.format("GetValue:SUCCESS: Module Update  mod:[%s] paramId:[%s] value[%s]",tostring(name),tostring(paramId),tostring(value)))
        return true
    end

end