Dynamic generators

[dirkzerwas]

BEGINRELEASENOTES

• Transform the "handwritten" loading of the generator modules, the writing of the datacards and the executable shell script into a dynamic loading triggered by the input name of the generator

ENDRELEASENOTES

[dirkzerwas]

@tmadlener : converting Generators.py I had to change the "import" syntax/structure of the Sherpa.py etc files. Is that a side effect of dynamic loading through importlib?

[tmadlener]

Suggested change

	generator = importlib.import_module(generatorName)
	generator = importlib.import_module(f".{generatorName}", package=__package__)

Should technically preserve enough of the package context to make things work without changing the generator .py files.

Alternatively the following might also work:

generator = importlib.import_module(f"Generators.{generatorName}")

[dirkzerwas]

generator = importlib.import_module(f"Generators.{generatorName}")
does not seem to work, will check why and try the other version

[tmadlener]

Ah sorry, that was my mistake. I didn't realize this happens inside the Generators package / module. In that case, I think only the other option works (i.e. the one with the . relative import, specifying the package dynamically as well)

[dirkzerwas]

I was imprecise, with a more detailed try structure:
` def initialize_generators(self):

    for generatorName in self.generator_list:
        # get the module
        try:
            generator   = importlib.import_module(f"Generators.{generatorName}")
            # get the ClassObject
            try:
                generatorClass = getattr(generator,generatorName)
                # execute the object
                try:
                    generatorObj = generatorClass(self.proc_info, self.settings)
                    #writing file
                    print("execution ok")
                    generatorObj.write_file()
                    #writing key4hep file
                    print("wrote file")
                    generatorObj.write_key4hepfile()
                except:
                    print(f"Requested Generator: {generatorName} object could not be executed, output files not written for {self.proc_info.get('_proclabel')}")
            except:
                print(f"Requested Generator: {generatorName} class could not be loaded {self.proc_info.get('_proclabel')}")
        except:
            print(f"Requested Generator: {generatorName} could not be configured for {self.proc_info.get('_proclabel')}")

`

I get:
Requested Generator: Sherpa object could not be executed, output files not written for e- e+ -> mu- mu+.....
That would mean that:

• import worked
• getting the class worked
• but the execution of the class failed?
  (or could the problem be hidden somewhere else?

[dirkzerwas]

If I change the import structure of Sherpa.py from:
from .GeneratorBase import GeneratorBase
from . import SherpaProcDB
to
from .GeneratorBase import GeneratorBase
from .SherpaProcDB import SherpaProcDB

it seems to work better, will check with all gens and the CI

[dirkzerwas]

It actually works for all the generators, but why is:
from .MadgraphProcDB import MadgraphProcDB
needed when
from Particles import Particle as part
seems to work?

[tmadlener]

Most likely Particles is visible from the PYTHONPATH because the top level python is available. The full (absolute) import would be Generators.MadgraphProcDB.

The difference between

from . import SherpaProcDB`

and

from .SherpaProcDB import SherpaProcDB

is that for the first case you import the SherpaProcDB module from the current scope, whereas in the second case you import the SherpaProcDB object (class, function, variable) from the local module SherpaProcDB. Hence the following two things are equivalent:

from . import SherpaProcDB
sherpa = SherpaProcDB.SherpaProcDB()

and

from .SherpaProcDB import SherpaProcDB
sherpa = SherpaProcDB()