[request] Dynamic Source File Lists

[Suudy]

It appears the file lists in the Bender.yml are static. Perhaps some support to call a hook (Python? Shell Script? Command?) that returns a list of source files.

The specific use case is for the Xilinx IP generators where one might invoke Vivado, create a core, then export the list of files.

Perhaps syntax such as:

sources:
  - @(<command to execute>)
  - $(<command to execute>)

Not being a Rust developer myself, I'm unsure how to add the behavior myself and generate a pull request.

[andreaskurth]

Hi @Suudy, thanks for your suggestion!

I can see the use of constructing the source file list through commands. However, I think letting Bender execute an arbitrary command provided in a manifest file would be catastrophic from a security perspective, as it essentially forms a Shellcode threat.

Could your use case instead be implemented by adding an include command to the sources section of the Bender manifest? Upon encountering such an include command with a path argument, Bender would read the file at the path and insert each line in that file as source path in the manifest, at the position of the include command. The file to be included could then be generated through any command the user desires, but the user has to execute the command themselves (or at least the command has to be part of a build flow that the user executes, and it is not provided by some dependency).

[Suudy]

I’m not seeing a concern for shell code exploit. This is all executed within the context of a project which are under the control of developer. Unless you are concerned with remote repositories owned/maintained by third parties?

in that case, perhaps a global enable that has to be turned on to support execution of code?

This feature is desirable within bender because it automatically determines which components need dynamic file lists without a manual step. I’m not sure how a large project might navigate the generation of these include lists.

[Suudy]

An include list could work if we allow a command to be called as a separate step. And that could be limited by a global flag. Something like:

sources:
  - $(include <file name>)

commands:
  - “<command line>”
  - “<command line>”

[andreaskurth]

Unless you are concerned with remote repositories owned/maintained by third parties?

Yes. You would have to trust the developers of any dependency (and of its dependencies, etc.) to run any command on your machine.

This feature is desirable within bender because it automatically determines which components need dynamic file lists without a manual step. I’m not sure how a large project might navigate the generation of these include lists.

What makes the file list dynamic? Is RTL code generated from higher-level code? Can the RTL code files not be enumerated a priori and/or can they not be checked into a repository that forms a Bender package?

[Suudy]

The example I can give is in Vivado. What I typically do is write some TCL to generate a core and then use a TCL command to export the list of files. That list of files can change depending on the version of the core, whether it is configured for simulation or synthesis, etc. Normally I do a single TCL file for the whole project, then call individual TCL files for each generated core. I then gather all the sources files, calling get_files to get the source files.

While they could be enumerated in advance, the list can change depending on the context or even on the version of the core.

[Suudy]

One other example. We have two entities depending upon whether it is a debug component or not.

For example:

entity mycomp is
port
(
  clk : in std_logic;
  reset : in std_logic;

#ifdef DEBUG
  dbg_en : in std_logic;
#endif

  ...
);
begin
  constant DEBUG : boolean :=
#ifdef DEBUG
  TRUE;
#else
  FALSE;
#endif
end entity mycomp;

And the architecture:

architecture rtl of mycomp is
...
begin
  dbg_gen : if DEBUG generate
  begin
    ...
  end generate dbg_gen;
end architecture rtl;

This component, which may be multiple layers down in the hierarchy needs to be pre-processed. If I have to rely upon the top level to run commands to pre-process (or generate different file lists), then I've tightly coupled my top level with the components in my hierarchy.

[micprog]

Hi @Suudy,
We generally use bender solely for dependency management and the resulting source file list, and make use of targets to alter specifics of the defines set within the code or include different files. Regarding vivado IPs, we generally use tcl scripting to generate the Vivado specific IPs and simply source the bender generated source file script within, for example in PULP.