building modules: using python scripts

[pjdobrowolski]

Added function for CMake to build modules using python scripts
Now using -l switch triggers CMakeLists for module passed after switch.
f.e.
xtensa-build-zephyr.py -u mtl -k ../keys/mtl_private_key.pem -l smart_amp_test

We can build more then one module just by adding more module names.

---------------------> update 08.02.2024 <--------------------------

I've separate building script from xtensa, I also create utils for generic functions used in that spaghetti script. More to come in next commits.

---------------------> update 14.02.2024 <--------------------------

To build modules using pythons were created several tools making
building easier.
For future deployment there is a tool extracting headers descibed in
manifest and creating zip file from them.

modules are build now:

python scripts/lmdk/libraries_build.py -l <modules_names> -k <key_directory>
python lmdk/scripts/libraries_build.py -l <modules_names> -k <key_directory>
To create deployment pack of headers use
python scripts/lmdk/header_pack.py
python lmdk/scripts/header_pack.py
we can build multiple modules by adding them one after another.

---------------------> update 19.04.2024 <--------------------------
Scripts updated with removed env copy part from extensa building python scripts.
Scripts moved to lmdk directory to not create multiple locations for the same application.

[marc-hb]

I understand how this can be useful but please don't embed this new, unrelated loop inside xtensa-build-zephyr.py. Isn't a separate build one of the most important LMDK design choices?

Also, CMake's GLOB is generally not recommended but this case looks like a good exception and a reason to implement this with a top-level CMake instead of Python. Multiple ExternalProject_Add() will build all libraries in parallel: much faster.

[pjdobrowolski]

I understand how this can be useful but please don't embed this new, unrelated loop inside xtensa-build-zephyr.py. Isn't a separate build one of the most important LMDK design choices?

You are right. I wasn't sure what community would be say about that.

How about deployment - release build? It is being prepared by python I presume. Modules binaries and exported headers should be gathered in the same place as sof binary. Can it be integrated inside python scripts?

[lgirdwood]

Separate helper script is fine, I assume we may need to share logic from the baseFW script ? if so how ?

[pjdobrowolski]

From my perspective python script for building base_fw needs refactor. It is too long and it has to many functions so it might be a good excuse for clean up, add extra functions etc.

[marc-hb]

Yes, xtensa-build-zephyr.py is ugly and no one likes it. It's ugly because it does all the ugly things that no beautiful person wants to do:

• configuring the Cadence toolchains with their ugly and evil environment variables.
• building the ugly rimage and others
• configuring rimage
• iterating over the list of platforms
• "installing" /lib/firmware/ for Linux. Creating symbolic links
• running west update. That one was a mistake but I lost that fight. Never use -u.
• computing checksums for reproducible builds
• taking out the trash with -p. Don't use that either, delete the build directories instead.

I won't name anyone but I even know people who refuse to use it (and then wonder why their firmware is sligthly different from everyone else's... I digress)

However, all these ugly jobs have one very important thing in common: it's the list of everything that is required to build the "base firmware" - and it's very convenient to have all these in a single place. So there is a tiny bit of beautiful consistency!

So let's draw a line and let's not overload that boat with things required by a new, separate and totally different and independent build.

Divide and conquer - one ugly build at a time! https://mesonbuild.com/Design-rationale.html

A software developer's most important tool is the editor. If you talk to coders about the editors they use, you are usually met with massive enthusiasm and praise. You will hear how Emacs is the greatest thing ever or how vi is so elegant or how Eclipse's integration features make you so much more productive. You can sense the enthusiasm and affection that the people feel towards these programs.

The second most important tool, even more important than the compiler, is the build system.

Those are pretty much universally despised.

The most positive statement on build systems you can usually get (and it might require some coaxing) is something along the lines of well, it's a terrible system, but all other options are even worse.

Immediately contradicting myself a little bit: if there are some specific bits of xtensa-build-zephyr.py that you'd want to re-use then maybe we could split them into some new and shared xtensa-build-library.py or something. But let's not make this giant script even bigger with stuff from a totally separate build system.

Or maybe not because “a little copying is better than a little dependency” (Go proverb)

[pjdobrowolski]

I agree. There are a lot of useful things. I suggest to separate building rimage, deployment to new files, document what are they doing and import them to main script. It would take some time but it be easier to understand and use in future.

[softwarecki]

.exe?

[marc-hb]

So let's draw a line and let's not overload that boat with things required by a new, separate and totally different and independent build.

[pjdobrowolski]

@softwarecki if .exe is not available use binary version.

[pjdobrowolski]

@marc-hb I removed building from main script.

done

[lgirdwood]

@pjdobrowolski what the plan to integrate lmdk builds into top level src/audio for module that support lmdk ? Having this a separate stand alone directory is a bit confusing.

[marc-hb]

ping. This serves a purpose where you copied it from but here the env_change thing does nothing, doesn't it?

stale review

[marc-hb]

I don't think that it works for programs out of PATH.

print(shutil.which("C:\\Users\\paweldox\\zephyrproject\\build-rimage\\rimage"))
None

Right, so using the path argument works:

print(shutil.which("rimage", path="C:/Users/paweldox/zephyrproject/build-rimage/"))

=> 'C:/Users/paweldox/zephyrproject/build-rimage/rimage.EXE'

[pjdobrowolski]

Yes, I've checked it on linux and windows to be sure.

[marc-hb]

I think you missed my point. I tested the path argument on Windows. You don't need is_file().

[marc-hb]

You stripped down this function to the point where... it does not print the command executed, which was its entire purpose!?

But there are some bits left that don't do anything...

I don't know why you started by blindly copying huge parts of xtensa-build-zephyr.py instead of just the bits you need one by one. After way too many force-pushes there's extremely little code copied from xtensa-build-zephyr.py now. This has been been really difficult to make sense of and time consuming to review.

[pjdobrowolski]

Look at comment to PR. I've started that script as a part of xtensa-build-zephyr.py . After your's suggestions I've separated parts I needed to external pythons scripts without changes from original spaghetti code.

It all started as simple function in xtensa-build-zephyr.py. I just followed suggestions from you and it looks as it looks.

[marc-hb]

Process digressions aside: what does this execute_command() function do now? It does not do what what the obsolete pydoc says it does. If it does nothing any more, why is it kept?

[kv2019i]

This seems almost ready to go, can you @pjdobrowolski check the last few comments in libraries_build.py script

[pjdobrowolski]

@kv2019i it would be nice, because I would move to loadable up_down_mixer.

[kv2019i]

@pjdobrowolski Can you check the CI fails? There seems to be a lot of failures, e.g. https://github.com/thesofproject/sof/actions/runs/8780400627/job/24090240938?pr=8828

[pjdobrowolski]

Cleaned up.

[lgirdwood]

This PR LGTM, but my big sticking point before v2.10 is to get the lmdk directory merged into existing modules, scripts, include directories so we are not duplicating and/or causing confusion. We are probably at the stage today where this is easy enough todo.

[kv2019i]

Ready to be merged once require CI is ok. @pjdobrowolski can you check the issue with quickbuild?

[lgirdwood]

Ready to be merged once require CI is ok. @pjdobrowolski can you check the issue with quickbuild?

Hang on, @pjdobrowolski and I have an update thats pending some internal discussion.

[marc-hb]

Some weird things still present in the Python code.

[marc-hb]

Process digressions aside: what does this execute_command() function do now? It does not do what what the obsolete pydoc says it does. If it does nothing any more, why is it kept?

[marc-hb]

I think you missed my point. I tested the path argument on Windows. You don't need is_file().

[lyakh]

no need to put this under #ifndef SOF_MONOLITHIC_BUILD and it makes little sense that way - if it isn't defined, then we define it... From this commit I don't see any other places where this can be defined, so this doesn't seem to protect against redefinition.