Building from source os works

[philipandag]

No description provided.

[philipandag]

I am not sure if such automatic test would make sense and how could we implement it well. It is possible that it will remain a manual-only.

[pietrushnic]

@philipandag I don't have the context, but manual tests must be avoided. Anything manual is a huge technical debt that holds scalability.

Are you saying we cannot build automatically from the source? Please explain what is the issue here.

[philipandag]

Are you saying we cannot build automatically from the source? Please explain what is the issue here.

The matter is that this test does not exactly test the Dasharo firmware, but the documentation. Unless the build steps are standardized, for example by using a universal build script (wip), doing such test automatically wouldn't make much sense. It should be performed manually to check if following the steps from the documentation works on a Fresh install of an OS to see if a user would succeed in building the firmare by following it on his own OS.

[pietrushnic]

The matter is that this test does not exactly test the Dasharo firmware, but the documentation.

I agree, but we should test everything we promise to deliver. Every procedure in the Dasharo documentation and terms of service should be tested; otherwise, we potentially do not deliver according to what we offer.

Naming in the test and description could be improved to accommodate that. Currently, it may be misleading to suggest we do different things than we do e.g., Build and Flash

Unless the build steps are standardized, for example by using a universal build script (wip), doing such test automatically wouldn't make much sense.

./build.sh supports most platforms, so I don't know what the precise issue is. If we are missing something, there should be an open blocker issue.

It should be performed manually to check if following the steps from the documentation works on a Fresh install of an OS to see if a user would succeed in building the firmare by following it on his own OS.

I still need to understand why. What you wrote doesn't seem to explain manual testing. I need to include some context here. Manual testing is evil and should be avoided. Your arguments are not enough to weave that rule.

[macpijan]

We have discussed this matter offline with @filipleple

It is good that such a need has been identified.

This looks more like automated validation of documentation, not firmware.

The plan could be:

• we could test each PR to docs and verify that all build guides work in QEMU
• we would need to parse documentation text (either .md or HTML) to get the releveant commands
  • we might need some updates so it's easier to parse them out
• we could compare hash of the produced file to the release one - this way, there is no need to test this binary on hardware

@filipleple Has identified at least one problem during his tests on one platform: Dasharo/dasharo-issues#1143

Another problem has been reported by Thierry not so long ago: Dasharo/dasharo-issues#1139

This shows that such tests are important for us, and can be automated.

@BeataZdunczyk CC

[pietrushnic]

@macpijan, there is also another option. Since the documentation format is static and the command repeats, we could use Jinja or another template language, and the input for generating documentation could also be input for CI. This is more complex but provides better abstraction; potentially, there will be fewer errors if the format or markdown changes over time.

[mkopec]

another solution: have a build system that abstract away all the commands, works for all releases, can be tested automatically, reduces the documentation to just a couple of commands, and is versioned outside of the source code repo