Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PC Engines Legacy support #213

Open
wants to merge 14 commits into
base: configs-rework
Choose a base branch
from
Open

PC Engines Legacy support #213

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
07b4b0a
platform-configs/include/pcengines.robot: add initial pcengines lib
mkopec Mar 4, 2024
4177c3e
platform-configs/pcengines-apu4.robot: add initial apu4 config
mkopec Mar 4, 2024
6ac8f6b
README.md: alphabetize DUT list
mkopec Mar 4, 2024
382f8fb
variables.robot: update apu4 config
mkopec Mar 4, 2024
a1c7643
README.md: add apu4 to DUT list
mkopec Mar 4, 2024
1e9c081
platform-configs/include/pcengines.robot: enable UEFI Shell tests
mkopec Mar 4, 2024
6e40eae
platform-configs/include/pcengines.robot: remove redundant definitions
mkopec Mar 4, 2024
0a0910e
platform-configs/include/pcengines.robot: adjust test scope
mkopec Mar 4, 2024
44b2236
platform-configs/include/pcengines.robot: remove unsupported tests
mkopec Mar 4, 2024
34b7b4f
README.md: split some docs to a subdirectory
mkopec Mar 4, 2024
cff516a
docs/adding-new-platforms.md: add file
mkopec Mar 4, 2024
9e3ed55
platform-configs/include/pcengines.robot: remove unsupported tests
mkopec Mar 4, 2024
5c8776c
platform-configs/include/pcengines.robot: add back USB write tests
mkopec Mar 4, 2024
1e1685b
initial support for PC Engines legacy testing
pietrushnic Mar 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
222 changes: 18 additions & 204 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,8 @@ appear here.
- [Running tests](#running-tests)
- [Running tests via wrapper](#running-tests-via-wrapper)
- [Running regressions tests](#running-regressions-tests)
* [QEMU workflow](#qemu-workflow)
- [Booting](#booting)
* [Contributing](#contributing)
- [Code](#code)
- [Issues](#issues)
* [Guidelines](#guidelines)
- [Pre-commit and CI checks](#pre-commit-and-ci-checks)
- [Code style](#code-style)
- [Keywords](#keywords)
- [Documentation](#documentation)
* [Useful refactoring tools](#useful-refactoring-tools)
* [RCS Talos II platforms](#rcs-talos-ii-platform)
* [Additional documents](#additional-documents)

## Lab architecture

Expand Down Expand Up @@ -69,15 +59,17 @@ different platform) may vary.

| Manufacturer | Platform | Firmware | $CONFIG |
|--------------|----------------------|--------------------------|----------------------------------------|
| NovaCustom | NV41MZ | Dasharo | `novacustom-nv41mz` |
| NovaCustom | NV41MB | Dasharo | `novacustom-nv41mb` |
| MSI | PRO Z690 A DDR5 | Dasharo | `msi-pro-z690-a-ddr5` |
| MSI | PRO Z690 A WIFI DDR4 | Dasharo | `msi-pro-z690-a-wifi-ddr4` |
| NovaCustom | NS50MU | Dasharo | `novacustom-ns50mu` |
| NovaCustom | NS70MU | Dasharo | `movacustom-ns70mu` |
| NovaCustom | NV41PZ | Dasharo | `novacustom-nv41pz` |
| NovaCustom | NS50PU | Dasharo | `novacustom-ns50pu` |
| NovaCustom | NS70MU | Dasharo | `movacustom-ns70mu` |
| NovaCustom | NS70PU | Dasharo | `novacustom-ns70pu` |
| MSI | PRO Z690 A WIFI DDR4 | Dasharo | `msi-pro-z690-a-wifi-ddr4` |
| MSI | PRO Z690 A DDR5 | Dasharo | `msi-pro-z690-a-ddr5` |
| NovaCustom | NV41MB | Dasharo | `novacustom-nv41mb` |
| NovaCustom | NV41MZ | Dasharo | `novacustom-nv41mz` |
| NovaCustom | NV41PZ | Dasharo | `novacustom-nv41pz` |
| PC Engines | apu3 | Dasharo | `pcengines-apu3` |
| PC Engines | apu4 | Dasharo | `pcengines-apu4` |
| Protectli | V1210 | Dasharo | `protectli-v1210` |
| Protectli | V1410 | Dasharo | `protectli-v1410` |
| Protectli | V1610 | Dasharo | `protectli-v1610` |
Expand All @@ -86,9 +78,9 @@ different platform) may vary.
| Protectli | VP4630 | Dasharo | `protectli-vp4630` |
| Protectli | VP4650 | Dasharo | `protectli-vp4650` |
| Protectli | VP4670 | Dasharo | `protectli-vp4670` |
| QEMU | Q35 | Dasharo (UEFI) | `qemu` |
| Raptor-CS | TalosII | Dasharo | `raptor-cs_talos2` |
| Raspberry Pi | RaspberryPi 3B | Yocto | `rpi-3b` |
| QEMU | Q35 | Dasharo (UEFI) | `qemu` |

## Getting started

Expand Down Expand Up @@ -193,146 +185,6 @@ platform config file.
FW_FILE=$FW_FILE DEVICE_IP=$DEVICE_IP RTE_IP=$RTE_IP CONFIG=$CONFIG ./scripts/regression.sh
```

## QEMU workflow

> Make sure to proceed with [Getting started section](#getting-started) first.

Many of the test and keywords can be tested in emulation environment. This
can greatly increase the development speed:
* there is no need to acquire hardware,
* there is no need to flash hardware, or resolve other hardware-related
problems,
* the boot time (and responsivness in general) is much faster.

### Booting

Following script assume that you have `OVMF_CODE.fd` and `OVMF_VARS.fd` in you
current working directory. If those binaries will not be found script will
download latest release of Dasharo (UEFI) for QEMU Q35.

If you want to use script in development workflow, most likely you have already built
Dasharo (UEFI) for QEMU Q35 according to
[this instruction](https://docs.dasharo.com/variants/qemu_q35/building-manual/).
In that case you would like to provide directory with Dasharo (UEFI) binaries as
environment variable (`DIR`).

You may also decide to not use graphics user interface for QEMU. In that case
choose mode `nographic`. If you run QEMU on a remote machine you may consider
to use mode `vnc` with default port for graphical output being `5900`.

Dasharo (UEFI) in QEMU can be started with:

```bash
./scripts/ci/qemu-run.sh graphic firmware
```

In this mode, a graphical QEMU window would popup, so you can observe the test
flow, or control it manually. The actual testing will happen over
serial, which is exposed via telnet. For more modes and options, please refer
to the script's help text.

You may also build customized Dasharo firmware for QEMU (e.g. with some Dasharo
options enabled or disabled). In such a case, please refer to:
* [Building Manual in Dasharo for QEMU documentation](https://docs.dasharo.com/variants/qemu_q35/building-manual/)
* [Development section in Dasharo for QEMU documentation](https://docs.dasharo.com/variants/qemu_q35/development/)

Refer to the [latest releases](https://github.com/Dasharo/edk2/releases/latest/)
to see which test have been proven to work on QEMU so far.

You may also refer to the `./scripts/ci/qemu-self-test.sh`, where we aim to
keep testing common keywords, to ensure of their correct operation.

## Contributing

### Code

* Install pre-commit hooks after cloning repository:

```bash
pre-commit install
```

### Issues

If you are certain that the issue is related to this repository, create issue
directly
[here](https://github.com/Dasharo/open-source-firmware-validation/issues/new/choose).
Otherwise, create an issue in
[dasharo-issues repisotory](https://github.com/Dasharo/dasharo-issues/issues/new/choose).

## Guidelines

A list of guidelines we shall follow during transition to improve the quality
of this repository. We start with getting rid of duplicated keywords, reducing
the size of `keywords.robot` file, and improving their overall quality.

There are other areas of interest that we will look into in the next steps
and add as guidelines:
* variables (use Python/YAML, not robot syntax),
* platform-configs (get rid of duplication, and unused data),
* separate test for different OS into different suites,
* prepare the OS for running test suite via some dedicated tools (e.g. Ansible),
rather than implementing keywords for that from scratch,
* reduce the number of unnecessary power events, so tests can finish quicker,
* improve overall code quality by enabling back more
[robocop checks we cannot pass right now](https://github.com/Dasharo/open-source-firmware-validation/blob/main/robocop.toml),
* To Be Continued.

### Pre-commit and CI checks

1. Make sure to use `pre-commit` locally. All pre-commit and other CI checks
must pass of course, prior requesting for review. Please check the status of
checks in your PR. If the failure is questionable, provide your arguments
for that, rather than silently ignoring this fact.

### Code style

1. It is automatically handled by
[robotidy](https://robotidy.readthedocs.io/en/stable/). The current rules
can be found
[here](https://github.com/Dasharo/open-source-firmware-validation/blob/main/.robotidy).

### Keywords

1. No new keywords in `keywords.robot` will be accepted
* new keywords must be placed in a logically divided modules, under `lib/`
directory
- see
[openbmc-test-automation](https://github.com/openbmc/openbmc-test-automation/tree/master/lib)
as a reference
* if you need to modify something in `keywords.robot`, you should create a new
module under `lib/`
* if you add new keyword module, you should review the `keywords.module` and
move related keywords there as well, if suitable
1. If keyword from keywords.robot can be reused or improved, do that instead
of creating a new one
- keyword duplication will not be accepted,
- you will be asked to use/improve existing keywords instead.
1. You are encouraged to use Python for more sophisticaed or complex keywords
(e.g. more convoluted data parsing and processing). We are not forced to use
RF for all keywords. Especially when it is simply easier to use Python.
1. For reading from terminal (no matter if it is Telnet, or SSH),
following keywords must be used:
- `Read From Terminal Until Prompt`
- `Read From Terminal Until`
- `Read From Terminal`
Usage of other keywords is prohibited. Whenever you modify a test/keyword,
you should rework it to use one of the above.
1. For writing into terminal, following keywords must be used:
- `Execute Command In Terminal`
- `Write Into Terminal`
- `Write Bare Into Terminal`
Usage of other keywords is prohibited. Whenever you modify a test/keyword,
you should rework it to use one of the above.
You should use `Execute Command In Terminal` unless you have a very good
reason not to. Thanks to that, your keyword will not leave floating output
in buffer to be received by another keywords, not expecting that.

### Documentation

* Each new (or modified) file, test, keyword, must have a `[Documentation]`
section.

## Useful refactoring tools

* [sherlock](https://github.com/MarketSquare/robotframework-sherlock)
Expand All @@ -341,50 +193,12 @@ and add as guidelines:
* [Renaming Test Cases](https://robotidy.readthedocs.io/en/stable/transformers/RenameTestCases.html)
* [Renaming Variables](https://robotidy.readthedocs.io/en/stable/transformers/RenameVariables.html)

## RCS Talos II platform

When running tests on Talos II platform use the following commands:

**WARNING** The support state of this platform in the `main` branch may vary.
We should have a single documentation for all platforms. This effort is tracked in
[this issue](https://github.com/Dasharo/open-source-firmware-validation/issues/112).

* For running a single test case:

```bash
robot -L TRACE -v device_ip:$DEVICE_IP -v config:raptor-cs_talos2 -v fw_file:$FW_FILE \
-v bootblock_file:$BOOTBLOCK_FILE -v zImage_file:$ZIMAGE_FILE -v pnor_file:$PNOR_FILE \
-t $TEST_CASE_ID $TEST_MODULE/$TEST_SUITE
```
## Additional documents

* For running single test suite:

```bash
robot -L TRACE -v device_ip:$DEVICE_IP -v config:raptor-cs_talos2 -v fw_file:$FW_FILE \
-v bootblock_file:$BOOTBLOCK_FILE -v zImage_file:$ZIMAGE_FILE -v pnor_file:$PNOR_FILE \
$TEST_MODULE/$TEST_SUITE
```

* For running single test module:

```bash
robot -L TRACE -v device_ip:$DEVICE_IP -v config:raptor-cs_talos2 -v fw_file:$FW_FILE \
-v bootblock_file:$BOOTBLOCK_FILE -v zImage_file:$ZIMAGE_FILE -v pnor_file:$PNOR_FILE \
./$TEST_MODULE
```

Parameters should be defined as follows:

* $DEVICE_IP - OBMC IP address (currently `192.168.20.9`),
* $FW_FILE - path to and name of the coreboot firmware file,
* $BOOTBLOCK_FILE - path to and name of the bootblock file,
* $ZIMAGE_FILE - path to and name of the zImage file,
* $PNOR_FILE - path to and name of the pnor file,
* $TEST_MODULE - name of the test module (i.e. `dasharo-compatibility`),
* $TEST_SUITE - name of the test suite (i.e. `coreboot-base-port`),
* $TEST_CASE_ID - ID of the requested to run test case (i.e. `CBP001.001`).
Note that after test case ID asterisk should be added. This is necessary due
to the construction of the flag `-t` (or `--test`)

You can also run tests with `-v snipeit:no` in order to skip checking whether
the platform is available on snipeit.
- [Adding new platforms](docs/adding-new-platforms.md) - Instructions for adding
support for new platforms
- [Contributing](docs/contributing.md) - Instructions for first-time
contributors
- [Raptor CS Talos II](docs/raptor-talos-2.md) - Documentation specific to the
Raptor Computing Systems Talos II mainboard
- [QEMU](docs/qemu.md) - Documentation for running tests in QEMU
81 changes: 81 additions & 0 deletions docs/adding-new-platforms.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
## Adding new platforms

Depending on what type of platform you're adding, the instructions here will
vary.

### Adding a brand new platform

- Create a new file for your mainboard in `platform-configs/`. For most
platforms this file will be called `[platform-vendor]-[platform-model].robot`.
- Add the following at the top of your platform config:

```robot
*** Settings ***
Resource default.robot
```

- Copy the contents of `include/default.robot` to your platform config
- Modify the file for your platform:
- Modify the settings appropriately for your mainboard
- Remove any unmodified lines - they will be sourced from `default.robot`
- Add the platform configuration to `variables.robot:
- Create a new configuration of RTE, if you are using one, e.g.:

```robot
&{RTE11}= ip=192.168.10.174 cpuid=02c000426621f7ea pcb_rev=1.0.0
... platform=apu4 board-revision=4d env=prod
... platform_vendor=PC Engines firmware_type=UEFI
```

- Add the RTE to the list:

```
@{RTE_LIST}= &{RTE05}
... &{RTE06} &{RTE07} &{RTE08} &{RTE09} &{RTE10}
... &{RTE11}
```

- Do the same for any modules installed in the platform
- Create a new CONFIG contaning the RTE and modules used for testing, and
append it to the list:

```
@{CONFIG04}= &{RTE11} &{SSD06} &{CARD06} &{USB03}
... &{MODULE06} &{ADAPTER01} &{MODULE10}

@{CONFIG_LIST}= @{CONFIG01} @{CONFIG02} @{CONFIG03} @{CONFIG04}
```

- Run a simple test to verify the config is working correctly - for example
UEFI Shell:

```
robot -v snipeit:no -L TRACE -v rte_ip:192.168.10.174 -v device_ip:0.0.0.0 -v config:pcengines-apu4 dasharo-compatibility/uefi-shell.robot
```

### Adding new variant of an existing platform

Some boards come in multiple variants, where the majority of properties and
features can be shared. For these cases, we have shared "base" configs in
`platform-configs/include/`. This way we don't need to copy-paste entire config
files, making maintenance easier. In this example we'll be adding a new PC
Engines apu variant, using an existing pcengines base config:

- Create a config file in `platform-configs` for your platform
- Add the following to your platform config

```
*** Settings ***
Resource include/pcengines.robot
```

- Add variant-specific settings for your platform - in this case, only the
SMBIOS product name field:

```
*** Variables ***
${DMIDECODE_PRODUCT_NAME}= apu4
```

- Proceed with adding the platform to `variables.robot` as per
[Adding a brand new platform](#adding-a-brand-new-platform).
Loading
Loading