From ebe0e094ff92649d0bda1988b0d1c1b08403aea4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Malte=20Grebe-L=C3=BCth?= <104556762+belagertem@users.noreply.github.com> Date: Tue, 15 Oct 2024 11:20:38 +0200 Subject: [PATCH] no subdirectories command line option added (#214) added command line option to not create timestamped directories for test runs. # Checklist The following aspects have been respected by the author of this pull request, confirmed by both pull request assignee **and** reviewer: * Adherence to coding conventions * [x] Pull Request Assignee * [x] Reviewer * Adherence to javadoc conventions * [x] Pull Request Assignee * [x] Reviewer * Changelog update (necessity checked and entry added or not added respectively) * [x] Pull Request Assignee * [x] Reviewer * README update (necessity checked and entry added or not added respectively) * [x] Pull Request Assignee * [x] Reviewer * config update (necessity checked and entry added or not added respectively) * [x] Pull Request Assignee * [x] Reviewer * SDCcc executable ran against a test device (if necessary) * [x] Pull Request Assignee * [x] Reviewer --- CHANGELOG.md | 1 + README.md | 129 ++++++++++++------ .../com/draeger/medical/sdccc/TestSuite.java | 12 +- .../configuration/CommandLineOptions.java | 15 ++ .../sdccc/configuration/TestRunConfig.java | 26 ++-- 5 files changed, 124 insertions(+), 59 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fedbe4f0..59f20e76 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - support for Kotlin - preconditions which observe mdib changes during the test run - storing of IP addresses of inbound messages in the database +- a command line parameter to not create subdirectories in the test run directory ### Changed diff --git a/README.md b/README.md index 40ea1e9c..56615c34 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,15 @@ # SDCcc -This test tool aims to evaluate the conformity of medical devices with selected parts of "ISO/IEEE 11073-20702", +This test tool aims to evaluate the conformity of medical devices with selected parts of "ISO/IEEE 11073-20702", "ISO/IEEE 11073-10207", "OASIS DPWS 1.1" and "ISO/IEEE 11073-20701". ## Introduction of the Project and Test approach + The test tool assumes the Role of an SDC Service Consumer to connect to the device under test (DUT) and interacts with the device during the test run. To use the test tool, a one-to-one connection between the test tool and the DUT is required, i.e. via an isolated network where only the DUT and the test tool are connected. -All offered reports and streams are subscribed to, and all inbound and outbound messages exchanged are stored in a database. +All offered reports and streams are subscribed to, and all inbound and outbound messages exchanged are stored in a +database. [More information on SDC.](https://en.wikipedia.org/wiki/IEEE_11073_service-oriented_device_connectivity) @@ -26,15 +28,17 @@ The test run is broadly divided into four phases. During the first phase, the te remote procedure call declared by the DUT, except for calls to the SetService. If no errors occur and the provider behaves correctly, the message `SDC Basic Messaging Check completed successfully.` is printed at the end of phase one, otherwise the message says `SDC Basic Messaging Check completed with errors.`. -During the second phase, the direct tests are performed. The third phase checks whether all preconditions for the invariant -tests are fulfilled. If this is not the case, the corresponding messages are triggered or manipulations are performed -before the connection is terminated. The invariant tests are executed in the fourth phase. +During the second phase, the direct tests are performed. The third phase checks whether all preconditions for the +invariant tests are fulfilled. If this is not the case, the corresponding messages are triggered or manipulations are +performed before the connection is terminated. The invariant tests are executed in the fourth phase. ## Test Consumer Configuration + The test consumer can be configured for the test run by modifying the *config.toml* file. It is located in the *configuration* directory. ### TLS Configuration + TLS is **mandatory** to use the test tool. All files required for TLS must be located in the same directory. @@ -60,6 +64,7 @@ A specific naming is required for all tls related files: | CA certificate | ca_certificate.pem | Different combinations can be used to establish a connection: + * keystore and truststore * keystore and ca_certificate * participant_public, participant_private and ca_certificate @@ -69,13 +74,16 @@ Optionally the TLS protocol versions to be enabled can be specified as well as t the TLS protocol. An example can be found in configuration/config.toml, the values there are also the default values. ### Network setup + To select the network interface that should be used, the interface address can be set under + ``` [SDCcc.Network] InterfaceAddress="interfaceAddress"` ``` The maximum waiting time in seconds to find and connect to the target device. + ``` [SDCcc.Network] MaxWait=timeInSeconds @@ -83,12 +91,14 @@ MaxWait=timeInSeconds The time to live (the number of routers an IP packet may pass before it is discarded) of multicast packets used for Discovery defaults to 128. When other values are needed, it can be configured using the following option + ``` [SDCcc.Network] MulticastTTL=196 ``` ### Target Device (DUT) configuration + In order for the test tool to connect to the DUT, appropriate filter criteria have to be set. It is possible to combine the following filter criteria: @@ -100,61 +110,76 @@ It is possible to combine the following filter criteria: - DeviceLocationRoom - DeviceLocationBed -All of them are optional. +All of them are optional. In case that all of them are not set, the first device encountered will be connected to. -In case at least one of them is set, +In case at least one of them is set, all the given filter criteria have to be fulfilled for initiating a connection. For example the configuration + ``` [SDCcc.Consumer] DeviceEpr="urn:uuid:857bf583-8a51-475f-a77f-d0ca7de69b11" ``` + will make only those devices match during discovery that have the EPR "urn:uuid:857bf583-8a51-475f-a77f-d0ca7de69b11", the configuration + ``` [SDCcc.Consumer] DeviceEpr="urn:uuid:857bf583-8a51-475f-a77f-d0ca7de69b11" DeviceLocationBed="bed32" ``` + only those that have the EPR "urn:uuid:857bf583-8a51-475f-a77f-d0ca7de69b11" and the bed "bed32" in the location query, the configuration + ``` [SDCcc.Consumer] DeviceLocationBed="bed32" ``` + only those that have the bed "bed32" in the location query, and the configuration + ``` [SDCcc.Consumer] ``` + will make all devices match during discovery. ### Manipulation API + The test tool uses *T2IAPI* version `4.1.0`. The *T2IAPI* is required for some test cases to put the DUT in a certain state, or to trigger a certain behavior. When using SDCcc with automated manipulations, it must be ensured that the same -version of *T2IAPI* is used for the test execution by both parties. It must also be ensured that the device's -manipulations are implemented according to the descriptions in the T2IAPI sources. Further information can be found +version of *T2IAPI* is used for the test execution by both parties. It must also be ensured that the device's +manipulations are implemented according to the descriptions in the T2IAPI sources. Further information can be found in the changelog. If an automated manipulation is not possible, the fallback manipulation takes effect. For each manipulation a fallback manipulation must be provided. A graphical user interface is displayed and the user can confirm that the -manipulation was performed manually or reject performing it. The fallback manipulation can also be performed via the command-line interface, +manipulation was performed manually or reject performing it. The fallback manipulation can also be performed via the +command-line interface, when *GraphicalPopups* is disabled: + ``` [SDCcc] GraphicalPopups=false ``` -To see which requirement test requires which manipulation, see Section **Which Manipulation is required for which test**. +To see which requirement test requires which manipulation, see Section +**Which Manipulation is required for which test**. ### CI Mode + In order to prevent the need for user interaction, the test tool can be run in continuous integration mode. This mode can be enabled under + ``` [SDCcc] CIMode = true ``` + A test fails in CI mode if this test requires manipulations and these have not been automated by the test engineer. ### Further Configuration Options @@ -167,7 +192,6 @@ TestExecutionLogging=true TestExecutionLogging can be enabled, to get more information on which test case is currently executed. When enabled, SDCcc will log when a test case for a requirement has started and finished. - ``` [SDCcc] EnableMessageEncodingCheck=true @@ -176,7 +200,7 @@ SummarizeMessageEncodingErrors=true EnableMessageEncodingCheck defaults to true and allows the user to control whether SDCcc checks the encoding and mimeType specified in the messages received from the DUT. Note that disabling the MessageEncodingCheck -causes SDCcc to decode all messages as UTF-8. +causes SDCcc to decode all messages as UTF-8. SummarizeMessageEncodingErrors defaults to true and allows the user to control how encoding and mimeType problems are presented during an SDCcc TestRun. Note that devices that have encoding problems usually produce these errors @@ -192,30 +216,34 @@ Some test cases require individual parameters that can be overwritten in the *te [TestParameter] Biceps547TimeInterval=5 ``` -When running biceps:5-4-7 tests the Biceps547TimeInterval parameter is used to pause between the SetMetricStatus -manipulation calls with a default of 5 seconds. The report that follows a SetMetricStatus manipulation is expected + +When running biceps:5-4-7 tests the Biceps547TimeInterval parameter is used to pause between the SetMetricStatus +manipulation calls with a default of 5 seconds. The report that follows a SetMetricStatus manipulation is expected within the specified seconds. ## Running SDCcc + The following command line options are supported by the test tool, the first two need to be provided. -| **Option** | **Short** | **Argument** | **Required** | -|----------------------|-----------|------------------------------------------------------------------------------------------------------|--------------| -| config | c | path to the *config.toml* | yes | -| testconfig | t | path to the *test_configuration.toml* | yes | -| testparam | p | path to the *test_parameter.toml* | no | -| device_epr | de | the epr of the target provider, overrides setting from configuration if provided | no | -| device_facility | fac | the facility of the target provider, overrides setting from configuration if provided | no | -| device_building | bldng | the building of the target provider, overrides setting from configuration if provided | no | -| device_point_of_care | poc | the point of care of the target provider, overrides setting from configuration if provided | no | -| device_floor | flr | the floor of the target provider, overrides setting from configuration if provided | no | -| device_room | rm | the room of the target provider, overrides setting from configuration if provided | no | -| device_bed | bed | the bed of the target provider, overrides setting from configuration if provided | no | -| ipaddress | ip | ip address of the adapter to use for communication, overrides setting from configuration if provided | no | -| test_run_directory | d | base directory to store test runs in, creates a timestamped SDCcc run | no | -| file_log_level | fll | log level to be used for the log file being created, e.g. DEBUG, defaults to INFO | no | +| **Option** | **Short** | **Argument** | **Required** | +|------------------------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------| +| --config | -c | path to the *config.toml* | yes | +| --testconfig | -t | path to the *test_configuration.toml* | yes | +| --testparam | -p | path to the *test_parameter.toml* | no | +| --device_epr | -de | the epr of the target provider, overrides setting from configuration if provided | no | +| --device_facility | -fac | the facility of the target provider, overrides setting from configuration if provided | no | +| --device_building | -bldng | the building of the target provider, overrides setting from configuration if provided | no | +| --device_point_of_care | -poc | the point of care of the target provider, overrides setting from configuration if provided | no | +| --device_floor | -flr | the floor of the target provider, overrides setting from configuration if provided | no | +| --device_room | -rm | the room of the target provider, overrides setting from configuration if provided | no | +| --device_bed | -bed | the bed of the target provider, overrides setting from configuration if provided | no | +| --ipaddress | -ip | ip address of the adapter to use for communication, overrides setting from configuration if provided | no | +| --test_run_directory | -d | base directory to store test runs in, creates a timestamped SDCcc run | no | +| --no_subdirectories | -ns | if set to "true", no directories are created in the directory configured with test_run_directory. The configured directory must be empty if no_subdirectories is set to "true" | no | +| --file_log_level | -fll | log level to be used for the log file being created, e.g. DEBUG, defaults to INFO | no | ### Enabling Tests + The *test_configuration.toml* file contains the identifiers of all implemented requirement tests. It is located in the configuration directory. These are grouped according to the SDC standards. Replaced or derived requirements are marked with an underscore, e.g. instead of BICEPS `R0025` it would be `R0025_0`. @@ -224,43 +252,50 @@ To disable an unwanted test for the test run, it can simply be set to *false*. E.g.: to disable BICEPS R0021 set + ``` [BICEPS] R0021=true ``` + to + ``` [BICEPS] R0021=false ``` ### Where to find results + After a test run, a folder *testruns* is created in the same directory as the executable by default. The directory can be changed with the command line argument `test_run_directory` (see Section **Running SDCcc**) and the results are -saved in that folder. Each test run gets its own directory named according to the scheme -*SDCcc-Testrun_YYYY-MM-DDTHH-mm-SS*. Inside this directory there is a subdirectory *Database* and three files -*SDCcc.log*, *TEST-SDCcc_direct.xml* and *TEST-SDCcc_invariant.xml*. *Database* is a database in which all messages -exchanged during the test run are recorded. *SDCcc.log* is the complete log file of the test run +saved in that folder. Each test run has its own directory, which is named according to the following scheme +*SDCcc-Testrun_YYYY-MM-DDTHH-mm-SS*, unless the command line argument `no_subdirecotries` is set, in which case the test +run files are saved directly in the *testruns* folder. Inside this directory there is a subdirectory *Database* and +three files *SDCcc.log*, *TEST-SDCcc_direct.xml* and *TEST-SDCcc_invariant.xml*. *Database* is a database in which all +messages exchanged during the test run are recorded. *SDCcc.log* is the complete log file of the test run in accordance to the log file log level. The test results are located in the two result XML files, *TEST-SDCcc_direct.xml* for the direct tests and *TEST-SDCcc_invariant.xml* for the invariant tests. ## Prerequisites for building SDCcc + Maven >= 3.8.1 and Java 17 are required to build the project. ## Limitations + The test tool has the following limitations. If the DUT falls under these limitations, the test tool **cannot** be used. Where it is possible to detect when a DUT falls under these limitations, SDCcc's test cases are designed to fail in this case in order to minimize the risk of such an invalid application going unnoticed. [General] -| **Limitation** | -|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| The SDCcc tool does not support the following HTTP Headers in test cases which use messages stored in the database: | -| The ArchiveService is not supported and will be ignored by the test tool. | -| Safe data transmission (MDPWS Ch. 9) is not supported | -| Multipart/Related Content-Type is not supported | +| **Limitation** | +|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| The SDCcc tool does not support the following HTTP Headers in test cases which use messages stored in the database: | +| The ArchiveService is not supported and will be ignored by the test tool. | +| Safe data transmission (MDPWS Ch. 9) is not supported | +| Multipart/Related Content-Type is not supported | [MDPWS] @@ -271,6 +306,7 @@ this case in order to minimize the risk of such an invalid application going unn | ArchiveService messages are not supported | R0006 | ## Which Manipulation is required for which test + [BICEPS] | **Requirement** | **T2IAPI Manipulation** | @@ -338,17 +374,20 @@ this case in order to minimize the risk of such an invalid application going unn SDCcc's exitCode should be interpreted as follows: -| **ExitCode** | **Semantics** | -|--------------|----------------------------------------------------------------------------------------------------------------------| -| 0 | Success - Test run execution was successful and the device under test satisfies all tested requirements | -| 1 | Failure - Test run execution was successful, but the device under test violated requirements | -| 2 | Error - Test run execution was not successful | +| **ExitCode** | **Semantics** | +|--------------|---------------------------------------------------------------------------------------------------------| +| 0 | Success - Test run execution was successful and the device under test satisfies all tested requirements | +| 1 | Failure - Test run execution was successful, but the device under test violated requirements | +| 2 | Error - Test run execution was not successful | ## Notices + SDCcc is not intended for use in medical products, clinical trials, clinical studies, or in clinical routine. ### ISO 9001 + SDCcc was not developed according to ISO 9001. ## License + [MIT](https://choosealicense.com/licenses/mit/) diff --git a/sdccc/src/main/java/com/draeger/medical/sdccc/TestSuite.java b/sdccc/src/main/java/com/draeger/medical/sdccc/TestSuite.java index a37117ac..23b17b9e 100644 --- a/sdccc/src/main/java/com/draeger/medical/sdccc/TestSuite.java +++ b/sdccc/src/main/java/com/draeger/medical/sdccc/TestSuite.java @@ -754,7 +754,17 @@ public static void runWithArgs( throws IOException { // setup logging final var testRunDir = TestRunConfig.createTestRunDirectory( - cmdLine.getTestRunDirectory().orElse(null)); + cmdLine.getTestRunDirectory().orElse(null), cmdLine.getNoSubdirectories()); + if (cmdLine.getNoSubdirectories() && testRunDir.isDirectory()) { + final var files = testRunDir.list(); + if (files != null && files.length > 0) { + throw new RuntimeException(String.format( + "The specified test run directory %s was not empty, " + + "although the command line option --no_subdirectories was set to true. Please make sure that " + + "you use --test_run_directory to configure an empty test run directory.", + testRunDir)); + } + } final var logConfig = LoggingConfigurator.loggerConfig(testRunDir, cmdLine.getFileLogLevel()); checkLogConfig(logConfig); diff --git a/sdccc/src/main/java/com/draeger/medical/sdccc/configuration/CommandLineOptions.java b/sdccc/src/main/java/com/draeger/medical/sdccc/configuration/CommandLineOptions.java index de0448f2..ec2ce56d 100644 --- a/sdccc/src/main/java/com/draeger/medical/sdccc/configuration/CommandLineOptions.java +++ b/sdccc/src/main/java/com/draeger/medical/sdccc/configuration/CommandLineOptions.java @@ -43,6 +43,7 @@ public class CommandLineOptions { private static final String DEVICE_LOCATION_BED = "device_bed"; private static final String IP_ADDRESS = "ipaddress"; private static final String TEST_RUN_DIRECTORY = "test_run_directory"; + private static final String NO_SUBDIRECTORIES = "no_subdirectories"; private static final String FILE_LOG_LEVEL = "file_log_level"; private final Path configPath; private final Path testConfigPath; @@ -56,6 +57,7 @@ public class CommandLineOptions { private final String deviceBed; private final String ipAddress; private final String testRunDirectory; + private final Boolean noSubdirectories; private final Level fileLogLevel; /** @@ -103,6 +105,7 @@ public CommandLineOptions(final String[] commandLineArguments) { this.deviceBed = cmd.getOptionValue(DEVICE_LOCATION_BED); this.ipAddress = cmd.getOptionValue(IP_ADDRESS); this.testRunDirectory = cmd.getOptionValue(TEST_RUN_DIRECTORY); + this.noSubdirectories = Boolean.parseBoolean(cmd.getOptionValue(NO_SUBDIRECTORIES)); this.fileLogLevel = Level.toLevel(cmd.getOptionValue(FILE_LOG_LEVEL), Level.INFO); } @@ -186,6 +189,14 @@ private Options setupOptions() { testRunDirectoryOpt.setRequired(false); options.addOption(testRunDirectoryOpt); } + { + final String description = + "If set to true, no directories are created in the directory configured with test_run_directory"; + final var noSubdirectoriesOpt = new Option("ns", NO_SUBDIRECTORIES, true, description); + noSubdirectoriesOpt.setRequired(false); + noSubdirectoriesOpt.setType(Boolean.class); + options.addOption(noSubdirectoriesOpt); + } { final String description = "The log level to be used for the log file. e.g. DEBUG . The default is INFO."; final var fileLogLevelOpt = new Option("fll", FILE_LOG_LEVEL, true, description); @@ -258,6 +269,10 @@ public Optional getTestRunDirectory() { return Optional.ofNullable(testRunDirectory); } + public Boolean getNoSubdirectories() { + return noSubdirectories; + } + public Level getFileLogLevel() { return this.fileLogLevel; } diff --git a/sdccc/src/main/java/com/draeger/medical/sdccc/configuration/TestRunConfig.java b/sdccc/src/main/java/com/draeger/medical/sdccc/configuration/TestRunConfig.java index 8842663f..dfabeeb2 100644 --- a/sdccc/src/main/java/com/draeger/medical/sdccc/configuration/TestRunConfig.java +++ b/sdccc/src/main/java/com/draeger/medical/sdccc/configuration/TestRunConfig.java @@ -1,6 +1,6 @@ /* * This Source Code Form is subject to the terms of the MIT License. - * Copyright (c) 2023 Draegerwerk AG & Co. KGaA. + * Copyright (c) 2023-2024 Draegerwerk AG & Co. KGaA. * * SPDX-License-Identifier: MIT */ @@ -46,17 +46,22 @@ protected void defaultConfigure() { * Also creates the 'testruns' parent directory if it doesn't exist when using default directory. * * @param baseDirectory directory to create test run dirs in, working directory if null + * @param noSubdirectories if true, no test run dirs are created * @return File pointing to the new directory. * @throws RuntimeException in case directory could not be created or already exists */ - public static File createTestRunDirectory(final @Nullable String baseDirectory) { - // create test run dir and bind it as config parameter - final String testRunTimestamp = ZonedDateTime.now(ZoneOffset.UTC) - .format(DateTimeFormatter.ISO_DATE_TIME) - .replace(":", "-"); + public static File createTestRunDirectory(final @Nullable String baseDirectory, final Boolean noSubdirectories) { + if (noSubdirectories) { + return createTestRunDirectory(baseDirectory, ""); + } else { + // create test run dir and bind it as config parameter + final String testRunTimestamp = ZonedDateTime.now(ZoneOffset.UTC) + .format(DateTimeFormatter.ISO_DATE_TIME) + .replace(":", "-"); - final String testRunName = "SDCcc_Testrun_" + testRunTimestamp; - return createTestRunDirectory(baseDirectory, testRunName); + final String testRunName = "SDCcc_Testrun_" + testRunTimestamp; + return createTestRunDirectory(baseDirectory, testRunName); + } } private static File createTestRunDirectory(final @Nullable String baseDirectory, final String testRunName) { @@ -69,11 +74,6 @@ private static File createTestRunDirectory(final @Nullable String baseDirectory, dirPath = Path.of(System.getProperty("user.dir"), "testruns", testRunName); } final File runDir = dirPath.toFile(); - if (runDir.exists()) { - if (!runDir.mkdir()) { - throw new RuntimeException("Directory for test result data compromised."); - } - } if (!runDir.exists()) { if (!runDir.mkdirs()) { throw new RuntimeException("Could not create directory for test result data");