Merge remote-tracking branch 'origin/main' into feature/#515-dependen…

…cy-upgrade

.gitignore

@@ -25,7 +25,6 @@ dependency-reduced-pom.xml
 .DS_Store
 *.swp
 local
-Scripts
 .dbeaver*
 **/*.log
 .directory

README.md

@@ -54,5 +54,5 @@ See the [User Guide](doc/user_guide/user_guide.md) for details and for non-Maven
 - [Developer Guide](doc/developer_guide/developer_guide.md)
 - [Dependencies](dependencies.md)
 - [Changelog](doc/changes/changelog.md)
-- [Features & Requirements](doc/system_requirements.md)
-- [Design](doc/design.md)
+- [Features & Requirements](doc/requirements/system_requirements.md)
+- [Design](doc/requirements/design.md)

doc/changes/changes_3.0.0.md

@@ -4,7 +4,7 @@ Code name: Support Java 17 builds
 
 ## Summary
 
-This release adds support for using different Java versions for running Maven and for compiling/testing projects. We use the `maven-toolchains-plugin` to implement this. See [requirements](../system_requirements.md#support-building-with-multiple-java-versions) and [software design](../design.md#use-maven-toolchain) for implementation details.
+This release adds support for using different Java versions for running Maven and for compiling/testing projects. We use the `maven-toolchains-plugin` to implement this. See [requirements](../requirements/system_requirements.md#support-building-with-multiple-java-versions) and [software design](../requirements/design.md#use-maven-toolchain) for implementation details.
 
 This is a breaking change as it requires PK users to install both JDK versions 11 and 17 and create `~/.m2/toolchains.xml`. See the [user guide](../user_guide/user_guide.md#prerequisites-for-using-project-keeper) and [troubleshooting](../user_guide/user_guide.md#troubleshooting) for details.
 

doc/changes/changes_4.0.0.md

@@ -6,7 +6,7 @@ Code name: Automatic Security Updates
 
 ## Features
 
-* #515: Added automatic dependency upgrade
+* #515: Added requirements and design for automatic dependency upgrade
 
 ## Dependency Updates
 

doc/developer_guide/developer_guide.md

@@ -16,8 +16,8 @@ Since version 2.7.0 PK will automatically install `go-licenses` if required.
 ## Requirements and Design
 
 Design documents are located at
-* [system_requirements.md](../system_requirements.md)
-* [design.md](../design.md)
+* [requirements/system_requirements.md](../requirements/system_requirements.md)
+* [requirements/design.md](../requirements/design.md)
 
 After modifying the `.plantuml` files in `doc/images/` please generate the `.svg` diagrams by running the following command and commit them to Git:
 

doc/design.md → doc/requirements/design.md

@@ -711,7 +711,8 @@ Covers:
 PK generates the `release.yml` workflow so that it runs PK in `verify-release` mode, see [`dsn~verify-release-mode~1`](#verify-release-mode).
 
 Rationale:
-This ensures that all preconditions for the release are met (e.g. current release date). In the previous process this was checked by release-droid.
+* This ensures that all preconditions for the release are met (e.g. current release date). In the previous process this was checked by release-droid.
+* Checking the release date allows skipping a release. I.e. when no release is planned when updating the `main` branch, the user can leave the release date undefined, e.g. `2024-??-??`. This will let `verify-release` fail and the release is cancelled.
 
 Covers:
 * [`dsn~release-workflow~1`](#generate-releaseyml-workflow)

doc/requirements/images/dependencies_update_process.plantuml

@@ -0,0 +1,53 @@
+@startuml dependencies_update_process
+
+start
+:**dependencies_check.yml** workflow
+(triggered daily);
+note right
+  Pass information about created
+  issues & vulnerabilities to
+  **dependencies_update.yml**
+end note
+if(Vulnerable dependencies found?) then (yes)
+  group **dependencies_update.yml** workflow
+    note right
+      GitHub Workflow
+      generated by PK
+    end note
+    :Run PK **update-dependencies** as Maven plugin;
+    group PK **update-dependencies**
+      note right
+        Implement
+        in PK
+      end note
+      if(Latest version already released?) then (yes)
+        :Increment project version;
+      else (no)
+      endif
+      :Update dependencies;
+      :Run PK fix;
+      note right
+        Add dependency changes to
+        changelog, Update list of
+        dependencies, ...
+      end note
+      :Update changlog: add fixed vulnerabilites;
+      if(pom.xml contains artifact-reference-checker-maven-plugin) then (yes)
+        :Run artifact-reference-checker-maven-plugin:fix;
+      else (no)
+      endif
+    end group
+    :Create branch, commit, push &
+    create pull request;
+    note right
+      Needs information about
+      issues & vulnerabilities
+    end note
+    :Send Slack notification
+    for success and failure;
+  end group
+else (no)
+  stop
+endif
+stop
+@enduml

doc/requirements/images/release_process.plantuml

@@ -0,0 +1,51 @@
+@startuml release_process
+
+start
+note right
+  Triggered manually or
+  on pushes to main branch.
+end note
+group **release.yml** workflow
+  note right
+    Generated
+    by PK
+  end note
+  :Run PK **verify-release** as Maven plugin;
+  group PK **verify-release**
+    note right
+      Implemented
+      in PK
+    end note
+    if(Release date up-to-date?\n(allow skipping the release)) then (up-to-date)
+      :Run PK **verify**;
+      :Run additional release checks;
+      :Write changelog content to file;
+      note right
+        Required for creating
+        the GitHub release
+      end note
+    else (invalid/outdated)
+      :Fail build;
+      stop
+    endif
+  end group
+  :Run **mvn verify**;
+  if(Maven Central deployment required) then (required)
+    :Run **mvn deploy**;
+  endif
+  :Calculate checksums for release artifacts;
+  :Create GitHub release;
+  note right
+    Reads changelog
+    content from file
+  end note
+  :Attach release artifacts and
+  checksums to GitHub release;
+  note right
+    Customizable
+  end note
+  :Send Slack notification
+  for success & failure;
+end group
+stop
+@enduml

doc/system_requirements.md → doc/requirements/system_requirements.md

@@ -315,25 +315,25 @@ Needs: dsn
 ### Automatic Dependency Update Process
 `feat~automatic-dependency-update-process~1`
 
-PK supports the automatic dependency process. This speeds up fixing vulnerabilities in third party dependencies and creating releases.
+PK supports a process for automated dependency update. This speeds up fixing vulnerabilities in third party dependencies and creating releases.
 
 Rationale:
 
-We have 130+ projects in the integration team that often require dependency updates due to security issues that are found in the dependencies or transitive dependencies. Most of the time the update is a combination of pulling the latest source, updating the dependencies, updating the change log, running the tests locally, on success pushing the branch, running CI and creating a release.
+The Exasol integration team maintains more than 130 projects that often require dependency updates due to security issues that are found in direct or transitive dependencies. In most cases the update requires pulling the latest source, updating the dependencies, updating the change log, running the tests locally, on success pushing the branch, running CI and creating a release as shown in the following bullet-list.
 
-* Auto-update dependencies
-* Automatically create change log entry
-* Automatically run local tests
-* Automatically push branch
-* Automatically run CI
+* Update dependencies
+* Create change log entry
+* Run local tests
+* Push branch
+* Run CI
 * Release
 
 Needs: req
 
 #### Auto-update dependencies
 `req~auto-update-dependencies~1`
 
-PK automatically updates dependencies when a new vulnerability is found.
+PK automatically updates dependencies when the `dependencies_check.yml` workflow finds a new vulnerability.
 
 Covers:
 * [`feat~automatic-dependency-update-process~1`](#automatic-dependency-update-process)
@@ -343,11 +343,11 @@ Needs: dsn
 #### Automatically create change log entry
 `req~auto-create-changelog~1`
 
-PK automatically generates the change log for fixed vulnerabilities.
+PK generates an entry in the changes file for fixed vulnerabilities.
 
 Rationale:
 
-The change log for fixed vulnerabilities always has the same structure and can be easily automated to avoid manual work.
+The changes file entries for fixed vulnerabilities always have the same structure and can be easily automated to avoid manual work.
 
 Covers:
 * [`feat~automatic-dependency-update-process~1`](#automatic-dependency-update-process)
@@ -362,7 +362,7 @@ PK creates a new Pull Request after upgrading dependencies.
 Rationale:
 
 A pull requests allows to
-* automatically run tests using the upgraded dependencies to verify if the upgrade caused any problems
+* automatically run tests using the updated dependencies to verify if the upgrade caused any problems
 * review and approve changes
 * manually modify files in case of problems
 
@@ -378,7 +378,8 @@ PK automatically builds a new release whenever the `main` branch is updated.
 
 Rationale:
 
-This reduces manual work, it's not necessary any more to manually run release-droid.
+* This reduces manual work, it's not necessary any more to manually run release-droid.
+* Optionally the user can indicate to apply the changes, but postpone creating a release.
 
 Covers:
 * [`feat~automatic-dependency-update-process~1`](#automatic-dependency-update-process)
@@ -391,7 +392,7 @@ Needs: dsn
 PK allows customizing the `ci-build.yml` and `release.yml` workflows with project-specific build steps.
 
 Rationale:
-Some projects use customized workflows and exclude them from PK generation. Allowing to customize workflows will simplify maintenance of GH workflows.
+Currently some projects are already using customized workflows but needed to exclude them from PK generation. Allowing to customize workflows will simplify maintenance of GH workflows.
 
 Needs: req
 
@@ -411,7 +412,7 @@ Covers:
 #### Customize Build Process
 `req~customize-build-process~0`
 
-PK allows adding pre and post steps during the build process as well as customize the actual build step.
+PK allows adding pre and post steps during the build process as well as customizing the actual build step.
 
 Rationale:
 Some projects need to

scripts/build_diagrams.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+# This script builds all design diagrams
+
+base_dir="$( cd "$(dirname "$0")/.." >/dev/null 2>&1 ; pwd -P )"
+readonly base_dir
+
+readonly diagrams_dir="$base_dir/doc/requirements/images"
+readonly output_dir="$base_dir/doc/requirements/images"
+
+if [[ "$(ls -A "$output_dir"/*.svg)" ]]; then
+  echo "Deleting diagrams from $output_dir..."
+  rm "$output_dir"/*.svg
+fi
+
+expected_diagram_count=$(find "$diagrams_dir" -name "*.plantuml" | wc --lines)
+readonly expected_diagram_count
+
+echo "Building $expected_diagram_count diagrams..."
+plantuml -tsvg -output "$output_dir" -failonerror -failonwarn -failfast2 "$diagrams_dir/**/*.plantuml"
+
+actual_diagram_count=$(find "$output_dir" -name "*.svg" | wc --lines)
+readonly actual_diagram_count
+
+if [[ "$expected_diagram_count" -ne "$actual_diagram_count" ]]; then
+    echo "ERROR: Expected $expected_diagram_count diagrams but $actual_diagram_count were generated"
+    exit 1
+fi
+
+echo "All $actual_diagram_count diagrams were built successfully in $output_dir."