release-please-action invoked from the GitHub CI workflow generates pull requests containing updated CHANGELOG.md and version.txt files based on these commit messages. Merging the pull request will result in a new release version being published by creating a GitHub release describing the changes.
release-please-action invoked from the GitHub CI workflow generates pull requests containing updated CHANGELOG.md and version.txt files based on these commit messages. Merging the pull request will
+result in a new release version being published; this includes publishing the image to Docker Hub, and creating a GitHub release describing the changes.
picocli: Used for fcli command implementations; perform fcli actions, process command line options, generate usage information, generate command output, …
+picocli: Process command line options, generate usage information, …
+Micronaut: Dependency injection, features for GraalVM native image generation
Jackson: Parse and generate data in JSON and other formats
@@ -141,7 +128,7 @@Lombok: Please see https://projectlombok.org/setup/overview for more information on how to add Lombok support to your IDE
Annotation processors: This project requires various annotation processors to be run, like the picocli annotation processor. Please see your IDE documentation on how to enable annotation processing
+Micronaut & picocli: These frameworks require annotation processors to be run during builds; please see your IDE documentation on how to enable annotation processing
GraalVM Native Image needs to know ahead-of-time the reflectively accessed program elements; see GraalVM Native Image & Reflection for details.
+GraalVM Native Image needs to know ahead-of-time the reflectively accessed program elements; see GraalVM Native Image & Reflection for details. Micronaut allows for generating the necessary GraalVM reflection configuration using the @ReflectiveAccess annotation if necessary.
In fcli, the following needs to be taken into account to avoid reflection issues in native fcli binaries:
+If a command runs fine on a regular JVM but not when running as a native image, then quite likely this is caused by missing reflection configuration which may be fixed by adding the @ReflectiveAccess annotation to the appropriate classes. Also see below for some reflection-related considerations specific to picocli and Jackson.
Picocli and it’s annotation processor should in theory generate the necessary reflection configuration as described above. However, the annotation processor doesn’t seem to take class hierarchies into account; see GraalVM: Options from superclass not visible. As a work-around, we can complement the picocli-generated reflection configuration by using Micronaut’s @ReflectiveAccess annotation. In general the following classes should probably be annotated with @ReflectiveAccess:
All data classes that are serialized or deserialized using Jackson must be annotated with the @Reflectable annotation. For de-serialization, a public no-args constructor is required as well. To avoid any potential issues, it is highly recommend to have both @Reflectable and @NoArgsConstructor annotations on such classes, preferably on the same line for easy identification.
Failure to do so will cause errors when serializing or deserializing data to or from classes that lack this annotation.
+All classes annotated with @Command (this likely duplicates the reflection configuration generated by picocli, but doesn’t hurt)
Usually this only affects a small group of commands or a single command, possibly even only under certain conditions, and as such any missing annotations may be difficult to detect.
+All super-classes of all classes annotated with @Command, in particular if they contain any picocli annotations like @Mixin, @ArgGroup, or @Option
Most notably, the following classes will need the @Reflectable annotation:
All classes used as an @Mixin or @ArgGroup, and their super-classes
The following native image runtime behavior may indicate a missing @ReflectiveAccess annotation:
Most *Descriptor classes.
Classes holding REST request or response data.
+Options and sub-commands not listed in help output, and not recognized when entered on the command line
Classes holding data stored in the fcli data folder, like configuration data classes.
+Exceptions related to picocli trying to access classes, methods and fields using reflection
Jackson is heavily based on reflection to perform object serialization and deserialization. All classes that need to be (de-)serialized with Jackson therefore need to be annotated with @ReflectiveAccess to allow reflection-based (de-)serialization.
If a command runs fine on a regular JVM but not when running as a native image, then quite likely this is caused by missing reflection configuration which may be fixed by adding the @Reflectable annotation to the appropriate classes.
Note that Micronaut provides the @Introspected annotation that should allow for reflection-free (de-)serialization. However experience learns that there are just too many differences between reflection-based
+and introspection-based (de-)serialization. When running in a normal JVM these differences are often not visible as Jackson can easily fall back to reflection-based (de-)serialization. In native images, Jackson cannot fall back to reflection-based (de-)serialization unless properly configured, and even when properly configured there are still some differences in behavior.
As such, it has been decided to disable Jackson introspection-based (de-)serialization using the jackson.bean-introspection-module configuration property in application.yml. This also means that we will not be using the @Introspected annotation on classes that need to be (de-)serialized with Jackson (unless of course introspection-based access is used for non Jackson-related purposes).
User documentation is generated automatically by the buildRoot/app/fcli-doc project. Part of the documentation consists of manual pages in various formats, generated using picocli. Other user documentation is generated from the src/docs/asciidoc directory in the before-mentioned project. This consists of static AsciiDoc files, used to generate the top-level contents of the fcli GitHub Pages site, and 'versioned' AsciiDoc files that describe the functionality provided by a particular fcli version.
Documentation can be generated using the Gradle distDocs task, ending up in the dist directory of the aforementioned project. The global dist task includes the distDocs task.
The GitHub Actions workflow defined in .github/workflows/ci.yml is responsible for publishing the documentation:
The build job builds the documentation artifacts and archives them as artifacts
The release job publishes docs-html.zip and docs-manpage.zip to the release artifacts (when building a release or development version)
The publishPages job published the output of the asciidoctorJekyll and asciidoctorGHPages to the appropriate directories on the GitHub Pages site, and updates the version index in the Jekyll _data directory (when building a release or development version)
All HTML-formatted documentation described above is generated using the doc-resources/templates/html5/document.html.erb template. This template is based on the official AsciiDoctor template with various modifications. Based on the attributes provided in the relevant Gradle tasks:
User documentation is generated automatically from the following three locations:
For Jekyll output:
+AsciiDoc located in the repository doc-resources/asciidoc/gh-pages directory
Add Jekyll front matter
-Add a Jekyll include to include additional content in the HTML <head> section; mostly used for applying stylesheets
Add a Jekyll include to include the site-wide banner and (version) navigation bar
+Published to the root directory of the GitHub Pages site
For offline HTML output:
+AsciiDoc located in the repository doc-resources/asciidoc/versioned directory
Add hardcoded custom styling
+Published to a version-specific directory on the GitHub Pages site
Add hardcoded banner and version bar
+Published to docs-html.zip in release assets
The offline HTML documentation is supposed to be self-contained, i.e., pages should render correctly, without having to extract the full contents, if users open any HTML file from docs-html.zip. In particular, this means that styles and images need to be embedded inside the HTML files. Of course, links to other documentation files will not work unless the full zip-file is extracted.
For now, the hardcoded banner and navigation bar in the offline documentation is similar to the banner included by Jekyll. However:
-Manual pages generated from the fcli code
Stylesheets and images are linked rather than being included in the HTML page, allowing for better browser cache utilization
+Published to a version-specific directory on the GitHub Pages site
The navigation bar in the offline documentation contains just a static version number, whereas the navigation bar in the online documentation allows for navigating to different versions
+Published to docs-html.zip in release assets
We can potentially add more advanced (navigation) functionalities in the online documentation
-We can easily update the banner for the online documentation to have a new layout/styling, for example to apply OpenText styling; this will be automatically applied to all existing online documentation pages
+Published to docs-manpage.zip in release assets
Usually it shouldn’t be necessary to update the documentation contents for existing release versions. However, if necessary, and assuming the build.gradle file is compatible with older versions, potentially a command like the following can be used to regenerate the documentation for the given versions:
-for v in 1.0.0 1.0.1 1.0.2 1.0.3 1.0.4 1.0.5 1.1.0 1.2.0 1.2.1 1.2.2; do (git restore . && git clean -fd && git checkout v$v && cp -r ../fcli-fork/doc-resources ../fcli-fork/build.gradle . && ./gradlew clean distDocs -Pversion=$v && mkdir -p ~/Downloads/fcli-docs/$v && cp build/dist/docs-html.zip ~/Downloads/fcli-docs/$v && cd ../fcli-pages/v$v && echo $pwd && rm -rf * && unzip ../../fcli/build/dist/docs-jekyll.zip && cd - && git restore . && git clean -fd); done-
This command iterates over the given version numbers, regenerates the documentation for each version (using latest build.gradle and doc-resources), copies the docs-html.zip to a separate directory for later upload to the corresponding release assets, and updates the GitHub Pages site, based on the following assumptions:
Current directory is a clone of the fcli repository
-../fcli-fork would contain the latest version of doc-resources and build.gradle
../fcli-pages would be a clone of the fcli repository with the gh-pages branch checked out
Note that project directory structure may change beteen fcli versions, possibly requiring extra work to make the above work without issues.
-Common Java coding conventions should be used for fcli source code, taking the following into consideration:
+The Gradle build includes various tasks for generating this documentation, following are the main tasks:
Indentation is done using 4 spaces; fcli source code should not contain tabs.
+generateManpageAsciiDoc: Generate man-page style AsciiDoc documentation from fcli code
Use of System.out and System.err should be avoided, except for code that explicitly handles output.
-Command output should be generated through the output framework provided in the common module.
+asciiDoctorManPage: Convert man-page style AsciiDoc to Linux man-page format
The logging framework should be used for outputting warning messages for example.
+asciiDoctorHtml: Convert both man-page style AsciiDoc and versioned user documentation to offline HTML format
If you use System.out for debugging, potentially commenting out or removing these statements once done with debugging, consider using the logging framework for debug logging. If you need this information for debugging, it may be useful to permanently have this information included in debug logs.
-asciidoctorJekyll: Convert both man-page style AsciiDoc and versioned user documentation to Jekyll HTML format for publishing on the GitHub Pages site
Avoid having commented out source code.
+asciidoctorGHPages: Convert AsciiDoc files from doc-resources/asciidoc/gh-pages to Jekyll HTML format for publishing on the GitHub Pages site
Use the 'Organize Imports` feature of the IDE to remove any unused imports.
-Avoid having unused variables, methods, …
-Avoid unsafe type conversions. In particular, when using Jackson for deserializing generic types, use TypeReference instead of the generic type class.
distDocs: Calls of the tasks above and packages the output from these tasks into separate docs-*.zip files in the build/dist directory
Most fcli command modules use the package structure described below. Note that there may be slight variations between product-specific modules that interact with a remote system, and fcli-specific modules like config and tool.
com.fortify.cli.<module>
Root package for the given module
-The GitHub Actions workflow defined in .github/workflows/ci.yml is responsible for publishing the documentation:
com.fortify.cli.<module>._main.cli.cmd
Contains <module>Commands class listing all entity commands for the given module.
The build job builds the documentation artifacts and archives them as artifacts
May contain command implementations that operate at module-level rather than entity-level, like the ConfigClearCommand.
The release job publishes docs-html.zip and docs-manpage.zip to the release artifacts (when building a release or development version)
com.fortify.cli.<module>.<entity>
Root package for the given module entity
+The publishPages job published the output of the asciidoctorJekyll and asciidoctorGHPages to the appropriate directories on the GitHub Pages site, and updates the version index in the Jekyll _data directory (when building a release or development version)
com.fortify.cli.<module>.<entity>.cli
Root package for Picocli-based code, like command implementations and mixins
-All HTML-formatted documentation described above is generated using the doc-resources/templates/html5/document.html.erb template. This template is based on the official AsciiDoctor template with various modifications. Based on the attributes provided in the relevant Gradle tasks:
com.fortify.cli.<module>.<entity>.cli.cmd
Contains the <module><entity>Commands class, listing all sub-commands for the given entity
Contains the individual entity action command classes
-Where appropriate, sub-packages may be used to group related action commands
-com.fortify.cli.<module>.<entity>.cli.mixin
For Jekyll output:
Contains classes used as Mixin classes, for example defining reusable options and parameters, which may be used by commands in the current entity but also by other entities.
+Add Jekyll front matter
May contain classes used as ArgGroups, but these should be used sparingly as noted in ArgGroup Annotations
+Add a Jekyll include to include additional content in the HTML <head> section; mostly used for applying stylesheets
Usually contains a <module><entity>ResolverMixin class, containing inner classes that allow for resolving one or more <entity> instances based on command-line options and/or positional parameters. Each inner class name describes the provided functionality, like RequiredOption, OptionalOption, RequiredPositionalParameter, …
Add a Jekyll include to include the site-wide banner and (version) navigation bar
com.fortify.cli.<module>.<entity>.helper
For offline HTML output:
Contains entity-related helper classes, for example for loading entity data, deleting entities, …
-Contains *Descriptor classes that hold entity-related data
Classes in this package should not contain any picocli-related functionality; they should be designed in such a way that they could potentially be used in non-picocli applications
-Add hardcoded custom styling
com.fortify.cli.<module>._common.*
Root package for module-specific generic functionality, like connecting to, and authenticating with, the remote system, generic request/response transformations, … Please see existing modules for example structure and contents.
+Add hardcoded banner and version bar
The following sections provide information on implementing fcli commands.
+The offline HTML documentation is supposed to be self-contained, i.e., users can open any HTML file from docs-html.zip without extracting the full contents, and the page will render correctly. Of course, links to other documentation files will not work unless the full zip-file is extracted.
In general, we try to adhere to the following fcli command structure:
-fcli <module> <entity> <action>-
For now, the hardcoded banner and navigation bar in the offline documentation is similar to the banner included by Jekyll. However:
<module> represents either a product like ssc or fod, or an fcli-specific module like config, state, tool or util.
Stylesheets and images are linked rather than being included in the HTML page, allowing for better browser cache utilization
<entity> represents the entity on which the <action> sub-commands operate, like app, appversion or appversion-attribute. Virtually every entity should have its own top-level command inside a <module>, we usually don’t use nested entities like app→version→attribute.
The navigation bar in the offline documentation contains just a static version number, whereas the navigation bar in the online documentation allows for navigating to different versions
<action> represents the action to be taken on the <entity> and is usually a verb like list, get, set, delete, update, …
If there are multiple variants of a particular command, <action> may include a suffix after the verb, like download-by-id and download-state, or purge-by-id and purge-by-date.
We can potentially add more advanced (navigation) functionalities in the online documentation
Aliases should be used to maintain backward compatibility if needed. For example, if there is already a delete command that deletes by id, and a new command for deleting by date needs to be added, the original command would be renamed to delete-by-id with alias delete.
We can easily update the banner for the online documentation to have a new layout/styling, for example to apply OpenText styling; this will be automatically applied to all existing online documentation pages
Fcli commands should be atomic and specific in nature. Each command should only do one thing, and do it well. A clear example are the wait-for commands that provide a lot of wait-related options, rather than having the wait-related options on the command that initiated the action that we’re waiting for.
In general, exclusive options that influence the outcome of a command are an indication that a command is not specific enough; in such cases you may want to consider having multiple variants of the same command as described above. For example, if you are considering a single purge command with exclusive options --id <id> and --older-than <date>, then having more specific purge-by-id and purge-by-date or even purge-older-than commands would be more appropriate.
As usual, there are some exceptions to this rule, in particular for commands that are consistently named across fcli modules. For example, login commands often allow for logging in with either user or token credentials; for consistency we just have a single login command that provides options for either approach.
-In general, each container command should contain either only leaf commands, or only container commands. For example, the top-level <module> command should usually only contain <entity> container commands and no leaf commands, whereas <entity> commands should usually only contain leaf commands. There are some exceptions to this rule though, for example if a command operates on all entities within a module, like the fcli config clear command.
Most or all product-specific leaf command implementations should have the following generic structure:
+Usually it shouldn’t be necessary to update the documentation contents for existing release versions. However, if necessary, and assuming the build.gradle file is compatible with older versions, potentially a command like the following can be used to regenerate the documentation for the given versions:
@Command(name = [Module]OutputHelperMixins.<Action>.CMD_NAME)
-public class <Module><Entity><Action>Command extends <SuperClass> implements <CommonInterfaces> {
- @Getter @Mixin private [Module]OutputHelperMixins.<Action> outputHelper;
- // Options, positional parameters, other fields; see next sections for info on options and parameters
-
- // Overrides for interfaces, for example methods generating the output data,
- // record transformations, ...
-
- @Override
- public boolean isSingular() {
- return <false if potentially returning multiple records, true if always returning single record>;
- }
-}<Module>: Corresponds to the module in which this class is located; for product-specific commands this would correspond to the product name like SSC or FoD.
[Module]: Commands can use IOutputHelper implementations from either the generic OutputHelperMixins or the module-specific [Module]OutputHelperMixins
<Entity> represents the entity that the command is operating on, like App, AppVersion, User, …
<Action>: Represents the action performed by this command; should be one of the classes in the [Module]OutputHelperMixins class, like Get, List, Delete, …
Every action should have a corresponding inner class in either the generic OutputHelperMixins class, or the module-specific [Module]OutputHelperMixins class.
In general, only concrete command implementations should declare references to *OutputHelperMixins classes, both in the @Command declaration and the outputHelper field. Defining an outputHelper field in an (abstract) superclass will result in any aliases defined on *OutputHelperMixins inner classes not being applied to the concrete command implementations.
<SuperClass> is usually Abstract<Module>OutputCommand or one of its abstract subclasses. Most modules provide Abstract<Module>JsonNodeOutputCommand and Abstract<Module>BaseRequestOutputCommand as bases classes for commands that generate JsonNode or HttpRequest instances respectively. Indirectly, virtually all leaf commands should extend from AbstractOutputCommand.
<CommonInterfaces> is a list of interfaces that define how output is being generated and processed. Following are some commonly used interfaces; see JavaDoc for details on usage:
IBaseRequestSupplier: Supply a Unirest HttpRequest instance for retrieving command output data.
IJsonNodeSupplier: Supply a JsonNode instance representing the command output.
IActionCommandResultSupplier: Supply data for a result column to be included in the output, like DELETED, CREATED, …
IInputTransformer: Allows for transforming the full JSON data before it is being processed for output.
IRecordTransformer: Allows for transforming individual records before they are being processed for output.
Leaf commands in non-product modules usually have a similar structure, but some details may be different. Container commands, i.e. commands that represent <module> and <entity> have a very different (easier) structure as they don’t have any actual functionality associated to them; please refer to existing fcli container commands to serve as an example.
Commands (including aliases) and options should use kebab-case names, i.e., lower-case names with words separated by a dash.
+for v in 1.0.0 1.0.1 1.0.2 1.0.3 1.0.4 1.0.5 1.1.0 1.2.0 1.2.1 1.2.2; do (git restore . && git clean -fd && git checkout v$v && cp -r ../fcli-fork/doc-resources ../fcli-fork/build.gradle . && ./gradlew clean distDocs -Pversion=$v && mkdir -p ~/Downloads/fcli-docs/$v && cp build/dist/docs-html.zip ~/Downloads/fcli-docs/$v && cd ../fcli-pages/v$v && echo $pwd && rm -rf * && unzip ../../fcli/build/dist/docs-jekyll.zip && cd - && git restore . && git clean -fd); done
Acceptable command/option names: app, appversion-attribute
Invalid command/option names: App, appversionAttribute, appversion_attribute
Commands may have one or more aliases, and multiple names may be defined for options. Having multiple names for a single command or option may be useful for various reasons, for example:
-For providing shorter names, resulting in less typing for the user.
-To maintain backward compatibility when renaming an existing command or option
-As a 'default' command or option name if there are multiple variants of the same command or option, for example purge as the an alias for purge-by-id to have this as the default 'purge' operation if there are also other purge-* commands.
In general:
+This command iterates over the given version numbers, regenerates the documentation for each version (using latest build.gradle and doc-resources), copies the docs-html.zip to a separate directory for later upload to the corresponding release assets, and updates the GitHub Pages site, based on the following assumptions:
Each command and option should have at most two names; the full name and a shorter form. For options, the shorter form is usually a single-letter option.
-Having more than two names is only allowed if needed for backward compatibility; the backward-compatible name(s) should be removed on the next major release.
-Options should always have a full name to describe their meaning, short (single-letter) name is optional.
-Single-letter options are preceded by - (single dash), multi-letter options are preceded by -- (double dash).
Current directory is a clone of the fcli repository
It was decided that ArgGroup annotations should be used sparingly (Issue #89). ArgGroup annotations are mostly used for generic options, like logging, help, output and query options. In general, command-specific options should use ArgGroup annotations only for defining exclusive options, not for creating a separate section in the help output.
It was decided that every fcli command should have at most one positional parameter definition to specify the primary entity id or name that the command operates on. For example, <entity> delete commands can have a positional parameter for specifying the entity name or id to be deleted, usually matching the <entity> command group that the command is located at. For example, an app delete command would take a positional parameter for identifying the app to be deleted. Potentially the positional parameter may be an array or collection, for example if the app delete command allows for deleting multiple apps in a single operation.
For everything else, including parent entities, options should be used. As an example, the fcli ssc appversion-attribute set command takes the parent entity (application version) as an option named --appversion, whereas the attribute names and values to be set are taken as a positional parameter Map instance.
In general, standard picocli conventions should be used for locating i18n message keys for options, positional parameters, command descriptions, and so on. In some cases, it may be necessary to configure explicit messages keys in picocli annotations, for example for options and parameters defined in command super-classes or mixins.
-As an example, take the @Option descriptionKey attributes in the AbstractToolInstallCommand, specifying a fixed description key for all sub-commands. This way, the option description only needs to be specified once in the resource bundle, whereas using the standard picocli conventions would potentially have resulted in requiring the option description to be repeated for every individual command that extends from AbstractToolInstallCommand.
Ideally, all commands, options and positional parameters should have a proper description. The following Linux commands can be used to find missing descriptions for options and commands:
-Check which options don’t have a description:
-./gradlew build generateManpageAsciiDoc && grep -e "^\\*-.*::" -A 1 build/generated-picocli-docs/*.adoc | grep -e "^build.*-\s*$" -B 1
../fcli-fork would contain the latest version of doc-resources and build.gradle
Check which commands don’t have a usage header (inheriting the header of the root command):
-./gradlew build && java -jar build/libs/fcli.jar util all-commands list --include-parents | fgrep "Command-line interface for working with"
../fcli-pages would be a clone of the fcli repository with the gh-pages branch checked out
Where applicable, option and positional parameter descriptions should include references to other related fcli commands, in particular when these related commands are in a separate command tree. For example, available attribute names and values that can be specified on the fcli ssc appversion-attribute set command can be found through the fcli ssc attribute-definition * commands. Being in a separate command tree, this may not be obvious to users and as such should be documented on the fcli ssc appversion-attribute set command.
Comparing this to the 'application name or id' to be passed to the fcli ssc app get command; available application id’s can be found through the fcli ssc app list command under the same app parent command, so this doesn’t need to be documented as it should be obvious to users.