Viash 0.9.0-RC1

Viash 0.9.0-RC1 (2024-01-26): Restructure platforms into runners and engines

This release restructures the platforms field into runners and engines.
Additionally changes are made to sanitize the built config output and include additional relevant meta data.

BREAKING CHANGES

• runners and engines: The usage of platforms is deprecated and instead these are split into runners and engines (PR #510).
  The platforms field is still supported but will be removed in a future release.
  In brief, the native platform became a native engine and docker platform became a docker engine.
  Additionally, the native platform and docker platform became a executable runner, nextflow platform became a nextflow runner.
  The fields of docker platform is split between docker engine and docker runner: port, workdir, setup_strategy, and run_args are captured by the runner as they define how the component is run. The other fields are captured by the engine as they define the environment in which the component is run. One exception is chown which is rarely set to false and is now always enabled.

• arguments: Merge arguments into argument_groups during a json decode prepare step (PR #574). The --parse_argument_groups option from ns list and config view is deprecated as it is now always enabled.

NEW FUNCTIONALITY

• export json_schema: Add a --strict option to output a subset of the schema representing the internal structure of the Viash config (PR #564).

• config view and ns list: Do not output internal functionality fields (#564). Additionally, add a validation that no internal fields are present when reading a Viash config file.

• project config: Add fields in the project config to specify default values for component config fields (PR #612). This allows for a more DRY approach to defining the same values for multiple components.

MINOR CHANGES

• testbenches: Add testbenches for local dependencies (PR #565).

• testbenches: Refactor testbenches helper functions to uniformize them (PR #565).

• logging: Preserve log order of StdOut and StdErr messages during reading configs in namespaces (PR #571).

• Java 21 support: Update Scala to 2.13.12 and update dependencies (PR #602).

• project config: Output the project config under the default name ProjectConfig instead of Project during schema export (PR #631). This is now important as the project config is now part of the component config. Previously this was overridden as the class name was ViashProject which was less descriptive.

BUG FIXES

• __merge__: Handle invalid yaml during merging (PR #570). There was not enough error handling during this operation. Switched to the more advanced Convert.textToJson helper method.

• config: Anonymize paths in the config when outputting the config (PR #625).

Viash 0.8.4 (2024-01-15): Bug fix

Fix building components with dependencies that have symlinks in their paths.

BUG FIXES

• dependencies: Fix dependencies with paths using symlinks (PR #621). The resolution for the .build.vsh was changed to use the toRealPath previously, so dependency resolution must take account possible resolved symlinks too.

Viash 0.8.3 (2024-01-08): Bug fixes

Viash 0.8.3 (2024-01-08): Bug fixes

Fix some edge cases and improve usability.

BUG FIXES

• NextflowPlatform: properly resolve paths when a nextflow workflow has another nextflow
  workflow as dependency and the worktree contains a directory that is a symlink (PR #614).

• Main: Fixes a bug added by #294 which causes Viash to print a stacktrace instead of a helpful error message when viash is run without any arguments (#617, PR #618).
  Thanks @mberacochea for pointing out this oversight!

• Dependency: When an alias is defined, pass the alias as a key to the .run() function (#601, PR #606).

Viash 0.8.2

Viash 0.8.2 (2023-12-14): Minor changes and bug fixes

This release fixes a few bugs regarding dependencies and how the Nextflow platform handles Paths.

MINOR CHANGES

• NextflowTestHelper: Do not hardcode a version of Nextflow in the testbench,
  but use the version of Nextflow that is installed on the system (PR #593).

• GitHub Actions: Test different versions of Nextflow (22.04.5, latest, and latest-edge) (PR #593).
  Testing the latest Edge version of Nextflow will allow us to catch notice changes in Nextflow earlier.

• Updates to the documentation and templates in the Git repo (#598, PR #600):

  • Add contributing guidelines.

  • Add issue templates.

  • Reworked the pull request template.

BUG FIXES

• config: Fix the main level of a component config not enforcing strict mode and instead allowing any field to be specified (PR #585).

• dependencies: Allow the user to define a local dependency with specifying repository: local as sugar syntax (PR #609). A local repository is the default value so it's not required to be filled in, but allowing it with a sensible sugar syntax makes sense.

• Repositories: Fix a structural issue where a repository defined directly in a dependency would require the name field to be set (PR #607). Repository variants are created with and without the name field. Repositories under .functionality.dependencies[] use repositories without the name field, while repositories under .functionality.repositories[] use repositories with the name field.

• NextflowPlatform: Do not resolve remote paths relative to the --param_list file (PR #592).

• NextflowPlatform: Allow finding .build.yaml file when worktree contains a directory that is a symlink (PR #611).

Viash 0.8.1

Viash 0.8.1 (2023-11-20): Minor bug fix to Nextflow workflows

This release fixes a bug in the Nextflow platform where calling a workflow with the .run() function without specifying the fromState argument would result in an error when the input channel contained tuples with more than two elements.

BUG FIXES

NextflowPlatform: Fix error when using .run() without using fromState and the input channel holds tuples with more than two elements (PR #587).

Viash 0.8.0

Nextflow workflows definitions are picked up by Viash and assembled into a functional Nextflow workflow, reducing the amount of boilerplate code needed to be written by the user.
It also adds a new runIf argument to the NextflowPlatform which allows for conditional execution of modules.
We added new 'dependencies' functionality to allow for more advanced functionality to be offloaded and re-used in components and workflows.

BREAKING CHANGES

• NextflowPlatform: Changed the default value of auto.simplifyOutput from true to false (#522, PR #518). With simplifyOutput set to true, the resulting Map could be simplified into a File or a List[File] depending on the number of outputs. To replicate the previous behaviour, add the following config mod to _viash.yaml:

  config_mods: |
    .platforms[.type == 'nextflow'].auto.simplifyOutput := true

• VDSL3Helper.nf: Removed from the Viash jar file (PR #518). Its functions have been moved to WorkflowHelper.nf.

• DataflowHelper.nf: Added deprecation warning to functions from this file (PR #518).

• preprocessInputs() in WorkflowHelper.nf: Added deprecation warning to preprocessInputs() because this function causes a synchronisation event (PR #518).

• author.props: Removed deprecated props field (PR #536). Deprecated since 0.7.4.

NEW FUNCTIONALITY

• dependencies: Add dependencies and repositories to functionality (PR #509).
  The new functionality allows specifying dependencies and where to retrieve (repositories) them in a component, and subsequentially allows advanced functionality to be offloaded and re-used in scripts and projects. This is alike e.g. npm, pip and many others. A big difference is that we aim to provide the needed boilerplate code to ease the usage of the dependencies in scripts, workflows and pipelines.
  Note that the dependency is required to be a built Viash component or project and not a random file or code project found externally. This is needed to provide the necessary background information to correctly link dependencies into a component.

• NextflowScript & NextflowPlatform: Merged code for merging the main.nf files for VDSL3 components and wrapped Nextflow workflows (PR #518).
  By aligning the codebase for these two, wrapped Nextflow workflows are more similar to VDSL3 components. For example, you can override the behaviour of a
  wrapped Nextflow workflow using the .run() method. Status of a workflows .run() arguments:

  • Works as intended: auto.simplifyInput, auto.simplifyOutput, fromState, toState, map, mapData, mapPassthrough, filter, auto.publish = "state"
  • Does not work (yet): auto.transcript, auto.publish = true, directives, debug.

  In a next PR, each of the dependencies will have their values overridden by the arguments of the .run.

• NextflowPlatform: The data passed to the input of a component and produced as output by the component are now validated against the arguments defined in the Viash config (PR #518).

• NextflowPlatform: Use stageAs to allow duplicate filenames to be used automatigically (PR #518).

• NextflowPlatform: When wrapping Nextflow workflows, throw an error if the IDs of the output channel doesn't match the IDs of the input channel (PR #518).
  If they don't, the workflow should store the original ID of the input tuple in the in the _meta.join_id field inside the state as follows:
  Example input event: ["id", [input: file(...)]],
  Example output event: ["newid", [output: file(...), _meta: [join_id: "id"]]]

• NextflowPlatform: Added new .run() argument runIf - a function that determines whether the module should be run or not (PR #553).
  If the runIf closure evaluates to true, then the module will be run. Otherwise it will be passed through without running.

MAJOR CHANGES

• WorkflowHelper.nf: The workflow helper was split into different helper files for each of the helper functions (PR #518).
  For now, these helper files are pasted together to recreate the WorkflowHelper.nf.
  In Viash development environments, don't forget to run ./configure to start using the updated Makefile.

• NextflowPlatform: Set default tag to "$id" (#521, PR #518).

• NextflowPlatform: Refactoring of helper functions (PR #557).

  • Cleaned up processConfig(): Removed support for functionality.inputs and functionality.outputs
  • Cleaned up processConfig(): Removed support for .functionality.argument_groups[].argument containing a list of argument ids as opposed to the arguments themselves.
  • Rewrote --param_list parser.
  • Removed unused function applyConfig() and applyConfigToOneParamSet().
  • Refactored channelFromParams() to make use of new helper functions.
  • Removed deprecated paramsToChannel(), paramsToList(), viashChannel().
  • Deprecated preprocessInputs() -- use the wrapped Viash Nextflow functionality instead.
  • Refactored preprocessInputs() to make use of new helper functions.
  • Reprecated run arguments map, mapData, mapPassthrough, renameKeys.

MINOR CHANGES

• NextflowPlatform: Throw error when unexpected keys are passed to the .run() method (#512, PR #518).

• NextflowPlatform: Use moduleDir instead of projectDir to determine the resource directory.

• NextflowPlatform: Rename internal VDSL3 variables to be more consistent with regular Viash component variables and avoid naming clashes (PR #553).

DOCUMENTATION

• Minor fixes to VDSL3 reference documentation (PR #508).

BUG FIXES

• WorkflowHelper.nf: Only set default values of output files which are not already set, and if the output file argument is not required (PR #514).

• NextflowPlatform: When using fromState and toState, do not throw an error when the state or output is missing an optional argument (PR #515).

• export cli_autocomplete: Fix output script format and hide --loglevel and --colorize (PR #544). Masked arguments are usable but might not be very useful to always display in help messages.

Viash 0.8.0-RC6 (2023-10-11): fix race condition issue in runEach

This part of the changelog will be removed.

Viash 0.8.0-RC5 (2023-10-11): Fix run workflow

This part of the changelog will be removed.

• fix run workflow

• fix 'safe'join helper function

• fix 'safe'join helper once more

• fix wf factory

Viash 0.8.0-RC4 (2023-10-10): Refactor Nextflow helper functions

MAJOR CHANGES

• NextflowPlatform: Refactoring of helper functions (PR #557).
  • Cleaned up processConfig(): Removed support for functionality.inputs and functionality.outputs
  • Cleaned up processConfig(): Removed support for .functionality.argument_groups[].argument containing a list of argument ids as opposed to the arguments themselves.
  • Rewrote --param_list parser.
  • Removed unused function applyConfig() and applyConfigToOneParamSet().
  • Refactored channelFromParams() to make use of new helper functions.
  • Removed deprecated paramsToChannel(), paramsToList(), viashChannel().
  • Deprecated preprocessInputs() -- use the wrapped Viash Nextflow functionality instead.
  • Refactored preprocessInputs() to make use of new helper functions.
  • Reprecated run arguments map, mapData, mapPassthrough, renameKeys.

Viash 0.8.0-RC2 (2023-10-04): Some bugfixes

Some bugfixes related to the new dependencies and Nextflow code generation functionality.

This part of the changelog will be removed.