Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nf-core configs create sub command #2852

Draft
wants to merge 102 commits into
base: dev
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
513de95
WIP: First stab at writing some Textual
ewels Aug 20, 2023
0e8204e
Refactor: Break into multiple files
ewels Aug 21, 2023
e37d2e5
Remove unusued imports
ewels Aug 21, 2023
ee12eb5
show error message when pressing enter
mirpedrol Aug 22, 2023
dd18bcc
show failure messages when button is pressed
mirpedrol Aug 22, 2023
486f180
add usegenomicdata screen (TODO on_button_pressed)
mirpedrol Aug 22, 2023
5b8cfd5
add custom pipeline switch table to select pipeline features
mirpedrol Aug 23, 2023
0a40dab
create new widget PipelineFeature to display feature buttons and help…
mirpedrol Aug 24, 2023
39b9e3c
add more custom features
mirpedrol Aug 24, 2023
25c9fa1
add template features to skip to TEMPLATE_CONFIG
mirpedrol Aug 25, 2023
610cbd9
animate help box
mirpedrol Aug 28, 2023
6677144
add type_nfcore screen and use PipelineCreate to create a pipeline
mirpedrol Aug 29, 2023
8bac0be
modify PipelineCreate to accept a template dictionary
mirpedrol Aug 29, 2023
43db7f6
start modifying nf-core create command to use TUI and not CLI
mirpedrol Aug 30, 2023
99f426a
small bug fixes and add a deprecation message for nf-core create command
mirpedrol Aug 31, 2023
09bd5db
create command fails if one but not all arguments are provided
mirpedrol Aug 31, 2023
9f121f7
ask if launching TUI when using deprecated nf-core create command
mirpedrol Aug 31, 2023
0c77029
add final_details screen for version and force
mirpedrol Aug 31, 2023
bf9a541
add nfcorepipeline grid with features to select
mirpedrol Aug 31, 2023
fec5e3e
first ask if the pipeline will be nf-core or custom before basicdetails
mirpedrol Aug 31, 2023
0ef16b5
use kwargs
mirpedrol Aug 31, 2023
238767f
remove inexisten variable version from send-tweet github workflow
mirpedrol Aug 21, 2023
7b4a342
fix typo in gh_badges jinja variable to github_badges
mirpedrol Aug 24, 2023
b3fa25e
bump version of nf-test snap files
mirpedrol Aug 30, 2023
6ff69b2
update changelog
mirpedrol Aug 30, 2023
17ba971
add outdir textinput and update config
mirpedrol Sep 1, 2023
0c64de8
refactor jinja template 'branded' to 'is_nfcore'
mirpedrol Sep 1, 2023
7720117
refactor params_dict to jinja_params and use that instead of the config
mirpedrol Sep 1, 2023
cb62be3
refactor function obtain_jinja_params_dict
mirpedrol Sep 1, 2023
da1c68f
run create command from the App, mv nf_core/create.py to nf_core/pipe…
mirpedrol Sep 1, 2023
3c14dd7
add new screen to create a github repo and push the new pipeline (not…
mirpedrol Oct 3, 2023
e401a6a
Some minor CSS tweaks
ewels Nov 2, 2023
8b9ad35
Add logging handler and final screen to visualise logs
mirpedrol Nov 3, 2023
4c237e2
fix githubrepo buttons handled by the same function
mirpedrol Nov 3, 2023
8318e0c
fix pytests
mirpedrol Nov 6, 2023
c7e057d
Merge branch 'dev' of https://github.com/nf-core/tools into textual-c…
mirpedrol Nov 6, 2023
c2c0561
fix test_remove_patch
mirpedrol Nov 6, 2023
ef9ea81
add types-requests to requirements-dev
mirpedrol Nov 6, 2023
798dad9
add full logging screen to show logging messages
mirpedrol Nov 10, 2023
4758c7d
autopopulate github credentials and hide password
mirpedrol Nov 13, 2023
665d378
add button to show or hide password
mirpedrol Nov 13, 2023
19f60bc
update textual and textual-dev versions
mirpedrol Nov 14, 2023
887f372
add first snapshot tests
mirpedrol Nov 14, 2023
7758698
add snapshots for all screens
mirpedrol Dec 5, 2023
0dbe982
Merge branch 'dev' of https://github.com/nf-core/tools into textual-c…
mirpedrol Dec 5, 2023
9c25603
Merge branch 'dev' of https://github.com/nf-core/tools into textual-c…
mirpedrol Jan 5, 2024
3908163
ruff code modifications
mirpedrol Jan 5, 2024
208d832
update snapshots
mirpedrol Jan 5, 2024
bebf067
ask if we have to create a github repo in a different screen
mirpedrol Jan 5, 2024
7ade926
add snapshot for github repo question and exit message
mirpedrol Jan 5, 2024
7049277
add work threads for creating a pipeline and a github repo
mirpedrol Jan 5, 2024
41fcc27
add loading screen when creating a pipeline
mirpedrol Jan 5, 2024
cfa0804
show logging only at the end
mirpedrol Jan 5, 2024
a85f1dd
add loading screen for repo creation
mirpedrol Jan 5, 2024
456a4be
update app snapshots
mirpedrol Jan 5, 2024
bb6cc97
fix typing error
mirpedrol Jan 8, 2024
abdffef
ignroe snapshot files with editorconfig
mirpedrol Jan 8, 2024
2e396d9
add exclusive=True to work threads
mirpedrol Jan 8, 2024
3329411
remove logging and add completed screen instead
mirpedrol Jan 8, 2024
f5551bc
Merge branch 'dev' of https://github.com/nf-core/tools into textual-c…
mirpedrol Jan 8, 2024
cbce046
show an error message if the pipeline already exists
mirpedrol Jan 8, 2024
62c4b3e
fix nf-core.pipelines.create import
mirpedrol Jan 8, 2024
f04ad5e
GHA use new nf-core pipelines create command and fixed default org
mirpedrol Jan 8, 2024
baa3412
fix pytests
mirpedrol Jan 8, 2024
acd7201
add loading messages
mirpedrol Jan 9, 2024
6b4ed9c
default version to 1.0.0dev
mirpedrol Jan 9, 2024
97bca8b
add logging instead of loading screen and see logging messages on rea…
mirpedrol Jan 19, 2024
0525d9c
deprecation error if 'nf-core create' is used
mirpedrol Feb 19, 2024
29afad0
more formatting and docs
mirpedrol Feb 19, 2024
2dcb0f1
add button to go back
mirpedrol Feb 19, 2024
17bb9e9
disable button to close logs while pipeline is not created
mirpedrol Feb 19, 2024
5445b74
more tweaking of logging buttons and screen
mirpedrol Feb 20, 2024
4ecb445
Merge branch 'dev' of https://github.com/nf-core/tools into textual-c…
mirpedrol Feb 21, 2024
536e3be
update create app tests according to last changes
mirpedrol Feb 21, 2024
599d2e0
Merge branch 'dev' into textual-create
ewels Mar 4, 2024
2943098
some fixes after merging dev to textual-create branch
mirpedrol Mar 6, 2024
3cfdc78
fix packaging. Thanks @mashehu for the debugging help
mirpedrol Mar 6, 2024
e7687e9
add changed from dev branch to create.py script
mirpedrol Mar 7, 2024
1debc4b
fix back button after validation of basic details
mirpedrol Mar 7, 2024
41e99a1
move button to close logging screen to the bottom
mirpedrol Mar 7, 2024
67db794
Merge branch 'dev' of https://github.com/nf-core/tools into textual-c…
mirpedrol Mar 7, 2024
8417de6
Add base command to main
jfy133 Mar 18, 2024
12dacdd
Rename to configs for consistency
jfy133 Mar 18, 2024
650188f
Copy some basic structure from pipelinexcreate
jfy133 Mar 18, 2024
e91dfa4
We have a welcome screen!
jfy133 Mar 18, 2024
e144ad7
Fix help message
jfy133 Mar 18, 2024
ea4e8ae
First question: infrastructure. Extra commit message courtesy of copi…
jfy133 Mar 18, 2024
6cb7485
Start adding basic details (confignameauthor,description), plus with …
jfy133 Mar 18, 2024
5225fb7
Run basic details screen
jfy133 Mar 18, 2024
b786e9c
Add infra type question (hpc or local))
jfy133 Mar 18, 2024
9ee8b43
Add github handle question for basic details
jfy133 Mar 18, 2024
e69c536
Add infratype and env modules questions, rename basic details to nfco…
jfy133 Mar 19, 2024
3378340
Add maxparams and other locations questions
jfy133 Mar 19, 2024
62c0d2e
Add misc. questions screen
jfy133 Mar 19, 2024
9fd60b1
Initialise a basic config and attempt to update config
jfy133 Mar 20, 2024
6ee0088
Add schedulertype question
georgiesamaha Mar 20, 2024
5788b14
Add validation for config github and URL
jfy133 Mar 20, 2024
a2a1f51
Merge branch 'configbuilder' of github.com:jfy133/nf-core-tools into …
jfy133 Mar 20, 2024
d46f0b6
Start adding further validation checks
jfy133 Mar 20, 2024
8ce44c7
Add attempt for writing file for Julia to fix :Grimace:
jfy133 Mar 20, 2024
0ffbe8c
first test writing to a file
mirpedrol Mar 20, 2024
37a2ed7
Add a self reminder here
jfy133 Mar 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,11 @@ indent_style = unset

[**/Makefile]
indent_style = unset

[tests/__snapshots__/*]
charset = unset
end_of_line = unset
insert_final_newline = unset
trim_trailing_whitespace = unset
indent_style = unset
indent_size = unset
2 changes: 1 addition & 1 deletion .github/actions/create-lint-wf/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ runs:
run: |
mkdir -p create-lint-wf && cd create-lint-wf
export NXF_WORK=$(pwd)
nf-core --log-file log.txt create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface" --plain
nf-core --log-file log.txt pipelines create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface"

# Try syncing it before we change anything
- name: nf-core sync
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/create-test-lint-wf-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ jobs:
- name: create a pipeline from the template ${{ matrix.TEMPLATE }}
run: |
cd create-test-lint-wf
nf-core --log-file log.txt create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface" --template-yaml ${{ matrix.TEMPLATE }}
nf-core --log-file log.txt pipelines create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface" --template-yaml ${{ matrix.TEMPLATE }}

- name: run the pipeline
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/create-test-wf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ jobs:
run: |
mkdir create-test-wf && cd create-test-wf
export NXF_WORK=$(pwd)
nf-core --log-file log.txt create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface" --plain
nf-core --log-file log.txt pipelines create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface"
nextflow run nf-core-testpipeline -profile test,self_hosted_runner --outdir ./results

- name: Upload log file artifact
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -115,3 +115,6 @@ ENV/
pip-wheel-metadata
.vscode
.*.sw?

# Textual
snapshot_report.html
4 changes: 3 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.2.2
rev: v0.3.1
hooks:
- id: ruff # linter
args: [--fix, --exit-non-zero-on-fix] # sort imports and fix
Expand All @@ -9,6 +9,8 @@ repos:
rev: "v3.1.0"
hooks:
- id: prettier
additional_dependencies:
- [email protected]

- repo: https://github.com/editorconfig-checker/editorconfig-checker.python
rev: "2.7.3"
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

### Template

- Update templates to use nf-core/setup-nextflow v2

### Linting

### Components
Expand All @@ -12,6 +14,7 @@

- Update CI to use nf-core/setup-nextflow v2
- Changelog bot: handle also patch version before dev suffix ([#2820](https://github.com/nf-core/tools/pull/2820))
- update prettier to 3.2.5 ([#2830](https://github.com/nf-core/tools/pull/2830))
- Update GitHub Actions ([#2827](https://github.com/nf-core/tools/pull/2827))

## [v2.13.1 - Tin Puppy Patch](https://github.com/nf-core/tools/releases/tag/2.13) - [2024-02-29]
Expand Down
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ include nf_core/assets/logo/nf-core-repo-logo-base-lightbg.png
include nf_core/assets/logo/nf-core-repo-logo-base-darkbg.png
include nf_core/assets/logo/placeholder_logo.svg
include nf_core/assets/logo/MavenPro-Bold.ttf
include nf_core/pipelines/create/create.tcss
2 changes: 1 addition & 1 deletion nf_core/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
""" Main nf_core module file.
"""Main nf_core module file.

Shouldn't do much, as everything is under subcommands.
"""
Expand Down
211 changes: 162 additions & 49 deletions nf_core/__main__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#!/usr/bin/env python
""" nf-core: Helper tools for use with nf-core Nextflow pipelines. """
"""nf-core: Helper tools for use with nf-core Nextflow pipelines."""

import logging
import os
import sys
Expand Down Expand Up @@ -35,6 +36,7 @@
"commands": [
"list",
"launch",
"configs",
"create-params-file",
"download",
"licences",
Expand All @@ -44,7 +46,7 @@
{
"name": "Commands for developers",
"commands": [
"create",
"pipelines",
"lint",
"modules",
"subworkflows",
Expand All @@ -55,6 +57,18 @@
],
},
],
"nf-core configs": [
{
"name": "Config commands",
"commands": ["create"],
},
],
"nf-core pipelines": [
{
"name": "Pipeline commands",
"commands": ["create"],
},
],
"nf-core modules": [
{
"name": "For pipelines",
Expand Down Expand Up @@ -302,6 +316,35 @@ def launch(
sys.exit(1)


# nf-core configs
@nf_core_cli.group()
@click.pass_context
def configs(ctx):
"""
Commands to manage nf-core pipelines.
"""
# ensure that ctx.obj exists and is a dict (in case `cli()` is called
# by means other than the `if` block below)
ctx.ensure_object(dict)


@configs.command("create")
def create_configs():
"""
Command to interactively create a nextflow config
"""
from nf_core.configs.create import ConfigsCreateApp

try:
log.info("Launching interactive nf-core configs creation tool.")
app = ConfigsCreateApp()
app.run()
sys.exit(app.return_code or 0)
except UserWarning as e:
log.error(e)
sys.exit(1)


# nf-core create-params-file
@nf_core_cli.command()
@click.argument("pipeline", required=False, metavar="<pipeline name>")
Expand Down Expand Up @@ -466,53 +509,6 @@ def licences(pipeline, json):
sys.exit(1)


# nf-core create
@nf_core_cli.command()
@click.option(
"-n",
"--name",
type=str,
help="The name of your new pipeline",
)
@click.option("-d", "--description", type=str, help="A short description of your pipeline")
@click.option("-a", "--author", type=str, help="Name of the main author(s)")
@click.option("--version", type=str, default="1.0dev", help="The initial version number to use")
@click.option(
"-f",
"--force",
is_flag=True,
default=False,
help="Overwrite output directory if it already exists",
)
@click.option("-o", "--outdir", help="Output directory for new pipeline (default: pipeline name)")
@click.option("-t", "--template-yaml", help="Pass a YAML file to customize the template")
@click.option("--plain", is_flag=True, help="Use the standard nf-core template")
def create(name, description, author, version, force, outdir, template_yaml, plain):
"""
Create a new pipeline using the nf-core template.

Uses the nf-core template to make a skeleton Nextflow pipeline with all required
files, boilerplate code and best-practices.
"""
from nf_core.create import PipelineCreate

try:
create_obj = PipelineCreate(
name,
description,
author,
version=version,
force=force,
outdir=outdir,
template_yaml_path=template_yaml,
plain=plain,
)
create_obj.init_pipeline()
except UserWarning as e:
log.error(e)
sys.exit(1)


# nf-core lint
@nf_core_cli.command()
@click.option(
Expand Down Expand Up @@ -630,6 +626,123 @@ def lint(
sys.exit(1)


# nf-core pipelines subcommands
@nf_core_cli.group()
@click.pass_context
def pipelines(ctx):
"""
Commands to manage nf-core pipelines.
"""
# ensure that ctx.obj exists and is a dict (in case `cli()` is called
# by means other than the `if` block below)
ctx.ensure_object(dict)


# nf-core pipelines create
@pipelines.command("create")
@click.pass_context
@click.option(
"-n",
"--name",
type=str,
help="The name of your new pipeline",
)
@click.option("-d", "--description", type=str, help="A short description of your pipeline")
@click.option("-a", "--author", type=str, help="Name of the main author(s)")
@click.option("--version", type=str, default="1.0.0dev", help="The initial version number to use")
@click.option(
"-f",
"--force",
is_flag=True,
default=False,
help="Overwrite output directory if it already exists",
)
@click.option("-o", "--outdir", help="Output directory for new pipeline (default: pipeline name)")
@click.option("-t", "--template-yaml", help="Pass a YAML file to customize the template")
@click.option(
"--organisation",
type=str,
default="nf-core",
help="The name of the GitHub organisation where the pipeline will be hosted (default: nf-core)",
)
def create_pipeline(ctx, name, description, author, version, force, outdir, template_yaml, organisation):
"""
Create a new pipeline using the nf-core template.

Uses the nf-core template to make a skeleton Nextflow pipeline with all required
files, boilerplate code and best-practices.
\n\n
Run without any command line arguments to use an interactive interface.
"""
from nf_core.pipelines.create import PipelineCreateApp
from nf_core.pipelines.create.create import PipelineCreate

if (name and description and author) or (template_yaml):
# If all command arguments are used, run without the interactive interface
try:
create_obj = PipelineCreate(
name,
description,
author,
version=version,
force=force,
outdir=outdir,
organisation=organisation,
)
create_obj.init_pipeline()
except UserWarning as e:
log.error(e)
sys.exit(1)
elif name or description or author or version != "1.0.0dev" or force or outdir or organisation != "nf-core":
log.error(
"Command arguments are not accepted in interactive mode.\n"
"Run with all command line arguments to avoid using an interactive interface"
"or run without any command line arguments to use an interactive interface."
)
sys.exit(1)
else:
log.info(
"Launching interactive nf-core pipeline creation tool."
"\nRun with all command line arguments to avoid using an interactive interface."
)
app = PipelineCreateApp()
app.run()
sys.exit(app.return_code or 0)


# nf-core create (deprecated)
@nf_core_cli.command(hidden=True, deprecated=True)
@click.option(
"-n",
"--name",
type=str,
help="The name of your new pipeline",
)
@click.option("-d", "--description", type=str, help="A short description of your pipeline")
@click.option("-a", "--author", type=str, help="Name of the main author(s)")
@click.option("--version", type=str, help="The initial version number to use")
@click.option(
"-f",
"--force",
is_flag=True,
default=False,
help="Overwrite output directory if it already exists",
)
@click.option("-o", "--outdir", help="Output directory for new pipeline (default: pipeline name)")
@click.option("-t", "--template-yaml", help="Pass a YAML file to customize the template")
@click.option("--plain", is_flag=True, help="Use the standard nf-core template")
def create(name, description, author, version, force, outdir, template_yaml, plain):
"""
DEPRECATED
Create a new pipeline using the nf-core template.

Uses the nf-core template to make a skeleton Nextflow pipeline with all required
files, boilerplate code and best-practices.
"""
log.error("[bold][green]nf-core create[/] command is deprecated. Use [green]nf-core pipelines create[/].[/]")
sys.exit(0)


# nf-core modules subcommands
@nf_core_cli.group()
@click.option(
Expand Down
7 changes: 3 additions & 4 deletions nf_core/components/components_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
The ComponentsTest class handles the generation and testing of nf-test snapshots.
"""


import logging
import os
import re
Expand Down Expand Up @@ -91,9 +90,9 @@ def run(self) -> None:
"""Run build steps"""
self.check_inputs()
os.environ["NFT_DIFF"] = "pdiff" # set nf-test differ to pdiff to get a better diff output
os.environ[
"NFT_DIFF_ARGS"
] = "--line-numbers --expand-tabs=2" # taken from https://code.askimed.com/nf-test/docs/assertions/snapshots/#snapshot-differences
os.environ["NFT_DIFF_ARGS"] = (
"--line-numbers --expand-tabs=2" # taken from https://code.askimed.com/nf-test/docs/assertions/snapshots/#snapshot-differences
)
with nf_core.utils.set_wd(Path(self.dir)):
self.check_snapshot_stability()
if len(self.errors) > 0:
Expand Down
1 change: 0 additions & 1 deletion nf_core/components/create.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
The ComponentCreate class handles generating of module and subworkflow templates
"""


import glob
import json
import logging
Expand Down
1 change: 0 additions & 1 deletion nf_core/components/lint/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
in nf-core pipelines
"""


import logging
import operator
import os
Expand Down
1 change: 1 addition & 0 deletions nf_core/components/nfcore_component.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""
The NFCoreComponent class holds information and utility functions for a single module or subworkflow
"""

import logging
import re
from pathlib import Path
Expand Down
1 change: 1 addition & 0 deletions nf_core/configs/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from .create import ConfigsCreateApp
Loading