Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task runner plugin docs are loaded too eagerly (replace script plugin docs sometimes) #183

Open
anna-geller opened this issue Oct 28, 2024 · 0 comments
Labels
area/frontend Needs frontend code changes bug Something isn't working

Comments

@anna-geller
Copy link
Member

anna-geller commented Oct 28, 2024

Issue description

Nothing major, just a small annoyance that often the task runner plugin docs takes precedence when it shouldn't. To reproduce, try to navigate to any script plugin property defined after task runner -- instead of seeing script plugin docs, you'll see task runner docs.

Demo https://share.descript.com/view/p5dGR4nODx7

reproducer - navigate the cursor to script or commands and open plugin docs. You'll see task runner docs even though you expect to see dbt or python docs:

id: dwh_and_analytics
namespace: tutorial

tasks:
  - id: dbt
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: clone_repository
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/dbt-demo
        branch: main
      - id: dbt_build
        type: io.kestra.plugin.dbt.cli.DbtCLI
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
        containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
        commands:
          - dbt deps
          - dbt build
        profiles: |
          jaffle_shop:
            outputs:
              dev:
                type: duckdb
                path: dbt.duckdb
                extensions: 
                  - parquet
                fixed_retries: 1
                threads: 16
                timeout_seconds: 300
            target: dev      
      - id: python
        type: io.kestra.plugin.scripts.python.Script
        outputFiles:
          - "*.csv"
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
        containerImage: ghcr.io/kestra-io/duckdb:latest
        script: >
          import duckdb
          import pandas as pd

          conn = duckdb.connect(database='dbt.duckdb', read_only=False)

          tables_query = "SELECT table_name FROM information_schema.tables WHERE
          table_schema = 'main';"

          tables = conn.execute(tables_query).fetchall()

          # Export each table to CSV, excluding tables that start with 'raw' or
          'stg'

          for table_name in tables:
              table_name = table_name[0]
              # Skip tables with names starting with 'raw' or 'stg'
              if not table_name.startswith('raw') and not table_name.startswith('stg'):
                  query = f"SELECT * FROM {table_name}"
                  df = conn.execute(query).fetchdf()
                  df.to_csv(f"{table_name}.csv", index=False)

          conn.close()
@anna-geller anna-geller added bug Something isn't working area/frontend Needs frontend code changes area/backend Needs backend code changes labels Oct 28, 2024
@kestrabot kestrabot bot added this to Issues Oct 28, 2024
@github-project-automation github-project-automation bot moved this to Backlog in Issues Oct 28, 2024
@anna-geller anna-geller added the good first issue Great issue for new contributors label Oct 28, 2024
@Skraye Skraye removed area/backend Needs backend code changes good first issue Great issue for new contributors labels Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/frontend Needs frontend code changes bug Something isn't working
Projects
Status: Backlog
Development

No branches or pull requests

2 participants