diff --git a/.github/workflows/helm-charts.yaml b/.github/workflows/helm-charts.yaml index caf1a956ac..fa0b791630 100644 --- a/.github/workflows/helm-charts.yaml +++ b/.github/workflows/helm-charts.yaml @@ -41,6 +41,7 @@ jobs: exit 1 fi - name: Publish Helm chart to GHCR + working-directory: charts if: ${{ github.event_name == 'push' || github.event_name == 'workflow_dispatch' }} run: helm push ${{ matrix.chart }}-*.tgz oci://ghcr.io/flyteorg/helm-charts diff --git a/charts/flyte-binary/README.md b/charts/flyte-binary/README.md index c81a0f9e72..46b62377a9 100644 --- a/charts/flyte-binary/README.md +++ b/charts/flyte-binary/README.md @@ -155,3 +155,4 @@ Chart for basic single Flyte executable deployment | serviceAccount.imagePullSecrets | list | `[]` | | | serviceAccount.labels | object | `{}` | | | serviceAccount.name | string | `""` | | + diff --git a/charts/flyte-core/README.md b/charts/flyte-core/README.md index 0be1360f43..7150f049ce 100644 --- a/charts/flyte-core/README.md +++ b/charts/flyte-core/README.md @@ -261,4 +261,4 @@ helm install gateway bitnami/contour -n flyte | webhook.serviceAccount.create | bool | `true` | Should a service account be created for the webhook | | webhook.serviceAccount.imagePullSecrets | list | `[]` | ImagePullSecrets to automatically assign to the service account | | workflow_notifications | object | `{"config":{},"enabled":false}` | **Optional Component** Workflow notifications module is an optional dependency. Flyte uses cloud native pub-sub systems to notify users of various events in their workflows | -| workflow_scheduler | object | `{"config":{},"enabled":false,"type":""}` | **Optional Component** Flyte uses a cloud hosted Cron scheduler to run workflows on a schedule. The following module is optional. Without, this module, you will not have scheduled launchplans / workflows. Docs: https://docs.flyte.org/en/latest/howto/enable_and_use_schedules.html#setting-up-scheduled-workflows | \ No newline at end of file +| workflow_scheduler | object | `{"config":{},"enabled":false,"type":""}` | **Optional Component** Flyte uses a cloud hosted Cron scheduler to run workflows on a schedule. The following module is optional. Without, this module, you will not have scheduled launchplans / workflows. Docs: https://docs.flyte.org/en/latest/howto/enable_and_use_schedules.html#setting-up-scheduled-workflows | diff --git a/charts/flyte-deps/README.md b/charts/flyte-deps/README.md index 4d21e6e18e..0cbf338fed 100644 --- a/charts/flyte-deps/README.md +++ b/charts/flyte-deps/README.md @@ -119,4 +119,4 @@ helm upgrade -f values.yaml flyte . | sparkoperator | object | `{"enabled":false}` | Optional: Spark Plugin using the Spark Operator | | sparkoperator.enabled | bool | `false` | - enable or disable Sparkoperator deployment installation | | webhook.enabled | bool | `true` | | -| webhook.service | object | `{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"}` | Service settings for the webhook | \ No newline at end of file +| webhook.service | object | `{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"}` | Service settings for the webhook | diff --git a/charts/flyte-sandbox/README.md b/charts/flyte-sandbox/README.md index 47821a9eac..2196a66bb4 100644 --- a/charts/flyte-sandbox/README.md +++ b/charts/flyte-sandbox/README.md @@ -101,3 +101,4 @@ A Helm chart for the Flyte local sandbox | sandbox.proxy.image.pullPolicy | string | `"Never"` | | | sandbox.proxy.image.repository | string | `"envoyproxy/envoy"` | | | sandbox.proxy.image.tag | string | `"sandbox"` | | + diff --git a/charts/flyte/README.md b/charts/flyte/README.md index 922d184c0c..75f8912e0d 100644 --- a/charts/flyte/README.md +++ b/charts/flyte/README.md @@ -249,4 +249,4 @@ helm upgrade -f values-sandbox.yaml flyte . | redoc.service | object | `{"type":"ClusterIP"}` | Service settings for Minio | | redoc.tolerations | list | `[]` | tolerations for Minio deployment | | sparkoperator | object | `{"enabled":false}` | Optional: Spark Plugin using the Spark Operator | -| sparkoperator.enabled | bool | `false` | - enable or disable Sparkoperator deployment installation | \ No newline at end of file +| sparkoperator.enabled | bool | `false` | - enable or disable Sparkoperator deployment installation | diff --git a/charts/flyteagent/README.md b/charts/flyteagent/README.md index 10f65508f1..523312ac17 100644 --- a/charts/flyteagent/README.md +++ b/charts/flyteagent/README.md @@ -35,3 +35,4 @@ A Helm chart for Flyte agent | serviceAccount.create | bool | `true` | Should a service account be created for flyteagent | | serviceAccount.imagePullSecrets | list | `[]` | ImagePullSecrets to automatically assign to the service account | | tolerations | list | `[]` | tolerations for flyteagent deployment | + diff --git a/rsts/deployment/agents/bigquery.rst b/rsts/deployment/agents/bigquery.rst new file mode 100644 index 0000000000..ca2b704986 --- /dev/null +++ b/rsts/deployment/agents/bigquery.rst @@ -0,0 +1,90 @@ +.. _deployment-agent-setup-bigquery: + +Google BigQuery Agent +====================== + +This guide provides an overview of setting up BigQuery agent in your Flyte deployment. +Please note that the BigQuery agent requires Flyte deployment in the GCP cloud; +it is not compatible with demo/AWS/Azure. + +Set up the GCP Flyte cluster +---------------------------- + +* Ensure you have a functional Flyte cluster running in `GCP `__. +* Create a service account for BigQuery. For more details, refer to: https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries. +* Verify that you have the correct kubeconfig and have selected the appropriate Kubernetes context. +* Confirm that you have the correct Flytectl configuration at ``~/.flyte/config.yaml``. + +Specify agent configuration +---------------------------- + +.. tabs:: + + .. group-tab:: Flyte binary + + Edit the relevant YAML file to specify the agent. + + .. code-block:: yaml + :emphasize-lines: 7,11 + + tasks: + task-plugins: + enabled-plugins: + - container + - sidecar + - k8s-array + - bigquery + default-for-task-types: + - container: container + - container_array: k8s-array + - bigquery_query_job_task: agent-service + + .. group-tab:: Flyte core + + Create a file named ``values-override.yaml`` and add the following configuration to it. + + .. code-block:: yaml + + configmap: + enabled_plugins: + # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) + tasks: + # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) + task-plugins: + # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend + enabled-plugins: + - container + - sidecar + - k8s-array + - bigquery + default-for-task-types: + container: container + sidecar: sidecar + container_array: k8s-array + bigquery_query_job_task: agent-service + +Ensure that the propeller has the correct service account for BigQuery. + +Upgrade the Flyte Helm release +------------------------------ + +.. tabs:: + + .. group-tab:: Flyte binary + + .. code-block:: bash + + helm upgrade flyteorg/flyte-binary -n --values + + Replace ```` with the name of your release (e.g., ``flyte-backend``), + ```` with the name of your namespace (e.g., ``flyte``), + and ```` with the name of your YAML file. + + .. group-tab:: Flyte core + + .. code-block:: bash + + helm upgrade flyte/flyte-core -n --values values-override.yaml + + Replace ```` with the name of your release (e.g., ``flyte``) + and ```` with the name of your namespace (e.g., ``flyte``). diff --git a/rsts/deployment/agents/index.rst b/rsts/deployment/agents/index.rst new file mode 100644 index 0000000000..57232864a7 --- /dev/null +++ b/rsts/deployment/agents/index.rst @@ -0,0 +1,26 @@ +.. _deployment-agent-setup: + +Agent Setup +=========== + +.. tags:: Agent, Integration, Data, Advanced + +Discover the process of setting up Agents for Flyte. + +.. panels:: + :header: text-center + :column: col-lg-12 p-2 + + .. link-button:: deployment-agent-setup-bigquery + :type: ref + :text: Bigquery Agent + :classes: btn-block stretched-link + ^^^^^^^^^^^^ + Guide to setting up the Bigquery agent. + +.. toctree:: + :maxdepth: 1 + :name: Agent setup + :hidden: + + bigquery diff --git a/rsts/deployment/index.rst b/rsts/deployment/index.rst index 9417c28204..8f569eb843 100644 --- a/rsts/deployment/index.rst +++ b/rsts/deployment/index.rst @@ -31,6 +31,15 @@ plugins, authentication, performance tuning, and maintaining Flyte as a producti --- + .. link-button:: deployment-agent-setup + :type: ref + :text: 🤖 Agent Setup + :classes: btn-block stretched-link + ^^^^^^^^^^^^ + Enable Flyte agents to extend Flyte's capabilities, including features like File sesnor, Databricks job, and Snowflake query services. + + --- + .. link-button:: deployment-configuration :type: ref :text: 🎛 Cluster Configuration diff --git a/rsts/index.rst b/rsts/index.rst index 2821c89218..a3fdd219a0 100644 --- a/rsts/index.rst +++ b/rsts/index.rst @@ -31,6 +31,7 @@ deployment/index deployment/deployment/index deployment/plugins/index + deployment/agents/index deployment/configuration/index deployment/configuration/generated/index deployment/security/index