Deployment documentation for Bigquery agent

.github/workflows/helm-charts.yaml

@@ -41,6 +41,7 @@ jobs:
               exit 1
           fi
       - name: Publish Helm chart to GHCR
+        working-directory: charts
         if: ${{ github.event_name == 'push' || github.event_name == 'workflow_dispatch' }}
         run:
           helm push ${{ matrix.chart }}-*.tgz oci://ghcr.io/flyteorg/helm-charts

charts/flyte-binary/README.md

@@ -155,3 +155,4 @@ Chart for basic single Flyte executable deployment
 | serviceAccount.imagePullSecrets | list | `[]` |  |
 | serviceAccount.labels | object | `{}` |  |
 | serviceAccount.name | string | `""` |  |
+

charts/flyte-core/README.md

@@ -261,4 +261,4 @@ helm install gateway bitnami/contour -n flyte
 | webhook.serviceAccount.create | bool | `true` | Should a service account be created for the webhook |
 | webhook.serviceAccount.imagePullSecrets | list | `[]` | ImagePullSecrets to automatically assign to the service account |
 | workflow_notifications | object | `{"config":{},"enabled":false}` | **Optional Component** Workflow notifications module is an optional dependency. Flyte uses cloud native pub-sub systems to notify users of various events in their workflows |
-| workflow_scheduler | object | `{"config":{},"enabled":false,"type":""}` | **Optional Component** Flyte uses a cloud hosted Cron scheduler to run workflows on a schedule. The following module is optional. Without, this module, you will not have scheduled launchplans / workflows. Docs: https://docs.flyte.org/en/latest/howto/enable_and_use_schedules.html#setting-up-scheduled-workflows |
+| workflow_scheduler | object | `{"config":{},"enabled":false,"type":""}` | **Optional Component** Flyte uses a cloud hosted Cron scheduler to run workflows on a schedule. The following module is optional. Without, this module, you will not have scheduled launchplans / workflows. Docs: https://docs.flyte.org/en/latest/howto/enable_and_use_schedules.html#setting-up-scheduled-workflows |

charts/flyte-deps/README.md

@@ -119,4 +119,4 @@ helm upgrade -f values.yaml flyte .
 | sparkoperator | object | `{"enabled":false}` | Optional: Spark Plugin using the Spark Operator |
 | sparkoperator.enabled | bool | `false` | - enable or disable Sparkoperator deployment installation |
 | webhook.enabled | bool | `true` |  |
-| webhook.service | object | `{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"}` | Service settings for the webhook |
+| webhook.service | object | `{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"}` | Service settings for the webhook |

charts/flyte-sandbox/README.md

@@ -101,3 +101,4 @@ A Helm chart for the Flyte local sandbox
 | sandbox.proxy.image.pullPolicy | string | `"Never"` |  |
 | sandbox.proxy.image.repository | string | `"envoyproxy/envoy"` |  |
 | sandbox.proxy.image.tag | string | `"sandbox"` |  |
+

charts/flyte/README.md

@@ -249,4 +249,4 @@ helm upgrade -f values-sandbox.yaml flyte .
 | redoc.service | object | `{"type":"ClusterIP"}` | Service settings for Minio |
 | redoc.tolerations | list | `[]` | tolerations for Minio deployment |
 | sparkoperator | object | `{"enabled":false}` | Optional: Spark Plugin using the Spark Operator |
-| sparkoperator.enabled | bool | `false` | - enable or disable Sparkoperator deployment installation |
+| sparkoperator.enabled | bool | `false` | - enable or disable Sparkoperator deployment installation |

charts/flyteagent/README.md

@@ -35,3 +35,4 @@ A Helm chart for Flyte agent
 | serviceAccount.create | bool | `true` | Should a service account be created for flyteagent |
 | serviceAccount.imagePullSecrets | list | `[]` | ImagePullSecrets to automatically assign to the service account |
 | tolerations | list | `[]` | tolerations for flyteagent deployment |
+

rsts/deployment/agents/bigquery.rst

@@ -0,0 +1,90 @@
+.. _deployment-agent-setup-bigquery:
+
+Google BigQuery Agent
+======================
+
+This guide provides an overview of setting up BigQuery agent in your Flyte deployment.
+Please note that the BigQuery agent requires Flyte deployment in the GCP cloud;
+it is not compatible with demo/AWS/Azure.
+
+Set up the GCP Flyte cluster
+----------------------------
+
+* Ensure you have a functional Flyte cluster running in `GCP <https://docs.flyte.org/en/latest/deployment/gcp/index.html#deployment-gcp>`__.
+* Create a service account for BigQuery. For more details, refer to: https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries.
+* Verify that you have the correct kubeconfig and have selected the appropriate Kubernetes context.
+* Confirm that you have the correct Flytectl configuration at ``~/.flyte/config.yaml``.
+
+Specify agent configuration
+----------------------------
+
+.. tabs::
+
+  .. group-tab:: Flyte binary
+
+    Edit the relevant YAML file to specify the agent.
+
+    .. code-block:: yaml
+      :emphasize-lines: 7,11
+
+      tasks:
+        task-plugins:
+          enabled-plugins:
+            - container
+            - sidecar
+            - k8s-array
+            - bigquery
+          default-for-task-types:
+            - container: container
+            - container_array: k8s-array
+            - bigquery_query_job_task: agent-service
+
+  .. group-tab:: Flyte core
+
+    Create a file named ``values-override.yaml`` and add the following configuration to it.
+
+    .. code-block:: yaml
+
+        configmap:
+          enabled_plugins:
+            # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
+            tasks:
+              # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
+              task-plugins:
+                # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend
+                enabled-plugins:
+                  - container
+                  - sidecar
+                  - k8s-array
+                  - bigquery
+                default-for-task-types:
+                  container: container
+                  sidecar: sidecar
+                  container_array: k8s-array
+                  bigquery_query_job_task: agent-service
+
+Ensure that the propeller has the correct service account for BigQuery.
+
+Upgrade the Flyte Helm release
+------------------------------
+
+.. tabs::
+
+  .. group-tab:: Flyte binary
+
+    .. code-block:: bash
+
+      helm upgrade <RELEASE_NAME> flyteorg/flyte-binary -n <YOUR_NAMESPACE> --values <YOUR_YAML_FILE>
+
+    Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte-backend``),
+    ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``),
+    and ``<YOUR_YAML_FILE>`` with the name of your YAML file.
+
+  .. group-tab:: Flyte core
+
+    .. code-block:: bash
+
+      helm upgrade <RELEASE_NAME> flyte/flyte-core -n <YOUR_NAMESPACE> --values values-override.yaml
+
+    Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte``)
+    and ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``).

rsts/deployment/agents/index.rst

@@ -0,0 +1,26 @@
+.. _deployment-agent-setup:
+
+Agent Setup
+===========
+
+.. tags:: Agent, Integration, Data, Advanced
+
+Discover the process of setting up Agents for Flyte.
+
+.. panels::
+    :header: text-center
+    :column: col-lg-12 p-2
+
+    .. link-button:: deployment-agent-setup-bigquery
+       :type: ref
+       :text: Bigquery Agent
+       :classes: btn-block stretched-link
+    ^^^^^^^^^^^^
+    Guide to setting up the Bigquery agent.
+
+.. toctree::
+    :maxdepth: 1
+    :name: Agent setup
+    :hidden:
+
+    bigquery

rsts/deployment/index.rst

@@ -31,6 +31,15 @@ plugins, authentication, performance tuning, and maintaining Flyte as a producti
 
     ---
 
+    .. link-button:: deployment-agent-setup
+       :type: ref
+       :text: 🔥 Agent Setup
+       :classes: btn-block stretched-link
+    ^^^^^^^^^^^^
+    Enable Flyte agents to extend Flyte's capabilities, including features like File sesnor, Databricks job, and Snowflake query services.
+
+    ---
+
     .. link-button:: deployment-configuration
        :type: ref
        :text: 🎛 Cluster Configuration