Merged master

Signed-off-by: Kevin Su <[email protected]>
flyteorg · Aug 9, 2023 · fbb0760 · fbb0760
2 parents 93e86df + 06f0d40
commit fbb0760
Show file tree

Hide file tree

Showing 63 changed files with 3,862 additions and 2,114 deletions.
diff --git a/CHANGELOG/CHANGELOG-v1.8.0.md b/CHANGELOG/CHANGELOG-v1.8.0.md
@@ -0,0 +1,59 @@
+# Flyte v1.8.0 Release
+
+## Flytekit
+* Support configuring pip index url with the image spec by @yini7777 in https://github.com/flyteorg/flytekit/pull/1692
+* Improve error message for pyflyte-fast-execute by @pingsutw in https://github.com/flyteorg/flytekit/pull/1686
+* Feat: Allow disabling rich tracebacks via env var by @fg91 in https://github.com/flyteorg/flytekit/pull/1695
+* Fix task type version in pytorch and mpi task plugin  by @yubofredwang in https://github.com/flyteorg/flytekit/pull/1690
+* get_transformer returns pickle transformer if type is unsupported  by @pingsutw in https://github.com/flyteorg/flytekit/pull/1694
+* Add Cuda to ImageSpec by @pingsutw in https://github.com/flyteorg/flytekit/pull/1688
+* Feat: Warn when doing local torch elastic training with nnodes > 1 by @fg91 in https://github.com/flyteorg/flytekit/pull/1697
+* Fix config of user facing execution parameters in spawning elastic tasks by @fg91 in https://github.com/flyteorg/flytekit/pull/1677
+* Union | optional return types supported by @kumare3 in https://github.com/flyteorg/flytekit/pull/1703
+* Fail registration if output isn't Optional when using map tasks with min_success_ratio < 1 by @pingsutw in https://github.com/flyteorg/flytekit/pull/1709
+* Add bigquery project and location to metadata by @pingsutw in https://github.com/flyteorg/flytekit/pull/1702
+* Generate decks at local execution by @pingsutw in https://github.com/flyteorg/flytekit/pull/1699
+* Improve task type hint by @pingsutw in https://github.com/flyteorg/flytekit/pull/1711
+* Detect upstream nodes from container nested promises by @wild-endeavor in https://github.com/flyteorg/flytekit/pull/1707
+* Add Iterator Transformer by @pingsutw in https://github.com/flyteorg/flytekit/pull/1710
+* Always a default image and streamline image handling serialize vs package by @wild-endeavor in https://github.com/flyteorg/flytekit/pull/1610
+* Remove ensure literal by @wild-endeavor in https://github.com/flyteorg/flytekit/pull/1706
+* Fix a bug in gx integration by @XinEDprob in https://github.com/flyteorg/flytekit/pull/1675
+* Update pythonbuild.yml: add hugging face plugin to CI by @cosmicBboy in https://github.com/flyteorg/flytekit/pull/1684
+* add imagespec cache by @RichhLi in https://github.com/flyteorg/flytekit/pull/1717
+* Add requirements to imageSpec by @pingsutw in https://github.com/flyteorg/flytekit/pull/1698
+* Memory reduction change by @wild-endeavor in https://github.com/flyteorg/flytekit/pull/1716
+* use getattr to access task_def.disable_deck in entrypoint by @cosmicBboy in https://github.com/flyteorg/flytekit/pull/1724
+* Add a couple tests by @wild-endeavor in https://github.com/flyteorg/flytekit/pull/1722
+* Csvtransform by @ChungYujoyce in https://github.com/flyteorg/flytekit/pull/1671
+
+## Flyteadmin
+* Add a user-specifiable root to upload link request by @wild-endeavor in https://github.com/flyteorg/flyteadmin/pull/577
+* Bump propeller version by @pingsutw in https://github.com/flyteorg/flyteadmin/pull/580
+* Upgrade go 1.19 in Dockerfile by @hamersaw in https://github.com/flyteorg/flyteadmin/pull/581
+
+## Flytepropeller
+* Update flyteplugins to v1.0.67 by @bstadlbauer in https://github.com/flyteorg/flytepropeller/pull/575
+* Correctly validating error code on aborting terminal launchplans by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/574
+* Use GetExecutionData instead by @EngHabu in https://github.com/flyteorg/flytepropeller/pull/573
+* Propagating environment variables through launchplans by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/576
+* upgrade dockerfile go to 1.19 by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/580
+* Bump flyteplugins version by @pingsutw in https://github.com/flyteorg/flytepropeller/pull/581
+* Support for cross-project secrets for GCP by @jeevb in https://github.com/flyteorg/flytepropeller/pull/582
+* Added IsFailurePermanent flag on DynamicTaskStatus by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/567
+
+## Flyteplugins
+* Don't add master replica log link when doing elastic pytorch training by @fg91 in https://github.com/flyteorg/flyteplugins/pull/356
+* [Bigquery] Add support for impersonation of GSA bound to task's KSA by @jeevb in https://github.com/flyteorg/flyteplugins/pull/355
+* Fix initial dask job state by @bstadlbauer in https://github.com/flyteorg/flyteplugins/pull/357
+* Feat: Add pod start and finish time in RFC3339 time format to logging link templating variables #minor by @fg91 in https://github.com/flyteorg/flyteplugins/pull/360
+* fix v1 pytorch job plugin with elastic policy by @yubofredwang in https://github.com/flyteorg/flyteplugins/pull/359
+* Allow using pod start time in kubeflow plugin log links by @fg91 in https://github.com/flyteorg/flyteplugins/pull/362
+* Pass location to the BigQuery request by @pingsutw in https://github.com/flyteorg/flyteplugins/pull/365
+* Fix map task cache misses by @bstadlbauer in https://github.com/flyteorg/flyteplugins/pull/363
+* Switch to official dask operator by @bstadlbauer in https://github.com/flyteorg/flyteplugins/pull/366
+* Fix duplicate env vars in container by @hamersaw in https://github.com/flyteorg/flyteplugins/pull/358
+* Set scheduler restart policy to Always by @bstadlbauer in https://github.com/flyteorg/flyteplugins/pull/367
+
+## Flyteconsole
+* Fix launch plan icon https://github.com/flyteorg/flyteconsole/pull/777
diff --git a/boilerplate/flyte/end2end/run-tests.py b/boilerplate/flyte/end2end/run-tests.py
@@ -190,8 +190,10 @@ def run(
 
     # For a given release tag and priority, this function filters the workflow groups from the flytesnacks
     # manifest file. For example, for the release tag "v0.2.224" and the priority "P0" it returns [ "core" ].
-    manifest_url = "https://raw.githubusercontent.com/flyteorg/flytesnacks/" \
-                   f"{flytesnacks_release_tag}/cookbook/flyte_tests_manifest.json"
+    manifest_url = (
+        "https://raw.githubusercontent.com/flyteorg/flytesnacks/"
+        f"{flytesnacks_release_tag}/flyte_tests_manifest.json"
+    )
     r = requests.get(manifest_url)
     parsed_manifest = r.json()
 

diff --git a/charts/flyte-binary/README.md b/charts/flyte-binary/README.md
@@ -161,11 +161,17 @@ Chart for basic single Flyte executable deployment
 | ingress.grpcAnnotations | object | `{}` |  |
 | ingress.grpcExtraPaths.append | list | `[]` |  |
 | ingress.grpcExtraPaths.prepend | list | `[]` |  |
+| ingress.grpcIngressClassName | string | `""` |  |
+| ingress.grpcTls | list | `[]` |  |
 | ingress.host | string | `""` |  |
 | ingress.httpAnnotations | object | `{}` |  |
 | ingress.httpExtraPaths.append | list | `[]` |  |
 | ingress.httpExtraPaths.prepend | list | `[]` |  |
+| ingress.httpIngressClassName | string | `""` |  |
+| ingress.httpTls | list | `[]` |  |
+| ingress.ingressClassName | string | `""` |  |
 | ingress.labels | object | `{}` |  |
+| ingress.tls | list | `[]` |  |
 | nameOverride | string | `""` |  |
 | rbac.annotations | object | `{}` |  |
 | rbac.create | bool | `true` |  |

diff --git a/charts/flyte-binary/templates/ingress/grpc.yaml b/charts/flyte-binary/templates/ingress/grpc.yaml
@@ -22,6 +22,11 @@ metadata:
     {{- tpl ( .Values.ingress.grpcAnnotations | toYaml ) . | nindent 4 }}
     {{- end }}
 spec:
+  {{- if .Values.ingress.grpcIngressClassName }}
+  ingressClassName: {{ .Values.ingress.grpcIngressClassName | quote }}
+  {{- else if .Values.ingress.ingressClassName }}
+  ingressClassName: {{ .Values.ingress.ingressClassName | quote }}
+  {{- end }}
   rules:
   - http:
       paths:
@@ -118,4 +123,9 @@ spec:
     {{- if .Values.ingress.host }}
     host: {{ tpl .Values.ingress.host . | quote }}
     {{- end }}
+    {{- if .Values.ingress.grpcTls }}
+    tls: {{- tpl ( .Values.ingress.grpcTls | toYaml ) . | nindent 6 }}
+    {{- else if .Values.ingress.tls }}
+    tls: {{- tpl ( .Values.ingress.tls | toYaml ) . | nindent 6 }}
+    {{- end }}
 {{- end }}
diff --git a/charts/flyte-binary/templates/ingress/http.yaml b/charts/flyte-binary/templates/ingress/http.yaml
@@ -22,6 +22,11 @@ metadata:
     {{- tpl ( .Values.ingress.httpAnnotations | toYaml ) . | nindent 4 }}
     {{- end }}
 spec:
+  {{- if .Values.ingress.httpIngressClassName }}
+  ingressClassName: {{ .Values.ingress.httpIngressClassName | quote }}
+  {{- else if .Values.ingress.ingressClassName }}
+  ingressClassName: {{ .Values.ingress.ingressClassName | quote }}
+  {{- end }}
   rules:
   - http:
       paths:
@@ -167,4 +172,9 @@ spec:
     {{- if .Values.ingress.host }}
     host: {{ tpl .Values.ingress.host . | quote }}
     {{- end }}
+    {{- if .Values.ingress.httpTls }}
+    tls: {{- tpl ( .Values.ingress.httpTls | toYaml ) . | nindent 6 }}
+    {{- else if .Values.ingress.tls }}
+    tls: {{- tpl ( .Values.ingress.tls | toYaml ) . | nindent 6 }}
+    {{- end }}
 {{- end }}
diff --git a/charts/flyte-binary/values.yaml b/charts/flyte-binary/values.yaml
@@ -321,6 +321,18 @@ ingress:
   httpAnnotations: {}
   # grpcAnnotations Add annotations to grpc ingress resource
   grpcAnnotations: {}
+  # ingressClassName Ingress class to use with all ingress resources
+  ingressClassName: ""
+  # httpIngressClassName Ingress class to use with all http ingress resource. Overrides `ingressClassName`
+  httpIngressClassName: ""
+  # grpcIngressClassName Ingress class to use with all grpc ingress resource. Overrides `ingressClassName`
+  grpcIngressClassName: ""
+  # tls Add TLS configuration to all ingress resources
+  tls: []
+  # httpTls Add TLS configuration to http ingress resource. Overrides `tls`
+  httpTls: []
+  # grpcTls Add TLS configuration to grpc ingress resource. Overrides `tls`
+  grpcTls: []
   # httpExtraPaths Add extra paths to http ingress rule
   httpExtraPaths:
     prepend: []

diff --git a/charts/flyte-core/README.md b/charts/flyte-core/README.md
@@ -90,8 +90,8 @@ helm install gateway bitnami/contour -n flyte
 | configmap.clusters.clusterConfigs | list | `[]` |  |
 | configmap.clusters.labelClusterMap | object | `{}` |  |
 | configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"}` | Configuration for Flyte console UI |
-| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.30","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration |
-| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.30","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) |
+| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.32","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration |
+| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.32","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) |
 | configmap.core | object | `{"manager":{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"},"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration |
 | configmap.core.manager | object | `{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/manager/config#Config). |
 | configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). |
@@ -125,7 +125,7 @@ helm install gateway bitnami/contour -n flyte
 | datacatalog.extraArgs | object | `{}` | Appends extra command line arguments to the main command |
 | datacatalog.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy |
 | datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment |
-| datacatalog.image.tag | string | `"v1.0.46"` | Docker image tag |
+| datacatalog.image.tag | string | `"v1.0.48"` | Docker image tag |
 | datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment |
 | datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods |
 | datacatalog.priorityClassName | string | `""` | Sets priorityClassName for datacatalog pod(s). |
@@ -157,7 +157,7 @@ helm install gateway bitnami/contour -n flyte
 | flyteadmin.extraArgs | object | `{}` | Appends extra command line arguments to the serve command |
 | flyteadmin.image.pullPolicy | string | `"IfNotPresent"` |  |
 | flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment |
-| flyteadmin.image.tag | string | `"v1.1.105"` |  |
+| flyteadmin.image.tag | string | `"v1.1.110"` |  |
 | flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create |
 | flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment |
 | flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods |
@@ -202,7 +202,7 @@ helm install gateway bitnami/contour -n flyte
 | flyteconsole.ga.tracking_id | string | `"G-0QW4DJWJ20"` |  |
 | flyteconsole.image.pullPolicy | string | `"IfNotPresent"` |  |
 | flyteconsole.image.repository | string | `"cr.flyte.org/flyteorg/flyteconsole"` | Docker image for Flyteconsole deployment |
-| flyteconsole.image.tag | string | `"v1.8.4"` |  |
+| flyteconsole.image.tag | string | `"v1.8.5"` |  |
 | flyteconsole.nodeSelector | object | `{}` | nodeSelector for Flyteconsole deployment |
 | flyteconsole.podAnnotations | object | `{}` | Annotations for Flyteconsole pods |
 | flyteconsole.priorityClassName | string | `""` | Sets priorityClassName for flyte console pod(s). |
@@ -222,7 +222,7 @@ helm install gateway bitnami/contour -n flyte
 | flytepropeller.extraArgs | object | `{}` | Appends extra command line arguments to the main command |
 | flytepropeller.image.pullPolicy | string | `"IfNotPresent"` |  |
 | flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment |
-| flytepropeller.image.tag | string | `"v1.1.95"` |  |
+| flytepropeller.image.tag | string | `"v1.1.106"` |  |
 | flytepropeller.manager | bool | `false` |  |
 | flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment |
 | flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods |
@@ -242,7 +242,7 @@ helm install gateway bitnami/contour -n flyte
 | flytescheduler.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files |
 | flytescheduler.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy |
 | flytescheduler.image.repository | string | `"cr.flyte.org/flyteorg/flytescheduler"` | Docker image for Flytescheduler deployment |
-| flytescheduler.image.tag | string | `"v1.1.105"` | Docker image tag |
+| flytescheduler.image.tag | string | `"v1.1.110"` | Docker image tag |
 | flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment |
 | flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods |
 | flytescheduler.priorityClassName | string | `""` | Sets priorityClassName for flyte scheduler pod(s). |

diff --git a/charts/flyte-core/values.yaml b/charts/flyte-core/values.yaml
@@ -16,7 +16,7 @@ flyteadmin:
   image:
     # -- Docker image for Flyteadmin deployment
     repository: cr.flyte.org/flyteorg/flyteadmin # FLYTEADMIN_IMAGE
-    tag: v1.1.105 # FLYTEADMIN_TAG
+    tag: v1.1.110 # FLYTEADMIN_TAG
     pullPolicy: IfNotPresent
   # -- Additional flyteadmin container environment variables
   #
@@ -97,7 +97,7 @@ flytescheduler:
     # -- Docker image for Flytescheduler deployment
     repository: cr.flyte.org/flyteorg/flytescheduler # FLYTESCHEDULER_IMAGE
     # -- Docker image tag
-    tag: v1.1.105 # FLYTESCHEDULER_TAG
+    tag: v1.1.110 # FLYTESCHEDULER_TAG
     # -- Docker image pull policy
     pullPolicy: IfNotPresent
   # -- Default resources requests and limits for Flytescheduler deployment
@@ -151,7 +151,7 @@ datacatalog:
     # -- Docker image for Datacatalog deployment
     repository: cr.flyte.org/flyteorg/datacatalog # DATACATALOG_IMAGE
     # -- Docker image tag
-    tag: v1.0.46 # DATACATALOG_TAG
+    tag: v1.0.48 # DATACATALOG_TAG
     # -- Docker image pull policy
     pullPolicy: IfNotPresent
   # -- Default resources requests and limits for Datacatalog deployment
@@ -274,7 +274,7 @@ flytepropeller:
   image:
     # -- Docker image for Flytepropeller deployment
     repository: cr.flyte.org/flyteorg/flytepropeller # FLYTEPROPELLER_IMAGE
-    tag: v1.1.95 # FLYTEPROPELLER_TAG
+    tag: v1.1.106 # FLYTEPROPELLER_TAG
     pullPolicy: IfNotPresent
   # -- Default resources requests and limits for Flytepropeller deployment
   resources:
@@ -332,7 +332,7 @@ flyteconsole:
   image:
     # -- Docker image for Flyteconsole deployment
     repository: cr.flyte.org/flyteorg/flyteconsole # FLYTECONSOLE_IMAGE
-    tag: v1.8.4 # FLYTECONSOLE_TAG
+    tag: v1.8.5 # FLYTECONSOLE_TAG
     pullPolicy: IfNotPresent
   # -- Default resources requests and limits for Flyteconsole deployment
   resources:
@@ -657,7 +657,7 @@ configmap:
         # -- Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig)
         co-pilot:
           name: flyte-copilot-
-          image: cr.flyte.org/flyteorg/flytecopilot:v0.0.30 # FLYTECOPILOT_IMAGE
+          image: cr.flyte.org/flyteorg/flytecopilot:v0.0.32 # FLYTECOPILOT_IMAGE
           start-timeout: 30s
 
   # -- Core propeller configuration

diff --git a/charts/flyte-sandbox/README.md b/charts/flyte-sandbox/README.md
@@ -92,6 +92,10 @@ A Helm chart for the Flyte local sandbox
 | postgresql.volumePermissions.enabled | bool | `true` |  |
 | postgresql.volumePermissions.image.pullPolicy | string | `"Never"` |  |
 | postgresql.volumePermissions.image.tag | string | `"sandbox"` |  |
+| sandbox.buildkit.enabled | bool | `true` |  |
+| sandbox.buildkit.image.pullPolicy | string | `"Never"` |  |
+| sandbox.buildkit.image.repository | string | `"moby/buildkit"` |  |
+| sandbox.buildkit.image.tag | string | `"sandbox"` |  |
 | sandbox.dev | bool | `false` |  |
 | sandbox.proxy.enabled | bool | `true` |  |
 | sandbox.proxy.image.pullPolicy | string | `"Never"` |  |

diff --git a/charts/flyte-sandbox/templates/_helpers.tpl b/charts/flyte-sandbox/templates/_helpers.tpl
@@ -89,6 +89,14 @@ Name of PersistentVolume and PersistentVolumeClaim for Minio
 {{- printf "%s-minio-storage" .Release.Name -}}
 {{- end }}
 
+{{/*
+Selector labels for Buildkit
+*/}}
+{{- define "flyte-sandbox.buildkitSelectorLabels" -}}
+{{ include "flyte-sandbox.selectorLabels" . }}
+app.kubernetes.io/component: buildkit
+{{- end }}
+
 {{/*
 Selector labels for Envoy proxy
 */}}