Add async delete function in base_agent #1800
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Future Outlier <[email protected]>
Future-Outlier
requested review from
wild-endeavor,
kumare3,
eapolinario,
pingsutw and
cosmicBboy
as code owners
pingsutw
approved these changes
Aug 18, 2023
pingsutw
approved these changes
Aug 19, 2023
8 tasks
jeevb
pushed a commit
that referenced
this pull request
Sep 20, 2023
Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Signed-off-by: Jeev B <[email protected]>
Future-Outlier
added a commit
to Future-Outlier/flytekit
that referenced
this pull request
Oct 3, 2023
Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Signed-off-by: Future Outlier <[email protected]>
jeevb
added a commit
that referenced
this pull request
Nov 1, 2023
* pip through to container Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * move around Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * add asserts Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * delete bad line Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * switch to abc and add support for gpu unpartitioned Signed-off-by: Jeev B <[email protected]> * Add Azure-specific headers when uploading to blob storage (#1784) * Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <[email protected]> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <[email protected]> --------- Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add async delete function in base_agent (#1800) Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add support for execution name prefixes (#1803) Signed-off-by: troychiu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove ref in output (#1794) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Inherit directly from DataClassJsonMixin instead of using @dataclass_json for improved static type checking (#1801) * Inherit directly from DataClassJsonMixin instead of @dataclass_json for improved static type checking As it says in the dataclasses-json README: https://github.com/lidatong/dataclasses-json/blob/89578cb9ebed290e70dba8946bfdb68ff6746755/README.md?plain=1#L111-L129, we can use inheritance for improved static type checking; this one change eliminates something like 467 pyright errors from the flytekit module Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async file sensor (#1790) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager workflows to support async workflows (#1579) * Eager workflows to support async workflows Signed-off-by: Niels Bantilan <[email protected]> * move array node maptask to experimental/__init__.py Signed-off-by: Niels Bantilan <[email protected]> * clean up docs Signed-off-by: Niels Bantilan <[email protected]> * clean up Signed-off-by: Niels Bantilan <[email protected]> * more clean up Signed-off-by: Niels Bantilan <[email protected]> * docs cleanup Signed-off-by: Niels Bantilan <[email protected]> * Update test_eager_workflows.py * clean up timeout handling Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Enable SecretsManager.get to load and return bytes (#1798) * fix secretsmanager Signed-off-by: Yue Shang <[email protected]> * fix lint issue Signed-off-by: Yue Shang <[email protected]> * add doc Signed-off-by: Yue Shang <[email protected]> * fix github check Signed-off-by: Yue Shang <[email protected]> --------- Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Jeev B <[email protected]> * Batch upload flyte directory (#1806) * Batch upload flyte directory Signed-off-by: Kevin Su <[email protected]> * Update get method Signed-off-by: Kevin Su <[email protected]> * Move batch size to type engine Signed-off-by: Kevin Su <[email protected]> * comment Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * Update flytekit/core/type_engine.py Co-authored-by: Eduardo Apolinario <[email protected]> * Add test Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * Better error messaging for overrides (#1807) - using incorrect type of overrides - using incorrect type for resources - using promises in overrides Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Run remote Launchplan from `pyflyte run` (#1785) * Beautified pyflyte run even for every task and workflow - identify a task or a workflow - task or workflow help menus show types and use rich to beautify Signed-off-by: Ketan Umare <[email protected]> * one more improvement Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated command Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * updated formatting Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * bug fixed in types Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add is none function (#1757) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Dynamic workflow should not throw nested task warning (#1812) Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add a manual image building GH action (#1816) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * catch abfs protocol in data_persistence.py/get_filesystem and set anon to False (#1813) Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Jeev B <[email protected]> * None doesnt work Signed-off-by: Jeev B <[email protected]> * unpartitioned selector Signed-off-by: Jeev B <[email protected]> * Fix list of annotated structured dataset (#1817) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit (#1819) * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit Signed-off-by: Chao-Heng Lee <[email protected]> * remove helper of getting env var. Signed-off-by: Chao-Heng Lee <[email protected]> * refactor variable name. Signed-off-by: Chao-Heng Lee <[email protected]> --------- Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async agent delete function for while loop case (#1802) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * refactor Signed-off-by: Jeev B <[email protected]> * fix docs warnings (#1827) Signed-off-by: Jeev B <[email protected]> * Fix extract_task_module (#1829) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Add type support for pydantic BaseModels (#1660) Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * add test for unspecified mig Signed-off-by: Jeev B <[email protected]> * add support for overriding accelerator Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * move from core to extras Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Make FlyteRemote slightly more copy/pastable (#1830) Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte meta inputs (#1823) * Re-orgining pyflyte run Signed-off-by: Ketan Umare <[email protected]> * Pyflyte beautified and simplified Signed-off-by: Ketan Umare <[email protected]> * fixed unit test Signed-off-by: Ketan Umare <[email protected]> * Added Launch options Signed-off-by: Ketan Umare <[email protected]> * lint fix Signed-off-by: Ketan Umare <[email protected]> * test fix Signed-off-by: Ketan Umare <[email protected]> * fixing docs failure Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Use mashumaro to serialize/deserialize dataclass (#1735) Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Databricks Agent (#1797) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Prometheus metrics (#1815) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte register optionally activates schedule (#1832) * Pyflyte register auto activates schedule Signed-off-by: Ketan Umare <[email protected]> * comment addressed Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove versions 3.9 and 3.10 (#1831) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Snowflake agent (#1799) Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: Jeev B <[email protected]> * Update agent metric name (#1835) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * MemVerge MMCloud Agent (#1821) Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add download badges in readme (#1836) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager local entrypoint and support for offloaded types (#1833) * implement eager workflow local entrypoint, support offloaded types Signed-off-by: Niels Bantilan <[email protected]> * wip local entrypoint Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add local entrypoint tests Signed-off-by: Niels Bantilan <[email protected]> * update eager unit tests, delete test script Signed-off-by: Niels Bantilan <[email protected]> * clean up tests Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * remove push step Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * update requirements and add snowflake agent to api reference (#1838) * update requirements and add snowflake agent to api reference Signed-off-by: Samhita Alla <[email protected]> * update requirements Signed-off-by: Samhita Alla <[email protected]> * remove versions Signed-off-by: Samhita Alla <[email protected]> * remove tensorflow-macos Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * downgrade sphinxcontrib-youtube package Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Jeev B <[email protected]> * Fix: Make sure decks created in elastic task workers are transferred to parent process (#1837) * Transfer decks created in the worker process to the parent process Signed-off-by: Fabio Graetz <[email protected]> * Add test for decks in elastic tasks Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> --------- Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Jeev B <[email protected]> * add accept grpc (#1841) * add accept grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * unpin setup.py grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Revert "add accept grpc" This reverts commit 2294592. Signed-off-by: Jeev B <[email protected]> * default headers interceptor Signed-off-by: Jeev B <[email protected]> * setup.py Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fmt Signed-off-by: Jeev B <[email protected]> * move prometheus-client import Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Co-authored-by: Jeev B <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Enable `flytekit` to authenticate with proxy in front of FlyteAdmin (#1787) * Introduce authenticator engine and make proxy auth work Signed-off-by: Fabio Grätz <[email protected]> * Use proxy authed session for client credentials flow Signed-off-by: Fabio Grätz <[email protected]> * Don't use authenticator engine but do proxy authentication via existing external command authenticator Signed-off-by: Fabio Grätz <[email protected]> * Add docstring to AuthenticationHTTPAdapter Signed-off-by: Fabio Grätz <[email protected]> * Address todo in docstring Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided in get_token Signed-off-by: Fabio Grätz <[email protected]> * Refresh proxy creds in session when not existing without triggering 401 Signed-off-by: Fabio Grätz <[email protected]> * Add test for get_session Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Add test for upgrade_channel_to_proxy_authenticated Signed-off-by: Fabio Grätz <[email protected]> * Auth helper tests without use of responses package Signed-off-by: Fabio Grätz <[email protected]> * Feat: Add plugin for generating GCP IAP ID tokens via external command (#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <[email protected]> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <[email protected]> * Remove comments Signed-off-by: Fabio Grätz <[email protected]> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <[email protected]> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <[email protected]> * Minor corrections README.md Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> * Use proxy auth'ed session for device code auth flow Signed-off-by: Fabio Grätz <[email protected]> * Fix token client tests Signed-off-by: Fabio Grätz <[email protected]> * Make poll token endpoint test more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator_with_custom_scopes test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Implement subcommand to generate id tokens for service accounts Signed-off-by: Fabio Graetz <[email protected]> * Test id token generation from service accounts Signed-off-by: Fabio Graetz <[email protected]> * Fix plugin requirements Signed-off-by: Fabio Graetz <[email protected]> * Document usage of generate-service-account-id-token subcommand Signed-off-by: Fabio Grätz <[email protected]> * Document alternative ways to obtain service account id tokens Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Jeev B <[email protected]> * bump flyteidl Signed-off-by: Jeev B <[email protected]> * make requirements Signed-off-by: Jeev B <[email protected]> * fix failing tests Signed-off-by: Jeev B <[email protected]> * move gpu accelerator to flyteidl.core.Resources Signed-off-by: Jeev B <[email protected]> * Use ResourceExtensions for extended resources Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Switch to using ExtendedResources in TaskTemplate Signed-off-by: Jeev B <[email protected]> * cleanups Signed-off-by: Jeev B <[email protected]> * update flyteidl Signed-off-by: Jeev B <[email protected]> * Replace _core_task imports with tasks_pb2 Signed-off-by: Jeev B <[email protected]> * less verbose definitions Signed-off-by: Jeev B <[email protected]> * Attempt at less confusing syntax Signed-off-by: Jeev B <[email protected]> * Streamline UX Signed-off-by: Jeev B <[email protected]> * Run make fmt Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Future Outlier <[email protected]> Signed-off-by: troychiu <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Victor Delépine <[email protected]> Co-authored-by: Future-Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Yi Chiu <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Yue Shang <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Keqiu Hu <[email protected]> Co-authored-by: Jan Fiedler <[email protected]> Co-authored-by: Chao-Heng Lee <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: Arthur Böök <[email protected]> Co-authored-by: Katrina Rogan <[email protected]> Co-authored-by: Po Han(Hank) Huang <[email protected]> Co-authored-by: Edwin Yu <[email protected]> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Fabio Grätz <[email protected]>
ringohoffman
added a commit
to ringohoffman/flytekit
that referenced
this pull request
Nov 24, 2023
* pip through to container Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * move around Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * add asserts Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * delete bad line Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * switch to abc and add support for gpu unpartitioned Signed-off-by: Jeev B <[email protected]> * Add Azure-specific headers when uploading to blob storage (flyteorg#1784) * Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <[email protected]> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <[email protected]> --------- Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add async delete function in base_agent (flyteorg#1800) Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add support for execution name prefixes (flyteorg#1803) Signed-off-by: troychiu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove ref in output (flyteorg#1794) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Inherit directly from DataClassJsonMixin instead of using @dataclass_json for improved static type checking (flyteorg#1801) * Inherit directly from DataClassJsonMixin instead of @dataclass_json for improved static type checking As it says in the dataclasses-json README: https://github.com/lidatong/dataclasses-json/blob/89578cb9ebed290e70dba8946bfdb68ff6746755/README.md?plain=1#L111-L129, we can use inheritance for improved static type checking; this one change eliminates something like 467 pyright errors from the flytekit module Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async file sensor (flyteorg#1790) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager workflows to support async workflows (flyteorg#1579) * Eager workflows to support async workflows Signed-off-by: Niels Bantilan <[email protected]> * move array node maptask to experimental/__init__.py Signed-off-by: Niels Bantilan <[email protected]> * clean up docs Signed-off-by: Niels Bantilan <[email protected]> * clean up Signed-off-by: Niels Bantilan <[email protected]> * more clean up Signed-off-by: Niels Bantilan <[email protected]> * docs cleanup Signed-off-by: Niels Bantilan <[email protected]> * Update test_eager_workflows.py * clean up timeout handling Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Enable SecretsManager.get to load and return bytes (flyteorg#1798) * fix secretsmanager Signed-off-by: Yue Shang <[email protected]> * fix lint issue Signed-off-by: Yue Shang <[email protected]> * add doc Signed-off-by: Yue Shang <[email protected]> * fix github check Signed-off-by: Yue Shang <[email protected]> --------- Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Jeev B <[email protected]> * Batch upload flyte directory (flyteorg#1806) * Batch upload flyte directory Signed-off-by: Kevin Su <[email protected]> * Update get method Signed-off-by: Kevin Su <[email protected]> * Move batch size to type engine Signed-off-by: Kevin Su <[email protected]> * comment Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * Update flytekit/core/type_engine.py Co-authored-by: Eduardo Apolinario <[email protected]> * Add test Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * Better error messaging for overrides (flyteorg#1807) - using incorrect type of overrides - using incorrect type for resources - using promises in overrides Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Run remote Launchplan from `pyflyte run` (flyteorg#1785) * Beautified pyflyte run even for every task and workflow - identify a task or a workflow - task or workflow help menus show types and use rich to beautify Signed-off-by: Ketan Umare <[email protected]> * one more improvement Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated command Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * updated formatting Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * bug fixed in types Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add is none function (flyteorg#1757) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Dynamic workflow should not throw nested task warning (flyteorg#1812) Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add a manual image building GH action (flyteorg#1816) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * catch abfs protocol in data_persistence.py/get_filesystem and set anon to False (flyteorg#1813) Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Jeev B <[email protected]> * None doesnt work Signed-off-by: Jeev B <[email protected]> * unpartitioned selector Signed-off-by: Jeev B <[email protected]> * Fix list of annotated structured dataset (flyteorg#1817) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit (flyteorg#1819) * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit Signed-off-by: Chao-Heng Lee <[email protected]> * remove helper of getting env var. Signed-off-by: Chao-Heng Lee <[email protected]> * refactor variable name. Signed-off-by: Chao-Heng Lee <[email protected]> --------- Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async agent delete function for while loop case (flyteorg#1802) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * refactor Signed-off-by: Jeev B <[email protected]> * fix docs warnings (flyteorg#1827) Signed-off-by: Jeev B <[email protected]> * Fix extract_task_module (flyteorg#1829) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Add type support for pydantic BaseModels (flyteorg#1660) Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * add test for unspecified mig Signed-off-by: Jeev B <[email protected]> * add support for overriding accelerator Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * move from core to extras Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Make FlyteRemote slightly more copy/pastable (flyteorg#1830) Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte meta inputs (flyteorg#1823) * Re-orgining pyflyte run Signed-off-by: Ketan Umare <[email protected]> * Pyflyte beautified and simplified Signed-off-by: Ketan Umare <[email protected]> * fixed unit test Signed-off-by: Ketan Umare <[email protected]> * Added Launch options Signed-off-by: Ketan Umare <[email protected]> * lint fix Signed-off-by: Ketan Umare <[email protected]> * test fix Signed-off-by: Ketan Umare <[email protected]> * fixing docs failure Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Use mashumaro to serialize/deserialize dataclass (flyteorg#1735) Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Databricks Agent (flyteorg#1797) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Prometheus metrics (flyteorg#1815) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte register optionally activates schedule (flyteorg#1832) * Pyflyte register auto activates schedule Signed-off-by: Ketan Umare <[email protected]> * comment addressed Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove versions 3.9 and 3.10 (flyteorg#1831) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Snowflake agent (flyteorg#1799) Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: Jeev B <[email protected]> * Update agent metric name (flyteorg#1835) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * MemVerge MMCloud Agent (flyteorg#1821) Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add download badges in readme (flyteorg#1836) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager local entrypoint and support for offloaded types (flyteorg#1833) * implement eager workflow local entrypoint, support offloaded types Signed-off-by: Niels Bantilan <[email protected]> * wip local entrypoint Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add local entrypoint tests Signed-off-by: Niels Bantilan <[email protected]> * update eager unit tests, delete test script Signed-off-by: Niels Bantilan <[email protected]> * clean up tests Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * remove push step Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * update requirements and add snowflake agent to api reference (flyteorg#1838) * update requirements and add snowflake agent to api reference Signed-off-by: Samhita Alla <[email protected]> * update requirements Signed-off-by: Samhita Alla <[email protected]> * remove versions Signed-off-by: Samhita Alla <[email protected]> * remove tensorflow-macos Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * downgrade sphinxcontrib-youtube package Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Jeev B <[email protected]> * Fix: Make sure decks created in elastic task workers are transferred to parent process (flyteorg#1837) * Transfer decks created in the worker process to the parent process Signed-off-by: Fabio Graetz <[email protected]> * Add test for decks in elastic tasks Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> --------- Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Jeev B <[email protected]> * add accept grpc (flyteorg#1841) * add accept grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * unpin setup.py grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Revert "add accept grpc" This reverts commit 2294592. Signed-off-by: Jeev B <[email protected]> * default headers interceptor Signed-off-by: Jeev B <[email protected]> * setup.py Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fmt Signed-off-by: Jeev B <[email protected]> * move prometheus-client import Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Co-authored-by: Jeev B <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Enable `flytekit` to authenticate with proxy in front of FlyteAdmin (flyteorg#1787) * Introduce authenticator engine and make proxy auth work Signed-off-by: Fabio Grätz <[email protected]> * Use proxy authed session for client credentials flow Signed-off-by: Fabio Grätz <[email protected]> * Don't use authenticator engine but do proxy authentication via existing external command authenticator Signed-off-by: Fabio Grätz <[email protected]> * Add docstring to AuthenticationHTTPAdapter Signed-off-by: Fabio Grätz <[email protected]> * Address todo in docstring Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided in get_token Signed-off-by: Fabio Grätz <[email protected]> * Refresh proxy creds in session when not existing without triggering 401 Signed-off-by: Fabio Grätz <[email protected]> * Add test for get_session Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Add test for upgrade_channel_to_proxy_authenticated Signed-off-by: Fabio Grätz <[email protected]> * Auth helper tests without use of responses package Signed-off-by: Fabio Grätz <[email protected]> * Feat: Add plugin for generating GCP IAP ID tokens via external command (flyteorg#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <[email protected]> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <[email protected]> * Remove comments Signed-off-by: Fabio Grätz <[email protected]> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <[email protected]> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <[email protected]> * Minor corrections README.md Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> * Use proxy auth'ed session for device code auth flow Signed-off-by: Fabio Grätz <[email protected]> * Fix token client tests Signed-off-by: Fabio Grätz <[email protected]> * Make poll token endpoint test more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator_with_custom_scopes test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Implement subcommand to generate id tokens for service accounts Signed-off-by: Fabio Graetz <[email protected]> * Test id token generation from service accounts Signed-off-by: Fabio Graetz <[email protected]> * Fix plugin requirements Signed-off-by: Fabio Graetz <[email protected]> * Document usage of generate-service-account-id-token subcommand Signed-off-by: Fabio Grätz <[email protected]> * Document alternative ways to obtain service account id tokens Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Jeev B <[email protected]> * bump flyteidl Signed-off-by: Jeev B <[email protected]> * make requirements Signed-off-by: Jeev B <[email protected]> * fix failing tests Signed-off-by: Jeev B <[email protected]> * move gpu accelerator to flyteidl.core.Resources Signed-off-by: Jeev B <[email protected]> * Use ResourceExtensions for extended resources Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Switch to using ExtendedResources in TaskTemplate Signed-off-by: Jeev B <[email protected]> * cleanups Signed-off-by: Jeev B <[email protected]> * update flyteidl Signed-off-by: Jeev B <[email protected]> * Replace _core_task imports with tasks_pb2 Signed-off-by: Jeev B <[email protected]> * less verbose definitions Signed-off-by: Jeev B <[email protected]> * Attempt at less confusing syntax Signed-off-by: Jeev B <[email protected]> * Streamline UX Signed-off-by: Jeev B <[email protected]> * Run make fmt Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Future Outlier <[email protected]> Signed-off-by: troychiu <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Victor Delépine <[email protected]> Co-authored-by: Future-Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Yi Chiu <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Yue Shang <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Keqiu Hu <[email protected]> Co-authored-by: Jan Fiedler <[email protected]> Co-authored-by: Chao-Heng Lee <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: Arthur Böök <[email protected]> Co-authored-by: Katrina Rogan <[email protected]> Co-authored-by: Po Han(Hank) Huang <[email protected]> Co-authored-by: Edwin Yu <[email protected]> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Fabio Grätz <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
As title.
Type
Are all requirements met?
Complete description
Async Delete Function
Tracking Issue
flyteorg/flyte#3936
Follow-up issue
NA