Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase more memory limits in flyteagent #5550

Merged
merged 1 commit into from
Jul 10, 2024

Conversation

Future-Outlier
Copy link
Member

@Future-Outlier Future-Outlier commented Jul 10, 2024

Tracking issue

#3936

Why are the changes needed?

I got OOM killed When I started the flyte sandbox with the agent pod.
It is not a good user experience for agent users when we run lots os agent tasks concurrently.
As we support more agents in flytekit, we will get OOM killed.

What changes were proposed in this pull request?

Add more memory limits for flyteagent pod.

How was this patch tested?

build an image of flyte sandbox.

cd docker/sandbox-bundled
make build

Setup process

Screenshots

Agent Pod has to restart due to OOM Killed

image

Agent Pod doesn't have to restart after giving more memories

image

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Copy link

codecov bot commented Jul 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 60.98%. Comparing base (b7e6959) to head (edfc9c9).
Report is 113 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #5550   +/-   ##
=======================================
  Coverage   60.98%   60.98%           
=======================================
  Files         796      796           
  Lines       51647    51647           
=======================================
  Hits        31498    31498           
  Misses      17249    17249           
  Partials     2900     2900           
Flag Coverage Δ
unittests-datacatalog 69.31% <ø> (ø)
unittests-flyteadmin 58.73% <ø> (ø)
unittests-flytecopilot 17.79% <ø> (ø)
unittests-flytectl 67.44% <ø> (ø)
unittests-flyteidl 79.06% <ø> (ø)
unittests-flyteplugins 61.85% <ø> (ø)
unittests-flytepropeller 57.42% <ø> (ø)
unittests-flytestdlib 65.68% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pingsutw pingsutw merged commit 81afb76 into master Jul 10, 2024
57 checks passed
@pingsutw pingsutw deleted the give-more-memory-to-flyte-agents branch July 10, 2024 17:28
vlibov pushed a commit to vlibov/flyte that referenced this pull request Aug 16, 2024
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Vladyslav Libov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants