Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prefect-aws: Extend worker log with aws-task-id #16116

Open
mch-sb opened this issue Nov 26, 2024 · 0 comments
Open

Prefect-aws: Extend worker log with aws-task-id #16116

mch-sb opened this issue Nov 26, 2024 · 0 comments
Labels
enhancement An improvement of an existing feature integrations Related to integrations with other services

Comments

@mch-sb
Copy link

mch-sb commented Nov 26, 2024

Describe the current behavior

Hey!
If something strange happens with a prefect-flow started by a ecs-worker, it is a little cumbersome to find the full cloud-watch-logs.

For instance in a case like this:
image

Here the task crashed withouth outputing the full logs to the prefect-ui, and we need to dig into the cloud-watch-logs.
But finding the correct cloud-watch-stream is currently not straight forward. We either need to search for something like the flow-name (might not be unique), or around the timestamp the issue occured. If there are a lot of other streams around the same time, this is more timeconsuming than it needs to be.

Describe the proposed behavior

For instance extend https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-aws/prefect_aws/workers/ecs_worker.py#L796 with the task_arn. With this we would have a precise way of finding the correct log-stream

Example Use

aws logs get-log-events \
    --log-group-name '<MY_KNOWN_LOG_GROUP_NAME>' \
    --log-stream-name '<MY_KNOWN_LOG_STREAM_PREFIX>/<MY_UNKNOWN_TASK_ID>' \
    --start-from-head

We could then use the MY_UNKNOWN_TASK_ID logged in the prefect-cloud-ui by the worker, to see if there are some more log-events in cloudwatch

Additional context

If all logs from the log-stream could be delivered to the prefect-ui, that would also be a nice alternative

@mch-sb mch-sb added the enhancement An improvement of an existing feature label Nov 26, 2024
@zzstoatzz zzstoatzz added the integrations Related to integrations with other services label Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature integrations Related to integrations with other services
Projects
None yet
Development

No branches or pull requests

2 participants