Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix model analysis weekly pipeline failures #975

Merged
merged 1 commit into from
Dec 27, 2024

Conversation

chandrasekaranpradeep
Copy link
Contributor

The PR will fix below issues found in the Model Analysis Weekly pipeline - Link

  1. Unique Op tests are not generated for models like Llama, Gemma and mistral because of the Huggingface access token issue so adding the HF_Token as env in the model analysis weekly yml
  2. Update the exception rule by checking the failed test logs of the unknown compiler component or N/A
  3. Add the module_name for the stereo model and update the module_name for the mistral and clip model in the compile function

Copy link
Contributor

@nvukobratTT nvukobratTT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Once this one is merged, let's kick one more run right away to complete over the weekends :))

Few minor details I spot:

  • PCC issues for E2E run are treated as FFE errors. I think in most of the cases right now this is a consequence of TTNN. So let's move it into that group. Once we fix our Golden checks, we can have a customer error separating the two.
  • When we have unknown errors, let's subtract that percentage from all 3 component passing score results. This is mostly to emphasize that when passing rate is 99%, that it's not actually that close if we have 20% in N/A field. However, this isn't a blocker for this PR.

@chandrasekaranpradeep
Copy link
Contributor Author

chandrasekaranpradeep commented Dec 27, 2024

Once this one is merged, let's kick one more run right away to complete over the weekends :))

FYI, there is weekly scheduled on serbia time: saturday at 12:00 AM
Would you like to trigger it after this PR merge and cancel the weekly scheduled?

@nvukobratTT
Copy link
Contributor

Once this one is merged, let's kick one more run right away to complete over the weekends :))

FYI, there is nightly scheduled on serbia time: saturday at 12:00 AM Would you like to trigger it after this PR merge and cancel the nightly scheduled?

If that si the case, than there is no need to run one manually. Let's wait for regular one to run over the weekend :))

@chandrasekaranpradeep chandrasekaranpradeep force-pushed the pchandrasekaran/fix_model_analysis branch from 6232917 to 2d36123 Compare December 27, 2024 11:23
@chandrasekaranpradeep
Copy link
Contributor Author

PCC issues for E2E run are treated as FFE errors. I think in most of the cases right now this is a consequence of TTNN. So let's move it into that group. Once we fix our Golden checks, we can have a customer error separating the two.

Moved the pcc issues exception rule from Forge Compiler Component to TT-Metal Compiler Component.

When we have unknown errors, let's subtract that percentage from all 3 component passing score results. This is mostly to emphasize that when passing rate is 99%, that it's not actually that close if we have 20% in N/A field. However, this isn't a blocker for this PR.

The unknown compiler component percentage is subtracted from the other compiler components pass percentage.
Compiler Component Pass/fail updater
Percentage Calculation for the compiler components

@chandrasekaranpradeep chandrasekaranpradeep force-pushed the pchandrasekaran/fix_model_analysis branch from 2d36123 to 24882fc Compare December 27, 2024 11:35
@chandrasekaranpradeep chandrasekaranpradeep force-pushed the pchandrasekaran/fix_model_analysis branch from 24882fc to a42139e Compare December 27, 2024 11:55
@chandrasekaranpradeep chandrasekaranpradeep merged commit 4355d7a into main Dec 27, 2024
8 checks passed
@chandrasekaranpradeep chandrasekaranpradeep deleted the pchandrasekaran/fix_model_analysis branch December 27, 2024 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants