CRIB-542: crib-integration-tests GH workflow running on push to develop #15221
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rafaelfelix
force-pushed
the
CRIB-542/crib-integration-tests-on-push-to-develop
branch
2 times, most recently
from
cf49bb2 to
3502efa
Compare
|
Instead of re-adding all of the CRIB logic here, you could use a similar for to the breaking changes GQL check: https://github.com/smartcontractkit/chainlink/blob/develop/.github/workflows/operator-ui-ci.yml#L16 |
rafaelfelix
force-pushed
the
CRIB-542/crib-integration-tests-on-push-to-develop
branch
4 times, most recently
from
37c9801 to
6786640
Compare
rafaelfelix
force-pushed
the
CRIB-542/crib-integration-tests-on-push-to-develop
branch
from
6786640 to
cb98d95
Compare
Because our crib-integration-tests runs only on schedule, the PR issuer never really gets the feedback that their change broke CRIB when/after merging to develop, and the CRIB team gets notified at most once a day if `develop` is broken. That + timezone differences between team mates can prolong the resolution of the problem. This PR embeds the crib-integration-tests.yml workflow into the already existing build-publish-develop-pr.yml, since `workflow_run` doesn't provide feedback in the context of a PR/push as well (see https://stackoverflow.com/questions/63343937/how-to-use-the-github-actions-workflow-run-event#comment134992476_65081720) we'd have a check running per PR/push to develop but completely detached from the commit ref, which defeats the purpose of establishing a feedback loop. Keeping these workflows splitted would incur in potential race conditions, such as running tests by pulling a docker tag that doesn't yet exist, or worse: exists but with a different content than the one from the actual commit ref.
running on every commit to every PR would increase our GH runner costs too much
rafaelfelix
force-pushed
the
CRIB-542/crib-integration-tests-on-push-to-develop
branch
5 times, most recently
from
3613228 to
e3c8879
Compare
rafaelfelix
force-pushed
the
CRIB-542/crib-integration-tests-on-push-to-develop
branch
from
e3c8879 to
b63e15d
Compare
rafaelfelix
force-pushed
the
CRIB-542/crib-integration-tests-on-push-to-develop
branch
from
58b467a to
21ed52c
Compare
lukaszcl
approved these changes
Dec 16, 2024
| workflow_call: | ||
| inputs: |
There was a problem hiding this comment.
Please correct me if I'm wrong, but is this going to trigger the test every time we merge into the develop branch? If that's true, a quick check shows there are 10+ PRs daily, resulting in at least 300 runs per month. This could cost us a significant amount of money, I suppose.
My questions are:
- Do we want to spend money on a broken test? It will spin up some pods every time, only to fail.
- Does it really make sense to enable it for every push to develop, considering we can't make it a mandatory check and we don't expect people to fix CRIB?
Maybe it would make sense to completely disable it for now. Once it's fixed, we could run it on a schedule (e.g., once daily) or trigger it by adding a specific label to PRs.
|
closing it after recent discussions. we'll stick with the scheduled run for now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Because our crib-integration-tests runs only on schedule, the PR issuer never really gets the feedback that their change broke CRIB when/after merging to develop, and the CRIB team gets notified at most once a day if
developis broken. That + timezone differences between team mates can prolong the resolution of the problem.This PR
embedscalls the crib-integration-test.yml workflowinto the already existingfrom build-publish-develop-pr.yml, sinceworkflow_rundoesn't provide feedback in the context of a PR/push as well (see https://stackoverflow.com/questions/63343937/how-to-use-the-github-actions-workflow-run-event#comment134992476_65081720) we'd have a check running per PR/push to develop but completely detached from the commit ref, which defeats the purpose of establishing a feedback loop.Keeping these workflows split would incur in potential race conditions, such as running tests by pulling a docker tag that doesn't yet exist, or worse: exists but with a different content than the one from the actual commit ref.