Skip to content

Latest commit

 

History

History
64 lines (53 loc) · 3.23 KB

README-ci-failures.md

File metadata and controls

64 lines (53 loc) · 3.23 KB
Raw

Jobs not starting

CI jobs are dispatched by a script which responds to both the ci-build-me label and comments by MinaProtocol organization members containing exactly !ci-build-me. If your CI job has not started after adding the ci-build-me label, please comment on the pull request with !ci-build-me to attempt to re-trigger the script. If no CI jobs started, check that your membership to O(1) Labs/mina organisation is public. If your membership is private, the jobs will not started and !ci-build-me won't have an impact.

If CI jobs are not running after applying both the ci-build-me label and comment, you may be able to find and fix the error in the script. The script lives in frontend/ci-build-me/src/index.js, and instructions for deploying the new version are in the readme at frontend/ci-build-me/README.md. You should still follow normal procedure: submit a pull request and await approval for the changes before attempting to deploy the fixed script.

Integration test failures

If your CI error is related to a timeout logged by one of the integration test runnners, this is a known issue and re-running the test in the Buildkite interface will usually succeed.

If an issue arises, please post an update in both development on the Mina Protocol discord and engineering-internal on the O(1) Labs discord with the details and links to the failures.

CI environment mismatch

The CI runs its jobs in multiple Docker images. The images that it is using are specified in buildkite/src/Constants/COntainerImages.dhall: the CI uses all Debian images prefixed by minaToolchainBuster.

Theses images are generated by the CI itself, in particular based on the content of the dockerfiles directory and the opam.export file (which describes versions of OCaml packages). If you PR modifies how the images are generated (for example by changing a package version in opam.export), then the CI will not automatically use these new images and will potentially fail.

In this case, you should:

  1. Let the CI run once with the old images. Jobs may fail, but ensure that jobs of the form Docker: toolchain-* succeed. This is the step that creates new Docker images from your PR.
  2. Look for the newly generated images on Google Cloud where they have been automatically uploaded. Look for your branch name and commit hash in the second column. You should find several images suffixed with -buster, -bullseye, -stretch and -focal (Debian versions).
  3. For each such image, retrieve its full name and hash by hovering its link and clicking the Copy full image name tooltip that appears (or retrieve it on the image's page).
  4. Edit buildkite/src/Constants/ContainerImages.dhall and replace the name of each minaToolchain-* image by the ones you retrieved.
  5. Commit the change, the CI should now be using the new images.

Contact

If you have a bugfix for failing CI, or are seeing a CI failure across multiple PRs, the best people to contact are:

  • @bkase (bkase#2492 on discord) (Europe - misc.)
  • @lk86 (linuskrom#2287 on discord) (US West Coast)
  • @OxO1 (awilson#6424 on discord) (US West Coast)
  • @mrmr1993 (matthew#4797 on discord) (UK)