Capture errors leads to a message about --allow-console-errors

[tmeasday]

If your build has component/story errors, the output is:

Chromatic CLI v5.0.0
https://www.chromatic.com/docs/cli

  ✔ Authenticated with Chromatic
    → Using project token '*******1fim'
  ✔ Retrieved git information
    → Commit '5d2ec3d' on branch 'next'; found 1 baseline commit
  ✔ Collected Storybook metadata
    → Storybook v6.0.0-beta.9 for React; supported addons found: Actions, Backgrounds, Docs, Links, Storyshots, Viewport
  ✔ Storybook built in 15 seconds
    → View build log at /Users/tom/Chroma/chromatic/examples/simple-app/build-storybook.log
  ✔ Publish complete in 15 seconds
    → View your Storybook at https://5a2777b5ac397a00201ccd82-ulbsaigmwu.chromatic.com
  ✔ Started build 617
    → View build details at https://www.chromatic.com/build?appId=5a2777b5ac397a00201ccd82&number=617
  ✔ Build 617 failed after 7 seconds
    → Found 3 errors (1 component, 2 stories, 5 snapshots); exiting with status code 2

ℹ Speed up Continuous Integration
Your project is linked to GitHub so Chromatic will report results there.
This means you can pass the --exit-once-uploaded flag to skip waiting for build results.
Read more here: https://www.chromatic.com/docs/cli#chromatic-options

✖ Encountered 3 build errors: failing with exit code 2
Pass --allow-console-errors to succeed this command regardless of runtime build errors.

A few things here:

• --allow-console-errors used to (when we used JSDOM) mean that we'd continue even if we observed errors logged in the JS console when gathering a story list (i.e. at load time). The reason we had that behaviour in the first place (stopping if we saw console errors), was due to console errors many times leading to people's Storybooks not loading properly, and us capturing an incomplete Storybook, which is a bad experience in Chromatic.
• Do we still stop on console errors now we get the story list w/ puppeteer? I think we probably should for the reason above (and because we are much less likely to get spurious console errors which happened a lot due to JSDOM). I think we lost this functionality in translation.
• In any case --allow-console-errors shouldn't have anything to do with the above errors which are render-time "capture" errors. Passing the flag shouldn't change the behaviour in this case, and if it does it should be named differently (--exit-zero-on-errors would be consistent with --exit-zero-on-changes).
• Finally, did we decide to call this kind of error a "build error"? ("Encountered 3 build errors"). This seems confusing as these are errors for specific components/stories, rather than errors at the build level, e.g. during building or at load time.

┆Issue is synchronized with this Asana task by Unito

[piperchester]

Thanks for opening @tmeasday!

[ghengeveld]

Since we no longer have JSDOM, the whole concept of a "runtime error" can be removed, and the --allow-console-errors flag should be removed as well, since it doesn't do anything anymore.

We could add --exit-zero-on-errors if that's something people need, but that would be a separate feature request.

[paulelliott]

We'll do this as part of #1115