idea: run expensive automated tests daily rather than on each PR

[ichub]

No description provided.

[artwyman]

How long do our CI tests take? To require this I feel like they'd need to be long enough to significantly delay merges, or the team would have to be big enough for the merge queue to fall behind.

[ichub]

this issue in particular is about figuring out a threshold duration for tests (i.e. a max length one would reasonably want to wait for tests before being able to hit merge) and then making tests that take longer happen once daily rather than on every merge. it was inspired by @robknight's answer to one of my question on his #1403 PR, where I asked if it would be possible to run the bundling/npm-publishing test script in CI, to which his answer was that it would probably take too long.

I am not proposing we move all our tests into a daily CI.

[robknight]

I don't really know what "too long" is, but I do know that we had an issue for making CI faster, which suggests that we don't want it to go significantly slower than it does now.

One other option might be to have different CI jobs for pull_request and merge_queue, with the latter having extra tests that we wouldn't want to run normally.

[robknight]

@robknight's answer to one of my question on his #1403 PR, where I asked if it would be possible to run the bundling/npm-publishing test script in CI, to which his answer was that it would probably take too long.

I have discovered a way of making this much faster, fwiw.