Running inference lib tests

[iboB]

The problem is here that the tests take a lot of time, seconds, and if the test runner doesn't have hardware acceleration they may take tens of seconds.

If we create integration tests which also download models, the time would be even longer.

Adding more inference libs will make this worse.

We need to devise a strategy of how to handle these tests. Are there any better ideas than "Suck it up"?

[zah]

The classic approach here is define many of the tests as separate CI jobs that can be executed in parallel on different machines.

[pminev]

Do we need to test with large models? If we just need to test the SDK, wouldn’t it be sufficient to use a small model that runs faster? If we can do that we can pack the model in the app.