Travis builds timing out due to module tests running > 10 mins

[shardulm94]

Travis kills a build when there is no output received from the build for 10 mins. As we grow our test cases (especially Spark), we will hit this timeout limit as Gradle by default does not produce an output if tests are working as intended.

I hit this issue in #1189 when I updated read tests to also test the vectorized codepaths. https://travis-ci.org/github/apache/iceberg/jobs/707083672#L1408. The spark2 module tests seem to run for > 10 mins now on Travis nodes.

There are a few ways I think we can tackle this issue.

1. Reduce the tests we have (Not ideal in the long term)
2. Use travis_wait as recommended by Travis for long running tasks. https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received The issue I have with travis_wait is that it suppresses the log from the child command till it is complete. Also timeout supplied to travis_wait is for the whole child command (in our case ./gradlew check which already takes about 20 minutes today.
3. Run a background process to output a line to the log every minute. This similar to what I have done in the past at https://github.com/linkedin/transport/blob/6fc9a9274c147e999b8b52ee286973a57bbbd7f2/.travis.yml#L21-L25. We can also use the travis_jigger command which does a similar thing (and can have a timeout too)
4. Make Gradle print out the name of every test it ran so that it produces some output as it progresses through the tests by modifying

   iceberg/build.gradle

   Line 120 in 063f926

   events "failed"

   This will produce some noise when Gradle tests are run though (even locally, although unsure if there is a way we can apply this only for Travis)

[rdblue]

I've been thinking about reducing the number of test classes to make tests run faster. When I run tests locally, the Spark start-up phase seems to take a long time, and we start a new session in lots of different tests. By merging some of those tests together, we could avoid that cost, making tests faster and not hitting this timeout issue.

[rdsr]

I don't mind too much if tests take a little long to run, especially on travis. It doesn't hamper my dev cycle where I mostly run specific tests while I'm developing and testing my changes. For travis to not time out can we simply print standard error and standard out on the console through Gradle? This will dump the logs when running the tests. The configuration for enabling that is

// show standard out and standard error of the test JVM(s) on the console
  testLogging.showStandardStreams = true

And we should be able to enable this configuration on Travis by checking Travis specific env variables, if this produces too much noise on our local machines.

[shardulm94]

Enabling stdout and stderr will produce way too much noise in the Jenkins log. If we have to go by the approach of making Gradle print something, printing the test name it ran (suggestion No. 4 from the original post) is pretty straightforward. If a single test method is taking > 10 mins, it is not an ideal test case.

[rdblue]

I opened a PR with option 4: #1210

[rdblue]

Fixed by #1210. Thanks for finding all of the options to fix this, @shardulm94!