Spark 3.4: Use Awaitility instead of Thread.sleep()

Spark 3.4: Use Awaitility instead of Thread.sleep() #21430