An "Auto Try Again?" will appear when the hangout is full. When clicked, you can try again automatically and it will auto join when there is a free spot.
The minimum delay is 5 seconds, but you can increase that in options. This number has been set this so that you wont get banned by Google and not stress the Google Hangout servers.