Fix ec2 CI testing framework

[samuel40791765]

Issues:

Resolves P113131493

Description of changes:

This fixing ec2 CI testing framework we've turned off. I tried introducing intentional test failures to see if the framework will fail correctly (as shown in the commit history of this PR). I also introduced 10 builds to run simultaneously, to see if we would get the sporadic test failures we had been having.
The general run fails early in the build and the FIPS specific one fails sooner since it's being ran after the sanitizer tests. Good news is all runs fail and succeed as anticipated without the original sporadic issues we've been having.

Call-outs:

1. Apparently the docker container needs to be in "privileged-mode" for the TSAN tests to work, so I've turned that on in the SSM document.
2. An sporadic error message E: Unable to lock the administration directory (/var/lib/dpkg/) is another process using it? occurs when we try to call apt-get update right after the instance is spun up. Calling killall apt apt-get effectively gets around the issue.
   • https://askubuntu.com/questions/15433/unable-to-lock-the-administration-directory-var-lib-dpkg-is-another-process

More details on debugging can be found in thread of P113131493

Testing:

CI changes

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.27%. Comparing base (10a389e) to head (9a75435).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1541      +/-   ##
==========================================
- Coverage   77.33%   77.27%   -0.07%     
==========================================
  Files         424      424              
  Lines       71452    71452              
==========================================
- Hits        55260    55217      -43     
- Misses      16192    16235      +43     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andrewhop]

Instead of just waiting can this check some SSM agent status and ensure it's up and running before starting?

[samuel40791765]

It's not that SSM isn't accessible after the ec2 instance is spun up, we've been using SSM right after in our past test runs. The weird thing is that SSM will suddenly end the session and is unavailable for a brief moment at times. This only happens when the instance has been recently launched. It's the main source of sporadic issues we've been running into (I left a comment on it under the thread of P113131493).
I'd rather sacrifice the pending 5 minutes for stability, rather than have the inconsistent failures happen again.