✨Autoscaling: Start using EBS-based EC2 buffers 🚨

[sanderegg]

What do these changes do?

• implements Autoscaling: Use EBS-backed buffers #6045

In effect enable usage of EBS-based buffer machines (but still to be manually tested first of everything).
this was manually tested within AWS directly through /tests/manual and following README.md there.

Details:

• added test to run autoscaling against AWS through pytest --external-env flag
• buffer instances that are failing their cloud-init status test are deemed as broken and are auto-terminated
• autoscaling task now uses buffer machines if available to fulfill cluster needs
• logging improvements
• added a fixture to auto-skip a test if the external-envfile is not set
• reduce noise in autoscaling (werkzeug library)

Important Note:

This does not remove the old buffering system, but adds a new one that is currently not in use in any deployment.

Related issue/s

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 75.67568% with 18 lines in your changes missing coverage. Please review.

Project coverage is 88.1%. Comparing base (cafbf96) to head (906a23a).
Report is 460 commits behind head on master.

Files	Patch %	Lines
...e_service_autoscaling/modules/auto_scaling_core.py	62.5%	9 Missing ⚠️
...e_autoscaling/modules/buffer_machines_pool_core.py	76.6%	7 Missing ⚠️
...ackages/aws-library/src/aws_library/ssm/_client.py	83.3%	0 Missing and 1 partial ⚠️
...ice_autoscaling/utils/buffer_machines_pool_core.py	87.5%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #6143      +/-   ##
=========================================
+ Coverage    84.5%   88.1%    +3.6%     
=========================================
  Files          10    1466    +1456     
  Lines         214   60912   +60698     
  Branches       25    1505    +1480     
=========================================
+ Hits          181   53718   +53537     
- Misses         23    6879    +6856     
- Partials       10     315     +305     

Flag	Coverage Δ
integrationtests	64.9% <ø> (?)
unittests	86.1% <75.6%> (+1.5%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...ackages/aws-library/src/aws_library/ec2/_client.py	100.0% <100.0%> (ø)
...ckages/aws-library/src/aws_library/ssm/__init__.py	100.0% <ø> (ø)
...ackages/aws-library/src/aws_library/ssm/_errors.py	100.0% <100.0%> (ø)
...rc/simcore_service_autoscaling/core/application.py	100.0% <ø> (ø)
...oscaling/src/simcore_service_autoscaling/models.py	100.0% <100.0%> (ø)
...ackages/aws-library/src/aws_library/ssm/_client.py	97.6% <83.3%> (ø)
...ice_autoscaling/utils/buffer_machines_pool_core.py	93.7% <87.5%> (ø)
...e_autoscaling/modules/buffer_machines_pool_core.py	90.6% <76.6%> (ø)
...e_service_autoscaling/modules/auto_scaling_core.py	92.8% <62.5%> (ø)

... and 1411 files with indirect coverage changes

[matusdrobuliak66]

Thanks! This is the first time we start to use ssm ?

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud