TODO: Create architectural decision record issue to summarize why we want to use latest images:

[ckadner]

TODO: Create architectural decision record issue to summarize why we want to use latest images:

Cons of using latest:

• MM image builds might fail unexpectedly
• hard to debug/trace back when failing changes happened
• last working image may not be possible to recreate for a new MM release without any changes
• last working image might not be available even if identified
• even if builds succeed, there might be unexpected changes in behavior which happens quite often with 3rd party dependencies of Python packages (on a tangent: when we update to Go 1.19 a while ago, the go get CLI tools install commands "succeed" but no longer installed executables, though we would not use ubi8/go-toolset:latest)

Pros of using latest:

• keeping specific version tag, we keep outdated code, old vulnerabilities
• we don't have to keep updating to newer patched versions of base images when upstream base images get updated frequently

Builder vs final images:

• using latest on a final build stage (not builder image) should be fairly "safe", no 3rd party libraries are installed, just files copied from builder image.
• however, it's the base of the builder images where most problems/vulnerabilities are coming in/need to be fixed, are getting fixed upstream,
• so really, all build stages need to build on a latest upstream (UBI) base image

Finally, we should follow the same strategy for all MM images consistently.

Originally posted by @ckadner in kserve/rest-proxy#43 (comment)

[ckadner]

FYI @spolti @Jooho @njhill @tjohnson31415 @rafvasq

[ckadner]

We could add a nightly build to keep updated about the build status -- to not be surprised by failing builds when we start working on a release.

• ci: Add nightly build twice a week #513
• ci: Add nightly build twice a week modelmesh#141
• ci: Add nightly builds twice a week modelmesh-runtime-adapter#88
• ci: Add nightly builds twice a week rest-proxy#44

Additionally, failing builds could trigger notifications to be send to the "modelmesh-build" team. Team to be created.

Have a build matrix even to build several versions of UBI 8/9

[ckadner]

@spolti with respect to FIPS we stay with UBI 8 for the time being. All the remaining MM images use UBI 8 latest for final build stage. Thanks!

• chore: Use latest ubi image tag #506
• chore: Use latest ubi image tag modelmesh#140
• chore: Use latest image tag modelmesh-runtime-adapter#86
• chore: Use latest image tag rest-proxy#42
• kserve/modelmesh-minio-examples

[spolti]

About the FIPS, there is no public document yet, but the work still on going.