docker: add distroless faq (#969)

Signed-off-by: Patrick Stephens <[email protected]>

Signed-off-by: Patrick Stephens <[email protected]>

installation/docker.md

@@ -51,7 +51,7 @@ It is strongly suggested that you always use the latest image of Fluent Bit.
 
 ## Multi Architecture Images
 
-Our production stable images are based on [Distroless](https://github.com/GoogleContainerTools/distroless) focusing on security containing just the Fluent Bit binary and minimal system libraries and basic configuration. 
+Our production stable images are based on [Distroless](https://github.com/GoogleContainerTools/distroless) focusing on security containing just the Fluent Bit binary and minimal system libraries and basic configuration.
 We also provide **debug** images for all architectures (from 1.9.0+) which contain a full (Debian) shell and package manager that can be used to troubleshoot or for testing purposes.
 
 From a deployment perspective, there is no need to specify an architecture, the container client tool that pulls the image gets the proper layer for the running architecture.
@@ -88,3 +88,40 @@ Alpine Linux uses Musl C library instead of Glibc. Musl is not fully compatible
 * Alpine Linux Musl functions bootstrap have a compatibility issue when loading Golang shared libraries, this generate problems when trying to load Golang output plugins in Fluent Bit.
 * Alpine Linux Musl Time format parser does not support Glibc extensions
 * Maintainers preference in terms of base image due to security and maintenance reasons are Distroless and Debian.
+
+### Why use distroless containers ?
+
+Briefly tackled in a [blog post](https://calyptia.com/2022/01/26/all-your-arch-are-belong-to-us/#security) which links out to the following possibly opposing views:
+
+* <https://hackernoon.com/distroless-containers-hype-or-true-value-2rfl3wat>
+* <https://www.redhat.com/en/blog/why-distroless-containers-arent-security-solution-you-think-they-are>
+
+The reasons for using Distroless are fairly well covered here: <https://github.com/GoogleContainerTools/distroless#why-should-i-use-distroless-images>
+
+* Only include what you need, reduce the attack surface available.
+* Reduces size so improves perfomance as well.
+* Reduces false positives on scans (and reduces resources required for scanning).
+* Reduces supply chain security requirements to just what you need.
+* Helps prevent unauthorised processes or users interacting with the container.
+* Less need to harden the container (and container runtime, K8S, etc.).
+* Faster CICD processes.
+
+With any choice of course there are downsides:
+
+* No shell or package manager to update/add things.
+  * Generally though dynamic updating is a bad idea in containers as the time it is done affects the outcome: two containers started at different times using the same base image may perform differently or get different dependencies, etc.
+  * A better approach is to rebuild a new image version but then you can do this with Distroless, however it is harder requiring multistage builds or similar to provide the new dependencies.
+* Debugging can be harder.
+  * More specifically you need applications set up to properly expose information for debugging rather than rely on traditional debug approaches of connecting to processes or dumping memory. This can be an upfront cost vs a runtime cost but does shift left in the development process so hopefully is a reduction overall.
+* Assumption that Distroless is secure: nothing is secure (just more or less secure) and there are still exploits so it does not remove the need for securing your system.
+* Sometimes you need to use a common base image, e.g. with audit/security/health/etc. hooks integrated, or common base tooling (this could still be Distroless though).
+
+One other important thing to note is that `exec`'ing into a container will potentially impact resource limits.
+
+For debugging, debug containers are available now in K8S: <https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container>
+
+* This can be a quite different container from the one you want to investigate (e.g. lots of extra tools or even a different base).
+* No resource limits applied to this container - can be good or bad.
+* Runs in pod namespaces, just another container that can access everything the others can.
+* May need architecture of the pod to share volumes, etc.
+* Requires more recent versions of K8S and the container runtime plus RBAC allowing it.