How to use Fluxcd with microservices where multiple gitrepos and multiple branches and their history come in place

[prathamesh-0909]

I am currently using gitlab and we had a decision of placing our k8s deployment files into the code itself under a folder named deployments. As we plan to connect gitlab agent with our k8s cluster. I wonder how to flux bootstrap flux for my 20+ microservices (each being a different project in gitlab under the same group).
Previously we used to deploy k8s deployment files from a single git repo for each env (dev, staging, QA) but now we have decided to place deployment files in the code itself.

I have no idea, how to bootstrap flux for all projects under a group in gitlab and how to bootstrap flux to multiple branches like dev, testing, and etc. As deployment files merging to dev branch in the code will affect the configurations of resources at dev env in my cluster whereas as changes to the deployment files at testing branch in the same repo will change the configurations of the resources in testing env and so on.

How should I flux bootstrap flux for multiple git repos and multiple git branches in each repo and how should I bootstrap flux whenever I need to introduce a new service in the project. Do I need to bootstrap every service again, or bootstrap the only new service will work?

Is there any way I can select all projects under a gitlab group or subgroup?
Also let me know if storing deployment files in the code repo is a good idea or bad idea?
Thank you.

[stefanprodan]

Flux bootstrap can be run only once per cluster, you would do this for a fleet repo which has a single branch e.g. main and from where you onboard all the apps by creating a GitRepo/Kustomization pair for each one. Depending on the cluster env, you will use Kustomize overlays in the fleet repo to change things like the branch name and any other env-specific app settings.

An example of this can be found here: https://github.com/fluxcd/flux2-multi-tenancy

And a more complex example here: https://github.com/controlplaneio-fluxcd/d1-fleet

[prathamesh-0909]

Thank you for your quick response. I would really love to hear from you about how should we maintain k8s deployment files for all env.
Placing the deployment files in the code itself is a bad idea? Because I need to create 20+ GitRepo/Kustomization pair for every service or creating the folder structure like given below. It will have all the deployment files in a single gitRepo local-cluster and I can kustomization object to build the deployment files and obtain reusability in the code.
I would like to know your thoughts on this.

[stefanprodan]

The way you structure the Flux configs should match your org culture, teams composition and SRE practices. If your dev teams are on-call and they have an active role in operating their apps on Kubernetes, then having the definitions next to the code makes sense. If the SRE team is decoupled from dev, to fix anything they will need to ask the devs to do it in their app repos, which may result in a lot of friction if your org has poor communication between teams. Also if incident management and SRE UX is a focus for your org, then you should have dedicated Flux Kustomization per app, along with alerts, dashboards with Flux metrics, etc. You should also consider using OCIRepo instead of GitRepos for the apps desired state storage, e.g. run flux push artifact from the apps CI workflows. This would limit the amount of SSH keys you need to manager and reduce the Git traffic.