[kit-fixups] RFC: Prepare new MARK releases

[geaaru]

To organize development and update the tree we will be going to create different releases:

• next: the existing release from Funtoo will be left as the base arrives for ex-Funtoo users and without major updates. We will maintain the current autogenned packages and Security fix

• mark-i: this will be the first MARK release based on next branch where removing all references to gentoo-staging stuff. We could consider updating this tree only from mark-testing after a testing period.

• mark-ii: The MARK II release will be based on only autogenned packages or curated packages. It will be our super-stable release. Could be possible to maintain this with only pinned packages that are updated only after a testing chain.

• mark-testing: The Mark Testing release could be used to push packages candidates for production and/or without a major impact if not explicit in the Planned project. We will start from next with the first mission to testing the operation that will be moved to mark-i after a testing period.

• mark-unstable: The MARK Unstable release is the world of the chaos. We can push PR to upgrade and break everything. But better in an organized mode if it's possible.

Open to comments and reflections

[geaaru]

The management of the releases could be done directly in the master branch of the kit-fixups using different release directories. This will simplify the merge of the autogenned specs from one release and and another and will be clearer to users without hopping between tons of repositories.
Maybe we could move gnome-kit-fixups, qt-kit-fixups, and the other external *-fixups in kit-fixups.
We are a little group and I don't think that we have time to work on different branches in every external *-fixups repos.

[linuxexplorer]

Personally, I would only have 3: next mark and mark-testing

[geaaru]

I think that next could be the first branch that will die because strictly related to Funtoo that will die too. It is safe to leave there now as a starting point but It's good to have MARK-related branches. The mark-unstable will replace the harvester-like branch without losing time on merge and drop the branch after N months.

[geaaru]

I'm going to create new releases and I'm trying to share notes about the way we want to use in this first step.

The kit-fixups supports multiple releases directly without branches but using the same tree has the following cons and pros:

1. to manage multiple releases at the same time you need to create multiple directories for every kit, for example:

core-kit/mark/<cat>/<pkg>
core-kit/mark-testing/<cat>/<pkg>

This could be fast to compare with diff and to migrate (with cp command) but could be more complex to propagate the name of the contributors.

2. Using the same branch means to have tons of the same autogen.yaml replicated for every release in the same branch but in different directories. This is pretty equal also with multiple branches but only divided into different branches.

3. If we use branches we can have only one release directory with the name mark:

core-kit/mark/<cat>/<pkg>

and eventually curated directory for the particular use case. This will permit easy use of cherry-pick or rebase and reusing the contributor commit as-is.

4. If we use branches we can use goreleaser as is already done in the Macaroni repositories to tag and supply the list of the changes done from the previous release. This will evidence the names of the contributors and their efforts.

5. If we use a single branch the tagging will be useless because the commits of the branch will be with the changes of all releases and more complex to read.

6. Using a single branch will be few usable the curated directories

So, considering that is important to me to give evidence of the efforts of our contributors because they are important for the life of this community I'm going to choose the multiple branches way for now.
We will modify this behavior eventually later if it is necessary.

[geaaru]

So in conclusion these are the rules to add in the README:

1. We will use the following branches:

• master (current): for next release and shortly in dismission
• mark-testing: for MARK testing release
• mark-unstable: for MARK unstable release
• mark-i: for MARK I stable release (starting from next) but with the target to remove all gentoo-staging stuff.
• mark-ii: for MARK II release with only autogen release and pinned version

2. The kit-fixups repository will use the mark release name in every branch. This will help on cherry-pick:

core-kit/mark/<cat>/<pkg>
dev-kit/mark/<cat>/pkg>
...

3. The commits of the mark-testing and mark-unstable branches could be merged on mark-i branch when things are stable.

4. Considering that on the mark-ii branch we will have only pinned versions probably we will create a specific commit (maybe from CD/CI task) after a complete testing pipeline or from specific and validated PRs. For this branch, it's difficult that will be possible to use cherry-pick.

[cuantar]

Responding to the RFC, perhaps late, but that's okay. Your plan seems sensible to me. However, I have one comment, regarding PRs to the stable tree.

I agree generally with the idea that PRs should go through testing before landing in stable, so we know that at least two things are true for every change:

• the tree regens, i.e. merge-kits completes without error; and
• metro can still build stages

IMO: We should strive never to break stage1-stage3 builds or tree regens for the stable tree. This policy might need to be revisited for stage4 builds like kde/gnome/xfce; but I'd start off with the "rule" that we ensure our basic infrastructure still works before pushing things to stable. To have this policy will mean that we keep testing fairly stable, too --- if the tree breaks from one change, we can't test other changes that are waiting to be regenned in metro builds...

I think it's okay to break unstable --- we can be less careful there --- but we should be attentive to it and ensure that when PRs are merged, we're aware of their outcome in the next ~24 hours. That will be a responsibility of contributors. We won't want the unstable tree broken for longer than it takes someone to fix a bug introduced in a change. However! The point of unstable (IMO) is to give people who don't have their own local infrastructure a place to test changes that might not work.

[cuantar]

So now, finally, is my point: We will need to merge updates to keep the mark-i tree stable, when upstream moves something or when a point release (like NetworkManager`) rudely introduces new dependencies that break builds.

How do you want to proceed? On the one hand, we can test these changes in testing first --- but meanwhile, mark-i is broken in some way for the interim period. On the other hand, if we allow PRs directly to mark-1 (what I called stable in my background note above), the changes might not get tested and could break things in different ways than the original Bug. On the gripping hand, maybe we can mitigate some of the trouble by using Github's tools, and allow PRs to mark-i directly only after a real code review with an approval from one of the other maintainers.

I'm leaning in favor of the third option, presently.

[geaaru]

Hey @cuantar, thanks for these comments. I like this way of sharing ideas and thoughts.

About ensuring that things will never be broken, this is pretty impossible. I have worked in the community for a lot of years, as I have shared in my last post on the Macaroni website there is an unhappy decrease in all contributions in a lot of opensource projects. In other cases, the Opensource project changes its license to a Business license with a community release that is useless and/or with few features, probably for the reduction of the contributors.

So, if you think that the few contributors possibly take care of the consequences of their PRs, done fortunately to fix their issues and give back to the community something but without having a more global vision it's easy that a correct fix will break others packages. It will happen for sure that a lot of contributions break things but it's the developer of the community that will fix this, a developer with a more global vision and more skill to fix correctly this.

A community normally is managed by people who give their free time to create something but it's not a job. It's very a cool thing if this could be a job. But it's not, we aren't a company. We can't ensure a fix in 24h or more, we can just do our best considering the two main rules for all Macaroni Developers: family and life have major priorities.

Obviously, the PRs must be validated and checked with attention but it is also true that I don't want that the few contributors will hate to help the community. Personally, I had bad experiences creating PRs for the Gentoo where different reviewers blocked my contributions for spaces or incongruent opinions based on the reviewer of the specific moment. I like to be positive about the incoming contributions (I hope that bad contributions like xz lib will never happen here) and not just create tons of rules that will push away people.

This is the reason because we need to have mark-testing and mark-unstable release. Based on the specific contributions we could address the contributor to push in the right place for him to update for a specific and correct testing phase.

Second, now you can see things also in a different way, we will create more relationships between MARK and Anise stack, which will permit users to fix broken trees using MARK-tagged tree from Anise.

About the production environment, I just discourage from compiling things in a production node, we have containers Docker, LXD, Singularity, etc. that could be prepared before, tested, tagged, and validated for a Production service. This will permit a reproducible way to configure the same service multiple times without having issues compiling things from a broken tree. Using binary packages built from MARK is also a plus because could be used to speed up things and compile with arch-specific flags only the packages that need to be optimized.
So, if at a specific time, things could be broken, we have the time to fix them without stress.

[geaaru]

About, the merge mode, we need to study the workflow. Personally, we can avoid autogen the mark-i tree and update it with a CD/CI using the version from the testing tree with specific rules for specific packages. I remember that on Macaroni I already update the packages with anise-portage-converter specs with rules in automatic. We can prepare things in an organized mode instead of just updating to the last available version and/or pin to the previous version when it's broken.
For specific packages, we will always need to create a manual PR or a more controlled one, for example for GCC, glibc, etc.

[geaaru]

But before going ahead what is needed is:

• stabilize the branch-based release mode
• try to drop all references to gentoo-staging tree with curated and/or autogen packages

After this, for big changes, like for example OpenSSL, Gnome, and QT6 we can split these big changes into Projects and Milestones, and proceed in a structured way without having to do too many things at the same time.

[geaaru]

Just thinking about another big issue. At the moment the MARK/Funtoo Desktop tree is lovely old. This could be a pain for upgrading packages. So, instead of using ego profiles to enable Gnome, KDE, etc. environments until we have moved things ahead for major Desktop packages could be better to have a mark-desktop branch where pin packages that break old Desktop packages.

[geaaru]

As described in the blog going to create the following branches:

Branch	Kits-Fixup Branch	Python Kit Profile	Target	Description
next	master	next	server,desktop	Funtoo landing branch before move to MARK. Slowly deprecated branch.
mark-testing	mark-testing	mark	server,desktop	The Mark Testing release could be used to push packages candidates for production and/or without a major impact if not explicit in the Planned project.
mark-i	mark-i	mark	server,container	The Mark-I release could be considered the stable release more similar to the Funtoo next release. We will create it in the near future after the stabilization of MARK tools. Our target is to remove all gentoo-staging stuff from this release.
mark-ii	mark-ii	mark	server,container	The Mark-II release will be based on only autogenned packages or curated packages. It will be our super-stable release. Could be possible to maintain this with only pinned packages that are updated only after a testing chain and/or normally PR tested by our CD/CI pipeline.
mark-v	mark-v	mark	server,container	The Mark-V release aslo knows as the Suitcase Suit wants to be specialized on improve Containers and Server rootfs.
mark-xl	mark-xl	mark	server,container	The Mark-XL release aslo knows as the Shotgun is the testing release of Mark-V.
mark-unstable	mark-unstable	mark	server,desktop	The MARK Unstable release is the world of the chaos. We can push PR to upgrade and break everything. But better in an organized mode if it's possible. :)
mark-funtoo	mark-funtoo	next	server	The MARK Funtoo branch is used to help Funtoo on upgrade/maintain server packages over exiting Funtoo Kits. Will be based on Macaroni next branch. This will not be autogen but only managed on Macaroni kit-fixups repository. The autogen is done just for testing ebuild generation.

This is our starting plan.
The way about how the branches will be synced will be described later and/or based on the evolution of our tools.

[geaaru]

The branch mark-v , mark-ii, and mark-i will be created when the merging tool will be available.
Closing this issue considering that the branches are been described in blog page too.