ci: Further configuration for FedoraProject

[LecrisUT]

So this is already a continuation of packit/packit#201. Seems that implementing packit would resolve the issue of keeping the spec file up to date. E.g. I have pushed a tag of v0.2.3-rc1 on my fork, and as you can see in the copr build it updates to that latest version: generated spec file for reference.

The main thing that packit does here:

• copr_build Submit builds to copr (using the latest version of the source)
• propose_downstream, create a PR on src.fedoraproject.org with the relevant changes (files defined in files_to_sync)
• koji_build and bodhi_update are only used downstream

There are still a few things to address:

• Move the builds to a scikit-build group. I've requested to have this group created
• Packit requires the installation of the packit app to the repo. Can check the branch for how it looks in action
• Move the build conditions to the main branch. But do we want it on every commit/PR or release?
• Implement propose_downstream

Depends on: packit/packit#247

[codecov]

Codecov Report

Merging #241 (f90d4ec) into main (3262d55) will decrease coverage by 0.89%.
The diff coverage is 72.00%.

@@            Coverage Diff             @@
##             main     #241      +/-   ##
==========================================
- Coverage   89.04%   88.15%   -0.89%     
==========================================
  Files          50       51       +1     
  Lines        2164     2238      +74     
==========================================
+ Hits         1927     1973      +46     
- Misses        237      265      +28     

Impacted Files	Coverage Δ
src/scikit_build_core/metadata/setuptools_scm.py	76.47% <33.33%> (-23.53%)	⬇️
src/scikit_build_core/settings/sources.py	82.67% <38.09%> (-3.14%)	⬇️
src/scikit_build_core/builder/sysconfig.py	80.26% <50.00%> (-0.82%)	⬇️
...rc/scikit_build_core/metadata/fancy_pypi_readme.py	80.95% <60.00%> (-19.05%)	⬇️
...cikit_build_core/settings/skbuild_read_settings.py	94.28% <66.66%> (-4.08%)	⬇️
src/scikit_build_core/settings/_load_provider.py	91.66% <91.66%> (ø)
src/scikit_build_core/settings/metadata.py	93.33% <94.11%> (-6.67%)	⬇️
src/scikit_build_core/build/wheel.py	80.16% <100.00%> (ø)
src/scikit_build_core/builder/get_requires.py	90.16% <100.00%> (+0.16%)	⬆️
src/scikit_build_core/settings/skbuild_model.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[LecrisUT]

Ok, I've tested the release trigger as well reference. The only thing I don't know if it takes care of is version number update, but we can check that at next release. (PS: rc release tag is a bit wonky right now on packit)

[hrnciar]

Personally, I don't have any experience with packit since I don't have any project that I am actively developing that I could use it for. But I have one small suggestion regarding the general workflow of shipping updates in Fedora.

The main thing that packit does here:

* Submit builds to copr (using the latest version of the source)

* propose_downstream, create a PR on `src.fedoraproject.org` with the relevant changes (files defined in `files_to_sync`)

In my team, we try to check the impact of new releases on the Fedora ecosystem. We want to avoid a situation when the update of package foo breaks package bar which is depending on it. Hence, for every new package update, we rebuild also all of its depending packages in copr. This way we can find potential breakages before the update is shipped and help with fixing them (or at least opening bugzillas for package maintainers so they are informed). I think, this is especially important for build systems like scikit-build/scikit-build-core, because in future a lot of packages could depend on it. Here is the document covering the process of how we do impact checks with all commands. I am willing to help you with this, feel free to add me as a co-maintainer and I'll try to review the PRs. This document describe how we do reviews.

* koji_build and bodhi_update are only used downstream

---

I've also enable the tracking of scikit-build-core in Anitya. This should create a bugzilla when new release is available.

[LecrisUT]

I understand the general flow of building all dependents, but I don't quite understand the workflow that we need to adjust here. I am considering initially 2 workflows to consider:

• copr_build that is run on every commit/PR. This is primarily for internal development to make sure fedora packaging will succeed
• propose_downstream + any dist_git jobs. At this point we propose releases to koji and bohdi so that further downstream can test their fedora packaging if it fails or not

I do not know if either bohdi or koji can automatically trigger rebuilds of dependents or if it automatically handled by rawhide. Maybe we can add a third copr workflow to a staging project triggered by release that other downstream copr projects can link to and trigger rebuilds when we push there, but we need a bit of documentation on how to instruct downstream to design their workflow for that. What do you think about this?

[hrnciar]

I wanted to point out that building the dependents should happen before koji build, that's the reason I put my text in between the bullets of the workflow. Once the package was built in koji it's available in rawhide repos and it can break stuff (on stable releases there is a waiting period, but not in rawhide). All the testing should happen before the PR is merged.

As far as I know, currently, there is no automated tooling for building dependents. We would love to have it, but currently, we are using a set of commands as described in the guide as a part of the PR review.

[LecrisUT]

Of course. propose_downstream only creates a PR, it is our responsibility to check others work. I don't know how to make the PR depend on an external CI, or at least if packit can create a custom message that we can link to other build statuses.

I think what we want is the 3rd workflow centered on copr. It seems the external dependency is possible, it is just the triggering of rebuild that needs to be done. I can add a job for this as soon as we have scikit-build group where we can create the relevant projects.

As far as I know, currently, there is no automated tooling for building dependents. We would love to have it, but currently, we are using a set of commands as described in the guide as a part of the PR review.

I will look into that later, but tldr, on this package side, we just need to push a release copr so others can pull from right? Also over on matrix there is the suggestion of koschei and there is talk about koschei on copr

Edit: I have enable python-scikit-build-core on koschei so we can at least detect some errors on rawhide

[LecrisUT]

@hrnciar Can I confirm about the guide you mentioned:

• Periodically repoquery to get all packages that require python3-scikit-build-core and save a list of it
• At each release MANUALLY submit copr builds for al dependent packages?

I think that would be a bit though to maintain. Maybe we can temporarily maintain it for rawhide, but isn't koschei already doing that for us? At least can we put it as a script, then maybe we can automate it here as a packit action

[hrnciar]

@hrnciar Can I confirm about the guide you mentioned:

* Periodically `repoquery` to get all packages that require `python3-scikit-build-core` and save a list of it

Yes.

* At each release MANUALLY submit copr builds for al dependent packages?

If you have a list of packages depending on python3-scikit-build-core you can easily rebuild them with these two commands:

# Using parallel from the moreutils-parallel package.
$ parallel copr add-package-distgit $COPR --webhook-rebuild on --commit rawhide --name -- $(cat packages.txt)
$ parallel copr build-package $COPR --background --nowait --name -- $(cat packages.txt

The tricky part comes later when you need to investigate the failures. You need to differentiate between the failures related to changes in the latest release and unrelated failures due to some other changes in the distro. What we usually do is that we rebuild all depending packages in the so-called "control" copr without the change that we are testing. When you compare these two coprs, you will have:
a) packages that succeeded in the main copr, and succeeded also in the control one - no breakage
b) packages that failed in the main copr, but succeeded in the control one - these packages need investigation because they are broken with the new release of the package
c) packages that succeeded in the main copr, and succeeded in the control one - ignore them, these packages were broken prior to your update

I think that would be a bit though to maintain. Maybe we can temporarily maintain it for rawhide, but isn't koschei already doing that for us? At least can we put it as a script, then maybe we can automate it here as a packit action

I understand it is complicated, but we want to avoid making people unhappy by breaking stuff. When you mentioned packit and automation I started to worry about the automatic shipping of updates without testing its impact. I am trying to provide the context of why it is necessary. Of course, you can decide to ship the update and react retroactively when users start to file bugs.

You are right that Koschei helps you detect the problem, but when the package is "red" in Koschei it's already after it was shipped to the users and we would like to avoid that.

As I said before, I am offering my help. Feel free to ping me and I'll do the testing for you. It won't be my top priority unless I won't need the update for some other package I maintain, but if you ping me, I'll try to make time and help with the impact check. It might sound complicated, but in reality, for easy updates, it takes a couple of minutes to run the scripts and then some time to check the results. When we discover a higher number of broken packages, we usually write a Change Proposal so maintainers are informed about what's happening.

[LecrisUT]

I understand your concern, but we also should not abuse the copr resources unnecessarily. IMO, we should be fine for rawhide at the very least. That version is meant for development.

Also note that I wrote the automation to update only rawhide and most recent release, so effect on older versions should be minimal. Maybe we can change it and only update latest release on request where we run your script. And when Fedora is forked, all packages are rebuilt right? So I don't think it is that drastic.

I would suggest to provide a copr project, e.g. @scikit-build/release, and instruct downstream to link to it and follow it, and we only run the script on those that don't have copr presence, although realistically they should all have. But could you write a script so that this can be automated to maybe a temporary project @scikit-build/downstream?

As for Change Proposal, that sounds like a good idea, but that would mean that we have to only release large changes so we don't pollute it with each package update? Maybe we are jumping a head too quickly on this since we have very few packages that depend on this. For spglib I am going to link it, and as soon as Koschei for copr is available or I can create a manual webhook, or there is any alternative I will implement it.

[hrnciar]

I understand your concern, but we also should not abuse the copr resources unnecessarily.

This is exactly what is copr intended for. We communicate with copr folks daily and they are happy when people use it. I am regularly submitting thousands of builds when rebuilding all Fedora packages with development versions of Python 3.12. Just use --background option when submitting builds to give them lower priority in the queue.

IMO, we should be fine for rawhide at the very least. That version is meant for development.

Also note that I wrote the automation to update only rawhide and most recent release, so effect on older versions should be minimal. Maybe we can change it and only update latest release on request where we run your script. And when Fedora is forked, all packages are rebuilt right? So I don't think it is that drastic.

Everything I've written above was written with rawhide in mind, I wasn't talking about stable releases.

I would suggest to provide a copr project, e.g. @scikit-build/release, and instruct downstream to link to it and follow it, and we only run the script on those that don't have copr presence, although realistically they should all have. But could you write a script so that this can be automated to maybe a temporary project @scikit-build/downstream?

Sorry, I don't understand this part. Downstream usually follows official releases on Pypi. That's why I enabled tracking.

I've also enable the tracking of scikit-build-core in Anitya. This should create a bugzilla when new release is available.

As for Change Proposal, that sounds like a good idea, but that would mean that we have to only release large changes so we don't pollute it with each package update? Maybe we are jumping a head too quickly on this since we have very few packages that depend on this. For spglib I am going to link it, and as soon as Koschei for copr is available or I can create a manual webhook, or there is any alternative I will implement it.

I wouldn't worry about a Change Proposal for now. I wanted to give you some perspective on what we do when there is a big impact on the distro. Upstream developers usually don't talk to us about their plans. So when the new version is released we evaluate the impact and then we decide if it's okay to just ship it or if it needs a more careful coordinated approach and then we create a Change Proposal so other maintainers are aware.

[LecrisUT]

Thanks for the pointers I'll keep them in mind when reviewing PRs.

Sorry, I don't understand this part. Downstream usually follows official releases on Pypi. That's why I enabled tracking.

So let's look at what I am implementing with this PR using packit:

• copr_build triggered on PRs and commits. This will submit builds to be CI tested to the copr @group/project: @scikit-build/PR and/or @scikit-build/latest. This is only for scikit-build purpose
• propose_downstream triggered on release. This will create a PR on src.fedoraproject.org to be reviewed. Whether or not we will merge it into rawhide depends on the release and the next stage
• copr_build triggered on release. This will submit builds to @scikit-build/release. Other copr projects are recommended to include that project as a dependency so they can run tests on their build
  • Ideally if there is Koschei or equivalent way to notify others, have them review the changes and work accordingly
  • Otherwise and what we can do for now, get all of the dependent packages using scikit-build-core and trigger a rebuild according to their dist-git in @scikit-build/downstream. But for this one, again, I would like your help on making a script to incorporate it here.

[LecrisUT]

Ok this PR is ready for review. Note that it depends on packit/packit#247. I should also point out that packit does not have PR approval yet, so maybe we should hold off until that is introduced? If the build does not require internet maybe it is not as critical though.

Also @henryiii @hrnciar should I add any of you or anyone else to the @scikit-build group for copr?

I will also add an equivalent workflow to scikit-build if all is ok over here.

[LecrisUT]

Played around with packit and it is quite powerful. Do we want to add some smoke tests, and if so what/how do we want to test there?

[henryiii]

I'm happy to add packit for the repo (done). Something just to make sure the recipe is mostly correct is fine.

[henryiii]

I've set up a Fedora account, I think that might allow it to work now.

[henryiii]

Another rebase should trigger a rebuild so we can see if the build is now allowed. I just merged #251.

[LecrisUT]

Something just to make sure the recipe is mostly correct is fine.

You mean like making sure the dependencies are correct and all that, or that the package tests pass? The latter should already be handled by %check, but the former, we have to write a shell script to run a test.

[henryiii]

Tests passing would be nice, but not required, depends on how much trouble it is & if it slows things down much. I'm more worried about dependencies and such being forgotten. Tests passing is (very nice) icing on the cake. :)

[LecrisUT]

Another rebase should trigger a rebuild so we can see if the build is now allowed. I just merged #251.

Thanks for setting it up. Can you comment /packit verify-fas <my-fas-username> in this issue before we rebase? We should be able to retrigger here with /packit build

[henryiii]

/packit build

[packit-as-a-service]

Based on your Packit configuration the settings of the @scikit-build/scikit-build-core Copr project would need to be updated as follows:

field	old value	new value
chroots	['fedora-rawhide-x86_64']	['fedora-38-x86_64', 'fedora-rawhide-x86_64']

Diff of chroots:

+fedora-38-x86_64

Packit was unable to update the settings above as it is missing admin permissions on the @scikit-build/scikit-build-core Copr project.

To fix this you can do one of the following:

• Grant Packit admin permissions on the @scikit-build/scikit-build-core Copr project on the permissions page.
• Change the above Copr project settings manually on the settings page to match the Packit configuration.
• Update the Packit configuration to match the Copr project settings.

Please retrigger the build, once the issue above is fixed.

[LecrisUT]

/packit build

[LecrisUT]

Ok, everything should be set up now. The tests have been passing last time I tried, and the dependencies should be picked up automatically from pyprojet.toml. The only thing is that we could add manual smoke tests to check if the rpm installation is sane and has all dependencies. If you have a suggestion of a simple shell script to do that I can add it here or in another PR

[henryiii]

If we do add a small test script, or run the package tests, that can be a followup.