Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for pro archives #53

Closed
wants to merge 8 commits into from
Closed

Add support for pro archives #53

wants to merge 8 commits into from

Conversation

woky
Copy link
Contributor

@woky woky commented May 2, 2023

This PR adds support for FIPS repositories along with preparatory changes:

  • Bump Go to 1.20 and go mod tidy. Some new code uses errors.Join.
  • No longer limit the archives to a single archive named "ubuntu".
  • Do not tie packages to individual archives. Instead, select the most recent version from all archives.
  • Add support for archive priorities. If a package to be downloaded exists in both archive A and archive B and archive A has higher priority than archive B, the package is downloaded from archive A even if archive B has a newer version. If two or more archives have the same priority, the most recent version of the package in any of these archive is selected, if any. Otherwise, archives with lower priority are tried.
  • Support parsing of archive credentials from configuration files in /etc/apt/auth.conf.d/. This is not enabled by default, see below.
  • Support Pro archives by pro property in chisel.yaml. When a pro property is specified, it has to be either "fips" or "fips-updates". The repository URL is inferred from the value. Also, credentials for these URLs are searched (see above). If credentials are not found, the corresponding pro archive is silently disabled. Otherwise, all requests to the archive are sent with the credentials found.

Currently, fips and fips-updates exists only for focal for which we don't have chisel-releases yet. I've created a fork with reduced set of slices. You can try this implementation with it provided you have pro fips-updates credentials available on your machine.

% cd /path/to/chisel/sources
% git clone -b ubuntu-20.04 https://github.com/woky/chisel-releases
% cat auth.conf.d/90ubuntu-advantage
machine esm.ubuntu.com/fips-updates/ login bearer password [...REDACTED...]  # ubuntu-advantage-tools
% export CHISEL_AUTH_DIR=$PWD/auth.conf.d
% mkdir -p out
% ./chisel cut --release ./chisel-releases --root out openssl_bins
2023/05/05 09:47:56 Processing ../chisel-releases release...
2023/05/05 09:47:56 Selecting slices...
2023/05/05 09:47:56 Fetching ubuntu 20.04 focal suite details...
2023/05/05 09:47:57 Release date: Thu, 23 Apr 2020 17:33:17 UTC
2023/05/05 09:47:57 Fetching index for ubuntu 20.04 focal main component...
2023/05/05 09:47:57 Fetching index for ubuntu 20.04 focal universe component...
2023/05/05 09:47:57 Fetching ubuntu 20.04 focal-security suite details...
2023/05/05 09:47:58 Release date: Fri, 05 May 2023  6:19:24 UTC
2023/05/05 09:47:58 Fetching index for ubuntu 20.04 focal-security main component...
2023/05/05 09:47:58 Fetching index for ubuntu 20.04 focal-security universe component...
2023/05/05 09:47:58 Fetching ubuntu 20.04 focal-updates suite details...
2023/05/05 09:47:58 Release date: Fri, 05 May 2023  7:29:11 UTC
2023/05/05 09:47:58 Fetching index for ubuntu 20.04 focal-updates main component...
2023/05/05 09:47:58 Fetching index for ubuntu 20.04 focal-updates universe component...
2023/05/05 09:47:59 Fetching ubuntu-fips-updates 20.04 focal-updates suite details...
2023/05/05 09:47:59 Release date: Fri, 28 Apr 2023 11:20:11 UTC
2023/05/05 09:47:59 Fetching index for ubuntu-fips-updates 20.04 focal-updates main component...
2023/05/05 09:47:59 Fetching pool/main/g/glibc/libc6_2.31-0ubuntu9.9_amd64.deb...
2023/05/05 09:47:59 Fetching pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.fips.18_amd64.deb...
2023/05/05 09:48:00 Fetching pool/main/o/openssl/openssl_1.1.1f-1ubuntu2.fips.18_amd64.deb...
2023/05/05 09:48:00 Extracting files from package "libc6"...
2023/05/05 09:48:03 Extracting files from package "libssl1.1"...
2023/05/05 09:48:05 Extracting files from package "openssl"...

woky added 3 commits May 2, 2023 16:47
Do not tie packages to individual archives and pick the package with
highest version from all defined archives.
@woky woky force-pushed the fips2 branch 2 times, most recently from d6cbd4a to 2b18c18 Compare May 5, 2023 05:50
@woky woky force-pushed the fips2 branch 2 times, most recently from 08ce9bd to b5b3396 Compare May 5, 2023 07:45
@woky woky added the Priority Look at me first label May 5, 2023
Copy link
Collaborator

@cjdcordeiro cjdcordeiro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for this. I've left some nitpicks and small remarks, but in general it lgtm

internal/archive/archive.go Show resolved Hide resolved
internal/archive/archive.go Show resolved Hide resolved
internal/archive/archive.go Show resolved Hide resolved
internal/archive/credentials_test.go Show resolved Hide resolved
internal/archive/credentials_test.go Show resolved Hide resolved
internal/setup/setup.go Show resolved Hide resolved
internal/setup/setup.go Show resolved Hide resolved
internal/setup/setup_test.go Show resolved Hide resolved
internal/slicer/slicer.go Show resolved Hide resolved
Copy link
Contributor

@niemeyer niemeyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per early conversations around collaboration, this PR needs to be broken down into finer grained changes so we can review and merge them more timely. The description itself already makes it clear that this is several independent changes bundled into one, which is a pattern that creates very long PRs, very long reviews, and frustration because it takes a many roundtrips of independent reviews until we can merge any one of the changes. On the other hands, with smaller and to-the-point PRs, we can discuss the issue at hand, adapt, and merge.

@woky
Copy link
Contributor Author

woky commented May 10, 2023

I'm gonna finish review with Cris in this PR and then I'll split it.

Copy link
Member

@rebornplusplus rebornplusplus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I looked through the changes and have a few suggestions.

I too think this should be split into a few more PRs, specially the Go version upgrade one. Additionally, I think it can be further split into the non-Pro archive updates such as having multiple archives, priority, not tying packages to archive, and the new Pro archive features. Of course, the PR/commit with the Pro feature will be depending on the non-Pro updates PR/commit.

internal/archive/credentials.go Outdated Show resolved Hide resolved
internal/archive/credentials.go Show resolved Hide resolved
internal/archive/credentials.go Show resolved Hide resolved
internal/archive/credentials.go Show resolved Hide resolved
internal/archive/credentials.go Show resolved Hide resolved
Comment on lines -19 to +20
Exists(pkg string) bool
Version(pkg string) string
Copy link
Member

@rebornplusplus rebornplusplus May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should leave Exists as it is, and just add another function called Version.
They might just do the same thing as it seems, but it feels a bit weird to check if package is there by checking it's version, which I realize now is what's being done inside selectPackage itself, but still Exists seems intuitive.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the code where Exists was used doesn't exist anymore. And if I did Exists followed by Version it'd be two lookups instead of one.

Comment on lines -80 to 86
func (a *ubuntuArchive) Exists(pkg string) bool {
_, _, err := a.selectPackage(pkg)
return err == nil
func (a *ubuntuArchive) Version(pkg string) string {
_, _, version := a.selectPackage(pkg)
return version
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above. Let's keep both I think.

Copy link
Contributor Author

@woky woky May 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment above. The Exists would be unused.

Comment on lines -85 to +88
func (a *ubuntuArchive) selectPackage(pkg string) (control.Section, *ubuntuIndex, error) {
func (a *ubuntuArchive) selectPackage(pkg string) (control.Section, *ubuntuIndex, string) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, I think we can keep this signature same and return an error like before as the third return value. We can deduce the Version from control.Section that will be returned as the first return value, e.g.

section, _, err := a.selectPackage(pkg)
if err != nil {
    ...
}
version := section.Get("Version")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And section.Get parses the section text every time it's called. Se when we get the version inside selectPackage, why parse the section block again right after that?

I don't understand the intention. This is internal function used only few lines above and below.

cmd/chisel/cmd_cut.go Show resolved Hide resolved
internal/slicer/slicer.go Show resolved Hide resolved
@woky woky removed the Priority Look at me first label May 12, 2023
@woky woky marked this pull request as draft May 15, 2023 19:59
@cjdcordeiro cjdcordeiro mentioned this pull request May 22, 2023
1 task
@cjdcordeiro cjdcordeiro mentioned this pull request Jun 1, 2023
1 task
@niemeyer
Copy link
Contributor

niemeyer commented Jun 2, 2023

My understanding is that this PR is being broken down into individual PRs at the moment, so I'll go ahead and close it so we don't mix it up.

@niemeyer niemeyer closed this Jun 2, 2023
@woky woky mentioned this pull request Jun 28, 2023
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants