Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

maint: release process updates for friendlier ARNs #103

Merged
merged 16 commits into from
Oct 13, 2022

Conversation

robbkidd
Copy link
Member

@robbkidd robbkidd commented Oct 3, 2022

Which problem is this PR solving?

Release woes, enumerated below with the changes.

Short description of the changes

Reorganize releasing

Make is good at knowing how to make files. This moves the layer contents
zip creation from the publish script to the Makefile. There's some duplication,
but I think that keeps the Making more readable. CI updated to use the zip
make target to build the binaries and the zip files for artifact storage and
workflow pass-along.

Publish script updated to record the results of AWS API calls to files instead of
to script variables or the console. With the results recorded on the filesystem,
we can see success/failures after the fact and thereby build release.json from
the data in the API responses.

Use the CIRCLE_TAG as the version for a build

Include the build version in the user-agent (formatting updated to spec).

If the CI workflow doesn't have a tag, default to an informative version
derived from the current state of the git repository. For example, this
git describe command returns the following from the branch this commit
will land on:

» git describe --always --tags --match "v[0-9]*" HEAD
v10.3.0-9-g834dbcb

Which is:

  • the nearest tag in history that matches the pattern
  • the number of commits the branch's HEAD is away from that tag/commit
  • g + the sha of current HEAD

Test against only a single Go version

This is a project released and consumed as a binary. We only need to test
against the version of Go we build and release with.

@robbkidd robbkidd force-pushed the robb-n-jamie.mo-betta-version branch from 3ed63ea to 40c08d0 Compare October 3, 2022 23:17
@robbkidd robbkidd self-assigned this Oct 11, 2022
robbkidd and others added 5 commits October 11, 2022 14:18
We'll inject the semver release version for the build like we do all our
other Go binaries. Non-tagged non-release builds will have a version
of "dev".

Co-authored-by: Jamie Danielson <[email protected]>
I'm back on my Makefile shenanigans.
* updated make target for building to perform the cross-compile once
* rename the amd64 bits to x86_64 like AWS wants

* GOOS defaults to linux; can override in dev if you want a darwin build
If the CI workflow doesn't have a tag, default to an informative version
derived from the current state of the git repository. For example, this
git describe command returns the following from the branch this commit
will land on:

    » git describe --always --tags --match "v[0-9]*" HEAD
    v10.3.0-9-g834dbcb

Which is:
* the nearest tag in history that matches the pattern
* the number of commits the branch's HEAD is away from that tag/commit
* g + the sha of current HEAD
@robbkidd robbkidd force-pushed the robb-n-jamie.mo-betta-version branch from bea010f to 9d6554e Compare October 11, 2022 18:30
Make is good at knowing how to make files. This moves the layer contents
zip creation to make. There's some duplication, but I think that keeps
the Making more readable.

Publish script updated to check for zip existence instead of creating
them. A little layer_name declaration refactor in preparation for
version-specific layer names in the future.

CI updated to use the `zip` make target to build the binaries and the
zip files for artifact storage and pass-along.
Remove the old publish target that wouldn't work with the new locations
for build files.

Add a publish_sandbox that could be useful in dev sandboxes for testing.
Rub lots of guards on it, because we don't intend (yet?) to publish with
Make.
Determine version from a make target for consistency; one thing knows
how to figure out a version number and that's make. Sadly AWS doesn't
accept periods in layer names, so we have to tweak it further before
using it in the published name.
@robbkidd robbkidd force-pushed the robb-n-jamie.mo-betta-version branch from f8158c9 to d35b14d Compare October 12, 2022 21:53
With the results recorded on the filesystem, we can store these and see
success/failures and also build release.json from the data in the API
responses.

For example:

jq '{ region: (.LayerArn | split(":")[3]),
        arch: (.LayerArn | split(":")[6] | split("-")[3]),
        arn: .LayerVersionArn
    }' publishing/publish-*.json
The combination of all of the regions we've been publishing to matches
the list of regions that support both arm64 and x86_64 at the AWS Lambda
pricing page as of 2022/10/12. Moving existing regions to the WITH_ARCH
list.

NO_ARCH is currently empty, but one new-to-us region is online and
currently only supports x86_64. We can add it to NO_ARCH when we've
granted the publishing credentials permission to publish layers there.
# REGIONS_NO_ARCH=(me-central-1) # listed on pricing page, but we need grant our publishing credentials permission to publish there first
REGIONS_NO_ARCH=()
# Regions with x86_64 & arm64
REGIONS_WITH_ARCH=(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Showing my work on determining this new list.

  • Copied the <ul> lists of x86_64 and arm64 supporting regions from the AWS Lambda pricing page. (ಠ_ಠ)
  • Trimmed the content down from the HTML to only the region identifiers in those lists.
  • Diff'd the lists.
» diff --ignore-matching-lines --side-by-side regions-arm64.txt regions-x86_64.txt
16a17
> me-central-1
20a22,23
> us-gov-east-1
> us-gov-west-1
(fuller diff)
 » diff --side-by-side regions-arm64.txt regions-x86_64.txt
af-south-1							af-south-1
ap-east-1							ap-east-1
ap-northeast-1							ap-northeast-1
ap-northeast-2							ap-northeast-2
ap-northeast-3							ap-northeast-3
ap-south-1							ap-south-1
ap-southeast-1							ap-southeast-1
ap-southeast-2							ap-southeast-2
ap-southeast-3							ap-southeast-3
ca-central-1							ca-central-1
eu-central-1							eu-central-1
eu-north-1							eu-north-1
eu-south-1							eu-south-1
eu-west-1							eu-west-1
eu-west-2							eu-west-2
eu-west-3							eu-west-3
							      >	me-central-1
me-south-1							me-south-1
sa-east-1							sa-east-1
us-east-1							us-east-1
us-east-2							us-east-2
							      >	us-gov-east-1
							      >	us-gov-west-1
us-west-1							us-west-1
us-west-2							us-west-2

OK. Only Middle East (UAE) and GovCloud in the x86_64-only camp.

How does the new list of regions with arm64 support compare with our previous lists? They match.

irb> NEW_REGIONS_WITH_ARM64 == (REGIONS_NO_ARCH + REGIONS_WITH_ARCH)
=> true
Details: Set comparisons with Ruby
# Because Ruby is a tool near to my hand …
» irb
irb(main):001:1] REGIONS_NO_ARCH=Set.new(%w[eu-north-1 us-west-1 eu-west-3 ap-northeast-2 sa-east-1 ca-central-1 af-south-1
irb(main):002:1]                 ap-east-1 eu-south-1 me-south-1 ap-southeast-3 ap-northeast-3])
=>
#<Set:
...
irb(main):003:1] REGIONS_WITH_ARCH=Set.new(%w[ap-south-1 eu-west-2 us-east-1 eu-west-1 ap-northeast-1 ap-southeast-1
irb(main):004:1]                    ap-southeast-2 eu-central-1 us-east-2 us-west-2])
=> #<Set: {"ap-south-1", "eu-west-2", "us-east-1", "eu-west-1", "ap-northeast-1", "ap-southeast-1", "ap-southeast-2", "eu-central-1", "us-east-2", "us-west-2"}>
irb(main):005:1] NEW_REGIONS_WITH_ARM64=Set.new(%w[
irb(main):006:1]   af-south-1
irb(main):007:1]   ap-east-1
irb(main):008:1]   ap-northeast-1
irb(main):009:1]   ap-northeast-2
irb(main):010:1]   ap-northeast-3
irb(main):011:1]   ap-south-1
irb(main):012:1]   ap-southeast-1
irb(main):013:1]   ap-southeast-2
irb(main):014:1]   ap-southeast-3
irb(main):015:1]   ca-central-1
irb(main):016:1]   eu-central-1
irb(main):017:1]   eu-north-1
irb(main):018:1]   eu-south-1
irb(main):019:1]   eu-west-1
irb(main):020:1]   eu-west-2
irb(main):021:1]   eu-west-3
irb(main):022:1]   me-south-1
irb(main):023:1]   sa-east-1
irb(main):024:1]   us-east-1
irb(main):025:1]   us-east-2
irb(main):026:1]   us-west-1
irb(main):027:1]   us-west-2])
=>
#<Set:
...
irb(main):028:0> NEW_REGIONS_WITH_ARM64 == (REGIONS_NO_ARCH + REGIONS_WITH_ARCH)
=> true

@robbkidd robbkidd marked this pull request as ready for review October 13, 2022 13:57
@robbkidd robbkidd requested review from a team and emilyashley October 13, 2022 13:57
@robbkidd robbkidd added the type: maintenance The necessary chores to keep the dust off. label Oct 13, 2022
@robbkidd
Copy link
Member Author

The layer name changes in this PR (and all subsequent releases!). That's a change users will need to adjust for, but even so I don't think the release version number needs a major version bump. I think we can make the new situation known in release notes that the ARN for the layer version is more dramatically different than an increased layer version number. Which I suspect users would notice if they went to look up the latest layer version and see that it went from 11 to 1.

@robbkidd robbkidd changed the title maint: managing versions maint: release process updates for friendlier ARNs Oct 13, 2022
publish.sh Outdated
# Region list update from AWS Lambda pricing page as of 2022/10/12
#
# regions with x86_64 only support
# REGIONS_NO_ARCH=(me-central-1) # listed on pricing page, but we need grant our publishing credentials permission to publish there first
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need to do the role permissions grant to publish in Middle East (UAE) before adding this region to the x86_64 only list.

The whole version (which would include any "dev" present) is a part of
the layer name now, so we don't need to change the root name of the
layer.
Copy link
Contributor

@JamieDanielson JamieDanielson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ch-ch-changes

We went through the whole process with a dev tag, this seems to be in good shape!

@robbkidd robbkidd added the version: bump major A PR that breaks backwards compatibility. label Oct 13, 2022
@robbkidd
Copy link
Member Author

Opting to major version bump to draw attention to the new layer naming scheme.

@robbkidd robbkidd merged commit 0ddd948 into main Oct 13, 2022
@robbkidd robbkidd deleted the robb-n-jamie.mo-betta-version branch October 13, 2022 16:43
robbkidd added a commit that referenced this pull request Oct 18, 2022
This is a follow-up fix to the build/release overhaul in #103.

A similar change was made in multiple places, but the publish_s3 job was
missed:

* #103 for an OS-specific directory ZIP files will be created in[1]

[1] https://github.com/honeycombio/honeycomb-lambda-extension/pull/103/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R61-R90

* #108 for publishing assets to the GitHub Release from the
  Linux-specific directory[2]

[2] https://github.com/honeycombio/honeycomb-lambda-extension/pull/108/files#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47R76

With this PR's change, the built artifacts will appear in S3 directly
under the `honeycomb-lambda-extension/<VERSION>` subdirectory of the
honeycomb-builds bucket to match the contents for past releases as
opposed to in a `honeycomb-lambda-extension/<VERSION>/linux`
subdirectory. This gets things back into alignment with the download
URL given in docs for retrieving the layer contents for use in custom
Docker images[3].

[3] https://docs.honeycomb.io/getting-data-in/integrations/aws/aws-lambda/#installing-in-a-container-image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: maintenance The necessary chores to keep the dust off. version: bump major A PR that breaks backwards compatibility.
Projects
None yet
2 participants