-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
maint: release process updates for friendlier ARNs #103
Conversation
3ed63ea
to
40c08d0
Compare
We'll inject the semver release version for the build like we do all our other Go binaries. Non-tagged non-release builds will have a version of "dev". Co-authored-by: Jamie Danielson <[email protected]>
We're a binary project!
I'm back on my Makefile shenanigans.
* updated make target for building to perform the cross-compile once * rename the amd64 bits to x86_64 like AWS wants * GOOS defaults to linux; can override in dev if you want a darwin build
If the CI workflow doesn't have a tag, default to an informative version derived from the current state of the git repository. For example, this git describe command returns the following from the branch this commit will land on: » git describe --always --tags --match "v[0-9]*" HEAD v10.3.0-9-g834dbcb Which is: * the nearest tag in history that matches the pattern * the number of commits the branch's HEAD is away from that tag/commit * g + the sha of current HEAD
bea010f
to
9d6554e
Compare
Make is good at knowing how to make files. This moves the layer contents zip creation to make. There's some duplication, but I think that keeps the Making more readable. Publish script updated to check for zip existence instead of creating them. A little layer_name declaration refactor in preparation for version-specific layer names in the future. CI updated to use the `zip` make target to build the binaries and the zip files for artifact storage and pass-along.
Remove the old publish target that wouldn't work with the new locations for build files. Add a publish_sandbox that could be useful in dev sandboxes for testing. Rub lots of guards on it, because we don't intend (yet?) to publish with Make.
Determine version from a make target for consistency; one thing knows how to figure out a version number and that's make. Sadly AWS doesn't accept periods in layer names, so we have to tweak it further before using it in the published name.
f8158c9
to
d35b14d
Compare
With the results recorded on the filesystem, we can store these and see success/failures and also build release.json from the data in the API responses. For example: jq '{ region: (.LayerArn | split(":")[3]), arch: (.LayerArn | split(":")[6] | split("-")[3]), arn: .LayerVersionArn }' publishing/publish-*.json
The combination of all of the regions we've been publishing to matches the list of regions that support both arm64 and x86_64 at the AWS Lambda pricing page as of 2022/10/12. Moving existing regions to the WITH_ARCH list. NO_ARCH is currently empty, but one new-to-us region is online and currently only supports x86_64. We can add it to NO_ARCH when we've granted the publishing credentials permission to publish layers there.
# REGIONS_NO_ARCH=(me-central-1) # listed on pricing page, but we need grant our publishing credentials permission to publish there first | ||
REGIONS_NO_ARCH=() | ||
# Regions with x86_64 & arm64 | ||
REGIONS_WITH_ARCH=( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Showing my work on determining this new list.
- Copied the
<ul>
lists of x86_64 and arm64 supporting regions from the AWS Lambda pricing page. (ಠ_ಠ) - Trimmed the content down from the HTML to only the region identifiers in those lists.
- Diff'd the lists.
» diff --ignore-matching-lines --side-by-side regions-arm64.txt regions-x86_64.txt
16a17
> me-central-1
20a22,23
> us-gov-east-1
> us-gov-west-1
(fuller diff)
» diff --side-by-side regions-arm64.txt regions-x86_64.txt
af-south-1 af-south-1
ap-east-1 ap-east-1
ap-northeast-1 ap-northeast-1
ap-northeast-2 ap-northeast-2
ap-northeast-3 ap-northeast-3
ap-south-1 ap-south-1
ap-southeast-1 ap-southeast-1
ap-southeast-2 ap-southeast-2
ap-southeast-3 ap-southeast-3
ca-central-1 ca-central-1
eu-central-1 eu-central-1
eu-north-1 eu-north-1
eu-south-1 eu-south-1
eu-west-1 eu-west-1
eu-west-2 eu-west-2
eu-west-3 eu-west-3
> me-central-1
me-south-1 me-south-1
sa-east-1 sa-east-1
us-east-1 us-east-1
us-east-2 us-east-2
> us-gov-east-1
> us-gov-west-1
us-west-1 us-west-1
us-west-2 us-west-2
OK. Only Middle East (UAE) and GovCloud in the x86_64-only camp.
How does the new list of regions with arm64 support compare with our previous lists? They match.
irb> NEW_REGIONS_WITH_ARM64 == (REGIONS_NO_ARCH + REGIONS_WITH_ARCH)
=> true
Details: Set comparisons with Ruby
# Because Ruby is a tool near to my hand …
» irb
irb(main):001:1] REGIONS_NO_ARCH=Set.new(%w[eu-north-1 us-west-1 eu-west-3 ap-northeast-2 sa-east-1 ca-central-1 af-south-1
irb(main):002:1] ap-east-1 eu-south-1 me-south-1 ap-southeast-3 ap-northeast-3])
=>
#<Set:
...
irb(main):003:1] REGIONS_WITH_ARCH=Set.new(%w[ap-south-1 eu-west-2 us-east-1 eu-west-1 ap-northeast-1 ap-southeast-1
irb(main):004:1] ap-southeast-2 eu-central-1 us-east-2 us-west-2])
=> #<Set: {"ap-south-1", "eu-west-2", "us-east-1", "eu-west-1", "ap-northeast-1", "ap-southeast-1", "ap-southeast-2", "eu-central-1", "us-east-2", "us-west-2"}>
irb(main):005:1] NEW_REGIONS_WITH_ARM64=Set.new(%w[
irb(main):006:1] af-south-1
irb(main):007:1] ap-east-1
irb(main):008:1] ap-northeast-1
irb(main):009:1] ap-northeast-2
irb(main):010:1] ap-northeast-3
irb(main):011:1] ap-south-1
irb(main):012:1] ap-southeast-1
irb(main):013:1] ap-southeast-2
irb(main):014:1] ap-southeast-3
irb(main):015:1] ca-central-1
irb(main):016:1] eu-central-1
irb(main):017:1] eu-north-1
irb(main):018:1] eu-south-1
irb(main):019:1] eu-west-1
irb(main):020:1] eu-west-2
irb(main):021:1] eu-west-3
irb(main):022:1] me-south-1
irb(main):023:1] sa-east-1
irb(main):024:1] us-east-1
irb(main):025:1] us-east-2
irb(main):026:1] us-west-1
irb(main):027:1] us-west-2])
=>
#<Set:
...
irb(main):028:0> NEW_REGIONS_WITH_ARM64 == (REGIONS_NO_ARCH + REGIONS_WITH_ARCH)
=> true
The layer name changes in this PR (and all subsequent releases!). That's a change users will need to adjust for, but even so I don't think the release version number needs a major version bump. I think we can make the new situation known in release notes that the ARN for the layer version is more dramatically different than an increased layer version number. Which I suspect users would notice if they went to look up the latest layer version and see that it went from |
publish.sh
Outdated
# Region list update from AWS Lambda pricing page as of 2022/10/12 | ||
# | ||
# regions with x86_64 only support | ||
# REGIONS_NO_ARCH=(me-central-1) # listed on pricing page, but we need grant our publishing credentials permission to publish there first |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll need to do the role permissions grant to publish in Middle East (UAE) before adding this region to the x86_64 only list.
The whole version (which would include any "dev" present) is a part of the layer name now, so we don't need to change the root name of the layer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Opting to major version bump to draw attention to the new layer naming scheme. |
This is a follow-up fix to the build/release overhaul in #103. A similar change was made in multiple places, but the publish_s3 job was missed: * #103 for an OS-specific directory ZIP files will be created in[1] [1] https://github.com/honeycombio/honeycomb-lambda-extension/pull/103/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R61-R90 * #108 for publishing assets to the GitHub Release from the Linux-specific directory[2] [2] https://github.com/honeycombio/honeycomb-lambda-extension/pull/108/files#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47R76 With this PR's change, the built artifacts will appear in S3 directly under the `honeycomb-lambda-extension/<VERSION>` subdirectory of the honeycomb-builds bucket to match the contents for past releases as opposed to in a `honeycomb-lambda-extension/<VERSION>/linux` subdirectory. This gets things back into alignment with the download URL given in docs for retrieving the layer contents for use in custom Docker images[3]. [3] https://docs.honeycomb.io/getting-data-in/integrations/aws/aws-lambda/#installing-in-a-container-image
Which problem is this PR solving?
Release woes, enumerated below with the changes.
Shortdescription of the changesReorganize releasing
amd64
tox86_64
#82honeycomb-lambda-extension-<arch>-<version tag>
so that our layer versions will always be:1
Make is good at knowing how to make files. This moves the layer contents
zip creation from the publish script to the Makefile. There's some duplication,
but I think that keeps the Making more readable. CI updated to use the
zip
make target to build the binaries and the zip files for artifact storage and
workflow pass-along.
Publish script updated to record the results of AWS API calls to files instead of
to script variables or the console. With the results recorded on the filesystem,
we can see success/failures after the fact and thereby build release.json from
the data in the API responses.
Use the CIRCLE_TAG as the version for a build
Include the build version in the user-agent (formatting updated to spec).
If the CI workflow doesn't have a tag, default to an informative version
derived from the current state of the git repository. For example, this
git describe command returns the following from the branch this commit
will land on:
Which is:
Test against only a single Go version
This is a project released and consumed as a binary. We only need to test
against the version of Go we build and release with.