AVRO-3423: Add build.sh step to encapsulate all the steps needed during release #1570

zcsizmadia · 2022-02-26T20:03:05Z

Make sure you have checked all steps below.

Jira

My PR addresses the following [Avro Jira](https://issues.apache.org/jira/browse/AVRO-3423
- https://issues.apache.org/jira/browse/AVRO-3423
- In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain Javadoc that explain what it does

zcsizmadia · 2022-02-26T20:07:58Z

@martin-g @RyanSkraba

This is a proposed way to simplify the release process for the C#. Step #9 in https://cwiki.apache.org/confluence/display/AVRO/How+To+Release#HowToRelease-Publishing could be reduced to the following if running from the ../lang/csharp folder:

NUGET_KEY="YOUR_KEY" ./build.sh release

This step requires that ./build.sh dist was executed earlier. That requirement can be eliminated by running dotnet pack --configuration Release Avro.sln as the first thing in release, however the dist step might be already done at the time when release is happening.

Other languages might follow this pattern so the release step would become a series of ./build.sh release from all language.

martin-g · 2022-02-26T20:27:45Z

lang/csharp/build.sh

@@ -76,6 +76,35 @@ do
      cp -pr build/doc/* ${ROOT}/build/avro-doc-${VERSION}/api/csharp
      ;;

+    release)
+      # "dist" step must be executed before this step


What do you think about the following:

extract all the logic in dist) to a function, so it will look like:

... dist) dist() release) release() ...

This way the first few lines of release() could check if there are artifacts created by dist() and call dist() if there are no such. Something like:

function release() { if [ ! -f .../some/file ]; then dist() fi # release stuff }

martin-g · 2022-02-26T20:28:18Z

lang/csharp/build.sh

+      # If not specified use default location
+      [ "$NUGET_SOURCE" ] || NUGET_SOURCE="https://api.nuget.org/v3/index.json"
+
+      # Set NUGET_KEY beofre executing script. E.g. `NUGET_KEY="YOUR_KEY" ./build.sh release`


typo in before

zcsizmadia · 2022-02-26T21:09:25Z

@martin-g I can check if the tar.gz exists and all the needed nupkgs exist, if not execute the dist() function as you described above. I have it implemented locally and I can commit it.

However, what about if there is no release action, but RELEASE=true NUGET_KEY="YOUR_KEY" ./build.sh dist needed to be executed:

  dist)
    ...
    if "$RELEASE"
    then
        ...
        # Bunch of nuget push ...
    fi
    ;;

martin-g · 2022-02-26T22:02:22Z

what about if there is no release action, but RELEASE=true NUGET_KEY="YOUR_KEY" ./build.sh dist

I like it!

zcsizmadia · 2022-02-26T23:39:17Z

This runs dist step and pushes the nupkgs to nuget.org:

RELEASE=1 NUGET_KEY="YOUR_KEY" ./build.sh dist

./build.sh dist is the same as it was before.

Is there any place where the NUGET_KEY is stored already on the "release" machine? SO I can default the value to it if NUGET_KEY is not set.

zcsizmadia · 2022-02-26T23:56:51Z

I still like the a new release step.This is another alternative to combine dist and release:

  dist|release)
     # dist steps
      ...
      if [ "$target" == "release" ]
      then
        # Push packages
        ...
      fi

zcsizmadia · 2022-02-27T16:13:27Z

Added a verify-release step to simplify and harden the verisifcation step after the release. It iterates through all the supported dotnet code SDKS to make sure that all the libs and tool can be installed from nuget for all the SDKs.

RyanSkraba · 2022-02-28T18:12:15Z

I really like one advantage of this -- I have tons of notes (in addition to the wiki) for releasing binary artifacts and doing the equivalent of verify-release. I've always intended to clean my notes up and contribute them back, and it would be really neat to do that in the build scripts themselves.

Unfortunately, I'm kind of skittish about delegating these steps to a script and generalizing them to all languages... I'd find it hard to "trust" doing a ./build.sh release from the root and have all of the binary artifacts deployed -- especially if there were failures midway or something unexpected happens (it always does).

Note that the Wiki process isn't immune to that either (check out the Python SDK release 1.9.2.1 as an example of fixing a bad deploy!)

I don't have any good ideas for that other than prompting (y/n/skip) at each command! Any ideas?

zcsizmadia · 2022-02-28T18:22:50Z

Adding a step function might be a good solution to help out skittish users ;)

function step
{
    ask if want to run [y/skip/abort]
    if no, return
    if abort, exit shole script
    execute $1
    if error code == 0 -> green message: step complete
    if error code != 0 -> red message: step failed (maybe question for retry/abort?)
}

   release)
       step "dotnet nuget push A.B.C.nupkg"
       step "dotnet nuget push A.B.C.D.nupkg"
       ...

Btw, couls you post how the release is happening now? Are you executing the steps from the wiki page in blocks, or each line one by one from those code snipptes for each step?

zcsizmadia · 2022-02-28T18:48:07Z

@RyanSkraba Added a ask function to the script and usage to the verify-release step as an example. It asks for each SDKs how to proceed. If there is any error anytime, the script stops.

Verify .NET SDK 3.1 ([y]es/[n]o/([a]bort)? y
...
Verify .NET SDK 5.0 ([y]es/[n]o/([a]bort)? y
...

RyanSkraba · 2022-03-08T11:08:10Z

(Please excuse the delay! I've been playing catch-up with emails)

Btw, could you post how the release is happening now? Are you executing the steps from the wiki page in blocks, or each line one by one from those code snipptes for each step?

Effectively, I've been doing commands from the wiki page -- the commands that can be reverted or backed out in blocks and the commands that (permanently) publish artifacts one at a time. Usually one SDK at a time. Even if it's a bit fussy, it's really only about an hours work!

I like the ask function and if we generalize this process (release and verify-release) to other language SDKs, we should probably also generalize that! If one day we get to the state where we're completely automating releases, this will be welcome. We probably don't need to ask to verify, right? It's fine though as it is!

Perhaps a future feature would be to not ask but to echo out the command instead of executing it.

zcsizmadia · 2022-03-08T12:24:28Z

I added soem better command line argument parsing and option fro dry run, answer yes to all questions .... It is better to have a more standard way of handling wehn the number of options start to inscrease.

It is possible to put most of these build.sh framework into /share/build-helper script, and pretty much all languages just source /share/build-helper and implenet their own commands with functions like function command_test {}. The buidl helper would execute bt something like this command_$1 for the supported commands.

…sizmadia/avro into avro-3424-add-build-sh-release-csharp

zcsizmadia · 2022-03-08T21:47:24Z

Added cd "$(dirname "$0")" to the top of the build.sh scripts. Previously it was inconsistent , some scripts had it, some did not. If the build.sh is executed from tha lang folder where the build.sh script is, it has no effect.
If the build.sh script is executed from a different folder, it will change into the proper folder and make sure the scripts will work. E.g. lang/csharp/build.sh dist will work.

zcsizmadia · 2022-03-08T21:51:59Z

Modded the C build.sh script as well. Only C++ is left. That script might need some rework, since it has some "non-standard" targets, like doc and 'format' ...

zcsizmadia · 2022-03-08T21:54:07Z

@martin-g Should be able to provide the CARGO_API_TOKEN from the command line as well? --cargo-api-token?
I was thinking to add support for language specific options to the build.sh script. This would perfectly fit there

martin-g · 2022-03-08T22:02:21Z

Env var should work too. I'm fine with both.

martin-g · 2022-05-12T10:58:02Z

lang/rust/build.sh

+{
+  execute cargo build --release --lib --all-features
+  execute cargo package
+  execute mkdir -p "$BUILD_ROOT/dist/rust"


Suggested change

execute mkdir -p "$BUILD_ROOT/dist/rust"

execute mkdir -p "$dist_dir"

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

martin-g · 2022-05-13T08:05:16Z

@jklamer Please review 69772c3
After some experimentation I concluded that cargo package cannot be used for Workspace project with Virtual manifests when a dependency (e.g. avro_derive) is not yet released.
The .crate (the source package) can be created only after cargo publishing the modules/manifests.

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

jklamer · 2022-05-15T17:27:56Z

@martin-g will review soon. Currently on vacation but will get to it ASAP

kojiromike · 2023-07-20T02:27:52Z

lang/py/build.sh


-clean() {
-  git clean -xdf '*.avpr' \
+function command_clean()


Please remove the function keyword from everywhere else instead of adding it here. There is very little practical difference, except that we should default to fewer characters when there is no reasonable argument for more.

kojiromike · 2023-09-19T14:22:59Z

lang/c/build.sh

+function prepare_build()
+{
+  command_clean
+  execute mkdir -p "$build_dir"


Where is execute defined?

share/build-helper.sh in the PR. The main reason behind the execute concept was to be able to have a --dry-run option, where the build.sh just prints out what would be running. The user can copy paste it and run it manually step by step if needed, or just simply checking the steps. Additionally there is a step option, IIRC I implemented it :) Wit enables to user to execute the build scripts step by step.

The build helper is pretty much the common code between all the build scripts and all build scripts include it. It has the execute implementation, the common help, the colored logging, etc.

kojiromike · 2023-09-19T14:24:17Z

lang/c/build.sh

+  execute make -C "$build_dir" package_source
+  execute rm "$root_dir"/docs/*.html
+  if [ ! -d "$dist_dir" ]; then
+    execute mkdir -p "$dist_dir"


If we're using mkdir -p we don't need to check if the dir already exists first.

Roger. I followed the original build.sh scripts as much to the letter as possible. At this stage my objective was to be able to easily compare roiginal build shells cripts with the new ones.

I understand but since you're adding execute doesn't that make it kind of moot?

Add build.sh release

10dfc37

github-actions bot added build C# labels Feb 26, 2022

zcsizmadia marked this pull request as draft February 26, 2022 20:03

Add quptes

041cea3

martin-g reviewed Feb 26, 2022

View reviewed changes

Call dist drom release if needed

9728e4e

zcsizmadia changed the title ~~AVRO-3424: Add build.sh step to encapsulate all the steps needed during release~~ AVRO-3423: Add build.sh step to encapsulate all the steps needed during release Feb 26, 2022

Use RELELEASE=1 approach

37969df

Zoltan Csizmadia added 2 commits February 26, 2022 17:59

Combine dist and release

d44f97c

Add verify-release

01b00b1

Add ask shell function

9cb6f7b

Zoltan Csizmadia added 3 commits February 28, 2022 12:51

Add disable ask

094e4bf

Add ask to release

24b9597

Add ask to release

ea34d82

Add dry run option to build.sh

58087e9

Zoltan Csizmadia added 3 commits March 8, 2022 06:27

Tweak interop-data-test

1b4bb56

Tweak test

4529ceb

Add build-helper.sh

ca37318

Zoltan Csizmadia added 2 commits March 8, 2022 21:37

Fix C build.sh

c9ca4bc

Merge branch 'avro-3424-add-build-sh-release-csharp' of github.com:zc…

30ed375

…sizmadia/avro into avro-3424-add-build-sh-release-csharp

github-actions bot added the C label Mar 8, 2022

Change dir into build.sh folder

3f09206

Zoltan Csizmadia added 8 commits March 8, 2022 16:07

Fix whitespaces

5f7621e

Lang specific options are set via env vars for now

b7c4e58

Remove BUILD_DESCRIPTION

c0cd424

Support extra commands

c9ced07

Measure build time

c9f26d5

Nothing to do

0e61c64

Fix shellcheck warnings

5fc2a2b

Use globbing instead of find to match files

3f127f6

martin-g mentioned this pull request Apr 18, 2022

AVRO-3479: [rust] Avro Schema Derive Proc Macro #1631

Merged

8 tasks

martin-g reviewed May 12, 2022

View reviewed changes

martin-g added 2 commits May 12, 2022 14:41

aa

7432a4c

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Merge branch 'master' into avro-3424-add-build-sh-release-csharp

4757ead

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

martin-g added 2 commits May 13, 2022 14:54

AVRO-3423: Update the dist and release steps for Rust module

69772c3

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

AVRO-3423: Fix Bash Shell syntax. Return true in the function

6ebfc85

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

kojiromike reviewed Jul 20, 2023

View reviewed changes

Zoltan Csizmadia added 2 commits September 18, 2023 16:15

Merge master

bdc773c

Add command_doc

388daa3

kojiromike reviewed Sep 19, 2023

View reviewed changes

zcsizmadia closed this by deleting the head repository Feb 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AVRO-3423: Add build.sh step to encapsulate all the steps needed during release #1570

AVRO-3423: Add build.sh step to encapsulate all the steps needed during release #1570

zcsizmadia commented Feb 26, 2022 •

edited

Loading

zcsizmadia commented Feb 26, 2022 •

edited

Loading

martin-g Feb 26, 2022

martin-g Feb 26, 2022

zcsizmadia commented Feb 26, 2022

martin-g commented Feb 26, 2022

zcsizmadia commented Feb 26, 2022

zcsizmadia commented Feb 26, 2022 •

edited

Loading

zcsizmadia commented Feb 27, 2022

RyanSkraba commented Feb 28, 2022

zcsizmadia commented Feb 28, 2022

zcsizmadia commented Feb 28, 2022

RyanSkraba commented Mar 8, 2022

zcsizmadia commented Mar 8, 2022

zcsizmadia commented Mar 8, 2022

zcsizmadia commented Mar 8, 2022

zcsizmadia commented Mar 8, 2022

martin-g commented Mar 8, 2022

martin-g May 12, 2022

martin-g commented May 13, 2022 •

edited

Loading

jklamer commented May 15, 2022

kojiromike Jul 20, 2023

kojiromike Sep 19, 2023

zcsizmadia Sep 19, 2023 •

edited

Loading

kojiromike Sep 19, 2023

zcsizmadia Sep 19, 2023

kojiromike Sep 20, 2023

	execute mkdir -p "$BUILD_ROOT/dist/rust"
	execute mkdir -p "$dist_dir"

AVRO-3423: Add build.sh step to encapsulate all the steps needed during release #1570

AVRO-3423: Add build.sh step to encapsulate all the steps needed during release #1570

Conversation

zcsizmadia commented Feb 26, 2022 • edited Loading

Jira

Tests

Commits

Documentation

zcsizmadia commented Feb 26, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zcsizmadia commented Feb 26, 2022

martin-g commented Feb 26, 2022

zcsizmadia commented Feb 26, 2022

zcsizmadia commented Feb 26, 2022 • edited Loading

zcsizmadia commented Feb 27, 2022

RyanSkraba commented Feb 28, 2022

zcsizmadia commented Feb 28, 2022

zcsizmadia commented Feb 28, 2022

RyanSkraba commented Mar 8, 2022

zcsizmadia commented Mar 8, 2022

zcsizmadia commented Mar 8, 2022

zcsizmadia commented Mar 8, 2022

zcsizmadia commented Mar 8, 2022

martin-g commented Mar 8, 2022

Choose a reason for hiding this comment

martin-g commented May 13, 2022 • edited Loading

jklamer commented May 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zcsizmadia Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zcsizmadia commented Feb 26, 2022 •

edited

Loading

zcsizmadia commented Feb 26, 2022 •

edited

Loading

zcsizmadia commented Feb 26, 2022 •

edited

Loading

martin-g commented May 13, 2022 •

edited

Loading

zcsizmadia Sep 19, 2023 •

edited

Loading