Feedback on the O3DE Release 22.10.0 (Feedback due by 10/31)

[tonybalandiuk]

This discussion will serve to collect feedback on the O3DE 22.10.0 release.
To help make your comment clear, please start out your comment with one of the following phrases:
"Something that went well..."
"Something that didn't go well..."
"Something that we should change..."

[tonybalandiuk]

From @Ulrick28 Something that we should change: "Let's re-examine the build rules. I think it's a good idea for the next stabilization branch we do a nightly build, regardless of changes. Noone is going to care that the installer is still the same from night to night as all they see to download is the most recent installer. This nightly build would catch any dependency changes that don't trigger a build. "

[tonybalandiuk]

The risk here is we get to the end of stabilization, and there's a 3P change that does not trigger a build. The process to update a 3P library, though, is to update the config file. There could be things we are not thinking about that could break a build. So this proposal is really a safety net. @amzn-changml is there a large financial impact to doing 1 extra build a day during the stabilization period?

[tonybalandiuk]

If the cost is low @amzn-changml then sig-release is in support of adding a nightly build for the stabilization branch.

[amzn-changml]

Yes, the cost is minimal if we do it nightly.

Note that there are changes to the O3DE repo whenever a 3P package change is made, as the package reference to a new version is updated in this path: https://github.com/o3de/o3de/tree/development/cmake/3rdParty/Platform.

This will trigger a nightly build, provided the change was made in the development or stabilization branch.

[vincent6767]

Thanks @amzn-changml ! This is solid action items for the next release @tonybalandiuk

[tonybalandiuk]

[Done] We have added this to the task here o3de/o3de#13208 that will be cloned for the next release.

[lmbr-pip]

Something that we should change is that stabilization branch also needs AR metrics and some form of automated alarm when success rate % falls below x. AR runs failed a lot on stabilization but it was unclear what the expectation should be and who should be keeping an eye on the stability of the build.

[amzn-changml]

We'll need the involvement of SIG-Build and SIG-Testing for the task of producing the tests. We do need a owning group to establish performance and stability thresholds, though some of this work may be repurposed from the existing testing framework if some thresholds already exist. This may be cross cutting with the TSC as well

[vincent6767]

We'll need the involvement of SIG-Build and SIG-Testing for the task of producing the tests.

Anything that we (SIG Release) can help to make happen so we can have it before the next release? @amzn-changml

[tonybalandiuk]

@vincent6767 please create a sig-build issue for "We do need additional metrics around the testing of the stabilization installer" and link it from here.

[vincent6767]

@amzn-changml I have created a GHI to track this effort: o3de/o3de#13307.

[tonybalandiuk]

[Done] action taken, issue has been created.

[lmbr-pip]

Something that we should change is there should be clear documentation and automation for issues seeking exception for release. Items tagged in a certain way should kick off workflow that auto attaches checklist, documentation links and next steps to take.

Folks had a lot of uncertainty about how to follow the process and it meant a lot of back and forth messaging to find out what the steps where, to get approvals etc.

[tonybalandiuk]

discussed in sig-release meeting 11/8
There are 2 concerns being raised

1. clear documentation - the current process is going to be documented here Define Release Exception Process #43
2. automation - We agree this is a good next step in improving the exception process. We will create a new issue for this. In creating the issue, we need to consider the different phases that stabilization goes through (where features require approval and bugs don't , then at a later phase even bugs may require approval)

[tonybalandiuk]

@AMZN-daimini to post a discussion to address (2) and link it here

[AMZN-daimini]

Discussion is up #106
Topic is a little broader, but the first two points (Discovery and Tracking) both should cover this concern.

[tonybalandiuk]

[Done] action taken, discussion has been created.

[lmbr-pip]

Something that went well, I thought @tonybalandiuk and @vincent6767 did an excellent job of communicating as release managers. Vincent esp had great bias-for-action and proactively pinged SIG chairs for things needed.

[vincent6767]

Thanks @lmbr-pip !

[tonybalandiuk]

[Done] Thanks! @lmbr-pip

[lmbr-pip]

Something to change, we should have release expectations for SIG(s) written down and shared that include expectations around roadmap issues, feature matrix, exception approvals etc. o3de/community#161 and how it relates to release process.

We seem to be working off tribal knowledge and as SIG-chair(s) are volunteer positions, we should be very clear about expectations so folks can plan their time/contribution effort accordingly.

[vincent6767]

@lmbr-pip By release expectations do you mean what each SIG plan to release on a particular version or what SIG's responsibilities are during the release?

[tonybalandiuk]

@tonybalandiuk to create issue for this and work on it after posting V1 of the current release process

[tonybalandiuk]

[Done] issue created to capture this request #112

[hultonha]

Something that we should change is making the exception process a little more streamlined. The GitHub board helped to triage exceptions and find them, but chasing many sig chairs was time consuming and stressful. Setting-up a communication channel for sig chairs to monitor might be a possible solution. People do not respond to tags/@ on the issue which results in lots of manual chasing (DMs/calls etc...)

[tonybalandiuk]

@AMZN-daimini please include in your discussion about streamlining the exception process, and link to it here

[AMZN-daimini]

I have added this issue (3. Notification) to the discussion and proposed a possible solution similar to the one above. Feel free to comment on it or add an alternative solution if you have ideas!
#106

[tonybalandiuk]

[Done] added to #106 above

[hultonha]

Something that went well was the overall communication and updates from sig-release (especially big shout out to @tonybalandiuk for the regular communication and updates).

[tonybalandiuk]

[Done] Thanks Tom!

[tonybalandiuk]

Something we should change - address the build jobs that were red for the merge to o3de-atom-sampleviewer repo. Potentially separate this merge from the editor release

[vincent6767]

Hi @galibzon and @thefranke
On the last 22.10.0 release, the SIG release was helping with the ASV release. Since this is a standalone product and not critical to the O3DE release in general, we would like to recommend SIG Graphics Audio as the project owner to also handle the release. Thoughts?

[galibzon]

@vincent6767 , yes sig-graphics-audio can and should manage ASV releases.

[vincent6767]

Thanks @galibzon ! SIG Release won't manage ASV starting from the next release then. @tonybalandiuk Is there some documentation that needs to be updated with this decision?

[tonybalandiuk]

@vincent6767 I think this discussion is a good document the decision. I will also update this in the documentation that I'm working on right now to reflect this. Thanks!

[tonybalandiuk]

[DONE] noted as a todo under release doc task #45

[Ulrick28]

Something that should change. Atom Sample Viewer should be it's own release, unrelated to O3DE releases due to size of project and issues with build pipeline customizations. This product is also not owned by the release sig and instead is owned by sig-graphics-audio

[tonybalandiuk]

@vincent6767 this is related to the above.

[tonybalandiuk]

[Done] as noted above @Ulrick28 - SIG-Release will no longer manage releases of ASV

[brianherrera]

Something that we should change is to run the AR for the merge PRs the day before the scheduled launch. This will help avoid issues with unreliable tests or any other failures that are encountered during the AR and delay the launch. Also a clean build is typically required due to the number of changes getting merged in which takes around 3 hours to complete.

Something that we should change regarding ASV, is to have the same level of review for branch health if we keep it in sync with the O3DE launch. For example, the main branch for ASV was already red prior to the release merge so it was difficult to determine if the failures we encountered should be addressed or ignored. I do agree with the comments that its launch should be separated.

[vincent6767]

Agree to run the AR before and makes sure the build is green before the launch day.

Something that we should change regarding ASV, is to have the same level of review for branch health if we keep it in sync with the O3DE launch. For example, the main branch for ASV was already red prior to the release merge so it was difficult to determine if the failures we encountered should be addressed or ignored. I do agree with the comments that its launch should be separated.

I think this needs to be followed up with SIG Graphics Audio as it belongs to each SIG to maintain the branch health #101 (reply in thread)

[tonybalandiuk]

(1) regarding AR being run in advance, we have added that to the description of this release task, o3de/o3de#11098 This task will be copied forward for the next release.
(2) @vincent6767 if we keep ASV in the release process, perhaps we need to enforce the health of the repo. Include in your discussion with sig-graphics

[tonybalandiuk]

@galibzon FYI there were concerns about the health of ASV repo this last release 22.10.0 Please read the discussion here since sig-graphics-audio will own releasing ASV. Thanks.

[tonybalandiuk]

[Done] (1) added to release task (2) @galibzon notified for future releases.

[tonybalandiuk]

Something that didn't go well...We had some late work come into stabilization that ended up stranded there -meaning the code only lives in stabilization and no other branch
See thread here https://discord.com/channels/805939474655346758/816043601754325023/1034970558275780631

[vincent6767]

@tonybalandiuk Have we ever known why it happens? Isn't a review needed before they can merge into the stabilization branch?

[tonybalandiuk]

@vincent6767 it happened because our process currently relies on people knowing and following the manual process. This one slipped through. It would be better if the stabilization process was driven by a process that wasn't manual

[vincent6767]

This one slipped through. It would be better if the stabilization process was driven by a process that wasn't manual

To confirm. Do you mean there are reviewers who approve it?

[tonybalandiuk]

I am assuming the people involved in this PR (creation/merge) were not aware of the stabilization process. I don't think they were aware that only certain code was being allowed in, and that every PR needed multiple approvals.

[tonybalandiuk]

[Done] We have a separate discussion about adding automation around the merge stabilization process.

[vincent6767]

Something that went well

1. On release day went quite well even though there are issues with the AR and ASV. But all of those are solvable quite easily. Thanks to the team!

Something that didn't go well

1. There is a surprise in the marketing committee on the O3DE 22.10.0. release date. This results in the blog updates done after the release.
2. Features grid is not updated on the release day. It was done a day after. This is mainly due to each late feature grid update.

Something that we should change.

1. The way we do release notes right now. I propose the release notes are updated regularly, e.g. monthly instead of at the end of the release as people tend to miss the context of the work done months before.

[vincent6767]

@tonybalandiuk One question related to marketing. Do we also own the release of the O3DE websites? I assume no, but would like to double-confirm with you.

[tonybalandiuk]

@vincent6767 No, we do not own the release of the Website. It is owned by Marketing.
Note our charter does not mention the Website https://github.com/o3de/sig-release/blob/main/governance/SIG%20Release%20Charter.md

We include an update to o3debinaries.org during our release, but that's just for the downloads.

[tonybalandiuk]

@vincent6767
"I propose the release notes are updated regularly"
The release notes discussion is here: #107

If we pursue a bot, or other ongoing way to capture release notes as features are submitted then we will essentially have a constantly-updated set of release notes. At that point we can just have a query that allows the community to easily see "what are the new features with release notes since the last release" which I think will be the most scalable way to address this.

[tonybalandiuk]

[Done] I think this suggested change is captured under the release notes discussion @vincent6767 .

[vincent6767]

Got it. Thanks!

[tonybalandiuk]

All items under this discussion are either address or accounted for in other tasks.
At this point SIG-Release considers this conversation "done".