deploy-deployment-preview Action Fails on Rebase/Force Push (0.8.0)

[annieastronomer]

on behalf of Aledade
This is related to #82, but the rebase/force push presents a slightly different form of how $GITHUB_REF is formed coming into the "Get Deploy Type" step.

Steps to reproduce:

Setup
Our Github action is as follows:

on:
  pull_request:
...
branch-push:
    if: github.ref != 'refs/heads/main' && github.ref != 'refs/heads/develop'
    env:
      ## Sets dev deployment API credential as an environment variable (created by devops)
      ASTRO_API_TOKEN: ${{ secrets.DEV_ASTRO_API_TOKEN }}
    runs-on:
      - self-hosted
      - linux
    steps:
      - name: Deploy to Astro
        id: Deploy
        uses: astronomer/[email protected]
        with:
          action: deploy-deployment-preview
          deployment-id: .... ##This is the template deployment id
          wake-on-deploy: true
          parse: true

Test Steps

1. Create and push a branch with some minor change to a dag or plugin (lets call the branch new-cool-branch)
2. On the local repository delete that named branch new-cool-branch and then make a new branch with a slightly different change also called new-cool-branch (simulates a rebase or a cleaned up version if things got really really messy... at any rate the hashes on the commit history are all now different)
3. Force push the now very different branch git push --force-with-lease -u origin new-cool-branch
4. Deploy action will fail.
   (edited)
   3:22
   The patched form that DOES work is for rebase/force pushes is:
   action_v0.8.0_patched.yaml

name: "Deploy Apache Airflow DAGs to Astro"
description: "Test your DAGs and deploy your Astro project to a Deployment on Astro, Astronomer's managed Airflow service."
author: "Astronomer"
branding:
icon: "upload-cloud"
color: "purple"
inputs:
root-folder:
required: false
default: ""
description: "Path to the Astro project, or dbt project for dbt deploys."
parse:
required: false
default: false
description: "If true DAGs will be parsed before deploying to Astro."
pytest:
required: false
default: false
description: "if true custom pytests will be ran before deploying to Astro."
pytest-file:
required: false
default: tests/
description: "Specify custom pytest files to run with the pytest command."
force:
required: false
default: false
description: "If true your code will be force deployed to Astronomer. Mostly used to skip parse test on image deploys."
image-name:
required: false
default: no-custom-image
description: "Specify a custom built image to deploy to an Asto Deployment. To be used with 'deploy-type' set to 'image-and-dags' or 'infer'"
action:
required: false
default: deploy
description: "Specify what action you would like to take. Use this option to create or delete deployment previews. Specify either 'deploy', 'create-deployment-preview', 'delete-deployment-preview', or 'deploy-deployment-preview'. If using 'deploy' or 'deploy-deployment-preview' one should also specify 'deploy-type'."
deployment-name:
required: false
description: "The name of the Deployment you want to make preview from or are deploying to."
deployment-id:
required: false
description: "The id of the Deployment you to make a preview from or are deploying to."
workspace:
required: false
description: "Workspace id to select. Only required when ASTRO_API_TOKEN is given an organization token."
preview-name:
required: false
description: "Custom preview name. By default this is ."
copy-connections:
required: false
default: true
description: "Copy connections from the original Deployment to the new deployment preview."
copy-airflow-variables:
required: false
default: true
description: "Copy Airflow variables from the original Deployment to the new deployment preview."
copy-pools:
required: false
default: true
description: "Copy pools from the original Deployment to the new deployment preview."
cli-version:
required: false
default: ""
description: "The desired Astro CLI version to use"
checkout:
required: false
default: true
description: "Whether to checkout the repo as the first step. Set this to false if you want to modify repo contents before invoking the action"
description:
required: false
description: >
A description to set for deploying to Astro. This is equivalent to running astro deploy --description "..." with the Astro CLI.
The description is visible in the Deploy History tab in your Astro Deployment, and can be helpful to explain what triggered a deploy.
​
For example, to display the most recent commit that resulted in an Astro deploy, you could configure
description: "Deployed from commit ..." with the value of github.event.after.
This would display e.g. "Deployed from commit da39a3ee5e6b4b0d3255bfef95601890afd80709".
Reference: https://docs.github.com/en/webhooks/webhook-events-and-payloads#push.
deploy-image:
required: false
default: false
description: "If true image and DAGs will deploy. NOTE: This option is deprecated and will be removed in a future release. Use deploy-type: image-and-dags instead."
deploy-type:
required: false
default: "infer"
description: "Specify the type of deploy you would like to do. Use this option to deploy only DAGs or the image or DBT project. Specify either 'infer', 'dags-only', 'image-and-dags', or 'dbt'. 'infer' option would infer between DAG only deploy and image and DAG deploy based on updated files."
build-secrets:
required: false
description: "Mimics docker build --secret flag. See https://docs.docker.com/build/building/secrets/ for more information. Example input 'id=mysecret,src=secrets.txt'"
mount-path:
required: false
default: ""
description: "Path to mount dbt project in Airflow, for reference by DAGs. Default /usr/local/airflow/dbt/{dbt project name}"
checkout-submodules:
required: false
default: false
description: "Whether to checkout submodules when cloning the repository: false to disable (default), true to checkout submodules or recursive to recursively checkout submodules. Works only when checkout is set to true."
wake-on-deploy:
required: false
default: false
description: "If true, the deployment will be woken up from hibernation before deploying. NOTE: This option overrides the deployment's hibernation override spec."
outputs:
preview-id:
description: "The ID of the created deployment preview. Only works when action=create-deployment-preview"
value: ${{ steps.deployment-preview.outputs.FINAL_DEPLOYMENT_ID }}
​
runs:
using: "composite"
steps:
- name: checkout repo
uses: actions/checkout@v4
if: inputs.checkout == 'true'
with:
fetch-depth: 0 # Fetch all history
ref: ${{ github.event.after }}
clean: false
submodules: ${{ inputs.checkout-submodules }}
- name: Warn about deprecated deploy-image option
if: inputs.deploy-image == true
shell: bash
run: |
echo "The 'deploy-image' option is deprecated and will be removed in 1.0 release. Use 'deploy-type: image-and-dags' instead."
- name: Install Astro CLI
run: |
​
echo ::group::Install Astro CLI
CLI_VERSION=${{ inputs.cli-version }}
# if Astro CLI is pre-installed, fetch it's version
if command -v astro &> /dev/null; then
# "astro version" commands returns the CLI version, its output is of the form: "Astro CLI Version: 1.29.0"
CLI_VERSION=$(astro version | awk '{print $4}')
fi
​
# Check if the Astro CLI version is less than 1.28.1 for dbt-deploy
if [[ "${{ inputs.deploy-type }}" == "dbt" && $CLI_VERSION != "" ]]; then
#install semver to compare versions
npm install -g semver
​
REQUIRED_VERSION="1.28.1"
CURRENT_VERSION=$(astro version | awk '{print $4}')
if ! semver -r ">${REQUIRED_VERSION}" "${CURRENT_VERSION}"; then
echo "DBT Deploy requires Astro CLI version $REQUIRED_VERSION or higher"
exit 1
fi
fi
​
# skip Astro CLI installation if already installed
if command -v astro &> /dev/null; then
echo "Astro CLI is already installed"
exit 0
fi
​
# check if CLI_VERSION does not starts with "v", then add "v" to it
if [[ $CLI_VERSION != "" && $CLI_VERSION != v* ]]; then
CLI_VERSION=v$CLI_VERSION
fi
​
if [[ $CLI_VERSION == "" ]]; then
curl -sSL https://install.astronomer.io | sudo bash -s
else
curl -sSL https://install.astronomer.io | sudo bash -s -- $CLI_VERSION
fi
echo ::endgroup::
shell: bash
- name: Determine Deploy Deployment
run: |
​
echo ::group::Determine Deploy Deployment
​
# validate action input
if [[ ${{ inputs.action }} != create-deployment-preview && ${{ inputs.action }} != delete-deployment-preview && ${{ inputs.action }} != deploy-deployment-preview && ${{ inputs.action }} != deploy ]]; then
echo ERROR: you specified an improper action input. Action must be deploy, deploy-deployment-preview, create-deployment-preview, or delete-deployment-preview.
exit 1 # terminate and indicate error
fi
​
# Select workspace if specified
if [[ "${{ inputs.workspace }}" != "" ]]; then
astro workspace switch ${{ inputs.workspace }}
fi
# error if both deployment name and id are used
if [[ "${{ inputs.deployment-name }}" != "" && "${{ inputs.deployment-id }}" != "" ]]; then
echo ERROR: cannot specify both a Deployment ID and Name
exit 1 # terminate and indicate error
fi
# figure out deployment id
if [[ "${{ inputs.deployment-name }}" != "" ]]; then
# get deployment-id
DEPLOYMENT_ID="$(astro deployment inspect --clean-output -n "${{ inputs.deployment-name }}" --key metadata.deployment_id)"
fi
​
if [[ "${{ inputs.deployment-id }}" != "" ]]; then
DEPLOYMENT_ID="${{ inputs.deployment-id }}"
fi
# create deployment preview if action is create-deployment-preview
if [[ ${{ inputs.action }} == create-deployment-preview ]]; then
​
if [[ "${{ inputs.deployment-name }}" == "" && "${{ inputs.deployment-id }}" == "" ]]; then
echo ERROR: cannot create a deployment preview without specifying a deployment name or id
exit 1 # terminate and indicate error
fi
​
if [[ "${{ inputs.preview-name }}" != "" ]]; then
BRANCH_DEPLOYMENT_NAME="${{ inputs.preview-name }}"
else
# get branch name
DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --clean-output --key configuration.name)"
if [[ ${GITHUB_HEAD_REF##/} != "" ]]; then
BRANCH_DEPLOYMENT_NAME=${GITHUB_HEAD_REF##/}$DEPLOYMENT_NAME
else
branch_ref="${{ github.ref }}"
branch_name="${branch_ref##/}"
BRANCH_DEPLOYMENT_NAME=${branch_name}$DEPLOYMENT_NAME
fi
BRANCH_DEPLOYMENT_NAME="${BRANCH_DEPLOYMENT_NAME// /}"
echo $BRANCH_DEPLOYMENT_NAME
fi
​
# Create template of deployment to be copied with
astro deployment inspect $DEPLOYMENT_ID --clean-output --template > deployment-preview-template.yaml # automatically creates deployment-preview-template.yaml file
​
# Add name to deployment template file
sed -i "s| name:.| name: ${BRANCH_DEPLOYMENT_NAME}|g" deployment-preview-template.yaml
​
# Create new deployment preview based on the deployment template file
astro deployment create --deployment-file deployment-preview-template.yaml
​
# TODO: we need to add wait for deployment to be created, otherwise operation to copy airflow resources like connection is flaky if webserver is not up by then
​
# Get final Deployment ID
echo "FINAL_DEPLOYMENT_ID=$(astro deployment inspect --clean-output -n $BRANCH_DEPLOYMENT_NAME --key metadata.deployment_id)" >> $GITHUB_OUTPUT
​
# Get original Deployment ID
echo "ORIGINAL_DEPLOYMENT_ID=$DEPLOYMENT_ID" >> $GITHUB_OUTPUT
fi
​
# delete deployment preview and skip deploy if action is delete-deployment-preview
if [[ ${{ inputs.action }} == delete-deployment-preview ]]; then
​
if [[ "${{ inputs.deployment-name }}" == "" && "${{ inputs.deployment-id }}" == "" ]]; then
echo ERROR: cannot delete a deployment preview without specifying a deployment name or id
exit 1 # terminate and indicate error
fi
​
if [[ "${{ inputs.preview-name }}" != "" ]]; then
BRANCH_DEPLOYMENT_NAME="${{ inputs.preview-name }}"
​
else
DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --clean-output --key configurat...

The patch should be compared to (hash e6e4fdf currently tagged as v0.8.0)

[annieastronomer]

other patch version
diff --git a/action.yaml b/action.yaml
index d729914..c9cd595 100644
--- a/action.yaml
+++ b/action.yaml
@@ -361,24 +361,43 @@ runs:
cd ${{ inputs.root-folder }}
fi

•    branch=$(echo "${GITHUB_REF#refs/heads/}")
  

•    echo "Branch pushed to: $branch"
  

•    git fetch origin $branch
  

•    ##### START ARCH-23156 PATCH
  

•    # Switch initialization of these variables to the "off" case as there are more than one case where that is what we want.
     DAGS_ONLY_DEPLOY=false
  

•    SKIP_IMAGE_OR_DAGS_DEPLOY=true
  

•    SKIP_IMAGE_OR_DAGS_DEPLOY=false
  

•    files=()
  

•    ##### END ARCH-23156 PATCH     
  
     # case when the triggered event is a new branch or tag creation, we would need to deploy the image because we cannot determine that it does not need to be deployed
     if [[ "${{ github.event.before }}" == "0000000000000000000000000000000000000000" ]]
     then
  

•      DAGS_ONLY_DEPLOY=false
  

•      SKIP_IMAGE_OR_DAGS_DEPLOY=false
  

•      files=()
  

•    else
  

•      echo "github.event.before detected to be: 0000000000000000000000000000000000000000"
  

•    ##### START ARCH-23156 PATCH
  

•    elif [[ "$GITHUB_REF" == refs/tags/* ]]
  

•    then     
  

•      # This branch is how things were solved in Issue 82 and is very likely redundant 
  

•      # with the "${{ github.event.before }}" == "0000000000000000000000000000000000000000" case above
  

•      tag_name="${GITHUB_REF#refs/tags/}"
  

•      echo "Tag Push Event. Tag name: $tag_name"                    
  

•    elif [[ "$GITHUB_REF" == refs/pull/*/merge ]]; then
  

•      pr_number="${GITHUB_REF#refs/pull/}"
  

•      pr_number="${pr_number%%/*}"
  

•      echo "Pull Request number: $pr_number"                       
  

•    elif [[ "$GITHUB_REF" == refs/heads/* ]]; 
  

•    then
  

•      branch=$(echo "${GITHUB_REF#refs/heads/}")
  

•      echo "Branch pushed to: $branch"
  

•      git fetch origin $branch       
  

•      SKIP_IMAGE_OR_DAGS_DEPLOY=true          
  

•      ##### END ARCH-23156 PATCH        
       files=$(git diff --name-only ${{ github.event.before }} ${{ github.event.after }})
       echo "files changed: $files"
  

•    # ARCH-23156 PATCH
  

•    else
  

•      echo "Unknown event type. $GITHUB_REF" 
  

•    # END ARCH-23156 PATCH
     fi
  

•    for file in $files; do
       if [[ $file =~ ^"${{ inputs.root-folder }}".* ]]; then
         echo $file is part of the input root folder
  

@@ -402,6 +421,24 @@ runs:
fi
fi
done
+
+

•    ##### START ARCH-23156 PATCH
  

•    set -x
  

•    echo "======================"
  

•    echo "DEBUGGING INFORMATION"
  

•    echo "======================"
  

•    echo "GITHUB_REF=$GITHUB_REF"
  

•    echo "github.event.before=${{ github.event.before }}"
  

•    echo "github.event.after=${{ github.event.after }}"
  

•    echo "inputs.image-name = ${{ inputs.image-name }}"
  

•    echo "SKIP_IMAGE_OR_DAGS_DEPLOY = $SKIP_IMAGE_OR_DAGS_DEPLOY"
  

•    echo "steps.dag-deploy-enabled.outputs.DAG_DEPLOY_ENABLED = ${{ steps.dag-deploy-enabled.outputs.DAG_DEPLOY_ENABLED }}"
  

•    echo "inputs.action = ${{ inputs.action }}"
  

•    echo "========================="
  

•    echo "END DEBUGGING INFORMATION"
  

•    echo "========================="
  

•    ##### END ARCH-23156 PATCH
  
     # Note: the order of these following checks is important to ensure that we skip/trigger deploy correctly in following cases:
     # 1. When there is no change in the input root folder we should skip deploy, but not when it's deployment preview create action
  

Collapse