Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add support to build image from docker file #69

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

feat: add support to build image from docker file #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b88be99
feat: add support to build image from docker file
dineshkumar-j-genea Oct 4, 2024
a9b3515
ci: added yaml to json converter package
dineshkumar-j-genea Oct 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,9 @@ test:smoketests:
alias: docker
before_script:
- apk --update --no-cache add bash gawk curl aws-cli tree xz jq wget xdelta3 openssl1.1-compat gcompat
- wget https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_amd64 -O yq
- chmod +x yq
- mv yq /usr/local/bin
script:
- ./tests/app/run.sh
- ./tests/gen/run.sh
Expand Down Expand Up @@ -108,6 +111,9 @@ test:acceptancetests:
- curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
- chmod +x ./kubectl
- mv ./kubectl /usr/bin/kubectl
- wget https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_amd64 -O yq
- chmod +x yq
- mv yq /usr/local/bin

script:
- ./acceptance-tests/run.sh
Expand Down
20 changes: 14 additions & 6 deletions acceptance-tests/scenarios.d/03.basic.failed.images.not.present.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,20 @@ function test_phase_run_images_not_present() {
--platform linux/amd64 \
--orchestrator docker-compose \
--manifests-dir acceptance-tests/data/manifests-1-broken \
--application-name myapp0a || return 1
echo "images_not_present: checking install rc"
mender install "$artifact_file" && return 2 # we expect a failure
echo "images_not_present: checking for running containers"
docker ps -q | grep -q . && return 3 # and we expect nothing to be running
return 0
--application-name myapp0a # we expect a failure here now as we are using docker-compose to pull the images now

if [ $? -ne 0 ]; then
echo "images_not_present: app-gen artifact generation failed which is expected"
return 0
else
echo "Unexpected success, we were expecting failure"
return 1
fi
# echo "images_not_present: checking install rc"
# mender install "$artifact_file" && return 2 # we expect a failure
# echo "images_not_present: checking for running containers"
# docker ps -q | grep -q . && return 3 # and we expect nothing to be running
# return 0
Comment on lines +54 to +67
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably should not modify the existing tests, why do we need to do it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in this test case, if you closely observe the existing flow, you will that we used image tag to pull image and then mender install it uses the docker compose to validate the image and expected a failure, as the image argument and docker compose totally defined with different image name.

but right now, as per the requirement we are using the docker-compose also to pull an image, which means image argument is essentially becomes the optional one.

with that, during the artifact generation itself, the script will throw an error in the new logic.

}

function test_failed_hook_phase_run_images_not_present() {
Expand Down
25 changes: 17 additions & 8 deletions acceptance-tests/scenarios.d/06.rollback.on.images.not.present.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,14 +65,23 @@ function test_phase_run_rollback_on_images_not_present() {
--platform linux/amd64 \
--orchestrator docker-compose \
--manifests-dir acceptance-tests/data/manifests-1-broken \
--application-name myapp0b || return 1
echo "images_not_present: checking install rc"
mender install "$artifact_file" && return 2 # we expect a failure
sleep $timeout_s
echo "images_not_present: checking for running containers"
docker ps
diff "${temp_dir}/before-rollback-$$" <(docker ps --format '{{.Image}}')
return 0
--application-name myapp0b # we expect a failure here now as we are using docker-compose to pull the images now

if [ $? -ne 0 ]; then
echo "images_not_present: app-gen artifact generation failed which is expected"
return 0
else
echo "Unexpected success, we were expecting failure"
return 1
fi

# echo "images_not_present: checking install rc"
# mender install "$artifact_file" && return 2 # we expect a failure
# sleep $timeout_s
# echo "images_not_present: checking for running containers"
# docker ps
# diff "${temp_dir}/before-rollback-$$" <(docker ps --format '{{.Image}}')
# return 0
Comment on lines +78 to +84
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a bit of commented out code in this pull request, could you clean that?

}

function test_failed_hook_phase_run_rollback_on_images_not_present() {
Expand Down
60 changes: 52 additions & 8 deletions gen/app-gen
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,12 +212,11 @@ if [ ${#images[@]} -lt 1 ]; then
if [[ "${orchestrator}" == "${DOCKER_COMPOSE_ORCHESTRATOR}" ]]; then
echo "No specific images specified. Will try to extract from docker-compose.yaml file." >&2

while read -r img; do
images+=("$img")
images+=("$img")
done < <( docker compose --project-directory "$manifests_dir" config --images)
# Check the docker compose has atleast 1 image specified.
img_count=$(docker compose --project-directory "$manifests_dir" config --images | wc -l)

if [ ${#images[@]} -lt 1 ]; then
# If img_count
if [ "$img_count" -lt 1 ]; then
echo "Image extraction from docker-compose.yaml failed. Aborting." >&2
show_help_and_exit_error
fi
Expand Down Expand Up @@ -260,12 +259,14 @@ get_image_sha() {

pull_image() {
local -r url="$1"
# use the provided platform or fall back to global
local platform_arg="${2:-$platform}"

[[ "$url" == "" ]] && { return 0; }
if [[ "${orchestrator}" == "${DOCKER_COMPOSE_ORCHESTRATOR}" ]]; then
docker pull "$url" --platform "$platform"
docker pull "$url" --platform "$platform_arg" || exit 2
else
ctr image pull "$url" --platform "$platform"
ctr image pull "$url" --platform "$platform_arg" || exit 2
fi
}

Expand Down Expand Up @@ -489,9 +490,46 @@ prepare_images() {
local url_current
local url_new
local output_dir
local services

# Pull images specified in args
for ((i = 0; i < ${#images[@]}; i++)); do
pull_image "${images[${i}]}"
done

# Extract services from the Docker Compose file
if [[ "${orchestrator}" == "${DOCKER_COMPOSE_ORCHESTRATOR}" ]]; then
echo "Using Docker Compose"
services=$(docker compose --project-directory "$manifests_dir" config --services)
for service in $services; do
# Convert Docker Compose config to JSON
config_json=$(docker compose --project-directory "$manifests_dir" config | yq eval -o=json)
# Check if the service has a build context
build_context=$(echo "$config_json" | jq -r ".services.\"$service\".build.context // empty")
if [ -n "$build_context" ]; then
echo "Building Image from Project"
# Build the image locally using Docker Compose
docker compose --project-directory "$manifests_dir" build "$service"
image_name=$(echo "$config_json" | jq -r ".services.\"$service\".image")
images+=("$image_name")
images+=("$image_name")
else
echo "Pulling Image from Directory"
# Get the image name from the Docker Compose file
image_name=$(echo "$config_json" | jq -r ".services.\"$service\".image")
images+=("$image_name")
images+=("$image_name")

# Extract platform from Docker Compose config (if specified)
service_platform=$(echo "$config_json" | jq -r ".services.\"$service\".platform // empty")
# Pull the image from the registry, using the extracted platform if available
pull_image "$image_name" "$service_platform"
fi
# images_shas+=($(get_image_sha "$image_name"))
done
fi

for ((i = 0; i < ${#images[@]}; i++)); do
images_shas+=($(get_image_sha "${images[${i}]}"))
done
declare -p images_shas
Expand Down Expand Up @@ -565,7 +603,13 @@ generate_metadata() {

prepare_images "${temp_dir}/images"
( cd "${temp_dir}" && tar czvf images.tar.gz images )
cp -a "${manifests_dir}" "${temp_dir}/manifests"
mkdir "${temp_dir}/manifests"
for file in "${manifests_dir}/docker-compose.yaml" "${manifests_dir}/docker-compose.yml"; do
if [ -e "$file" ]; then
cp -a "$file" "${temp_dir}/manifests/docker-compose.yaml"
break # Exit the loop after copying the first found file
fi
done
( cd "${temp_dir}" && tar czvf "manifests.tar.gz" manifests )
generate_metadata "${temp_dir}/metadata.json"

Expand Down
2 changes: 2 additions & 0 deletions tests/gen/data/docker/manifests/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ services:
- "65536"
db:
image: docker.io/library/memcached:1.6.18-alpine
# this images doesn't have the support for linux/arm/v7, test case failes for as we are passing platform 01_deep_delta_generate.run.sh
platform: linux/amd64
networks:
- local_net
environment:
Expand Down