Considered using Kaniko?

[f0o]

Kaniko supports building images without requiring privileged mode.

https://github.com/GoogleContainerTools/kaniko

[vito]

Yep! Just haven't had time to dig into this yet. It may also be possible to use it to resolve #1. Some context here: concourse/docker-image-resource#190 (comment)

[jmccann]

I've used kaniko and ran into multiple issues where a Dockerfile that works with docker build didn't work with kaniko. This meant updating the Dockerfile to workaround kaniko bugs. Last I tried kaniko was several months ago. So I'd personally suggest not using it based on my own recent experience.

[f0o]

@jmccann Can you disclose the Dockerfile that needed hacks? I havent experienced any of those incompatibilities. Using it for almost a year now on the daily for many images

[jmccann]

@f0o I don't have an example readily available as I've given up for a few months and would have to go back and dig into and/or try again. Also not sure I'm at liberty to share.

But I will say personally I've ran into about 3 separate Dockerfiles I tried to convert to using kaniko that I ran into issues with. I've personally had about a 50% success rate. I've seen others run into issues as well. I will say majority of people seem to be able to use it without issue. But I'm worried that multiple times things that work completely fine with docker build did not work with kaniko.

Unfortunately the cases I've ran into where there were issues with kaniko and not docker were not maybe "typical" builds. However, I wouldn't say they were complete weird edgecases either. But maybe not something that'd be easy to reproduce more generically quickly to share.

I would just say if you decide to move that direction be very aware of if/when people start opening issues for failed builds. 😄

[marcaurele]

I've used kaniko and ran into multiple issues where a Dockerfile that works with docker build didn't work with kaniko. This meant updating the Dockerfile to workaround kaniko bugs. Last I tried kaniko was several months ago. So I'd personally suggest not using it based on my own recent experience.

? "Last I tried kaniko was several months ago" and "recent experience" from a single user and which are conflicting arguments IMO.

I'm surprised that a single comment without any clear example on the issue seems to block the change forward kaniko. Personally I've been using it for 1-2 years without any issue in all Dockefile. Being able to build without elevated privileges is worth the change.

[jmccann]

I guess it could be considered conflicting 😆

I hope I'm not blocking (and don't think I am). I'm just giving my experience.

[vito]

Definitely isn't blocking - if it was this would be closed. The only thing blocking this is the number of hours in the day and the fairly dramatic code change it would imply, which makes the barrier to entry pretty high for me. Anyone could create an alternative Kaniko-flavored image builder though; there's no monopoly.

[marcaurele]

I'm new to concourse and I already setup my build with kaniko without specifying a custom image, using the one from gcr.io. What is the added value of creating a custom one like this project on top of such a tool?

[kardashov]

@marcaurele could you share concourse build using kaniko container snippet here?
I'm trying to build image using private image from gcr.io/ as base image, then push resulting image back to gcr.io/ using kaniko container, but struggling to mount gcp_credentials.json file properly.

[marcaurele]

I do in 2 steps:

1. Create the credentials file for the registry
2. Build the image: tasks/build.yml

---
# Kaniko build: `--force` is required on concourse for unknown reason, otherwise
# it does not detect it's inside a container.
platform: linux

image_resource:
  type: registry-image
  source:
    repository: gcr.io/kaniko-project/executor
    tag: debug

params:
  BRANCH: dev
  PUSH:
  TAG_SUFFIX:

inputs:
  - name: repo
  - name: credentials
    optional: true

run:
  path: sh
  args:
    - -ecx
    - |
      CREDENTIALS_FILE=credentials/config.json
      test -f $CREDENTIALS_FILE && cp $CREDENTIALS_FILE /kaniko/.docker/config.json || echo "Missing credentials file!"
      FILE_SHORT_REF=repo/.git/short_ref
      git_short_ref=$(test -f $FILE_SHORT_REF && cat $FILE_SHORT_REF || echo "dev")
      FILE_REF=repo/.git/ref
      git_ref=$(test -f $FILE_REF && cat $FILE_REF || echo "dev")
      destination_flag=$(test -z $PUSH && echo "--no-push" || \
          echo "--destination $REPOSITORY:$git_short_ref$TAG_SUFFIX --destination $REPOSITORY:latest$TAG_SUFFIX")
      /kaniko/executor --force \
          --build-arg "version=$git_ref" \
          --single-snapshot \
          --label org.opencontainers.image.version=$BRANCH \
          --label org.opencontainers.image.revision=$git_ref \
          --label org.opencontainers.image.created=$(date -Iseconds -u) \
          --context repo/ \
          --skip-unused-stages \
          --target $IMAGE_TARGET \
          $destination_flag

[kardashov]

Thanks @marcaurele!
So the trick is to use DEBUG version on kaniko image which has shell installed https://github.com/GoogleContainerTools/kaniko#debug-image. Works great for me.
In fact, using kaniko in Concourse is so flexible and powerful that it deserves at least mentioning in this guide: https://concourse-ci.org/container-image-guides.html

[marcaurele]

I think this ticket can be closed as kaniko will require a large effort¹ to provide multi-arch build, therefore it's not a good option to follow.

Footnotes

1. https://github.com/GoogleContainerTools/kaniko/issues/786#issuecomment-1069377414 ↩