Regression: Docker deployment from private registry appears broken (after upgrade to concourse 3.9.2 from 3.8.0)

[r-chris]

Hi, I have upgraded to 3.9.2 from 3.8.0 and am experiencing fatal errors when trying to deploy any docker image from a private registry. The error as shown below.

For some reason the provided CF_DOCKER_PASSWORD is no longer being found or the input arguments to cf push are messed up. Any help or currently working examples with private docker registries would be much appreciated.

Related: nulldriver/cf-cli-resource#24

[r-chris]

Manifest file:

applications:
- name: test-deploy-private
  docker:
    image: PRIVATE_DOCKER_REGISTRY_NAME/test-app:latest
    username: ***
  command: null
  memory: 128M
  instances: 1
  disk_quota: 5G
  health-check-type: process
  routes:
    - route: test-deploy-private.***

Deployment plan:

- name: deploy-build-in-cf-private
  plan:
  - get: git
    passed: [build-private]
    trigger: false
  - put: cf-build-in
    params:
      manifest: git/manifest-private.yml
      docker_username: ((docker-registry-username))
      docker_password: ((docker-registry-password))
      environment_variables:
        CF_DOCKER_PASSWORD: ((docker-registry-password))

Error message:

Incorrect Usage: '--docker-image, -o' and '--docker-username' must be used together.
FAILED

NAME:
   push - Push a new app or sync changes to an existing app

USAGE:
   cf push APP_NAME [-b BUILDPACK_NAME] [-c COMMAND] [-f MANIFEST_PATH | --no-manifest] [--no-start]
   [-i NUM_INSTANCES] [-k DISK] [-m MEMORY] [-p PATH] [-s STACK] [-t HEALTH_TIMEOUT] [-u (process | port | http)]
   [--no-route | --random-route | --hostname HOST | --no-hostname] [-d DOMAIN] [--route-path ROUTE_PATH]

   cf push APP_NAME --docker-image [REGISTRY_HOST:PORT/]IMAGE[:TAG] [--docker-username USERNAME]
   [-c COMMAND] [-f MANIFEST_PATH | --no-manifest] [--no-start]
   [-i NUM_INSTANCES] [-k DISK] [-m MEMORY] [-t HEALTH_TIMEOUT] [-u (process | port | http)]
   [--no-route | --random-route | --hostname HOST | --no-hostname] [-d DOMAIN] [--route-path ROUTE_PATH]

   cf push -f MANIFEST_WITH_MULTIPLE_APPS_PATH [APP_NAME] [--no-start]

ALIAS:
   p

OPTIONS:
   -b                           Custom buildpack by name (e.g. my-buildpack) or Git URL (e.g. 'https://github.com/cloudfoundry/java-buildpack.git') or Git URL with a branch or tag (e.g. 'https://github.com/cloudfoundry/java-buildpack.git#v3.3.0' for 'v3.3.0' tag). To use built-in buildpacks only, specify 'default' or 'null'
   -c                           Startup command, set to null to reset to default start command
   -d                           Domain (e.g. example.com)
   --docker-image, -o           Docker-image to be used (e.g. user/docker-image-name)
   --docker-username            Repository username; used with password from environment variable CF_DOCKER_PASSWORD
   -f                           Path to manifest
   --health-check-type, -u      Application health check type (Default: 'port', 'none' accepted for 'process', 'http' implies endpoint '/')
   --hostname, -n               Hostname (e.g. my-subdomain)
   -i                           Number of instances
   -k                           Disk limit (e.g. 256M, 1024M, 1G)
   -m                           Memory limit (e.g. 256M, 1024M, 1G)
   --no-hostname                Map the root domain to this app
   --no-manifest                Ignore manifest file
   --no-route                   Do not map a route to this app and remove routes from previous pushes of this app
   --no-start                   Do not start an app after pushing
   -p                           Path to app directory or to a zip file of the contents of the app directory
   --random-route               Create a random route for this app
   --route-path                 Path for the route
   -s                           Stack to use (a stack is a pre-built file system, including an operating system, that can run apps)
   -t                           Time (in seconds) allowed to elapse between starting up an app and the first healthy response from the app

ENVIRONMENT:
   CF_STAGING_TIMEOUT=15        Max wait time for buildpack staging, in minutes
   CF_STARTUP_TIMEOUT=5         Max wait time for app instance startup, in minutes
   CF_DOCKER_PASSWORD=          Password used for private docker repository

SEE ALSO:
   apps, create-app-manifest, logs, ssh, start
error running command: exit status 1

[r-chris]

I found these issues and a workaround here:

1. Providing docker_username: XXX as part of the parameters doesn't work as it creates a cf command such as the one below, which is missing the --docker-image argument - even though that image name was actually provided as part of the manifest file:
   cf push -f manifest-private.yml --docker-username XXX

2. Providing CF_DOCKER_PASSWORD explicitly through the parameters environment_variables: doesn't work - you have to provide it through docker_password: ZZZ

The only working combination of manifest file that specifies a private registry is to provide the docker username inside the manifest file (as shown above) and to only provide the docker registry password docker_password: ZZZ and not also the docker username docker_username: XXX

[r-chris]

The deployment plan now has to be changed to this:

- name: deploy-build-in-cf-private
  plan:
  - get: git
    passed: [build-private]
    trigger: false
  - put: cf-build-in
    params:
      manifest: git/manifest-private.yml
      docker_password: ((docker-registry-password))

docker_username: ((docker-registry-username))

[r-chris]

As the error above shows, the underlying issue is that the cf push command expects you to also provide --docker-image as an argument when providing --docker-username and isn't smart enough to try and find that information inside the manifest file. This resource can not set --docker-image though and you probably want to decide if you should remove --docker-username altogether or add support for --docker-image.