Resumable Upload problem

[MiltiadisKoutsokeras]

It seems that the upload functionality returns an error due to a wrong host name returned by the server to the client during resumable uploads.

The same client Python program works as expected when run from the host system and uses the exposed port of the GCP Emulator server.

Here is the minimal case that showcases the problem:

compose.yaml

services:
    google_storage:
        image: oittaa/gcp-storage-emulator
        ports:
            # Exposed in port 9023 of localhost
            - "127.0.0.1:9023:9023/tcp"
        entrypoint: gcp-storage-emulator
        command: ["start",
            "--host=0.0.0.0", "--port=9023", "--in-memory",
            "--default-bucket=localtesting_bucket" ]

    upload:
        image: python:3.7-buster
        environment:
            STORAGE_EMULATOR_HOST: "http://google_storage:9023"
        entrypoint: /entrypoint.sh
        volumes:
            - ./entrypoint.sh:/entrypoint.sh:ro
            - ./upload.py:/upload.py:ro

entrypoint.sh

#!/usr/bin/env bash

# Install Python requirements
pip install google-cloud-storage==1.31.2

echo "STORAGE_EMULATOR_HOST: ${STORAGE_EMULATOR_HOST}"

# Test upload data, 10MiB file to force resumable upload
dd if=/dev/zero of=/test.data bs=1024 count=10240

exec python3 /upload.py

upload.py

import sys
from google.auth.credentials import AnonymousCredentials
from google.cloud import storage

# Upload
client = storage.Client(credentials=AnonymousCredentials(),
                                project='localtesting')
bucket_obj = client.bucket('localtesting_bucket')
blob_obj = bucket_obj.blob('test/test.data')
blob_obj.upload_from_filename('/test.data')

# Print bucket contents
for bucket in client.list_buckets():
    print(f'Bucket: {bucket}')
    for blob in bucket.list_blobs():
        print(f'|_Blob: {blob}')

sys.exit(0)

Put these 3 files in a directory and then docker-compose up to run. The initial connection to the GCP emulator service succeeds:

google_storage_1  | "POST /upload/storage/v1/b/localtesting_bucket/o?uploadType=resumable HTTP/1.1" 200 -

but then an error occurs during connection, with host 0.0.0.0 and port 9023:

upload_1          | Traceback (most recent call last):
upload_1          |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 170, in _new_conn
upload_1          |     (self._dns_host, self.port), self.timeout, **extra_kw
upload_1          |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 96, in create_connection
upload_1          |     raise err
upload_1          |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 86, in create_connection
upload_1          |     sock.connect(sa)
upload_1          | ConnectionRefusedError: [Errno 111] Connection refused
upload_1          | 
upload_1          | During handling of the above exception, another exception occurred:
upload_1          | 
upload_1          | Traceback (most recent call last):
upload_1          |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
upload_1          |     chunked=chunked,
upload_1          |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
upload_1          |     conn.request(method, url, **httplib_request_kw)
upload_1          |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 234, in request
upload_1          |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
upload_1          |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
upload_1          |     self._send_request(method, url, body, headers, encode_chunked)
upload_1          |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
upload_1          |     self.endheaders(body, encode_chunked=encode_chunked)
upload_1          |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
upload_1          |     self._send_output(message_body, encode_chunked=encode_chunked)
upload_1          |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
upload_1          |     self.send(msg)
upload_1          |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
upload_1          |     self.connect()
upload_1          |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 200, in connect
upload_1          |     conn = self._new_conn()
upload_1          |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
upload_1          |     self, "Failed to establish a new connection: %s" % e
upload_1          | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe241769150>: Failed to establish a new connection: [Errno 111] Connection refused
upload_1          | 
upload_1          | During handling of the above exception, another exception occurred:
upload_1          | 
upload_1          | Traceback (most recent call last):
upload_1          |   File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
upload_1          |     timeout=timeout
upload_1          |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
upload_1          |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
upload_1          |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 574, in increment
upload_1          |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
upload_1          | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='0.0.0.0', port=9023): Max retries exceeded with url: /upload/storage/v1/b/localtesting_bucket/o?uploadType=resumable&upload_id=localtesting_bucket%3Atest%2Ftest.data%3A2021-09-08+10%3A13%3A34.413826 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe241769150>: Failed to establish a new connection: [Errno 111] Connection refused'))
upload_1          | 
upload_1          | During handling of the above exception, another exception occurred:
upload_1          | 
upload_1          | Traceback (most recent call last):
upload_1          |   File "/upload.py", line 10, in <module>
upload_1          |     blob_obj.upload_from_filename('/test.data')
upload_1          |   File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 2348, in upload_from_filename
upload_1          |     checksum=checksum,
upload_1          |   File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 2235, in upload_from_file
upload_1          |     checksum=checksum,
upload_1          |   File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 2082, in _do_upload
upload_1          |     checksum=checksum,
upload_1          |   File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1949, in _do_resumable_upload
upload_1          |     response = upload.transmit_next_chunk(transport, timeout=timeout)
upload_1          |   File "/usr/local/lib/python3.7/site-packages/google/resumable_media/requests/upload.py", line 503, in transmit_next_chunk
upload_1          |     timeout=timeout,
upload_1          |   File "/usr/local/lib/python3.7/site-packages/google/resumable_media/requests/_request_helpers.py", line 136, in http_request
upload_1          |     return _helpers.wait_and_retry(func, RequestsMixin._get_status_code, retry_strategy)
upload_1          |   File "/usr/local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 188, in wait_and_retry
upload_1          |     raise error
upload_1          |   File "/usr/local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 177, in wait_and_retry
upload_1          |     response = func()
upload_1          |   File "/usr/local/lib/python3.7/site-packages/google/auth/transport/requests.py", line 486, in request
upload_1          |     **kwargs
upload_1          |   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
upload_1          |     resp = self.send(prep, **send_kwargs)
upload_1          |   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
upload_1          |     r = adapter.send(request, **kwargs)
upload_1          |   File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
upload_1          |     raise ConnectionError(e, request=request)
upload_1          | requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=9023): Max retries exceeded with url: /upload/storage/v1/b/localtesting_bucket/o?uploadType=resumable&upload_id=localtesting_bucket%3Atest%2Ftest.data%3A2021-09-08+10%3A13%3A34.413826 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe241769150>: Failed to establish a new connection: [Errno 111] Connection refused'))

I think the problem resides to the URL returned by the server to resume the upload. It contains the bind IP which breaks the HTTP client on the uploader side.

[MiltiadisKoutsokeras]

Small workaround:

diff --git a/compose.yaml b/compose.yaml
index dc45fdc..98050e7 100644
--- a/compose.yaml
+++ b/compose.yaml
@@ -6,7 +6,7 @@ services:
             - "127.0.0.1:9023:9023/tcp"
         entrypoint: gcp-storage-emulator
         command: ["start",
-            "--host=0.0.0.0", "--port=9023", "--in-memory",
+            "--host=google_storage", "--port=9023", "--in-memory",
             "--default-bucket=localtesting_bucket" ]
 
     upload:

[agates4]

Thank you @MiltiadisKoutsokeras !