'MockRawResponse' object has no attribute 'raw_headers' error when using moto with s3fs

[nielstenboom]

Reporting Bug

Hi there!

I'm getting a 'MockRawResponse' object has no attribute 'raw_headers' error in my test using s3fs.

stacktrace:

result= s3fs_client.ls(f"somerandombucket/scans")

tests/test_ingestion.py:200: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/fsspec/asyn.py:118: in wrapper
    return sync(self.loop, func, *args, **kwargs)
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/fsspec/asyn.py:103: in sync
    raise return_result
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/fsspec/asyn.py:56: in _runner
    result[0] = await coro
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/s3fs/core.py:962: in _ls
    files = await self._lsdir(path, refresh, versions=versions)
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/s3fs/core.py:699: in _lsdir
    async for c in self._iterdir(
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/s3fs/core.py:749: in _iterdir
    async for i in it:
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/aiobotocore/paginate.py:30: in __anext__
    response = await self._make_request(current_kwargs)
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/aiobotocore/client.py:366: in _make_api_call
    http, parsed_response = await self._make_request(
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/aiobotocore/client.py:391: in _make_request
    return await self._endpoint.make_request(
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/aiobotocore/endpoint.py:97: in _send_request
    success_response, exception = await self._get_response(
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/aiobotocore/endpoint.py:139: in _get_response
    success_response, exception = await self._do_get_response(
../../../../Library/Caches/pypoetry/virtualenvs/klZ997Wd-py3.10/lib/python3.10/site-packages/aiobotocore/endpoint.py:191: in _do_get_response
    response_dict = await convert_to_response_dict(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

http_response = <botocore.awsrequest.AWSResponse object at 0x125b0abc0>, operation_model = OperationModel(name=ListObjectsV2)

    async def convert_to_response_dict(http_response, operation_model):
        """Convert an HTTP response object to a request dict.
    
        This converts the requests library's HTTP response object to
        a dictionary.
    
        :type http_response: botocore.vendored.requests.model.Response
        :param http_response: The HTTP response from an AWS service request.
    
        :rtype: dict
        :return: A response dictionary which will contain the following keys:
            * headers (dict)
            * status_code (int)
            * body (string or file-like object)
    
        """
        response_dict = {
            # botocore converts keys to str, so make sure that they are in
            # the expected case. See detailed discussion here:
            # https://github.com/aio-libs/aiobotocore/pull/116
            # aiohttp's CIMultiDict camel cases the headers :(
            'headers': HTTPHeaderDict(
                {
                    k.decode('utf-8').lower(): v.decode('utf-8')
>                   for k, v in http_response.raw.raw_headers
                }
            ),
            'status_code': http_response.status_code,
            'context': {
                'operation_name': operation_model.name,
            },
        }
E       AttributeError: 'MockRawResponse' object has no attribute 'raw_headers'

To reproduce

conftest.py

os.environ["AWS_DEFAULT_REGION"] = "eu-west-1"
os.environ["AWS_ACCESS_KEY_ID"] = "test"
os.environ["AWS_SECRET_ACCESS_KEY"] = "test"

BUCKET = "testbucket"

@pytest.fixture(scope="function")
def s3fs_client():
    with mock_s3():
        # create bucket
        client =  boto3.resource("s3")
        bucket = client.Bucket(BUCKET)
        bucket.create(CreateBucketConfiguration={"LocationConstraint": "eu-west-1"})


        client = s3fs.S3FileSystem()

        yield client

mytest_test.py

def test_my_thing(s3fs_client):
    # upload a file to the s3 bucket with boto3
    

     # then ls the bucket with the s3fs client, and the error should occur
     results = s3fs_client.ls(f"{BUCKET}")

I would've expected to see the uploaded file in the results variable and not receive an error 👍

Versions installed:

• moto - 4.2.3
• boto3 - 1.28.17
• botocore - 1.31.17
• s3fs - 2023.9.0

All installed and ran via poetry.

Not sure if this is anything you guys would like to cover since it happens in the s3fs package, if that's the case feel free to close this issue, but figured I'd want to let you guys know!

[bblommers]

Hi @nielstenboom! It has been reported before here, but it's a bug in aiobotocore actually.

There are some workarounds in their repo: aio-libs/aiobotocore#755

[nielstenboom]

Ah cool, thanks!

Sorry for the duplicate then, no issues popped up when I searched for this error!

[bblommers]

No problem! I just realized that s3fs actually uses Moto for their own tests, but using the MotoServer - so that's another workaround you could use.

See the docs here: http://docs.getmoto.org/en/latest/docs/server_mode.html

[nielstenboom]

Ahhh good to know, thanks!