Issue creating h5py wheel when running on M1 MacBook - trying to 'make run' in api directory #1683
Labels
type: bug
Something isn't working
Comments
|
Hi @mamsterla 👋, Thanks for reporting. |
|
That's what I do. Are you on a Mac M1?On Aug 5, 2024 12:31 AM, Felix Dittrich ***@***.***> wrote:
Hi @mamsterla 👋,
Thanks for reporting.
Unfortunately i wasn't able to reproduce your issue :/
``
cd api
make run
INFO: Will watch for changes in these directories: ['/app']
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO: Started reloader process [1] using WatchFiles
2024-08-05 07:28:19.085788: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-08-05 07:28:19.086951: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-08-05 07:28:19.103180: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-08-05 07:28:19.103199: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-08-05 07:28:19.103729: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-08-05 07:28:19.106522: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-08-05 07:28:19.106658: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-08-05 07:28:19.470437: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
INFO: Started server process [8]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: 172.18.0.1:57114 - "GET / HTTP/1.1" 404 Not Found
INFO: 172.18.0.1:57114 - "GET /favicon.ico HTTP/1.1" 404 Not Found
INFO: 172.18.0.1:57114 - "GET /docs HTTP/1.1" 200 OK
/usr/local/lib/python3.9/site-packages/pydantic/json_schema.py:2179: PydanticJsonSchemaWarning: Default value [File(PydanticUndefined)] is not JSON serializable; excluding default from JSON schema [non-serializable-default]
warnings.warn(message, PydanticJsonSchemaWarning)
INFO: 172.18.0.1:57114 - "GET /openapi.json HTTP/1.1" 200 OK
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Hi @mamsterla 👋, No i am on a linux machine (unfortunately i haven't a mac) but with If you try to install h5py standalone with pip on your machine does it work ? |
|
Direct install on my machine works only the build does not. Unfortunately there is no other way to run the API. I think this a Mac specific problem and you will need one to reproduce. Looks like the build needs a way to specify the HDF5_DIR for the MacOn Aug 5, 2024 11:04 PM, Felix Dittrich ***@***.***> wrote:
Hi @mamsterla 👋,
No i am on a linux machine (unfortunately i haven't a mac) but with make run the api is containerized via docker so this shouldn't make a difference.
If you try to install h5py standalone with pip on your machine does it work ?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
For what it's worth, I can reproduce @mamsterla's error on an m2 mac running 13.6.7. It appears that the OS does make a difference, from what I can see. `104.2 note: This error originates from a subprocess, and is likely not a problem with pip.
|
|
That is exactly what I saw. I know that it is an issue with building the container, but I could not figure how to pass in what it needed to complete the build.
… On Aug 13, 2024, at 4:32 PM, mmroden ***@***.***> wrote:
For what it's worth, I can reproduce @mamsterla <https://github.com/mamsterla>'s error on an m2 mac running 13.6.7.
It appears that the OS does make a difference, from what I can see.
`104.2 note: This error originates from a subprocess, and is likely not a problem with pip.
104.2 ERROR: Failed building wheel for h5py
104.2 Building wheel for langdetect (setup.py): started
104.5 Building wheel for langdetect (setup.py): finished with status 'done'
104.5 Created wheel for langdetect: filename=langdetect-1.0.9-py3-none-any.whl size=993221 sha256=7a17c910eac746923e68559f44c31c768cceedf182566fa4065cc2fdf74190fc
104.5 Stored in directory: /root/.cache/pip/wheels/d1/c1/d9/7e068de779d863bc8f8fc9467d85e25cfe47fa5051fff1a1bb
104.5 Building wheel for python-doctr (pyproject.toml): started
104.7 Building wheel for python-doctr (pyproject.toml): finished with status 'done'
104.7 Created wheel for python-doctr: filename=python_doctr-0.9.1a0-py3-none-any.whl size=299517 sha256=9aa29437f23f6d867fb00432855690150e1541d7309e9b80e36377428aa06449
104.7 Stored in directory: /root/.cache/pip/wheels/92/73/83/54e1a3424f829899ce15c8dce692a3cb5680f0ffd71524fa01
104.7 Successfully built langdetect python-doctr
104.7 Failed to build h5py
104.7 ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (h5py)
failed to solve: process "/bin/sh -c pip install --upgrade pip setuptools wheel && make lock && pip install -r /app/requirements.txt && pip cache purge && rm -rf /root/.cache/pip" did not complete successfully: exit code: 1
make: *** [run] Error 17`
—
Reply to this email directly, view it on GitHub <#1683 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABVHDQHREUEKUCMIAC5OJLZRKJRNAVCNFSM6AAAAABL5QNCXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBXGM4DOMZVG4>.
You are receiving this because you were mentioned.
|
|
@mamsterla @mmroden The key difference is that we don't need to build This is definitely not recommended |
|
Felix:
Sorry for taking so long to try this. I did and got the following error (Rosetta is being used because it thinks this is x86_64):
13.06 2024-08-21 18:40:59 (10.8 MB/s) - 'Miniconda3-latest-Linux-x86_64.sh' saved [146836934/146836934]
13.06
13.07 PREFIX=/root/miniconda3
13.43 Unpacking payload ...
13.44 rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
13.44 Miniconda3-latest-Linux-x86_64.sh: line 442: 43 Exit 141 extract_range $boundary1 $boundary2
13.44 44 Trace/breakpoint trap | CONDA_QUIET="$BATCH" "$CONDA_EXEC" constructor --extract-tarball --prefix "$PREFIX"
------
failed to solve: process "/bin/sh -c wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && bash Miniconda3-latest-Linux-x86_64.sh -b -p /root/miniconda3 && rm -f Miniconda3-latest-Linux-x86_64.sh" did not complete successfully: exit code: 133
make: *** [run] Error 17
… On Aug 14, 2024, at 3:14 AM, Felix Dittrich ***@***.***> wrote:
@mamsterla <https://github.com/mamsterla> @mmroden <https://github.com/mmroden>
Could you try to replace the Dockerfile with the following (dirty try) and run cd api && make run again ?
FROM nvidia/cuda:12.1.0-base-ubuntu22.04
WORKDIR /app
# Set up PATH for Conda
ENV PATH="/root/miniconda3/bin:${PATH}"
RUN apt-get -y update \
&& apt-get install --no-install-recommends git ffmpeg libsm6 libxext6 make wget -y \
&& apt-get autoremove -y \
&& rm -rf /var/lib/apt/lists/*
# Copy project files
COPY app /app/app
# Install Miniconda
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& bash Miniconda3-latest-Linux-x86_64.sh -b -p /root/miniconda3 \
&& rm -f Miniconda3-latest-Linux-x86_64.sh
# Initialize Conda and create environment
RUN /root/miniconda3/bin/conda init bash \
&& /bin/bash -c "source /root/miniconda3/etc/profile.d/conda.sh && conda create -n doctr-api python=3.9 -y"
# Activate environment and install dependencies
RUN /bin/bash -c "source /root/miniconda3/etc/profile.d/conda.sh && conda activate doctr-api && conda install h5py -y"
RUN /bin/bash -c "source /root/miniconda3/etc/profile.d/conda.sh && conda activate doctr-api && pip install ***@***.***+https://github.com/mindee/doctr.git fastapi uvicorn python-multipart"
# Create the entrypoint script directly within the Dockerfile
RUN echo '#!/bin/bash' > /entrypoint.sh && \
echo 'source /root/miniconda3/etc/profile.d/conda.sh' >> /entrypoint.sh && \
echo 'conda activate doctr-api' >> /entrypoint.sh && \
echo 'exec "$@"' >> /entrypoint.sh && \
chmod +x /entrypoint.sh
# Set the entrypoint to use the script
ENTRYPOINT ["/entrypoint.sh"]
The key difference is that we don't need to build h5py here we make use of the already compiled binary from anaconda
This is definitely not recommended
—
Reply to this email directly, view it on GitHub <#1683 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABVHDSR7JRT3XFROTM3HMDZRMUXLAVCNFSM6AAAAABL5QNCXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBYGM3TAMRTGQ>.
You are receiving this because you were mentioned.
|
|
We probably need to use ARM64 to make it cleaner on Mac M1 chips (all Apple silicon).
…-MA
On Aug 14, 2024, at 3:14 AM, Felix Dittrich ***@***.***> wrote:
@mamsterla <https://github.com/mamsterla> @mmroden <https://github.com/mmroden>
Could you try to replace the Dockerfile with the following (dirty try) and run cd api && make run again ?
FROM nvidia/cuda:12.1.0-base-ubuntu22.04
WORKDIR /app
# Set up PATH for Conda
ENV PATH="/root/miniconda3/bin:${PATH}"
RUN apt-get -y update \
&& apt-get install --no-install-recommends git ffmpeg libsm6 libxext6 make wget -y \
&& apt-get autoremove -y \
&& rm -rf /var/lib/apt/lists/*
# Copy project files
COPY app /app/app
# Install Miniconda
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& bash Miniconda3-latest-Linux-x86_64.sh -b -p /root/miniconda3 \
&& rm -f Miniconda3-latest-Linux-x86_64.sh
# Initialize Conda and create environment
RUN /root/miniconda3/bin/conda init bash \
&& /bin/bash -c "source /root/miniconda3/etc/profile.d/conda.sh && conda create -n doctr-api python=3.9 -y"
# Activate environment and install dependencies
RUN /bin/bash -c "source /root/miniconda3/etc/profile.d/conda.sh && conda activate doctr-api && conda install h5py -y"
RUN /bin/bash -c "source /root/miniconda3/etc/profile.d/conda.sh && conda activate doctr-api && pip install ***@***.***+https://github.com/mindee/doctr.git fastapi uvicorn python-multipart"
# Create the entrypoint script directly within the Dockerfile
RUN echo '#!/bin/bash' > /entrypoint.sh && \
echo 'source /root/miniconda3/etc/profile.d/conda.sh' >> /entrypoint.sh && \
echo 'conda activate doctr-api' >> /entrypoint.sh && \
echo 'exec "$@"' >> /entrypoint.sh && \
chmod +x /entrypoint.sh
# Set the entrypoint to use the script
ENTRYPOINT ["/entrypoint.sh"]
The key difference is that we don't need to build h5py here we make use of the already compiled binary from anaconda
This is definitely not recommended
—
Reply to this email directly, view it on GitHub <#1683 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABVHDSR7JRT3XFROTM3HMDZRMUXLAVCNFSM6AAAAABL5QNCXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBYGM3TAMRTGQ>.
You are receiving this because you were mentioned.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug description
I do a git clone of of the repo and then cd in the api directory and then type 'make run' - I get the following output. I have tried many ways to define HDF5_DIR which do not seem to pass through to the build process:
.....
54.26 running build_ext
54.26 Building h5py requires pkg-config unless the HDF5 path is explicitly specified using the environment variable HDF5_DIR. For more information and details, see https://docs.h5py.org/en/stable/build.html#custom-installation
54.26 error: pkg-config probably not installed: FileNotFoundError(2, 'No such file or directory')
54.26 [end of output]
54.26
54.26 note: This error originates from a subprocess, and is likely not a problem with pip.
54.26 ERROR: Failed building wheel for h5py
54.26 Building wheel for langdetect (setup.py): started
54.56 Building wheel for langdetect (setup.py): finished with status 'done'
54.56 Created wheel for langdetect: filename=langdetect-1.0.9-py3-none-any.whl size=993222 sha256=50e7a1af9ba0ef70e13edfcfc87566758157cf2f97746d1707b9bb01bb286397
54.56 Stored in directory: /root/.cache/pip/wheels/d1/c1/d9/7e068de779d863bc8f8fc9467d85e25cfe47fa5051fff1a1bb
54.57 Building wheel for python-doctr (pyproject.toml): started
54.78 Building wheel for python-doctr (pyproject.toml): finished with status 'done'
54.78 Created wheel for python-doctr: filename=python_doctr-0.9.0a0-py3-none-any.whl size=299068 sha256=80c9749bc081db15f718107d6df35d9c5f6f946fd85f7777b2b506d0be77bef3
54.78 Stored in directory: /root/.cache/pip/wheels/4a/b6/9f/1a0c3bf05bcc97eec54dbf7529ae100ea6e1dbff8a74675bd9
54.78 Successfully built langdetect python-doctr
54.78 Failed to build h5py
54.78 ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (h5py)
failed to solve: process "/bin/sh -c pip install --upgrade pip setuptools wheel && make lock && pip install -r /app/requirements.txt && pip cache purge && rm -rf /root/.cache/pip" did not complete successfully: exit code: 1
make: *** [run] Error 17
Code snippet to reproduce the bug
cd api; make run
Error traceback
Michaels-MacBook-Pro:api mamsterla$ make run
docker compose up -d --build
WARN[0000] /Users/mamsterla/projects/doctr/api/docker-compose.yml: the attribute
versionis obsolete, it will be ignored, please remove it to avoid potential confusion[+] Building 55.9s (10/11) docker:desktop-linux
=> [web internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 672B 0.0s
=> [web internal] load metadata for docker.io/tiangolo/uvicorn-gunicorn-fastapi:python3.9-slim 0.4s
=> [web internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [web 1/7] FROM docker.io/tiangolo/uvicorn-gunicorn-fastapi:python3.9-slim@sha256:4b52dafa4cf3560a06e8744adae01d1a2cbac771af2f 0.0s
=> [web internal] load build context 0.0s
=> => transferring context: 442B 0.0s
=> CACHED [web 2/7] WORKDIR /app 0.0s
=> CACHED [web 3/7] RUN apt-get update && apt-get install --no-install-recommends git ffmpeg libsm6 libxext6 make -y && 0.0s
=> CACHED [web 4/7] COPY pyproject.toml /app/pyproject.toml 0.0s
=> CACHED [web 5/7] COPY Makefile /app/Makefile 0.0s
=> ERROR [web 6/7] RUN pip install --upgrade pip setuptools wheel && make lock && pip install -r /app/requirements.txt 55.4s
failed to solve: process "/bin/sh -c pip install --upgrade pip setuptools wheel && make lock && pip install -r /app/requirements.txt && pip cache purge && rm -rf /root/.cache/pip" did not complete successfully: exit code: 1
make: *** [run] Error 17
Environment
MacBook Pro M1 - Sonoma 14.5
Pyenv installed - configured with Python 3.10.14
Deep Learning backend
N/A
The text was updated successfully, but these errors were encountered: