Mac OS wheels are not compatible with Monterey version 12.1

[w-tim]

Please provide all mandatory information!

Describe the bug (mandatory)

I have used your PyMuPDF for one year! Thanks for your effort.

I upgraded system to MacOS 12.1 today, then something about Xcode upgraded automatically. Simultaneously, I upgraded PyMuPDF from 1.19.2 to 1.19.3. After that ,I run my routine self-design app but failed. Showing Fitz error. Considering it is working properly on another Intel MBP.(all libraries are the same,incl versions), I supposed maybe 1.19.3 was incompatible for M1 MBP. I tried to downgrade or remove but failed, so deleted Fitz & PyMuPDF library manually and tried to reinstall it.

Now, always failed and showing below. (via Pycharm and terminal, all failed)

Collecting PyMuPDF==1.19.3
Using cached PyMuPDF-1.19.3.tar.gz (2.3 MB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Using legacy 'setup.py install' for PyMuPDF, since package 'wheel' is not installed.
Installing collected packages: PyMuPDF
Running setup.py install for PyMuPDF: started
Running setup.py install for PyMuPDF: finished with status 'error'

ERROR: Command errored out with exit status 1:
 command: /usr/local/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/_h/3svyvztx0vs_x29qml4vw29r0000gn/T/pip-install-qha8k814/pymupdf_58dd1a65f67141f38d5812b54be8f029/setup.py'"'"'; __file__='"'"'/private/var/folders/_h/3svyvztx0vs_x29qml4vw29r0000gn/T/pip-install-qha8k814/pymupdf_58dd1a65f67141f38d5812b54be8f029/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/_h/3svyvztx0vs_x29qml4vw29r0000gn/T/pip-record-uwufdbgk/install-record.txt --single-version-externally-managed --compile --install-headers /Library/Frameworks/Python.framework/Versions/3.9/include/python3.9/PyMuPDF
     cwd: /private/var/folders/_h/3svyvztx0vs_x29qml4vw29r0000gn/T/pip-install-qha8k814/pymupdf_58dd1a65f67141f38d5812b54be8f029/
Complete output (24 lines):
running install
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running build
running build_py
creating build
creating build/lib.macosx-10.9-universal2-3.9
creating build/lib.macosx-10.9-universal2-3.9/fitz
copying fitz/__init__.py -> build/lib.macosx-10.9-universal2-3.9/fitz
copying fitz/fitz.py -> build/lib.macosx-10.9-universal2-3.9/fitz
copying fitz/utils.py -> build/lib.macosx-10.9-universal2-3.9/fitz
copying fitz/__main__.py -> build/lib.macosx-10.9-universal2-3.9/fitz
warning: build_py: byte-compiling is disabled, skipping.

running build_ext
building 'fitz._fitz' extension
creating build/temp.macosx-10.9-universal2-3.9
creating build/temp.macosx-10.9-universal2-3.9/fitz
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/usr/local/include/mupdf -I/usr/local/include -Imupdf/thirdparty/freetype/include -I/usr/local/include/freetype2 -I/usr/X11R6/include/freetype2 -I/opt/homebrew/include -I/opt/homebrew/include/mupdf -I/opt/homebrew/include/freetype2 -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c fitz/fitz_wrap.c -o build/temp.macosx-10.9-universal2-3.9/fitz/fitz_wrap.o
fitz/fitz_wrap.c:2755:10: fatal error: 'fitz.h' file not found
#include <fitz.h>
         ^~~~~~~~
1 error generated.
error: command '/usr/bin/gcc' failed with exit code 1
----------------------------------------

ERROR: Command errored out with exit status 1: /usr/local/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/_h/3svyvztx0vs_x29qml4vw29r0000gn/T/pip-install-qha8k814/pymupdf_58dd1a65f67141f38d5812b54be8f029/setup.py'"'"'; file='"'"'/private/var/folders/_h/3svyvztx0vs_x29qml4vw29r0000gn/T/pip-install-qha8k814/pymupdf_58dd1a65f67141f38d5812b54be8f029/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/_h/3svyvztx0vs_x29qml4vw29r0000gn/T/pip-record-uwufdbgk/install-record.txt --single-version-externally-managed --compile --install-headers /Library/Frameworks/Python.framework/Versions/3.9/include/python3.9/PyMuPDF Check the logs for full command output.

I tried to install other lib, successful. So maybe it is something about Fitz or PyMuPDF?

To Reproduce (mandatory)

install, always failed

Expected behavior (optional)

install successfully and no error on loading.

Screenshots (optional)

described as above

Your configuration (mandatory)

• MacOS 12.1
• Python 3.9.9
• PyMuPDF 1.19.3, via PyCharm

For example, the output of print(sys.version, "\n", sys.platform, "\n", fitz.__doc__) would be sufficient (for the first two bullets).

Additional context (optional)

[w-tim]

I ignored importing PyMuPDF, but app not working as well. I found the similar error on Numpy. Because the first time I found error is that it showed PyMuPDF (or maybe Fitz, I forgot it) xxxxx.

Original error was: dlopen(/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so, 0x0002): tried: '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/lib/_multiarray_umath.cpython-39-darwin.so' (no such file)

[JorjMcKie]

I am not sure about what MacOS 12 means, but if it is not the Intel version (but that M1 stuff), there exist no wheels for you!
There can nothing be done about it in that case. You must build MuPDF, then PyMuPDF from sources.

This is what PIP on your machine is trying to do - and fails because it does not find MuPDF.

[JorjMcKie]

I just confirmed that the OSX wheels on PyPI work fine: they run on a Mac OSX 11 machine ...

Please confirm my assumption, so I can close the issue.
You may find help here.

[w-tim]

Thanks for your quick reply. I am also testing now.

I am pretty sure that PyMuPDF works fine find under MacOS Monterey 12.0.1, for both M1 mbp and Intel mbp. Because I use this module everyday. Errors happened after upgrading to 12.1. (Intel works fine, M1 mbp failed)

Actually I use Tkinter app built by apple script (programed by python). I found it is more complex than I thought.
I reinstalled the whole python and every libs one by one. All successful except PyMuPDF. So I # this module now and launched my Tkinter app. Launched successfully but button width, window width, color all changed. So I am afraid something should be changed under new system.(All normal under intel mbp)

[JorjMcKie]

I see ... and PyMuPDF installation failed? Or did it install and running it failed?

[w-tim]

Failed to install. I tried many versions. From 1.18.x to 1.19.3, all failed.

[JorjMcKie]

Ok. Then I would try an installation from sources ...: first MuPDF, then PyMuPDF.
You also upgraded PIP to the newest version? Sometimes helps.
Anyway, PIP looks at the wheel file platform tag and decides that no wheel would work on your machine, so tries a source installation.

[w-tim]

Yes, my pip is the latest, 21.3.1
I even test it based on different Python version (3.9.7 & 3.9.9), same result.

[JorjMcKie]

Do you know the platform tag for that new MacOS? You could try a dirty hack: download the wheel and rename it accordingly ... 😉

[w-tim]

Sorry. I am not a professional programmer. Even not sure how to build from source. I install all libraries from Pycharm pip only...

[JorjMcKie]

Well, in that case ... I am out of good advice.

I would like to close and postpone this issue until the cibuildwheel package and the GitHub Actions service support this Mac version.
I am using these features for building wheels on the various platforms.

[w-tim]

Thank you so much. No problem, you could close this issue now. Thanks again for creating this awesome library.

[JorjMcKie]

What I meant was this:

• download this file.
• rename the file PyMuPDF-1.19.3-cp39-cp39-macosx_10_9_x86_64.whl to PyMuPDF-1.19.3-cp39-cp39-macosx_11_0_x86_64.whl.
• then execute pip install PyMuPDF-1.19.3-cp39-cp39-macosx_11_0_x86_64.whl

When looking at how numpy wheels are named, this trick may work. But it is dirty as I said ...

Or maybe instead 11_0 try 10_10 or 12_0 or 12_1

[w-tim]

Thanks. However,

ERROR: PyMuPDF-1.19.3-cp39-cp39-macosx_12_1_x86_64.whl is not a supported wheel on this platform.
ERROR: PyMuPDF-1.19.3-cp39-cp39-macosx_12_0_x86_64.whl is not a supported wheel on this platform.
ERROR: PyMuPDF-1.19.3-cp39-cp39-macosx_11_0_x86_64.whl is not a supported wheel on this platform.

[kalaomer]

I have build my own wheel on Docker for python:3.8.12-bullseye image from my M1 device.

Here is my path to build it:

Enter docker image with this:
docker run -it python:3.8.12-bullseye bash

Then run this commands to install dependencies

apt update

apt install python3-swiglpk swig libjbig-dev libjbig-dev libjbig2dec0-dev libjbig2dec0 libjbig-dev jbig2dec libgumbo-dev libgumbo1 python3-gumbo libharfbuzz-dev libharfbuzz-bin libghc-gi-harfbuzz-dev libmujs-dev mujs python3-mujson libmujs1

After run this

wget https://mupdf.com/downloads/archive/mupdf-1.19.0-source.tar.gz
tar -zxvf mupdf-1.19.0-source.tar.gz

git clone https://github.com/pymupdf/PyMuPDF.git --depth 1

cd mupdf-1.19.0-source
# replace files in mupdf source
cp ../PyMuPDF/fitz/_config.h include/mupdf/fitz/config.h

export CFLAGS="-fPIC"
# install some prerequirement
apt install pkg-config python-dev

make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no HAVE_LEPTONICA=no HAVE_TESSERACT=no -j 3 prefix=/usr/local
make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no HAVE_LEPTONICA=no HAVE_TESSERACT=no -j 3 prefix=/usr/local install

cd ..

cd PyMuPDF

python setup.py build
python setup.py install
python setup.py bdist_wheel

Note: -j parameter is optimized with my docker config. Please change this with your cpu count for dedicated to Docker.

After this, go to dist folder and store wheel file. Also I uploaded my wheel you can download it from here.
PyMuPDF-1.19.3-cp38-cp38-linux_aarch64.whl.zip

[JorjMcKie]

Thanks a lot @kalaomer !

[bushig]

Here is working Dockerfile example that can be used as base image (kudos to @kalaomer ):

# FIXME: you are free to change python version
FROM python:3.6

RUN apt update && apt install -y python3-swiglpk swig libjbig-dev libjbig-dev libjbig2dec0-dev libjbig2dec0 libjbig-dev jbig2dec libgumbo-dev libgumbo1 python3-gumbo libharfbuzz-dev libharfbuzz-bin libghc-gi-harfbuzz-dev libmujs-dev mujs python3-mujson libmujs1
RUN apt install -y pkg-config python-dev

WORKDIR /src/mupdf
# FIXME: you can change mupdf version
RUN wget https://mupdf.com/downloads/archive/mupdf-1.19.0-source.tar.gz
RUN tar -zxvf mupdf-1.19.0-source.tar.gz

# FIXME: here you can change pymupdf version by setting git branch
RUN git clone https://github.com/pymupdf/PyMuPDF.git --depth 1 --branch 1.19.4

RUN ls -lah
# replace files in mupdf source
RUN cp PyMuPDF/fitz/_config.h mupdf-1.19.0-source/include/mupdf/fitz/config.h

WORKDIR /src/mupdf/mupdf-1.19.0-source
RUN ls -lah

ENV CFLAGS="-fPIC"

RUN make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no HAVE_LEPTONICA=no HAVE_TESSERACT=no -j 4 prefix=/usr/local
RUN make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no HAVE_LEPTONICA=no HAVE_TESSERACT=no -j 4 prefix=/usr/local install

WORKDIR /src/mupdf/PyMuPDF

RUN python setup.py build
RUN python setup.py install
RUN python setup.py bdist_wheel

[w-tim]

I resolved it. Learnt it from Stackoverflow.
Install Home-brew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Then
brew install mupdf swig freetype
pip3 install https://github.com/pymupdf/PyMuPDF/archive/master.tar.gz

Done