Building universal binaries (x86_64+arm64) on macOS

[nviolini]

The error occurs when including the ncrystal.h file in a newly-written Vitess module - YES, we're trying to include NCrystal in the package! -

I get an error due to the library having architecture arm64, where the required one for my Mac is x86_64.
I tried to rebuild the library using the cmake option for x86_64 or a universal library, too, but still the resulting file is arm64.
cmake . -DCMAKE_INSTALL_PREFIX=/../ncrystal_package -DCMAKE_OSX_ARCHITECTURES="x86_64;arm64"

So to me it seems that NCrystal is failing to build the x86_64 arch.

Can you assist resolving this issue?

@tkittel

[tkittel]

That sounds weird, you should not need to mess around with MAKE_OSX_ARCHITECTURES unless you are cross compiling, I would have thought.

How did you install NCrystal? From PyPI or conda-forge packages?

[tkittel]

Btw., sounds great about vitess+ncrystal plans :-)

[tkittel]

Btw., I would have thought M3 would imply arm64, not x86_64?

[nviolini]

I cloned the repo and installed manually with cmake.

Yeah, sorry, my Mac has arm64 not x86_64. The Vitess build is looking for both.
Even if I deactivated the universal option, I still get an error due to the libNCrystal.3.9.7.dylib library being arm64, when Vitess requires also x86_64.
Can Ncrystal build x86_64?

[tkittel]

Yes of course it can, we are routinely using it on mac in both environments. We also have both PyPI and conda-forge packages available for both:

• https://pypi.org/project/ncrystal/#files
• https://anaconda.org/conda-forge/ncrystal/files

So if you are either in a python or conda environment, you should simply be able to install with a single command without any need to mess around with manual cmake commands.

However if you need to, it simply sounds to me like you have a bit of a mixed and unusual development environment, so your problem most likely comes from that.

It is hard for me to debug your environment for you though, I don't have any experience with osx and universal binaries. But I suspect your issue would be the same if you try to build other projects than NCrystal, since NCrystal's CMake code really doesn't have many platform-specific hacks in this regard.

cheers,
Thomas

[nviolini]

Alright, thanks for your help so far. We will investigate locally.

[willend]

@nviolini what toolchain are you using to build ncrystal (i.e. "bare macOS dev tools + cmake", a conda env, homebrew or something else)?

I tried myself using my "usual" conda setup - which however turned out to have an arm64-only libc.

I'have instead now installed CMake to /Applications and unloaded any "non-vanilla" systems and simply run CMake + developer tools - and this seems to work:

hostname:build username$ /Applications/CMake.app/Contents/bin/cmake  .. -DCMAKE_INSTALL_PREFIX=/../ncrystal_package -DCMAKE_OSX_ARCHITECTURES="x86_64;arm64"
-- The CXX compiler identification is AppleClang 16.0.0.16000026
-- The C compiler identification is AppleClang 16.0.0.16000026
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- NCrystal-cfg: NCRYSTAL_ENABLE_EXAMPLES=OFF
-- NCrystal-cfg: NCRYSTAL_ENABLE_GEANT4=OFF
-- NCrystal-cfg: NCRYSTAL_ENABLE_PYTHON=ON
-- NCrystal-cfg: NCRYSTAL_ENABLE_DATA=ON
-- NCrystal-cfg: NCRYSTAL_ENABLE_SETUPSH=OFF
-- NCrystal-cfg: NCRYSTAL_MODIFY_RPATH=ON
-- NCrystal-cfg: NCRYSTAL_ENABLE_DYNLOAD=IFAVAILABLE
-- NCrystal-cfg: NCRYSTAL_SKIP_PYMODINST=OFF
-- NCrystal-cfg: NCRYSTAL_BUILD_STRICT=OFF
-- NCrystal-cfg: NCRYSTAL_ENABLE_CPACK=OFF
-- NCrystal-cfg: NCRYSTAL_QUIET=OFF
-- NCrystal-cfg: NCRYSTAL_SKIP_INSTALL=OFF
-- NCrystal-cfg: NCRYSTAL_ENABLE_SOVERSION=ON
-- NCrystal-cfg: NCRYSTAL_BUILTIN_PLUGINS=
-- NCrystal-cfg: NCRYSTAL_NAMESPACE=
-- Configuring done (2.8s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/username/Projects/ncrystal/build

hostname:build username$ make -j12 
[  1%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCAtomDBExtender.cc.o
[  2%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCAtomData.cc.o
[  3%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCAtomUtils.cc.o
[  4%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCCfgManip.cc.o
[  5%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCCfgVars.cc.o
[  6%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCCompositionUtils.cc.o
[  7%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCDefs.cc.o
[  8%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCDump.cc.o
[  9%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCCfgTypes.cc.o
[ 10%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCBkgdExtCurve.cc.o
[ 11%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCDebyeMSD.cc.o
[ 12%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCDataSources.cc.o
[ 13%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCDynInfoUtils.cc.o
[ 14%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCDynLoader.cc.o
[ 15%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCElIncScatter.cc.o
[ 16%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCElIncXS.cc.o
[ 17%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCEqRefl.cc.o
[ 18%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFact.cc.o
[ 19%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCException.cc.o
[ 20%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFactImpl.cc.o
[ 21%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFactRequests.cc.o
[ 22%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFactTypes.cc.o
[ 23%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFactoryUtils.cc.o
[ 24%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFactory_NCMAT.cc.o
[ 25%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFastConvolve.cc.o
[ 26%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFileUtils.cc.o
[ 27%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFillHKL.cc.o
[ 28%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFmt.cc.o
[ 29%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFreeGas.cc.o
[ 30%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCFreeGasUtils.cc.o
[ 31%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCGasMixFact.cc.o
[ 32%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCGasMixUtils.cc.o
[ 33%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCGaussMos.cc.o
[ 34%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCGaussOnSphere.cc.o
[ 35%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCInfo.cc.o
[ 36%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCInfoBuilder.cc.o
[ 37%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCIofQHelper.cc.o
[ 38%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCKinUtils.cc.o
[ 39%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCLCBragg.cc.o
[ 40%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCLCRefModels.cc.o
[ 41%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCLCUtils.cc.o
[ 42%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCLatticeUtils.cc.o
[ 43%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCLauLazyFact.cc.o
[ 44%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCLazy.cc.o
[ 45%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCLoadNCMAT.cc.o
[ 46%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCMatCfg.cc.o
[ 47%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCMath.cc.o
[ 48%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCMatrix.cc.o
[ 50%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCMem.cc.o
[ 51%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCNCMATData.cc.o
[ 52%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCOrientUtils.cc.o
[ 53%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCPCBragg.cc.o
[ 54%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCParseNCMAT.cc.o
[ 55%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCPlaneProvider.cc.o
[ 56%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCPluginMgmt.cc.o
[ 57%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCPointwiseDist.cc.o
[ 58%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCProc.cc.o
[ 59%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCProcImpl.cc.o
[ 60%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCQuickFact.cc.o
[ 61%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCRNG.cc.o
[ 62%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCRandUtils.cc.o
[ 63%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCRomberg.cc.o
[ 64%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCRotMatrix.cc.o
[ 65%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABData.cc.o
[ 66%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABEval.cc.o
[ 67%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABExtender.cc.o
[ 68%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABFactory.cc.o
[ 69%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABIntegrator.cc.o
[ 70%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABSampler.cc.o
[ 71%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABSamplerModels.cc.o
[ 72%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABScatter.cc.o
[ 73%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABUCN.cc.o
[ 74%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABUtils.cc.o
[ 75%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSABXSProvider.cc.o
[ 76%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSANSFact.cc.o
[ 77%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSANSSphScat.cc.o
[ 78%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSANSUtils.cc.o
[ 79%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSCBragg.cc.o
[ 80%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSCOrientation.cc.o
[ 81%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCScatKnlData.cc.o
[ 82%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCSpline.cc.o
[ 83%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCStdAbsFact.cc.o
[ 84%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCStdMPScatFact.cc.o
[ 85%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCStdScatFact.cc.o
[ 86%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCStrView.cc.o
[ 87%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCString.cc.o
[ 88%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCTDProd.cc.o
[ 89%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCTextData.cc.o
[ 90%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCTypes.cc.o
[ 91%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCVDOSEval.cc.o
[ 92%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCVDOSGn.cc.o
[ 93%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCVDOSToScatKnl.cc.o
[ 94%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCVector.cc.o
[ 95%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/NCVersion.cc.o
[ 96%] Building CXX object CMakeFiles/NCrystal.dir/ncrystal_core/src/ncrystal.cc.o
[ 97%] Linking CXX shared library libNCrystal.dylib
[100%] Built target NCrystal
hostname:build username$ file libNCrystal.dylib 
libNCrystal.dylib: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [arm64]
libNCrystal.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64
libNCrystal.dylib (for architecture arm64):	Mach-O 64-bit dynamically linked shared library arm64

[nviolini]

@willend Thanks for investigating this.
As I'm not on a conda env, I used cmake to build ncrystal:

cmake . -DCMAKE_INSTALL_PREFIX=/.../ncrystal_package

and tried also with the option -DCMAKE_OSX_ARCHITECTURES="x86_64;arm64"

but in my case the library is still arm64 only:

file libNCrystal.dylib libNCrystal.dylib: Mach-O 64-bit dynamically linked shared library arm64

Anyways, for now I compile Vitess not for universal, but for arm64 only and it seems at least to not prompt errors anymore.
Not sure how stable this workaround is. And at the moment, the ncrystal code is not actually executed. I'll investigate further.

[tkittel]

@nviolini Do you by any chance have any sort of homebrew/ports/conda/... environment active on your machine? How did you install CMake? Same way as Peter?

What does which cmake say?

Perhaps if you post your cmake output Peter will be able to spot something useful?

[nviolini]

No conda env of any kind on my Mac.
I believe the CMake comes with the distributed version from our ITS group, but I will check with them.
I don't have memory of having installed it myself. If so, it's too long ago to remember how.
which cmake
gives
/usr/local/bin/cmake

[nviolini]

cmake . -DCMAKE_INSTALL_PREFIX=/.../ncrystal_package
-- The CXX compiler identification is AppleClang 16.0.0.16000026
-- The C compiler identification is AppleClang 16.0.0.16000026
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test ncrystal_compiler_supports_nomatherrno_flag
-- Performing Test ncrystal_compiler_supports_nomatherrno_flag - Success
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Thread support will be enabled.
-- NCrystal-cfg: NCRYSTAL_ENABLE_EXAMPLES=OFF
-- NCrystal-cfg: NCRYSTAL_ENABLE_GEANT4=OFF
-- NCrystal-cfg: NCRYSTAL_ENABLE_PYTHON=ON
-- NCrystal-cfg: NCRYSTAL_ENABLE_DATA=ON
-- NCrystal-cfg: NCRYSTAL_ENABLE_SETUPSH=OFF
-- NCrystal-cfg: NCRYSTAL_MODIFY_RPATH=ON
-- NCrystal-cfg: NCRYSTAL_ENABLE_DYNLOAD=IFAVAILABLE
-- NCrystal-cfg: NCRYSTAL_SKIP_PYMODINST=OFF
-- NCrystal-cfg: NCRYSTAL_BUILD_STRICT=OFF
-- NCrystal-cfg: NCRYSTAL_ENABLE_CPACK=OFF
-- NCrystal-cfg: NCRYSTAL_QUIET=OFF
-- NCrystal-cfg: NCRYSTAL_SKIP_INSTALL=OFF
-- NCrystal-cfg: NCRYSTAL_ENABLE_SOVERSION=ON
-- NCrystal-cfg: NCRYSTAL_ENABLE_THREADS=IFAVAILABLE
-- NCrystal-cfg: NCRYSTAL_BUILTIN_PLUGINS=
-- NCrystal-cfg: NCRYSTAL_NAMESPACE=
-- Configuring done (0.9s)
-- Generating done (0.0s)
-- Build files have been written to: /.../ncrystal

[tkittel]

/usr/local/bin/cmake sounds a bit like homebrew or something like that. Do you have the command "brew" available in your shell?

Perhaps you could try to download a fresh CMake application bundle, and then invoke it with the full path /Applications/CMake.app/Contents/bin/cmake like Peter did?

[nviolini]

Ok, it seems to have solved now.
I cleaned first.
make clean
then removed the files
rm -rf CMakeCache.txt CMakeFiles

then built with cmake giving the full path to cmake:
/usr/local/bin/cmake . -DCMAKE_INSTALL_PREFIX=/.../ncrystal -DCMAKE_OSX_ARCHITECTURES="x86_64;arm64"

and finally
make -j12

file libNCrystal.dylib libNCrystal.dylib: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [arm64] libNCrystal.dylib (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64 libNCrystal.dylib (for architecture arm64): Mach-O 64-bit dynamically linked shared library arm64

[nviolini]

I didn't install cmake again, but I think giving the full path was the key.

[tkittel]

Great! :-)

But I am pretty sure that it made no difference that you gave the full path.

However, what IS always important with CMake (especially if something is not working) is to ALWAYS clear the entire build directory and try again. Otherwise you have no idea what you are debugging. In fact, I would recommend doing a proper rm -rf ./mycmakebuilddir and not just trying to surgically remove parts like you were doing (of course, it means that you should use out-of-source buld dirs with CMake, which is also a general recommendation).

I am going to rename this issue and convert it into a discussion, to serve as documentation for anyone else wanting to build universal binaries on OSX.