Failing cross compilation aarch64-unknown-linux-musl

[zonyitoo]

  -- Configuring done
  -- Generating /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build
     Called from: [1]	/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/CMakeLists.txt
  -- Generating done
  -- Build files have been written to: /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build
  running: "cmake" "--build" "." "--target" "snmallocshim-1mib-rust" "--config" "Release" "--"
  /usr/local/bin/cmake -S/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc -B/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build --check-build-system CMakeFiles/Makefile.cmake 0
  /usr/bin/make  -f CMakeFiles/Makefile2 snmallocshim-1mib-rust
  make[1]: Entering directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  /usr/local/bin/cmake -S/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc -B/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build --check-build-system CMakeFiles/Makefile.cmake 0
  /usr/local/bin/cmake -E cmake_progress_start /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles 1
  /usr/bin/make  -f CMakeFiles/Makefile2 CMakeFiles/snmallocshim-1mib-rust.dir/all
  make[2]: Entering directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  /usr/bin/make  -f CMakeFiles/snmallocshim-1mib-rust.dir/build.make CMakeFiles/snmallocshim-1mib-rust.dir/depend
  make[3]: Entering directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  cd /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc /cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles/snmallocshim-1mib-rust.dir/DependInfo.cmake --color=
  Dependee "/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles/snmallocshim-1mib-rust.dir/DependInfo.cmake" is newer than depender "/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles/snmallocshim-1mib-rust.dir/depend.internal".
  Dependee "/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles/snmallocshim-1mib-rust.dir/depend.internal".
  Scanning dependencies of target snmallocshim-1mib-rust
  make[3]: Leaving directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  /usr/bin/make  -f CMakeFiles/snmallocshim-1mib-rust.dir/build.make CMakeFiles/snmallocshim-1mib-rust.dir/build
  make[3]: Entering directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  [100%] Building CXX object CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o
  /usr/local/bin/aarch64-linux-musl-g++  -DSNMALLOC_EXPORT="__attribute__((visibility(\"default\")))" -I/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/src  -ffunction-sections -fdata-sections -fPIC -O3 -DNDEBUG -fvisibility=hidden   -Wall -Wextra -Werror -Wundef -fno-exceptions -fno-rtti -g -fomit-frame-pointer -ftls-model=initial-exec -mcx16 -std=gnu++1z -o CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o -c /cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/src/override/rust.cc
  CMakeFiles/snmallocshim-1mib-rust.dir/build.make:85: recipe for target 'CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o' failed
  make[3]: Leaving directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  CMakeFiles/Makefile2:1189: recipe for target 'CMakeFiles/snmallocshim-1mib-rust.dir/all' failed
  make[2]: Leaving directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  CMakeFiles/Makefile2:1196: recipe for target 'CMakeFiles/snmallocshim-1mib-rust.dir/rule' failed
  make[1]: Leaving directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  Makefile:581: recipe for target 'snmallocshim-1mib-rust' failed

  --- stderr
  CMake Warning at CMakeLists.txt:79 (message):
    Not generating clangformat target, no clang-format tool found
  Call Stack (most recent call first):
    CMakeLists.txt:366 (clangformat_targets)


     Called from: [2]	/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/CMakeLists.txt
                  [1]	/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/CMakeLists.txt
  CMake Warning:
    Manually-specified variables were not used by the project:

      CMAKE_ASM_COMPILER
      CMAKE_ASM_FLAGS


  aarch64-linux-musl-g++: error: unrecognized command line option '-mcx16'
  make[3]: *** [CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o] Error 1
  make[2]: *** [CMakeFiles/snmallocshim-1mib-rust.dir/all] Error 2
  make[1]: *** [CMakeFiles/snmallocshim-1mib-rust.dir/rule] Error 2
  make: *** [snmallocshim-1mib-rust] Error 2
  thread 'main' panicked at '
  command did not execute successfully, got: exit status: 2

  build script failed, must exit now', /cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.45/src/lib.rs:894:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I built it with cross.

[ryancinsight]

can you also try --features build_cc as well so we can see if there are other errors than -mcx16

[zonyitoo]

Build with build_cc produces hundreds of lines of warnings, here are some of them

  cargo:warning=snmalloc/src/override/../mem/alloc.h:1067:41:   required from 'snmalloc::CapPtr<void, (snmalloc::capptr_bounds)5u> snmalloc::Allocator<NeedsInitialisation, InitThreadAllocator, MemoryProvider, ChunkMap, IsQueueInline>::small_alloc(size_t) [with snmalloc::ZeroMem zero_mem = (snmalloc::ZeroMem)0u; bool (* NeedsInitialisation)(void*) = snmalloc::needs_initialisation; void* (* InitThreadAllocator)(snmalloc::function_ref<void*(void*)>) = snmalloc::init_thread_allocator; MemoryProvider = snmalloc::MemoryProviderStateMixin<snmalloc::PALLinux, snmalloc::DefaultArenaMapTemplate<snmalloc::PALLinux, snmalloc::GlobalPagemapTemplate<snmalloc::default_auth_pagemap, snmalloc::ForAuthmap> > >; ChunkMap = snmalloc::DefaultChunkMap<>; bool IsQueueInline = true; size_t = long unsigned int]'
  cargo:warning=snmalloc/src/override/../mem/alloc.h:180:51:   required from 'void* snmalloc::Allocator<NeedsInitialisation, InitThreadAllocator, MemoryProvider, ChunkMap, IsQueueInline>::alloc(size_t) [with snmalloc::ZeroMem zero_mem = (snmalloc::ZeroMem)0u; bool (* NeedsInitialisation)(void*) = snmalloc::needs_initialisation; void* (* InitThreadAllocator)(snmalloc::function_ref<void*(void*)>) = snmalloc::init_thread_allocator; MemoryProvider = snmalloc::MemoryProviderStateMixin<snmalloc::PALLinux, snmalloc::DefaultArenaMapTemplate<snmalloc::PALLinux, snmalloc::GlobalPagemapTemplate<snmalloc::default_auth_pagemap, snmalloc::ForAuthmap> > >; ChunkMap = snmalloc::DefaultChunkMap<>; bool IsQueueInline = true; size_t = long unsigned int]'
  cargo:warning=snmalloc/src/override/malloc.cc:46:54:   required from here
  cargo:warning=snmalloc/src/override/../mem/alloc.h:986:7: error: invalid use of void expression
  cargo:warning=       super->init(public_state());
  cargo:warning=       ^~~~~
  cargo:warning=snmalloc/src/override/../mem/alloc.h: In member function 'void* snmalloc::Allocator<NeedsInitialisation, InitThreadAllocator, MemoryProvider, ChunkMap, IsQueueInline>::external_pointer(void*) [with snmalloc::Boundary location = (snmalloc::Boundary)0u; bool (* NeedsInitialisation)(void*) = snmalloc::needs_initialisation; void* (* InitThreadAllocator)(snmalloc::function_ref<void*(void*)>) = snmalloc::init_thread_allocator; MemoryProvider = snmalloc::MemoryProviderStateMixin<snmalloc::PALLinux, snmalloc::DefaultArenaMapTemplate<snmalloc::PALLinux, snmalloc::GlobalPagemapTemplate<snmalloc::default_auth_pagemap, snmalloc::ForAuthmap> > >; ChunkMap = snmalloc::DefaultChunkMap<>; bool IsQueueInline = true]':
  cargo:warning=snmalloc/src/override/../mem/alloc.h:418:7: warning: control reaches end of non-void function [-Wreturn-type]
  cargo:warning=       }
  cargo:warning=       ^
  cargo:warning=snmalloc/src/override/../mem/alloc.h: In member function 'void* snmalloc::Allocator<NeedsInitialisation, InitThreadAllocator, MemoryProvider, ChunkMap, IsQueueInline>::external_pointer(void*) [with snmalloc::Boundary location = (snmalloc::Boundary)2u; bool (* NeedsInitialisation)(void*) = snmalloc::needs_initialisation; void* (* InitThreadAllocator)(snmalloc::function_ref<void*(void*)>) = snmalloc::init_thread_allocator; MemoryProvider = snmalloc::MemoryProviderStateMixin<snmalloc::PALLinux, snmalloc::DefaultArenaMapTemplate<snmalloc::PALLinux, snmalloc::GlobalPagemapTemplate<snmalloc::default_auth_pagemap, snmalloc::ForAuthmap> > >; ChunkMap = snmalloc::DefaultChunkMap<>; bool IsQueueInline = true]':
  cargo:warning=snmalloc/src/override/../mem/alloc.h:418:7: warning: control reaches end of non-void function [-Wreturn-type]
  cargo:warning=       }
  cargo:warning=       ^
  cargo:warning=In file included from snmalloc/src/override/../mem/../ds/helpers.h:3:0,
  cargo:warning=                 from snmalloc/src/override/../mem/globalalloc.h:3,
  cargo:warning=                 from snmalloc/src/override/../mem/slowalloc.h:3,
  cargo:warning=                 from snmalloc/src/override/malloc.cc:1,
  cargo:warning=                 from snmalloc/src/override/rust.cc:2:
  cargo:warning=snmalloc/src/override/../mem/../ds/bits.h: In function 'size_t snmalloc::bits::clz(size_t)':
  cargo:warning=snmalloc/src/override/../mem/../ds/bits.h:91:5: warning: control reaches end of non-void function [-Wreturn-type]
  cargo:warning=     }
  cargo:warning=     ^
  cargo:warning=In file included from snmalloc/src/override/../mem/../mem/sizeclass.h:3:0,
  cargo:warning=                 from snmalloc/src/override/../mem/allocstats.h:4,
  cargo:warning=                 from snmalloc/src/override/../mem/alloc.h:10,
  cargo:warning=                 from snmalloc/src/override/../mem/globalalloc.h:4,
  cargo:warning=                 from snmalloc/src/override/../mem/slowalloc.h:3,
  cargo:warning=                 from snmalloc/src/override/malloc.cc:1,
  cargo:warning=                 from snmalloc/src/override/rust.cc:2:
  cargo:warning=snmalloc/src/override/../mem/../mem/../pal/pal.h: At global scope:
  cargo:warning=snmalloc/src/override/../mem/../mem/../pal/pal.h:83:3: warning: 'std::enable_if_t<(! aal_supports<StrictProvenance, AAL>), snmalloc::CapPtr<T, capptr_export_type<B>()> > snmalloc::capptr_export(snmalloc::CapPtr<T, B>) [with PAL = snmalloc::PALLinux; AAL = snmalloc::AAL_Generic<snmalloc::AAL_NoStrictProvenance<snmalloc::AAL_arm> >; T = snmalloc::Mediumslab; snmalloc::capptr_bounds B = (snmalloc::capptr_bounds)2u]' used but never defined
  cargo:warning=   capptr_export(CapPtr<T, B> p)
  cargo:warning=   ^~~~~~~~~~~~~
  exit status: 1

  --- stderr


  error occurred: Command "aarch64-linux-musl-g++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "snmalloc/src" "-Wall" "-Wextra" "-O3" "-fno-exceptions" "-fno-rtti" "-g" "-fomit-frame-pointer" "-fpermissive" "-std=c++17" "-ftls-model=initial-exec" "-o" "/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-58af90f0d695a805/out/snmalloc/src/override/rust.o" "-c" "snmalloc/src/override/rust.cc" with args "aarch64-linux-musl-g++" did not execute successfully (status code exit status: 1).

Compiliation failed because of

snmalloc/src/override/../mem/alloc.h:986:7: error: invalid use of void expression
       super->init(public_state());
       ^~~~~

[ryancinsight]

Thanks, I've seen the majority of warnings go away using clang, g++ tends to be a little aggressive, you can change by or musl equivalent
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++

[zonyitoo]

Ok thanks. Let me build a docker image and try again.

[zonyitoo]

BTW, it seems that snmalloc doesn't work well on MUSL: #102 .

Let me try to make a docker image with a clang wrapper of MUSL.

[ryancinsight]

yeah but your original error and architecture were a little different so I was getting curious to why there was a different error lol

[zonyitoo]

Well, after searching on Google, there is no existed well maintained project that help to build a MUSL toolchain with clang.

I would give up right here, since even if I have succeeded make it built, it won't work well with MUSL.

[ryancinsight]

I wouldn't close yet, I know upstream was thinking on trying to implement a workaround for the previous musl issue so they may want to know someone is still interested

[mjp41]

Yeah. It is possible to write a work around to the issue mentioned. It isn't on my path at the moment, but I would be happy to review, and help someone if they want to take on #102.

#102 (comment)

[mjp41]

The -mcx16 should not be set on ARM.

  if(${CMAKE_SYSTEM_PROCESSOR} STREQUAL "x86_64")
   target_compile_options(snmalloc_lib INTERFACE -mcx16)
  elseif(${CMAKE_SYSTEM_PROCESSOR} STREQUAL "amd64")
   target_compile_options(snmalloc_lib INTERFACE -mcx16)
  elseif(${CMAKE_SYSTEM_PROCESSOR} STREQUAL "AMD64")
   target_compile_options(snmalloc_lib INTERFACE -mcx16)
  elseif(${CMAKE_SYSTEM_PROCESSOR} STREQUAL "x86")
   target_compile_options(snmalloc_lib INTERFACE -mcx16)
  # XXX elseif ARM?
  endif()

This code should support cross compiling. Perhaps, this is a Rust CMake issue? Though, our CI does not check the cross compiling case.

The warning:

  cargo:warning=In file included from snmalloc/src/override/../mem/../ds/helpers.h:3:0,
  cargo:warning=                 from snmalloc/src/override/../mem/globalalloc.h:3,
  cargo:warning=                 from snmalloc/src/override/../mem/slowalloc.h:3,
  cargo:warning=                 from snmalloc/src/override/malloc.cc:1,
  cargo:warning=                 from snmalloc/src/override/rust.cc:2:
  cargo:warning=snmalloc/src/override/../mem/../ds/bits.h: In function 'size_t snmalloc::bits::clz(size_t)':
  cargo:warning=snmalloc/src/override/../mem/../ds/bits.h:91:5: warning: control reaches end of non-void function [-Wreturn-type]
  cargo:warning=     }

Looks like the constexpr is not resolving to a definition on the platform, which seems strange to me.

@nwf @davidchisnall

[ryancinsight]

OK I was able to build with clang available so Ill attach the dockerfile to help anyone interested,
The simplified clang error is cstddef not found on aarch64 musl:
running: "clang++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=aarch64-unknown-linux-musl" "-I" "snmalloc/src" "-Wall" "-Wextra" "-o" "/snmalloc-rs/target/aarch64-unknown-linux-musl/debug/build/snmalloc-sys-83e4008035e1f873/out/snmalloc/src/override/rust.o" "-c" "snmalloc/src/override/rust.cc"
cargo:warning=In file included from snmalloc/src/override/rust.cc:2:
cargo:warning=In file included from snmalloc/src/override/malloc.cc:1:
cargo:warning=In file included from snmalloc/src/override/../mem/slowalloc.h:3:
cargo:warning=In file included from snmalloc/src/override/../mem/globalalloc.h:3:
cargo:warning=In file included from snmalloc/src/override/../mem/../ds/helpers.h:3:
cargo:warning=snmalloc/src/override/../mem/../ds/bits.h:3:10: fatal error: 'cstddef' file not found
cargo:warning=#include
cargo:warning= ^~~~~~~~~
cargo:warning=1 error generated.
exit code: 1

--- stderr

error occurred: Command "clang++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=aarch64-unknown-linux-musl" "-I" "snmalloc/src" "-Wall" "-Wextra" "-o" "/snmalloc-rs/target/aarch64-unknown-linux-musl/debug/build/snmalloc-sys-83e4008035e1f873/out/snmalloc/src/override/rust.o" "-c" "snmalloc/src/override/rust.cc" with args "clang++" did not execute successfully (status code exit code: 1).
Dockerfile.txt