ggml: Add run-time detection of neon, i8mm and sve

[eddnjjn]

This patch adds run-time detection of the Arm instructions set features Arm® Neon™, i8mm and sve for Linux and Apple build targets. The run-time detection is enabled for aarch64 builds and done in ggml_init. The data is stored in a global struct instance to be later used by the ggml_cpu_has_* functions.

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[ggerganov]

For clarity, this call should be within the is_first_call section above

[eddnjjn]

Thanks for the review @ggerganov . I've rebased the patch and addressed your comment by moving the invocation of ggml_init_aarch64_features to the is_first_call section. I also updated ggml_init_aarch64_features to not check for first invocation since this is done in ggml_init.

Please let me know if you have additional comments.

[ggerganov]

This looks incorrect because ARM NEON presence is now associated with __aarch64__, but this is not always the case AFAIK. For example, here we have support for __ARM_NEON && !__aarch64__, such as Raspberry Pi:

llama.cpp/ggml/src/ggml-cpu-impl.h

Lines 132 to 167 in 64c6af3

	#if defined(__ARM_NEON)
	// if YCM cannot find <arm_neon.h>, make a symbolic link to it, for example:
	//
	// $ ln -sfn /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include/arm_neon.h ./src/
	//
	#include <arm_neon.h>
	#ifdef _MSC_VER
	typedef uint16_t ggml_fp16_internal_t;
	#define ggml_vld1q_u32(w,x,y,z) { ((w) + ((uint64_t)(x) << 32)), ((y) + ((uint64_t)(z) << 32)) }
	#else
	typedef __fp16 ggml_fp16_internal_t;
	#define ggml_vld1q_u32(w,x,y,z) { (w), (x), (y), (z) }
	#endif // _MSC_VER
	#if !defined(__aarch64__)
	// 32-bit ARM compatibility
	// vaddlvq_s16
	// vpaddq_s16
	// vpaddq_s32
	// vaddvq_s32
	// vaddvq_f32
	// vmaxvq_f32
	// vcvtnq_s32_f32
	// vzip1_u8
	// vzip2_u8

[eddnjjn]

I've addressed this comment in the update. Non aarch64 Linux builds fallback to using compile time flags for detecting whether or not Neon is enabled.

[ggerganov]

Move down in the global data section, around line 438

[eddnjjn]

Done!

[eddnjjn]

I've addressed @ggerganov's latest comment and rebased the patches on latest master. Please let me know if you have additional comments.

[ggerganov]

This function is better to not be facing the public API. Will merge this for now, but consider making it private.

[wtarreau]

This series breaks the build on Linux (Ubuntu 20.04 here) due to HWCAP2_I8MM not being known (probably depends on libc or kernel source version):

ggml/src/ggml.c: In function "ggml_init_arm_arch_features"
ggml/src/ggml.c:3696:51: error: "HWCAP2_I8MM" undeclared (first use in this function)
 3696 |     ggml_arm_arch_features.has_i8mm = !!(hwcap2 & HWCAP2_I8MM);
      |                                                   ^~~~~~~~~~~
ggml/src/ggml.c:3696:51: note: each undeclared identifier is reported only once for each function it appears in
make: *** [Makefile:1049: ggml/src/ggml.o] Error 1

This fixes it:

diff --git a/ggml/src/ggml.c b/ggml/src/ggml.c
index fac4466e..854949cb 100644
--- a/ggml/src/ggml.c
+++ b/ggml/src/ggml.c
@@ -3693,7 +3693,9 @@ static void ggml_init_arm_arch_features(void) {
-    uint32_t hwcap2 = getauxval(AT_HWCAP2);
+    uint32_t hwcap2 __attribute__((unused)) = getauxval(AT_HWCAP2);
 
     ggml_arm_arch_features.has_neon = !!(hwcap & HWCAP_ASIMD);
+#if defined(HWCAP2_I8MM)
     ggml_arm_arch_features.has_i8mm = !!(hwcap2 & HWCAP2_I8MM);
+#endif
     ggml_arm_arch_features.has_sve  = !!(hwcap & HWCAP_SVE);
 
 #if defined(__ARM_FEATURE_SVE)

But I think it would be cleaner to just define the missing HWCAP* flags before such as this, which is easy to extend to new flags when needed (and works as well):

diff --git a/ggml/src/ggml.c b/ggml/src/ggml.c
index fac4466e..f85de4a1 100644
--- a/ggml/src/ggml.c
+++ b/ggml/src/ggml.c
@@ -3687,6 +3687,10 @@ static inline int ggml_up(int n, int m) {
 #include <sys/sysctl.h>
 #endif

+#if !defined(HWCAP2_I8MM)
+#define HWCAP2_I8MM 0
+#endif
+
 static void ggml_init_arm_arch_features(void) {
 #if defined(__linux__) && defined(__aarch64__)
     uint32_t hwcap = getauxval(AT_HWCAP);