SmallVector.h: avoid casting away constness #1439

Job	Run time
linux-coverage	20m 29s
bindings	27m 9s
conan	11m 5s
build (clang-sanitize, ubuntu-22.04)	1h 29m 21s
build (gcc-release, ubuntu-22.04)	19m 56s
build (gcc-10-release, ubuntu-22.04)	19m 44s
build (gcc-debug-shared, ubuntu-22.04)	11m 36s
build (gcc-debug-noexcept, ubuntu-22.04)	13m 30s
build (macos-release, macos-13)	6m 13s
build (win64-release, windows-latest)	17m 0s
build (win64-debug-shared, windows-latest)	14m 24s
build (win64-debug-noexcept, windows-latest)	11m 10s
	4h 21m 37s

Provide feedback