Merge #6138: backport: merge bitcoin#22840, bitcoin#22937, bitcoin#23446

, bitcoin#23522, bitcoin#24026, bitcoin#24104, bitcoin#24167, bitcoin#20744, partial bitcoin#23469, bitcoin#24169 (replace boost::filesystem with std::filesystem)

0f23920 partial bitcoin#24169: Add --enable-c++20 option (Kittywhiskers Van Gogh)
a3b7926 merge bitcoin#20744: Use std::filesystem. Remove Boost Filesystem & System (Kittywhiskers Van Gogh)
be7ac49 merge bitcoin#24167: consistently use fsbridge:: for ifstream / ofstream (Kittywhiskers Van Gogh)
7ffea43 merge bitcoin#24104: Make compatible with boost 1.78 (Kittywhiskers Van Gogh)
7c270e6 merge bitcoin#24026: Block unsafe std::string fs::path conversion copy_file calls (Kittywhiskers Van Gogh)
b0d2484 merge bitcoin#23522: Improve fs::PathToString documentation (Kittywhiskers Van Gogh)
20d359b partial bitcoin#23469: Remove Boost build note from build-unix.md (Kittywhiskers Van Gogh)
193f6fd merge bitcoin#23446: Mention that BerkeleyDB is for legacy wallet in build-unix (Kittywhiskers Van Gogh)
ecfac10 merge bitcoin#22937: Forbid calling unsafe fs::path(std::string) constructor and fs::path::string() method (Kittywhiskers Van Gogh)
23fe7e2 chore: dashify symbols in some unit tests (Kittywhiskers Van Gogh)
28b96a0 merge bitcoin#22840: fix unoptimized libraries in depends (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * Depends on #6085
  * Depends on #6137
  * Dependency for #6150

  ## Breaking Changes

  None observed.

  ## Checklist:

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation **(note: N/A)**
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  knst:
    utACK  0f23920
  PastaPastaPasta:
    utACK 0f23920

Tree-SHA512: 4b8f0ae55185ece27d8084a5339196b7ed993c8138f4c59a0db3e16729d4edf4a59a68a8c7309c32df57734c07182821c4878b55c253da5df763204ad7158426

build-aux/m4/l_filesystem.m4

@@ -0,0 +1,47 @@
+dnl Copyright (c) 2022 The Bitcoin Core developers
+dnl Distributed under the MIT software license, see the accompanying
+dnl file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+# GCC 8.1 and earlier requires -lstdc++fs
+# Clang 8.0.0 (libc++) and earlier requires -lc++fs
+
+m4_define([_CHECK_FILESYSTEM_testbody], [[
+  #include <filesystem>
+
+  namespace fs = std::filesystem;
+
+  int main() {
+    (void)fs::current_path().root_name();
+    return 0;
+  }
+]])
+
+AC_DEFUN([CHECK_FILESYSTEM], [
+
+  AC_LANG_PUSH(C++)
+
+  AC_MSG_CHECKING([whether std::filesystem can be used without link library])
+
+  AC_LINK_IFELSE([AC_LANG_SOURCE([_CHECK_FILESYSTEM_testbody])],[
+      AC_MSG_RESULT([yes])
+    ],[
+      AC_MSG_RESULT([no])
+      SAVED_LIBS="$LIBS"
+      LIBS="$SAVED_LIBS -lstdc++fs"
+      AC_MSG_CHECKING([whether std::filesystem needs -lstdc++fs])
+      AC_LINK_IFELSE([AC_LANG_SOURCE([_CHECK_FILESYSTEM_testbody])],[
+          AC_MSG_RESULT([yes])
+        ],[
+          AC_MSG_RESULT([no])
+          AC_MSG_CHECKING([whether std::filesystem needs -lc++fs])
+          LIBS="$SAVED_LIBS -lc++fs"
+          AC_LINK_IFELSE([AC_LANG_SOURCE([_CHECK_FILESYSTEM_testbody])],[
+            AC_MSG_RESULT([yes])
+        ],[
+            AC_MSG_FAILURE([cannot figure out how to use std::filesystem])
+          ])
+        ])
+    ])
+
+  AC_LANG_POP
+])

ci/test/00_setup_env_native_asan.sh

@@ -6,7 +6,7 @@
 
 export LC_ALL=C.UTF-8
 
-export PACKAGES="clang llvm python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev bsdmainutils libboost-dev libboost-filesystem-dev libboost-test-dev libdb5.3++-dev libminiupnpc-dev libzmq3-dev libqrencode-dev"
+export PACKAGES="clang llvm python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev bsdmainutils libboost-dev libboost-test-dev libdb5.3++-dev libminiupnpc-dev libzmq3-dev libqrencode-dev"
 export NO_DEPENDS=1
 export TEST_RUNNER_EXTRA="--timeout-factor=4"  # Increase timeout because sanitizers slow down
 export FUNCTIONAL_TESTS_CONFIG="--exclude wallet_multiwallet.py" # Temporarily suppress ASan heap-use-after-free (see issue #14163)

ci/test/00_setup_env_native_fuzz.sh

@@ -7,7 +7,7 @@
 export LC_ALL=C.UTF-8
 
 export CONTAINER_NAME=ci_native_fuzz
-export PACKAGES="clang llvm python3 libevent-dev bsdmainutils libboost-dev libboost-filesystem-dev libboost-test-dev"
+export PACKAGES="clang llvm python3 libevent-dev bsdmainutils libboost-dev libboost-test-dev"
 export DEP_OPTS="NO_UPNP=1 DEBUG=1"
 export CPPFLAGS="-DDEBUG_LOCKORDER -DARENA_DEBUG"
 export CXXFLAGS="-Werror -Wno-unused-command-line-argument -Wno-unused-value -Wno-deprecated-builtins"

ci/test/00_setup_env_native_fuzz_with_valgrind.sh

@@ -7,7 +7,7 @@
 export LC_ALL=C.UTF-8
 
 export CONTAINER_NAME=ci_native_fuzz_valgrind
-export PACKAGES="clang llvm python3 libevent-dev bsdmainutils libboost-dev libboost-system-dev libboost-filesystem-dev libboost-test-dev valgrind"
+export PACKAGES="clang llvm python3 libevent-dev bsdmainutils libboost-dev libboost-test-dev valgrind"
 export NO_DEPENDS=1
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false

ci/test/00_setup_env_native_ubsan.sh

@@ -8,7 +8,7 @@
 export LC_ALL=C.UTF-8
 
 export CONTAINER_NAME=ci_native_ubsan
-export PACKAGES="clang-16 llvm-16 python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev bsdmainutils libboost-filesystem-dev libboost-test-dev libboost-thread-dev libdb5.3++-dev libminiupnpc-dev libzmq3-dev libqrencode-dev"
+export PACKAGES="clang-16 llvm-16 python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev bsdmainutils libboost-test-dev libboost-thread-dev libdb5.3++-dev libminiupnpc-dev libzmq3-dev libqrencode-dev"
 export DEP_OPTS="NO_UPNP=1 DEBUG=1"
 export GOAL="install"
 export BITCOIN_CONFIG="--enable-zmq --enable-reduce-exports --enable-crash-hooks --with-sanitizers=undefined CC=clang-16 CXX=clang++-16"

ci/test/00_setup_env_native_valgrind.sh

@@ -6,7 +6,7 @@
 
 export LC_ALL=C.UTF-8
 
-export PACKAGES="valgrind clang llvm python3-zmq libevent-dev bsdmainutils libboost-dev libboost-system-dev libboost-filesystem-dev libboost-test-dev libdb5.3++-dev libminiupnpc-dev libzmq3-dev"
+export PACKAGES="valgrind clang llvm python3-zmq libevent-dev bsdmainutils libboost-dev libboost-test-dev libdb5.3++-dev libminiupnpc-dev libzmq3-dev"
 export USE_VALGRIND=1
 export NO_DEPENDS=1
 export TEST_RUNNER_EXTRA="--exclude rpc_bind --timeout-factor=4"  # Excluded for now, see https://github.com/bitcoin/bitcoin/issues/17765#issuecomment-602068547

configure.ac

@@ -72,16 +72,19 @@ AC_ARG_ENABLE([c++20],
   [use_cxx20=$enableval],
   [use_cxx20=no])
 
-dnl Require C++17 or C++20 compiler (no GNU extensions)
-if test "x$use_cxx20" = xyes; then
-  AX_CXX_COMPILE_STDCXX([20], [noext], [mandatory])
-else
+dnl Require C++17 compiler (no GNU extensions)
+if test "$use_cxx20" = "no"; then
   AX_CXX_COMPILE_STDCXX([17], [noext], [mandatory])
+else
+  AX_CXX_COMPILE_STDCXX([20], [noext], [mandatory])
 fi
 
 dnl Check if -latomic is required for <std::atomic>
 CHECK_ATOMIC
 
+dnl check if additional link flags are required for std::filesystem
+CHECK_FILESYSTEM
+
 dnl Unless the user specified OBJCXX, force it to be the same as CXX. This ensures
 dnl that we get the same -std flags for both.
 m4_ifdef([AC_PROG_OBJCXX],[
@@ -1472,8 +1475,6 @@ dnl and will generate warnings with newer compilers.
 dnl See: https://github.com/boostorg/container_hash/issues/22.
 BOOST_CPPFLAGS="$BOOST_CPPFLAGS -DBOOST_NO_CXX98_FUNCTION_BASE"
 
-AX_BOOST_FILESYSTEM
-
 dnl Opt-in to Boost Process
 if test "x$boost_process" != xno; then
 AC_MSG_CHECKING(for Boost Process)
@@ -1488,7 +1489,7 @@ if test x$suppress_external_warnings != xno; then
     BOOST_CPPFLAGS=SUPPRESS_WARNINGS($BOOST_CPPFLAGS)
 fi
 
-BOOST_LIBS="$BOOST_LDFLAGS $BOOST_SYSTEM_LIB $BOOST_FILESYSTEM_LIB"
+BOOST_LIBS="$BOOST_LDFLAGS"
 fi
 
 dnl Check for reduced exports

contrib/valgrind.supp

@@ -118,12 +118,6 @@
    fun:__wcsnlen_sse4_1
    fun:wcsnrtombs
 }
-{
-   Suppress wcsnrtombs warning (remove after removing boost::fs)
-   Memcheck:Cond
-   ...
-   fun:_ZN5boost10filesystem6detail11unique_pathERKNS0_4pathEPNS_6system10error_codeE
-}
 {
    Suppress boost warning
    Memcheck:Leak
@@ -134,21 +128,6 @@
    fun:_ZN5boost9unit_test14unit_test_mainEPFbvEiPPc
    fun:main
 }
-{
-   Suppress boost::filesystem warning (fixed in boost 1.70: https://github.com/boostorg/filesystem/commit/bbe9d1771e5d679b3f10c42a58fc81f7e8c024a9)
-   Memcheck:Cond
-   fun:_ZN5boost10filesystem6detail28directory_iterator_incrementERNS0_18directory_iteratorEPNS_6system10error_codeE
-   ...
-   obj:*/libboost_filesystem.so.*
-}
-{
-   Suppress boost::filesystem warning (could be related: https://stackoverflow.com/questions/9830182/function-boostfilesystemcomplete-being-reported-as-possible-memory-leak-by-v)
-   Memcheck:Leak
-   match-leak-kinds: reachable
-   fun:_Znwm
-   ...
-   fun:_ZN5boost10filesystem8absoluteERKNS0_4pathES3_
-}
 {
    Suppress boost still reachable memory warning
    Memcheck:Leak

depends/packages/bdb.mk

@@ -15,7 +15,7 @@ $(package)_config_opts_netbsd=--with-pic
 $(package)_config_opts_openbsd=--with-pic
 $(package)_config_opts_android=--with-pic
 $(package)_cflags+=-Wno-error=implicit-function-declaration
-$(package)_cxxflags=-std=c++17
+$(package)_cxxflags+=-std=c++17
 $(package)_cppflags_mingw32=-DUNICODE -D_UNICODE
 endef
 

depends/packages/boost.mk

@@ -26,8 +26,8 @@ $(package)_toolset_$(host_os)=clang
 else
 $(package)_toolset_$(host_os)=gcc
 endif
-$(package)_config_libraries=filesystem,test
-$(package)_cxxflags=-std=c++11
+$(package)_config_libraries=test
+$(package)_cxxflags+=-std=c++11
 $(package)_cxxflags_linux=-fPIC
 $(package)_cxxflags_freebsd=-fPIC
 $(package)_cxxflags_openbsd=-fPIC

depends/packages/zeromq.mk

@@ -16,7 +16,7 @@ define $(package)_set_vars
   $(package)_config_opts_netbsd=--with-pic
   $(package)_config_opts_openbsd=--with-pic
   $(package)_config_opts_android=--with-pic
-  $(package)_cxxflags=-std=c++17
+  $(package)_cxxflags+=-std=c++17
 endef
 
 define $(package)_preprocess_cmds