Failed to build virtual switch with SONIC_BUILD_JOBS=4

[dnsflood]

Hi，

When SONIC_BUILD_JOBS is set to 4, make all results in an error, whereas the build proceeds normally when SONIC_BUILD_JOBS is set to 1. Is this a bug?

git checkout master

make init
make configure PLATFORM=vs
make SONIC_BUILD_JOBS=4 all

Below is the error log：

[ FAIL LOG START ] [ target/sonic-vs.bin__vs__rfs.squashfs ]
Build start time: Fri Jul 5 12:25:11 UTC 2024
++ '[' -f .arch ']'
++ cat .arch

• CONFIGURED_ARCH=amd64
• DOCKER_VERSION=5:24.0.2-1debian.12bookworm
• CONTAINERD_IO_VERSION=1.6.21-1
• LINUX_KERNEL_VERSION=6.1.0-11-2
• FILESYSTEM_ROOT=./fsroot
• PLATFORM_DIR=platform
• HOSTNAME=sonic
• DEFAULT_USERINFO='Default admin user,,,'
• BUILD_TOOL_PATH=src/sonic-build-hooks/buildinfo
• TRUSTED_GPG_DIR=src/sonic-build-hooks/buildinfo/trusted.gpg.d
• . ./onie-image.conf
  ++ : 32768
  ++ : x86_64
  ++ : vs
  ++ ONIEIMAGE_VERSION=r0
  ++ FILESYSTEM_ROOT=./fsroot-vs
  ++ FILESYSTEM_SQUASHFS=fs.squashfs
  ++ INSTALLER_PAYLOAD=fs.zip
  ++ FILESYSTEM_DOCKERFS=dockerfs.tar.gz
  ++ DOCKERFS_DIR=docker
  ++ DOCKER_RAMFS_SIZE=2500M
  ++ OUTPUT_ONIE_IMAGE=target/sonic-vs.bin
  ++ OUTPUT_RAW_IMAGE=target/sonic-vs.raw
  ++ RAW_IMAGE_DISK_SIZE=10700
  ++ OUTPUT_KVM_IMAGE=target/sonic-vs.img
  ++ KVM_IMAGE_DISK_SIZE=16
  ++ OUTPUT_ABOOT_IMAGE=target/sonic-aboot-vs.swi
  ++ ABOOT_BOOT_IMAGE=.sonic-boot.swi
  ++ OUTPUT_KVM_4ASIC_IMAGE=target/sonic-4asic-vs.img
  ++ OUTPUT_KVM_6ASIC_IMAGE=target/sonic-6asic-vs.img
  ++ OUTPUT_DSC_IMAGE=target/sonic-vs.tar
• '[' -n 32768 ']'
• '[' -n fs.zip ']'
• '[' -n fs.squashfs ']'
• '[' onie = aboot ']'
• [[ n != y ]]
• [[ -d ./fsroot-vs ]]
• sudo rm -rf ./fsroot-vs
• mkdir -p ./fsroot-vs
• mkdir -p ./fsroot-vs/platform
• touch ./fsroot-vs/platform/firsttime
• bootloader_packages=
• '[' grub '!=' aboot ']'
• mkdir -p ./fsroot-vs/platform/grub
• bootloader_packages=grub2-common
• sudo mount proc /proc -t proc
  mount: /proc: proc already mounted on /proc.
  dmesg(1) may have more information after failed mount system call.
• true
• echo '[INFO] Build host debian base system...'
  [INFO] Build host debian base system...
• TARGET_PATH=target
• scripts/build_debian_base_system.sh amd64 bookworm ./fsroot-vs
  mknod: /sonic/fsroot-vs/test-dev-null: File exists
  E: Cannot install into target '/sonic/fsroot-vs' mounted with noexec or nodev
  [ FAIL LOG END ] [ target/sonic-vs.bin__vs__rfs.squashfs ]
  make: *** [slave.mk:1299: target/sonic-vs.bin__vs__rfs.squashfs] Error 1
  make[1]: *** [Makefile.work:604: all] Error 2

[mrafatpanah]

Hello. I had the same problem. I also don't know if it is a bug.
I built it with SONIC_BUILD_JOBS=4 till this error happened and then I didn't use SONIC_BUILD_JOBS and built with its default mode.

[dnsflood]

Hello. I had the same problem. I also don't know if it is a bug. I built it with SONIC_BUILD_JOBS=4 till this error happened and then I didn't use SONIC_BUILD_JOBS and built with its default mode.

Hi, I think this might not be considered a bug. The issue arises when multiple final targets are being built in parallel, as these concurrent build tasks use some of the same files, including the root file system, scripts, etc. The solution is to run the build tasks separately for each final target, and you can still use the SONIC_BUILD_JOBS=x option.

[ahsan-kaukab]

Hello
I am getting below error

`
Build Configuration
"CONFIGURED_PLATFORM" : "vs"
"CONFIGURED_ARCH" : "amd64"
"SONIC_CONFIG_PRINT_DEPENDENCIES" : ""
"SONIC_BUILD_JOBS" : "1"
"SONIC_CONFIG_MAKE_JOBS" : "8"
"USE_NATIVE_DOCKERD_FOR_BUILD" : ""
"SONIC_USE_DOCKER_BUILDKIT" : ""
"USERNAME" : "ahsan"
"PASSWORD" : "YourPaSsWoRd"
"CHANGE_DEFAULT_PASSWORD" : "n"
"SECURE_UPGRADE_MODE" : "no_sign"
"SECURE_UPGRADE_DEV_SIGNING_KEY" : ""
"SECURE_UPGRADE_SIGNING_CERT" : ""
"SECURE_UPGRADE_PROD_SIGNING_TOOL": ""
"SECURE_UPGRADE_PROD_TOOL_ARGS" : ""
"ONIE_IMAGE_PART_SIZE" : ""
"SHUTDOWN_BGP_ON_START" : ""
"ENABLE_PFCWD_ON_START" : ""
"SONIC_BUFFER_MODEL" : ""
"INSTALL_DEBUG_TOOLS" : ""
"ROUTING_STACK" : "frr"
"FRR_USER_UID" : "300"
"FRR_USER_GID" : "300"
"ENABLE_SYNCD_RPC" : ""
"SAITHRIFT_V2" : ""
"ENABLE_ORGANIZATION_EXTENSIONS" : "y"
"HTTP_PROXY" : ""
"HTTPS_PROXY" : ""
"NO_PROXY" : ""
"ENABLE_ZTP" : ""
"INCLUDE_PDE" : ""
"SONIC_DEBUGGING_ON" : ""
"SONIC_PROFILING_ON" : ""
"KERNEL_PROCURE_METHOD" : "build"
"BUILD_TIMESTAMP" : "20241011.153004"
"BUILD_LOG_TIMESTAMP" : "none"
"SONIC_IMAGE_VERSION" : "master.0-074caea"
"BLDENV" : "bookworm"
"VS_PREPARE_MEM" : "yes"
"INCLUDE_MGMT_FRAMEWORK" : "y"
"INCLUDE_ICCPD" : "n"
"INCLUDE_SYSTEM_TELEMETRY" : "n"
"INCLUDE_SYSTEM_GNMI" : "y"
"INCLUDE_SYSTEM_EVENTD" : "y"
"ENABLE_HOST_SERVICE_ON_START" : "y"
"INCLUDE_RESTAPI" : "n"
"INCLUDE_SFLOW" : "y"
"INCLUDE_NAT" : "y"
"INCLUDE_DHCP_RELAY" : "y"
"INCLUDE_DHCP_SERVER" : "n"
"INCLUDE_P4RT" : "n"
"INCLUDE_KUBERNETES" : "n"
"INCLUDE_KUBERNETES_MASTER" : "n"
"INCLUDE_MACSEC" : "y"
"INCLUDE_MUX" : "y"
"INCLUDE_TEAMD" : "y"
"INCLUDE_ROUTER_ADVERTISER" : "y"
"INCLUDE_BOOTCHART : "y"
"ENABLE_BOOTCHART : "n"
"INCLUDE_FIPS" : "y"
"ENABLE_TRANSLIB_WRITE" : ""
"ENABLE_NATIVE_WRITE" : "y"
"ENABLE_DIALOUT" : ""
"ENABLE_AUTO_TECH_SUPPORT" : "y"
"PDDF_SUPPORT" : "y"
"MULTIARCH_QEMU_ENVIRON" : "n"
"SONIC_VERSION_CONTROL_COMPONENTS": "none"
"ENABLE_ASAN" : "n"
"DEFAULT_CONTAINER_REGISTRY" : ""
"BUILD_MULTIASIC_KVM" : "n"
"CROSS_BUILD_ENVIRON" : "n"
"LEGACY_SONIC_MGMT_DOCKER" : "y"
"INCLUDE_EXTERNAL_PATCHES" : "n"
"PTF_ENV_PY_VER" : "mixed"

mv: cannot move '/etc/apt/sources.list.d/debian.sources' to '/etc/apt/sources.list.d/debian.sources.back': Permission denied
"SONIC_DPKG_CACHE_METHOD" : "none"

[ FAIL LOG START ] [ target/debs/bookworm/frr_8.5.4-sonic-0_amd64.deb ]
Build start time: Fri Oct 11 10:30:14 UTC 2024
[ REASON ] : target/debs/bookworm/frr_8.5.4-sonic-0_amd64.deb does not exist NON-EXISTENT PREREQUISITES: target/debs/bookworm/libsnmp-dev_5.9.3+dfsg-2_amd64.deb-install target/debs/bookworm/libyang2_2.0.112-6_amd64.deb-install target/debs/bookworm/libyang2-dev_2.0.112-6_amd64.deb-install
[ FLAGS FILE ] : []
[ FLAGS DEPENDS ] : []
[ FLAGS DIFF ] : []
make[1]: Entering directory '/sonic/src/sonic-frr'

pushd ./frr
git checkout -b frr/8.5 origin/frr/8.5 || git checkout frr/8.5
stg branch --create stg_temp.20241011.103014 frr-8.5.4
stg import -s ../patch/series
gbp dch --ignore-branch --new-version=8.5.4-sonic-0 --dch-opt="--force-bad-version" --commit --git-author
cp ../dplane_fpm_sonic/dplane_fpm_sonic.c zebra/
dpkg-buildpackage -rfakeroot -b -us -uc -Ppkg.frr.nortrlib -j8 --admindir /sonic/dpkg/tmp.jAGhDXD4df
stg undo || true
git clean -xfdf
git checkout frr/8.5
stg branch --delete --force stg_temp.20241011.103014
git rev-parse --short HEAD | xargs git checkout
git checkout master
git branch -D frr/8.5
popd
mv frr-pythontools_8.5.4-sonic-0_all.deb frr-dbgsym_8.5.4-sonic-0_amd64.deb frr-snmp_8.5.4-sonic-0_amd64.deb frr-snmp-dbgsym_8.5.4-sonic-0_amd64.deb frr_8.5.4-sonic-0_amd64.deb /sonic/target/debs/bookworm/
/sonic/src/sonic-frr/frr /sonic/src/sonic-frr
fatal: 'origin/frr/8.5' is not a commit and a branch 'frr/8.5' cannot be created from it
error: pathspec 'frr/8.5' did not match any file(s) known to git
make[1]: *** [Makefile:15: /sonic/target/debs/bookworm/frr_8.5.4-sonic-0_amd64.deb] Error 1
make[1]: Leaving directory '/sonic/src/sonic-frr'
[ FAIL LOG END ] [ target/debs/bookworm/frr_8.5.4-sonic-0_amd64.deb ]
make: *** [slave.mk:744: target/debs/bookworm/frr_8.5.4-sonic-0_amd64.deb] Error 1
make[1]: *** [Makefile.work:605: all] Error 2
make[1]: Leaving directory '/home/ahsan/Desktop/Ahsan-DreamBig/sonic-buildimage'
make: *** [Makefile:47: all] Error 2
ahsan@Ahsan:~/Desktop/
`
I used following commands

`git checkout master

make init
make configure PLATFORM=vs
make all`

[AlexFdv]

The same issue for me, with SONIC_BUILD_JOBS=4 have "couldn't find these debs: ..." in the logs:

+ TARGET_PATH=target
+ scripts/build_debian_base_system.sh amd64 bookworm ./fsroot-vs
I: Target architecture can be executed
I: Retrieving InRelease 
I: Checking Release signature
I: Valid Release signature (key id 4D64FEC119C2029067D6E791F8D2585B8783D481)
I: Retrieving Packages 
I: Validating Packages 
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
E: Couldn't find these debs:  adduser apt base-files base-passwd bash bsdutils coreutils dash debconf debian-archive-keyring debianutils diffutils dpkg e2fsprogs findutils gpgv grep gzip hostname init-system-helpers libacl1 libapt-pkg6.0 libattr1 libaudit1 libblkid1 libbz2-1.0 libc-bin libc6 libcap-ng0 libcom-err2 libcrypt1 libdb5.3 libdebconfclient0 libext2fs2 libgcc-s1 libgmp10 libgnutls30 liblzma5 libmd0 libmount1 libpam-modules libpam-modules-bin libpam-runtime libpam0g libpcre2-8-0 libseccomp2 libselinux1 libsemanage2 libsmartcols1 libss2 libstdc++6 libsystemd0 libtinfo6 libudev1 libuuid1 libzstd1 login logsave mawk mount ncurses-base ncurses-bin passwd perl-base sed sysvinit-utils tar tzdata usr-is-merged util-linux util-linux-extra zlib1g 
[  FAIL LOG END  ] [ target/sonic-vs.bin__vs__rfs.squashfs ]
make: *** [slave.mk:1308: target/sonic-vs.bin__vs__rfs.squashfs] Error 1
make[1]: *** [Makefile.work:605: all] Error 2
make[1]: Leaving directory '/home/oleksii.fadieiev/projects/sonic-buildimage'
make: *** [Makefile:47: all] Error 2

Using just make all I have another error:

cd /sonic/src/p4lang/p4lang-p4c-1.2.4.2/build/ir && /usr/bin/cmake -P CMakeFiles/ir.dir/cmake_clean_target.cmake
cd /sonic/src/p4lang/p4lang-p4c-1.2.4.2/build/ir && /usr/bin/cmake -E cmake_link_script CMakeFiles/ir.dir/link.txt --verbose=1
/usr/bin/ar qc libir.a CMakeFiles/ir.dir/base.cpp.o CMakeFiles/ir.dir/dbprint.cpp.o "CMakeFiles/ir.dir/dbprint-expression.cpp.o" "CMakeFiles/ir.dir/dbprint-stmt.cpp.o" "CMakeFiles/ir.dir/dbprint-type.cpp.o" "CMakeFiles/ir.dir/dbprint-p4.cpp.o" CMakeFiles/ir.dir/dump.cpp.o CMakeFiles/ir.dir/expression.cpp.o CMakeFiles/ir.dir/ir.cpp.o CMakeFiles/ir.dir/irutils.cpp.o CMakeFiles/ir.dir/json_parser.cpp.o CMakeFiles/ir.dir/node.cpp.o CMakeFiles/ir.dir/pass_manager.cpp.o CMakeFiles/ir.dir/type.cpp.o CMakeFiles/ir.dir/v1.cpp.o CMakeFiles/ir.dir/visitor.cpp.o CMakeFiles/ir.dir/write_context.cpp.o
/usr/bin/ranlib libir.a
make[6]: Leaving directory '/sonic/src/p4lang/p4lang-p4c-1.2.4.2/build'
[ 22%] Built target ir
make[5]: Leaving directory '/sonic/src/p4lang/p4lang-p4c-1.2.4.2/build'
make[4]: *** [Makefile:169: all] Error 2
make[4]: Leaving directory '/sonic/src/p4lang/p4lang-p4c-1.2.4.2/build'
dh_auto_install: error: cd build && make -j8 install DESTDIR=/sonic/src/p4lang/p4lang-p4c-1.2.4.2/debian/p4lang-p4c AM_UPDATE_INFO_DIR=no "INSTALL=install --strip-program=true" DESTDIR=/sonic/src/p4lang/p4lang-p4c-1.2.4.2/debian/p4lang-p4c install returned exit code 2
make[3]: *** [debian/rules:47: override_dh_auto_install] Error 25
make[3]: Leaving directory '/sonic/src/p4lang/p4lang-p4c-1.2.4.2'
make[2]: *** [debian/rules:16: binary] Error 2
make[2]: Leaving directory '/sonic/src/p4lang/p4lang-p4c-1.2.4.2'
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
make[1]: *** [Makefile:38: /sonic/target/debs/bookworm/p4lang-p4c_1.2.4.2-2_amd64.deb] Error 2
make[1]: Leaving directory '/sonic/src/p4lang'
[  FAIL LOG END  ] [ target/debs/bookworm/p4lang-p4c_1.2.4.2-2_amd64.deb ]
make: *** [slave.mk:744: target/debs/bookworm/p4lang-p4c_1.2.4.2-2_amd64.deb] Error 1
make[1]: *** [Makefile.work:605: all] Error 2
make[1]: Leaving directory '/home/oleksii.fadieiev/projects/sonic-buildimage'
make: *** [Makefile:47: all] Error 2

[AlexFdv]

I have the same issue, it is a challenging to build a master branch on my environment: I had to rerun make with and without SONIC_BUILD_JOBS=4 parameter after each failure. But then fails on the next component(s).