Building artifacts should fail fast when errors occur

[dliappis]

Description

When trying to build release artifacts from main branch (commit: ) using Buildkite containers -- as opposed to a VM -- we observed that an error during the :bootstrap task gets ignored, resulting in a missing snakeyaml-engine.jar in the deb artifact.

Detailed Logs

Below is an excerpt from the Buildkite output:

[artifact:deb] building deb package for x86_64
--
  | WARNING: Unknown module: org.jruby.dist specified to --add-opens
  | WARNING: Unknown module: org.jruby.dist specified to --add-opens
  | WARNING: Unknown module: org.jruby.dist specified to --add-opens
  | WARNING: Unknown module: org.jruby.dist specified to --add-opens
  | To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.5/userguide/gradle_daemon.html#sec:disabling_the_daemon.
  | Daemon will be stopped at the end of the build
  |  
  | > Task :downloadPreviousJRuby UP-TO-DATE
  | Download https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.3.9.0/jruby-dist-9.3.9.0-bin.tar.gz
  |  
  | > Task :downloadJRuby UP-TO-DATE
  | Download https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.4.3.0/jruby-dist-9.4.3.0-bin.tar.gz
  |  
  | > Task :bootstrap
  | 2023-09-27T15:13:23.844Z [Execution worker Thread 5] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
  | Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.
  | Error Errno::ENOENT, retrying 1/10
  | No such file or directory - /buildkite/builds/bk-agent-prod-k8s-1695827479442731913/elastic/logstash-dra-pipeline-ci/vendor/bundle/jruby/3.1.0/gems/psych-5.1.0-java/deps.lst

Initial analysis

Upon further inspection and after comparing with an artifact built on a VM using the same process, we can see that the following is missing:

$ tree usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/psych-5.1.0-java/lib/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/psych-5.1.0-java/lib/
├── org
│   └── snakeyaml
│       └── snakeyaml-engine
│           └── 2.6
│               └── snakeyaml-engine-2.6.jar

Under the hood the sequence of execution is:

The bash script .buildkite/scripts/dra/build_packages.sh gets executed (env var WORKFLOW_TYPE=snapshot is set):
--> runs rake artifact:all
--> runs Rake's artifact:deb
--> runs Gradle

and somewhere in this chain, errors don't get propagated properly.

The team has looked a bit more into it and the initial thoughts are that bundler isn’t loading the jar-dependencies / ruby-maven stuff, so the pom file isn’t being processed at gem install time, and the jar doesn't get installed.

Steps to reproduce:

1. Build a docker image using docker build -t dockerbundlerissue . and the following Dockerfile

Dockerfile

FROM ubuntu:jammy
ARG DEBIAN_FRONTEND=noninteractive

# Increase default number of retries for apt
RUN echo 'Acquire::Retries "5";' > /etc/apt/apt.conf.d/80-retries

RUN apt-get clean && \
    apt-get update && \
    apt-get -y upgrade && \
    apt-get -y install apt-transport-https apt-utils ca-certificates \
        git curl git gnupg2 jq libmagic1 openssh-client retry rsync \
# for dra
    rpm \
# for rbenv
    autoconf bison patch build-essential rustc libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libgmp-dev libncurses5-dev libffi-dev libgdbm6 libgdbm-dev libdb-dev uuid-dev \
# java for jruby
    openjdk-17-jdk

# Install rbenv
RUN retry -t 5 -- git clone https://github.com/rbenv/rbenv.git /usr/local/rbenv \
    && retry -t 5 -- git clone https://github.com/rbenv/ruby-build.git /usr/local/rbenv/plugins/ruby-build \
    && /usr/local/rbenv/plugins/ruby-build/install.sh

RUN echo '# rbenv setup' > /etc/profile.d/rbenv.sh \
    && echo 'export RBENV_ROOT=/usr/local/rbenv' >> /etc/profile.d/rbenv.sh \
    && echo 'export PATH="$RBENV_ROOT/bin:$PATH"' >> /etc/profile.d/rbenv.sh \
    && echo 'eval "$(rbenv init -)"' >> /etc/profile.d/rbenv.sh \
    && echo 'eval "$(rbenv init -)"' >> /root/.bashrc

ENV PATH /root/.rbenv/shims:/usr/local/rbenv/bin:$PATH:/root/go/bin

RUN retry -t 5 -- rbenv install jruby-9.3.10.0

RUN rbenv rehash; rbenv global jruby-9.3.10.0

2. Start a container using the new docker image:

   docker run --rm -ti dockerbundlerissue:latest /bin/bash
   

3. Run the following commands in the container:

   git clone https://github.com/elastic/logstash && cd logstash
   export WORKFLOW_TYPE=snapshot
   .buildkite/scripts/dra/build_packages.sh

4. Observe after ./gradlew assemble and [artifact:deb] building deb package for x86_64 (doesn't take very long, depending on network speed):

   > Task :bootstrap
   2023-09-28T08:29:42.361Z [Execution worker Thread 11] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
   Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.
   Error Errno::ENOENT, retrying 1/10
   No such file or directory - /logstash/vendor/bundle/jruby/3.1.0/gems/psych-5.1.0-java/deps.lst
   

Related issue

#15356