Docker image builds fail with go: modules disabled by GO111MODULE=off; see 'go help modules' -- remove GO111MODULE=off

[dliappis]

Since Mar 7 2024, we are seeing failures in CI steps that build Docker images, e.g. DRA jobs and the acceptance/docker group of the exhaustive pipeline:

------
Dockerfile:2
--------------------
   1 |     FROM golang:1
   2 | >>> RUN go env -w GO111MODULE=off && (for i in 0 1 2 3 4 5; do sleep "$i"; go get gopkg.in/yaml.v2 && break; done)
   3 |     WORKDIR /usr/local/src/env2yaml
   4 |     CMD ["go", "build"]
--------------------
ERROR: failed to solve: process "/bin/sh -c go env -w GO111MODULE=off && (for i in 0 1 2 3 4 5; do sleep \"$i\"; go get gopkg.in/yaml.v2 && break; done)" did not complete successfully: exit code: 1
make: *** [Makefile:205: golang] Error 1

Example failures:

• DRA Snapshot: https://buildkite.com/elastic/logstash-dra-snapshot-pipeline/builds/494#018d868e-b429-4734-9cd5-666eeec81c0e
• Exhaustive acceptance/docker: https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/211#018d8486-c72b-4e05-992d-82a0b1d21ca2/113-5410

The root cause is that we use golang:1 as the base docker image, but recently 1.22 was released and

go get is no longer supported outside of a module in the legacy GOPATH mode (that is, with GO111MODULE=off). Other build commands, such as go build and go test, will continue to work indefinitely for legacy GOPATH programs.

The quick solution would be to pin to 1.21, but we should install the module without GO111MODULE=off i.e. not using the legacy GOPATH.