Impossible to install on Amazonlinux

[marc-guenther]

Since several years I have run watchman successfully in an Amazonlinux-2 Docker Container, by installing from the https://download.opensuse.org/repositories/home:crantila:watchman/CentOS_7/home:crantila:watchman.repo Repository:

FROM amazonlinux:2

RUN yum -y update && yum clean all && rm -rf /var/cache/yum

RUN curl -L https://download.opensuse.org/repositories/home:crantila:watchman/CentOS_7/home:crantila:watchman.repo >/etc/yum.repos.d/watchman.repo
RUN yum -y install watchman
RUN mkdir /var/run/watchman
RUN chmod 2777 /var/run/watchman

Last month I needed to make some changes to the container, and the build failed, because that repository has vanished in the mean time. 😮

So I try to install the official binaries:

RUN curl -OL https://github.com/facebook/watchman/releases/download/v2023.04.10.00/watchman-v2023.04.10.00-linux.zip
RUN yum install -y unzip
RUN unzip watchman-*-linux.zip
RUN mkdir -p /usr/local/{bin,lib} /usr/local/var/run/watchman
RUN cp watchman-v*.*.*.*-linux/bin/* /usr/local/bin
RUN cp watchman-v*.*.*.*-linux/lib/* /usr/local/lib
RUN chmod 755 /usr/local/bin/watchman
RUN chmod 2777 /usr/local/var/run/watchman
RUN yum install -y openssl11

This then fails due to wrong glibc version:

watchman: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by watchman)
watchman: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by watchman)

I then tried to upgrade the container to Amazonlinux-2023, which has a higher glibc version, but it is missing the required openssl package:

watchman: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory

I then tried to install through Homebrew, which blew up the size of my container from 180MB to 3.6GB, took like 8min to install, and in the end produced an executable which looks like it's working, but never notices even a single file change.

RUN yum install -y git tar
RUN /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
RUN eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)" ; brew install watchman
RUN ln -s /home/linuxbrew/.linuxbrew/bin/watchman* /usr/local/bin/

I then spent more than an entire day trying to compile from source code, because:

bash-4.2# ./install-system-packages.sh 
++ dirname ./install-system-packages.sh
+ python3 ./build/fbcode_builder/getdeps.py install-system-deps --recursive watchman
I don't know how to install any packages on this system linux-amazon-2

it seems I have to find out required packages by myself:

RUN yum install -y git
RUN git clone https://github.com/facebook/watchman.git
RUN yum install -y python3 python-devel
RUN yum install -y cargo clang make which
RUN yum install -y openssl11 openssl11-devel
RUN yum install -y m4
RUN yum install -y help2man
RUN yum install -y perl-Thread-Queue perl-Data-Dumper
RUN (cd watchman;./autogen.sh)

This initially (a month ago?) failed due to cpp: too many input files while compiling boost. I never got any further.

Today as I wanted to report this, that error was gone, but a long time later it fails with:

‘TCP_ULP’ was not declared in this scope

This happens while compiling fizz, and has been reported here 3 month ago:

• Build failure: ‘TCP_ULP’ was not declared in this scope facebookincubator/fizz#85

As I already spent more than a day on this, and it was super slow, blew up my container size, and depended on like a million external dependencies, all of which had their own problems, I gave up at this point. This is a complete nightmare, and I don't want this in my build pipeline.

In the end, I extracted the binaries from the working container which luckily I had not deleted yet, and checked them into my Git repo:

COPY watchman/* /usr/bin/

This is the only thing that ever worked, and it's only 244KB. I have no idea where these binaries come from, and how they were built. But by now, I don't care anymore.

The full Dockerfile and a small test script is available here:

• https://github.com/marc-guenther/watchman-amazonlinux-problems

Lessons learned:

• Never ever depend on external resources, as they will suddenly change or completely vanish. Check everything into version control or some local artifact repository manager under your control. If your build depends on a working internet connection, then you are screwed sooner or later.
• Don't trust a software that doesn't do stable releases. Seriously, what?
• Apparently, building a 228k binary requires downloading half the internet, installing two dozen build tools from scratch and even more completely unrelated libraries and what not, fails in a myriad of unexpected ways, and in the end the whole process is more complex than compiling the Linux kernel.

[trey-jones]

I agree with you. I'm still using v4.9.0 because building the weekly releases seems to be more trouble than I have time for. It seems very locked into "works on my machine" at this point. As of 8/2023 getting 504 Gateway Timeout just trying to download Boost via autogen... Surely it could be easier for non-maintainers to make a build...

I know this isn't a helpful comment, I just don't know where else to offer feedback. Watchman was really good about 3 years ago.

[chadaustin]

The Linux binary release isn't guaranteed compatible with every distribution. It has libc version requirements, and may only work on a range of Ubuntu releases, as you found.

We don't have the bandwidth to support builds for the long tail of Linux distributions, but I wonder if you could play with the release actions and see if you can get it to build an Amazonlinux-compatible release in a Docker image:

https://github.com/facebook/watchman/blob/main/.github/workflows/generate-release-yml.rs

https://github.com/facebook/watchman/tree/main/watchman/build/package