Skip to content

Commit

Permalink
Dockerfile and docker compose updates (NationalSecurityAgency#965)
Browse files Browse the repository at this point in the history
  • Loading branch information
DonResnik authored Oct 11, 2024
1 parent e12b323 commit 2576ceb
Show file tree
Hide file tree
Showing 9 changed files with 158 additions and 254 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/maven-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
with:
context: .
push: false
file: contrib/docker/Dockerfile.ci
file: contrib/docker/Dockerfile
build-args: |
java_version=${{ matrix.java-version }}
java_compiler=${{ matrix.java-compiler }}
Expand All @@ -77,7 +77,7 @@ jobs:
with:
context: .
push: false
file: contrib/docker/Dockerfile.ci
file: contrib/docker/Dockerfile
build-args: |
target_os=centos7
verify_build=false
Expand Down
37 changes: 24 additions & 13 deletions DEVELOPING.md
Original file line number Diff line number Diff line change
Expand Up @@ -615,12 +615,18 @@ Alternatively, we can use Docker directly. First run a full maven build and then
mvn clean install -Pdist
docker build -f contrib/docker/Dockerfile . -t emissary
```
There are three os profiles in the Dockerfile that can be used to build Emissary docker
- default - ubi8 docker image
- centos7 - centos7 docker image
- alpine3 - alpine3 docker image

For a lightweight emissary docker image the Dockerfile.alpine can be used.
This image is 1/3 of the size of the normal centOS 7 based image
centos7 and alpine images can be selected by using ```--build-arg target_os=[centos7|apline3]``` in the ``docker build``` command

For example, a lightweight emissary docker can use the alpine3 image from the Dockerfile.
This image is much smaller than the Dockerfile default ubi8 based image
```
mvn clean install -Pdist
docker build -f contrib/docker/Dockerfile.alpine . -t emissary-light
docker build -f contrib/docker/Dockerfile . --build-arg target_os=alpine3 -t emissary-light
```

### Run Emissary with Docker
Expand Down Expand Up @@ -648,9 +654,14 @@ Emissary, run the sample command:
docker run -it --rm -v ${PWD}/target/data:/opt/emissary/target/data:Z -v ${PWD}/target/localoutput:/opt/emissary/localoutput:Z --name emissary emissary
```

If you are running into permission issues on Linux, an option is to run the container with your uid and gid by adding
If you are running into permission issues on Linux (which sometimes presents as an error related to moving files to the 'target/data/HoldData/' directory), an option is to run the container with your uid and gid by adding
```--user $(id -u):$(id -g)``` to the above command.

So it looks like this:
```
docker run -it --rm --user $(id -u):$(id -g) -v ${PWD}/target/data:/opt/emissary/target/data:Z -v ${PWD}/target/localoutput:/opt/emissary/localoutput:Z --name emissary emissary
```

Once Emissary starts up, we should see a log line that says: "Started EmissaryServer at http://localhost:8001." We now
can copy files into the input directory for Emissary to process:

Expand Down Expand Up @@ -690,36 +701,36 @@ docker run -it --rm --name emissary --hostname emissary-001 -p 8001:8001 emissar

Then from a browser, assuming container is running locally, go to http://localhost:8001/ to see the endpoints.

### Cluster Mode using Docker Compose
### Run Emissary in Cluster Mode using Docker Compose
We can use a Docker compose file to simulate cluster mode. We'll start a feeder and two workers by default. To start the
cluster, run the sample docker-compose.yml file. From the root of the project, run:
cluster, run the sample docker-compose.cluster.yml file. From the root of the project, run:

```
docker-compose -f contrib/docker/docker-compose.yml up
docker compose -f contrib/docker/docker-compose.cluster.yml up
```

Use docker copy to run a file through Emissary:

```
docker cp emissary-knight.png docker_emissary-feeder_1:/opt/emissary/target/data/InputData/
docker cp emissary-knight.png docker-emissary-feeder-1:/opt/emissary/target/data/InputData/
```

### Optionally Build and Test Emissary with a Docker Dev Image
### Optionally Build and Test Emissary with a Docker Alpine Image

Let's use the dev image to build Emissary with Maven and Java:
Let's use the alpine image to build Emissary with Maven and Java:
```
docker build . -t emissary:test -f contrib/docker/Dockerfile.dev
docker build . -t emissary:test --build-arg target_os=alpine3 -f contrib/docker/Dockerfile
```
Once the build succeeds, we can start a container:
```
docker run -it --rm -p 8001:8001 --hostname emissary --name emissary emissary:test
```

### Build and Run Emissary UBI8 Docker Image
### Run Emissary server with default UBI8 Docker Image and Docker Compose

From the root of the project, simply run:
```
docker compose -f contrib/docker/docker-compose.ubi8.yml up --build
docker compose -f contrib/docker/docker-compose.server.yml up --build
```
Then from a browser, assuming container is running locally, go to http://localhost:8001/ to see the endpoints.

Expand Down
142 changes: 125 additions & 17 deletions contrib/docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,28 +1,136 @@
FROM centos:7
ARG target_os=ubi8

RUN yum update -y \
&& rpm --import https://yum.corretto.aws/corretto.key \
&& curl -L -o /etc/yum.repos.d/corretto.repo https://yum.corretto.aws/corretto.repo \
&& yum install -y java-11-amazon-corretto-devel \
&& yum clean all -y \
&& rm -rf /var/cache/yum

ADD target/emissary-*-dist.tar.gz /opt

RUN ls -al /opt && \
version=`ls /opt | grep emissary- | awk '{ print $1 }'` && \
echo "Linking /opt/${version} to /opt/emissary" && \
ln -s /opt/${version} /opt/emissary && \
mkdir -p /opt/emissary/localoutput && \
chmod -R a+rw /opt/emissary
FROM redhat/ubi8:8.9 AS base_ubi8

ENV JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8
ENV PROJECT_BASE=/opt/emissary

WORKDIR /opt/emissary
ARG java_version=11
ARG user=emissary
ARG group=emissary
ARG uid=1000
ARG gid=1000
RUN dnf update -y \
&& dnf upgrade -y \
&& rpm --import https://yum.corretto.aws/corretto.key \
&& curl -L -o /etc/yum.repos.d/corretto.repo https://yum.corretto.aws/corretto.repo \
&& dnf install -y java-${java_version}-amazon-corretto-devel \
&& dnf install -y glibc-langpack-en lsof git \
&& groupadd -g ${gid} ${group} \
&& useradd -u ${uid} -g ${group} -m -s /bin/sh ${user} \
&& dnf clean all -y \
&& rm -rf /var/cache/yum

VOLUME /opt/emissary/localoutput

EXPOSE 8000 8001

FROM centos:7 AS base_centos7

ENV JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8
ENV PROJECT_BASE=/opt/emissary

ARG java_version=11
ARG user=emissary
ARG group=emissary
ARG uid=1000
ARG gid=1000
RUN sed -i s/mirrorlist.centos.org/vault.centos.org/g /etc/yum.repos.d/CentOS-*.repo \
&& sed -i s/mirror.centos.org/vault.centos.org/g /etc/yum.repos.d/CentOS-*.repo \
&& sed -i s/#baseurl/baseurl/g /etc/yum.repos.d/CentOS-*.repo \
&& yum update -y \
&& yum upgrade -y \
&& rpm --import https://yum.corretto.aws/corretto.key \
&& curl -L -o /etc/yum.repos.d/corretto.repo https://yum.corretto.aws/corretto.repo \
&& yum install -y java-${java_version}-amazon-corretto-devel \
&& yum install -y which lsof git \
&& groupadd -g ${gid} ${group} \
&& useradd -u ${uid} -g ${group} -m -s /bin/bash ${user} \
&& yum clean all -y \
&& rm -rf /var/cache/yum



FROM alpine:3 AS base_alpine3

ENV JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8
ENV PROJECT_BASE=/opt/emissary

ARG java_version=11
ARG user=emissary
ARG group=emissary
ARG uid=1000
ARG gid=1000
RUN apk update \
&& apk upgrade \
&& apk add ca-certificates \
&& update-ca-certificates \
&& apk add --update coreutils && rm -rf /var/cache/apk/* \
&& apk add --update openjdk${java_version} tzdata curl unzip bash which lsof git \
&& apk add --no-cache nss \
&& addgroup --gid ${gid} ${group} \
&& adduser --disabled-password --uid ${uid} --ingroup ${group} ${user} \
&& rm -rf /var/cache/apk/*



FROM base_${target_os} AS build

ENV MAVEN_OPTS -Xms512M -Xmx1024M -Xss1M -Djava.awt.headless=true
ENV MAVEN_HOME /opt/maven

ARG maven_version=3.9.6
RUN curl -L -o /tmp/maven.tar.gz https://dlcdn.apache.org/maven/maven-3/${maven_version}/binaries/apache-maven-${maven_version}-bin.tar.gz \
&& tar xvf /tmp/maven.tar.gz -C /opt \
&& ln -s /opt/apache-maven-${maven_version} ${MAVEN_HOME} \
&& ln -s ${MAVEN_HOME}/bin/mvn /usr/bin/mvn

COPY . ${PROJECT_BASE}
RUN chown -R ${user}:${group} ${PROJECT_BASE} \
&& chmod -R 744 ${PROJECT_BASE}

USER ${user}
WORKDIR ${PROJECT_BASE}

RUN git init --quiet \
&& git config user.email "[email protected]" \
&& git config user.name "emissary" \
&& git add . \
&& git commit --quiet -am.

ARG java_compiler=11
ARG maven_phases='clean verify'
ARG maven_profiles='-Pdist'
RUN --mount=type=cache,uid=${uid},gid=${gid},target=/home/${user}/.m2 \
mvn -V -B -e -ntp "-Dstyle.color=always" -Dmaven.compiler.release=${java_compiler} ${maven_phases} ${maven_profiles}

ARG verify_build=true
RUN if ${verify_build} ; then ./contrib/ci/detect-changes.sh ; fi



FROM base_${target_os}

COPY --from=build ${PROJECT_BASE}/target/emissary-*-dist.tar.gz /tmp

RUN tar -xf /tmp/emissary-*-dist.tar.gz -C /opt/ \
&& version=`ls /opt | grep emissary- | awk '{ print $1 }'` \
&& echo "Linking /opt/${version} to ${PROJECT_BASE}" \
&& ln -s /opt/${version} ${PROJECT_BASE} \
&& mkdir -p ${PROJECT_BASE}/localoutput \
&& mkdir -p ${PROJECT_BASE}/target/data \
&& chmod -R a+rw ${PROJECT_BASE} \
&& chown -R ${user}:${group} ${PROJECT_BASE}* \
&& rm -f /tmp/*

USER ${user}

WORKDIR ${PROJECT_BASE}

VOLUME ${PROJECT_BASE}/target/data
VOLUME ${PROJECT_BASE}/localoutput

EXPOSE 8001

ENTRYPOINT ["./emissary"]

Expand Down
31 changes: 0 additions & 31 deletions contrib/docker/Dockerfile.alpine

This file was deleted.

Loading

0 comments on commit 2576ceb

Please sign in to comment.