Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update docker images and GHA to use miniforge #740

Merged
merged 8 commits into from
Aug 30, 2024
Merged

Update docker images and GHA to use miniforge #740

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
205b08d
update conda GHAs to miniforge
jashapiro Aug 30, 2024
59703b1
update ewings dockerfile to miniforge
jashapiro Aug 30, 2024
e1f2dd5
update doublet detection and docker docs
jashapiro Aug 30, 2024
b52bcab
Merge remote-tracking branch 'upstream/main' into 711-713-docker-gha-…
jashapiro Aug 30, 2024
60b1b2d
Make dockerfiles more uniform
jashapiro Aug 30, 2024
c4248c7
update installation link
jashapiro Aug 30, 2024
6317655
Merge remote-tracking branch 'upstream/main' into 711-713-docker-gha-…
jashapiro Aug 30, 2024
b87bdae
Add miniforge to dict
jashapiro Aug 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/run_hello-python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ jobs:

- name: Set up conda
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-version: latest

- name: Install and activate locked conda environment
run: |
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/run_metacells.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ jobs:

- name: Set up conda
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-version: latest

- name: Install and activate locked conda environment
run: |
Expand Down
28 changes: 15 additions & 13 deletions analyses/cell-type-ewings/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ ENV RENV_CONFIG_CACHE_ENABLED=FALSE
# set a name for the conda environment
ARG ENV_NAME=openscpca-cell-type-ewings

# set environment variables to install miniconda
# set environment variables to install conda
ENV PATH="/opt/conda/bin:${PATH}"

# Install dependencies for renv library
Expand All @@ -27,18 +27,20 @@ RUN apt-get -y update && \
wget \
&& rm -rf /var/lib/apt/lists/*

# Install miniconda
# adapted from https://github.com/ContinuumIO/docker-images/blob/main/miniconda3/debian/Dockerfile
RUN curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& bash Miniconda3-latest-Linux-x86_64.sh -b -p /opt/conda \
&& rm -f Miniconda3-latest-Linux-x86_64.sh \
&& ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh \
&& echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc \
&& find /opt/conda/ -follow -type f -name '*.a' -delete \
&& find /opt/conda/ -follow -type f -name '*.js.map' -delete
# Install conda via miniforge
# adapted from https://github.com/conda-forge/miniforge-images/blob/master/ubuntu/Dockerfile
RUN curl -L "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" -o /tmp/miniforge.sh \
&& bash /tmp/miniforge.sh -b -p /opt/conda \
&& rm -f /tmp/miniforge.sh \
&& conda clean --tarballs --index-cache --packages --yes \
&& find /opt/conda -follow -type f -name '*.a' -delete \
&& find /opt/conda -follow -type f -name '*.pyc' -delete \
&& conda clean --force-pkgs-dirs --all --yes \
&& echo ". /opt/conda/etc/profile.d/conda.sh && conda activate base" >> /etc/skel/.bashrc \
&& echo ". /opt/conda/etc/profile.d/conda.sh && conda activate base" >> ~/.bashrc

# add conda-lock
RUN conda install conda-lock
RUN conda install conda-lock && conda clean --all --yes

# Install renv to enable later package installation
RUN Rscript -e "install.packages('renv')"
Expand All @@ -47,8 +49,8 @@ RUN Rscript -e "install.packages('renv')"
COPY conda-lock.yml conda-lock.yml

# restore from conda-lock.yml file and clean up to reduce image size
RUN conda-lock install -n ${ENV_NAME} conda-lock.yml && \
conda clean --all --yes
RUN conda-lock install -n ${ENV_NAME} conda-lock.yml \
&& conda clean --all --yes

# Activate conda environment on bash launch
RUN echo "conda activate ${ENV_NAME}" >> ~/.bashrc
Expand Down
28 changes: 15 additions & 13 deletions analyses/doublet-detection/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,21 +11,23 @@ ARG ENV_NAME=openscpca-doublet-detection
# Set an environment variable to allow checking if we are in an OpenScPCA container
ENV OPENSCPCA_DOCKER=TRUE

# set environment variables to install miniconda
# set environment variables to install conda
ENV PATH="/opt/conda/bin:${PATH}"

# Install Miniconda
# adapted from https://github.com/ContinuumIO/docker-images/blob/main/miniconda3/debian/Dockerfile
RUN curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& bash Miniconda3-latest-Linux-x86_64.sh -b -p /opt/conda \
&& ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh \
&& echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc \
&& rm -f Miniconda3-latest-Linux-x86_64.sh \
&& find /opt/conda/ -follow -type f -name '*.a' -delete \
&& find /opt/conda/ -follow -type f -name '*.js.map' -delete
# Install conda via miniforge
# adapted from https://github.com/conda-forge/miniforge-images/blob/master/ubuntu/Dockerfile
RUN curl -L "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" -o /tmp/miniforge.sh \
&& bash /tmp/miniforge.sh -b -p /opt/conda \
&& rm -f /tmp/miniforge.sh \
&& conda clean --tarballs --index-cache --packages --yes \
&& find /opt/conda -follow -type f -name '*.a' -delete \
&& find /opt/conda -follow -type f -name '*.pyc' -delete \
&& conda clean --force-pkgs-dirs --all --yes \
&& echo ". /opt/conda/etc/profile.d/conda.sh && conda activate base" >> /etc/skel/.bashrc \
&& echo ". /opt/conda/etc/profile.d/conda.sh && conda activate base" >> ~/.bashrc

# Install conda-lock
RUN conda install --channel=conda-forge --name=base conda-lock
RUN conda install --channel=conda-forge --name=base conda-lock && conda clean --all --yes

# Install pandoc dependency
RUN apt-get -y update && \
Expand All @@ -44,8 +46,8 @@ ENV RENV_CONFIG_CACHE_ENABLED=FALSE
COPY conda-lock.yml conda-lock.yml

# restore from conda-lock.yml file and clean up to reduce image size
RUN conda-lock install -n ${ENV_NAME} conda-lock.yml && \
conda clean --all --yes
RUN conda-lock install -n ${ENV_NAME} conda-lock.yml \
&& conda clean --all --yes

# Copy the renv.lock file from the host environment to the image
COPY renv.lock renv.lock
Expand Down
28 changes: 15 additions & 13 deletions docs/ensuring-repro/docker/docker-images.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,21 +147,23 @@ FROM bioconductor/r-ver:3.19
# set a name for the conda environment
ARG ENV_NAME=openscpca-analysis

# set environment variables to install miniconda
# set environment variables to install conda
ENV PATH="/opt/conda/bin:${PATH}"

# Install Miniconda
# adapted from https://github.com/ContinuumIO/docker-images/blob/main/miniconda3/debian/Dockerfile
RUN curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& bash Miniconda3-latest-Linux-x86_64.sh -b -p /opt/conda \
&& ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh \
&& echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc \
&& rm -f Miniconda3-latest-Linux-x86_64.sh \
&& find /opt/conda/ -follow -type f -name '*.a' -delete \
&& find /opt/conda/ -follow -type f -name '*.js.map' -delete
# Install conda via miniforge
# adapted from https://github.com/conda-forge/miniforge-images/blob/master/ubuntu/Dockerfile
RUN curl -L "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" -o /tmp/miniforge.sh \
&& bash /tmp/miniforge.sh -b -p /opt/conda \
&& rm -f /tmp/miniforge.sh \
&& conda clean --tarballs --index-cache --packages --yes \
&& find /opt/conda -follow -type f -name '*.a' -delete \
&& find /opt/conda -follow -type f -name '*.pyc' -delete \
&& conda clean --force-pkgs-dirs --all --yes \
&& echo ". /opt/conda/etc/profile.d/conda.sh && conda activate base" >> /etc/skel/.bashrc \
&& echo ". /opt/conda/etc/profile.d/conda.sh && conda activate base" >> ~/.bashrc

# Install conda-lock
RUN conda install --channel=conda-forge --name=base conda-lock
RUN conda install --channel=conda-forge --name=base conda-lock && conda clean --all --yes
Copy link
Member

@sjspielman sjspielman Aug 30, 2024 •
edited by jashapiro
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just spacing

Suggested change
RUN conda install --channel=conda-forge --name=base conda-lock && conda clean --all --yes
RUN conda install --channel=conda-forge --name=base conda-lock \
&& conda clean --all --yes


# Install renv
RUN Rscript -e "install.packages('renv')"
Expand All @@ -173,8 +175,8 @@ ENV RENV_CONFIG_CACHE_ENABLED FALSE
COPY conda-lock.yml conda-lock.yml

# restore from conda-lock.yml file and clean up to reduce image size
RUN conda-lock install -n ${ENV_NAME} conda-lock.yml && \
conda clean --all --yes
RUN conda-lock install -n ${ENV_NAME} conda-lock.yml \
&& conda clean --all --yes

# Copy the renv.lock file from the host environment to the image
COPY renv.lock renv.lock
Expand Down
2 changes: 2 additions & 0 deletions templates/workflows/run_conda-module.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ jobs:
- name: Set up conda
# Note that this creates and activates an environment named 'test' by default
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-version: latest

- name: Install and activate locked conda environment
run: |
Expand Down
2 changes: 2 additions & 0 deletions templates/workflows/run_conda-renv-module.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ jobs:
- name: Set up conda
# Note that this creates and activates an environment named 'test' by default
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-version: latest

- name: Install conda-lock and activate locked conda environment
run: |
Expand Down
Loading