Streamline dependencies of docker CI images #951
Conversation
|
Is it necessary to create an extra stage like HDF5 and MOAB, given that these are also external dependencies? I think we can remove these and instead update our workflow to create a binary/final stage to store only the binaries with minimal size, and we may choose the tag as dagmc. |
|
|
||
| ARG INSTALL_DIR | ||
|
|
||
| COPY --from=dagmc_test ${INSTALL_DIR} ${INSTALL_DIR} |
There was a problem hiding this comment.
What files are you copying here and why? If they aren't installed by make install then they probably aren't necessary.
There was a problem hiding this comment.
In this step, I am using an empty image to copy all the installed files from the DAGMC step to the exact location. Therefore, in this image, we only contain the files available in /opt/*. There are no extra files or dependencies in this new image, which keeps the image size minimal, less than 90MB.
There was a problem hiding this comment.
In this step we have, DAGMC, MOAB, HDF5, Double Down, Embree, Geant4. However, We can make it more minimal by copying only /opt/dagmc folder. I have chosen to copy the full folder as I am not sure if they are core dependency of DAGMC or only necessary to build DAGMC since we have all the binaries available that works without them.
There was a problem hiding this comment.
By applying this step, we can update our docker_publish workflow to add a new tag as dagmc:latest upon PR merge, and dagmc:version/stable upon new release. This can help streamline other projects as well, such as PyNE and NukeLab. We can also update our documentations to use the docker version of DAGMC as well for the general users. Let me know if you find the idea useful.
There was a problem hiding this comment.
Ah! You've already merged this PR. I was waiting for your response. The binary stage won't run until we update the docker_publish file. There are currently no instructions for this. I think I should create a new PR for it.
I think these stages mostly appear for historical reasons to limit the amount of rebuilding of images that must occur. The philosophy is that things that occur earlier in the Dockerfile change least often and least under our control. Thus, having them installed in early as a different stage can mean fewer instances of rebuilding those sections of the file. GEANT4 is the most resource intensive to build so one of the main things to avoid. |
I understand. We've included these stages to minimize the need for rebuilding HDF5 and MOAB whenever we make changes to the Dockerfile. For instance, if we modify the MOAB part, Docker will trace back to the HDF5 stage and then build MOAB. Currently, GitHub workflow supports a full chase log. This means that the steps where we make changes in the Dockerfile will be traced accordingly. |
| cd build && \ | ||
| ../hdf5-${HDF5_VERSION}/configure --enable-shared \ | ||
| --prefix=${hdf5_install_dir} \ | ||
| ../hdf5/configure --enable-shared \ |
There was a problem hiding this comment.
@ahnaf-tahmid-chowdhury - are you familiar with configure and autotools? The tarball that we retrieve with wget has the configure file already generated. If we pull from the repo, we need to do those steps ourselves. Do they support a CMake based build? That might be more generalizable.
There was a problem hiding this comment.
Not that much as I know cmake. I think autotools are based on autoconf similar to cmake. Handle things in an easier way than cmake.
There was a problem hiding this comment.
I have noticed we have installed autoconf and libtool thus I haven't tried with cmake for this step. I think I can give this a try. To make it more simplified.
Description
This PR aims to optimize the dependencies of the docker CI images by:
git clonefrom the source.git cloneto download dependencies, removing the dependency onwget.Impact
These changes are expected to streamline the dependencies of the docker CI images, reducing the image size and potentially improving performance.
Related Issues
Closes #750.