From 1803e9536fb2f53112472604737d6d8ff51b26a5 Mon Sep 17 00:00:00 2001 From: Charl Smit Date: Thu, 11 Apr 2024 12:03:10 +0200 Subject: [PATCH 1/5] Generate windows exe --- .github/workflows/release_actions.yml | 28 +++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/.github/workflows/release_actions.yml b/.github/workflows/release_actions.yml index cf7f7f8..b195197 100644 --- a/.github/workflows/release_actions.yml +++ b/.github/workflows/release_actions.yml @@ -4,8 +4,8 @@ on: types: [published] jobs: - generate_release_assets: - name: Generate release assets + generate_linux_bin: + name: Generate Linux binary as release asset runs-on: ubuntu-22.04 steps: - name: Checkout repository @@ -24,3 +24,27 @@ jobs: with: files: "./dist/linux/*" repo-token: ${{ secrets.GITHUB_TOKEN }} + + generate_windows_exe: + name: Generate Windows exe as release asset + runs-on: windows-latest + steps: + - name: Checkout repository + uses: actions/checkout@v2 + + - name: Install pyinstaller + shell: pwsh + run: python -m pip install pyinstaller + + - name: Generate exe + shell: pwsh + run: | + pip install commcare-export + pip install -r build_exe/requirements.txt + pyinstaller --dist ./dist/windows commcare-export.spec + + - name: Upload release assets + uses: AButler/upload-release-assets@v3.0 + with: + files: "./dist/windows/*" + repo-token: ${{ secrets.GITHUB_TOKEN }} From fc2a7d19792150488aa4a1c5d371d5eade2aba84 Mon Sep 17 00:00:00 2001 From: Charl Smit Date: Thu, 11 Apr 2024 12:05:57 +0200 Subject: [PATCH 2/5] Don't use docker image for pyinstaller --- .github/workflows/release_actions.yml | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/.github/workflows/release_actions.yml b/.github/workflows/release_actions.yml index b195197..d4fee0b 100644 --- a/.github/workflows/release_actions.yml +++ b/.github/workflows/release_actions.yml @@ -11,13 +11,16 @@ jobs: - name: Checkout repository uses: actions/checkout@v2 - - name: Pull pyinstaller docker image - run: | - docker pull dimagi/commcare-export-pyinstaller-linux + - name: Install pyinstaller + shell: bash + run: python -m pip install pyinstaller - - name: Compile linux binary + - name: Generate exe + shell: bash run: | - docker run -v "$(pwd):/src/" dimagi/commcare-export-pyinstaller-linux + pip install commcare-export + pip install -r build_exe/requirements.txt + pyinstaller --dist ./dist/linux commcare-export.spec - name: Upload release assets uses: AButler/upload-release-assets@v3.0 From bb49f901c14da97a3159d4abaabcf608266c3662 Mon Sep 17 00:00:00 2001 From: Charl Smit Date: Thu, 11 Apr 2024 12:13:17 +0200 Subject: [PATCH 3/5] Remove docker-related files and update README --- build_exe/README.md | 69 +++++++--------------------- build_exe/linux/Dockerfile-py3-amd64 | 59 ------------------------ build_exe/linux/entrypoint-linux.sh | 14 ------ 3 files changed, 16 insertions(+), 126 deletions(-) delete mode 100644 build_exe/linux/Dockerfile-py3-amd64 delete mode 100644 build_exe/linux/entrypoint-linux.sh diff --git a/build_exe/README.md b/build_exe/README.md index afebee0..3cbbcbb 100644 --- a/build_exe/README.md +++ b/build_exe/README.md @@ -1,54 +1,17 @@ # Compiling DET to running executable -This folder contains relevant files needed (dockerfiles and scripts) for compiling the DET into an executable file. -The file structure is segmented into the different operating systems the resultant executable will -be compatible on. - -(Currently only Linux is supported; Windows coming soon) - - -## How it works -In order to compile the DET script into a working executable we use [pyinstaller](https://github.com/pyinstaller/pyinstaller) in a containerized -environment. The dockerfile is an edited version from [cdrx/docker-pyinstaller](https://github.com/cdrx/docker-pyinstaller) -which is slightly modified to suit our use-case. - -When a new release of the DET is published, a workflow is triggered which automatically compiles an executable from the latest -code using the custom built docker image, `dimagi/commcare-export-pyinstaller-linux`, then uploads it to the release as an asset. - -If you ever have to compile the executable yourself you can follow the section below, *Compiling executable files locally*, on how to compile an executable locally. - - -Compiling executable files locally ------------------------------------ -The DET executable files are compiled using a tool called [pyinstaller](https://pyinstaller.org/en/stable/). -Pyinstaller is very easy to use, but only works out-of-the-box for Linux as support for cross-compilation was -dropped in earlier releases. Another tool, [wine](https://www.winehq.org/), can be used in conjuction with -pyinstaller to compile the Windows exe files (not yet supported). - -Luckily in the world we live containerization is a thing. We use a docker container, `dimagi/commcare-export-pyinstaller-linux` -(based on [docker-pyinstaller](https://github.com/cdrx/docker-pyinstaller)), which allows you to seamlessly compile the Linux binary, so we don't ever have to worry about installing any additional packages ourselves. - -To compile a new linux binary, first make sure you have the docker image used to generate the executable: -> docker pull dimagi/commcare-export-pyinstaller-linux:latest - -Now it's really as simple as running -> docker run -v "$(pwd):/src/" dimagi/commcare-export-pyinstaller-linux - -Once you're done, the compiled file can be located at `./dist/linux/commcare-export`. - -The tool needs two files to make the process work: -1. `commcare-export.spec`: this file is used by `pyinstaller` and is already defined and sits at the top of this project. -It shouldn't be necessary for you to change any parameters in the file. -2. `requirements.txt`: this file lists all the necessary packages needed for running commcare-export. - - -## Updating the docker image -Are you sure you need to update the image? - -Just checking... - - -If it's needed to make any changes (for whatever reason) to the docker image you can rebuild the image as follows: -> docker build -f ./build_exe/linux/Dockerfile-py3-amd64 -t dimagi/commcare-export-pyinstaller-linux:latest . - -Now upload the new image to dockerhub (remember to log in to the account first!): -> docker image push dimagi/commcare-export-pyinstaller-linux:latest +This folder contains relevant files needed for compiling the DET into an executable file. +The executable is generated on after every release of the DET and the resultant files are uploaded +to the release as assets. + +## Testing locally +In the event that you want to test the exe compilation locally you can simply run +> pip install -r build_exe/requirements.txt + +Now create the executable (if you're running this on a Linux machine you can only compile the binary for this type of +OS) +> pyinstaller --dist ./dist/linux commcare-export.spec + +The resultant executable file can be located under `./dist/linux/`. + +Note that the argument `commcare-export.spec`. This is a simple configuration file used by +pyinstaller which you ideally shouldn't have to ever change. \ No newline at end of file diff --git a/build_exe/linux/Dockerfile-py3-amd64 b/build_exe/linux/Dockerfile-py3-amd64 deleted file mode 100644 index 1c33b42..0000000 --- a/build_exe/linux/Dockerfile-py3-amd64 +++ /dev/null @@ -1,59 +0,0 @@ -FROM ubuntu:20.04 -SHELL ["/bin/bash", "-i", "-c"] - -ARG PYTHON_VERSION=3.9.18 -ARG PYINSTALLER_VERSION=6.4 - -ENV PYPI_URL=https://pypi.python.org/ -ENV PYPI_INDEX_URL=https://pypi.python.org/simple -ENV PYENV_VERSION=${PYTHON_VERSION} - -COPY ./build_exe/linux/entrypoint-linux.sh /entrypoint.sh - -RUN \ - set -x \ - # update system - && apt-get update \ - # install requirements - && apt-get install -y --no-install-recommends \ - build-essential \ - ca-certificates \ - curl \ - wget \ - git \ - libbz2-dev \ - libreadline-dev \ - libsqlite3-dev \ - libssl-dev \ - zlib1g-dev \ - libffi-dev \ - # required because openSSL on Ubuntu 12.04 and 14.04 run out of support versions of OpenSSL - && mkdir openssl \ - && cd openssl \ - # latest version, there won't be anything newer for this - && wget https://www.openssl.org/source/openssl-1.0.2u.tar.gz \ - && tar -xzvf openssl-1.0.2u.tar.gz \ - && cd openssl-1.0.2u \ - && ./config --prefix=$HOME/openssl --openssldir=$HOME/openssl shared zlib \ - && make \ - && make install \ - # install pyenv - && echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc \ - && echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc \ - && source ~/.bashrc \ - && curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash \ - && echo 'eval "$(pyenv init -)"' >> ~/.bashrc \ - && source ~/.bashrc \ - # install python - && PATH="$HOME/openssl:$PATH" CPPFLAGS="-O2 -I$HOME/openssl/include" CFLAGS="-I$HOME/openssl/include/" LDFLAGS="-L$HOME/openssl/lib -Wl,-rpath,$HOME/openssl/lib" LD_LIBRARY_PATH=$HOME/openssl/lib:$LD_LIBRARY_PATH LD_RUN_PATH="$HOME/openssl/lib" CONFIGURE_OPTS="--with-openssl=$HOME/openssl" PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install $PYTHON_VERSION \ - && pyenv global $PYTHON_VERSION \ - && pip install --upgrade pip \ - # install pyinstaller - && pip install pyinstaller==$PYINSTALLER_VERSION \ - && mkdir /src/ \ - && chmod +x /entrypoint.sh - -VOLUME /src/ -WORKDIR /src/ - -ENTRYPOINT ["/entrypoint.sh"] diff --git a/build_exe/linux/entrypoint-linux.sh b/build_exe/linux/entrypoint-linux.sh deleted file mode 100644 index 5d781fb..0000000 --- a/build_exe/linux/entrypoint-linux.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/bin/bash -i - -# Fail on errors. -set -e - -# Make sure .bashrc is sourced -. /root/.bashrc - -cd /src - -pip install commcare-export -pip install -r build_exe/requirements.txt - -pyinstaller --clean -y --dist ./dist/linux --workpath /tmp *.spec From 2c936d449ff61c4ee4b2bc68674ae55ba1819eeb Mon Sep 17 00:00:00 2001 From: Charl Smit Date: Thu, 11 Apr 2024 13:36:47 +0200 Subject: [PATCH 4/5] Update README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 19a0fd8..b9ace25 100644 --- a/README.md +++ b/README.md @@ -645,10 +645,10 @@ https://pypi.python.org/pypi/commcare-export https://github.com/dimagi/commcare-export/releases -Once the release is published a GitHub workflow is kicked off that compiles an executable of the DET compatible with -running on a Linux machine (Windows coming soon), adding it as a release asset. +Once the release is published a GitHub workflow is kicked off that compiles executables of the DET compatible with +Linux and Windows machines, adding it to the release as assets. -If you decide to download and use the executable file, please make sure the file has the executable permission enabled, +[For Linux-based users] If you decide to download and use the executable file, please make sure the file has the executable permission enabled, after which it can be invoked like any other executable though the command line. From 6fcbfddd1a61272ae5cb8f27fdd5b4e041e92e36 Mon Sep 17 00:00:00 2001 From: Charl Smit Date: Thu, 11 Apr 2024 13:43:08 +0200 Subject: [PATCH 5/5] Fix README --- build_exe/README.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/build_exe/README.md b/build_exe/README.md index 3cbbcbb..28997f3 100644 --- a/build_exe/README.md +++ b/build_exe/README.md @@ -4,14 +4,15 @@ The executable is generated on after every release of the DET and the resultant to the release as assets. ## Testing locally -In the event that you want to test the exe compilation locally you can simply run -> pip install -r build_exe/requirements.txt +In the event that you want to test the exe compilation locally you can follow the steps below: -Now create the executable (if you're running this on a Linux machine you can only compile the binary for this type of -OS) +Install `pyinstaller`: +> python -m pip install pyinstaller + +Now create the executable (assuming you're running this on a Linux machine): > pyinstaller --dist ./dist/linux commcare-export.spec The resultant executable file can be located under `./dist/linux/`. -Note that the argument `commcare-export.spec`. This is a simple configuration file used by +The argument, `commcare-export.spec`, is a simple configuration file used by pyinstaller which you ideally shouldn't have to ever change. \ No newline at end of file