Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support python3.11 #130

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Support python3.11 #130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:

strategy:
matrix:
python-version: [3.8, 3.9]
python-version: [3.8, 3.9, 3.11]

steps:
- uses: actions/checkout@v2
Expand All @@ -53,22 +53,26 @@ jobs:
standalone_spark3_with_S3:
runs-on: ubuntu-latest

strategy:
matrix:
python-version: [ 3.9, 3.11 ]

steps:
- uses: actions/checkout@v2

- name: Build spark-docker
run: docker build -t spark-docker ./examples/spark-with-S3 --build-arg SPARK_INPUT_VERSION=3.2.2 --build-arg PYTHON_VERSION=3.9.15
run: docker build -t spark-docker ./examples/spark-with-S3 --build-arg SPARK_INPUT_VERSION=3.2.2 --build-arg PYTHON_VERSION=${{ matrix.python-version }}

- name: Build the docker-compose stack
run: |
export PYTHON_VERSION=3.9
export PYTHON_VERSION=${{ matrix.python-version }}
docker compose -f ./examples/spark-with-S3/docker-compose.yml up -d

- name: Check running containers
run: docker ps -a

- name: Run spark Job
run: docker exec spark-master ./examples/spark-with-S3/scripts/run_spark_example.sh python3.9 3.2.2
run: docker exec spark-master ./examples/spark-with-S3/scripts/run_spark_example.sh python${{ matrix.python-version }} 3.2.2

# hadoop_hdfs:
# runs-on: ubuntu-latest
Expand Down Expand Up @@ -107,7 +111,7 @@ jobs:

strategy:
matrix:
python-version: [3.9]
python-version: [3.9, 3.11]

steps:
- uses: actions/checkout@v2
Expand Down
21 changes: 11 additions & 10 deletions examples/spark-with-S3/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,17 @@ ARG SPARK_INPUT_VERSION=3.2.2
ARG PYTHON_VERSION=3.9.15

# Python
RUN apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
libreadline-dev libsqlite3-dev wget curl unzip llvm libncurses5-dev libncursesw5-dev \
xz-utils tk-dev libffi-dev liblzma-dev
RUN curl https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz --output /usr/python.tgz
RUN tar xf /usr/python.tgz -C /usr && rm /usr/python.tgz
RUN cd /usr/Python-${PYTHON_VERSION} \
&& ./configure --enable-shared \
&& make -j8 \
&& make altinstall \
&& cd -
RUN apt-get install -y make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev \
libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
RUN curl https://pyenv.run | bash

ENV PATH=/root/.pyenv/bin:$PATH
RUN eval "$(pyenv init -)"
RUN eval "$(pyenv virtualenv-init -)"

RUN pyenv install ${PYTHON_VERSION}
RUN pyenv global ${PYTHON_VERSION}

# Java
RUN apt-get install -y openjdk-11-jre \
Expand Down
Loading