exec: "curl": executable file not found in $PATH: unknown

[thuybt]

How can I config this with airflow for automation,
Furthermore, I have some issue with curl when exec post command

[PardhuMadipalli]

I am facing this issue too.

[teplydat]

"Furthermore, I have some issue with curl when exec post command"
I have the same issue. So basically the example "Run Spark Job" (https://github.com/JahstreetOrg/spark-on-kubernetes-helm#run-spark-job) does not work.

I tried to install curl but I got errors like:

apt update

E: Repository 'https://deb.debian.org/debian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'

apt install curl -y

E: Failed to fetch https://deb.debian.org/debian/pool/main/o/openldap/libldap-common_2.4.47+dfsg-3+deb10u2_all.deb  404  Not Found [IP: 151.101.14.132 443]

I already have problems to install curl in the first base image sasnouskikh/spark:3.0.1_2.12-hadoop_3.2.0_cloud somehow apt is locked already:

RUN apt-get update -y

E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)

#solved by
USER root
#but then same error like above

@jahstreet
What's wrong and how can I fix that? I mean what's the point of giving a curl example if curl is not installed in livy container...

@jahstreet also another question: is that Livy compatible with Spark 3.3.1?

Any hints are welcome :)

[teplydat]

Okay, I found a workaround regarding that curl problem:

kubectl exec -it --namespace livy livy-0 -- bash
apt update --allow-releaseinfo-change
apt install curl -y
curl --version

or permanent via Dockerfile:

FROM sasnouskikh/livy:0.8.0-incubating-spark_3.0.1_2.12-hadoop_3.2.0_cloud

RUN apt-get update --allow-releaseinfo-change
RUN apt install curl -y

[jahstreet]

Hi guys, sorry for the delayed responses...

How can I config this with airflow for automation

Some time ago I've written about that in Stackoverflow. Unless the new Airflow operators has been released since then it still should be actual info.

I have some issue with curl when exec post command
Okay, I found a workaround regarding that curl problem

Thx for checking. Could you please suggest the PR with the fix? Would really appreciate the contribution.

also another question: is that Livy compatible with Spark 3.3.1?

I'm not sure, probably not fully and we would need to rebuild it with the proper support of Spark 3.x. Going to give it a closer look in the coming months since getting back to the project related activities.

[teplydat]

Thx for checking. Could you please suggest the PR with the fix? Would really appreciate the contribution.
@jahstreet
Sorry for the delay. Here is the PR to fix it: JahstreetOrg/spark-on-kubernetes-docker#19

[lhoss]

also another question: is that Livy compatible with Spark 3.3.1?

I guess @teplydat meant Spark 3.1.1. Ourselves would also prefer to have a build using 3.1.1 (same version currently used in the spark k8s operator, we would like to integrate with @jahstreet 's Livy deployment)