Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pyspark is not required when install koalas #2221

Open
bingwork opened this issue May 13, 2022 · 0 comments
Open

pyspark is not required when install koalas #2221

bingwork opened this issue May 13, 2022 · 0 comments

Comments

@bingwork
Copy link

koalas is a great package.

when I install the package, all requirements are as below:
pip install koalas==1.8.2 Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting koalas==1.8.2 Using cached https://pypi.tuna.tsinghua.edu.cn/packages/28/9a/d69cf12ea62116873b427e5843be8ae8431b18f2a0714d6f4eec3ee4cda6/koalas-1.8.2-py3-none-any.whl (390 kB) Requirement already satisfied: numpy>=1.14 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from koalas==1.8.2) (1.21.5) Requirement already satisfied: pandas>=0.23.2 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from koalas==1.8.2) (1.3.5) Requirement already satisfied: pyarrow>=0.10 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from koalas==1.8.2) (7.0.0) Requirement already satisfied: pytz>=2017.3 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from pandas>=0.23.2->koalas==1.8.2) (2021.1) Requirement already satisfied: python-dateutil>=2.7.3 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from pandas>=0.23.2->koalas==1.8.2) (2.8.2) Requirement already satisfied: six>=1.5 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas>=0.23.2->koalas==1.8.2) (1.16.0) Installing collected packages: koalas Successfully installed koalas-1.8.2

but it also needs pyspark, for example when I start a service:
`ImportError: Unable to import pyspark - consider doing a pip install with [spark] extra to install pyspark with pip
Traceback (most recent call last):
File "/Users/celential-bing/.pyenv/versions/time_machine/lib/python3.8/site-packages/databricks/koalas/init.py", line 49, in assert_pyspark_version
import pyspark
ModuleNotFoundError: No module named 'pyspark'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/celential-bing/time-machine/timemachine/app.py", line 1, in
from timemachine import app, dapp
File "/Users/celential-bing/time-machine/timemachine/init.py", line 301, in
app, dapp, schema = create_app()
File "/Users/celential-bing/time-machine/timemachine/init.py", line 57, in create_app
raise e
File "/Users/celential-bing/time-machine/timemachine/init.py", line 54, in create_app
return TimeMachineInitializer(app).init_app()
File "/Users/celential-bing/time-machine/timemachine/init.py", line 212, in init_app
self.init_app_in_ctx()
File "/Users/celential-bing/time-machine/timemachine/init.py", line 176, in init_app_in_ctx
self.init_views()
File "/Users/celential-bing/time-machine/timemachine/init.py", line 69, in init_views
from timemachine.views.base import (
File "/Users/celential-bing/time-machine/timemachine/views/base.py", line 13, in
from timemachine.models.base import Module, Lambda
File "/Users/celential-bing/time-machine/timemachine/models/base.py", line 14, in
from timemachine.engines import current_engine, DF
File "/Users/celential-bing/time-machine/timemachine/engines/init.py", line 9, in
from databricks.koalas import DataFrame as SparkDataFrame
File "/Users/celential-bing/.pyenv/versions/time_machine/lib/python3.8/site-packages/databricks/koalas/init.py", line 72, in
assert_pyspark_version()
File "/Users/celential-bing/.pyenv/versions/time_machine/lib/python3.8/site-packages/databricks/koalas/init.py", line 51, in assert_pyspark_version
raise ImportError(
ImportError: Unable to import pyspark - consider doing a pip install with [spark] extra to install pyspark with pip`

so I suggest adding the pyspark in the requirement.txt. I didn't find the file, so pull an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant