Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#58 Made the pytest-slc plugin using the pyexasol_connection #59

Merged
merged 7 commits into from
Sep 23, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion justfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
PROJECTS := "pytest-extension pytest-slc pytest-backend pytest-saas pytest-itde"
PROJECTS := "pytest-slc pytest-extension pytest-backend pytest-saas pytest-itde"

# Default target
default:
Expand Down
31 changes: 28 additions & 3 deletions pytest-slc/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,32 @@ pip install pytest-exasol-slc

## Usage in Tests

Below is an example of a test that requires a script language container to be built and uploaded into the database.
Note, that by default this test will run twice - once for each backend.
Below is an example of a test that requires a script language container to be built and deployed in the database.
Here, we override the default `language_alias` fixture providing a "meaningful" name. The language container will be
activated with this language alias. Note, that by default the test will run twice - once for each backend.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Here, we override the default `language_alias` fixture providing a "meaningful" name. The language container will be
activated with this language alias. Note, that by default the test will run twice - once for each backend.
The example test case overrides two fixtures that are used by fixture `deployed_slc` from this plugin:
* Fixture `language_alias` to provide a meaningful name.
* and fixture `slc_builder` to define the SLC to be used in the example test case.
The language container will be activated with value returned by fixture `language_alias`.
Note, that by default the test will run twice - once for each backend.


```python
import pytest

@pytest.fixture(scope='session')
def language_alias():
return "MY_LANGUAGE_ALIAS"

@pytest.fixture(scope='session')
def slc_builder(use_onprem, use_saas):
if use_onprem or use_saas:
with language_container_factory() as container_builder:
yield container_builder
else:
yield None

def test_something_with_slc(deployed_slc):
...
```

Alternatively, the language container can be deployed using the function version of this fixture. The function
can be called multiple times providing an opportunity to activate the language container with different
aliases.

```python
import pytest
Expand All @@ -28,6 +52,7 @@ def slc_builder(use_onprem, use_saas):
else:
yield None

def test_something_with_slc(upload_slc):
def test_something_with_slc(deploy_slc):
deploy_slc("MY_LANGUAGE_ALIAS")
...
```
2 changes: 1 addition & 1 deletion pytest-slc/doc/changes/changes_0.3.0.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# 0.3.0 - 2024-09-20
# 0.3.0 - 2024-09-23

## Summary

Expand Down
42 changes: 37 additions & 5 deletions pytest-slc/exasol/pytest_slc/__init__.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
from __future__ import annotations
from pathlib import Path
import getpass
from exasol.pytest_backend import paralleltask
import pytest

import exasol.bucketfs as bfs
from exasol.python_extension_common.deployment.language_container_deployer import LanguageContainerDeployer
from exasol.python_extension_common.deployment.language_container_deployer import (
LanguageContainerDeployer, LanguageActivationLevel)
from exasol.python_extension_common.deployment.language_container_builder import LanguageContainerBuilder

BFS_CONTAINER_DIRECTORY = 'container'
Expand Down Expand Up @@ -62,19 +64,49 @@ def export_slc(slc_builder, export_slc_async) -> Path | None:
return Path(export_info.cache_file)


@pytest.fixture(scope='session')
def language_alias(project_short_tag):
"""
The user can override this fixture if they want to provide a more meaningful
name for the language alias.
"""
owner = getpass.getuser()
return f"PYTHON3-{project_short_tag or 'EXTENSION'}-{owner}"


@pytest.fixture(scope="session")
def upload_slc(slc_builder, export_slc,
def deploy_slc(slc_builder, export_slc,
pyexasol_connection, backend_aware_bucketfs_params):
"""
The fixture provides a function uploading the language container to a database,
according to the selected backends.
The fixture provides a function deploying the language container in a database.
The function can be called multiple times, allowing to activate the container with
multiple aliases. However, the container will be uploaded only once.
"""
bucket_file_path = ''

def func(language_alias: str) -> None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, you changed it to that, such that we can isolate the aliases per test, if it necessary. I could assume, in case of the deployer tests and the extension tests that could be necessary. Or, did you think of a different reason.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the motivation is described in the related ticket #58:

If an extension test uses both pyexasol_connection fixture from the pytest-extension and upload_slc fixture from the pytest-slc then the container may not get activated for the pyexasol_connection session if the pyexasol_connection fixture is created first. To make sure this doesn't happen the pytest-slc should depend on the pytest-extension and use its pyexasol_connection fixture instead of creating its own connection.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I have some more questions:

  1. Could we make the function func use the language_alias that is already known to the slc_builder?
  2. Why does the fixture return a function instead of just uploading the SLC?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My question was related to the function and not the Pyexasol connection

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used to take the language_alias from the LanguageContainerBuilder. As we found, the language_alias is not actually required to build a language container; it is only for testing it, which we don't do. So it was removed from the LanguageContainerBuilder. But we do need it for the container deployer. Hence is the upload_slc function. We can provide the language_alias there.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But good point , maybe we need a fixture that returns a function that activates the SLC for a connection

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used to take the language_alias from the Builder ... language_alias is not actually required to build a language container ... But for the deployer. ...

Ah - very good, thank you!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision:

  • Rename fixture upload_slc to deploy_slc
  • Add fixture deployed_slc which uses a dummy fixture for alias which needs be overwritten by the extension
  • the function returned by deploy_slc fixture uses a state variable stored in deploy_slc fixture to upload the slc only once and only activate it in subsequent calls

nonlocal bucket_file_path
if (slc_builder is not None) and (export_slc is not None):
bucketfs_path = bfs.path.build_path(**backend_aware_bucketfs_params,
path=BFS_CONTAINER_DIRECTORY)
deployer = LanguageContainerDeployer(pyexasol_connection=pyexasol_connection,
bucketfs_path=bucketfs_path,
language_alias=language_alias)
deployer.run(container_file=export_slc, alter_system=True, allow_override=True)
if bucket_file_path:
# The container has already been uploaded and just needs to be activated
for alter_type in [LanguageActivationLevel.Session, LanguageActivationLevel.System]:
deployer.activate_container(bucket_file_path, alter_type, allow_override=True)
else:
bucket_file_path = export_slc.name
deployer.run(container_file=export_slc, bucket_file_path=bucket_file_path,
alter_system=True, allow_override=True)
return func


@pytest.fixture(scope="session")
def deployed_slc(deploy_slc, language_alias) -> None:
"""
The fixture calls deploy_slc() once, with the language_alis defined in the fixture
with the corresponded name.
"""
deploy_slc(language_alias)
20 changes: 14 additions & 6 deletions pytest-slc/test/integration/pytest_slc_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@

pytest_plugins = ["pytester"]

LANGUAGE_ALIAS = 'PYTHON3_PYTEST_SLC'
MAIN_LANGUAGE_ALIAS = 'PYTHON3_PYTEST_SLC'
ALT_LANGUAGE_ALIAS = 'PYTHON3_PYTEST_SLC_ALT'

_test_code = dedent(fr"""
import pyexasol
Expand All @@ -20,11 +21,15 @@ def slc_builder() -> LanguageContainerBuilder:
container_builder.prepare_flavor(project_directory)
yield container_builder

def assert_udf_running(conn: pyexasol.ExaConnection):
@pytest.fixture(scope='session')
def language_alias() -> str:
return "{MAIN_LANGUAGE_ALIAS}"

def assert_udf_running(conn: pyexasol.ExaConnection, lang_alias: str) -> None:
with temp_schema(conn) as schema:
udf_name = 'TEST_UDF'
udf_create_sql = (
f'CREATE OR REPLACE {LANGUAGE_ALIAS} SCALAR SCRIPT "{{schema}}"."{{udf_name}}"() '
f'CREATE OR REPLACE {{lang_alias}} SCALAR SCRIPT "{{schema}}"."{{udf_name}}"() '
'RETURNS BOOLEAN AS '
'def run(ctx): '
'return True '
Expand All @@ -34,9 +39,12 @@ def assert_udf_running(conn: pyexasol.ExaConnection):
result = conn.execute(f'SELECT "{{schema}}"."{{udf_name}}"()').fetchall()
assert result[0][0] is True
ckunki marked this conversation as resolved.
Show resolved Hide resolved

def test_upload_slc(upload_slc, backend_aware_database_params):
upload_slc("{LANGUAGE_ALIAS}")
assert_udf_running(pyexasol.connect(**backend_aware_database_params))
def test_deploy_slc(deploy_slc, deployed_slc, backend_aware_database_params):
# We will activate the SLC also with an alternative alias and check that both
# the main and the alternative alias work.
deploy_slc("{ALT_LANGUAGE_ALIAS}")
for lang_alias in ["{MAIN_LANGUAGE_ALIAS}", "{ALT_LANGUAGE_ALIAS}"]:
assert_udf_running(pyexasol.connect(**backend_aware_database_params), lang_alias)
""")


Expand Down
Loading