Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server : replace behave with pytest #10416

Merged
merged 17 commits into from
Nov 26, 2024
Merged
2 changes: 1 addition & 1 deletion .devops/nix/python-scripts.nix
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ let

# server tests
openai
behave
pytest
prometheus-client
];
in
Expand Down
9 changes: 5 additions & 4 deletions .github/workflows/server.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,14 +122,14 @@ jobs:
id: server_integration_tests
run: |
cd examples/server/tests
PORT=8888 ./tests.sh
./tests.sh
- name: Slow tests
id: server_integration_tests_slow
if: ${{ (github.event.schedule || github.event.inputs.slow_tests == 'true') && matrix.build_type == 'Release' }}
run: |
cd examples/server/tests
PORT=8888 ./tests.sh --stop --no-skipped --no-capture --tags slow
SLOW_TESTS=1 ./tests.sh
server-windows:
Expand Down Expand Up @@ -180,11 +180,12 @@ jobs:
run: |
cd examples/server/tests
$env:PYTHONIOENCODING = ":replace"
behave.exe --summary --stop --no-capture --exclude 'issues|wrong_usages|passkey' --tags llama.cpp
pytest -v -x
- name: Slow tests
id: server_integration_tests_slow
if: ${{ (github.event.schedule || github.event.inputs.slow_tests == 'true') && matrix.build_type == 'Release' }}
run: |
cd examples/server/tests
behave.exe --stop --no-skipped --no-capture --tags slow
$env:SLOW_TESTS = "1"
pytest -v -x
1 change: 1 addition & 0 deletions examples/server/tests/.gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
.venv
tmp
12 changes: 1 addition & 11 deletions examples/server/tests/README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,9 @@
# Server tests

Python based server tests scenario using [BDD](https://en.wikipedia.org/wiki/Behavior-driven_development)
and [behave](https://behave.readthedocs.io/en/latest/):

* [issues.feature](./features/issues.feature) Pending issues scenario
* [parallel.feature](./features/parallel.feature) Scenario involving multi slots and concurrent requests
* [security.feature](./features/security.feature) Security, CORS and API Key
* [server.feature](./features/server.feature) Server base scenario: completion, embedding, tokenization, etc...
Python based server tests scenario using [pytest](https://docs.pytest.org/en/stable/).

Tests target GitHub workflows job runners with 4 vCPU.

Requests are
using [aiohttp](https://docs.aiohttp.org/en/stable/client_reference.html), [asyncio](https://docs.python.org/fr/3/library/asyncio.html)
based http client.

Note: If the host architecture inference speed is faster than GitHub runners one, parallel scenario may randomly fail.
To mitigate it, you can increase values in `n_predict`, `kv_size`.

Expand Down
15 changes: 15 additions & 0 deletions examples/server/tests/conftest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import pytest
from utils import *


# ref: https://stackoverflow.com/questions/22627659/run-code-before-and-after-each-test-in-py-test
@pytest.fixture(autouse=True)
ngxson marked this conversation as resolved.
Show resolved Hide resolved
def stop_server_after_each_test():
# do nothing before each test
yield
# stop all servers after each test
instances = set(
server_instances
) # copy the set to prevent 'Set changed size during iteration'
for server in instances:
server.stop()
66 changes: 0 additions & 66 deletions examples/server/tests/features/ctx_shift.feature

This file was deleted.

113 changes: 0 additions & 113 deletions examples/server/tests/features/embeddings.feature

This file was deleted.

71 changes: 0 additions & 71 deletions examples/server/tests/features/environment.py

This file was deleted.

36 changes: 0 additions & 36 deletions examples/server/tests/features/infill.feature

This file was deleted.

5 changes: 0 additions & 5 deletions examples/server/tests/features/issues.feature

This file was deleted.

Loading
Loading