Skip to content

Commit

Permalink
Hotfix/database initialization improvements (#217)
Browse files Browse the repository at this point in the history
* Fix docker compose command

* Add sqlachemy-utils python package

* Update init database script

* Improve setup script so test database is also configured

* Rework python client info into separate readme

* Update curl commands

* Update init db and testing text

* Update text

* Remove unused docker compose

* Improve readme
  • Loading branch information
nathanfranklin authored Sep 16, 2024
1 parent b345f37 commit 577f028
Show file tree
Hide file tree
Showing 8 changed files with 154 additions and 192 deletions.
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ help: ## Display this help screen

.PHONY: start
start:
docker-compose -f devops/docker-compose.local.yml --env-file .env up
docker compose -f devops/docker-compose.local.yml --env-file .env up

.PHONY: stop
stop:
docker-compose -f devops/docker-compose.local.yml --env-file .env down
docker compose -f devops/docker-compose.local.yml --env-file .env down

.PHONY: restart-workers
restart-workers: ## Restart workers
docker-compose -f devops/docker-compose.local.yml --env-file .env restart workers
docker compose -f devops/docker-compose.local.yml --env-file .env restart workers

.PHONY: build
build:
Expand Down
27 changes: 27 additions & 0 deletions PYTHON_CLIENT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
## Python client

[![PyPI version](https://badge.fury.io/py/geoapi-client.svg)](https://badge.fury.io/py/geoapi-client)

The python package can be found at [PyPi](https://pypi.org/project/geoapi-client/).

**Note:** This Python client is no longer actively maintained or updated.

### Python client generation

Python client is generated from the swagger definition of GeoAPI. The following steps can be used to get swagger definition:
```
docker exec -it geoapi python output_swagger.py swagger.json
docker cp geoapi:/app/geoapi/swagger.json .
```

Using the swagger definition, the following steps create python client and upload the python client to PyPi
```
git clone --depth 1 https://github.com/swagger-api/swagger-codegen.git client/swagger-codegen
cp client/*.mustache client/swagger-codegen/modules/swagger-codegen/src/main/resources/python/.
# Convert
docker run --rm -v ${PWD}:/local -w=/local swaggerapi/swagger-codegen-cli generate -i swagger.json -l python -o client/geoapi_client -c client/config.json -t client/swagger-codegen/modules/swagger-codegen/src/main/resources/python/
cd client/geoapi_client
python3 setup.py sdist bdist_wheel
twine check dist/*
python3 -m twine upload dist/*
```
79 changes: 18 additions & 61 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
# GeoAPI

[![PyPI version](https://badge.fury.io/py/geoapi-client.svg)](https://badge.fury.io/py/geoapi-client)

## Overview and Architecture

GeoAPI is a restful API to create geospatial features in a PostGIS database. Users create a map "project" then
can add features to it. The development docker-compose file has 3 containers:
* a PostGIS database exposing 5432,
* the api which exposes port 8000 behind gunicorn
* an nginx server to serve static files and proxy to the api, running on port 8080.
* a nginx server to serve static files and proxy to the api, running on port 8080.

See https://github.com/TACC-Cloud/hazmapper which is an associated viewer application.

Expand All @@ -27,33 +25,30 @@ under gunicorn on port 8000
`make build`
`make start`

###### Initialize the database
###### Initialize the database (for local development and unit testing)

`docker exec -it geoapi python initdb.py`

###### Obtain a JWT

Refer to the confluence page or ask a colleague for assistance in obtaining a JWT.
### Example requests

###### Make some requests
You need a Tapis token for the appropriate tenant.

You need to add the following header for authentication:

`X-JWT-Assertion-designsafe` to equal the JWT obtained above
```bash
export JWT=your_access_token_string
```

###### Create a new map project
To create a new "map" project, send a POST request:

send a POST request to `localhost:8000/projects` with a body like this:
```
curl -X POST -H "Content-Type: application/json" -H "X-Tapis-Token: $JWT" http://localhost:8000/projects/ -d '{"name": "Test Project", "description": "This is a test project."}'
```

```json
{
"name": "Awesome Project",
"description": "Cool project"
}
To view all projects, including the newly created one, send a GET request:

```
curl -v -H "Content-Type: application/json" -H "X-Tapis-Token: $JWT" http://localhost:8000/projects/
```

send a GET request to `localhost:8000/projects` and you should get that back.

### Client viewer

Expand All @@ -69,6 +64,8 @@ First, apply migrations:
docker exec -it geoapi alembic upgrade head
```

**Note:** The above step is also automatically performed when running `initdb.py`

Then, create migrations:

```
Expand All @@ -81,28 +78,8 @@ alembic revision --autogenerate

## Testing

Run route/service tests on the `api` container
```
docker-compose -f devops/docker-compose.test.yml -p geoapi_test run api pytest
```

Run worker-related tasks on the `workers` container
```
docker-compose -f devops/docker-compose.test.yml -p geoapi_test run workers pytest -m "worker"
Run directly in your running containers:
```

Note that images need to be rebuilt before running tests if they have been updated (e.g. packages):
```
make build
```

or run directly in your running containers:
```
docker exec -it geoapi_postgres psql -d postgres -U dev
CREATE DATABASE TEST;
\connect test;
CREATE EXTENSION postgis;
# then run tests in api
docker exec -it geoapi bash
APP_ENV=testing pytest
Expand All @@ -120,24 +97,4 @@ on kube commands and Jenkins deployment workflows.

## Python client

The python package can be found at [PyPi](https://pypi.org/project/geoapi-client/)

### Python client generation

Python client is generated from the swagger definition of GeoAPI. The following steps can be used to get swagger definition:
```
docker exec -it geoapi python output_swagger.py swagger.json
docker cp geoapi:/app/geoapi/swagger.json .
```

Using the swagger definition, the following steps create python client and upload the python client to PyPi
```
git clone --depth 1 https://github.com/swagger-api/swagger-codegen.git client/swagger-codegen
cp client/*.mustache client/swagger-codegen/modules/swagger-codegen/src/main/resources/python/.
# Convert
docker run --rm -v ${PWD}:/local -w=/local swaggerapi/swagger-codegen-cli generate -i swagger.json -l python -o client/geoapi_client -c client/config.json -t client/swagger-codegen/modules/swagger-codegen/src/main/resources/python/
cd client/geoapi_client
python3 setup.py sdist bdist_wheel
twine check dist/*
python3 -m twine upload dist/*
```
The python package can be found at [PyPi](https://pypi.org/project/geoapi-client/). More details can be found in [Python Client](./PYTHON_CLIENT.md)
74 changes: 0 additions & 74 deletions devops/docker-compose.test.yml

This file was deleted.

65 changes: 49 additions & 16 deletions devops/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion devops/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ ExifRead = "^2.3.2"
flake8 = "^5.0.4"
urllib3 = "<2"
pytest-mock = "^3.12.0"
mapillary-tools = "0.9.5"
mapillary-tools = {git = "https://github.com/mapillary/mapillary_tools.git"}
sqlalchemy-utils = "^0.41.2"

[tool.poetry.dev-dependencies]

Expand Down
10 changes: 3 additions & 7 deletions geoapi/db.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,12 @@
from geoapi.log import logger


CONNECTION_STRING = 'postgresql://{}:{}@{}/{}'.format(
settings.DB_USERNAME,
settings.DB_PASSWD,
settings.DB_HOST,
settings.DB_NAME
)
def get_db_connection_string(conf):
return f'postgresql://{conf.DB_USERNAME}:{conf.DB_PASSWD}@{conf.DB_HOST}/{conf.DB_NAME}'


def create_engine_for_context(context=None):
engine = create_engine(CONNECTION_STRING,
engine = create_engine(get_db_connection_string(settings),
echo=False, # default value
pool_pre_ping=True,
pool_reset_on_return=True)
Expand Down
Loading

0 comments on commit 577f028

Please sign in to comment.