Architecture refactoring (#61)

* chore(ci): remove the limitation on branches for automatic testing * chore(domain): replace optional model to new domain module * feat(tests): add tests for optional model * feat(domain): add file entity * feat(tests): add tests for file entity * chore(domain): replace and restructure task entities to new domain module * chore: change imports to absolute ones * chore(domain): add empty user submodule to domain module * fix(domain): add handling of incorrect case in optional model * chore(domain): add code documentation for tasks and file * chore(domain): add README files for domain modules * chore(use case): init use case module, exception and dto submodules * chore(use case): init file, task and user use case modules * chore: add pytest mock, pytest asyncio and cfgv to dependencies * fix: replace dto module to internal * feat(dto): add base schema for dto * feat(dto): add dto schemas for file * feat(dto): add dto schemas for file metadata * feat(dto): add dto schemas for dataset * feat(dto): add dto schemas for task * chore: init repository exceptions file * feat(dto): add dto schemas for task worker * chore: add readme file for dto module * feat: add file and dataset exceptions * fix: delete old use case exception module * feat(use case): add use case for file's content type checking * feat(tests): add tests for file's content type checking use case * feat: add Unit Of Work pattern implementation * feat(tests): add tests for unit of work * feat(use case): add use case for saving datasets * feat(tests): add tests for save dataset use case * feat(use case): add use case for file saving * feat(tests): add tests for save file use case * feat: add task use cases' exceptions * feat(use case): add use case for retrieving task * feat(tests): add tests for retrieve task use case * chore: add missing imports * feat(use case): add use case for setting task * feat(tests): add tests for set task use case * chore: add file for user's use case exceptions * chore: add readme file for usecase module * chore: init infrastructure, repository and worker modules * chore(infra): init data storage module * feat(infra): add settings for data storages * feat(infra): add context implementation for postgres * feat(infra): add base orm model for relational data bases * feat(infra): add base orm model for relational data bases * feat(infra): add file metadata model for relational data bases * feat(infra): add dataset model for relational data bases * feat(infra): add task model for relational data bases * chore(infra): add empty module for user relational models * chore: set new directories for test command in the makefile * chore: set new alembic.ini file's path for postgres migrations command in the makefile * feat(infra): add first empty postgres migration * feat(infra): add postgres db migration that creates file and dataset table * feat(infra): add postgres db migration that creates task table * feat(repo): add file repository implementation * chore(infra): rename and replace relational storage context * chore(repo): format code * fix(infra): remove typo in dataset relational model's fields * fix(uow): change the way to return a context from uow * feat(use case): add file metadata not found exception * feat(use case): add use case for retrieving datasets * feat(tests): add tests for retrieve dataset use case * fix(use case): remove unnecessery fields in profiling task worker interface * chore(use case): make usecase arguments callable only by key-value * feat(use case): add use case for updating task info * chore(dto): update task update schema * feat(tests): add tests for updating task info use case * feat(dto): add dataset and file metadata repositories' exceptions * feat(dto): add find and response schemas for csv files * feat(domain): add a function to get a primitive by its name * feat(use case): add use case for task profiling * feat(tests): add tests for profilie task use case * chore(infra): change name of relational context type * feat(infra): add postgres context and context maker * feat(repo): add find(read) csv file method to the file repository * feat(repo): add class with crud operations for relational repositories * feat(repo): add file metadata repository implementation * fix(tests): update conftest file, delete alembic tests * feat(infra): add necessary imports for migrations * feat(repo): add dataset repository implementation * feat(tests): update conftest file with database preparing fixture * feat(repo): add find with join for dataset repository * feat(repo): add task repository implementation * feat(infra): init background task and celery modules * feat(infra): add settings for background tasks * feat(infra): add celery config * feat(infra): add resource intensive celery task class * feat(infra): add dependencies for injection in celery tasks * feat(infra): add data profiling celery task * feat(infra): add the main celery worker * feat(worker): add data profiling worker implementation (for celery) * feat(rest): init rest module and its submodules * feat(test): add ping endpoint, common router * feat(rest): add dependencies for injection in endpoints * feat(rest): add dependencies for injection in file/dataset endpoints * feat(rest): add endpoint for upload csv datasets * feat(rest): add endpoint for retrieve datasets * feat(rest): add dependencies for injection in task endpoints * feat(rest): add endpoint for task setting * feat(rest): add endpoint for task retrieving * feat(rest): add di module for user * feat(rest): add main api router * feat: add main application (fastapi) object * feat: change path to app object and celery worker in makefile * fix(repo): typo in imports * fix(tests): remove typo in fixture name * chore: change name for celery worker in makefile * fix(infra): make the correct traceback processing * feat(rest): add use case's exceptions handling * chore(domain): add code documentation for math function * chore: add failed file read exception and its handler * chore: delete old app module * chore: format code * chore: change old module path in all makefile commands * fix(tests): remove typo in exception name and expande uow mocks * feat(infra): add flat file storage context and update use cases * feat(rest): update di after add new flat context * fix: typo in file metadata repository name * feat(tests): add tests for file repository * feat(tests): add tests for file metadata repository * feat(tests): add tests for dataset repository * feat(tests): add tests for task repository * fix(tests): add converter from UUID to string for file name * chore(domain): move task status and task failure reason classes in separate module * fix(dto): remove unnecessery generic in update schemas * chore: add and fix documentation for repo and use case exceptions * chore(infra): add readme file for data storage module * chore(infra): correct typing * chore: update new directories path in Dockerfile * chore(infra): add missed imports to postgres env.py file * chore: fix format errors * chore: delete sqlalchemy mixins from project dependencies * fix: type errors * chore: add command for type checking in makefile and rename old target names * chore: add type checking to ci * chore(infra): add relational/postgres context maker types * chore(infra): add helper functions for getting postgres context * feat(infra): implement an analogue of a transaction for local storage * feat(tests): add tests for new flat context * feat(infra): add relational add/delete models * chore(tests): divide the test directory into subdirectories for unit and integration testing * feat(infra): implement sync flush method for flat context * feat(infra): add universal context for storages * feat(tests): add tests for universal context * chore(usecase): add missed task usecases imports * feat(usecase): make a single unit of work in each use case and its usage in app * feat(repo): add processing for case when value is not found when updating in the database * feat(tests): add tests that check dependencies between modules * feat(tests): set up the correct system of fixtures with access to storage systems * feat(tests): add integration tests for all api usage scenarios * chore: format code, and some documentation and readme files
Desbordante · Oct 7, 2024 · 84a044b · 84a044b
1 parent 84d2e30
commit 84a044b
Show file tree

Hide file tree

Showing 192 changed files with 8,645 additions and 3,849 deletions.
diff --git a/.github/workflows/run-linter-and-tests.yaml b/.github/workflows/run-linter-and-tests.yaml
@@ -1,11 +1,7 @@
 name: Lint
 on:
   pull_request:
-    branches:
-      - main
   push:
-    branches:
-      - main
 jobs:
   run-linters-and-tests:
     runs-on: ubuntu-latest
@@ -24,6 +20,9 @@ jobs:
       - name: Run all linters and formatters
         run: make lint
 
+      - name: Run type check
+        run: make check-types
+
       - name: Up all containers
         run: docker compose -f dev-docker-compose.yaml up --build --force-recreate --remove-orphans -d
 

diff --git a/Dockerfile b/Dockerfile
@@ -78,7 +78,7 @@ EXPOSE ${APPLICATION_SERVER_PORT}
 USER 1001
 
 # Run the uvicorn application server.
-CMD exec uvicorn --workers 1 --host 0.0.0.0 --port $APPLICATION_SERVER_PORT app.main:app
+CMD exec uvicorn --workers 1 --host 0.0.0.0 --port $APPLICATION_SERVER_PORT internal:app
 
 FROM server-setup-build-stage as install-dependencies-build-stage
 # install [tool.poetry.dependencies]
@@ -93,4 +93,4 @@ FROM server-setup-build-stage as production-image
 COPY --chown=python_application:python_application --from=install-dependencies-build-stage /application_root/.venv /application_root/.venv
 
 # Copy application files
-COPY --chown=python_application:python_application /app /application_root/app/
+COPY --chown=python_application:python_application /internal /application_root/internal/
diff --git a/Makefile b/Makefile
@@ -1,4 +1,4 @@
-.PHONY: env install-deps up open-db revision migrate downgrade worker app init lint test
+.PHONY: env install-deps up open-db pg-revision pg-migrate pg-downgrade celery-worker app init lint test check-types
 
 ifeq ($(shell test -e '.env' && echo -n yes), yes)
 	include .env
@@ -27,43 +27,46 @@ open-db:
 	docker exec -it desbordante-postgres psql -d $(POSTGRES_DB) -U $(POSTGRES_USER)
 
 ## Create new revision file automatically
-revision:
-	poetry run alembic -c app/settings/alembic.ini revision --autogenerate $(args)
+pg-revision:
+	poetry run alembic -c internal/infrastructure/data_storage/relational/postgres/migrations/alembic.ini revision --autogenerate $(args)
 
 ## Make migrations in database
-migrate:
-	poetry run alembic -c app/settings/alembic.ini upgrade $(args)
+pg-migrate:
+	poetry run alembic -c internal/infrastructure/data_storage/relational/postgres/migrations/alembic.ini upgrade $(args)
 
 ## Downgrade database
-downgrade:
-	poetry run alembic -c app/settings/alembic.ini downgrade $(args)
+pg-downgrade:
+	poetry run alembic -c internal/infrastructure/data_storage/relational/postgres/migrations/alembic.ini downgrade $(args)
 
 ## Run celery worker in watch mode
-worker:
-	watchmedo auto-restart --directory=./ --pattern='*.py' --recursive -- celery -A app.worker worker --loglevel=info --concurrency=1
+celery-worker:
+	watchmedo auto-restart --directory=./ --pattern='*.py' --recursive -- celery -A internal.infrastructure.background_task.celery worker --loglevel=info --concurrency=1
 
 ## Run application server in watch mode
 app:
-	poetry run uvicorn --port 8000 app.main:app --reload
+	poetry run uvicorn --port 8000 internal:app --reload
 
 ## Initiate repository
 init:
 	make env install-deps
 
 ## Run all formatters and linters in project
 lint:
-	poetry run ruff check tests app \
-	& poetry run ruff format --check tests app \
-	& poetry run black --check tests app
+	poetry run ruff check tests internal \
+	& poetry run ruff format --check tests internal \
+	& poetry run black --check tests internal
 
 ## Reformat code
 format:
-	poetry run ruff format tests app & poetry run ruff check --fix & poetry run black tests app
-
+	poetry run ruff format tests internal & poetry run ruff check --fix & poetry run black tests internal
 
 ## Run all tests in project
 test:
-	poetry run pytest -o log_cli=true --verbosity=2 --showlocals --log-cli-level=INFO --test-alembic --cov=app --cov-report term
+	poetry run pytest -o log_cli=true --verbosity=2 --showlocals --log-cli-level=INFO --cov=internal --cov-report term
+
+## Check all types
+check-types:
+	poetry run pyright .
 
 .DEFAULT_GOAL := help
 # See <https://gist.github.com/klmr/575726c7e05d8780505a> for explanation.

diff --git a/README.md b/README.md
@@ -13,7 +13,7 @@ Execute `make` to see all available rules with documentation
 1. Activate virtual environment: `source .venv/bin/activate`
 2. Don't forget to change values in .env
 3. Run **development-only** containers: `make up`
-4. Run celery worker in watch mode: `make worker`
+4. Run celery worker in watch mode: `make celery-worker`
 5. Run application in watch mode: `make app`
 
 ## Docs

diff --git a/app/api/__init__.py b/app/api/__init__.py
diff --git a/app/api/dataset.py b/app/api/dataset.py
diff --git a/app/api/task.py b/app/api/task.py
diff --git a/app/db/__init__.py b/app/db/__init__.py
diff --git a/app/db/session.py b/app/db/session.py
diff --git a/app/domain/common/README.md b/app/domain/common/README.md
diff --git a/app/domain/common/optional_model.py b/app/domain/common/optional_model.py
diff --git a/app/domain/file/README.md b/app/domain/file/README.md
diff --git a/app/domain/file/dataset.py b/app/domain/file/dataset.py
diff --git a/app/domain/task/README.md b/app/domain/task/README.md
diff --git a/app/domain/task/__init__.py b/app/domain/task/__init__.py