✨ Add orchestration module

.github/workflows/docker-publish.yml

@@ -35,12 +35,34 @@ jobs:
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
-      - name: Build
+      - name: Build and publish viadot-lite image
         uses: docker/build-push-action@v3
         with:
           context: .
           file: docker/Dockerfile
           platforms: linux/amd64
           push: true
-          tags: ghcr.io/${{ github.repository }}/viadot:${{ github.event.inputs.tag }}
+          target: viadot-lite
+          tags: ghcr.io/${{ github.repository }}/viadot-lite:${{ github.event.inputs.tag }}
+
+      - name: Build and publish viadot-aws image
+        uses: docker/build-push-action@v3
+        with:
+          context: .
+          file: docker/Dockerfile
+          platforms: linux/amd64
+          push: true
+          target: viadot-aws
+          tags: ghcr.io/${{ github.repository }}/viadot-aws:${{ github.event.inputs.tag }}
+
+      - name: Build and publish viadot-azure image
+        uses: docker/build-push-action@v3
+        with:
+          context: .
+          file: docker/Dockerfile
+          platforms: linux/amd64
+          push: true
+          target: viadot-azure
+          tags: ghcr.io/${{ github.repository }}/viadot-azure:${{ github.event.inputs.tag }}
           build-args: INSTALL_DATABRICKS=${{ github.event.inputs.if_databricks }}
+

.gitignore

@@ -82,6 +82,9 @@ target/
 profile_default/
 ipython_config.py
 
+# pyenv
+.python-version
+
 # pipenv
 #   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
 #   However, in case of collaboration, if having platform-specific dependencies or dependencies
@@ -126,13 +129,14 @@ dmypy.json
 # Pyre type checker
 .pyre/
 
-# Linux
+# OS files
+.DS_Store
 .bash_history
 .bashrc
 .viminfo
 .netrwhist
 .ssh
-
+.python_history
 
 # Azure
 .azure
@@ -162,6 +166,17 @@ profiles.yaml
 # DataHub
 .datahub
 
+# local/env
+*.prefect/
+*.config/
+*.local/
+
+# VS Code
+.vscode-server/
+
+# Jupyter notebook
+*.ipynb
+
 # Git
 .gitconfig
 
@@ -171,4 +186,4 @@ profiles.yaml
 *.secret
 
 # AWS
-.aws
+.aws

CHANGELOG.md

@@ -7,8 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+
+- Added new version of `Genesys` connector and test files.
+- Added new version of `Outlook` connector and test files.
+- Added new version of `Hubspot` connector and test files.
+- Added `Mindful` connector and test file.
+
+
 ### Added
 
+- Added `sap_to_parquet` Prefect flow.
+- Added `duckdb_to_sql_server`, `duckdb_to_parquet`, `duckdb_transform` Prefect flows.
+- Added `bcp` and `duckdb_query` Prefect tasks.
+- Added `DuckDB` source class.
+- Added `sql_server_to_minio` flow for prefect.
+- Added `df_to_minio` task for prefect
+- Added handling for `DatabaseCredentials` and `Secret` blocks in `prefect/utlis.py:get_credentials`
+- Added `SQLServer` source and tasks `create_sql_server_table`, `sql_server_to_df`,`sql_server_query`
+- Added `basename_template` to `MinIO` source
 - Added `_empty_column_to_string` and `_convert_all_to_string_type` to convert data types to string.
 - Added `na_values` parameter to `Sharepoint` class to parse `N/A` values coming from the excel file columns.
 - Added `get_last_segment_from_url` function to sharepoint file.
@@ -35,6 +51,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- Changed location of `task_utils.py` and removed unused/prefect1-related tasks.
 - Changed the way of handling `NA` string values and mapped column types to `str` for `Sharepoint` source.
 - Added `SQLServerToDF` task
 - Added `SQLServerToDuckDB` flow which downloads data from SQLServer table, loads it to parquet file and then uploads it do DuckDB
@@ -53,6 +70,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Removed Prefect dependency from the library (Python library, Docker base image)
 - Removed `catch_extra_separators()` from `SAPRFCV2` class
 
+### Fixed
+- Fixed the typo in credentials in `SQLServer` source
+
 ## [0.4.3] - 2022-04-28
 
 ### Added

CONTRIBUTING.md

@@ -22,7 +22,43 @@ We provide the extensions, settings, and tasks for VSCode in the `.vscode` folde
 
 ### Development Docker container
 
-If you wish to develop in a Docker container, viadot comes with a VSCode task to make that simple. You can easily spin up a terminal in the container with the `Ctrl+Shift+B` shortcut. The container will have all of the contents of the root `viadot` directory mapped to `/home/viadot`.
+#### Bulding of containers
+
+To build all available containers, run the following command:
+
+**NOTE**: All the following commands must be execured from within the `viadot/docker/` directory. 
+
+```bash
+docker compose up -d 
+```
+If you want to build a specific one, add its name at the end of the command:
+
+```bash
+docker compose up -d viadot-azure
+```
+
+#### Building docker images
+
+All necessary Docker images are released in `ghcr.io` and are included in the `docker-compose.yml` file, but if you want to create your own custom Docker image, follow the following instructions.
+
+In the repository, we have three possible images to build:
+
+- `viadot-lite`
+- `viadot-azure`
+- `viadot-aws`
+
+To build an image, you have to be in the root directory of the repository and run the following command with selected target:
+
+```bash
+docker build --target viadot-azure -t <name of your image>:<version of your image> -f docker/Dockerfile .
+```
+
+
+#### Start of work inside the container 
+
+```bash
+docker exec -it viadot-azure bash
+```
 
 ### Environment variables
 

docker/Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.10-slim-bullseye
+FROM python:3.10-slim-bullseye AS base
 
 # Add user
 RUN useradd --non-unique --uid 1000 --create-home viadot && \
@@ -15,15 +15,28 @@ SHELL ["/bin/sh", "-c"]
 RUN groupadd docker && \
     usermod -aG docker viadot
 
-# Release File Error
-# https://stackoverflow.com/questions/63526272/release-file-is-not-valid-yet-docker
-RUN echo "Acquire::Check-Valid-Until \"false\";\nAcquire::Check-Date \"false\";" | cat > /etc/apt/apt.conf.d/10no--check-valid-until
-
 # System packages
-RUN apt update -q && yes | apt install -q gnupg vim unixodbc-dev build-essential \
-    curl python3-dev libboost-all-dev libpq-dev python3-gi sudo git software-properties-common
+RUN apt update -q && yes | apt install -q  gnupg vim curl git unixodbc
 ENV PIP_NO_CACHE_DIR=1
-RUN pip install --upgrade cffi
+
+# This one's needed for the SAP RFC connector.
+# It must be installed here as the SAP package does not define its dependencies,
+# so `pip install pyrfc` breaks if all deps are not already present.
+RUN pip install cython==0.29.24
+
+# Python env
+RUN pip install --upgrade pip
+
+
+ENV USER viadot
+ENV HOME="/home/$USER"
+ENV PATH="$HOME/.local/bin:$PATH"
+ENV RYE_HOME="$HOME/rye"
+ENV PATH="$RYE_HOME/shims:$PATH"
+
+# Install Rye and uv.
+RUN curl -sSf https://rye.astral.sh/get | RYE_TOOLCHAIN_VERSION="3.10" RYE_INSTALL_OPTION="--yes" bash && \
+    rye config --set-bool behavior.use-uv=true
 
 # Fix for old SQL Servers still using TLS < 1.2
 RUN chmod +rwx /usr/lib/ssl/openssl.cnf && \
@@ -40,6 +53,13 @@ RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - && \
 
 COPY docker/odbcinst.ini /etc
 
+
+####################
+### viadot-azure ###
+####################
+
+FROM base as viadot-azure
+
 ARG INSTALL_DATABRICKS=false
 
 # Databricks source setup
@@ -56,31 +76,72 @@ RUN if [ "$INSTALL_DATABRICKS" = "true" ]; then \
 
 ENV SPARK_HOME /usr/local/lib/python3.10/site-packages/pyspark
 
-# This one's needed for the SAP RFC connector.
-# It must be installed here as the SAP package does not define its dependencies,
-# so `pip install pyrfc` breaks if all deps are not already present.
-RUN pip install cython==0.29.24
-
-# Python env
-RUN pip install --upgrade pip
-
-ENV USER viadot
-ENV HOME="/home/$USER"
-ENV PATH="$HOME/.local/bin:$PATH"
+ARG INSTALL_DATABRICKS=false
 
 WORKDIR ${HOME}
 
 COPY --chown=${USER}:${USER} . ./viadot
 
+RUN rye lock --reset --features viadot-azure --pyproject viadot/pyproject.toml
+RUN sed '/-e/d' ./viadot/requirements.lock > ./viadot/requirements.txt
+RUN pip install --no-cache-dir -r ./viadot/requirements.txt
 
-# Dependecy install
 RUN if [ "$INSTALL_DATABRICKS" = "true" ]; then \
     pip install ./viadot/.[databricks]; \
-    else \
-    pip install ./viadot; \
     fi
 
+# Dependecy install
+RUN pip install ./viadot/.[viadot-azure]
+
 # Cleanup.
 RUN rm -rf ./viadot
 
 USER ${USER}
+
+
+###################
+### viadot-lite ###
+###################
+
+FROM base as viadot-lite
+
+WORKDIR ${HOME}
+
+COPY --chown=${USER}:${USER} . ./viadot
+
+RUN rye lock --reset --features viadot-lite --pyproject viadot/pyproject.toml
+RUN sed '/-e/d' ./viadot/requirements.lock > ./viadot/requirements.txt
+RUN pip install --no-cache-dir -r ./viadot/requirements.txt
+
+# Dependecy install
+RUN pip install ./viadot/
+
+# Cleanup.
+RUN rm -rf ./viadot
+
+USER ${USER}
+
+##################
+### viadot-aws ###
+##################
+
+FROM base as viadot-aws
+
+
+WORKDIR ${HOME}
+
+COPY --chown=${USER}:${USER} . ./viadot
+
+RUN rye lock --reset --features viadot-aws --pyproject viadot/pyproject.toml
+RUN sed '/-e/d' ./viadot/requirements.lock > ./viadot/requirements.txt
+RUN pip install --no-cache-dir -r ./viadot/requirements.txt
+
+# Dependecy install
+RUN pip install ./viadot/.[viadot-aws]
+
+# Cleanup.
+RUN rm -rf ./viadot
+
+USER ${USER}
+
+

docker/docker-compose.yml

@@ -1,13 +1,34 @@
 version: "3"
 
 services:
-  viadot_2:
-    image: ghcr.io/dyvenia/viadot/viadot:2.0-latest
-    container_name: viadot_2
+  viadot_2_lite:
+    image: ghcr.io/dyvenia/viadot/viadot-lite:2.0.0-beta.1
+    container_name: viadot_2_lite
     volumes:
       # - ${HOME}/.databricks-connect:/home/viadot/.databricks-connect
       # - ${HOME}/.config/viadot/config.yaml:/home/viadot/.config/viadot/config.yaml
       - ../:/home/viadot
     shm_size: "4gb"
     command: sleep infinity
     restart: "unless-stopped"
+  viadot_2_azure:
+    image: ghcr.io/dyvenia/viadot/viadot-azure:2.0.0-beta.1
+    container_name: viadot_2_azure
+    volumes:
+      # - ${HOME}/.databricks-connect:/home/viadot/.databricks-connect
+      # - ${HOME}/.config/viadot/config.yaml:/home/viadot/.config/viadot/config.yaml
+      - ../:/home/viadot
+    shm_size: "4gb"
+    command: sleep infinity
+    restart: "unless-stopped"
+  viadot_2_aws:
+    image: ghcr.io/dyvenia/viadot/viadot-aws:2.0.0-beta.1
+    container_name: viadot_2_aws
+    volumes:
+      # - ${HOME}/.databricks-connect:/home/viadot/.databricks-connect
+      # - ${HOME}/.config/viadot/config.yaml:/home/viadot/.config/viadot/config.yaml
+      - ../:/home/viadot
+    shm_size: "4gb"
+    command: sleep infinity
+    restart: "unless-stopped"
+

docs/advanced_usage/docker_containers.md

@@ -0,0 +1,5 @@
+Currently there are tree available containers to build:
+
+- `viadot-lite` - It has installed default dependencies and supports only non-cloud-specific sources.
+- `viadot-azure` - It has installed default and viadot-azure dependencies. Supports Azure-based sources and non-cloud-specific ones.
+- `viadot-aws` - It has installed default and aws-azure dependencies. Supports AWS-based sources and non-cloud-specific ones.

docs/getting_started/getting_started.md

@@ -0,0 +1,24 @@
+
+### Prerequisites
+
+We assume that you have [Rye](https://rye-up.com/) installed:
+
+```console
+curl -sSf https://rye-up.com/get | bash
+```
+
+### Installation
+
+Clone the `2.0` branch, and set up and run the environment:
+
+```console
+git clone https://github.com/dyvenia/viadot.git -b 2.0 && \
+  cd viadot && \
+  rye sync
+```
+
+### Configuration
+
+In order to start using sources, you must configure them with required credentials. Credentials can be specified either in the viadot config file (by default, `~/.config/viadot/config.yaml`), or passed directly to each source's `credentials` parameter.
+
+You can find specific information about each source's credentials in [the documentation](../references/sources/sql_sources.md).