OLAP database project for life admin.
As part of my life admin, I keep track of:
- Every transaction I make at an item level (since 2018-01-18)
- On the job, what I'm working on every 15 minutes (since 2019-04-23)
Note that the job tracker is my daily-tracker project.
Documentation is hosted at:
Since this is a personal project, I don't expect anyone else to do this, but I'm still documenting it for myself (I have a very, very bad memory).
This project requires the following three tools to be installed:
The required versions are specified in the badges at the top of this README, and also in:
After cloning the repo, install the dependencies and enable pre-commit:
poetry install --sync
pre-commit install --install-hooksPipeline changes need to be applied via a plan:
sqlmesh -p billiam_database plan
...and the pipeline can be run with the run command:
sqlmesh -p billiam_database run
The SQLMesh UI is great for doing these with a GUI:
sqlmesh -p billiam_database ui
Warning
The DuckDB driver for Metabase is a community driver. This means that it might not work in all circumstances.
Metabase is a tool for visualising data.
In this project, Metabase is run in a Docker container; run the following command:
# start
docker-compose --file docker-compose.yaml --project-name billiam-database up --detach
# stop
docker-compose --file docker-compose.yaml --project-name billiam-database down --remove-orphansThis will make Metabase available at:
Warning
There are two important notes about the current Metabase configuration:
- Since DuckDB only supports one connection at a time, you can't use Metabase and run the pipelines at the same time.
- The Metabase data is stored locally in the
dockerfiles/metaduck-datadirectory so that it persists between container restarts.