Python MD-Models

This is the Python package for the MDModels Rust crate and hosts a set of tools to work with metadata models defined in markdown.

Whats in the bag?

• Core - The core functionality to work with MDModels
• PyDantic - Generate Pydantic models (on steroids) from MDModels
• LLM tools - Use LLMs to extract, transform and explore data
• SQL tools - Create SQL databases and query them with SQLAlchemy
• Graph tools - Create graph databases and query them with SPARQL

Note: This package is work in progress and the API will likely change in the future. Issues and contributions are very welcome!

Installation

To install the package, you can use the following command:

pip install mdmodels

# LLM tools
pip install mdmodels[chat]

# Graph tools
pip install mdmodels[graph]

# SQL tools
pip install mdmodels[sql]

# All tools
pip install mdmodels[all]

Examples

To get you started, have a look at the examples folder, featuring notebooks that showcase the usage of the package. This is what is available right now:

• Core
  • Basic - Basic usage of the core functionality
• LLM tools
  • Question Answering - Use LLMs to answer questions about the data
  • Metadata Extraction - Use LLMs to extract metadata from text and to databases
  • Metadata Mapping - Use LLMs to map metadata from one format to another
  • Similarity Search - Use LLMs to find similar items in a database
• SQL tools
  • Basic - Create a SQL database and interact with it
• Graph tools
  • Basic - Create a graph database and interact with it

Development

To run the tests for the package, use the following command:

# Execute all tests defined in the project
poetry run pytest

To run the tests and generate a detailed coverage report, which shows how much of your code is tested, use:

# Run tests with coverage analysis and generate an HTML report
poetry run pytest --cov=mdmodels --cov-report=html

If you want to run the tests within a Docker container, which can help ensure a consistent environment, use the following commands:

# Build the Docker image, specifying the Python version to use
docker build --build-arg PYTHON_VERSION=3.12 -t mdmodels .

# Run the Docker container, mounting the current directory to /app in the container
docker run -v $(pwd):/app mdmodels

To skip tests that are considered expensive and require additional services (like databases or external APIs), you can run:

# Execute tests while excluding those marked as expensive
poetry run pytest -m "not expensive"