From e1d5dfebc411b1698ebdcacfb69465347505e663 Mon Sep 17 00:00:00 2001 From: sklavit Date: Sat, 27 Jul 2024 23:10:19 +0100 Subject: [PATCH] Some docs cleanup --- .idea/vcs.xml | 3 +- apps/streamlit_ds_chat/CHANGELOG.md | 7 +++ .../{readme.md => INSTALLATION.md} | 47 ++++++++++++------- apps/streamlit_ds_chat/README.md | 23 +++++++++ .../README_05_putting_all_togather.md | 3 +- 5 files changed, 64 insertions(+), 19 deletions(-) create mode 100644 apps/streamlit_ds_chat/CHANGELOG.md rename apps/streamlit_ds_chat/{readme.md => INSTALLATION.md} (67%) create mode 100644 apps/streamlit_ds_chat/README.md diff --git a/.idea/vcs.xml b/.idea/vcs.xml index 856008c..8306744 100644 --- a/.idea/vcs.xml +++ b/.idea/vcs.xml @@ -2,7 +2,6 @@ - - + \ No newline at end of file diff --git a/apps/streamlit_ds_chat/CHANGELOG.md b/apps/streamlit_ds_chat/CHANGELOG.md new file mode 100644 index 0000000..a4931d9 --- /dev/null +++ b/apps/streamlit_ds_chat/CHANGELOG.md @@ -0,0 +1,7 @@ + + + +- xxx Improving documentation +- 2024-07-27 Merged branch with POC streamlit data scientist assistant + +... diff --git a/apps/streamlit_ds_chat/readme.md b/apps/streamlit_ds_chat/INSTALLATION.md similarity index 67% rename from apps/streamlit_ds_chat/readme.md rename to apps/streamlit_ds_chat/INSTALLATION.md index e9c7ae4..2f4286a 100644 --- a/apps/streamlit_ds_chat/readme.md +++ b/apps/streamlit_ds_chat/INSTALLATION.md @@ -1,13 +1,3 @@ -Title -feat: add Streamlit app for DS chat with Google Vertex AI and Coiled integration - -Overview -Introduced a new Streamlit application for data science chat functionality using Google Vertex AI. -Added multiple supporting scripts and modules for handling various functionalities such as code generation, cloud execution, and data visualization. -Integrated Coiled for distributed computing and remote code execution. -Provided detailed documentation and examples for setting up the environment, using the application, and testing different components. - - # Setting up a local development environment Before we can actually start building Streamlit apps, we will first have to set up a development environment. @@ -15,8 +5,10 @@ Before we can actually start building Streamlit apps, we will first have to set Let's start by installing and setting up a conda environment. ## **Install conda** -- Install `conda` by going to https://docs.conda.io/en/latest/miniconda.html and choose your operating system (Windows, Mac or Linux). -- Download and run the installer to install `conda`. +- Install `conda`: + - visit https://docs.conda.io/en/latest/miniconda.html + - choose your operating system (Windows, Mac or Linux) + - download and run the installer to install `conda`. ## **Create a new conda environment** Now that you have conda installed, let's create a conda environment for managing all the Python library dependencies. @@ -46,6 +38,7 @@ conda activate stenv ## **Install the Streamlit library** It's now time to install the `streamlit` library: + ```bash conda install streamlit // or @@ -56,20 +49,25 @@ pip install streamlit ## Other dependencies ### For HuggingFace transformers + +```bash pip install transformers conda install pytorch +``` ### For Google vertex AI + +```bash pip install google-cloud-aiplatform // --upgrade ? sudo snap install google-cloud-cli --classic -pip3 install black -conda install numpy"<=2.0" - +//pip3 install black +//conda install numpy"<2.0" +``` ### Installing Ruff Ruff is available as ruff on PyPI: -``` +```bash pip install ruff ``` @@ -82,4 +80,21 @@ ruff format # Format all files in the current directory. ### Bokeh visualization +```bash pip install bokeh -- already installed? +``` + +## Dask and Coiled installation + +See https://github.com/Sklavit/pet_project/blob/c6278b67484558985faa9bd3a9a58e34e765cf6d/apps/streamlit_ds_chat/cloud_coiled_io/README.md + +```bash +pip3 install coiled "dask[complete]" +coiled login +``` + +Expected result: + +> Authentication successful 🎉 +> Credentials have been saved at /home/< USERNAME >/.config/dask/coiled.yaml + diff --git a/apps/streamlit_ds_chat/README.md b/apps/streamlit_ds_chat/README.md new file mode 100644 index 0000000..e0ebcf7 --- /dev/null +++ b/apps/streamlit_ds_chat/README.md @@ -0,0 +1,23 @@ +# POC of data scientist assistant + +Features and peculiarities: + +- This is Streamlit application for UI +- +- Use `pandas` for parsing CSV file +- Use Google Vertex AI (gemini-1.5-flash-001") to generate python code + from user requests +- Use `Dask` for secure and remote code execution +- use `Coiled` for management of the cloud of Dask workers +- ... + + +## [Installation](https://github.com/sklavit/pet_project/tree/main/apps/streamlit_ds_chat/INSTALLATION.md) + +See [Installation](https://github.com/sklavit/pet_project/tree/main/apps/streamlit_ds_chat/INSTALLATION.md) + +## [Changelog](https://github.com/sklavit/pet_project/tree/main/apps/streamlit_ds_chat/CHANGELOG.md) + +See [Changelog](https://github.com/sklavit/pet_project/tree/main/apps/streamlit_ds_chat/CHANGELOG.md) + + diff --git a/apps/streamlit_ds_chat/README_05_putting_all_togather.md b/apps/streamlit_ds_chat/README_05_putting_all_togather.md index 0f038ae..3d56e3a 100644 --- a/apps/streamlit_ds_chat/README_05_putting_all_togather.md +++ b/apps/streamlit_ds_chat/README_05_putting_all_togather.md @@ -15,8 +15,9 @@ It is time now to put all components togather. Example requests: - Draw number of likes for people and fruits +``` conda create -p ./env12 -c conda-forge coiled python=3.12 "numpy<2.0.0" streamlit google-cloud-aiplatform dask - +``` Known issues: - Bokeh is not needed for workers but it is still provided.