-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
27 changed files
with
316 additions
and
278 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,7 @@ | ||
| --- | ||
| name: Publish Develop Docs | ||
| name: Docs | ||
|
|
||
| on: | ||
| pull_request: | ||
| push: | ||
| branches: | ||
| - main | ||
|
|
@@ -30,6 +29,8 @@ jobs: | |
| run: | | ||
| git config user.email "[email protected]" | ||
| git config user.name "GitHub Action" | ||
| - name: Build Documentation | ||
| - name: Deploy Documentation | ||
| run: | | ||
| nox -s docs -- gh-deploy --force --remote-branch gh-pages | ||
| git fetch origin gh-pages:gh-pages | ||
| nox -s docs -- deploy --update-aliases dev | ||
| git push origin gh-pages | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,85 +1,147 @@ | ||
| <div align="center"> | ||
|
|
||
| <img src="./docs/images/eva-logo.png" width="400"> | ||
| <img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/eva-logo.png?raw=true" width="400"> | ||
|
|
||
| <br /> | ||
|
|
||
| _Oncology FM Evaluation Framework by kaiko.ai_ | ||
|
|
||
|
|
||
| <a href="https://www.apache.org/licenses/LICENSE-2.0"> | ||
| <img src="https://img.shields.io/badge/License-Apache%202.0-blue?style=flat-square" /> | ||
| </a> | ||
|
|
||
| <br /> | ||
| <br /> | ||
| [](https://pypi.python.org/pypi/kaiko-eva) | ||
| [](https://github.com/kaiko-ai/eva/actions?query=workflow%3ACI) | ||
| [](https://github.com/kaiko-ai/eva#license) | ||
|
|
||
| <p align="center"> | ||
| <a href="#installation">Installation</a> • | ||
| <a href="#how-to-use">How To Use</a> • | ||
| <a href="#datasets">Datasets</a> • | ||
| <a href="#contributing">Contribute</a> | ||
| <a href="https://github.com/kaiko-ai/eva#installation">Installation</a> • | ||
| <a href="https://github.com/kaiko-ai/eva#how-to-use">How To Use</a> • | ||
| <a href="https://kaiko-ai.github.io/eva/">Documentation</a> • | ||
| <a href="https://kaiko-ai.github.io/eva/dev/datasets/">Datasets</a> • | ||
| <a href="https://github.com/kaiko-ai/eva#benchmarks">Benchmarks</a> <br> | ||
| <a href="https://github.com/kaiko-ai/eva#contributing">Contribute</a> • | ||
| <a href="https://github.com/kaiko-ai/eva#acknowledgements">Acknowledgements</a> | ||
| </p> | ||
|
|
||
| </div> | ||
|
|
||
| --- | ||
|
|
||
| ### _About_ | ||
| <br /> | ||
|
|
||
| `eva` is [kaiko.ai](https://kaiko.ai/)'s evaluation framework for oncology foundation models (FMs). Check out the documentation (LINK TO BE ADDED) for more information. | ||
| _`eva`_ is an evaluation framework for oncology foundation models (FMs) by [kaiko.ai](https://kaiko.ai/). Check out the [documentation](https://kaiko-ai.github.io/eva/) for more information. | ||
|
|
||
| ### Highlights: | ||
| - Easy and reliable benchmark of Oncology FMs | ||
| - Automatic embedding inference and evaluation of a downstream task | ||
| - Native support of popular medical [datasets](https://kaiko-ai.github.io/eva/dev/datasets/) and models | ||
| - Produce statistics over multiple evaluation fits and multiple metrics | ||
|
|
||
| ## Installation | ||
|
|
||
| *Note: this section will be revised for the public package when publishing eva* | ||
| Simple installation from PyPI: | ||
| ```sh | ||
| # to install the core version only | ||
| pip install kaiko-eva | ||
|
|
||
| - Create and activate a virtual environment with Python 3.10+ | ||
| # to install the expanded `vision` version | ||
| pip install 'kaiko-eva[vision]' | ||
|
|
||
| - Install *eva* and the *eva-vision* package with: | ||
| # to install everything | ||
| pip install 'kaiko-eva[all]' | ||
| ``` | ||
|
|
||
| To install the latest version of the `main` branch: | ||
| ```sh | ||
| pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git" | ||
| ``` | ||
| pip install 'kaiko-eva[vision]' | ||
|
|
||
| You can verify that the installation was successful by executing: | ||
| ```sh | ||
| eva --version | ||
| ``` | ||
|
|
||
| - To be able to use the existing configs, download them from the [*eva* GitHub repo](https://github.com/kaiko-ai/eva/tree/main) and move them to directory where you installed *eva*. | ||
| ## How To Use | ||
|
|
||
| ### Run *eva* | ||
| _eva_ can be used directly from the terminal as a CLI tool as follows: | ||
| ```sh | ||
| eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml | ||
| ``` | ||
|
|
||
| Now you can run a complete *eva* workflow, for example with: | ||
| For example, to perform a downstream evaluation of DINO ViT-S/16 on the BACH dataset with linear probing by first inferring the embeddings and performing 5 sequential fits, execute: | ||
| ```sh | ||
| eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml | ||
| ``` | ||
| eva fit --config configs/vision/dino_vit/online/bach.yaml | ||
|
|
||
| > [!NOTE] | ||
| > All the datasets that support automatic download in the repo have by default the option to automatically download set to false. For automatic download you have to manually set download=true. | ||
|
|
||
| To view all the possibles, execute: | ||
| ```sh | ||
| eva --help | ||
| ``` | ||
| This will: | ||
|
|
||
| - Download and extract the dataset, if it has not been downloaded before. | ||
| - Fit a model consisting of the frozen FM-backbone and a classification head on the train split. | ||
| - Evaluate the trained model on the validation split and report the results. | ||
| For more information, please refer to the [documentation](https://kaiko-ai.github.io/eva/dev/user-guide/tutorials/offline_vs_online/) and [tutorials](https://kaiko-ai.github.io/eva/dev/user-guide/advanced/replicate_evaluations/). | ||
|
|
||
| For more information, documentation and tutorials, refer to the documentation (LINK TO BE ADDED). | ||
| ## Benchmarks | ||
|
|
||
| ## Datasets | ||
| In this section you will find model benchmarks which were generated with _eva_. | ||
|
|
||
| The following datasets are supported natively: | ||
| ### Table I: WSI patch-level benchmark | ||
|
|
||
| ### Vision | ||
| <br /> | ||
|
|
||
| <div align="center"> | ||
|
|
||
| | Model | BACH | CRC | MHIST | PCam/val | PCam/test | | ||
| |--------------------------------------------------|-------|-------|-------|----------|-----------| | ||
| | ViT-S/16 _(random)_ <sup>[1]</sup> | 0.410 | 0.617 | 0.501 | 0.753 | 0.728 | | ||
| | ViT-S/16 _(ImageNet)_ <sup>[1]</sup> | 0.695 | 0.935 | 0.831 | 0.864 | 0.849 | | ||
| | ViT-B/8 _(ImageNet)_ <sup>[1]</sup> | 0.710 | 0.939 | 0.814 | 0.870 | 0.856 | | ||
| | DINO<sub>(p=16)</sub> <sup>[2]</sup> | 0.801 | 0.934 | 0.768 | 0.889 | 0.895 | | ||
| | Phikon <sup>[3]</sup> | 0.725 | 0.935 | 0.777 | 0.912 | 0.915 | | ||
| | ViT-S/16 _(kaiko.ai)_ <sup>[4]</sup> | 0.797 | 0.943 | 0.828 | 0.903 | 0.893 | | ||
| | ViT-S/8 _(kaiko.ai)_ <sup>[4]</sup> | 0.834 | 0.946 | 0.832 | 0.897 | 0.887 | | ||
| | ViT-B/16 _(kaiko.ai)_ <sup>[4]</sup> | 0.810 | 0.960 | 0.826 | 0.900 | 0.898 | | ||
| | ViT-B/8 _(kaiko.ai)_ <sup>[4]</sup> | 0.865 | 0.956 | 0.809 | 0.913 | 0.921 | | ||
| | ViT-L/14 _(kaiko.ai)_ <sup>[4]</sup> | 0.870 | 0.930 | 0.809 | 0.908 | 0.898 | | ||
|
|
||
| _Table I: Linear probing evaluation of FMs on patch-level downstream datasets.<br> We report averaged balanced accuracy | ||
| over 5 runs, with an average standard deviation of ±0.003._ | ||
|
|
||
| #### Patch-level pathology datasets: | ||
| - [BACH](./docs/datasets/bach.md) | ||
| - [CRC](./docs/datasets/crc.md) | ||
| - [MHIST](./docs/datasets/mhist.md) | ||
| - [PatchCamelyon](./docs/datasets/patch_camelyon.md) | ||
| </div> | ||
|
|
||
| <br /> | ||
|
|
||
| #### Radiology datasets: | ||
| - [TotalSegmentator](./docs/datasets/total_segmentator.md) | ||
| _References_: | ||
| 1. _"Emerging properties in self-supervised vision transformers”_ | ||
| 2. _"Benchmarking self-supervised learning on diverse pathology datasets”_ | ||
| 3. _"Scaling self-supervised learning for histopathology with masked image modeling”_ | ||
| 4. _"Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”_ | ||
|
|
||
| ## Contributing | ||
|
|
||
| _eva_ is an open source project and welcomes contributions of all kinds. Please checkout the [developer](./docs/DEVELOPER_GUIDE.md) and [contributing guide](./docs/CONTRIBUTING.md) for help on how to do so. | ||
|
|
||
| All contributors must follow the [code of conduct](./docs/CODE_OF_CONDUCT.md). | ||
|
|
||
|
|
||
| ## Acknowledgements | ||
|
|
||
| Our codebase is built using multiple opensource contributions | ||
|
|
||
| <div align="center"> | ||
|
|
||
| [](https://github.com/pre-commit/pre-commit) | ||
| [](https://pytorch.org/get-started/locally/) | ||
| [](https://pytorchlightning.ai/)<br> | ||
| [](https://black.readthedocs.io/en/stable/) | ||
| [](https://pycqa.github.io/isort/) | ||
| [](https://github.com/astral-sh/ruff) | ||
| [](https://microsoft.github.io/pyright/)<br> | ||
| [](https://pdm-project.org) | ||
| [](https://github.com/wntrblm/nox) | ||
| [](https://squidfunk.github.io/mkdocs-material/) | ||
|
|
||
| </div> | ||
|
|
||
| --- | ||
| <div align="center"> | ||
| <img src="./docs/images/kaiko-logo.png" width="200"> | ||
| <img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/kaiko-logo.png?raw=true" width="200"> | ||
| </div> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.