Merge pull request #326 from usnistgov/develop

Develop

.github/workflows/test_build.yml

@@ -4,29 +4,32 @@ on: [push, pull_request]
 
 
 
-
 jobs:
-  build:
-    runs-on: ubuntu-latest
+  miniconda:
+    name: Miniconda ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+        matrix:
+            os: ["ubuntu-latest"]
     steps:
-    - uses: actions/checkout@v3
-    - name: Set up Python 3.9
-      uses: actions/setup-python@v4
-      with:
-        python-version: 3.9
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip setuptools
-        pip install pytest coverage==6.5.0 flake8 pymatgen
-        #pip install codecov
-        python setup.py develop
-        #python -m pip install -r requirements.txt
-        echo 'PIP FREEZE'
-        pip freeze
-    - name: Build site
-      run: |
-        # flake8 --ignore E203,W503 --exclude=tests --statistics --count --exit-zero jarvis_leaderboard
-        coverage run -m pytest
-        coverage report -m -i
-        #codecov
-        #codecov --token="eba3e27a-b0c9-421a-a7f6-4ff5111fc4f0"
+      - uses: actions/checkout@v2
+      - uses: conda-incubator/setup-miniconda@v2
+        with:
+          activate-environment: test
+          environment-file: environment.yml
+          python-version: "3.10"
+          auto-activate-base: false
+      - shell: bash -l {0}
+        run: |
+          conda info
+          conda list
+          python setup.py develop
+          echo 'PIP FREEZE'
+          pip freeze
+          echo 'CONDA LIST'
+          conda list
+          pip install pytest coverage==6.5.0 flake8
+          coverage run -m pytest
+          coverage report -m -i          
+              
+

docs/guide_short.md

@@ -0,0 +1,104 @@
+# Guide to JARVIS-Leaderboard
+
+## Introduction
+JARVIS-Leaderboard is an open-source, community-driven platform that facilitates benchmarking and enhances reproducibility in materials design. Users can set up benchmarks with custom tasks and contribute datasets, code, and meta-data. The platform covers five main categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC), and Experiments (EXP).
+
+## External Resources
+- [Powerpoint slides](https://lnkd.in/eNg4w6Cz)
+- [YouTube video](https://www.youtube.com/embed/QDx3jSIwpMo?autoplay=1&mute=1)
+
+## Terminologies
+
+### Categories
+- **AI:** Input data types include atomic structures, images, spectra, and text.
+- **ES:** Involves various ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiments.
+- **FF:** Focuses on multiple approaches for material property predictions.
+- **QC:** Benchmarks Hamiltonian simulations using various quantum algorithms and circuits.
+- **EXP:** Utilizes inter-laboratory approaches to establish benchmarks.
+
+### Sub-categories
+1. SinglePropertyPrediction
+2. SinglePropertyClass
+3. ImageClass
+4. textClass
+5. MLFF (machine learning force-field)
+6. Spectra
+7. EigenSolver
+
+### Benchmarks
+Ground truth data used to calculate metrics for specific tasks (e.g., a json.zip file).
+
+### Methods
+Precise specifications for evaluation against a benchmark (e.g., DFT with VASP-GGA-PAW-PBE in the ES category).
+
+### Contributions
+Individual data in the form of csv.zip files for each benchmark and method. Each contribution includes:
+- Method (e.g., AI)
+- Category (e.g., SinglePropertyPrediction)
+- Property (e.g., formation energy)
+- Dataset (e.g., dft_3d)
+- Data-split (e.g., test)
+- Metric (e.g., mae)
+
+## Directory and File Structure
+![Tree](https://raw.githubusercontent.com/usnistgov/jarvis_leaderboard/develop/jarvis_leaderboard/Tree.jpg)
+
+## How to Contribute
+
+### Adding a Contribution (csv.zip)
+1. Fork the JARVIS-Leaderboard repository on GitHub.
+2. Clone your forked repository:
+   ```bash
+   git clone https://github.com/USERNAME/jarvis_leaderboard
+   ```
+3. Create a Python environment:
+   ```bash
+   conda create --name leaderboard python=3.8
+   source activate leaderboard
+   ```
+4. Install the package:
+   ```bash
+   python setup.py develop
+   ```
+5. Add a contribution:
+   ```bash
+   cd jarvis_leaderboard/contributions/
+   mkdir vasp_pbe_teamX
+   cd vasp_pbe_teamX
+   cp ../vasp_optb88vdw/ES-SinglePropertyPrediction-bandgap_JVASP_1002_Si-dft_3d-test-mae.csv.zip .
+   vi ES-SinglePropertyPrediction-bandgap_JVASP_1002_Si-dft_3d-test-mae.csv.zip
+   ```
+6. Modify the prediction value in the csv file, add `metadata.json` and `run.sh` files.
+7. Rebuild the leaderboard:
+   ```bash
+   cd ../../../
+   python jarvis_leaderboard/rebuild.py
+   mkdocs serve
+   ```
+8. Commit and push your changes:
+   ```bash
+   git add jarvis_leaderboard/contributions/vasp_pbe_teamX
+   git commit -m 'Adding my PBE Si result.'
+   git push
+   ```
+9. Create a pull request on GitHub.
+
+### Adding a New Benchmark (json.zip)
+1. Create a `json.zip` file in the `jarvis_leaderboard/benchmarks` folder.
+2. Add a `.json` file with `train`, `val`, `test` keys.
+3. Add a corresponding `.md` file in the `jarvis_leaderboard/docs` folder.
+4. Follow instructions for "Adding model benchmarks to existing dataset".
+
+## Acronyms
+1. MAE: Mean Absolute Error
+2. ACC: Classification accuracy
+3. MULTIMAE: MAE sum of multiple entries, Euclidean distance
+
+## Help
+For questions or concerns, raise an [issue on GitHub](https://github.com/usnistgov/jarvis_leaderboard/issues) or email Kamal Choudhary ([email protected]).
+
+## Citation
+[JARVIS-Leaderboard: a large scale benchmark of materials design methods](https://www.nature.com/articles/s41524-024-01259-w)
+
+## License
+This template is served under the NIST license. Read the [LICENSE] file for more info.

jarvis_leaderboard/benchmarks/benchmark_dois.json

@@ -208,6 +208,9 @@
                 "https://www.nature.com/articles/s41524-020-00440-1",
                 "https://doi.org/10.48550/arXiv.2305.11842"
             ],
+            "dft_3d_Tc_supercon.json.zip": [
+                "https://www.nature.com/articles/s41524-022-00933-1"
+            ],
             "dft_3d_Tc_supercon_hydride_plus_bulk.json.zip": [
                 "https://iopscience.iop.org/article/10.1088/2752-5724/ad4a94",
                 "https://www.nature.com/articles/s41524-020-00440-1",

jarvis_leaderboard/benchmarks/descriptions.csv

@@ -48,6 +48,7 @@ AI,SinglePropertyPrediction,qm9_std_jctc_ZPVE,This is a benchmark to evaluate ho
 AI,SinglePropertyPrediction,qm9_std_jctc_alpha,This is a benchmark to evaluate how accurately an AI model can predict the isotropic polarizability using the QM9 dataset. The dataset contains the structures and properties of molecules. Here we use mean absolute error (MAE) to compare models with respect to DFT accuracy. External links: https://www.nature.com/articles/sdata201422,
 AI,SinglePropertyPrediction,dft_3d_Tc_supercon_hydride,This is a benchmark to evaluate how accurately an AI model can predict the superconducting transition temperature of hydride materials under pressure. The dataset contains the structures and DFT calculated Tc of the hydride materials under various amounts of pressure. Here we use mean absolute error (MAE) to compare models with respect to DFT accuracy. External links: https://iopscience.iop.org/article/10.1088/2752-5724/ad4a94,
 AI,SinglePropertyPrediction,dft_3d_Tc_supercon_hydride_plus_bulk,"This is a benchmark to evaluate how accurately an AI model can predict the superconducting transition temperature of hydride materials under pressure. The dataset contains the structures and DFT calculated Tc of the hydride materials under various amounts of pressure plus the DFT dataset of bulk hydride materials. Here we use mean absolute error (MAE) to compare models with respect to DFT accuracy. External links: https://iopscience.iop.org/article/10.1088/2752-5724/ad4a95, https://www.nature.com/articles/s41524-022-00933-1",
+AI,SinglePropertyPrediction,dft_3d_Tc_supercon,"This is a benchmark to evaluate how accurately an AI model can predict the superconducting transition temperature of 3D materials. Ref: https://www.nature.com/articles/s41524-022-00933-1.",
 AI,SinglePropertyPrediction,dft_3d_avg_elec_mass,This is a benchmark to evaluate how accurately an AI model can predict the average electron mass using the JARVIS-DFT (dft_3d) dataset. The dataset contains different types of chemical formula and atomic structures. Here we use mean absolute error (MAE) to compare models with respect to DFT accuracy.,
 AI,SinglePropertyPrediction,dft_3d_avg_hole_mass,This is a benchmark to evaluate how accurately an AI model can predict the average hole mass using the JARVIS-DFT (dft_3d) dataset. The dataset contains different types of chemical formula and atomic structures. Here we use mean absolute error (MAE) to compare models with respect to DFT accuracy.,
 AI,SinglePropertyPrediction,qmof_bandgap,This is a benchmark to evaluate how accurately an AI model can predict the bandgap using the QMOF dataset. The dataset contains different types of metal organic frameworks. Here we use mean absolute error (MAE) to compare models with respect to DFT accuracy. External links: https://github.com/Andrew-S-Rosen/QMOF,
@@ -297,4 +298,4 @@ FF,SinglePropertyPrediction,biobench_deltaF,This is a benchmark to evaluate how
 FF,SinglePropertyPrediction,biobench_left_handed_population,This is a benchmark to evaluate how accurately an FF model can predict bulk free energy difference of peptides.,
 FF,SinglePropertyPrediction,biobench_right_handed_population,This is a benchmark to evaluate how accurately an FF model can predict bulk free energy difference of peptides.,
 FF,SinglePropertyPrediction,lj_2d_liquid_viscosity,This is a benchmark to evaluate how accurately an FF model can predict viscosity of Lennard-Jones liquid.,
-QC,EigenSolver,dft_3d_electron_bands_JVASP_816_Al_WTBH,This is a benchmark to evaluate how accurately an QC model can predict electronic bands of aluminum (JVASP-816) produced using Wannier tight binding Hamiltonian approach.,
+QC,EigenSolver,dft_3d_electron_bands_JVASP_816_Al_WTBH,This is a benchmark to evaluate how accurately an QC model can predict electronic bands of aluminum (JVASP-816) produced using Wannier tight binding Hamiltonian approach.,

jarvis_leaderboard/contributions/atomgpt_model/metadata.json

@@ -0,0 +1,20 @@
+{
+    "model_name": "AtomGPT",
+    "project_url": "https://arxiv.org/abs/2405.03680",
+    "date_submitted": "05-23-2024",
+    "author_email": "[email protected]",
+    "database_version": "12-12-2022",
+    "team_name": "AtomGPT",
+    "time_taken_seconds": {
+        "AI-SinglePropertyClass-mbj_bandgap-dft_3d-test-acc.csv.zip": "",
+        "AI-SinglePropertyClass-optb88vdw_bandgap-dft_3d-test-acc.csv.zip": "",
+        "AI-SinglePropertyPrediction-formation_energy_peratom-dft_3d-test-mae.csv.zip": ""
+    },
+    "language": "python",
+    "os": "linux",
+    "software_used": "jarvis-tools,numpy,scipy,torch,atomgpt",
+    "hardware_used": "nisaba-cluster at NIST, V100 Tesla GPU",
+    "git_url": [
+        "https://github.com/usnistgov/atomgpt"
+    ]
+}

jarvis_leaderboard/contributions/atomgpt_model/run.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+pip install atomgpt
+python run.py