Adds LLM support to obtain the repository URL

.gitignore

@@ -41,6 +41,7 @@ prospector/install_fastext.sh
 prospector/nvd.ipynb
 prospector/data/nvd.pkl
 prospector/data/nvd.csv
+prospector/data_sources/reports
 .vscode/settings.json
 prospector/cov_html/*
 prospector/client/cli/cov_html/*
@@ -51,6 +52,7 @@ prospector/.coverage
 **/cov_html
 prospector/cov_html
 .coverage
+prospector/.venv
 prospector/prospector.code-workspace
 prospector/requests-cache.sqlite
 prospector/prospector-report.html

prospector/README.md

@@ -5,18 +5,29 @@ currently under development: the instructions below are intended for development
 
 :exclamation: Please note that **Windows is not supported** while WSL and WSL2 are fine.
 
-## Description
+## Table of Contents
+
+1. [Description](#description)
+2. [Quick Setup & Run](#setup--run)
+3. [Development Setup](#development-setup)
+4. [Contributing](#contributing)
+5. [History](#history)
+
+## 📖 Description
 
 Prospector is a tool to reduce the effort needed to find security fixes for
 *known* vulnerabilities in open source software repositories.
 
 Given an advisory expressed in natural language, Prospector processes the commits found in the target source code repository, ranks them based on a set of predefined rules, and produces a report that the user can inspect to determine which commits to retain as the actual fix.
 
-## Setup & Run
+## ⚡️ Quick Setup & Run
+
+Prerequisites:
 
-:warning: The tool requires Docker and Docker-compose, as it employes Docker containers for certain functionalities. Make sure you have Docker installed and running before proceeding with the setup and usage of Prospector.
+* Docker (make sure you have Docker installed and running before proceeding with the setup)
+* Docker-compose
 
-To quickly set up Prospector:
+To quickly set up Prospector, follow these steps. This will run Prospector in its containerised version. If you wish to debug or run Prospector's components individually, follow the steps below at [Development Setup](#development-setup).
 
 1. Clone the project KB repository
     ```
@@ -44,7 +55,52 @@ To quickly set up Prospector:
     By default, Prospector saves the results in a HTML file named *prospector-report.html*.
     Open this file in a web browser to view what Prospector was able to find!
 
-## Development Setup
+### 🤖 LLM Support
+
+To use Prospector with LLM support, set the `use_llm_<...>` parameters in `config.yaml`. Additionally, you must specify required parameters for API access to the LLM. These parameters can vary depending on your choice of provider, please follow what fits your needs:
+
+<details><summary><b>Use SAP AI CORE SDK</b></summary>
+
+You will need the following parameters in `config.yaml`:
+
+```yaml
+llm_service:
+    type: sap
+    model_name: <model_name>
+```
+
+`<model_name>` refers to the model names available in the Generative AI Hub in SAP AI Core. [Here](https://github.tools.sap/I343697/generative-ai-hub-readme#1-supported-models) you can find an overview of available models.
+
+In `.env`, you must set the deployment URL as an environment variable following this naming convention:
+```yaml
+<model_name (in capitals, and - changed to _)>_URL
+```
+
+</details>
+
+<details><summary><b>Use personal third party provider</b></summary>
+
+Implemented third party providers are **OpenAI**, **Google** and **Mistral**.
+
+1. You will need the following parameters in `config.yaml`:
+    ```yaml
+    llm_service:
+        type: third_party
+        model_name: <model_name>
+    ```
+
+    `<model_name>` refers to the model names available, for example `gpt-4o` for OpenAI. You can find a lists of available models here:
+   1. [OpenAI](https://platform.openai.com/docs/models)
+   2. [Google](https://ai.google.dev/gemini-api/docs/models/gemini)
+   3. [Mistral](https://docs.mistral.ai/getting-started/models/)
+
+2. Make sure to add your OpenAI API key to your `.env` file as `[OPENAI|GOOGLE|MISTRAL]_API_KEY`.
+
+</details>
+
+## 👩‍💻 Development Setup
+
+Following these steps allows you to run Prospector's components individually: [Backend database and worker containers](#starting-the-backend-database-and-the-job-workers), [RESTful Server](#starting-the-restful-server) for API endpoints, [Prospector CLI](#running-the-cli-version) and [Tests](#testing).
 
 Prerequisites:
 
@@ -53,6 +109,8 @@ Prerequisites:
 * gcc g++ libffi-dev python3-dev libpq-dev
 * Docker & Docker-compose
 
+### General
+
 You can setup everything and install the dependencies by running:
 ```
 make setup
@@ -81,11 +139,13 @@ your editor so that autoformatting is enforced "on save". The pre-commit hook en
 black is run prior to committing anyway, but the auto-formatting might save you some time
 and avoid frustration.
 
-If you use VSCode, this can be achieved by pasting these lines in your configuration file:
+If you use VSCode, this can be achieved by installing the Black Formatter extension and pasting these lines in your configuration file:
 
-```
-    "python.formatting.provider": "black",
-    "editor.formatOnSave": true,
+```json
+    "[python]": {
+        "editor.defaultFormatter": "ms-python.black-formatter",
+        "editor.formatOnSave": true,
+    }
 ```
 
 ### Starting the backend database and the job workers
@@ -94,17 +154,23 @@ If you run the client without running the backend you will get a warning and hav
 
 You can then start the necessary containers with the following command:
 
-`make docker-setup`
+```bash
+make docker-setup
+```
 
 This also starts a convenient DB administration tool at http://localhost:8080
 
 If you wish to cleanup docker to run a fresh version of the backend you can run:
 
-`make docker-clean`
+```bash
+make docker-clean
+```
 
 ### Starting the RESTful server
 
-`uvicorn api.main:app --reload`
+```bash
+uvicorn service.main:app --reload
+```
 
 Note, that it requires `POSTGRES_USER`, `POSTGRES_HOST`, `POSTGRES_PORT`, `POSTGRES_DBNAME` to be set in the .env file.
 
@@ -113,7 +179,9 @@ You might also want to take a look at `http://127.0.0.1:8000/docs`.
 
 *Alternatively*, you can execute the RESTful server explicitly with:
 
-`python api/main.py`
+```bash
+python api/main.py
+```
 
 which is equivalent but more convenient for debugging.
 
@@ -127,11 +195,13 @@ Prospector makes use of `pytest`.
 
 :exclamation: **NOTE:** before using it please make sure to have running instances of the backend and the database.
 
+## 🤝 Contributing
+
 If you find a bug, please open an issue. If you can also fix the bug, please
 create a pull request (make sure it includes a test case that passes with your correction
 but fails without it)
 
-## History
+## 🕰️ History
 
 The high-level structure of Prospector follows the approach of its
 predecessor FixFinder, which is described in:

prospector/cli/main.py

@@ -7,6 +7,7 @@
 
 from dotenv import load_dotenv
 
+import llm.llm_operations as llm
 from util.http import ping_backend
 
 path_root = os.getcwd()
@@ -32,10 +33,12 @@ def main(argv):  # noqa: C901
     with ConsoleWriter("Initialization") as console:
         config = get_configuration(argv)
         if not config:
-            logger.error("No configuration file found. Cannot proceed.")
+            logger.error(
+                "No configuration file found, or error in configuration file. Cannot proceed."
+            )
 
             console.print(
-                "No configuration file found.",
+                "No configuration file found, or error in configuration file. Check logs.",
                 status=MessageStatus.ERROR,
             )
             return
@@ -51,6 +54,16 @@ def main(argv):  # noqa: C901
             )
             return
 
+        if not config.repository and not config.use_llm_repository_url:
+            logger.error(
+                "Either provide the repository URL or allow LLM usage to obtain it."
+            )
+            console.print(
+                "Either provide the repository URL or allow LLM usage to obtain it.",
+                status=MessageStatus.ERROR,
+            )
+            sys.exit(1)
+
         # if config.ping:
         #     return ping_backend(backend, get_level() < logging.INFO)
 
@@ -63,6 +76,12 @@ def main(argv):  # noqa: C901
 
         logger.debug("Vulnerability ID: " + config.vuln_id)
 
+    # whether to use LLM support
+    if not config.repository:
+        config.repository = llm.get_repository_url(
+            llm_config=config.llm, vuln_id=config.vuln_id
+        )
+
     results, advisory_record = prospector(
         vulnerability_id=config.vuln_id,
         repository_url=config.repository,
@@ -88,7 +107,7 @@ def main(argv):  # noqa: C901
     )
 
     execution_time = execution_statistics["core"]["execution time"][0]
-    ConsoleWriter.print(f"Execution time: {execution_time:.3f}s")
+    ConsoleWriter.print(f"Execution time: {execution_time:.3f}s\n")
 
     return
 

prospector/config-sample.yaml

@@ -1,5 +1,3 @@
-
-
 # Wheter to preprocess only the repository's commits or fully run prospector
 preprocess_only: False
 
@@ -12,7 +10,7 @@ fetch_references: False
 use_nvd: True
 
 # The NVD API token
-nvd_token: Null
+# nvd_token: <your_nvd_api_token>
 
 # Wheter to use a backend or not: "always", "never", "optional"
 use_backend: optional
@@ -30,6 +28,13 @@ database:
 
 redis_url: redis://redis:6379/0
 
+# LLM Usage (check README for help)
+llm_service:
+  type: sap
+  model_name: gpt-4-turbo
+
+use_llm_repository_url: True # whether to use LLM's to obtain the repository URL
+
 # Report file format: "html", "json", "console" or "all"
 # and the file name
 report:
@@ -43,4 +48,4 @@ log_level: INFO
 git_cache: /tmp/gitcache
 
 # The GitHub API token
-github_token: Null
+# github_token: <your_api_token>

prospector/core/prospector.py

@@ -1,6 +1,7 @@
 # flake8: noqa
 
 import logging
+import os
 import re
 import sys
 import time
@@ -36,7 +37,7 @@
 ONE_YEAR = 365 * SECS_PER_DAY
 
 MAX_CANDIDATES = 2000
-DEFAULT_BACKEND = "http://localhost:8000"
+DEFAULT_BACKEND = "http://backend:8000"
 
 
 core_statistics = execution_statistics.sub_collection("core")
@@ -157,7 +158,14 @@ def prospector(  # noqa: C901
                     exc_info=get_level() < logging.WARNING,
                 )
                 if use_backend == "always":
-                    print("Backend not reachable: aborting")
+                    if backend_address == "http://localhost:8000" and os.path.exists(
+                        "/.dockerenv"
+                    ):
+                        print(
+                            "The backend address should be 'http://backend:8000' when running the containerised version of Prospector: aborting"
+                        )
+                    else:
+                        print("Backend not reachable: aborting")
                     sys.exit(1)
                 print("Backend not reachable: continuing")
 
@@ -227,7 +235,7 @@ def preprocess_commits(commits: List[RawCommit], timer: ExecutionTimer) -> List[
 
 
 def filter(commits: Dict[str, RawCommit]) -> Dict[str, RawCommit]:
-    with ConsoleWriter("\nCandidate filtering\n") as console:
+    with ConsoleWriter("\nCandidate filtering") as console:
         commits, rejected = filter_commits(commits)
         if rejected > 0:
             console.print(f"Dropped {rejected} candidates")

prospector/datamodel/nlp.py

@@ -139,23 +139,24 @@ def extract_ghissue_references(repository: str, text: str) -> Dict[str, str]:
         id = result.group(1)
         url = f"{repository}/issues/{id}"
         content = fetch_url(url=url, extract_text=False)
-        gh_ref_data = content.find_all(
-            attrs={
-                "class": ["comment-body", "markdown-title"],
-            },
-            recursive=False,
-        )
-        # TODO: when an issue/pr is referenced somewhere, the page contains also the "message" of that reference (e.g. a commit). This may lead to unwanted detection of certain rules.
-        gh_ref_data.extend(
-            content.find_all(
+        if content is not None:
+            gh_ref_data = content.find_all(
                 attrs={
-                    "id": re.compile(r"ref-issue|ref-pullrequest"),
-                }
+                    "class": ["comment-body", "markdown-title"],
+                },
+                recursive=False,
+            )
+            # TODO: when an issue/pr is referenced somewhere, the page contains also the "message" of that reference (e.g. a commit). This may lead to unwanted detection of certain rules.
+            gh_ref_data.extend(
+                content.find_all(
+                    attrs={
+                        "id": re.compile(r"ref-issue|ref-pullrequest"),
+                    }
+                )
+            )
+            refs[id] = " ".join(
+                [" ".join(block.get_text().split()) for block in gh_ref_data]
             )
-        )
-        refs[id] = " ".join(
-            [" ".join(block.get_text().split()) for block in gh_ref_data]
-        )
 
     return refs