Code

Evaluation Scripts#408

Code

Merged

copernico merged 131 commits intoSAP:mainSAP/project-kb:mainfrom lauraschauer:evaluation-scriptslauraschauer/project-kb:evaluation-scriptsCopy head branch name to clipboardSep 2, 2024

+11,560-62

Commits

Commits on Aug 16, 2024

• send one request to update security relevance of a batch of commits (instead of sending a request for each individual commit)

  Show description for 55e1420
  lauraschauer

  committed
  55e1420

  Copy full SHA for 55e1420

• Reduces throwing errors when the LLM returns a verbose answer in commit classification.

  Show description for 19d16fe
  lauraschauer

  committed
  19d16fe

  Copy full SHA for 19d16fe

• cleaner error logging when git command fails

  lauraschauer

  committed
  5af0501

  Copy full SHA for 5af0501

• Advisory References: commit

  Show description for 552509b
  lauraschauer

  committed
  552509b

  Copy full SHA for 552509b

• fixes bug when extracting text from Jira Refs

  lauraschauer

  committed
  3be03fb

  Copy full SHA for 3be03fb

• removes 'dubious ownership' error when trying to access git repositories with a different user'

  lauraschauer

  committed
  c7dbd7f

  Copy full SHA for c7dbd7f

• prints error to console instead of crashing at None, -1 returned from Prospector

  lauraschauer

  committed
  ba72de1

  Copy full SHA for ba72de1

Commits on Aug 26, 2024

• [FIX] makes sure that commit classification information is only retrieved when using backend is optional or always

  lauraschauer

  committed
  9108729

  Copy full SHA for 9108729

Commits on Aug 29, 2024

• sorts and cleans up gitignore

  lauraschauer

  committed
  a57769b

  Copy full SHA for a57769b

• orders and cleans up gitignore

  lauraschauer

  committed
  bcf7279

  Copy full SHA for bcf7279

• renames folder with pipeline code to pipeline

  lauraschauer

  committed
  e76a19a

  Copy full SHA for e76a19a

• comments out cluttering print statement

  lauraschauer

  committed
  3ed03ed

  Copy full SHA for 3ed03ed

• adds the pipeline folder as a volume

  lauraschauer

  committed
  2870cf3

  Copy full SHA for 2870cf3

• adds the full code folder as a volume to the worker container

  lauraschauer

  committed
  98074ff

  Copy full SHA for 98074ff

• adjusts import paths after renaming folder to pipeline

  lauraschauer

  committed
  53d53a2

  Copy full SHA for 53d53a2

• adds console output for steps of pipeline

  lauraschauer

  committed
  a409c12

  Copy full SHA for a409c12

• more adjustment after renaming to pipeline

  lauraschauer

  committed
  a07ea30

  Copy full SHA for a07ea30

• updates and sorts gitignore

  lauraschauer

  committed
  aa71baa

  Copy full SHA for aa71baa

• can successfully generate reports with the pipeline

  lauraschauer

  committed
  f077121

  Copy full SHA for f077121

• cleans up code

  lauraschauer

  committed
  04746d2

  Copy full SHA for 04746d2

• adds readme

  lauraschauer

  committed
  44c4d5b

  Copy full SHA for 44c4d5b

Commits on Aug 30, 2024

• adds LLM statistics to statistics object and

  Show description for 95bbacc
  lauraschauer

  committed
  95bbacc

  Copy full SHA for 95bbacc

• adds LLM statistics to statistics object and

  Show description for a670bc8
  lauraschauer

  committed
  a670bc8

  Copy full SHA for a670bc8

• adds matteo's script

  lauraschauer

  committed
  5697bfe

  Copy full SHA for 5697bfe

• script runs

  lauraschauer

  committed
  b2c6690

  Copy full SHA for b2c6690

• results from run

  lauraschauer

  committed
  aa22101

  Copy full SHA for aa22101

• adds collection of LLM stats to the LLM class

  lauraschauer

  committed
  03e1cef

  Copy full SHA for 03e1cef

• adds folder structure for reports and 1 first report generated in containerised Prospector with LLM

  lauraschauer

  committed
  ef1c1a2

  Copy full SHA for ef1c1a2

• breaks up matteo's script into smaller files

  lauraschauer

  committed
  67fca69

  Copy full SHA for 67fca69

• improves code

  lauraschauer

  committed
  edf0894

  Copy full SHA for edf0894

• adds LLM generated docstrings to start understanding what's happening

  lauraschauer

  committed
  1c5803d

  Copy full SHA for 1c5803d

• adds function to analyse llm statistics across Prospector reports

  lauraschauer

  committed
  aee608f

  Copy full SHA for aee608f

• adds CL argument for llm statistics evaluation

  lauraschauer

  committed
  ba799b6

  Copy full SHA for ba799b6

• makes analysis code more robust to JSON report files missing fields

  lauraschauer

  committed
  ab6f22b

  Copy full SHA for ab6f22b

• adds seaborn and matplotlib to dev requirements

  lauraschauer

  committed
  fe18896

  Copy full SHA for fe18896

• logs to console

  lauraschauer

  committed
  f18c644

  Copy full SHA for f18c644

• adds readme

  lauraschauer

  committed
  153ba2e

  Copy full SHA for 153ba2e

• bundels run variables in config.yaml and adds readme

  lauraschauer

  committed
  5d9fa02

  Copy full SHA for 5d9fa02

• this is the dataset used in D6.3

  lauraschauer

  committed
  5043ce6

  Copy full SHA for 5043ce6

• renames dataset

  lauraschauer

  committed
  35e93ae

  Copy full SHA for 35e93ae

• adds dataset description to readme

  lauraschauer

  committed
  c5f8b70

  Copy full SHA for c5f8b70

• mounts volume so that reports are available on the host

  lauraschauer

  committed
  7244462

  Copy full SHA for 7244462

• adds function to update latex table from D6.3 report

  lauraschauer

  committed
  910463e

  Copy full SHA for 910463e

• puts matteo's code into functions and writes results to files instead of console

  lauraschauer

  committed
  59c0c66

  Copy full SHA for 59c0c66

• puts matteo's code into functions and writes results to files instead of console

  lauraschauer

  committed
  4711219

  Copy full SHA for 4711219

• fixes precision of the rules function

  lauraschauer

  committed
  d3338c7

  Copy full SHA for d3338c7

• correctly display number of dispatched jobs

  lauraschauer

  committed
  f074472

  Copy full SHA for f074472

• moves execution time measurement for LLM from child to parent function

  Show description for d174f20
  lauraschauer

  committed
  d174f20

  Copy full SHA for d174f20

• adds function to stop all jobs on queue

  lauraschauer

  committed
  6cbb208

  Copy full SHA for 6cbb208

• removes cluttering console output from redis queue, adjusts folder path: one folder for execution with llm and one without

  lauraschauer

  committed
  eef559f

  Copy full SHA for eef559f

• moves functions to extract version intervals from cves to utils

  lauraschauer

  committed
  9402ba1

  Copy full SHA for 9402ba1

• writes llm stats analysis to file

  lauraschauer

  committed
  7727e01

  Copy full SHA for 7727e01

• corrects git cache path and adds docstring to do_clone

  lauraschauer

  committed
  c5d627a

  Copy full SHA for c5d627a

• adds progress bar to clone_repo_multiple

  lauraschauer

  committed
  1d81fc1

  Copy full SHA for 1d81fc1

• adds script to clone repos

  lauraschauer

  committed
  dc9deac

  Copy full SHA for dc9deac

• executes prospector container with host user so that created files (eg. gitcaches) can be modified on the host

  lauraschauer

  committed
  a9f961e

  Copy full SHA for a9f961e

• adds comments, docstrings and improves logging of repo cloning

  lauraschauer

  committed
  cfe7b7e

  Copy full SHA for cfe7b7e

• reformatting

  lauraschauer

  committed
  0beec15

  Copy full SHA for 0beec15

• creates own evaluation logger

  lauraschauer

  committed
  74e0df5

  Copy full SHA for 74e0df5

• restructuring to have better file hierarchy and avoid cyclic imports

  lauraschauer

  committed
  4ba005b

  Copy full SHA for 4ba005b

• adds script to trigger repository cloning for CVEs

  lauraschauer

  committed
  f727ae5

  Copy full SHA for f727ae5

• rewrites method to generate D6.3 summary execution table

  lauraschauer

  committed
  d36bbc4

  Copy full SHA for d36bbc4

• new function to analyse reports now yields (almost) correct results

  lauraschauer

  committed
  df890c5

  Copy full SHA for df890c5

• adds new cli flag to count which reports are missing

  lauraschauer

  committed
  2c64853

  Copy full SHA for 2c64853

• updates gitignore

  lauraschauer

  committed
  6dd5574

  Copy full SHA for 6dd5574

• changes config file to resemble the prospector config file, removes some unused code

  lauraschauer

  committed
  154a696

  Copy full SHA for 154a696

• added flag to remove jobs from queue

  lauraschauer

  committed
  1f8fa29

  Copy full SHA for 1f8fa29

• increased default timeout

  lauraschauer

  committed
  971f7b7

  Copy full SHA for 971f7b7

• adds script to compare pairs of prospector reports (eg. the reports from D6.3 and the currently generated ones)

  lauraschauer

  committed
  cb61e13

  Copy full SHA for cb61e13

• clean analysis code

  lauraschauer

  committed
  4d75870

  Copy full SHA for 4d75870

• set parameter so that already cloned repos are not cloned again

  lauraschauer

  committed
  7cb6e57

  Copy full SHA for 7cb6e57

• sets using LLMs to infer repo URLs to false in any case and refactors code to make it cleaner

  lauraschauer

  committed
  634b404

  Copy full SHA for 634b404

• adds two more folders to share between the container and host for the reports with mvi

  lauraschauer

  committed
  8578dfd

  Copy full SHA for 8578dfd

• removes unnecessary import

  lauraschauer

  committed
  312ee0c

  Copy full SHA for 312ee0c

• renames run_multiple to main and removes unused code from it

  lauraschauer

  committed
  2a69348

  Copy full SHA for 2a69348

• changes back to one reports path from config

  lauraschauer

  committed
  6ca5102

  Copy full SHA for 6ca5102

• moves cloning repositories to fill gitcache into evaluation folder

  lauraschauer

  committed
  2d9ea2c

  Copy full SHA for 2d9ea2c

• creates new script to analyse statistics

  lauraschauer

  committed
  8c8b01d

  Copy full SHA for 8c8b01d

• prints error to console instead of crashing at None, -1 returned from Prospector

  lauraschauer

  committed
  1e579e7

  Copy full SHA for 1e579e7

• starts function to create boxplot with statistics

  lauraschauer

  committed
  3452233

  Copy full SHA for 3452233

• configuration file for evaluation code

  lauraschauer

  committed
  41f7c98

  Copy full SHA for 41f7c98

• adds gitignore for evaluation folder

  lauraschauer

  committed
  fd99b6f

  Copy full SHA for fd99b6f

• adds script to extract lines with error reasons from log

  lauraschauer

  committed
  2dea986

  Copy full SHA for 2dea986

• updates readme and removes unused cl flags from main

  lauraschauer

  committed
  fee7a4a

  Copy full SHA for fee7a4a

• creates boxplot of overall execution time for all 4 categories

  lauraschauer

  committed
  3ad62e8

  Copy full SHA for 3ad62e8

• adds function to analyse which reports change category in different executions

  lauraschauer

  committed
  1a09525

  Copy full SHA for 1a09525

• [DOC] updates readme

  lauraschauer

  committed
  60ed59c

  Copy full SHA for 60ed59c

• [DOC] cleans up utils file

  lauraschauer

  committed
  62dcee1

  Copy full SHA for 62dcee1

• [ADD] adds function to plot commit classification times as boxplots

  lauraschauer

  committed
  2f2540e

  Copy full SHA for 2f2540e

• [ADD] function to check whether two ground truth datasets contain the same CVEs

  lauraschauer

  committed
  6deb2e5

  Copy full SHA for 6deb2e5

• [ADD] function to see correlation between time needed for execution and number of candidates

  lauraschauer

  committed
  7820c32

  Copy full SHA for 7820c32

• [DOC] adds logging statements for git commands and which period of tags will be retrieved

  lauraschauer

  committed
  fe67a7d

  Copy full SHA for fe67a7d

• [FIX] makes sure that only commits referenced in the advisory's references with a commit hash are treated as 'fixing commits', and not if the commit is eg. 'commit::master'

  lauraschauer

  committed
  ca24925

  Copy full SHA for ca24925

• [ADD] function to choose which set of strong rules should be used + function to analyse the category flows when there is no mutual exclusion between the high confidence categories

  lauraschauer

  committed
  ad94c05

  Copy full SHA for ad94c05

• [ADD] information about which CVEs are missing reports for both directories being compared

  lauraschauer

  committed
  22319bd

  Copy full SHA for 22319bd

• [DOC] restructures folders to have all reports not in prospector folder (as the docker container won't build because it's getting too big)

  lauraschauer

  committed
  49976cc

  Copy full SHA for 49976cc

• [FIX] fixes bug when extracting text from Jira Refs

  lauraschauer

  committed
  a9caf27

  Copy full SHA for a9caf27

• [FIX] fixes mistake in filtering out the 'commit::master' references

  lauraschauer

  committed
  27a61a3

  Copy full SHA for 27a61a3

• [DOC] updates readme

  lauraschauer

  committed
  06cce63

  Copy full SHA for 06cce63

• [DOC] removes unused code

  lauraschauer

  committed
  58ff072

  Copy full SHA for 58ff072

• [FIX] uses 20 instead of 2 workers

  lauraschauer

  committed
  f57080e

  Copy full SHA for f57080e

• [DOC] adds images for readme

  lauraschauer

  committed
  49301b3

  Copy full SHA for 49301b3

• [ADD] adds script to compare differently classified reports

  lauraschauer

  committed
  eea2a2e

  Copy full SHA for eea2a2e

• [ADD] options to select certain CVEs for analysis

  lauraschauer

  committed
  ff9591f

  Copy full SHA for ff9591f

• [FIX] mounts the code into the container instead of copying, so that changes on the host happen in the container as well

  lauraschauer

  committed
  aa76ef0

  Copy full SHA for aa76ef0

• [IMP] small code changes

  lauraschauer

  committed
  9afd5ee

  Copy full SHA for 9afd5ee

• [FIX] gives host permission to read and write to log files

  lauraschauer

  committed
  e2deaba

  Copy full SHA for e2deaba

• [FIX] new method to generate better flow analysis

  lauraschauer

  committed
  7d16871

  Copy full SHA for 7d16871

• [FIX] changes weight of commit classification rule to 16 instead of 32 (shows better results)

  lauraschauer

  committed
  7775f65

  Copy full SHA for 7775f65

• [IMP] refactors the code so that the batch of reports can be chosen in the config file

  lauraschauer

  committed
  158c0bb

  Copy full SHA for 158c0bb

• [ADD] adds comparison of keywords in reports to flow file

  lauraschauer

  committed
  e03ef04

  Copy full SHA for e03ef04

• [IMP] removes unused code

  lauraschauer

  committed
  36d1ac7

  Copy full SHA for 36d1ac7

• [ADD] function to produce checkmarks table as in D6.3

  lauraschauer

  committed
  45e449b

  Copy full SHA for 45e449b

• [FIX] makes sure that version interval is only supplied when 'version_interval' is true in config

  lauraschauer

  committed
  b3c1103

  Copy full SHA for b3c1103

• [ADD] function to create sankey diagram with flows between categories

  lauraschauer

  committed
  7f33fea

  Copy full SHA for 7f33fea

• [FIX] changes the config parameter use_comparison to batch, which can either be regular, old_code or old_reports

  lauraschauer

  committed
  08d1d58

  Copy full SHA for 08d1d58

• [FIX] changes to using clearer config variables for batch selection

  lauraschauer

  committed
  6383842

  Copy full SHA for 6383842

• [FIX] adjusts code so that with LLm support can run with 10 workers, and fixes bug where cc outcome would be taken from database regardless of the settings in config.yaml

  lauraschauer

  committed
  6c35088

  Copy full SHA for 6c35088

• [FIX] removes unused variables

  lauraschauer

  committed
  e0f16d8

  Copy full SHA for e0f16d8

• [FIX] adjusts titles of plots, uses different paths (alltogether unimportant)

  lauraschauer

  committed
  0ec2004

  Copy full SHA for 0ec2004

• [IMP] improves the execution time analysis plots by adding a violin plot and showing the boxplots on a logarithmic scale

  lauraschauer

  committed
  95f7111

  Copy full SHA for 95f7111

• [ADD] copies code for using the API to create jobs

  lauraschauer

  committed
  7c6bfd5

  Copy full SHA for 7c6bfd5

• [ADD] code to use the API to enqueue jobs

  lauraschauer

  committed
  64eeb01

  Copy full SHA for 64eeb01

• [FIX] temporary fix to not use database for cc when use_backend set to false

  lauraschauer

  committed
  36e9318

  Copy full SHA for 36e9318

• [DOC] cleans up unimportant files

  lauraschauer

  committed
  579a622

  Copy full SHA for 579a622

• [FIX] after rebasing on pipeline branch

  lauraschauer

  committed
  b5023d5

  Copy full SHA for b5023d5

• [IMP] changes main file to use registry pattern for selection of function to execute

  lauraschauer

  committed
  bfe4ffa

  Copy full SHA for bfe4ffa

• [IMP] adjust config.yaml file to structure changes from last commit

  lauraschauer

  committed
  5642c5f

  Copy full SHA for 5642c5f

• [IMP] adjusted all files and functions to the new registry pattern

  lauraschauer

  committed
  b56718c

  Copy full SHA for b56718c

• [IMP] cleans up code and removes all unused or unimportant files

  lauraschauer

  committed
  e0e4944

  Copy full SHA for e0e4944

Commits on Sep 2, 2024

• Merge branch 'main' into evaluation-scripts

  copernico

  authored
  09b92d9

  Copy full SHA for 09b92d9