From 535aaf91414dfb5878dabe5299831956e978632f Mon Sep 17 00:00:00 2001 From: Sean Corfield Date: Mon, 19 Aug 2024 16:13:24 -0700 Subject: [PATCH 1/2] fixes #104 by supporting properties via environment variables (#108) * fixes #104 by adding support for properties as env vars Signed-off-by: Sean Corfield --------- Signed-off-by: Sean Corfield --- .gitignore | 2 +- README.md | 69 +++++++++++++++++-- .../controller/dependency_check/scanner.clj | 43 +++++++++--- 3 files changed, 98 insertions(+), 16 deletions(-) diff --git a/.gitignore b/.gitignore index 859a4b4..f82852e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,7 @@ *.class *.iml *.jar -/.calva/output-window/ +/.calva/repl.calva-repl /.clj-kondo/.cache/ /.cpcache/ /.idea/ diff --git a/README.md b/README.md index 79b24cf..d58787a 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ clojure -M:clj-watson scan -p deps.edn The first time it runs, it will download the entire vulnerability database, which can take several minutes. Subsequent runs will be much faster. -> [!NOTE] +> [!NOTE] > The database is stored in the `/tmp/db/` folder (on macOS/Linux) - in case you ever need to delete that folder, if it looks like the database is corrupted. `clj-watson` can also be installed as a Clojure CLI tool: @@ -105,7 +105,8 @@ It is easy to [request an API key](https://github.com/jeremylong/DependencyCheck You can specify you key via: 1. The `nvd.api.key` Java system property on the command line -2. Or, an `nvd.api.key` entry in your `clj-watson.properties` file +2. Or, the `CLJ_WATSON_NVD_API_KEY` environment variable +3. Or, an `nvd.api.key` entry in your `clj-watson.properties` file > [!CAUTION] > Keeping your nvd api key secret is your responsibility. @@ -129,7 +130,39 @@ clojure -J-Dnvd.api.key= -Tclj-watson scan :p deps.edn Replace `` with your actual api key. > [!CAUTION] -> You could specify this system property under `:jvm-opts` in your `deps.edn` under your `:clj-watson` alias, but be careful not to commit it to version control. +> You could specify this system property under `:jvm-opts` in your `deps.edn` under your `:clj-watson` alias, but be careful not to commit it to version control. + +##### Via Environment Variable + +Example usage: + +```shell +CLJ_WATSON_NVD_API_KEY= clojure -M:clj-watson scan -p deps.edn +``` + +Or: + +```shell +export CLJ_WATSON_NVD_API_KEY= + +clojure -M:clj-watson scan -p deps.edn +``` + +Or: + +```shell +CLJ_WATSON_NVD_API_KEY= clojure -Tclj-watson scan :p deps.edn +``` + +Or: + +```shell +export CLJ_WATSON_NVD_API_KEY= + +clojure -Tclj-watson scan :p deps.edn +``` + +Replace `` with your actual api key. ##### Via the `clj-watson.properties` File @@ -154,7 +187,7 @@ Or: clojure -Tclj-watson scan :p deps.edn :clj-watson-properties ./clj-watson.properties ``` -> [!CAUTION] +> [!CAUTION] > Be careful not to commit your key to version control. ### GitHub Advisory Database [experimental] @@ -233,7 +266,7 @@ the `--suggest-fix` or `-s` option when running `clj-watson`. # Installation > [!IMPORTANT] -> You'll need to [setup your NVD API key](#nist-nvd-api). +> You'll need to [setup your NVD API key](#nist-nvd-api). `clj-watson` can be installed as a Clojure CLI tool, as shown above. While this is the easiest way to install the latest version and keep it up-to-date @@ -309,6 +342,8 @@ OPTIONS valid when database-strategy is dependency-check: See docs for configuration. [false] ``` +## Properties + By default, when using the DEPENDENCY-CHECK strategy, `clj-watson` will load its own `dependency-check.properties` file, and then look for a `clj-watson.properties` file on the classpath and load that if found, for @@ -324,6 +359,27 @@ that file and apply those properties to the dependency-check scan. This is in addition to the properties loaded from the `dependency-check.properties` or the `-d` file. This can be useful to override just a few properties. +In addition, relevant properties provided as Java system properties are +read by the underlying DependencyCheck scan, and take precedence over the +properties provided in these files. See the `-Dnvd.api.key=` example above. + +## Environment Variables + +`clj-watson` also supports environment variables that start with `CLJ_WATSON_`. +These are used to set properties that are not provided on the command line. +The `CLJ_WATSON_` prefix is removed, and the rest of the name is converted to +a lowercase property name with `_` replaced by `.` (e.g., `CLJ_WATSON_NVD_API_KEY`). +To specify a property name that contains an underscore, use two underscores +in the environment variable name, e.g., `CLJ_WATSON_DATA_FILE__NAME` to +set the `data.file_name` property. + +Properties set via environment variables take precedence over those set in +the properties files described above, but not over Java system properties +set on the command-line. + +Environment variables are often the most straightforward and most secure +way to provide sensitive information like API keys in various CI systems. + # Execution The minimum needed to run `clj-watson` is to provide the path to a `deps.edn` @@ -331,7 +387,7 @@ file, but it is recommended that you also provide the `-s` option so `clj-watson` will try to suggest remediations for any vulnerabilities found. > [!IMPORTANT] -> You'll need to first [setup your NVD API key](#nist-nvd-api). +> You'll need to first [setup your NVD API key](#nist-nvd-api). ```bash clojure -M:clj-watson -p deps.edn @@ -383,6 +439,7 @@ It writes settings and vulnerability findings to `stdout`. # Who uses it - [180 Seguros](https://180s.com.br) +- [org.clojure/tools.deps](https://github.com/clojure/tools.deps) - [World Singles Networks](https://worldsinglesnetworks.com/) # Development diff --git a/src/clj_watson/controller/dependency_check/scanner.clj b/src/clj_watson/controller/dependency_check/scanner.clj index 624b8f8..096fba1 100644 --- a/src/clj_watson/controller/dependency_check/scanner.clj +++ b/src/clj_watson/controller/dependency_check/scanner.clj @@ -21,16 +21,40 @@ ;; ensure (fake) API key is redacted: (sanitize-property "nvd.api.key=72a48765-90ab-5678-abcd-1234abcd5489")) +(defn ^:private env-var->property [env-var] + (-> env-var + (string/replace #"^CLJ_WATSON_" "") ; strip prefix + (string/lower-case) ; lowercase + (string/replace "_" ".") ; _ -> . + (string/replace ".." "_"))) ; allow __ -> .. -> _ + +(comment + (env-var->property "CLJ_WATSON_NVD_API_KEY") + (env-var->property "CLJ_WATSON_DATA_FILE__NAME")) + +(defn ^:private set-watson-env-vars-as-properties [] + (run! (fn [[env-var value]] + (when (string/starts-with? env-var "CLJ_WATSON_") + (let [property (env-var->property env-var) + p-value (System/getProperty property)] + (if p-value + (println (str "Ignoring " env-var " as " property " is already set.")) + (do + (println (str "Setting " property " from " env-var ".")) + (System/setProperty property value)))))) + (System/getenv)) + (println)) + (defn ^:private create-settings [^String properties-file-path ^String additional-properties-file-path] - (let [settings (Settings.)] - (let [props - (if properties-file-path - (->> properties-file-path File.) - (->> "dependency-check.properties" io/resource))] - (println (str "\nRead " (count (line-seq (io/reader props))) " dependency-check properties.")) - (if properties-file-path - (->> props (.mergeProperties settings)) - (->> props slurp .getBytes ByteArrayInputStream. (.mergeProperties settings)))) + (let [settings (Settings.) + props + (if properties-file-path + (->> properties-file-path File.) + (->> "dependency-check.properties" io/resource))] + (println (str "\nRead " (count (line-seq (io/reader props))) " dependency-check properties.")) + (if properties-file-path + (->> props (.mergeProperties settings)) + (->> props slurp .getBytes ByteArrayInputStream. (.mergeProperties settings))) (if-let [add-props (if additional-properties-file-path (->> additional-properties-file-path File.) @@ -50,6 +74,7 @@ (->> add-props (.mergeProperties settings)) (some->> add-props slurp .getBytes ByteArrayInputStream. (.mergeProperties settings)))) (println "No additional properties found.\n")) + (set-watson-env-vars-as-properties) settings)) (defn ^:private validate-settings From fbdad5b2cadc11ca399cbc38663ffb53aeb65617 Mon Sep 17 00:00:00 2001 From: Sean Corfield Date: Mon, 19 Aug 2024 16:33:33 -0700 Subject: [PATCH 2/2] address #103 Streamline dependency(-)check.properties overrides (#106) * address #103 Streamline dependency(-)check.properties overrides * note #103 in the changelog * match nvd-clojure per @lread Signed-off-by: Sean Corfield --------- Signed-off-by: Sean Corfield --- CHANGELOG.md | 9 +- README.md | 8 +- resources/dependency-check.properties | 138 ++++++-------------------- 3 files changed, 38 insertions(+), 117 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c55a82f..3461c80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # CHANGELOG -* Unreleased +* Unreleased (6.0.0): * Fix: show score and severity in dependency-check findings [#58](https://github.com/clj-holmes/clj-watson/issues/58) * Bump deps [#75](https://github.com/clj-holmes/clj-watson/issues/75) * Improve command line experience [#77](https://github.com/clj-holmes/clj-watson/issues/77) @@ -8,6 +8,9 @@ * Explicitly close the dependency-check engine when we are done with it [#86](https://github.com/clj-holmes/clj-watson/issues/86) * Respect dependency-check `odc.autoupdate` property [#88](https://github.com/clj-holmes/clj-watson/issues/88) * Replace deprecated clj-time dep with JDK8 java.time interop [#83](https://github.com/clj-holmes/clj-watson/issues/83) + * Allow properties to be specified via environment variables [#104](https://github.com/clj-holmes/clj-watson/issues/104) to make it easier to use `clj-watson` in CI/CD pipelines. + * Streamline `dependency-check.properties` file [#103](https://github.com/clj-holmes/clj-watson/issues/103) so that it only includes properties which need to be different from the defaults in the core DependencyCheck configuration. + * This changes the default location of the local database used for analysis from `/tmp/db` to a directory within your local Maven cache (DependencyCheck's default location), which makes `clj-watson` more CI-friendly since `~/.m2` is typically cached in CI. **The first time you run `clj-watson` 6.0.0, it will download the entire NIST NVD database!** * Improve feedback during scan * Stop suppressing all logging [#68](https://github.com/clj-holmes/clj-watson/issues/68) * Suppress noisy INFO level logging from Apache Commons JCS [#69](https://github.com/clj-holmes/clj-watson/issues/69) @@ -15,10 +18,10 @@ * v5.1.3 5812615 -- 2024-07-31 * Address [#60](https://github.com/clj-holmes/clj-watson/issues/60) by updating `org.owasp/dependency-check-core` to 10.0.3. - + * v5.1.2 ae20e1e -- 2024-03-20 * GitHub Advisory: fix matching CVE for allowlist via PR [#59](https://github.com/clj-holmes/clj-watson/pull/59) [@markomafs](https://github.com/markomafs). - + * v5.1.1 ad5fe07 -- 2024-01-15 * Address [#49](https://github.com/clj-holmes/clj-watson/issues/49) by improving the `-T` invocation to support short names, symbols for strings, and all the defaults. * Address [#48](https://github.com/clj-holmes/clj-watson/issues/48) by updating all of the project dependencies, including DependencyCheck to 9.0.8. diff --git a/README.md b/README.md index d58787a..3a13144 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ The first time it runs, it will download the entire vulnerability database, whic can take several minutes. Subsequent runs will be much faster. > [!NOTE] -> The database is stored in the `/tmp/db/` folder (on macOS/Linux) - in case you ever need to delete that folder, if it looks like the database is corrupted. +> The database is stored in your local Maven cache (on macOS/Linux, that's under `~/.m2/repository/org/owasp/dependency-check-utils/10.0.3/data/9.0/` currently) - in case you ever need to delete that folder, if it looks like the database is corrupted. `clj-watson` can also be installed as a Clojure CLI tool: @@ -82,7 +82,7 @@ can be shortened to `:p` (matching the `-p` short form of `--deps-edn-path`). [DependencyCheck](https://github.com/jeremylong/DependencyCheck) is the most widely used method among the Clojure/Java SCA tools. It: -1. Downloads a database of known vulnerabilities from [NIST NVD](https://nvd.nist.gov/), storing it locally under your `/tmp/db/` folder +1. Downloads a database of known vulnerabilities from [NIST NVD](https://nvd.nist.gov/), storing it locally (inside your local Maven cache, under `~/.m2/repository/org/owasp/dependency-check-utils/10.0.3/data/9.0/` currently). 3. Scans JARs from dependencies specified in your `deps.edn` 4. Composes a [Common Platform Enumeration (CPE)](https://nvd.nist.gov/products/cpe) based on your dependencies 5. Returns any matching vulnerabilities @@ -350,9 +350,7 @@ its own `dependency-check.properties` file, and then look for a additional properties to apply to the DependencyCheck scan. If you provide `-d` (or `--dependency-check-properties`) then `clj-watson` will -load that file instead of its own `dependency-check.properties` file so it -needs to be a complete properties file, not just the properties you want to -override. +load that file instead of its own `dependency-check.properties` file. If you provide `-w` (or `--clj-watson-properties`) then `clj-watson` will load that file and apply those properties to the dependency-check scan. This is diff --git a/resources/dependency-check.properties b/resources/dependency-check.properties index f817b20..06096f3 100644 --- a/resources/dependency-check.properties +++ b/resources/dependency-check.properties @@ -1,119 +1,39 @@ # replaced ${pom.*} with actual values: odc.application.name=clj-watson odc.application.version=5.0.0 -odc.autoupdate=true -odc.analysis.timeout=30 -odc.settings.mask=.*password.*,.*token.* -odc.reports.pretty.print=false -odc.ecosystem.maxquerylimit.native=1000 -odc.ecosystem.maxquerylimit.default=100 -engine.version.url=https://jeremylong.github.io/DependencyCheck/current.txt - -database.batchinsert.enabled=true -database.batchinsert.maxsize=3000 - -data.version=5.2 -data.directory=/tmp/db/ -data.connection_string=jdbc:h2:file:%s;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000; -data.writelock.shutdownhook=org.owasp.dependencycheck.utils.WriteLockCleanupHook -data.driver_name=org.h2.Driver - -proxy.disableSchemas=true - -# nvd.api.key must be provided by the user: -#nvd.api.key=... -nvd.api.check.validforhours=12 -nvd.api.datafeed.startyear=2002 -nvd.api.datafeed.validfordays=7 -nvd.api.delay=2000 -nvd.api.max.retry.count=10 -# unused nvd.api.* keys: -#nvd.api.datafeed.url= -#nvd.api.datafeed.user= -#nvd.api.datafeed.password= - -cve.url.modified.validfordays=7 -cve.check.validforhours=12 -cve.startyear=2002 -cve.url.original=https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.json.gz -cve.url.modified=https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.json.gz -cve.url.base=https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-%d.json.gz -cve.cpe.startswith.filter=cpe:2.3:a: - -nvd.newyear.grace.period=10 - -max.download.threads=8 - -cpe.validfordays=30 - -cpe.url=https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz - -central.content.url=https://search.maven.org/remotecontent?filepath= - -archive.scan.depth=10 - -downloader.quick.query.timestamp=true -downloader.tls.protocols=TLSv1.1,TLSv1.2,TLSv1.3 - -junit.fail.on.cvss=0 - -analyzer.nexus.url=https://repository.sonatype.org/service/local/ -analyzer.central.url=https://search.maven.org/solrsearch/select -analyzer.central.query=%s?q=1:%s&wt=xml -analyzer.central.retry.count=7 -analyzer.central.parallel.analysis=true -analyzer.central.use.cache=true -analyzer.ossindex.enabled=true -analyzer.ossindex.url=https://ossindex.sonatype.org -analyzer.ossindex.use.cache=true -analyzer.node.audit.url=https://registry.npmjs.org/-/npm/v1/security/audits -analyzer.node.audit.use.cache=false -analyzer.experimental.enabled=false -analyzer.retired.enabled=false -analyzer.jar.enabled=true -analyzer.archive.enabled=false -analyzer.node.package.enabled=false -analyzer.node.audit.enabled=false -analyzer.yarn.audit.enabled=true +# turn off all the analyzers that don't matter for JVM projects: +analyzer.artifactory.enabled=false +analyzer.assembly.enabled=false +analyzer.autoconf.enabled=false +analyzer.bundle.audit.enabled=false +analyzer.carthage.enabled=false +analyzer.cmake.enabled=false +analyzer.cocoapods.enabled=false +analyzer.composer.lock.enabled=false +analyzer.cpanfile.enabled=false +analyzer.dart.enabled=false analyzer.golang.dep.enabled=false -analyzer.retirejs.enabled=false -analyzer.retirejs.repo.validforhours=24 -analyzer.retirejs.repo.js.url=https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json -analyzer.retirejs.filternonvulnerable=false analyzer.golang.mod.enabled=false -analyzer.mix.audit.enabled=true -analyzer.composer.lock.enabled=false +analyzer.mix.audit.enabled=false +analyzer.msbuildproject.enabled=false +analyzer.nexus.proxy=false +analyzer.node.audit.enabled=false +analyzer.node.audit.use.cache=false +analyzer.node.package.enabled=false +analyzer.npm.cpe.enabled=false +analyzer.nugetconf.enabled=false +analyzer.nuspec.enabled=false +analyzer.openssl.enabled=false +analyzer.pip.enabled=false +analyzer.pipfile.enabled=false +analyzer.pnpm.audit.enabled=false +analyzer.poetry.enabled=false analyzer.python.distribution.enabled=false analyzer.python.package.enabled=false +analyzer.retirejs.enabled=false +analyzer.retirejs.filternonvulnerable=false analyzer.ruby.gemspec.enabled=false -analyzer.bundle.audit.enabled=false -analyzer.autoconf.enabled=false -analyzer.pip.enabled=false -analyzer.pipfile.enabled=false -analyzer.cmake.enabled=false -analyzer.assembly.enabled=false -analyzer.nuspec.enabled=false -analyzer.nugetconf.enabled=false -analyzer.msbuildproject.enabled=false -analyzer.openssl.enabled=true -analyzer.central.enabled=true -analyzer.nexus.enabled=false -analyzer.cocoapods.enabled=true analyzer.swift.package.manager.enabled=false -analyzer.nexus.proxy=false -analyzer.cpe.enabled=true -analyzer.npm.cpe.enabled=true -analyzer.cpesuppression.enabled=true -analyzer.dependencybundling.enabled=true -analyzer.dependencymerging.enabled=true -analyzer.falsepositive.enabled=true -analyzer.filename.enabled=true -analyzer.pe.enabled=true -analyzer.hint.enabled=true -analyzer.nvdcve.enabled=true -analyzer.vulnerabilitysuppression.enabled=true -updater.nvdcve.enabled=true -updater.versioncheck.enabled=true -analyzer.versionfilter.enabled=true -analyzer.artifactory.enabled=false +analyzer.swift.package.resolved.enabled=false +analyzer.yarn.audit.enabled=false