org.owasp.dependencycheck.analyzer.exception.AnalysisException: Failed to request component-reports due to blank lines in requirements.txt #6731

francisATgwn · 2024-06-18T20:22:15Z

Describe the bug
Dependency Check fails with a 400 error when requesting something from the Sonatype OSS Index, leading to fewer CVEs found in the report and warnings in the job log like An error occurred while analyzing requirements.txt (Sonatype OSS Index Analyzer).

Version of dependency-check used
The problem occurs using version 9.2.0 of the CLI -- both docker and brew packaging

Log file
https://gist.github.com/francisATgwn/ece673ba589b75110a3aeecc9354708e

To Reproduce
Steps to reproduce the behavior:

Use dependency check CLI with this CLI invocation

$ /usr/share/dependency-check/bin/dependency-check.sh --dbDriverName org.postgresql.Driver --connectionString "$DEPENDENCY_CHECK_CONNECTION" --dbUser "$DEPENDENCY_CHECK_USER" --dbPassword "$DEPENDENCY_CHECK_PASSWORD" --nvdApiKey "$NIST_NVD_API_KEY" --project $CI_PROJECT_NAME --out . --scan . --enableExperimental --format JUNIT --junitFailOnCVSS 4 --format HTML --failOnCVSS 4 --nodeAuditSkipDevDependencies $( [[ -e dependency-check-suppression.xml ]] && echo '--suppression dependency-check-suppression.xml' || echo '' )

Expected behavior

CVEs from Sonatype OSS Index are included in the report
[WARN] An error occurred while analyzing '...' (Sonatype OSS Index Analyzer). does not appear in log
no HTTP 400 resulting in org.owasp.dependencycheck.analyzer.exception.AnalysisException: Failed to request component-reports appearing in the Analysis Exceptions section of the HTML report

Additional context
This does not happen when run with the 8.4.3 docker image from the same environment on the same project at the same time.

The 8.4.3 CLI invocation is similar to the 9.2.0 invocation:

$ /usr/share/dependency-check/bin/dependency-check.sh --project $CI_PROJECT_NAME --out . --scan . --enableExperimental --format JUNIT --junitFailOnCVSS 4 --format HTML --failOnCVSS 4 --nodeAuditSkipDevDependencies --data dependency-check-data $( [[ -e dependency-check-suppression.xml ]] && echo '--suppression dependency-check-suppression.xml' || echo '' )

The text was updated successfully, but these errors were encountered:

francisATgwn · 2024-06-18T20:28:00Z

dependency-check-8.4.3-report.html.txt
dependency-check-9.2.0-report.html.txt

ahrys-serve · 2024-06-21T14:40:39Z

Having the same problem but only for some projects.
problematic requirements file: requirements.txt

[WARN] An error occurred while analyzing '/Users/[CUT]/Projects/[CUT]/app/init.py' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/Users/[CUT]/Projects/[CUT]/app/domain/api/facilities/init.py' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/Users/[CUT]/Projects/[CUT]/app/domain/api/departments/init.py' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/Users/[CUT]/Projects/[CUT]/app/domain/api/contacts/init.py' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/Users/[CUT]/Projects/[CUT]/app/domain/api/api_keys/init.py' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/Users/[CUT]/Projects/[CUT]/app/domain/api/init.py' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/Users/[CUT]/Projects/[CUT]/app/domain/init.py' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/Users/[CUT]/Projects/[CUT]/app/cli/init.py' (Sonatype OSS Index Analyzer).
[INFO] Finished Sonatype OSS Index Analyzer (4 seconds)
[INFO] Finished Vulnerability Suppression Analyzer (0 seconds)
[INFO] Finished Known Exploited Vulnerability Analyzer (0 seconds)
[INFO] Finished Dependency Bundling Analyzer (0 seconds)
[INFO] Finished Unused Suppression Rule Analyzer (0 seconds)
[INFO] Analysis Complete (7 seconds)
[INFO] Writing JSON report to: /Users/[CUT]/Projects/[CUT]/./dependency-check-report.json
[INFO] Writing HTML report to: /Users/[CUT]/Projects/[CUT]/./dependency-check-report.html
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports

francisATgwn · 2024-06-24T13:31:50Z

I've confirmed that it is a regression between 9.1.0 and 9.2.0.

9.1.0 finds 11 vulnerabilities among 5 dependencies

[INFO] Analysis Started
[INFO] Finished File Name Analyzer (0 seconds)
[INFO] Finished Python Package Analyzer (0 seconds)
[INFO] Finished pip Analyzer (0 seconds)
[INFO] Finished Poetry Analyzer (0 seconds)
[INFO] Finished Dependency Merging Analyzer (0 seconds)
[INFO] Finished Hint Analyzer (0 seconds)
[INFO] Finished Version Filter Analyzer (0 seconds)
[INFO] Created CPE Index (0 seconds)
[WARN] Hosted Suppressions file is empty or missing - attempting to force the update
[WARN] Empty Hosted Suppression file after update, results may contain false positives already resolved by the DependencyCheck project due to failed download of the hosted suppression file
[INFO] Finished NPM CPE Analyzer (0 seconds)
[INFO] Created CPE Index (1 seconds)
[INFO] Finished CPE Analyzer (3 seconds)
[INFO] Finished False Positive Analyzer (0 seconds)
[INFO] Finished NVD CVE Analyzer (0 seconds)
[INFO] Finished Sonatype OSS Index Analyzer (0 seconds)
[INFO] Finished Vulnerability Suppression Analyzer (0 seconds)
[INFO] Finished Known Exploited Vulnerability Analyzer (0 seconds)
[INFO] Finished Dependency Bundling Analyzer (0 seconds)
[INFO] Finished Unused Suppression Rule Analyzer (0 seconds)
[INFO] Analysis Complete (5 seconds)
[INFO] Writing HTML report to: /project/./dependency-check-report.html

9.2.0 finds 2 vulnerabilities among 1 dependency:

[INFO] Analysis Started
[INFO] Finished File Name Analyzer (0 seconds)
[INFO] Finished Python Package Analyzer (0 seconds)
[INFO] Finished pip Analyzer (0 seconds)
[INFO] Finished Poetry Analyzer (0 seconds)
[INFO] Finished Dependency Merging Analyzer (0 seconds)
[INFO] Finished Hint Analyzer (0 seconds)
[INFO] Finished Version Filter Analyzer (0 seconds)
[INFO] Created CPE Index (0 seconds)
[WARN] Hosted Suppressions file is empty or missing - attempting to force the update
[WARN] Empty Hosted Suppression file after update, results may contain false positives already resolved by the DependencyCheck project due to failed download of the hosted suppression file
[INFO] Finished NPM CPE Analyzer (0 seconds)
[INFO] Created CPE Index (1 seconds)
[INFO] Finished CPE Analyzer (3 seconds)
[INFO] Finished False Positive Analyzer (0 seconds)
[INFO] Finished NVD CVE Analyzer (0 seconds)
[WARN] An error occurred while analyzing '/project/src/mobilepairing/__init__.py' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/project/requirements.txt' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/project/requirements.txt' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/project/requirements.txt' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/project/requirements.txt' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/project/requirements.txt' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/project/requirements.txt' (Sonatype OSS Index Analyzer).
[WARN] An error occurred while analyzing '/project/requirements.txt' (Sonatype OSS Index Analyzer).
[INFO] Finished Sonatype OSS Index Analyzer (1 seconds)
[INFO] Finished Vulnerability Suppression Analyzer (0 seconds)
[INFO] Finished Known Exploited Vulnerability Analyzer (0 seconds)
[INFO] Finished Dependency Bundling Analyzer (0 seconds)
[INFO] Finished Unused Suppression Rule Analyzer (0 seconds)
[INFO] Analysis Complete (6 seconds)
[INFO] Writing JSON report to: /project/./dependency-check-report.json
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports
[ERROR] Failed to request component-reports

jellisgwn · 2024-06-24T14:23:19Z

@jeremylong any advice of where to look for this regression? the diff between 9.1.0 ... 9.20 is most dependabot and a new unrelated(?) analyzer.

Neumsy · 2024-07-05T13:27:55Z

This issue is also happening with version 10.0.1. It appears to be an issue with connecting to Sonatype OSS. The issue is happening if the OSS Variables (user, password, url) are supplied or not. I saw two issues with the OSS connection.

/opt/dependency-check/bin/dependency-check.sh --out . -s . -f HTML -f JSON --enableExperimental --exclude '**/.sonar/**' --exclude '**/.cache/**' -l tmp.log

First error which spams the output for just about every node_module file:

[WARN] An error occurred while analyzing '<full path>/node_modules/karma/static/context.js' (Sonatype OSS Index Analyzer).

Second Error:

[ERROR] Failed to request component-reports

Log Message:

DEBUG - Error requesting component reports
java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "A%"
        at java.base/java.net.URLDecoder.decode(URLDecoder.java:237)
        at java.base/java.net.URLDecoder.decode(URLDecoder.java:147)
        at org.sonatype.goodies.packageurl.PercentEncoding.decode(PercentEncoding.java:78)
        at org.sonatype.goodies.packageurl.PackageUrlParser.parseName(PackageUrlParser.java:135)
        at org.sonatype.goodies.packageurl.PackageUrlParser.parse(PackageUrlParser.java:106)
        at org.sonatype.goodies.packageurl.PackageUrl.parse(PackageUrl.java:293)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.parsePackageUrl(OssIndexAnalyzer.java:203)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$null$1(OssIndexAnalyzer.java:223)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
        at java.base/java.util.TreeMap$KeySpliterator.forEachRemaining(TreeMap.java:3064)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$requestReports$3(OssIndexAnalyzer.java:225)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.requestReports(OssIndexAnalyzer.java:221)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency(OssIndexAnalyzer.java:136)
        at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131)
        at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88)
        at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
2024-07-05 09:18:40,386 org.owasp.dependencycheck.AnalysisTask:90

Java Version:
openjdk 17.0.11 2024-04-16 LTS
OpenJDK Runtime Environment (Red_Hat-17.0.11.0.9-3) (build 17.0.11+9-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.11.0.9-3) (build 17.0.11+9-LTS, mixed mode, sharing)

andriyzhurba · 2024-07-16T14:38:30Z

I had the same issue too with the requirements.txt. The reason was blank lines in this file. Try to remove them.
I use the latest version 10.0.3. with the nvdApiKey.

Neumsy · 2024-07-16T19:18:51Z

The requirements.txt does not have any extra spaces in it, we generate it at the time with pip freeze > requirements.txt. Our build method is to use setup.py to contain all of our dependencies. We use that to install pacakges during the build into a venv and source the venv. The issue is being thrown from the packages under .venv and in node_modules.

This may be a documentation update request, but should node_modules and a virtual environment be excluded from the dependency check run?

andriyzhurba · 2024-07-18T12:41:22Z

It means you have a different error cause than Francis. "An error occurred while analyzing requirements.txt (Sonatype OSS Index Analyzer)."

I could reproduce it very easily. Adding a new empty line (in between, not at the end of the requirements.txt) causes the above mentioned error.

francisATgwn · 2024-07-18T12:55:05Z

If that truly is the cause of my error, which I haven't verified yet, then this is a bug report. My requirements.txt is generated by pip-compile and tools that read it should be resilient to the format it writes.

francisATgwn · 2024-07-22T15:18:42Z

I have upgraded to 10.0.3 and confirmed the error still occurs for me. I have also confirmed @andriyzhurba comment that removing any blank lines in requirements.txt resolves the issue.

I believe that the ODC code which reads requirements.txt should be resilient to the presence of blank lines. In my case, these blank lines are generated by pip-compile, a widely used tool in the python community.

jeremylong · 2024-07-23T10:02:03Z

Can a sample requirements.txt be provided that contains spaces?

andriyzhurba · 2024-07-23T10:42:34Z

@jeremylong It's pretty easy. It's not spaces, but empty lines.

requirements.txt - this file will work

requests==2.28.2
boto3==1.26.105
aws_lambda_powertools==2.26.0
aws_xray_sdk==2.12.1
langdetect==1.0.9
openai==1.13.3
orjson==3.9.15
numpy==1.26.4
types-orjson==3.6.2
openai==1.13.3

requirements.txt - this one will run into error:

requests==2.28.2
boto3==1.26.105

aws_xray_sdk==2.12.1
langdetect==1.0.9

orjson==3.9.15
numpy==1.26.4
types-orjson==3.6.2
openai==1.13.3

resovles #6731

francisATgwn · 2024-07-23T15:56:43Z

Here's an example generated by pip-compile. Just removing the blank line was sufficient to avoid the error, so I don't think the other non-package lines cause a problem but including it for completeness:

#
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
#    pip-compile
#
--index-url https://REDACTED

certifi==2024.6.2
    # via sentry-sdk
sentry-sdk==2.7.1
    # via sentry-python-example (pyproject.toml)
urllib3==2.2.2
    # via sentry-sdk

jeremylong · 2024-07-25T10:40:07Z

fix will be included in the next release.

francisATgwn · 2024-08-26T15:05:46Z

@jeremylong I'd like to make use of this fix. When can I expect the next release?

francisATgwn added the bug label Jun 18, 2024

jeremylong added a commit that referenced this issue Jul 23, 2024

fix: skip blank lines in requirements.txt

bf4fd08

resovles #6731

jeremylong mentioned this issue Jul 23, 2024

fix: skip blank lines in requirements.txt #6867

Merged

jeremylong linked a pull request Jul 25, 2024 that will close this issue

fix: skip blank lines in requirements.txt #6867

Merged

jeremylong closed this as completed Jul 25, 2024

seansmith39 mentioned this issue Aug 14, 2024

Add OWASP Secrets seansmith2600/H6060-Python-UrlLib3#3

Merged

github-actions bot locked as resolved and limited conversation to collaborators Dec 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

org.owasp.dependencycheck.analyzer.exception.AnalysisException: Failed to request component-reports due to blank lines in requirements.txt #6731

org.owasp.dependencycheck.analyzer.exception.AnalysisException: Failed to request component-reports due to blank lines in requirements.txt #6731

francisATgwn commented Jun 18, 2024 •

edited

Loading

francisATgwn commented Jun 18, 2024

ahrys-serve commented Jun 21, 2024

francisATgwn commented Jun 24, 2024

jellisgwn commented Jun 24, 2024

Neumsy commented Jul 5, 2024

andriyzhurba commented Jul 16, 2024 •

edited

Loading

Neumsy commented Jul 16, 2024 •

edited

Loading

andriyzhurba commented Jul 18, 2024

francisATgwn commented Jul 18, 2024

francisATgwn commented Jul 22, 2024

jeremylong commented Jul 23, 2024

andriyzhurba commented Jul 23, 2024 •

edited by jeremylong

Loading

francisATgwn commented Jul 23, 2024 •

edited

Loading

jeremylong commented Jul 25, 2024

francisATgwn commented Aug 26, 2024

org.owasp.dependencycheck.analyzer.exception.AnalysisException: Failed to request component-reports due to blank lines in requirements.txt #6731

org.owasp.dependencycheck.analyzer.exception.AnalysisException: Failed to request component-reports due to blank lines in requirements.txt #6731

Comments

francisATgwn commented Jun 18, 2024 • edited Loading

francisATgwn commented Jun 18, 2024

ahrys-serve commented Jun 21, 2024

francisATgwn commented Jun 24, 2024

jellisgwn commented Jun 24, 2024

Neumsy commented Jul 5, 2024

andriyzhurba commented Jul 16, 2024 • edited Loading

Neumsy commented Jul 16, 2024 • edited Loading

andriyzhurba commented Jul 18, 2024

francisATgwn commented Jul 18, 2024

francisATgwn commented Jul 22, 2024

jeremylong commented Jul 23, 2024

andriyzhurba commented Jul 23, 2024 • edited by jeremylong Loading

requirements.txt - this file will work

requirements.txt - this one will run into error:

francisATgwn commented Jul 23, 2024 • edited Loading

jeremylong commented Jul 25, 2024

francisATgwn commented Aug 26, 2024

francisATgwn commented Jun 18, 2024 •

edited

Loading

andriyzhurba commented Jul 16, 2024 •

edited

Loading

Neumsy commented Jul 16, 2024 •

edited

Loading

andriyzhurba commented Jul 23, 2024 •

edited by jeremylong

Loading

francisATgwn commented Jul 23, 2024 •

edited

Loading