Authentication failure after upgrade to 11.0.0 with Maven

[mdvalk-quintor]

Describe the bug
Since version 11.0.0 Apache Httpclient is used for web requests instead of plain java.
Configuration properties suppressionFileUser and suppressionFilePassword are no longer respected. (and the alternative configuration using suppressionFileServerId is neither respected).
By intercepting the http request the owasp dependency-check maven plugin tries to make, we can verify that it skips authentication, even when authentication is provided by the user!

Version of dependency-check used
The problem occurs using version 11.0.0 of the maven plugin

Log file

org.owasp.dependencycheck.exception.ExceptionCollection: One or more exceptions occurred during analysis:
        InitializationException: Warn initializing the suppression analyzer: Failed to load http://localhost:8087/suppressions.xml, caused by Unable to fetch the configured suppression file. 
                caused by SuppressionParseException: Failed to load http://localhost:8087/suppressions.xml, caused by Unable to fetch the configured suppression file.

To Reproduce
Steps to reproduce the behavior:

1. Configure maven dependency-check plugin with the following:

<configuration>
    <suppressionFiles>
        <suppressionFile>http://localhost:8087/suppressions.xml</suppressionFile>
    </suppressionFiles>
    <suppressionFileUser>my-username</suppressionFileUser>
    <suppressionFilePassword>my-secret-password</suppressionFilePassword>
</configuration>

2. On linux run netcat listening on port 8087 to intercept traffic and see what authentication maven dependency-check plugin comes up with: nc -l 8087
3. Run mvn dependency-check:check
4. Check your netcat interceptor and you will see something along the lines of:

GET /suppressions.xml HTTP/1.1
Accept-Encoding: gzip, x-gzip, deflate
Host: localhost:8087
Connection: keep-alive
User-Agent: Apache-HttpClient/5.4 (Java/21.0.2)
Upgrade: TLS/1.2
Connection: Upgrade

5. We can see that the Authorization request header is missing!!

Expected behavior
For step 4 in the reproduction steps above, we expect to see the Authorization header present on the request... e.g.:

GET /suppressions.xml HTTP/1.1
Authorization: Basic ........
Accept-Encoding: gzip, x-gzip, deflate
Host: localhost:8087
Connection: keep-alive
User-Agent: Apache-HttpClient/5.4 (Java/21.0.2)
Upgrade: TLS/1.2
Connection: Upgrade

Other comments

• If you run the reproduction steps again, but this time switch the plugin version from 11.0.0 to version 10.0.4 and you will see the Authorization header is present in the requests!

[aikebah]

4. is expected on first request, as it does not do pre-emptive authentication. It should properly authenticate when challenged for it by the server (HTTP 401 authentication required response)

HttpClient is handed over the credentials for the hostedSuppressionUrl.

Can you check whether your server responds with the authentication challenge?

[mdvalk-quintor]

4. is expected on first request, as it does not do pre-emptive authentication. It should properly authenticate when challenged for it by the server (HTTP 401 authentication required response)

HttpClient is handed over the credentials for the hostedSuppressionUrl.

Can you check whether your server responds with the authentication challenge?

Thank you for the clarification.
The server does respond with an authentication challenge, but this challenge contains a <!DOCTYPE> declaration which is rejected / disallowed by the SAXParser:

org.xml.sax.SAXException: Line=3, Column=10: DOCTYPE is disallowed when the feature "http://apache.org/xml/features/disallow-doctype-decl" set to true.
    at org.owasp.dependencycheck.xml.suppression.SuppressionErrorHandler.fatalError (SuppressionErrorHandler.java:71)
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError (ErrorHandlerWrapper.java:181)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError (XMLErrorReporter.java:400)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError (XMLErrorReporter.java:327)
    at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError (XMLScanner.java:1465)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next (XMLDocumentScannerImpl.java:898)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next (XMLDocumentScannerImpl.java:605)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next (XMLNSDocumentScannerImpl.java:114)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument (XMLDocumentFragmentScannerImpl.java:542)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse (XML11Configuration.java:889)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse (XML11Configuration.java:825)

See the following curl request as example of the scenario rejected by SAXParser:
curl -Lv https://dev.azure.com/xxx/xxx/_apis/git/repositories/xxxx/items?path=/suppressions.xml

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html lang="en-US">
<head><title>
	
            Azure DevOps Services | Sign In

[aikebah]

Check the accompanying HTTP code, because given your stacktrace it looks more like that page gets served with an HTTP responsecode of the 2xx or 3xx range

[mdvalk-quintor]

Check the accompanying HTTP code, because given your stacktrace it looks more like that page gets served with an HTTP responsecode of the 2xx or 3xx range

Correct these are the response details:

< HTTP/2 302 
< www-authenticate: Basic realm="https://tfs.app.visualstudio.com/"
< www-authenticate: Bearer

When sending a request to Microsoft Azure DevOps without providing authentication credentials, it redirects to the login page using status code 302 while at the same time requesting authentication.

If you however do provide invalid credentials, the response is 401 in combination with an authentication challenge:

< HTTP/2 401 
< www-authenticate: Basic realm="https://tfs.app.visualstudio.com/"

Does this simply mean that this plugin no longer supports authentication to Microsoft Azure DevOps services?

[martijndebruijn]

I'm facing the same problems. Reverted the plugin to 10.0.4 and the DOCTYPE errors are gone.

Trying to download the suppression file from a Azure Devops url.

Any workaround available?

[mdvalk-quintor]

bump!

anybody willing to answer my question above? or @martijndebruijn 's question?

[jeremylong]

@aikebah any clue on this one?

[aikebah]

Need to finish up my local pending changes to make apache httpclient use pre-emptive auth, but basically the root cause appears to be the server misbehaving in the context of HTTP protocol (not giving an http 401, but redirecting with 302 to webbased authentication page so that httpclient's credentials lookup never kicks in)

Hope to finish that up later this week and put up a branch that hopefully someone can build a private snapshot from to validate my assumption that pre-emptive auth will solve the issue.

[jeremylong]

Thanks - I wanted to make sure someone had this, even if it is going to take a bit of time to solve.

[aikebah]

@mdvalk-quintor Would you be able to test a privately built snapshot from the fix/httpclient5 branch in your Azure Devops environment in order to verify that indeed adding pre-emptive auth solves the issues your CI runs into?

[ftiercelin]

@aikebah
I gave it a try:

1. pull fix/httpclient5
2. mvn install
3. create a server that sends a auth challenge with a HTTP-302 (instead of proper 401)
4. create a demo project with:

    <suppressionFiles>
        <suppressionFile>http://xxxxxxxxx:8080/basic302/suppressions.xml</suppressionFile>
    </suppressionFiles>
    <suppressionFileUser>my-username</suppressionFileUser>
    <suppressionFilePassword>my-secret-password</suppressionFilePassword>

5.mvn dependency-check:check

dependency-chek failed: InitializationException: Warn initializing the suppression analyzer: Failed to load http://xxxxxxxxx:8080/basic302/suppressions.xml, caused by Unable to fetch the configured suppression file.

Suppression server logs:

< No Authorization header provided for Basic 302
  > HTTP-302 FOUND
  > Content-Type: [application/xml]
  > WWW-Authenticate: [Basic realm="hosted suppressions"]
< No Authorization header provided for Basic 302
  > HTTP-302 FOUND
  > Content-Type: [application/xml]
  > WWW-Authenticate: [Basic realm="hosted suppressions"]

➫ it seems the client reacts to the challenge but still without the credentials

same project, with a server that returns 401 for the challenge, works:

< No Authorization header provided for Basic
  > HTTP-401 UNAUTHORIZED
  > Content-Type: [application/xml]
  > WWW-Authenticate: [Basic realm="hosted suppressions"]
< Authorization: Basic bXktdXNlcm5hbWU6bXktc2VjcmV0LXBhc3N3b3Jk
  > HTTP-200 OK
  > Content-Type: [application/xml]

if you want I can propose a PR to fix this, let me know

[aikebah]

@ftiercelin HostedSuppression credentials are currently only supported via the properties file, so it's logical that your attempt failed. The suppressionFileUser/Password are used for configuring the user suppression files (during suppression analyzer run) download's authentication.

[mathze]

@aikebah sorry but I'm confused by your comment #7108 (comment).
What do you mean by "hostedSuppression credentials are currently only supported via properties file"? From my understanding, downloading a file from a web-source, like azure or a git repo, through the suppressionFile property is a hosted suppression, isn't it? At least it worked with 10.0.4 in the gradle-plugin by providing suppressionFileUser/Password.
Does it mean this is no longer supported?