Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/remove unirest #1

Open
wants to merge 47 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
c7a7074
unirest removed from repo meta analyzer files
mehab Jan 17, 2023
39f31f3
empty commit
mehab Jan 17, 2023
7e40086
more removal of unirest
mehab Jan 19, 2023
6cc00de
merged changes from upstream master
mehab Jan 19, 2023
2c99bf2
defect dojo and oss index changed
mehab Jan 20, 2023
b1dca1d
removed unirest client
mehab Jan 23, 2023
e691dda
cleanup
mehab Jan 23, 2023
89ab958
removed spdx based test that is not working and is not relevant
mehab Jan 23, 2023
f7d6c1c
added unit tests for vulndb
mehab Jan 24, 2023
9e45ee7
added tests
mehab Jan 27, 2023
d718caa
Merge branch 'master' of https://github.com/mehab/dependency-track in…
mehab Jan 27, 2023
c5b0e96
Merge branch 'DependencyTrack:master' into feature/removeUnirest
mehab Jan 30, 2023
d405ab8
Merge branch 'master' of https://github.com/DependencyTrack/dependenc…
mehab Jan 31, 2023
21d2ae5
Merge branch 'feature/removeUnirest' of https://github.com/mehab/depe…
mehab Jan 31, 2023
abba6a7
test updated
mehab Jan 31, 2023
abd98a9
test updated
mehab Jan 31, 2023
3ccef34
test changes
mehab Jan 31, 2023
7298bde
proper fix
mehab Jan 31, 2023
bedb383
tests added and workiing
mehab Jan 31, 2023
5d7141b
Merge branch 'master' of https://github.com/DependencyTrack/dependenc…
mehab Jan 31, 2023
7a7b938
Merge branch 'master' of https://github.com/DependencyTrack/dependenc…
mehab Jan 31, 2023
f24701a
PR review fixes
mehab Feb 1, 2023
f156265
removed changes for bugfix from this branch
mehab Feb 1, 2023
d2bdcad
removed problematic dependency
mehab Feb 1, 2023
18524f7
removed vulndbdatamirror library
mehab Feb 1, 2023
5c5da89
changes pulled in from citi fork
mehab Feb 2, 2023
5b2bcd0
some fixes
mehab Feb 2, 2023
3fa7d4f
sonatype fixes
mehab Feb 2, 2023
c34cec8
sonatype fixes
mehab Feb 2, 2023
9496c4c
PR review fixes
mehab Feb 3, 2023
4a34be1
Merge branch 'master' of https://github.com/DependencyTrack/dependenc…
mehab Feb 3, 2023
2a7f648
renamed package per standard
mehab Feb 3, 2023
119b8f5
changed package name to follow convention
mehab Feb 3, 2023
8a049f1
changes from merge master and pr review
mehab Feb 6, 2023
4309ca9
merged changes from master upstream
mehab Feb 6, 2023
2e8af4a
intermediate changes
mehab Feb 10, 2023
860c90e
all changes completed except additional unit test
mehab Feb 10, 2023
a119f8b
Merge branch 'master' of https://github.com/DependencyTrack/dependenc…
mehab Feb 10, 2023
9cfd580
fixed unit test
mehab Feb 11, 2023
58d863f
content type found and updated for defectdojo and fortifyssc
mehab Feb 12, 2023
b95545a
Merge branch 'DependencyTrack:master' into feature/removeUnirest
mehab Feb 12, 2023
3d780f4
Merge branch 'master' of https://github.com/DependencyTrack/dependenc…
mehab Feb 12, 2023
e2c6eee
resolved merged error
mehab Feb 12, 2023
46e6640
Merge branch 'feature/removeUnirest' of https://github.com/mehab/depe…
mehab Feb 12, 2023
c974b72
merged changes from main
mehab Feb 13, 2023
1c1b1d3
added tests with wiremock
mehab Feb 13, 2023
38b3115
added tests with wiremock
mehab Feb 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 20 additions & 18 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -92,11 +92,11 @@
<lib.lucene.version>8.11.2</lib.lucene.version>
<lib.packageurl.version>1.4.1</lib.packageurl.version>
<lib.pebble.version>3.2.0</lib.pebble.version>
<lib.unirest.version>3.14.1</lib.unirest.version>
<lib.vulndb-data-mirror.version>1.0.1</lib.vulndb-data-mirror.version>
<lib.resilience4j.version>2.0.1</lib.resilience4j.version>
<lib.woodstox.version>6.5.0</lib.woodstox.version>
<lib.junit-params.version>1.1.1</lib.junit-params.version>
<lib.signpost-core.version>2.1.1</lib.signpost-core.version>
<lib.httpclient.version>4.5.14</lib.httpclient.version>
<!-- JDBC Drivers -->
<lib.jdbc-driver.mssql.version>11.2.3.jre17</lib.jdbc-driver.mssql.version>
<!-- Leave at 8.0.29 until https://github.com/datanucleus/datanucleus-rdbms/issues/446 is resolved! -->
Expand Down Expand Up @@ -226,31 +226,33 @@
<artifactId>pebble</artifactId>
<version>${lib.pebble.version}</version>
</dependency>
<!-- VulnDB data mirroring and parsing -->

<dependency>
<groupId>us.springett</groupId>
<artifactId>vulndb-data-mirror</artifactId>
<version>${lib.vulndb-data-mirror.version}</version>
<!-- Exclude transitive version so Dependency-Track can manage independently -->
<exclusions>
<exclusion>
<groupId>io.github.openunirest</groupId>
<artifactId>open-unirest-java</artifactId>
</exclusion>
</exclusions>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>${lib.httpclient.version}</version>
</dependency>
<!-- Unirest -->

<dependency>
<groupId>com.konghq</groupId>
<artifactId>unirest-java</artifactId>
<version>${lib.unirest.version}</version>
<groupId>oauth.signpost</groupId>
<artifactId>signpost-core</artifactId>
<version>${lib.signpost-core.version}</version>
<scope>compile</scope>
</dependency>

<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>${lib.httpclient.version}</version>
</dependency>


<dependency>
<groupId>com.fasterxml.woodstox</groupId>
<artifactId>woodstox-core</artifactId>
<version>${lib.woodstox.version}</version>
</dependency>
<!-- Package version parsing -->

<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-artifact</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,25 @@
package org.dependencytrack.integrations.defectdojo;

import alpine.common.logging.Logger;
import kong.unirest.UnirestInstance;
import kong.unirest.HttpRequestWithBody;
import kong.unirest.HttpResponse;
import kong.unirest.JsonNode;
import kong.unirest.json.JSONArray;
import kong.unirest.json.JSONObject;
import org.dependencytrack.common.UnirestFactory;
import org.apache.http.HttpEntity;
import org.apache.http.HttpStatus;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.HttpMultipartMode;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.InputStreamBody;
import org.apache.http.entity.mime.content.StringBody;
import org.apache.http.util.EntityUtils;
import org.dependencytrack.common.HttpClientPool;
import org.json.JSONArray;
import org.json.JSONObject;

import java.io.IOException;
import java.io.InputStream;
import java.net.URISyntaxException;
import java.net.URL;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
Expand All @@ -40,86 +50,98 @@ public class DefectDojoClient {
private final DefectDojoUploader uploader;
private final URL baseURL;

public DefectDojoClient(final DefectDojoUploader uploader, final URL baseURL) {
public DefectDojoClient(final DefectDojoUploader uploader, final URL baseURL) {
this.uploader = uploader;
this.baseURL = baseURL;
}

public void uploadDependencyTrackFindings(final String token, final String engagementId, final InputStream findingsJson) {
LOGGER.debug("Uploading Dependency-Track findings to DefectDojo");
final UnirestInstance ui = UnirestFactory.getUnirestInstance();
final HttpRequestWithBody request = ui.post(baseURL + "/api/v2/import-scan/");
HttpPost request = new HttpPost(baseURL + "/api/v2/import-scan/");
InputStreamBody inputStreamBody = new InputStreamBody(findingsJson, ContentType.DEFAULT_BINARY, "findings.json");
request.addHeader("accept", "application/json");
request.addHeader("Authorization", "Token " + token);
HttpEntity data = MultipartEntityBuilder.create().setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
.addPart("file", inputStreamBody)
.addPart("engagement", new StringBody(engagementId, ContentType.MULTIPART_FORM_DATA))
.addPart("scan_type", new StringBody("Dependency Track Finding Packaging Format (FPF) Export", ContentType.MULTIPART_FORM_DATA))
.addPart("verified", new StringBody("true", ContentType.MULTIPART_FORM_DATA))
.addPart("active", new StringBody("true", ContentType.MULTIPART_FORM_DATA))
.addPart("minimum_severity", new StringBody("Info", ContentType.MULTIPART_FORM_DATA))
.addPart("close_old_findings", new StringBody("true", ContentType.MULTIPART_FORM_DATA))
.addPart("push_to_jira", new StringBody("push_to_jira", ContentType.MULTIPART_FORM_DATA))
.addPart("scan_date", new StringBody(DATE_FORMAT.format(new Date()), ContentType.MULTIPART_FORM_DATA))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

24% of developers fix this issue

JavaUtilDate: Date has a bad API that leads to bugs; prefer java.time.Instant or LocalDate.


ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage
@sonatype-lift ignore Leave out the above finding from this PR
@sonatype-lift ignoreall Leave out all the existing findings from this PR
@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
Click here to add LiftBot to another repo.


Help us improve LIFT! (Sonatype LiftBot external survey)

Was this a good recommendation for you? Answering this survey will not impact your Lift settings.

[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

.build();
request.setEntity(data);

final HttpResponse<String> response = request
.header("accept", "application/json")
.header("Authorization", "Token " + token)
.field("file", findingsJson, "findings.json")
.field("engagement", engagementId)
.field("scan_type", "Dependency Track Finding Packaging Format (FPF) Export")
.field("verified", "true")
.field("active", "true")
.field("minimum_severity", "Info")
.field("close_old_findings", "true")
.field("push_to_jira", "false")
.field("scan_date", DATE_FORMAT.format(new Date()))
.asString();
if (response.getStatus() == 201) {
LOGGER.debug("Successfully uploaded findings to DefectDojo");
} else {
LOGGER.warn("DefectDojo Client did not receive expected response while attempting to upload "
+ "Dependency-Track findings. HTTP response code: "
+ response.getStatus() + " - " + response.getStatusText());
uploader.handleUnexpectedHttpResponse(LOGGER, request.getUrl(), response.getStatus(), response.getStatusText());

try (CloseableHttpResponse response = HttpClientPool.getClient().execute(request)) {
if (response.getStatusLine().getStatusCode() == HttpStatus.SC_CREATED) {
LOGGER.debug("Successfully uploaded findings to DefectDojo");
} else {
uploader.handleUnexpectedHttpResponse(LOGGER, request.getURI().toString(), response.getStatusLine().getStatusCode(), response.getStatusLine().getReasonPhrase());
}
} catch (IOException ex) {
uploader.handleException(LOGGER, ex);
}
}

// Pulling DefectDojo 'tests' API endpoint with engagementID filter on, and retrieve a list of existing tests
public ArrayList getDojoTestIds(final String token, final String eid) {
LOGGER.debug("Pulling DefectDojo Tests API ...");
String tests_uri = "/api/v2/tests/";
final UnirestInstance ui = UnirestFactory.getUnirestInstance();
String testsUri = "/api/v2/tests/";
LOGGER.debug("Make the first pagination call");
HttpResponse<JsonNode> response = ui.get(baseURL + tests_uri + "?limit=100&engagement=" + eid)
.header("accept", "application/json")
.header("Authorization", "Token " + token)
.asJson();
if (response.getStatus() == 200) {
if (response.getBody() != null && response.getBody().getObject() != null) {
JsonNode root = response.getBody();
JSONObject dojoObj = root.getObject();
JSONArray dojoArray = dojoObj.getJSONArray("results");
ArrayList dojoTests = jsonToList(dojoArray);
String nextUrl = "";
while (dojoObj.get("next") != null ) {
nextUrl = dojoObj.get("next").toString();
LOGGER.debug("Making the subsequent pagination call on " + nextUrl);
response = ui.get(nextUrl)
.header("accept", "application/json")
.header("Authorization", "Token " + token)
.asJson();
nextUrl = dojoObj.get("next").toString();
root = response.getBody();
dojoObj = root.getObject();
dojoArray = dojoObj.getJSONArray("results");
dojoTests.addAll(jsonToList(dojoArray));
try {
URIBuilder uriBuilder = new URIBuilder(baseURL + testsUri);
uriBuilder.addParameter("limit", "100");
uriBuilder.addParameter("engagement", eid);
HttpGet request = new HttpGet(uriBuilder.build().toString());
request.addHeader("accept", "application/json");
request.addHeader("Authorization", "Token " + token);
try (CloseableHttpResponse response = HttpClientPool.getClient().execute(request)) {
if (response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
if (response.getEntity() != null) {
String stringResponse = EntityUtils.toString(response.getEntity());
JSONObject dojoObj = new JSONObject(stringResponse);
JSONArray dojoArray = dojoObj.getJSONArray("results");
ArrayList dojoTests = jsonToList(dojoArray);
String nextUrl = "";
while (dojoObj.get("next") != null) {
nextUrl = dojoObj.get("next").toString();
LOGGER.debug("Making the subsequent pagination call on " + nextUrl);
uriBuilder = new URIBuilder(nextUrl);
request = new HttpGet(uriBuilder.build().toString());
request.addHeader("accept", "application/json");
request.addHeader("Authorization", "Token " + token);
try (CloseableHttpResponse response1 = HttpClientPool.getClient().execute(request)) {
nextUrl = dojoObj.get("next").toString();
stringResponse = EntityUtils.toString(response1.getEntity());
}
dojoObj = new JSONObject(stringResponse);
dojoArray = dojoObj.getJSONArray("results");
dojoTests.addAll(jsonToList(dojoArray));
}
LOGGER.debug("Successfully retrieved the test list ");
return dojoTests;
}
} else {
LOGGER.warn("DefectDojo Client did not receive expected response while attempting to retrieve tests list "
+ response.getStatusLine().getStatusCode() + " - " + response.getStatusLine().getReasonPhrase());
}
LOGGER.debug("Successfully retrieved the test list ");
return dojoTests;
}
} else {
LOGGER.warn("DefectDojo Client did not receive expected response while attempting to retrieve tests list "
+ response.getStatus() + " - " + response.getBody());
} catch (IOException | URISyntaxException ex) {
uploader.handleException(LOGGER, ex);
}
return null;
return new ArrayList<>();
}

// Given the engagement id and scan type, search for existing test id
public String getDojoTestId(final String engagementID, final ArrayList dojoTests) {
public String getDojoTestId(final String engagementID, final ArrayList dojoTests) {
for (int i = 0; i < dojoTests.size(); i++) {
String s = dojoTests.get(i).toString();
JSONObject dojoTest = new JSONObject(s);
if (dojoTest.get("engagement").toString().equals(engagementID) &&
dojoTest.get("scan_type").toString().equals("Dependency Track Finding Packaging Format (FPF) Export")) {
if (dojoTest.get("engagement").toString().equals(engagementID) &&
dojoTest.get("scan_type").toString().equals("Dependency Track Finding Packaging Format (FPF) Export")) {
return dojoTest.get("id").toString();
}
}
Expand All @@ -140,32 +162,34 @@ public ArrayList<String> jsonToList(final JSONArray jsonArray) {
/*
* A Reimport will reuse (overwrite) the existing test, instead of create a new test.
* The Successfully reimport will also increase the reimport counter by 1.
*/
*/
public void reimportDependencyTrackFindings(final String token, final String engagementId, final InputStream findingsJson, final String testId) {
LOGGER.debug("Re-reimport Dependency-Track findings to DefectDojo per Engagement");
final UnirestInstance ui = UnirestFactory.getUnirestInstance();
final HttpRequestWithBody request = ui.post(baseURL + "/api/v2/reimport-scan/");
final HttpResponse<String> response = request
.header("accept", "application/json")
.header("Authorization", "Token " + token)
.field("file", findingsJson, "findings.json")
.field("engagement", engagementId)
.field("scan_type", "Dependency Track Finding Packaging Format (FPF) Export")
.field("verified", "true")
.field("active", "true")
.field("minimum_severity", "Info")
.field("close_old_findings", "true")
.field("push_to_jira", "false")
.field("test", testId)
.field("scan_date", DATE_FORMAT.format(new Date()))
.asString();
if (response.getStatus() == 201) {
LOGGER.debug("Successfully reimport findings to DefectDojo");
} else {
LOGGER.warn("DefectDojo Client did not receive expected response while attempting to reimport"
+ "Dependency-Track findings. HTTP response code: "
+ response.getStatus() + " - " + response.getStatusText());
uploader.handleUnexpectedHttpResponse(LOGGER, request.getUrl(), response.getStatus(), response.getBody());
HttpPost request = new HttpPost(baseURL + "/api/v2/reimport-scan/");
request.addHeader("accept", "application/json");
request.addHeader("Authorization", "Token " + token);
InputStreamBody inputStreamBody = new InputStreamBody(findingsJson, ContentType.DEFAULT_BINARY, "findings.json");
HttpEntity fileData = MultipartEntityBuilder.create().setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
.addPart("file", inputStreamBody)
.addPart("engagement", new StringBody(engagementId, ContentType.MULTIPART_FORM_DATA))
.addPart("scan_type", new StringBody("Dependency Track Finding Packaging Format (FPF) Export", ContentType.MULTIPART_FORM_DATA))
.addPart("verified", new StringBody("true", ContentType.MULTIPART_FORM_DATA))
.addPart("active", new StringBody("true", ContentType.MULTIPART_FORM_DATA))
.addPart("minimum_severity", new StringBody("Info", ContentType.MULTIPART_FORM_DATA))
.addPart("close_old_findings", new StringBody("true", ContentType.MULTIPART_FORM_DATA))
.addPart("push_to_jira", new StringBody("push_to_jira", ContentType.MULTIPART_FORM_DATA))
.addPart("test", new StringBody(testId, ContentType.MULTIPART_FORM_DATA))
.addPart("scan_date", new StringBody(DATE_FORMAT.format(new Date()), ContentType.MULTIPART_FORM_DATA))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

24% of developers fix this issue

JavaUtilDate: Date has a bad API that leads to bugs; prefer java.time.Instant or LocalDate.


ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage
@sonatype-lift ignore Leave out the above finding from this PR
@sonatype-lift ignoreall Leave out all the existing findings from this PR
@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
Click here to add LiftBot to another repo.


Help us improve LIFT! (Sonatype LiftBot external survey)

Was this a good recommendation for you? Answering this survey will not impact your Lift settings.

[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

.build();
request.setEntity(fileData);
try (CloseableHttpResponse response = HttpClientPool.getClient().execute(request)) {
if (response.getStatusLine().getStatusCode() == HttpStatus.SC_CREATED) {
LOGGER.debug("Successfully reimport findings to DefectDojo");
} else {
uploader.handleUnexpectedHttpResponse(LOGGER, request.getURI().toString(), response.getStatusLine().getStatusCode(), response.getStatusLine().getReasonPhrase());
}
} catch (IOException ex) {
uploader.handleException(LOGGER, ex);
}
}

Expand Down
Loading