Reduce multiplicities in .NET scans

[v6ak]

ODC groups libraries by file content (actually by hashes). This works well in Java world.

This is unfortunately not the case in the .NET world. When I resolve libraries using Nuget and perform a scan on it, I usually get many files for one library:

1. The nupkg file.
2. DLLs for multiple .NET versions in the nuget file
3. The same DLLs extracted (Easiest to get rid of, and ODC handles them well, because they are duplicates.)

My goal is to get rid of multiplied items in the reports. I can perform some preprocessing or modify the ODC code. What is the best approach for that? My preferred idea how to implement it:

Don't treat nupkg as a standard ZIP. Files in nupkg archive would be analyzed, but the evidence would be added to the nupkg bundle. This does not skip the extracted DLLs (the #3), but they might be addressed in multiple ways, e.g. by blacklisting hashes of files (maybe just DLLs) found in a nupkg archive or by removing the lib directory before running ODC.

I was also thinking about some special mode that treats a directory as a single item (with some Merkle-tree-based hashing), which should achieve similar results as grouping by nupkg, but this approach would be more complex and more general. While the increased complexity is rather obvious (not only for the Merkle-tree hashing, but mainly for deciding which directories to group), I am not sure if it has any real benefits.

Should I start implementing the nupkg grouping?

Original thread: https://groups.google.com/forum/#!topic/dependency-check/bOsmmCmKfnM

[jeremylong]

Sorry for the delayed response on this. Absolutely the dependencies should be combined. This is similar to an issue we have with the ruby analyzers - see #477. I think a similiar approach could work for both (see comments in the other issue on how we might combine these). However, if after looking at the code you think there is a better place to combine the analyzers let me know. If you have time to work on a PR I would truly appreciate it.

Best Regards,

--Jeremy

[v6ak]

Hi Jeremy,
the #477 approach looks to be essentially the same to the mine, just with mode detailed.

I'd like to work on it, but I won't probably do it this week.

Regards,
Vít Šesták 'v6ak'

[colezlaw]

I guess Dependency objects don't have a concept of a parent dependency? Trying to figure out the most elegant way to handle this, and I'm not sure the interior analyzers that "matter" (AssemblyAnalyzer, for example) have a way of "knowing" that this is actually within an extracted nupkg to add the evidence to that.

[jeremylong]

Other analyzers set the packagePath and then the dependency bundling
analyzer (?) Combines them. Not in front of code right now or i could
point to the exact lines. Problem is the assembly analyzer likely cant set
the package path.

Jeremy

On Nov 12, 2016 7:24 PM, "colezlaw" [email protected] wrote:

I guess Dependency objects don't have a concept of a parent dependency?
Trying to figure out the most elegant way to handle this, and I'm not sure
the interior analyzers that "matter" (AssemblyAnalyzer, for example) have a
way of "knowing" that this is actually within an extracted nupkg to add the
evidence to that.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#453 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA0qwqFdmUVrLW27bPGz2SbfdbD-7-eAks5q9liygaJpZM4Hiooo
.

[colezlaw]

Ah - it probably could, since it just sees a DLL - I just didn't know about
setPackagePath(dependency). I'm trying to decipher what the
DependencyBundlingAnalyzer does - maybe that will sort everything out for
me.

On Sat, Nov 12, 2016 at 8:57 PM Jeremy Long [email protected]
wrote:

Other analyzers set the packagePath and then the dependency bundling
analyzer (?) Combines them. Not in front of code right now or i could
point to the exact lines. Problem is the assembly analyzer likely cant set
the package path.

Jeremy

On Nov 12, 2016 7:24 PM, "colezlaw" [email protected] wrote:

I guess Dependency objects don't have a concept of a parent dependency?
Trying to figure out the most elegant way to handle this, and I'm not
sure
the interior analyzers that "matter" (AssemblyAnalyzer, for example)
have a
way of "knowing" that this is actually within an extracted nupkg to add
the
evidence to that.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<
#453 (comment)
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AA0qwqFdmUVrLW27bPGz2SbfdbD-7-eAks5q9liygaJpZM4Hiooo

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#453 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdyESAA0nvc_SKOgpmnIvusMOQdB5itks5q9m6cgaJpZM4Hiooo
.

[jeremylong]

I think the package path may be named incorrectly too. That came in as part
of a PR to solve this issue for Ruby and it was later extended for another
language (python?). Don't have the code in front of me again - I'll take a
look in the morning.

--Jeremy

On Mon, Nov 14, 2016 at 6:38 PM, colezlaw [email protected] wrote:

Ah - it probably could, since it just sees a DLL - I just didn't know about
setPackagePath(dependency). I'm trying to decipher what the
DependencyBundlingAnalyzer does - maybe that will sort everything out for
me.

On Sat, Nov 12, 2016 at 8:57 PM Jeremy Long [email protected]
wrote:

Other analyzers set the packagePath and then the dependency bundling
analyzer (?) Combines them. Not in front of code right now or i could
point to the exact lines. Problem is the assembly analyzer likely cant
set
the package path.

Jeremy

On Nov 12, 2016 7:24 PM, "colezlaw" [email protected] wrote:

I guess Dependency objects don't have a concept of a parent dependency?
Trying to figure out the most elegant way to handle this, and I'm not
sure
the interior analyzers that "matter" (AssemblyAnalyzer, for example)
have a
way of "knowing" that this is actually within an extracted nupkg to add
the
evidence to that.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<
#453
issuecomment-260157865
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/
AA0qwqFdmUVrLW27bPGz2SbfdbD-7-eAks5q9liygaJpZM4Hiooo

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#453 (comment)
260161383>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAdyESAA0nvc_
SKOgpmnIvusMOQdB5itks5q9m6cgaJpZM4Hiooo>

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#453 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA0qwlmalaeT_PMqFjhOByvF4UhXsBMiks5q-PETgaJpZM4Hiooo
.