Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert Microsoft Secure Score to OHDF #6007

Merged
merged 79 commits into from
Aug 7, 2024
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
dfdd047
draft of the microsoftSecureScore converter
meme112233 Jul 17, 2024
2621d63
bugfix: reading of secureScoreProfiles doc needed field for the array
meme112233 Jul 17, 2024
6bb8070
minor update map/filter
meme112233 Jul 17, 2024
da00d81
add threats to tags from profile matching controlScore
meme112233 Jul 18, 2024
074dd7a
add testing and update status check to use scoreInPercentage
meme112233 Jul 18, 2024
f9dd52f
update testing and fingerprinting
meme112233 Jul 18, 2024
3f12007
update fingerprinting for accept combined msftSecureScore document vi…
meme112233 Jul 18, 2024
1a18d6b
lint fix
meme112233 Jul 18, 2024
86592ed
msft-config-mapper-2.ts renamed without 2, cleanup commented out code
meme112233 Jul 18, 2024
62878a3
code cleanup. delete unused code
meme112233 Jul 18, 2024
cca15a3
tslint allow commented out code in test suite
meme112233 Jul 18, 2024
efb1109
revert jest version bump
meme112233 Jul 19, 2024
f96258e
revert package.json to minimal requirements being added (only typing …
meme112233 Jul 19, 2024
7842362
rename mapper for consistency. msft_secure_score_mapper
meme112233 Jul 19, 2024
7cad9c8
update test name and remove linting inline disble block for commented…
meme112233 Jul 19, 2024
a88392e
rename files for org per PR comments
meme112233 Jul 19, 2024
387fbf3
update tags. fix array of array issue. add tiers,services,userImpact …
meme112233 Jul 19, 2024
23b1171
bugfix: replace missed exports due to rename to MsftSecureScoreMapper
meme112233 Jul 19, 2024
353519e
bugfix/ typo
meme112233 Jul 22, 2024
8a003c8
console debugging
meme112233 Jul 22, 2024
f4f5120
Revert "console debugging"
meme112233 Jul 22, 2024
59bf07b
Passthrough fix; minor styling changes
charleshu-8 Jul 22, 2024
6b1b3e2
Merge branch 'master' into meme-working
charleshu-8 Jul 22, 2024
bb32263
actual sample msft secureScore.json test doc updated to have value: []
meme112233 Jul 22, 2024
97e9610
bugfix/populate code with control data and optionally profiles data
meme112233 Jul 22, 2024
5a3ab7a
Update libs/hdf-converters/src/msft-secure-score-mapper.ts
meme112233 Jul 22, 2024
946412b
remove 'summary' field as no value available
meme112233 Jul 22, 2024
ca595c8
Update libs/hdf-converters/src/msft-secure-score-mapper.ts
meme112233 Jul 22, 2024
be4b3a0
improve names of parameters in arrow functions
meme112233 Jul 22, 2024
7a1ea4a
Update libs/hdf-converters/src/msft-secure-score-mapper.ts
meme112233 Jul 22, 2024
8f7133b
remove unuded profile.version field as no value known from Microsoft
meme112233 Jul 22, 2024
a720dc4
code cleanup. remove unused import
meme112233 Jul 22, 2024
767e30f
lint and update test expected results
meme112233 Jul 22, 2024
96e0ea6
update test data
meme112233 Jul 22, 2024
7a270f0
Missed argument name change
charleshu-8 Jul 23, 2024
53a2ace
Linting
charleshu-8 Jul 23, 2024
363cc2f
implementationStatus -> controls.results.code_desc, remediation -> de…
meme112233 Jul 23, 2024
01c0a83
update test data for changes to mapper
meme112233 Jul 23, 2024
4995960
Update msft-secure-score-mapper.ts
ejaronne Jul 24, 2024
c78d45c
Merge pull request #6019 from mitre/secureScore_nist_default
meme112233 Jul 24, 2024
12c442c
lint fix and update test data
meme112233 Jul 24, 2024
e1cfe87
relocate NIST to be in tags
meme112233 Jul 24, 2024
3d7bec6
sort exports per PR comment
meme112233 Jul 29, 2024
4510d36
add Msft_Secure_mapper to supported formats README.md
meme112233 Jul 29, 2024
c979383
typo fixed
meme112233 Jul 29, 2024
3dba0d7
add run_time to mapper. required for downstream transformations that …
meme112233 Jul 30, 2024
44dfdf6
update delimeter on control title from ... to \n
meme112233 Aug 1, 2024
933fd37
rename tag: group->category in dederence to msft naming
meme112233 Aug 5, 2024
d5078b7
utilize lodash.uniq for tag.threats[]
meme112233 Aug 5, 2024
7225794
add profiles[].remediationImpact as descriptions[label:rationale]
meme112233 Aug 5, 2024
f8c5062
add secure score to FileReader.vue
meme112233 Aug 5, 2024
af1c4a3
merge from master
meme112233 Aug 5, 2024
9d8e43d
update merge of records by id/cat
meme112233 Aug 5, 2024
be3747b
fix rawdata passthrough
meme112233 Aug 5, 2024
4c07e73
update profiles.title include runID
meme112233 Aug 5, 2024
fb8203f
handle output readability better
meme112233 Aug 5, 2024
15c96a3
track secureScoreControlProfile.rank as control.tag.rank
meme112233 Aug 5, 2024
daa958a
remove run_time
meme112233 Aug 5, 2024
636d739
cleanup inports
meme112233 Aug 5, 2024
309b657
lint fix
meme112233 Aug 5, 2024
d228148
conditional includsion of tags
meme112233 Aug 5, 2024
13afed3
secureScoreResults used to output full OHDF report per secureScore re…
meme112233 Aug 6, 2024
185aa41
bugfix/ exports from msft-secure-score-mapper fixed
meme112233 Aug 6, 2024
98242f0
update exports msftSecureMapper
meme112233 Aug 6, 2024
9668cdc
export MsftSecureScoreMapper
meme112233 Aug 6, 2024
a8f4e55
Merge branch 'master' into meme-working
meme112233 Aug 6, 2024
ab6113e
add unmapped fields as tags or passthrough data
meme112233 Aug 6, 2024
0ddff07
update MsftSecureScoreResult type hints
meme112233 Aug 6, 2024
83ce9c1
update MsftSecureScoreResult type hints
meme112233 Aug 6, 2024
163ff01
add withRaw parameter to msft secure score results
meme112233 Aug 6, 2024
529ebcf
use utils.global constants for default NIST tags
meme112233 Aug 6, 2024
f18fd5d
remove duplicate tag 'rank'
meme112233 Aug 6, 2024
438dee0
convert forEach to for ... of
meme112233 Aug 6, 2024
68ead45
convert forEach to map(..)
meme112233 Aug 6, 2024
d8e142a
delete extra copy of combined_msft.json
meme112233 Aug 6, 2024
b7b8080
lint fix
meme112233 Aug 6, 2024
58357f7
private keyword isn't that useful
Amndeep7 Aug 7, 2024
710cdf4
memoized the getProfiles function so that the repeated calls to the f…
Amndeep7 Aug 7, 2024
3537d98
Merge branch 'master' into meme-working
Amndeep7 Aug 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions apps/frontend/src/store/report_intake.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import {
INPUT_TYPES,
IonChannelMapper,
JfrogXrayMapper,
MsftSecureScoreMapper,
NessusResults,
NetsparkerMapper,
NiktoMapper,
Expand Down Expand Up @@ -229,6 +230,8 @@ export class InspecIntake extends VuexModule {
switch (typeGuess) {
case INPUT_TYPES.JFROG:
return new JfrogXrayMapper(convertOptions.data).toHdf();
case INPUT_TYPES.MSFT_SEC_SCORE:
return new MsftSecureScoreMapper(convertOptions.data).toHdf();
case INPUT_TYPES.ASFF:
return Object.values(
new ASFFResultsMapper(convertOptions.data).toHdf()
Expand Down
1 change: 1 addition & 0 deletions libs/hdf-converters/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,4 @@ export * from './src/utils/fingerprinting';
export * from './src/veracode-mapper';
export * from './src/xccdf-results-mapper';
export * from './src/zap-mapper';
export * from './src/msft-secure-score-mapper';
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
5 changes: 5 additions & 0 deletions libs/hdf-converters/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
"xml2json": "tsx data/converters/xml2json.ts"
},
"dependencies": {
"@microsoft/microsoft-graph-types": "^2.40.0",
"@aws-sdk/client-config-service": "^3.95.0",
"@e965/xlsx": "^0.20.0",
"@mdi/js": "^7.0.96",
Expand All @@ -34,7 +35,9 @@
"@types/ms": "^0.7.31",
"@types/mustache": "^4.1.2",
"@types/papaparse": "^5.3.2",
"@types/revalidator": "^0.3.12",
"@types/triple-beam": "^1.3.2",
"@types/validator": "^13.12.0",
"@types/xml2js": "^0.4.9",
"axios": "^1.3.5",
"compare-versions": "^6.0.0",
Expand All @@ -48,10 +51,12 @@
"ms": "^2.1.3",
"mustache": "^4.2.0",
"papaparse": "^5.3.1",
"revalidator": "^0.3.1",
"run-script-os": "^1.1.6",
"semver": "^7.6.0",
"tailwindcss": "^3.3.3",
"tw-elements": "^1.0.0-beta2",
"validator": "^13.12.0",
"winston": "^3.6.0",
"xml-formatter": "^3.6.2",
"xml-parser-xo": "^4.1.1",
Expand Down

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

9,352 changes: 9,352 additions & 0 deletions libs/hdf-converters/sample_jsons/msft_secure_score_mapper/secure_score-hdf.json

Large diffs are not rendered by default.

202 changes: 202 additions & 0 deletions libs/hdf-converters/src/msft-secure-score-mapper.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
import {
SecureScore,
ControlScore,
SecureScoreControlProfile
} from '@microsoft/microsoft-graph-types';
import {ExecJSON} from 'inspecjs';
import {version as HeimdallToolsVersion} from '../package.json';
import {BaseConverter, ILookupPath, MappedTransform} from './base-converter';

export class MsftSecureScoreMapper extends BaseConverter {
withRaw: boolean;
profiles: SecureScoreControlProfile[];

private getProfiles(controlName: string): SecureScoreControlProfile[] {
return this.profiles.filter((p) => p.id === controlName);
}

mappings: MappedTransform<
ExecJSON.Execution & {passthrough: unknown},
ILookupPath
> = {
platform: {
name: 'Heimdall Tools',
release: HeimdallToolsVersion
},
version: HeimdallToolsVersion,
statistics: {},
profiles: [
{
name: 'Microsoft Secure Score',
meme112233 marked this conversation as resolved.
Show resolved Hide resolved
version: {path: 'v4'},
meme112233 marked this conversation as resolved.
Show resolved Hide resolved
title: {
transformer: (d: SecureScore) =>
meme112233 marked this conversation as resolved.
Show resolved Hide resolved
`Azure Secure Score report: TenantID: ${d.azureTenantId}`
meme112233 marked this conversation as resolved.
Show resolved Hide resolved
},
summary: 'NAME',
meme112233 marked this conversation as resolved.
Show resolved Hide resolved
supports: [],
attributes: [],
groups: [],
status: 'loaded',
controls: [
{
path: 'controlScores',
id: {
transformer: (d: ControlScore) =>
`${d.controlCategory}:${d.controlName}`
},
title: {
transformer: (d: ControlScore) => {
const titles = this.getProfiles(d.controlName || '')
.filter((p) => p.title !== undefined)
.map((p) => p.title);

if (titles.length > 0) {
return titles.join('... ');
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
} else {
return `${d.controlCategory || ''}:${d.controlName || ''}`;
}
}
},
desc: {
transformer: (d: ControlScore) => d.description || ''
},
meme112233 marked this conversation as resolved.
Show resolved Hide resolved
impact: {
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
transformer: (d: ControlScore) => {
// return controlCategory from the profile document where its id matches the controlName
const knownMaxScores = this.getProfiles(
d.controlName || ''
).map((p) => p.maxScore || 0);

if (knownMaxScores.length === 0) {
return 0.5;
}

const highMaxScore = Math.max(...knownMaxScores);
return highMaxScore / 10.0;
}
},
refs: [],
tags: {
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
group: {
meme112233 marked this conversation as resolved.
Show resolved Hide resolved
transformer: (d: ControlScore) => {
// return controlCategory from the profile document where its id matches the controlName
return this.getProfiles(d.controlName || '').map(
(p) => p.controlCategory
);
}
},
tiers: {
transformer: (d: ControlScore) => {
// return tiers from the profile document where its id matches the controlName
return this.getProfiles(d.controlName || '').map(
(p) => p.tier
);
}
},
threats: {
transformer: (d: ControlScore) => {
// return unique threats from the profile document where its id matches the controlName
const uniqs: Set<string> = new Set();
meme112233 marked this conversation as resolved.
Show resolved Hide resolved
this.getProfiles(d.controlName || '').forEach((p) =>
p.threats?.forEach((threat) => uniqs.add(threat))
);
return [...uniqs];
}
},
services: {
transformer: (d: ControlScore) => {
// return thrserviceeats from the profile document where its id matches the controlName
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
return this.getProfiles(d.controlName || '').map(
(p) => p.service
);
}
},
userImpacts: {
transformer: (d: ControlScore) => {
// return userImpacts from the profile document where its id matches the controlName
return this.getProfiles(d.controlName || '')
.filter((p) => p.userImpact !== undefined)
.map((p) => p.userImpact);
}
}
},
source_location: {},
code: {
transformer: (
d: ControlScore & {implementationStatus: string}
) => {
const profiles = this.getProfiles(d.controlName || '');

if (profiles?.length === 0) {
return d.implementationStatus;
charleshu-8 marked this conversation as resolved.
Show resolved Hide resolved
}
}
},
results: [
{
status: {
transformer: (
d: ControlScore & {scoreInPercentage: number}
) => {
if (d.scoreInPercentage === 100) {
return ExecJSON.ControlResultStatus.Passed;
}

const knownMaxScores = this.getProfiles(
d.controlName || ''
).map((p) => p.maxScore || 0);

const highMaxScore = Math.max(...knownMaxScores);

if (knownMaxScores.length === 0) {
// no Profile found matching the controlName
return ExecJSON.ControlResultStatus.Failed;
} else if (d.score === undefined) {
return ExecJSON.ControlResultStatus.Error;
} else if (d.score === highMaxScore) {
return ExecJSON.ControlResultStatus.Passed;
} else {
return ExecJSON.ControlResultStatus.Failed;
}
}
},
code_desc: {
transformer: (
d: ControlScore & {implementationStatus: string}
) => {
const remediations = this.getProfiles(d.controlName || '')
.filter((p) => p.remediation !== undefined)
.map((p) => p.remediation);

if (remediations.length > 0) {
return remediations.join('\n\n');
}
}
},
start_time: {transformer: () => this.data.createdDateTime}
}
]
}
],
sha256: ''
}
],
passthrough: {
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
transformer: (data: Record<string, any>): Record<string, unknown> => {
return {
auxiliary_data: [
{name: 'Microsoft Secure Score', data: this.profiles}
],
...(this.withRaw && {raw: data})
};
}
}
};
constructor(secureScore_and_profiles_combined: string, withRaw = false) {
const rawParams = JSON.parse(secureScore_and_profiles_combined);
super(rawParams.secureScore, true);
this.withRaw = withRaw;
this.profiles = rawParams.profiles.value as SecureScoreControlProfile[];
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
}
}
2 changes: 2 additions & 0 deletions libs/hdf-converters/src/utils/fingerprinting.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ export enum INPUT_TYPES {
GOSEC = 'gosec',
IONCHANNEL = 'ionchannel',
JFROG = 'jfrog',
MSFT_SEC_SCORE = 'msft_secure_score',
NIKTO = 'nikto',
SARIF = 'sarif',
SNYK = 'snyk',
Expand Down Expand Up @@ -36,6 +37,7 @@ const fileTypeFingerprints: Record<INPUT_TYPES, string[]> = {
'trigger_hash'
],
[INPUT_TYPES.JFROG]: ['total_count', 'data'],
[INPUT_TYPES.MSFT_SEC_SCORE]: ['secureScore', 'profiles'],
[INPUT_TYPES.NIKTO]: ['banner', 'host', 'ip', 'port', 'vulnerabilities'],
[INPUT_TYPES.SARIF]: ['$schema', 'version', 'runs'],
[INPUT_TYPES.SNYK]: [
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
import fs from 'fs';
import {MsftSecureScoreMapper} from '../../../src/msft-secure-score-mapper';
import {omitVersions} from '../../utils';

describe('msft_secure_score_mapper', () => {
it('Successfully converts Microsoft Secure Score reports', () => {
const mapper = new MsftSecureScoreMapper(
JSON.stringify({
Amndeep7 marked this conversation as resolved.
Show resolved Hide resolved
secureScore: JSON.parse(
fs.readFileSync(
'sample_jsons/msft_secure_score_mapper/sample_input_report/secureScore.json',
{
encoding: 'utf-8'
}
)
),
profiles: JSON.parse(
fs.readFileSync(
'sample_jsons/msft_secure_score_mapper/sample_input_report/profiles.json',
{
encoding: 'utf-8'
}
)
)
})
);

// fs.writeFileSync(
// 'sample_jsons/msft_secure_score_mapper/secure_score-hdf.json',
// JSON.stringify(mapper.toHdf(), null, 2)
// );

expect(omitVersions(mapper.toHdf())).toEqual(
omitVersions(
JSON.parse(
fs.readFileSync(
'sample_jsons/msft_secure_score_mapper/secure_score-hdf.json',
{
encoding: 'utf-8'
}
)
)
)
);
});
});

describe('msft_secure_score_mapper_withraw', () => {
it('Successfully converts withRaw flagged Microsoft Secure Score reports', () => {
const mapper = new MsftSecureScoreMapper(
JSON.stringify({
secureScore: JSON.parse(
fs.readFileSync(
'sample_jsons/msft_secure_score_mapper/sample_input_report/secureScore.json',
{
encoding: 'utf-8'
}
)
),
profiles: JSON.parse(
fs.readFileSync(
'sample_jsons/msft_secure_score_mapper/sample_input_report/profiles.json',
{
encoding: 'utf-8'
}
)
)
}),
true
);

// fs.writeFileSync(
// 'sample_jsons/msft_secure_score_mapper/secure_score-hdf-withraw.json',
// JSON.stringify(mapper.toHdf(), null, 2)
// );

expect(omitVersions(mapper.toHdf())).toEqual(
omitVersions(
JSON.parse(
fs.readFileSync(
'sample_jsons/msft_secure_score_mapper/secure_score-hdf-withraw.json',
{
encoding: 'utf-8'
}
)
)
)
);
});
});
Loading