Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes to DFG #1382

Draft
wants to merge 151 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
151 commits
Select commit Hold shift + click to select a range
55224ab
Experimental support to run passes in parallel
oxisto Nov 23, 2023
4b78fc3
Experimental support to run passes in parallel
oxisto Nov 23, 2023
a754912
Consider READWRITE of variables as last write location
KuechA Sep 29, 2023
fc0839f
Some fixes for pointer DFG
KuechA Oct 2, 2023
35505d4
Fix bugs
KuechA Oct 10, 2023
78a0979
More fixes
KuechA Oct 11, 2023
b654c02
Fix bugs
KuechA Oct 11, 2023
42b5bbc
Be more open!
KuechA Oct 24, 2023
bbf0629
Hotfix
KuechA Oct 27, 2023
0bd1413
Fix problem of unconnected dfg for unknown functions
KuechA Oct 27, 2023
d676dd7
Fix bug caused by inconsistencies
KuechA Oct 27, 2023
3713d7b
Use comment matcher also in CXX frontend
KuechA Nov 10, 2023
1d540c4
Try more fixes
KuechA Nov 16, 2023
a3841c2
Try more efficient flags
KuechA Nov 21, 2023
4f2db1d
Added comment to be careful about replacing
oxisto Nov 21, 2023
26e1e49
Fixed NPE
oxisto Nov 21, 2023
e909f92
more parallel
oxisto Nov 23, 2023
4f8c894
Rebased with parallel passes
oxisto Nov 24, 2023
34332d1
++
oxisto Nov 28, 2023
33d2d1a
No more DFG Edges to FunctionDeclaration
morbitzer Nov 28, 2023
e395527
Merge branch 'main' into small-dfg-fix
KuechA Jan 11, 2024
ba40f18
Added continueAfterHit flag to followNextDFGEdgesUntilHit
morbitzer Jan 17, 2024
006a8d8
Merge branch 'main' into small-dfg-fix
KuechA Mar 18, 2024
52bdc7e
Merge branch 'main' into small-dfg-fix
KuechA Mar 19, 2024
4ac2dbb
intellij formatting
morbitzer Mar 19, 2024
9075a5e
public methods#
KuechA Mar 20, 2024
6ec8133
Merge branch 'main' into small-dfg-fix
oxisto Mar 27, 2024
521771e
Multi path followDFG
KuechA Apr 4, 2024
bcb9be2
Merge branch 'main' into small-dfg-fix
oxisto Apr 10, 2024
945d414
Merge remote-tracking branch 'origin/main' into small-dfg-fix
oxisto Apr 10, 2024
65c59f9
Merge branch 'main' into small-dfg-fix
KuechA Apr 11, 2024
48d969f
Fix missing import
KuechA Apr 25, 2024
7abad4c
fixed broken EOG in do-while loops
morbitzer Apr 29, 2024
59f2b3e
prevent Functionsummaries from breaking in case of not enough arguments
morbitzer May 5, 2024
3edb188
Prevent CDG from looping endlessly
morbitzer May 5, 2024
5d726eb
fixed merge conflict
morbitzer May 5, 2024
532d2d8
switched alreadySeen to hashCode-List to avoid endless loops
morbitzer May 5, 2024
131f7cd
Merge branch 'main' into small-dfg-fix
KuechA May 7, 2024
bfe9bd6
Only propagate type updates in a `Reference` from FULL previous DFG n…
oxisto May 8, 2024
212c560
Merge branch 'dfg-propagate-fix' into small-dfg-fix
oxisto May 8, 2024
93b36d8
fixed broken alreadySeen checks
morbitzer May 13, 2024
0d628e2
avoid double DFG-Edges
morbitzer May 13, 2024
6ea4323
Node.followNextFullDFGEdgesUntilHit now doesn't stop anymore in neste…
morbitzer May 14, 2024
8ec6b2d
Very ugly hack to avoid loops
oxisto May 21, 2024
01e98d5
Merge remote-tracking branch 'origin/main' into small-dfg-fix
oxisto May 28, 2024
a326770
IncompleteType now has a language
oxisto May 28, 2024
472d4a4
Merge remote-tracking branch 'origin/main' into small-dfg-fix
oxisto Jun 3, 2024
bb6057d
fixed a typo
morbitzer Jun 3, 2024
7d01b9f
Merge branch 'main' into small-dfg-fix
morbitzer Jun 14, 2024
8784d6b
fixy fix
oxisto Jun 17, 2024
63804f4
Differentiate between CallingContexts for edgeProperties
morbitzer Jun 18, 2024
46cd61a
Trying to fix edgePropertiesMap with Aliases
morbitzer Jun 25, 2024
bc1fd32
Started adding PointerDFs
morbitzer Jul 22, 2024
101f617
Remove unneeded DFG-Edges for pointer access
morbitzer Jul 22, 2024
cce64b9
Revert "Remove unneeded DFG-Edges for pointer access"
morbitzer Jul 23, 2024
548b016
Revert "Started adding PointerDFs"
morbitzer Jul 23, 2024
fc41e34
Added PointerDataFlows
morbitzer Jul 26, 2024
983203f
Changed unwrapReference to not remove UnaryOperator
morbitzer Aug 2, 2024
08d4d06
started working on pointer stuff
morbitzer Aug 7, 2024
626c10e
removed valueAccess flag from PointerDataflowGranularity
morbitzer Sep 4, 2024
178213e
Added/Fixed PointerDataFlows
morbitzer Sep 4, 2024
49373d0
started changing DFG for C(++)
morbitzer Sep 4, 2024
d54eff1
Revert "started changing DFG for C(++)"
morbitzer Sep 5, 2024
abae76c
Revert "Added/Fixed PointerDataFlows"
morbitzer Sep 5, 2024
33a7f2d
Introduced PointerReference and PointerDereference Node
morbitzer Sep 5, 2024
8957f84
adapted unwrapReference for PointerReference/PointerDereference
morbitzer Sep 5, 2024
fe4c49b
added missing PointerDereference/PointerReference classes
morbitzer Sep 5, 2024
d3c930c
started adapting AliasCFS Pass to new pointer magic
morbitzer Sep 9, 2024
a713c3f
PointerDataFlowGranularity for PointerReferences
morbitzer Sep 10, 2024
c7a7556
Consider PointerReferences for AliasCFSDFG
morbitzer Sep 23, 2024
2c68aca
Pointer Stuff
morbitzer Oct 7, 2024
a38b853
Merge branch 'main' into small-dfg-fix
maximiliankaul Oct 9, 2024
2c1c054
add prev/nextPDG shorthand
maximiliankaul Oct 10, 2024
0e915fb
Merge branch 'mk/pdgshorthand' into small-dfg-fix
maximiliankaul Oct 10, 2024
fddcc87
Merge branch 'main' into small-dfg-fix
maximiliankaul Oct 10, 2024
218bb19
hashCode and equals for custom granularities
morbitzer Oct 14, 2024
104e91c
fixed hashCode()
morbitzer Oct 14, 2024
25cfa8b
replaced hashCodes with data class
morbitzer Oct 14, 2024
2e4c223
determine granularity based on edgePropertiesMap
morbitzer Oct 14, 2024
31546ab
added input to pointer(de)reference
morbitzer Oct 15, 2024
b899cc4
Try a new eog iteration
KuechA Oct 16, 2024
18edf60
Merge branch 'main' into small-dfg-fix
maximiliankaul Oct 30, 2024
3ba4f87
Fix one test
KuechA Oct 30, 2024
59f108f
Comment out test which does not make sense
KuechA Oct 30, 2024
833630c
Merge branch 'main' into small-dfg-fix
maximiliankaul Nov 6, 2024
0c96c4c
use refers to instead of aliases (which are not known at this point)
maximiliankaul Nov 6, 2024
1c5a7fe
allow other code to rewrite findAndSetProperties
maximiliankaul Nov 6, 2024
ee424f6
Document ignored tests
KuechA Nov 12, 2024
9178d65
Update fixpoint iteration to be faster
KuechA Nov 12, 2024
bfe6f66
Type propagation
KuechA Nov 12, 2024
bc5c5d4
Try to fix tests but without effect
KuechA Nov 12, 2024
c010192
Started on Points to Analysis
morbitzer Nov 15, 2024
b370219
continued on PointsToPass
morbitzer Nov 19, 2024
0bd9b5e
add location to BinaryOperator.toString()
maximiliankaul Nov 19, 2024
ab46dd1
continued again on PointsToPassTest
morbitzer Nov 19, 2024
e887a4b
initial version of PointerPassTest working
morbitzer Nov 20, 2024
9514d44
Added PointsToPassTest
morbitzer Nov 20, 2024
642d4fe
added conditions test to PointsToTest
morbitzer Nov 21, 2024
13c5396
started working on struct-support for PointsToPass
morbitzer Nov 21, 2024
a3d37b0
finished struct support for PointsToPass
morbitzer Nov 22, 2024
0e15fae
added array support for PointsToPass (no test yet)
morbitzer Nov 22, 2024
51513aa
Added Test for PointerPassTest for Arrays
morbitzer Nov 25, 2024
d23367c
started on memcpy support for PointsToPass
morbitzer Nov 25, 2024
de718ae
finished memcpy support for PointsToPass
morbitzer Nov 25, 2024
5e98082
Added PointerToPointer test for PointersToPass
morbitzer Nov 26, 2024
2df3476
TODOs for context sensitive analysis
KuechA Nov 27, 2024
6e04278
Initial try with EOG ordering in pass
KuechA Nov 27, 2024
9290c62
Update
KuechA Nov 27, 2024
518074f
Update
KuechA Nov 27, 2024
aa86831
started using PlaceholderMemoryValue and UnknownMemoryValue
morbitzer Nov 27, 2024
1863196
fix goto / label missing names
maximiliankaul Nov 28, 2024
242edfb
code review
maximiliankaul Nov 28, 2024
add4710
test++
maximiliankaul Nov 28, 2024
6f484a4
test++
maximiliankaul Nov 28, 2024
13ada9d
Added FunctionSummaries to PointsToPass
morbitzer Dec 3, 2024
f927edc
continued on FunctionSummaries
morbitzer Dec 3, 2024
93e9ad3
more functionSummaries handling
morbitzer Dec 4, 2024
505fcab
cleaned up
morbitzer Dec 4, 2024
a9ebfcb
Merge branch 'main' into small-dfg-fix
KuechA Dec 4, 2024
60b9c0c
Fix spotless
KuechA Dec 5, 2024
252c688
Fix query test
KuechA Dec 5, 2024
adefc32
Fix query test
KuechA Dec 5, 2024
1745f1f
Revert "Fix query test"
KuechA Dec 5, 2024
398b3ec
Revert "Fix query test"
KuechA Dec 5, 2024
1c51494
Revert "Fix spotless"
KuechA Dec 5, 2024
bacfefa
Merge branch 'main' into small-dfg-fix
KuechA Dec 5, 2024
478dc01
Improved on Function Summaries
morbitzer Dec 5, 2024
c55f51c
moved memoryValue to prevDFG
morbitzer Dec 9, 2024
ef0559d
got things running again
morbitzer Dec 9, 2024
3349ec6
continued on functionSummaries
morbitzer Dec 9, 2024
f9dc30b
tests work again but ugly
morbitzer Dec 10, 2024
451bf8d
Fix some bugs
KuechA Dec 10, 2024
1948ca3
Add value only once if unknown
KuechA Dec 10, 2024
c521a21
First working version of function Summaries
morbitzer Dec 10, 2024
d2224e2
finished functionSummaries
morbitzer Dec 11, 2024
3bfe2fc
fixed resolving of literals
morbitzer Dec 11, 2024
2558fb5
fixed GhidraTest
morbitzer Dec 12, 2024
c9df04a
attached function Summaries to functionDeclaration
morbitzer Dec 12, 2024
831547a
FunctionSummaries now also include partial writes
morbitzer Dec 13, 2024
63013f6
Merge branch 'main' into small-dfg-fix
KuechA Dec 17, 2024
92533e6
Add sanity check
KuechA Dec 17, 2024
b80db86
Some fixes
KuechA Dec 17, 2024
6ed5357
draw dfg edges via call expressions
morbitzer Dec 18, 2024
fde0c20
bugfix
morbitzer Dec 18, 2024
c3e20d4
More size checks
KuechA Dec 18, 2024
baa2a73
Fix missing edges to parameter values
KuechA Dec 18, 2024
2a98f1a
add check
KuechA Dec 18, 2024
70d0b10
better function summaries processing
morbitzer Dec 19, 2024
d637020
Remove duplicated code
KuechA Dec 19, 2024
9e05ca1
initial version using a set
KuechA Dec 19, 2024
f40fd06
working ghidra test
morbitzer Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
FunctionSummaries now also include partial writes
morbitzer committed Dec 13, 2024
commit 831547abe65a44ecac96109d7c99448655076964
Original file line number Diff line number Diff line change
@@ -34,7 +34,6 @@ import de.fraunhofer.aisec.cpg.graph.statements.*
import de.fraunhofer.aisec.cpg.graph.statements.expressions.Block
import de.fraunhofer.aisec.cpg.graph.statements.expressions.Expression
import de.fraunhofer.aisec.cpg.graph.types.Type
import de.fraunhofer.aisec.cpg.helpers.IdentitySet
import java.util.*
import org.apache.commons.lang3.builder.ToStringBuilder
import org.neo4j.ogm.annotation.Relationship
@@ -77,13 +76,13 @@ open class FunctionDeclaration : ValueDeclaration(), DeclarationHolder, EOGStart
}

/**
* Saves the information on which parameter(s) of the function are modified by the function. This
* is interesting since we need to add DFG edges between the modified parameter and the
* Saves the information on which parameter(s) of the function are modified by the function.
* This is interesting since we need to add DFG edges between the modified parameter and the
* respective argument(s). For each [ParameterDeclaration] as well as the
* [MethodDeclaration.receiver] that has some incoming DFG-edge within this
* [FunctionDeclaration], we store all previous DFG nodes.
*/
var functionSummary = mutableMapOf<Node, IdentitySet<Pair<Node, Boolean>>>()
var functionSummary = mutableMapOf<Node, MutableSet<Pair<Node, Boolean>>>()

/** Returns true, if this function has a [body] statement. */
fun hasBody(): Boolean {
Original file line number Diff line number Diff line change
@@ -84,7 +84,9 @@ class DFGPass(ctx: TranslationContext) : ComponentPass(ctx) {
callingContext = CallingContextOut(call)
)
(arg as? Reference)?.let {
it.access = AccessValues.READWRITE
// The access value stays on READ. Even if it's a pointer, only the
// dereference will be written.
// it.access = AccessValues.READWRITE
it.refersTo?.let { it1 -> it.nextDFGEdges += it1 }
}
}
Original file line number Diff line number Diff line change
@@ -132,8 +132,21 @@ class PointsToPass(ctx: TranslationContext) : EOGStarterPass(ctx, orderDependenc

private fun storeFunctionSummary(node: FunctionDeclaration, doubleState: PointsToState2) {
node.parameters.forEach { param ->
val addresses = doubleState.getAddresses(param)
val indexes = mutableSetOf<Node>()
doubleState.getAddresses(param).forEach { indexes.addAll(doubleState.getValues(it)) }
addresses.forEach { addr ->
indexes.addAll(doubleState.getValues(addr))
// Additionally check for partial writes to fields
if (addr is MemoryAddress) {
addr.fieldAddresses
.flatMap { it.value }
.forEach {
// indexes.addAll( doubleState.getValues(it)
// )
indexes.add(it)
}
}
}
indexes.forEach { index ->
val finalValue =
doubleState.declarationsState.elements
@@ -154,7 +167,7 @@ class PointsToPass(ctx: TranslationContext) : EOGStarterPass(ctx, orderDependenc
// If so, store the last write for the parameter in the FunctionSummary
?.forEach { value ->
node.functionSummary
.computeIfAbsent(param) { identitySetOf() }
.computeIfAbsent(param) { mutableSetOf() }
.add(Pair(value, true))
}
}
@@ -196,11 +209,12 @@ class PointsToPass(ctx: TranslationContext) : EOGStarterPass(ctx, orderDependenc
to add them to the FunctionSummary */
var doubleState = doubleState
if (currentNode.returnValues.isNotEmpty()) {
val parentFD = currentNode.scope?.parent?.astNode as? FunctionDeclaration
val parentFD =
currentNode.firstParentOrNull { it is FunctionDeclaration } as? FunctionDeclaration
if (parentFD != null) {
currentNode.returnValues.forEach { retval ->
parentFD.functionSummary
.computeIfAbsent(currentNode) { identitySetOf() }
.computeIfAbsent(currentNode) { mutableSetOf() }
.addAll(doubleState.getValues(retval).map { Pair(it, false) })
}
}
@@ -230,8 +244,8 @@ class PointsToPass(ctx: TranslationContext) : EOGStarterPass(ctx, orderDependenc
} else {
// Add a dummy function Summary so that we don't try this every time
// In this dummy, all parameters point to the return
val newValues: IdentitySet<Pair<Node, Boolean>> =
invoke.parameters.map { Pair(it, false) }.toIdentitySet()
val newValues: MutableSet<Pair<Node, Boolean>> =
invoke.parameters.map { Pair(it, false) }.toMutableSet()
invoke.functionSummary[ReturnStatement()] = newValues
}
}
@@ -385,10 +399,12 @@ class PointsToPass(ctx: TranslationContext) : EOGStarterPass(ctx, orderDependenc
/* No need to set the address, this already happens in the constructor */
val addresses = doubleState.getAddresses(currentNode)

val values =
(currentNode as? HasInitializer)?.initializer?.let { initializer ->
doubleState.getValues(initializer)
} ?: identitySetOf()
val values = identitySetOf<Node>()

(currentNode as? HasInitializer)?.initializer?.let { initializer ->
if (initializer is Literal<*>) values.add(initializer)
else values.addAll(doubleState.getValues(initializer))
}

var doubleState =
doubleState.push(
@@ -574,10 +590,6 @@ class PointsToPass(ctx: TranslationContext) : EOGStarterPass(ctx, orderDependenc

fun getValues(node: Node): Set<Node> {
return when (node) {
is MemoryAddress -> {
/* In these cases, we simply have to fetch the current value for the MemoryAddress from the DeclarationState */
fetchElementFromDeclarationState(node)
}
is PointerReference -> {
/* For PointerReferences, the value is the address of the input
* For example, the value of `&i` is the address of `i`
@@ -595,12 +607,7 @@ class PointsToPass(ctx: TranslationContext) : EOGStarterPass(ctx, orderDependenc
identitySetOf(UnknownMemoryValue(node.name))
}
val retVal = identitySetOf<Node>()
inputVal.forEach {
// If we have a literal here, we don't further resolve this b/c we don't
// know anything about memory addresses
if (it is Literal<*>) retVal.addAll(fetchElementFromDeclarationState(it))
else retVal.addAll(this.getValues(it))
}
inputVal.forEach { retVal.addAll(this.getValues(it)) }
retVal
}
is Declaration -> {
@@ -659,8 +666,8 @@ class PointsToPass(ctx: TranslationContext) : EOGStarterPass(ctx, orderDependenc
} else identitySetOf(UnknownMemoryValue(node.name))
} else identitySetOf(UnknownMemoryValue(node.name))
}
is Literal<*>, // -> identitySetOf(UnknownMemoryValue(Name(node.value.toString())))
is BinaryOperator -> identitySetOf(node)
/* In these cases, we simply have to fetch the current value for the MemoryAddress from the DeclarationState */
else -> fetchElementFromDeclarationState(node)
}
}
Original file line number Diff line number Diff line change
@@ -1079,6 +1079,16 @@ class PointsToPassTest {
tu.allChildren<Literal<*>> { it.location?.region?.startLine == 177 }.first()
assertNotNull(literal0Line177)

// MemberExpressions
val meLine201 =
tu.allChildren<MemberExpression> { it.location?.region?.startLine == 201 }.first()
assertNotNull(meLine201)

// CastExpressions
val ceLine201 =
tu.allChildren<CastExpression> { it.location?.region?.startLine == 201 }.first()
assertNotNull(ceLine201)

// Line 159
assertEquals(1, local_20Line159.prevDFG.size)
assertEquals(1, local_20Line159.prevDFG.size)
@@ -1105,26 +1115,43 @@ class PointsToPassTest {

// Line 179
assertEquals(3, local_28DerefLine179.prevDFG.size)
assertTrue(local_28DerefLine179.prevDFG.contains(literal10Line166))
assertTrue(local_28DerefLine179.prevDFG.contains(literal0Line177))
assertEquals(
1,
2,
local_28DerefLine179.prevDFG
.filterIsInstance<UnknownMemoryValue>()
.filter { it.name.localName == "0" }
.filter { it.name.localName == "0" || it.name.localName == "16" }
.size
)
assertTrue(local_28DerefLine179.prevDFG.contains(literal0Line177))
assertEquals(2, local_28DerefLine179.memoryAddress.size)
assertTrue(local_28DerefLine179.memoryAddress.contains(literal0Line167))
assertTrue(
local_28DerefLine179.memoryAddress
.filterIsInstance<UnknownMemoryValue>()
.filter { it.name.localName == "0" }
.size == 1
)
assertTrue(local_28DerefLine179.memoryAddress.contains(local_10Line172))

// Line 180
assertEquals(2, local_28Line180.prevDFG.size)
assertTrue(literal0Line167 in local_28Line180.prevDFG)
assertTrue(
local_28Line180.prevDFG
.filterIsInstance<UnknownMemoryValue>()
.filter { it.name.localName == "0" }
.size == 1
)
assertTrue(local_10Line172 in local_28Line180.prevDFG)

// Line 181
println(local_28DerefLine181)
assertEquals(5, local_28DerefLine181.prevDFG.size)
assertTrue(local_28DerefLine181.prevDFG.contains(meLine201))
assertTrue(local_28DerefLine181.prevDFG.contains(ceLine201))
assertTrue(
local_28DerefLine181.prevDFG.any {
it is UnknownMemoryValue && it.name.localName == "DAT_0011b1c8"
}
)

// Line 192
println(param_1Line193)