Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

port intraprocedural analysis and fix the long tail.. #664

Merged
merged 56 commits into from
Oct 26, 2023

Conversation

swissiety
Copy link
Collaborator

@swissiety swissiety commented Aug 8, 2023

for deca and other more elaborated BodyInterceptors that were added to soot in the meantime.

  • check if the used PriorityQueue is a better implementation than one from a Collections.
  • fix changing complete RightOp in ReplaceUseStmtVisitor
  • handle nulls e.g. from incomplete information in TypeAssigner
  • fix naming of Locals that are not referring to fields (i.e. the one starting with $ - which are now starting that way)
  • improve ClassHierarchy - introduce FallsThroughStmt and make Stmt, BranchingStmt and new FallsThroughStmt Interfaces
  • fix invalid internal state from MutableStmtGraph.insertBefore(...)
  • fix Aggregator
  • fix DeadAssignmentEliminator (it seems to remove more than necessary / does remove a complete branch-target of an IfStmt without fixing the missing branch - resulting in an invalid StmtGraph)
  • fix LocalSplitter - seems it starts already here - (or better port the newer/improved one in Soot?) as it does not (replace all former Local references in Stmts?) anyway it creates a precondition in a way that the DeadAssignmentEliminator wants to remove stmts which it does not when the LocalSplitter is not called. Which means it has leftovers in the StmtGraph or does something else wrong. Have a look at the newest LocalSplitter in Soot before fixing the current port as it seems to be a more precise algorithm. couldnt reproduce this assumption - deadassignmenteliminator failed in the mentioned commit already on its own
  • introduces new parameters to the putEdge() method for modifying the Stmtgraph edges which gives the developer the necessary power to modify the correct successor of a BranchingStmt - putEdge(FallsThroughStmt, Stmt) works like before and putEdge(Stmt, int, Stmt) needs a successorIdx according to the semantics of the respective BranchingStmt.
  • As a sideeffect we can introduce: putEdge(BranchingStmt, int, Stmt) to replace putEdge(Stmt, int, Stmt) all Stmts that have no successor like return/throw can't be used as a first Argument as they would fail already at compiletime.

closes #667, #692

@swissiety
Copy link
Collaborator Author

swissiety commented Sep 15, 2023

  • sets default ByteCodeClassLoadingOptions i.e. BodyInterceptor to create reasonable Jimple (still more Interceptors needed to port from Soot to improve the Jimple generation)
  • fixes multiple Issues in Aggregator / Copypropagator / TypeAssigner
  • we need to build sootUp more robust i.e. write algorithms that fail early if necessary but work as far as possible if we have incomplete information (we could log.info() in those cases were we detect missing information or implement a more elaborated system for imprecisions..?)
  • test default bodyinterceptors on different real inputs

@swissiety swissiety marked this pull request as ready for review September 15, 2023 11:09
@codecov
Copy link

codecov bot commented Sep 15, 2023

Codecov Report

Attention: 611 lines in your changes are missing coverage. Please review.

Comparison is base (baa460f) 65.37% compared to head (4669188) 64.05%.

Additional details and impacted files
@@              Coverage Diff              @@
##             develop     #664      +/-   ##
=============================================
- Coverage      65.37%   64.05%   -1.33%     
- Complexity      3408     3430      +22     
=============================================
  Files            313      317       +4     
  Lines          14973    15385     +412     
  Branches        2524     2613      +89     
=============================================
+ Hits            9789     9855      +66     
- Misses          4294     4622     +328     
- Partials         890      908      +18     
Files Coverage Δ
...a/sootup/callgraph/AbstractCallGraphAlgorithm.java 89.63% <100.00%> (ø)
...a/sootup/callgraph/RapidTypeAnalysisAlgorithm.java 89.77% <100.00%> (ø)
.../main/java/sootup/core/graph/MutableStmtGraph.java 77.77% <ø> (ø)
...ore/src/main/java/sootup/core/graph/StmtGraph.java 69.23% <ø> (+4.19%) ⬆️
....core/src/main/java/sootup/core/jimple/Jimple.java 87.23% <100.00%> (ø)
.../java/sootup/core/jimple/common/ref/JArrayRef.java 91.66% <ø> (ø)
.../java/sootup/core/jimple/common/ref/JFieldRef.java 50.00% <ø> (ø)
...otup/core/jimple/common/ref/JInstanceFieldRef.java 83.33% <ø> (ø)
...sootup/core/jimple/common/ref/JStaticFieldRef.java 72.72% <ø> (ø)
...ore/jimple/common/stmt/AbstractDefinitionStmt.java 81.25% <100.00%> (-13.49%) ⬇️
... and 66 more

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@swissiety
Copy link
Collaborator Author

swissiety commented Sep 15, 2023

  • fix AugEvalTest
  • one of the Interceptors the Aggregator scrambles the code order..
    e.g.
           r2 := @this: target.exercise2.FileNotClosedAliasing
	   specialinvoke $stack3.<target.exercise2.File: void <init>()>()
	   r0 = new target.exercise2.File

and i hope its only the (already) wrong order that makes a leftover of the $stack3 which should be the r0 as well..

@struce2
Copy link

struce2 commented Sep 26, 2023

When I use commit b789f9a153ffb9ba42a64d686be7863a9cdbee2d, it throw an exception, the error message as follows

	... 14 more
Caused by: java.lang.IllegalStateException: if $stack12 != 0: If must have '2' outgoing flow but has '1'.
	at sootup.core.graph.StmtGraph.validateStmtConnectionsInGraph(StmtGraph.java:215)
	... 15 more
Caused by: java.lang.IllegalStateException: if $stack12 != 0: If must have '2' outgoing flow but has '1'.

maybe ConditionalBranchFolder BodyInterceptor has an error, but I don't how t fix it 😅.

MutableStmtGraph.successors(JIfStmt) returned List size is 0 or 1, so maybe the stmtGraph generater has an error...

@swissiety
Copy link
Collaborator Author

When I use commit b789f9a153ffb9ba42a64d686be7863a9cdbee2d, it throw an exception, the error message as follows

	... 14 more
Caused by: java.lang.IllegalStateException: if $stack12 != 0: If must have '2' outgoing flow but has '1'.
	at sootup.core.graph.StmtGraph.validateStmtConnectionsInGraph(StmtGraph.java:215)
	... 15 more
Caused by: java.lang.IllegalStateException: if $stack12 != 0: If must have '2' outgoing flow but has '1'.

maybe ConditionalBranchFolder BodyInterceptor has an error, but I don't how t fix it 😅.

MutableStmtGraph.successors(JIfStmt) returned List size is 0 or 1, so maybe the stmtGraph generater has an error...

please share the complete StackTrace.

@struce2
Copy link

struce2 commented Sep 27, 2023

When I use commit b789f9a153ffb9ba42a64d686be7863a9cdbee2d, it throw an exception, the error message as follows

	... 14 more
Caused by: java.lang.IllegalStateException: if $stack12 != 0: If must have '2' outgoing flow but has '1'.
	at sootup.core.graph.StmtGraph.validateStmtConnectionsInGraph(StmtGraph.java:215)
	... 15 more
Caused by: java.lang.IllegalStateException: if $stack12 != 0: If must have '2' outgoing flow but has '1'.

maybe ConditionalBranchFolder BodyInterceptor has an error, but I don't how t fix it 😅.
MutableStmtGraph.successors(JIfStmt) returned List size is 0 or 1, so maybe the stmtGraph generater has an error...

please share the complete StackTrace.

When enable ConditionalBranchFolder BodyInterceptor, stacktrace as follows

java.lang.IllegalStateException: Failed to apply sootup.java.bytecode.interceptors.ConditionalBranchFolder@2b27cc70 to <org.apache.jsp.java_detection_samples: void _jspService(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)>
	at sootup.java.bytecode.frontend.AsmMethodSource.resolveBody(AsmMethodSource.java:261)
	at sootup.core.model.SootMethod.lazyBodyInitializer(SootMethod.java:98)
	at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:181)
	at sootup.core.model.SootMethod.getBody(SootMethod.java:177)
	at main.Main.run(Main.java:62)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at main.Main.main(Main.java:138)
Caused by: java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
	at java.base/java.util.Objects.checkIndex(Objects.java:359)
Caused by: java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1

	at java.base/java.util.ArrayList.get(ArrayList.java:427)
	at sootup.java.bytecode.interceptors.ConditionalBranchFolder.interceptBody(ConditionalBranchFolder.java:74)
	at sootup.java.bytecode.frontend.AsmMethodSource.resolveBody(AsmMethodSource.java:258)
	... 13 more

Execution failed for task ':Main.main()'.
> Process 'command '/home/home/applications/java/jdk-17.0.7/bin/java'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

when disable ConditionalBranchFolder BodyInterceptor, stacktrace as follows

java.lang.RuntimeException: StmtGraph of <org.apache.jsp.java_detection_samples: void _jspService(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)> is invalid.
	at sootup.core.model.Body$BodyBuilder.build(Body.java:554)
	at sootup.java.bytecode.frontend.AsmMethodSource.resolveBody(AsmMethodSource.java:264)
	at sootup.core.model.SootMethod.lazyBodyInitializer(SootMethod.java:98)
	at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:181)
	at sootup.core.model.SootMethod.getBody(SootMethod.java:177)
	at main.Main.run(Main.java:62)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at main.Main.main(Main.java:138)
Caused by: java.lang.IllegalStateException: visualize invalid StmtGraph: http://magjac.com/graphviz-visual-editor/....
	at sootup.core.graph.StmtGraph.validateStmtConnectionsInGraph(StmtGraph.java:242)
Caused by: java.lang.IllegalStateException: visualize invalid StmtGraph: http://magjac.com/graphviz-visual-editor/...

	at sootup.core.model.Body$BodyBuilder.build(Body.java:552)
	... 14 more
Caused by: java.lang.IllegalStateException: if $stack12 != 0: If must have '2' outgoing flow but has '1'.
	at sootup.core.graph.StmtGraph.validateStmtConnectionsInGraph(StmtGraph.java:215)
	... 15 more
Caused by: java.lang.IllegalStateException: if $stack12 != 0: If must have '2' outgoing flow but has '1'.


Execution failed for task ':Main.main()'.

the url is too long, so i deleted something like ?dot=digraph+G+%7B%0A%09compound%3Dtrue%0A%...

here is classes file

compiled.zip

sootUp setup

        JavaProject applicationProject =
                JavaProject.builder(new JavaLanguage(8))
                        .addInputLocation(
                                new JavaClassPathAnalysisInputLocation("compiled", SourceType.Application))
                        .build();

        JavaView view = applicationProject.createMutableView();
        view.configBodyInterceptors(analysisInputLocation -> BytecodeClassLoadingOptions.Default);

@struce2
Copy link

struce2 commented Sep 27, 2023

disable ConditionalBranchFolder BodyInterceptor mean comment out the new ConditionalBranchFolder() line in BytecodeBodyInterceptors.Default

@swissiety
Copy link
Collaborator Author

@struce2 can you check if the error still occurs with your input?

@struce2
Copy link

struce2 commented Oct 23, 2023

@struce2 can you check if the error still occurs with your input?

Works fine without error

…rs can not be added as they are FallsThroughStmt nor BranchingStmt
# Conflicts:
#	sootup.java.bytecode/src/test/java/sootup/java/bytecode/Soot1577.java
#	sootup.java.bytecode/src/test/java/sootup/java/bytecode/minimaltestsuite/java11/TypeInferenceLambdaTest.java
#	sootup.java.bytecode/src/test/java/sootup/java/bytecode/minimaltestsuite/java8/MethodAcceptingLamExprTest.java
@swissiety swissiety requested a review from JonasKlauke October 23, 2023 11:59
JonasKlauke and others added 5 commits October 25, 2023 09:51
# Conflicts:
#	sootup.java.bytecode/src/test/java/sootup/java/bytecode/interceptors/AggregatorTest.java
@swissiety swissiety merged commit 4a02db3 into develop Oct 26, 2023
6 of 8 checks passed
@swissiety swissiety deleted the add/intraprocedural branch October 26, 2023 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cleanup Jimple via BodyInterceptors - necessary for DECA LABS
4 participants