Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate some passes to Mini passes #11191

Merged
merged 90 commits into from
Oct 15, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
c9ef2c3
Implement skeleton of mini pass framework
Akirathan Sep 26, 2024
43eafae
Rename MiniPassManager to MiniPassChainer
Akirathan Sep 26, 2024
4a0fd85
Implement default IR.withNewChildren
Akirathan Sep 26, 2024
1bbf991
Add MiniPassTester and MiniPassTraverser
Akirathan Sep 26, 2024
70f0a1a
Add tests for OperatorToFunction mini pass
Akirathan Sep 26, 2024
808dbd8
Add necessary withNewChildren implementations
Akirathan Sep 26, 2024
1abc70d
Add necessary withNewChildren implementations
Akirathan Sep 26, 2024
cdf1d87
MiniIRPass has metadata type argument
Akirathan Sep 27, 2024
d7b6016
Add mini pass version of LambdaShorthandToLambda
Akirathan Sep 27, 2024
1bd2b40
Add tests of mini pass version of LambdaShorthandToLambda
Akirathan Sep 27, 2024
61fed92
Implement Prefix.withNewChildren
Akirathan Sep 27, 2024
40252a2
Add IRTest to test children and withNewChildren API
Akirathan Sep 30, 2024
76dbc23
Update default IR.withNewChildren
Akirathan Sep 30, 2024
4a66460
Copy of children of DefinitionArgument.Specified is handled specifica…
Akirathan Sep 30, 2024
be2724d
Add necessary withNewChildren implementations
Akirathan Sep 30, 2024
fd3a727
LambdaToShorthandMini ignores blank fn
Akirathan Sep 30, 2024
8a5c435
Tests that have different internal naming are handled in different te…
Akirathan Sep 30, 2024
995dea5
DefinitionArgument.Specified implements withNewChildren
Akirathan Oct 1, 2024
985b8c2
Change prepare API
Akirathan Oct 1, 2024
28edccb
MiniIRPass is not parametrized based on metadata type
Akirathan Oct 1, 2024
6121750
Dummy implementation of minipass chaining
Akirathan Oct 1, 2024
54e5bd0
Add test for BinaryOperator.newChildren
Akirathan Oct 1, 2024
6b4c6b3
IR.withNewChildren must be called with the same length as children
Akirathan Oct 1, 2024
01561a1
MiniIRPass.prepare has parent parameter
Akirathan Oct 1, 2024
dd42554
Remove parent parameter from MiniIRPass.prepare
Akirathan Oct 2, 2024
f85f017
Remove MiniIRPass.runsAfter - it will not be necessary for this PR
Akirathan Oct 2, 2024
a4aaf8e
Add docs to MiniIRPass
Akirathan Oct 2, 2024
836c4c2
Add skeleton of TailCallMini pass
Akirathan Oct 2, 2024
079543a
runtime-compiler depends on org.graalvm.sdk:collections
Akirathan Oct 2, 2024
5d19be4
Implement withNewChildren on Expression via mapExpressions
Akirathan Oct 2, 2024
a369ed8
Fix MatchError in TailCallMini
Akirathan Oct 3, 2024
487003f
Revert "Implement withNewChildren on Expression via mapExpressions"
Akirathan Oct 4, 2024
b2f7053
Revert "runtime-compiler depends on org.graalvm.sdk:collections"
Akirathan Oct 4, 2024
edec03b
Replace org.graalvm.collections.EconomicSet with java.util.IdentityHa…
Akirathan Oct 4, 2024
67ff454
Revert tests that depend on InlineContext config
Akirathan Oct 4, 2024
547e9e6
Add more tests that test consistency with old pass version
Akirathan Oct 4, 2024
925a2de
Add logging to MiniPassTraverser
Akirathan Oct 4, 2024
dc6e7f3
Implement more withNewChildren methods
Akirathan Oct 4, 2024
c56c2be
Add another tail call mini pass test
Akirathan Oct 4, 2024
ea1462a
TailCallMini respect initial isInTailPosition flag
Akirathan Oct 4, 2024
18f4cea
Implement TailCallMini.toString
Akirathan Oct 4, 2024
e2fc538
Using IR.mapExpressions in MiniPassTraverser
JaroslavTulach Oct 8, 2024
a58ecc8
Detect and use MiniPassFactory in PassFactory
JaroslavTulach Oct 8, 2024
1aaa589
Move OperatorToFunction mega pass to test
Akirathan Oct 9, 2024
8604acb
Move LambdaShorthandToLambda mega pass to test
Akirathan Oct 9, 2024
0ddb96b
Use pattern match instead of isInstanceOf check in PassManager
Akirathan Oct 9, 2024
a85679d
Move TailCall mega pass to test
Akirathan Oct 9, 2024
80c4545
Implement MiniPass trait for testing mini and mega passes
Akirathan Oct 9, 2024
9a5dfea
TailCallMegaPass uses metadata from TailCall, not this
Akirathan Oct 9, 2024
d5c6983
Change API of MiniIRPass to accept only Expression
Akirathan Oct 10, 2024
89cde89
OperatorToFunctionMini checks post condition
Akirathan Oct 10, 2024
7c9f370
Add failing test to OperatorToFunctionTest
Akirathan Oct 10, 2024
92ae4fe
Implement IRDumper.dumpWithSvg
Akirathan Oct 10, 2024
65560c4
Ensure MiniPassTraverser returns transformed Module
Akirathan Oct 10, 2024
dfc54db
IRDumper handles SugaredType
Akirathan Oct 10, 2024
8771d58
MiniPassTraverser logs IRs with hash code.
Akirathan Oct 10, 2024
bdff605
Refactor TailCallMini to the new API
Akirathan Oct 10, 2024
ce56abb
typo
Akirathan Oct 10, 2024
d42d3e9
Must preprocess the module even for miniPass
JaroslavTulach Oct 11, 2024
ba7ab9d
Using analyseModuleBinding from TailCallMegaPass
JaroslavTulach Oct 11, 2024
90139e7
Allow for comparing mega and mini IR trees
JaroslavTulach Oct 11, 2024
8558d5b
Merging with develop. 47 failing tests. Mostly because of advanturous…
JaroslavTulach Oct 12, 2024
05fd38a
Making the TailCall pass closer to original mega pass code
JaroslavTulach Oct 12, 2024
367fc9a
Properly annotate withClue sections
JaroslavTulach Oct 12, 2024
36ebf2b
Associate TailCall pass metadata
JaroslavTulach Oct 12, 2024
0a02563
Rewrite using tailCandidates
JaroslavTulach Oct 12, 2024
83f4a57
Removal of notTailMeta
JaroslavTulach Oct 12, 2024
69b3274
Execute megaPass.runExpression at the end of the test
JaroslavTulach Oct 12, 2024
dea6dd6
One TailCallTest failure down to six
JaroslavTulach Oct 12, 2024
6c2b51d
Down to five failing tests in TailCallTest
JaroslavTulach Oct 12, 2024
f27ce6f
Down to three failures
JaroslavTulach Oct 12, 2024
f4077de
Reordering the test and down to two failures
JaroslavTulach Oct 12, 2024
1b021c2
Making all TailCallTest pass on mega as well as mini pass
JaroslavTulach Oct 12, 2024
8ce42e0
No need for TailCallMini
JaroslavTulach Oct 12, 2024
64b013f
Cleaning dead code up
JaroslavTulach Oct 13, 2024
f54ba6d
SectionsToBinOp is mini IR pass
JaroslavTulach Oct 13, 2024
40ef345
Introducing MiniIRPass.combine
JaroslavTulach Oct 13, 2024
8124e1e
Hiding traversal behind MiniIRPass.compile
JaroslavTulach Oct 13, 2024
668fdfd
Not including IRDumper changes in this PR
JaroslavTulach Oct 13, 2024
2af068f
No need for MetadataStorage.size
JaroslavTulach Oct 13, 2024
95b1756
Properly clone ChainedMiniPass when first or second delegates changes
JaroslavTulach Oct 14, 2024
c84e881
Removing duplicated test - mini pass testing is already inside of ass…
JaroslavTulach Oct 14, 2024
0964711
Rewriting MiniIRPass processing to queue - much easier for profiling …
JaroslavTulach Oct 14, 2024
ffd27df
MiniIRPass.prepare(parent, child)
JaroslavTulach Oct 14, 2024
924e404
Keep all IRs in local variables for easier debugging
JaroslavTulach Oct 14, 2024
2736a76
Combining subsequent mini passes
JaroslavTulach Oct 15, 2024
762045a
Both MiniPassFactory and IRPass are on par by extending IRProcessingPass
JaroslavTulach Oct 15, 2024
ef3450c
Only update IR metadata in transform methods
JaroslavTulach Oct 15, 2024
248c0f4
Removing dead code
JaroslavTulach Oct 15, 2024
befef0c
TailCall rewritten to Java
JaroslavTulach Oct 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package org.enso.compiler.pass

import org.slf4j.LoggerFactory
import org.enso.compiler.context.{InlineContext, ModuleContext}
import org.enso.compiler.core.ir.{Expression, Module}
import org.enso.compiler.core.CompilerError
Expand All @@ -20,7 +21,8 @@ class PassManager(
passes: List[PassGroup],
passConfiguration: PassConfiguration
) {
val allPasses = verifyPassOrdering(passes.flatMap(_.passes))
private val logger = LoggerFactory.getLogger(classOf[PassManager])
val allPasses = verifyPassOrdering(passes.flatMap(_.passes))

/** Computes a valid pass ordering for the compiler.
*
Expand Down Expand Up @@ -89,7 +91,26 @@ class PassManager(

val passesWithIndex = passGroup.passes.zipWithIndex

passesWithIndex.foldLeft(ir) {
logger.debug(
"runPassesOnModule[{}@{}]",
moduleContext.getName(),
moduleContext.module.getCompilationStage()
)
var pendingMiniPasses: List[MiniPassFactory] = List()
def flushMiniPass(in: Module): Module = {
if (pendingMiniPasses.nonEmpty) {
val miniPasses = pendingMiniPasses.map(factory =>
factory.createForModuleCompilation(newContext)
)
val combinedPass = miniPasses.fold(null)(MiniIRPass.combine)
logger.trace(" flushing pending mini pass: {}", combinedPass)
pendingMiniPasses = List()
MiniIRPass.compile(classOf[Module], in, combinedPass)
} else {
in
}
}
val res = passesWithIndex.foldLeft(ir) {
case (intermediateIR, (pass, index)) => {
// TODO [AA, MK] This is a possible race condition.
passConfiguration
Expand All @@ -100,13 +121,36 @@ class PassManager(

pass match {
case miniFactory: MiniPassFactory =>
val miniPass = miniFactory.createForModuleCompilation(newContext)
MiniIRPass.compile(classOf[Module], intermediateIR, miniPass)
logger.trace(
" mini collected: {}",
pass
)
val combiningPreventedBy = pendingMiniPasses.find { p =>
p.asInstanceOf[IRPass].invalidatedPasses.contains(miniFactory)
}
val irForRemainingMiniPasses = if (combiningPreventedBy.isDefined) {
logger.trace(
" pass {} forces flush before {}",
combiningPreventedBy.orNull,
miniFactory
)
flushMiniPass(intermediateIR)
} else {
intermediateIR
}
Copy link
Member

@JaroslavTulach JaroslavTulach Oct 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 2736a76 commit changes the PassManager to combine subsequent mini passes unless the already collected passes prevent such combining by declaring newly collected pass to be invalidated by their changes. Imagine there is m.enso file:

import Standard.Base.Runtime.Ref.Ref

main =
    r = Ref.new "foo"
    r.modify (_+"boo")
    r.get

then the processing of the m.enso file logs following info:

runPassesOnModule[m@AFTER_IMPORT_RESOLUTION]
   mega running: MethodDefinitions
   mini collected: org.enso.compiler.pass.desugar.SectionsToBinOp@784abd3e
   mini collected: OperatorToFunction
   mini collected: LambdaShorthandToLambda
   pass OperatorToFunction forces flush before LambdaShorthandToLambda
   flushing pending mini pass: org.enso.compiler.pass.desugar.SectionsToBinOp$Mini:org.enso.compiler.pass.desugar.OperatorToFunctionMini
   flushing pending mini pass: org.enso.compiler.pass.desugar.LambdaShorthandToLambdaMini
   mega running: ImportSymbolAnalysis
   ...

e.g. three mini passes are collected in this PassGroup. Two of them are "flushed" together by a single IR traversal. The 3rd one is "flushed" separately before a subsequent mega pass is executed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LambdaShorthandToLambda is defined as being invalidated by OperatorToFunction pass as it is known that LambdaShorthandToLambda needs to see result of OperatorToFunction in its prepare method - that's only possible if the whole tree produced by OperatorToFunction is fed into LambdaShorthandToLambda - e.g. these two mini passes cannot be merged.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imagine there is m.enso file:

import Standard.Base.Runtime.Ref.Ref

main =
    r = Ref.new "foo"
    r.modify (_+"boo")
    r.get

then the enso --log-level trace --run m.enso file logs following info (after filtering the relevant parts):

[DEBUG] [2024-10-15T06:17:26.979] [org.enso.compiler.pass.PassManager] runPassesOnModule[m@INITIAL]
   mega running: ModuleAnnotations
   mega running: DocumentationComments
   mega running: Imports
   mega running: ComplexType
   mega running: FunctionBinding
   mega running: GenerateMethodBodies
   mega running: BindingAnalysis
   mega running: ModuleNameConflicts
[DEBUG] [2024-10-15T06:17:28.103] [org.enso.compiler.pass.PassManager] runPassesOnModule[m@AFTER_IMPORT_RESOLUTION]
   mega running: MethodDefinitions
   mini collected: org.enso.compiler.pass.desugar.SectionsToBinOp@784abd3e
   mini collected: OperatorToFunction
   mini collected: LambdaShorthandToLambda
   pass OperatorToFunction forces flush before LambdaShorthandToLambda
   flushing pending mini pass: org.enso.compiler.pass.desugar.SectionsToBinOp$Mini:org.enso.compiler.pass.desugar.OperatorToFunctionMini
   flushing pending mini pass: org.enso.compiler.pass.desugar.LambdaShorthandToLambdaMini
   mega running: ImportSymbolAnalysis
   mega running: AmbiguousImportsAnalysis
   mega running: org.enso.compiler.pass.analyse.PrivateModuleAnalysis@6540cf1d
   mega running: org.enso.compiler.pass.analyse.PrivateConstructorAnalysis@ec8f4b9
   mega running: ShadowedPatternFields
   mega running: UnreachableMatchBranches
   mega running: NestedPatternMatch
   mega running: IgnoredBindings
   mega running: TypeFunctions
   mega running: TypeSignatures
[DEBUG] [2024-10-15T06:17:28.141] [org.enso.compiler.pass.PassManager] runPassesOnModule[m@AFTER_GLOBAL_TYPES]
   mega running: ExpressionAnnotations
   mega running: AliasAnalysis
   mega running: FullyQualifiedNames
   mega running: GlobalNames
   mega running: TypeNames
   mega running: org.enso.compiler.pass.resolve.MethodCalls$@bc042d5
   mega running: org.enso.compiler.pass.resolve.FullyAppliedFunctionUses$@5484117b
   mega running: AliasAnalysis
   mega running: LambdaConsolidate
   mega running: AliasAnalysis
   mega running: SuspendedArguments
   mega running: OverloadsResolution
   mega running: AliasAnalysis
   mega running: DemandAnalysis
   mega running: AliasAnalysis
   mini collected: TailCall
   flushing pending mini pass: org.enso.compiler.pass.analyse.TailCall$Mini
   mega running: org.enso.compiler.pass.resolve.Patterns$@36c2b646
   mega running: org.enso.compiler.pass.analyse.PrivateSymbolsAnalysis@37df14d1
   mega running: AliasAnalysis
   mega running: FramePointerAnalysis
   mega running: DataflowAnalysis
   mega running: CachePreferenceAnalysis
   mega running: GenericAnnotations
   mega running: UnusedBindings
   mega running: org.enso.compiler.pass.lint.NoSelfInStatic$@7efb53af

pendingMiniPasses = pendingMiniPasses.appended(miniFactory)
irForRemainingMiniPasses
case _ =>
pass.runModule(intermediateIR, newContext)
val flushedIR = flushMiniPass(intermediateIR)
logger.trace(
" mega running: {}",
pass
)
pass.runModule(flushedIR, newContext)
}
}
}
flushMiniPass(res)
}

/** Executes all passes on the [[Expression]].
Expand Down Expand Up @@ -157,11 +201,7 @@ class PassManager(
pass match {
case miniFactory: MiniPassFactory =>
val miniPass = miniFactory.createForInlineCompilation(newContext)
MiniIRPass.compile(
classOf[Expression],
intermediateIR,
miniPass
)
MiniIRPass.compile(classOf[Expression], intermediateIR, miniPass)
case _ =>
pass.runExpression(intermediateIR, newContext)
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,41 +23,42 @@ class LambdaShorthandToLambdaMini(
parent: IR,
current: Expression
): LambdaShorthandToLambdaMini = {
if (shouldSkipBlanks(parent)) {
new LambdaShorthandToLambdaMini(freshNameSupply, true)
} else {
this
}
}

private def shouldSkipBlanks(parent: IR): Boolean = {
parent match {
case Application.Prefix(fn, args, _, _, _) =>
JaroslavTulach marked this conversation as resolved.
Show resolved Hide resolved
val hasBlankArg = args.exists {
case CallArgument.Specified(_, _: Name.Blank, _, _) => true
case _ => false
}
val hasBlankFn = fn.isInstanceOf[Name.Blank]
if (hasBlankArg || hasBlankFn) {
new LambdaShorthandToLambdaMini(freshNameSupply, true)
} else {
this
}
hasBlankArg || hasBlankFn
case Application.Sequence(items, _, _) =>
val hasBlankItem = items.exists {
case _: Name.Blank => true
case _ => false
}
if (hasBlankItem) {
new LambdaShorthandToLambdaMini(freshNameSupply, true)
} else {
this
}
hasBlankItem
case Case.Expr(_: Name.Blank, _, _, _, _) =>
new LambdaShorthandToLambdaMini(freshNameSupply, true)
case _ => this
true
case _ => false
}
}

override def transformExpression(ir: Expression): Expression = {
ir match {
val newIr = ir match {
case name: Name => desugarName(name)
case app: Application => desugarApplication(app)
case caseExpr: Case.Expr => desugarCaseExpr(caseExpr)
case _ => ir
}
newIr
}

/** Desugars an arbitrary name occurrence, turning isolated occurrences of
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ case object OperatorToFunction extends IRPass with MiniPassFactory {
override lazy val invalidatedPasses: Seq[IRPass] = List(
AliasAnalysis,
DataflowAnalysis,
DemandAnalysis
DemandAnalysis,
LambdaShorthandToLambda
)

override def runModule(
Expand Down
Loading