Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graphviz: Set total memory by default to 2^30 #190

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

tjerkw
Copy link

@tjerkw tjerkw commented Aug 9, 2022

Fixes #174

Set a high upper bound for total memory to prevent failure:

Cannot enlarge memory arrays. Either (1) compile with -s TOTAL_MEMORY=X with X higher than the current value 16777216, (2) compile with -s ALLOW_MEMORY_GROWTH=1 which allows increasing the size at runtime but prevents some optimizations, (3) set Module.TOTAL_MEMORY to a higher value before the program runs, or (4) if you want malloc to return NULL (0) instead of this abort, compile with -s ABORTING_MALLOC=0

Set a high upper bound for total memory to prevent failure:

> Cannot enlarge memory arrays. Either (1) compile with  -s TOTAL_MEMORY=X  with X higher than the current value 16777216, (2) compile with  -s ALLOW_MEMORY_GROWTH=1  which allows increasing the size at runtime but prevents some optimizations, (3) set Module.TOTAL_MEMORY to a higher value before the program runs, or (4) if you want malloc to return NULL (0) instead of this abort, compile with  -s ABORTING_MALLOC=0
@@ -28,7 +28,7 @@ import java.io.File
val dot = File(outputDirectory, generator.outputFileNameDot)
dot.writeText(graph.toString())

val graphviz = Graphviz.fromGraph(graph).run(generator.graphviz)
val graphviz = Graphviz.fromGraph(graph).totalMemory(999999999).run(generator.graphviz)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it really need to be that high?

@tjerkw
Copy link
Author

tjerkw commented Aug 9, 2022 via email

@SimonMarquis
Copy link
Contributor

SimonMarquis commented Aug 21, 2022

Hi, I'm not sure we want to add this default for 2 reasons.
It seems arbitrary, and not really the responsibility of this plugin.
Also, I feel like this value should be a power of 2 (I remembered testing it and failing with some values).

And finally, there already is a way to do that today, through the same mechanism in the configuration block:

graphviz = { it.totalMemory( /*...*/ ) }

The Graphviz instance configuration has been added here: #183

This should cover the need of #174

@vanniktech
Copy link
Owner

@SimonMarquis like #173 (comment) what do you think if we provide a reasonable default for totalMemory like 2^25? That should be enough, right?

@SimonMarquis
Copy link
Contributor

Yes we could provide such default, although I think bumping it to 2^25 would not change it that much. (2^24 is the current default)

Power Bits
2^24 16,777,216
2^25 33,554,432
2^26 67,108,864
2^27 134,217,728
2^28 268,435,456
2^29 536,870,912
2^30 1,073,741,824

@vanniktech
Copy link
Owner

Didn't know 2^24 is the default, then let's take 2^30?

@SimonMarquis
Copy link
Contributor

2.0.pow(30) 👍
We should also do the same for: src/main/kotlin/com/vanniktech/dependency/graph/generator/ProjectDependencyGraphGeneratorTask.kt

@vanniktech
Copy link
Owner

Yes it should be consistent. Let's go for 2^30

@SimonMarquis
Copy link
Contributor

@tjerkw do you need help to merge this?

I think we can add an extension like this in extensions.kt:

internal fun Graphviz.withDefaultTotalMemory() = totalMemory(2.0.pow(30).toInt())

And use it in both DependencyGraphGeneratorTask and ProjectDependencyGraphGeneratorTask like that:

val graphviz = Graphviz.fromGraph(graph).withDefaultTotalMemory().run(generator.graphviz)

@tjerkw
Copy link
Author

tjerkw commented Aug 28, 2022

@SimonMarquis Thanks! Updated it.

I missed the discussion but thanks for the help. MR is updated. Feel free to merge.

@vanniktech vanniktech changed the title #174 Update DependencyGraphGeneratorTask.kt Graphviz: Set total memory by default to 2^30 Aug 28, 2022
@tjerkw
Copy link
Author

tjerkw commented Aug 28, 2022

Sorry will fix

@SimonMarquis
Copy link
Contributor

SimonMarquis commented Aug 28, 2022

That's what I feared unfortunately.

* What went wrong:
Execution failed for task ':generateDependencyGraph'.
> java.lang.OutOfMemoryError: Direct buffer memory

This setting is not only a limit, but forces to somewhat allocate part of the memory, which triggers this OOM.

We could increase our Xmx config of the tests, but I'm not even sure GitHub Actions will allow this.

But keep in mind users would have to do the same, which is not very obvious from the error message.

@tonyaytlo
Copy link

Is there any updates? Are you going to merge it?

@SimonMarquis
Copy link
Contributor

As explained in my previous comment, this is not a "one size fits all" solution. Therefore and don't think we should merge this. Also, you can manually set it from your own configuration, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow increase in memory to accommodate larger modules
4 participants