Share build artifacts between zomes to speed up compilation

[pospi]

Hard to know whether this is a Holonix-level thing or a result of the way hc package is implemented. But I notice that when compiling a DNA, modules that are shared by the zomes within that DNA are re-compiled separately for every zome. It would be nice if possible for them to share the same build cache, so that subsequent compile steps can be skipped.

[pospi]

@thedavidmeister need to elevate priority on this. It is causing builds to time out on TravisCI and I can't see any way to work around that on a free plan... "The job exceeded the maximum time limit for jobs, and has been terminated."

https://travis-ci.org/holo-rea/holo-rea/builds/598477699

(CircleCI fails a couple of tests for no particular reason FWIW, which is why I'm moving away from it.)

[pospi]

@thedavidmeister is it at all possible that this can be achieved by manually altering the .hcbuild files?

As in, if I use the same directory for --target-dir for all zomes, they should be able to share build artifacts since they depend on the same crates... right?

The other question is- can artifact be generated to not end up inside --target-dir? I expect if I move the final WASM bundles hc build won't be able to find them...

[thedavidmeister]

@pospi potentially, test it.. i think there was some reason we're not doing that...

the big thing that trips people up is having the IDE also recompiling things away in the background to the same path, thus invalidating the artifacts they're building in the shell

[pospi]

I've experienced that problem as the cargo cache being locked when trying to build on CLI because of a stale editor cargo process. We talking about the same thing?

I actually think this setup might help with the IDE conflicts, because cargo in the IDE will be using its own default locations for the build cache, whereas hc package is following the config in .hcbuild files. Not sure if they were already the same but I figured they'd have differed anyway.

Anyway, made the changes, seems to work wonderfully. Console output confirms that shared modules are only built once. IDE seems to be playing at least as well as before.

Feel like documenting this as part of the Holonix docs? May even be worth adding a section on multi-DNA development...

[thedavidmeister]

it might be worth updating hc itself?

[pospi]

Not sure I understand. Why's that?

[thedavidmeister]

to update what the scaffolding generates?

[pospi]

FYI I'm not sure having both hc package and your IDE building into the same folder is a good idea... runaway processes & lock contention stuff going on in the background isn't visible, and it can result in editor feedback lagging for minutes at a time. So I'm following the app-spec pattern and setting all build folders to be based in /tmp for .hcbuild-related processes.

[thedavidmeister]

it's not a good idea, it's a mistake

rust builds don't play nice with each other

[pospi]

Hitting up against quirks in cloud provider platforms now. I wouldn't recommend setting a build folder in /tmp because your local machine may clear that when it sleeps (meaning it will slow you down); and on Google Cloud Platform at least, /tmp seems to get cleared and require a full rebuild quite regularly.

Perhaps the best location is just a reserved folder name in one's homedir?

[thedavidmeister]

hmm, so it was that way for a long time and the complaint was that rust produces double digit GBs of build artifacts that are never cleaned up (even though there is a flush command, people don't know/think/want to use it)

it was moved to /tmp specifically so that the machine would periodically force a cleanup on some schedule 🤔

[pospi]

Lol I see I see

…

On Thu, 5 Dec 2019 at 02:57, David Meister ***@***.***> wrote: hmm, so it was that way for a long time and the complaint was that rust produces double digit GBs of build artifacts that are never cleaned up (even though there is a flush command, people don't know/think/want to use it) it was moved to /tmp specifically so that the machine would periodically force a cleanup on some schedule 🤔 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <holochain/holonix#68>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADK2JE2ONGT23SKBBATJWTQW7OQPANCNFSM4I2U3LWQ> .

-- writings <http://pospi.spadgos.com/> | twitter <https://twitter.com/pospigos> | github <https://github.com/pospi> ← where to find me ValueFlows <https://www.valueflo.ws/> | HoloREA <https://github.com/holo-rea/> ← things I'm working on

[thedavidmeister]

@pospi the main rust native lever to pull (rather than whatever hc is doing) is to set $CARGO_HOME so i tend to defer to that

at the least we could document how to override $CARGO_HOME on a per-project basis (extending the shell hook) so that people can move it out of /tmp if they want something more permanent