custom-setup: build path doesn't show Cabal version from extra-deps

[sjakobi]

~/tmp $ stack --resolver lts-8.2 new cabal-extra-dep-repro simple-library
~/tmp $ cd cabal-extra-dep-repro/
~/t/cabal-extra-dep-repro $ vim cabal-extra-dep-repro.cabal # Add custom-setup with dependencies on base and Cabal, use build-type Custom
~/t/cabal-extra-dep-repro $ vim stack.yaml # Add Cabal-1.24.0.0 and directory-1.2.7.1 to extra-deps
~/t/cabal-extra-dep-repro $ stack build
Warning: File listed in cabal-extra-dep-repro.cabal file does not exist: README.md
directory-1.2.7.1: configure
Package directory uses a custom Cabal build, but does not use a custom-setup stanza
Not using the explicit setup deps approach based on configuration
Strongly recommend fixing the package's cabal file
directory-1.2.7.1: build
Package directory uses a custom Cabal build, but does not use a custom-setup stanza
Not using the explicit setup deps approach based on configuration
Strongly recommend fixing the package's cabal file
directory-1.2.7.1: copy/register
Package directory uses a custom Cabal build, but does not use a custom-setup stanza
Not using the explicit setup deps approach based on configuration
Strongly recommend fixing the package's cabal file
Package directory uses a custom Cabal build, but does not use a custom-setup stanza
Not using the explicit setup deps approach based on configuration
Strongly recommend fixing the package's cabal file
process-1.4.3.0: configure
Package process uses a custom Cabal build, but does not use a custom-setup stanza
Not using the explicit setup deps approach based on configuration
Strongly recommend fixing the package's cabal file
process-1.4.3.0: build
Package process uses a custom Cabal build, but does not use a custom-setup stanza
Not using the explicit setup deps approach based on configuration
Strongly recommend fixing the package's cabal file
process-1.4.3.0: copy/register
Package process uses a custom Cabal build, but does not use a custom-setup stanza
Not using the explicit setup deps approach based on configuration
Strongly recommend fixing the package's cabal file
Package process uses a custom Cabal build, but does not use a custom-setup stanza
Not using the explicit setup deps approach based on configuration
Strongly recommend fixing the package's cabal file
Cabal-1.24.0.0: configure
Cabal-1.24.0.0: build
Cabal-1.24.0.0: copy/register
cabal-extra-dep-repro-0.1.0.0: configure (lib)
[1 of 2] Compiling Main             ( /home/simon/tmp/cabal-extra-dep-repro/Setup.hs, /home/simon/tmp/cabal-extra-dep-repro/.stack-work/dist/x86_64-linux/Cabal-1.24.2.0/setup/Main.o )
[2 of 2] Compiling StackSetupShim   ( /home/simon/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /home/simon/tmp/cabal-extra-dep-repro/.stack-work/dist/x86_64-linux/Cabal-1.24.2.0/setup/StackSetupShim.o )
Linking /home/simon/tmp/cabal-extra-dep-repro/.stack-work/dist/x86_64-linux/Cabal-1.24.2.0/setup/setup ...
Configuring cabal-extra-dep-repro-0.1.0.0...
cabal-extra-dep-repro-0.1.0.0: build (lib)
Preprocessing library cabal-extra-dep-repro-0.1.0.0...
[1 of 1] Compiling Lib              ( src/Lib.hs, .stack-work/dist/x86_64-linux/Cabal-1.24.2.0/build/Lib.o )
Warning: File listed in cabal-extra-dep-repro.cabal file does not exist: README.md
cabal-extra-dep-repro-0.1.0.0: copy/register
Installing library in
/home/simon/tmp/cabal-extra-dep-repro/.stack-work/install/x86_64-linux/lts-8.2/8.0.2/lib/x86_64-linux-ghc-8.0.2/cabal-extra-dep-repro-0.1.0.0-Fua4G1E7sSb6CVOk8fjsxL
Registering cabal-extra-dep-repro-0.1.0.0...
Completed 4 action(s).

The build paths list Cabal-1.24.2.0 where I would expect Cabal-1.24.0.0.

The verbose logs do reveal though that the Setup.hs file was built with Cabal-1.24.0.0:

2017-02-22 21:44:31.640577: [debug] Run process: /home/simon/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-8.0.2 --make -odir /home/simon/tmp/cabal-extra-dep-repro/.stack-work/dist/x86_64-linux/Cabal-1.24.2.0/setup -hidir /home/simon/tmp/cabal-extra-dep-repro/.stack-work/dist/x86_64-linux/Cabal-1.24.2.0/setup -i -i. -clear-package-db -global-package-db -package-db=/home/simon/.stack/snapshots/x86_64-linux/lts-8.2/8.0.2/pkgdb -package-db=/home/simon/tmp/cabal-extra-dep-repro/.stack-work/install/x86_64-linux/lts-8.2/8.0.2/pkgdb -hide-all-packages -package-id=Cabal-1.24.0.0-Esh4VVEmqHRHKkyoOU0Nv -package-id=base-4.9.1.0 -optP-include -optP/home/simon/tmp/cabal-extra-dep-repro/.stack-work/dist/x86_64-linux/Cabal-1.24.2.0/setup/setup_macros.h /home/simon/tmp/cabal-extra-dep-repro/Setup.hs /home/simon/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs -main-is StackSetupShim.mainOverride -o /home/simon/tmp/cabal-extra-dep-repro/.stack-work/dist/x86_64-linux/Cabal-1.24.2.0/setup/setup -threaded

The warnings for directory and process seem a bit overkill too.

[mgsloan]

Hmm, yes I believe it is using the globally installed Cabal version instead. The support for the custom-setup stanza made it so that snapshot / extra-deps Cabal will actually be used.

I'm thinking it will make sense to almost always use the snapshot / extra-deps version of Cabal - see #3049 . Lets see how that all turns out before addressing this.

[mgsloan]

Hmm, this is particularly hard to fix, because in quite a few places stack assumes it is just using one version of Cabal. The custom-setup support means it now will potentially use two different versions of Cabal.

This means that for stack path --dist-dir to work right and yield a path with the right Cabal version, it will need to know which package we need the dist-dir for.

The only clean solution I can see to this is to remove envConfigCabalVersion, and remove all uses of cabalVersionL. In particular, this means paths will no longer contain the Cabal version.

@snoyberg Will bad things happen if we drop the Cabal version from the paths? If we don't trust it to gracefully handle using dist directories written by a different version, then perhaps we can store the Cabal version along with the config cache. If the version has changed, wipe out the dist dir.

This would also shorten paths, which is good for avoidance of MAX_PATH issues..

[sjakobi]

Will bad things happen if we drop the Cabal version from the paths?

My understanding is that the differentiation by Cabal version helped keep apart build artifacts from different resolvers, effectively providing separate "build caches" for different resolvers / project configurations.

Obviously these "separated build caches" worked imperfectly – when you switch from e.g. lts-8.3 to lts-8.4 you still have to rebuild everything (right?) because they use the same compiler and therefore the same Cabal version. (I hope I got this right.)

I'm wondering if – loosely speaking – implicit snapshots could provide a better "criterion" for separating build artifacts… but I'm a bit too tired right now, this might not make any sense at all.

EDIT:

Obviously these "separated build caches" worked imperfectly – when you switch from e.g. lts-8.3 to lts-8.4 you still have to rebuild everything (right?) because they use the same compiler and therefore the same Cabal version. (I hope I got this right.)

Actually this "imperfection" might be a good heuristic for determining which project configurations to keep apart, and where to allow the build artifacts from one configuration to overwrite those from another configuration.

[snoyberg]

I don't believe any change is needed here. The build path references the globally installed Cabal library, which is still being used for non-custom setups. For the custom setups which are using the local or snapshot version of Cabal, the snapshot itself is encoding which version is being used, and changing that Cabal version will either entail using a different snapshot or forcing recompilation of the local packages.

[mgsloan]

Hmm, true, we likely don't need to change anything here. It does seems weird to me that we're putting a different Cabal version in the path. If the mismatch doesn't matter, then it makes me think we don't need it in the paths at all.

[sjakobi]

I agree that the problem isn't severe enough to warrant making things more complicated.

If the mismatch doesn't matter, then it makes me think we don't need it in the paths at all.

I tried to explain in #3020 (comment) why I think that the Cabal version in the path (currently!) has its use.