WIP: Include caching improved #217
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This does not change any behaviour, but only passed these variables along through the entire stack of functions leading up to ObjFileIsUpToDate. This prepares for the next commit. Signed-off-by: Matthijs Kooijman <[email protected]>
If -debug-level=20 is passed, whenever the cached file is not usable for whatever reason, a message is displayed. This should help debug caching problems. The messages are hardcoded in the source and not put into `constants`, since they are only debug messages. Signed-off-by: Matthijs Kooijman <[email protected]>
This slightly cleans up a function call. Signed-off-by: Matthijs Kooijman <[email protected]>
This renames this function and adds a resultFile parameter. This resultFile is used for all timestamp checks against other files (source file, dep file, header files from the dep file), while the objectFile passed is now only used to check against the contents of the dep file. Both calls to this function still pass the same filename as objectFile and resultFile, so this commit should not change any behavior. Signed-off-by: Matthijs Kooijman <[email protected]>
matthijskooijman
force-pushed
the
include-caching-improved
branch
from
33fde6d to
3b4d090
Compare
Previously, the including detection process caching relied on the .d
files generated by compilation to know what .h files a given source
files depends on. This works correctly, but if a project does not
compile completely, not all .d files are generated, so not all cached
include detection results can be used.
In practice, this means that if a there is a compilation error in the
first file that is compiled, include detection will run again for all
files on the next run. If you have a few errors to solve and a big
project, this gets annoying quickly.
To fix this, the include detection process should generate .d files
itself. At first glance it appears that there is a problematic case
where the list of included header files changes, the include detection
overwrites the .d file and then compilation only sees the new list
(which was generated later than the .o file was generated). However,
since this implies that changes are made to an #include directive in the
source file itself or one of the files that are still included, this
should be detected normally. There is still a corner case when a file is
changed during the build, but that was already the case.
Since include detections uses `-o /dev/null`, the compiler generates a
slightly different .d file. During compilation, a file `foo.cpp.d` is
generated in the output directory starting with:
/path/to/foo.cpp.o: \
But when just passing `-MMD` to the preproc recipe, it generates a
`foo.d` file in the source directory starting with:
foo.o: \
To make these equal, `-MF` must be passed during include detection to
set the .d filename, and `-MT` must be passed to set the .o filename
inside the .d file.
To enable this feature, platform.txt should be modified by adding ` -MMD
-MF {dep_file} -MT {object_file}` to `preproc.macros.flags` (or
`recipe.preproc.macros`). Without any changes to platform.txt, behaviour
is unchanged.
To allow this, this adds `{dep_file}` and `{object_file}` variables to
the build properties for the preproc macros recipe. For consistency,
`{dep_file}` is also added during normal compilation, though it is not
currently used.
matthijskooijman
force-pushed
the
include-caching-improved
branch
from
3b4d090 to
f13ade8
Compare
|
✅ Build completed. ⬇️ Build URL: ℹ️ To test this build:
|
|
I can see a huge improvement while developing big projects, but the extra complexity seems a bit overwhelming for smaller ones 🙂 I remember that @ffissore had some funny times trying to find out a common dependency method which could work even for older compilers. If |
matthijskooijman
changed the title
|
✅ Build completed. ⬇️ Build URL: ℹ️ To test this build:
|
|
It seems reusing
Hence, a new recipe is probably a good idea. Not sure if Not that I have not pushed the rebased version yet, since it would completely clutter this PR with all of the commits of #236, so I'll push once that one gets merged. |
|
|
5 tasks
per1234
added
type: enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes an issue with the include detection caching where it would not use cached versions because of errors during compilation. This changes the include caching code to generate the same .d files as the compilation process, so they can be used even when compilation fails (which is convenient when working through multiple compilation errors in a big project).
The last commit in this PR is the core of it and is still a work-in-progress. It currently works, but it re-uses the same
recipe.preproc.macrosthat must be modified to take advantage of this new code. However, this recipe is also used by the "preprocess the sketch" code during prototype insertion which currently also overwrites the .d file (which is probably not ideal, perhaps this needs to be changed to pass/dev/nullinstead?). An alternative approach would be to add a new recipe for include detection and leave therecipe.preproc.macrosfor just generating a preprocessed version of a file, without any .d files being generated (and adding a new recipe that generates a .d file and errors, but not any output, perhaps using-MMinstead of-MMDto send .d file output to the file specified with -o).