feat: add support for .gitignore #1908

jbedard · 2024-09-04T01:22:36Z

What type of PR is this?

Feature

What package or component does this PR mostly affect?

all

What does this PR do? Why is it needed?

Allow users to opt-in to respecting the gitignore when collecting files for gazelle extensions to consume.

While bazel can access git-ignored files many repositories manually exclude git-ignored content by not including it in source file lists or globs in BUILDs or by duplicating the git-ignore entries as # gazelle:exclude.

Ideally users would put all bazel ignored content in .bazelignore but without glob support that is often not done.

This should remain opt-in for repositories wanting it but by default bazel can access git-ignored content so gazelle should as well.

Which issues(s) does this PR fix?

Fixes #1166

Other notes for review

jbedard · 2024-09-05T17:58:18Z

@fmeum can you confirm if adding this to the gazelle core is something we'd like to do? If so are there preferences for which library to use vs manual?

tyler-french · 2024-09-05T18:06:07Z

How is this different from using 'gazelle:exclude' directives?

jbedard · 2024-09-05T18:12:36Z

People won't want to copy+paste gitignore files into BUILD files...?

jbedard · 2024-09-05T18:14:33Z

Functionally gitignore also works a bit different with things like ! operators etc. That is one reason to use a library such as go-git instead of doing it ourselves but we could also potentially do it ourselves but don't support !...

fmeum · 2024-09-05T19:35:01Z

I kind of like this as it reduces the amount of Gazelle-specific configuration.

I don't know which library is best, but we should definitely use one rather than roll our own.

jbedard · 2024-09-05T20:32:35Z

I don't know which library is best, but we should definitely use one rather than roll our own.

I agree we should use a real library and not try to re-implement gitignore logic, otherwise we'll just end up with a never-ending stream "it doesn't work like git" bugs.

The go-git library is what I've used in all the aspect gazelle extensions and I've never had any issues so I think that's a good place to start. If we add appropriate tests then we should be able to change the implementation later.

jbedard · 2024-09-05T23:44:13Z

FWIW I've also tried other libraries which all had issues in the aspect gazelle extensions:

https://github.com/sabhiram/go-gitignore - too slow, don't recall if there were other bugs
https://github.com/denormal/go-gitignore - had multiple bugs ([Bug]: new gitignore module causes regressions in parsing aspect-build/aspect-cli#657)

Those both also seem to be unmaintained.

Others I have not tried seem to also be unmaintained and many look too simple or slow:

walk/config.go

jbedard · 2024-09-06T09:27:24Z

tests/BUILD.bazel

+            name = "rename_%s_%s" % (TESTS[i], j),
+            srcs = [TEST_GEN_FILES_IN[i][j]],
+            outs = [TEST_GEN_FILES_OUT[i][j]],
+            cmd = "cp $< $@",


Something here is not working on windows where this genrule is either not being generated or is not dropping the .test- from the path, I assume because of slashes.

Anyone see anything wrong here? Or have a windows box to debug it more?

I gave up and ignored this test on windows for now. I confirmed it is renaming the files but the .test-* files are still visible. I assume this is a windows+sandboxing issue?

walk/walk.go

walk/config.go

This reduces the noise a lot when adding more libraries such as in #1908 **What type of PR is this?** > Other **What package or component does this PR mostly affect?** > all **What does this PR do? Why is it needed?** **Which issues(s) does this PR fix?** Fixes # **Other notes for review**

DavidZbarsky-at · 2024-09-20T23:29:15Z

This looks great to me, @fmeum wdyt?

jbedard · 2024-09-21T01:55:51Z

walk/walk.go

+		entParts[len(entParts)-1] = base
+
+		// Skip git-ignored when the gitignore option is enabled.
+		if wc.isGitIgnored(entParts, ent.IsDir()) {


Note this is its own conditional and not embedded within isExcluded for a few reasons:

I don't think .gitignore can exclude the parent directory it is within the way # gazelle:exclude * can (that is the reason we must invoke isExcluded both above and within this for loop).

isGitIgnored requires the path as []string and needs to know IsDir(), I don't think we should override isExcluded with more params

… gazelle feature implementation (#6873) This aligns the implementation more with what I'm trying to merge into gazelle: bazel-contrib/bazel-gazelle#1908 There are no user facing changes here but the implementation is simpler and should improve performance for many reasons: * the .gitignore files are only `os.Open`ed once per directory (not per directory * per language) * the .gitignore files are only parsed once instead of once per gazelle language * the .gitignore patterns are only executed once because it is done within the (patched) gazelle walk * .gitignore-d directories are not recursed into at all because they are filtered in the gazelle fs walk * ... --- ### Changes are visible to end-users: no ### Test plan - Covered by existing test cases GitOrigin-RevId: 27fe8377ca89de91d3c95105ab37adea9347e6ab

fmeum · 2024-10-23T09:52:56Z

Based on aspect-build/aspect-cli#755, what do you expect to be the use case for this if we can't enable it by default due to performance issues? It's surprising to me that .gitignore processing should be costly - do you know what the hotspots are in the current implementation?

jbedard · 2024-10-23T19:29:07Z

I was suggesting it should be disabled by default because that is the ideal use case for both "correctness" and performance (although performance is only really a concern in very large projects globbing on hundreds of thousands of files).

However enabling it by default is probably a better initial experience for most projects so I'm fine with that as well.

Longer explanation, imo...

The ideal use case is when bazelignore specifies everything bazel should ignore, and gitignore specifies everything git should ignore. Those are not always 1-1... sometimes bazel might need to process things that are git ignored, very often things such as IDE config are in git but should be ignored by bazel etc.

In reality the more common use case is projects use gitignore and then (partially) duplicate that in glob([exclude]) + # gazelle:exclude, and bazelignore is almost unused. This is done either due to lack of awareness of bazelignore or the fact that bazelignore does not support globs so it is too much of a hassle or sometimes not even possible.

This feature would remove the need for many of those glob([exclude]) + # gazelle:exclude which I think makes this feature worth it for most projects. I essentially agree with everything in this comment: #1166 (comment)

Gitignore and generation-mode are both generic gazelle config and should require no code in plugins. Note that both of these are currently patched into gazelle while waiting for PRs which will allow the removal of all aspect code: bazel-contrib/bazel-gazelle#1921 bazel-contrib/bazel-gazelle#1908 --- ### Changes are visible to end-users: no ### Test plan - Covered by existing test cases GitOrigin-RevId: cdfe6690ff0ab2584e429419612cc32e94c6b379

jbedard · 2024-12-03T20:43:50Z

Rebased. @fmeum any updates with getting this reviewed/merged?

Close bazel-contrib#1166

jbedard force-pushed the gitignore branch from a568354 to d578c8b Compare September 4, 2024 01:36

jbedard mentioned this pull request Sep 4, 2024

Configure - Starlark Roadmap aspect-build/aspect-cli#740

Open

3 tasks

jbedard force-pushed the gitignore branch from d578c8b to cacc420 Compare September 6, 2024 00:21

dzbarsky reviewed Sep 6, 2024

View reviewed changes

walk/config.go Outdated Show resolved Hide resolved

dzbarsky reviewed Sep 6, 2024

View reviewed changes

walk/config.go Outdated Show resolved Hide resolved

dzbarsky reviewed Sep 6, 2024

View reviewed changes

walk/config.go Outdated Show resolved Hide resolved

jbedard force-pushed the gitignore branch 3 times, most recently from 7ed05a4 to 610ea22 Compare September 6, 2024 01:32

jbedard requested a review from dzbarsky September 6, 2024 01:36

jbedard mentioned this pull request Sep 6, 2024

chore: upgrade go mod dependencies #1911

Merged

jbedard force-pushed the gitignore branch 9 times, most recently from 698d1b1 to 52d7986 Compare September 6, 2024 08:53

jbedard commented Sep 6, 2024

View reviewed changes

jbedard force-pushed the gitignore branch from 52d7986 to b757d3e Compare September 6, 2024 09:55

dzbarsky reviewed Sep 6, 2024

View reviewed changes

walk/walk.go Outdated Show resolved Hide resolved

walk/walk.go Outdated Show resolved Hide resolved

walk/walk.go Outdated Show resolved Hide resolved

walk/config.go Outdated Show resolved Hide resolved

jbedard force-pushed the gitignore branch from ed09b76 to 0f82486 Compare September 16, 2024 07:58

jbedard force-pushed the gitignore branch from 0f82486 to fa4c3d6 Compare September 16, 2024 18:22

DavidZbarsky-at approved these changes Sep 20, 2024

View reviewed changes

jbedard force-pushed the gitignore branch 3 times, most recently from 55f9844 to cce54f9 Compare September 21, 2024 01:42

jbedard commented Sep 21, 2024

View reviewed changes

jbedard force-pushed the gitignore branch from cce54f9 to 200e7f4 Compare September 24, 2024 02:53

jbedard mentioned this pull request Oct 1, 2024

process: sync commits from Aspect-internal silo aspect-build/aspect-cli#749

Merged

jbedard mentioned this pull request Oct 10, 2024

[bug]: gazelle support for gitignore should be disabled by default aspect-build/aspect-cli#755

Open

jbedard force-pushed the gitignore branch from 200e7f4 to f620f24 Compare October 23, 2024 19:29

jbedard force-pushed the gitignore branch from f620f24 to 6db691c Compare December 3, 2024 20:42

feat: add support for .gitignore

961db30

Close bazel-contrib#1166

jbedard force-pushed the gitignore branch from 6db691c to 961db30 Compare December 17, 2024 00:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for .gitignore #1908

feat: add support for .gitignore #1908

jbedard commented Sep 4, 2024 •

edited

Loading

jbedard commented Sep 5, 2024

tyler-french commented Sep 5, 2024

jbedard commented Sep 5, 2024

jbedard commented Sep 5, 2024

fmeum commented Sep 5, 2024

jbedard commented Sep 5, 2024 •

edited

Loading

jbedard commented Sep 5, 2024

jbedard Sep 6, 2024

jbedard Sep 12, 2024

DavidZbarsky-at commented Sep 20, 2024

jbedard Sep 21, 2024

fmeum commented Oct 23, 2024

jbedard commented Oct 23, 2024

jbedard commented Dec 3, 2024

feat: add support for .gitignore #1908

Are you sure you want to change the base?

feat: add support for .gitignore #1908

Conversation

jbedard commented Sep 4, 2024 • edited Loading

jbedard commented Sep 5, 2024

tyler-french commented Sep 5, 2024

jbedard commented Sep 5, 2024

jbedard commented Sep 5, 2024

fmeum commented Sep 5, 2024

jbedard commented Sep 5, 2024 • edited Loading

jbedard commented Sep 5, 2024

jbedard Sep 6, 2024

Choose a reason for hiding this comment

jbedard Sep 12, 2024

Choose a reason for hiding this comment

DavidZbarsky-at commented Sep 20, 2024

jbedard Sep 21, 2024

Choose a reason for hiding this comment

fmeum commented Oct 23, 2024

jbedard commented Oct 23, 2024

jbedard commented Dec 3, 2024

jbedard commented Sep 4, 2024 •

edited

Loading

jbedard commented Sep 5, 2024 •

edited

Loading