PyInfo fields to aid static analysis (pyi files, original source files)

[rickeylev]

After seeing a few discussion about static analysis from several different places, I wanted to start a discussion about how the rules can better support such tools.

In particular, I'm interested in hearing from people implementing rules for e.g. mypy, other type checkers, or other tools, and what information would be useful for a py_* rule to provide.

The primary case I have in mind is type checkers. Strict dependency checking are a second case.

For a type checker, 3 pieces of information are needed:

• The original py source files.
• The pyi files
• Information to map a file to its module name.

For strict dependency checking, the same information is needed.

Right now...

The original py sources files aren't always directly available. While the sources are usually put in e.g PyInfo.transitive_sources et all, precompile settings may cause them to be omitted.

There are no fields to store pyi fields.

The PyInfo.imports attribute has enough information to map files to what their imported module name will become, so that part is covered.

So, I'm proposing adding 4 new fields to PyInfo:

• direct_original_sources and transitive_original_sources: these are depsets that always contain the original source files (e.g. srcs) regardless of e.g. precompile settings.
• direct_pyi_files, transitive_pyi_files: these contain the pyi files (or equivalent)

Within Google, we carry a patch for direct_original_sources to support dependency checking.

Within Google, for our Pytype-based type checking rules, those rules have a different way of getting equivalent information (this is for historical reasons). Something notable here is they don't use PyInfo.imports attribute for path mapping. Instead, a single File that tells how to map file paths to module names is used. I'm not sure what the pros/cons of this are compared to PyInfo.imports.

[mark-thm]

There are a few mypy rulesets out there, I created and maintain https://github.com/theoremlp/rules_mypy.

In terms of sources and typings:

• we end up getting local sources through the aspect’s access to the original rule’s attrs
• getting upstream typings demands loading runfiles from upstream deps, as there’s not a practical or consistent way to get typings (and, in other use cases dlls/so files)

Something I wish was stamped on external deps in PyInfo or some other provider is the canonical pypi requirement name — in rules_mypy we have a shoddy way of finding typing-extensions and mypy-extensions to exclude them because otherwise mypy will hard fail.

Being able to easily grab direct and transitive sources and pyi files would certainly make rule implementation easier.

It is not currently possible to put pyi files anywhere other than data in a py_* target, which additionally makes it difficult for code generators (like protobuf) to handle typings correctly.

[rickeylev]

Thanks for the feedback!

Being able to easily grab direct and transitive sources and pyi files would certainly make rule implementation easier.

Thanks. This provides some validation that the provider fields I proposed are a base line necessity. I'll clean up the PR that adds them.

hacks to identify typing-extensions/mypy-extensions to make mypy ignore them to prevent crashing

Is this rooted in the non-site-packages based layout of external dependencies?

In any case, yeah, attaching some sort of info like that to what comes from pypi is something we've talked about, but the details of what/how/etc are murky. (this has come up in the context of having a site-packages based layout instead of lots of directories on sys.path)

It is not currently possible to put pyi files anywhere other than data in a py_* target

Something I've gone back and forth on in my head is how to pass pyi files to e.g. py_library. Definitely not the data attribute -- that's intended for runtime data files. The two things I go between are:

1. A pyi_srcs attribute.
2. Allow .pyi in the regular srcs attribute. Under the hood, the rule logic would split them out and put them in the provider field.

I think I mostly prefer (1):

• It's familiar -- it's how our internal rules work
• They aren't "real, actual, runtime source code". So it seems wrong to put that in srcs.

Which reminds me: pyi_deps: I think this will be a necessity. Otherwise, if there are typing-only (build-time) dependencies, there's no way to express that without them also becoming part of the resulting output for runtime. I think this is further support for a pyi_srcs attribute.

https://github.com/theoremlp/rules_mypy implementation

I looked at the readme. Interesting approach!

separately published types package (from https://github.com/theoremlp/rules_mypy)

Thanks for reminding me of these. These might look a bit weird, but seem easy to model: just a py_library with pyi_srcs set and empty srcs. This seems like more support for a separate pyi_srcs attribute.

[aignas]

I like the pyi_srcs and pyi_deps idea. That said, I am curious about the pip.parse integration. Where would we add the type stubs to third party deps for the third party libs? In gazelle #2425 there is already some heuristics applied to connect things up.

[rickeylev]

Where would we add the type stubs to third party deps for the third party libs?

For libraries that come with pyi files, they'd be part of the same py_library the py files are in, and be automatically picked up.

For libraries that distribute a separate {name}-types package, then...hm. Well, then either of deps=["@pypi//foo_stubs:pkg"] or pyi_deps=["@pypi//foo_stubs:pkg"] would work. However, I'm not sure how we'd know when to use pyi_deps? So it'd probably just always be deps in the pypi-generated targets.