Make intersphinx (a.k.a. external references) more user friendly

[chrisjsewell]

Note I've already made a change in this direction (#12133) and plan some more, but I though it would be helpful to open a "meta" discussion to outline my thinking.

There is also a disclaimer, that I have already implemented some of the thing discussed below in MyST 😅 (https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html#cross-project-intersphinx-links)

Edit: see also https://github.com/orgs/sphinx-doc/discussions/12204, for a more specific discussion on changing the current objects.inv format

---

I think intersphinx is a great feature of Sphinx, and a key selling point:

That each built sphinx HTML site provides a centralised registry of references, that can be utilised in a more robust/concise way than directly adding URL references to documentation.

That said, I think it could be a bit easier for the user to understand / use...

Desirable (simple) user workflow

At the very simplest, each objects.inv (generated in an HTML build) provides a unique mapping from domain name -> object type -> target name -> relative URL (plus also an optional display name a.k.a implicit reference text).

Combine this with the intersphinx_mapping configuration, and you get a unique mapping from inv key -> domain name -> object type -> target name -> absolute URL

As a user, I feel the "minimal" work flow should be:

1. Have a sphinx CLI I can call, that reads the intersphinx_mapping then outputs a "human readable" file with all available mappings.

2. Pick something I want to reference.

3. Have a sphinx "syntax" that I can use, to specify a (inv key, domain name, object type, target name) "key set" that I want to reference, together with an optional "explicit display name".

4. Have sphinx check my "key set" is valid and, if so, render output the absolute URL, with the explicit or implicit display name (and warn if neither is available).

for brevity / flexibility, one might also want the syntax to allow for one or more of the (inv key, domain name, object type) to be omitted, and have sphinx check that there is exactly one match for the remaining keys.

Current sphinx implementation (and its problems)

For (1), this is semi-implemented with: python -m sphinx.ext.intersphinx <inv file/url>.
This is a good start, but:

1. I feel it would be nicer if there was a CLI that could just be pointed at the conf.py and output all mappings (possibly with some "filtering" options).
2. The current output format is a little "bespoke", outputing as something like YAML might be nicer.

For (3), you now have the external role;

• you can use full key sets, e.g. :external+inv_name:domain:type`compile`
• you can use partial key sets, e.g. :external:type:`compile`
• you can declare an explicit display name, e.g. :external:type`explicit display name <compile>`

BUT this is where it gets a bit problematic:

what the documentation does not tell you is that the external role expects a role name INSTEAD of an object type,
these are two "inter-linked" but distinct concepts.

The first problem with requiring role names (fixed in #12133), is that they are not always the same as the object type name, for example in the sphinx objects.inv, you have:

py:function
    compile                  : usage/domains/python.html#compile

Previous to #12133, :external:py:function:`compile` would fail because function is the object type, but func it the role name,
there is currently no obvious way for a user to know this.

The second problem (not yet fixed) is that an objects.inv may contain (domain name, object type) that do not have a corresponding role name, available in your local sphinx project.
For example in the sphinx objects.inv, you have:

std:confval
    add_function_parentheses     : usage/configuration.html#confval-add_function_parentheses

But, if you try to use :external:std:confval:`add_function_parentheses` in your local project, it will likely fail because confval is a bespoke object type created in sphinx/doc/conf.py::setup.
Again, it would not be obvious to the user why this is failing, and how they could fix it.
Fixing would anyway basically entail you have to ensure you have all the same extensions / bespoke setup as the project you are referencing, which is definitely not ideal.

Another issue related to (4), is that intersphinx reference resolution does not emit any warnings if an external reference has multiple matches, when the inv_key is not specified, it silently choses one (see use of main_inventory in intersphinx.py).
I don't see the justification for this.

So why does external use role names and not object types?

Well there is some advantages here, internally external actually locates/calls the role to generate the "reference" nodes.

1. This allows for "special features" of that role to be utilised

   For example, with py:meth you can prepend the target with . and/or ~ to implement partial target matching (see bottom of https://www.sphinx-doc.org/en/master/usage/domains/python.html)

2. This allows for "special formatting" of that role to be utilised

   For example, for :external:std:ref:`a` creates:

   <pending_xref refdoc="index" refdomain="std" refexplicit="False" reftarget="a" reftype="ref" refwarn="True" intersphinx="True" inventory="None">
       <inline classes="xref std std-ref">
           a

   whereas :external:std:keyword:`a` creates:

   <pending_xref refdoc="index" refdomain="std" refexplicit="False" reftarget="a" reftype="keyword" refwarn="True" intersphinx="True" inventory="None">
       <literal classes="xref std std-keyword">
           a

   (note one uses inline and the other uses literal for the inner node)

So yes, I understand why it uses role names, but I think this also can lead to issues and user confusion.

Another consequence to note, is that you can see above that external ends up creating pending_xref nodes,
which then need to be resolved in a later (post-processing) stage.
Unless I'm missing something, there is really no need to "postpone" the resolution of these references, since we already have all the information we need to resolve them (i.e. loaded inventories).

Proposed solutions

1. An extended CLI could be implemented, that reads the intersphinx_mapping from conf.py then outputs a "human readable" file with all available mappings (possibly with some "filtering" options).

2. The output from the CLI could additionally output what role names are available for each domain/object type in the local project (if any).

3. The external role should fallback to matching against the object type, if a role name is not available, with a "standard" reference format.
   A warning can be emitted for such fallbacks, but with a type/subtype that can be suppressed if desired.

4. The external role should emit a warning if an external reference has multiple matches, e.g. when the inv_key is not specified.

5. A related outstanding issue is that currently the math domain does not output anything in get_objects (used to create objects.inv), and so its not possible to reference any labelled math equations using external.

Additional thoughts 1 (user generated inventories)

Lets say you want to make lots of reference to a website that was not written in sphinx (boo) and so does not have an objects.inv.
It might be nice to allow users to create their own objects.inv (without having to look into the depths of sphinx, on how to create the binary file format).

We could either offer a CLI tool to do this, e.g. converting a YAML file to an objects.inv, or you could allow intersphinx to directly read a .yaml file.

Additional thoughts 2 (internal references)

If you think about it, internal references are generally just a special case of external reference.
It would be nice to give users a quick overview of all the internal references in their project, and how they can be referenced.
At present this is a bit convoluted:

1. you need to run the HTML builder to generate the objects.inv for your project
2. then you need to convert the generated objects.inv to a human-readable format
3. then (as discussed above) you need to understand how the object types relate to the role names

It would be nice if there was a builder, that just did all this in one.

The ideal output would also could be slightly different to external:

• rather than the URL path, really it would be nice just to have the file path to the local source document (see InventoryFile.dump where the docname is converted to the URI)
• ideally you would also get the source line number (if available) for the target, although this would require an "extended" Domain.get_objects method (perhaps Domain.get_objects_with_line) for domains to implement. a bit of a pain but achievable in theory.

This could also be useful for Language Service Providers to provide features like auto-completions and "jump to references/target"

[chrisjsewell]

Ok thats my ten cents, happy to hear feedback.

cc @picnixz @danieleades @jayaddison, as people I know actively working on sphinx,
also @jakobandersen who I know has worked on this before

[picnixz]

I'd be happy to have a CLI indeed. Especially when I want to know which ID I need to use when referring to sections for instance or whether something is generated as a class, data or anything. For instance, I'd be happy to write something like

intersphinx find 're.Match' --short

and get something like :class:`~re.Match` for instance. Without --short I would have re.Match. Or for something like

intersphinx find 'pkg.foo.bar.Nya' --namespace 'foo'

I would get, for isntance :class:`foo.bar.Nya <pkg.foo.bar.Nya>`. This would help me a lot for writing references. Sometimes just the name of the object is not sufficient but sometimes the fullname is too much so being able to indicate where the title should 'start' is also good. The name of the options need to be carefully chosen in the future though.

I would also expect the tool to be able to output something that is as short as possible in terms of characters if people are concerned about docstrings for instance. I agree also that we could have filters like --type function or --type func (here 'type' could be either a role name or its corresponding object type name) or --all to show all possible matches that are ordered by projects and within the project by similarity. We could also support regexes when searching for some references.

--

I feel it would be nicer if there was a CLI that could just be pointed at the conf.py and output all mappings (possibly with some "filtering" options).

Agreed.

The current output format is a little "bespoke", outputing as something like YAML might be nicer.

I prefer JSON over yml because it's in the library and easy to handle with jq externally (also YAML is not supported natively and needs an additional dependency).

Unless I'm missing something, there is really no need to "postpone" the resolution of these references, since we already have all the information we need to resolve them (i.e. loaded inventories).

No, because extensions can add additional information if needed or process them differently. I personally don't want to change that logic.

A related outstanding issue is that currently the math domain does not output anything in get_objects (used to create objects.inv), and so its not possible to reference any labelled math equations using external.

I'm not sure whether it's the same for images. But referencing labelled equations would be indeed good.

[jakobandersen]

Regarding the external role, based on my immediate recollection of our discussions:
The idea behind using roles was that users should not be too concerned with object types and the format of the inventories, rather they should write as if they were referring to entities in a local document. That is, if MyModule.MyClass is local, then :py:class:`MyModule.MyClass` , would work, and if it was external, then it would still work because intersphinx acts as backup resolver (which is another option for point (3) mentioned above).
The external role was thought of as a replacement for an icky syntax (that I don't recall the details of right now), which were also role based. The idea was that if one was sure this should be an external reference, then "just" sort-of prepend :external: and if would do the same.
Thus, the idea was not that people should inspect the inventories manually to figure out the reference to write, as the external document should hopefully be written in a manner that it is clear how to refer to the entities.
An additional point that we tried to address back then was to decouple intersphinx from how Python works, so support for other languages could be improved. This is why that the lookup logic was partially moved to the domains at roughly the same time. The thought was that intersphinx should only delegate work to the domains, but not contain core logic in it self. (There is a huge chunk of work still to be done in this area as intersphinx still is in the middle of writing and reading domain-specific content.)
As @picnixz noted, each domain can/should render reference how they see fit. E.g., the C and C++ domain does not just create a single link, but a link for each name, e.g., MyClass<std::vector<MyElement>> potentially has a link for each of MyClass, std, vector, and MyElements.

In regards to the proposed steps above:

1. (CLI for inspecting inventories) Sounds good to me, assuming point 2. This also would help address the use-case where the inventory is in fact not written by Sphinx (though as noted, writing them by hand is not easy)
2. (That inspection showing roles) Very good idea, I think it makes things much more complicated if object types should be a user-facing thing as well.
3. (external falling back to matching object types) I recommend not doing this, to keep object types away from users.
4. (external emitting warnings on duplicates) This is not just for external but also the implicit intersphinx lookups. This is something I think the domains should be responsible for. Only the domains knows how each language works and how lookups should work. E.g., in C++ you can have one external project with a primary class template, and then several other projects with each their set of (partial) specializations. The lookup should be in a unified symbol tree of all the external projects.
5. (deficiencies in the math domain) This seems orthogonal to the primary issue.

[picnixz]

(We included point (3) in master so maybe we should have waited for this discussion to be done)

@chrisjsewell Do you think we should revert the role ~ object name approach?

[chrisjsewell]

thanks for the response @jakobandersen

based on my immediate recollection of our discussions
that I don't recall the details of right now

well... this hits at a larger issue I have in mind to discuss/address, whereby it would really be ideal for at least some key aspects of sphinx to have some form of referencable "design" documentation
Otherwise, you just go round in circles with people questioning why things are as they are 😅
Obviously with some balance, that you don't want to introduce to much unnecessary red-tape for developers

I would note, I have already tried to introduce something very similar for MyST: https://mep.mystmd.org/en/latest/meps/mep-0002/

the external document should hopefully be written in a manner that it is clear how to refer to the entities.

I feel this is not particularly user friendly 😬:

1. it assumes the user actually has access to the raw/source external documentation, which is not always the case
2. It forces users to have to start searching through source documentation for "targets" and trying to decipher what roles they would relate to, which is not always obvious. I've had to do this personally, it is not fun 😅

As I've mentioned before, an example would be https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-suppress_warnings, which I wanted to reference in the MyST documentation.
Currently, there is no clear way as a user to work out how to reference this through intersphinx, even if you work out how to get to the source documentation, you see:

.. confval:: suppress_warnings

   A list of warning types to suppress arbitrary warning messages.

which is still not really clear (a) if it is referencable at all, (b) how to reference it specifically,
then even if you try :external:confval:`suppress_warnings`, it just gives you the warning: WARNING: role for external cross-reference not found: confval, which is no help

I recommend not doing this, to keep object types away from users.

I can understand you're view point here 👍 , but ...
As I've outlined initially, some object types just do not have a complimentary role available,
particularly for things in the inventory that come from extensions that the remote project used but the local project does not.

I don't feel it is user friendly, to just tell users that they cannot reference something that is clearly in the objects.inv.
What this ends up doing, is just pushing users away from using intersphinx, and adding direct URL (again, this has happened to me)

Whether its involves exposing users to object types or not, I still feel there should be a way to reference anything in the objects.inv within the intersphinx world (even if there is a "suppresable" warning)

Actually, one additional problem of relying on "local" roles that I did not mention before, is that currently the objects.inv give no information on the "environment" where it was built (e.g. what version of sphinx was used, and what extensions, etc),
and you are largely working on faith, that the local sphinx build will work the same as the remote build worked.
(here the "local" intersphinx does introduce some hard-coded translations between versions, for example std.cmdoption -> std:option)

(CLI for inspecting inventories) Sounds good to me,
(That inspection showing roles) Very good idea

great, good we agree 😄

Do you think we should revert the role ~ object name approach?

@picnixz Personally no, at present I don't think it needs reverting; it addresses a key user issue, without changing any of the existing logic (just providing a sensible fallback).
In the coming weeks I may alter it though, perhaps at least to emit a warning that it is falling back

[chrisjsewell]

intersphinx find 're.Match' --short ... and get something like :class:`~re.Match`

@picnixz the problem with this, is that there is currently no "declarative nature" of domains/roles,
that allows you to know which roles allow for this ~ syntax.
this is generally specific to domains that are focussed on codebase APIs

In fact, you are already taking on faith, that all "cross-reference" roles, have the standard syntax: :name:`text <target>`,
since strictly speaking they could interpret the content text <target> however they wanted

[picnixz]

Ah yes, but it was something that each domain would actually specify. I mean, we could have different output depending on whether the role is fine or not (though it might be a complex CLI at the end... and no more user-friendly)

[chrisjsewell]

Ah yes, but it was something that each domain would actually specify. I mean, we could have different output depending on whether the role is fine or not (though it might be a complex CLI at the end... and no more user-friendly)

That would be the dream 😄 but yeh not trivial to implement

[jakobandersen]

based on my immediate recollection of our discussions
that I don't recall the details of right now

My phrasing might not have been too clear, it was the old syntax which was icky. I think it was something like :roleName:`invName: normalRefStuff` . This would be sent to the domains to be parsed, which some couldn't which gave warnings, before intersphinx would interpret it. The external role was to be explicit that now you are asking intersphinx directly first, and it can then delegate to the domain.
But your point of having a design document is indeed good, but very probably rather difficult to maintain.

the external document should hopefully be written in a manner that it is clear how to refer to the entities.

I feel this is not particularly user friendly 😬:

I agree that it probably can't be the only tool.

Currently, there is no clear way as a user to work out how to reference [confval] through intersphinx,

The idea was that one should write the same as if the target is in the same document. If there is no role to cross-reference a confval, then I think that is the missing piece for this case.

I recommend not doing this, to keep object types away from users.

I can understand you're view point here 👍 , but ... As I've outlined initially, some object types just do not have a complimentary role available,

Then I suggest those roles to be added. In general, if an entity can be declared with some directive, then there should be a role to cross-reference it.

particularly for things in the inventory that come from extensions that the remote project used but the local project does not.

Due to domains being the only ones really knowing how to do lookups, I don't see a reasonable way in general to avoid that users most have those extensions as well to be able to cross-reference external projects.

I don't feel it is user friendly, to just tell users that they cannot reference something that is clearly in the objects.inv. What this ends up doing, is just pushing users away from using intersphinx, and adding direct URL (again, this has happened to me)

If there is better tooling to inspect inventories, including role information, wouldn't that help?

Actually, one additional problem of relying on "local" roles that I did not mention before, is that currently the objects.inv give no information on the "environment" where it was built (e.g. what version of sphinx was used, and what extensions, etc), and you are largely working on faith, that the local sphinx build will work the same as the remote build worked. (here the "local" intersphinx does introduce some hard-coded translations between versions, for example std.cmdoption -> std:option)

Part of the plan for generalizing intersphinx was to let domains write blobs with their objects, including versioning information. I think modernizing the inventory to include necessary information is the best first step for solving this.

Do you think we should revert the role ~ object name approach?

@picnixz Personally no, at present I don't think it needs reverting; it addresses a key user issue, without changing any of the existing logic (just providing a sensible fallback). In the coming weeks I may alter it though, perhaps at least to emit a warning that it is falling back

While it may address the problem, it seems to me that it complicates intersphinx, and makes it harder to generalize. Once there is a release with this, it becomes very hard to change without breaking projects.

[chrisjsewell]

But your point of having a design document is indeed good, but very probably rather difficult to maintain

well I'd suggest they do not need to be maintained at all 😄, because they should essentially be locked after they are written, i.e. the intent is to record the design decisions at one point in time, rather than be an evolving document (as per Python PEPs etc)

The idea was that one should write the same as if the target is in the same document.

This certainly makes sense and I like the way it works in general,
but I feel its moreso focused on writing programming package APIs, than for other uses of referencing 🤔

Naturally you would expect to have a unique references to fully qualified program objects (like :c:type:`a.b.c`), and don't want to have to care if they are located locally or remote,
and you would also expect to have the sphinx domain for your programming language installed locally.

For other forms of references, like the std and math domain, I feel this is less the case, you are more likely to want to specify a certain external source, and reference clashes are a lot more likely (indeed we already make a special case with intersphinx_disabled_reftypes = ['std:doc'])

Due to domains being the only ones really knowing how to do lookups, I don't see a reasonable way in general to avoid that users must have those extensions as well to be able to cross-reference external projects.

I just really feel this unecessarily restricts intersphinx;
for many use case, users will be absolutely fine just having something that simply creates a "standard reference" from a lookup key-set (which is simplistically what objects.inv is).
Users will certainly not want to be installing extension just to work around intersphinx (python dependency hell is already bad enough 😅)
and indeed this may not even be possible, since the extensions may not be publically available, and then you'd need start mocking extensions, which is even worse

A fallback for unknown/uninstalled domains/roles is a pragmatic approach IMO (which can have a supressable warning)

Part of the plan for generalizing intersphinx was to let domains write blobs with their objects, including versioning information. I think modernizing the inventory to include necessary information is the best first step for solving this.

Yeh I think this is reasonable 👍,
although there is probably no amount of data that will allow one to fully characterise the environment in which the remote documentation was built 😬

[jdillard]

In addition to the CLI, I'd also love to see the ability to add a page to projects (default could be on or off) that shows all the references available, something similar to intersphinx-untangled that @webknjaz created. hrm, I suppose that could be converted to an extension first 🤔 to prove things out, but I would love for it to be more of a default to promote cross-linking out in the wild.

I also created https://github.com/orgs/sphinx-doc/discussions/12151, coincidentally in tandem, and it's mostly covered here, so I closed that and moved it here. In summary, it would be nice to be able to use intersphinx to cross-reference confval etc without having to modify the conf.py (the main addition to the discussion being the example of how to get references to confval working):

def setup(app):
    app.add_object_type('confval', 'confval',
                        objname='Sphinx configuration value',
                        indextemplate='pair: %s; Sphinx configuration value')

(edit: as pointed out below, this code originally likely comes from #5562)

Semi-related, I think the confval should also be usable by default without having to modify the conf.py. Main use case is that I use confval to document the configuration values available to my extension and don't think that should be reserved for core Sphinx configuration values.

[bskinn]

Hey, @chrisjsewell -- sphobjinv author here.

FWIW, sphobjinv returning the directive form was an intentional design choice: providing the reference as it's stored in the objects.inv, without attempting any role conversion.

After doing quite a bit of research (that I think was similar to what's been discussed above), it became pretty clear that trying to translate from the directive form stored in objects.inv to an appropriate role form was going to be a protracted exercise in edge-case wrangling that I did not have the time or motivation to tackle. IIRC, I was also stymied by the lack of Sphinx version information stored in the inventory, as discussed above. And, if I understand correctly, emitting the correct role syntax would require knowledge of the Sphinx version in which that cross-reference was intended to be used, not only the version used to create the inventory.

So, ultimately, I decided to give users the raw data, even though it requires some interpretation. I figured better that than transforming it in some way that made it actively wrong for some users.

There's an open ticket about a possible angle to output role reST instead, bskinn/sphobjinv#234. Neither I nor OP have had a chance to chase it yet, though, and I'm still not convinced it's actually feasible without knowing the version of Sphinx that the reST is going to be used in. (Plus, there are other features/fixes that I personally would give higher priority in the limited time I have available to dedicate to it).

I expect this would be one significant advantage to having this functionality integrated directly into Sphinx: when you run a Sphinx tool, you know what version of Sphinx you're using. I appreciated it a lot when @AA-Turner reached out last year about pulling sphobjinv into the sphinx-doc org for more direct interface/integration into Sphinx, and I think there would be a lot of advantages. However, I stand by what I said in turning the offer down, in that I know that some of the sphobjinv userbase is using it outside the context of a Sphinx install, and I don't want those users to have to take on the entirety of Sphinx as a dependency to use it.

---

More generally, FWIW, I've been frustrated in my general use of Sphinx by the significant role collapse that Sphinx performs in generating the objects.inv file -- there are some targets that I simply have not been able to figure out how to properly cross-reference because of this collapse. I can't remember offhand what they are, it's been a while, but I think one of them was something to do with cmdoption. (Perhaps it's that very std:cmdoption -> std:option mentioned above!)

I've scanned the discussion in here, and part of me would love to see an objects.inv syntax v3 that fixes and smooths out a bunch of things, and makes. But, I recognize that the requirements and mechanics here are incredibly complicated, and achieving a net improvement will be a lot of work, and not guaranteed.

One thing that would make such a project easier, I think, is that the imperative for small objects.inv filesize that was part of the v2 inventory design is no longer nearly as strong. A v3 objects.inv should be compressed, yes... but maybe it's just gzipped JSON or TOML or something.

[chrisjsewell]

thanks for the feedback @bskinn,
and just to emphasise I mentioned sphobjinv to highlight the deficiency in sphinx, not to be critical of the tool itself 😅

[bskinn]

<grin>, no worries -- while there was a bit of "Hey! That was intentional, I didn't misunderstand!!1!" in there, primarily I went into all of it in the hope that the thought process was a useful contribution.

[chrisjsewell]

Oh indeed, very helpful 👍

FYI, you might want to look at #12190

[bskinn]

👀

Ooooh, ✨ indeed, most excellent!!