Error when building multiple latex documents

[marcel-22]

When I configure latex_documents[] in conf.py to output multiple documents, the following warning is generated and finally an exception occurs:

resolving references...
done
writing... WARNING: unknown node type: <NeedIncoming: <inline...>>
failed
Needs successfully exported

Exception occurred:
  File "C:\...\.venv\Lib\site-packages\docutils\nodes.py", line 2068, in unknown_departure
    raise NotImplementedError(
NotImplementedError: <class 'sphinx.writers.latex.LaTeXTranslator'> departing unknown node type: NeedIncoming

The latex_documents[] is setup to output two documents: 1) a document with the full output including all Sphinx pages, 2) a document with only a subset of the Sphinx pages.

latex_documents = [
    ('index', 'project.tex', 'documentation', author, 'manual', True),
    ('subset', 'project_subset.tex',  'subset', author, 'manual', True),
]

Both documents contain a page with the following sphinx-needs construct:

.. req:: Demo requirement
   :id: REQ_77bed70d
   :status: closed
   :collapse: False
   :tags: example

   Demo requirement.

.. req:: Link example
   :id: REQ_ae3d9643
   :status: closed
   :links: REQ_77bed70d
   :tags: example

   An example need with a link set to REQ_77bed70d.

When I delete above statements from the .rst file, the file being completely empty, the build succeeds without errors.
Also when I remove the second document with the partial output from latex_documents[], the build succeeds.

Traceback (most recent call last):
  File "C:\...\.venv\Lib\site-packages\sphinx\cmd\build.py", line 285, in build_main
    app.build(args.force_all, args.filenames)
  File "C:\...\.venv\Lib\site-packages\sphinx\application.py", line 353, in build
    self.builder.build_update()
  File "C:\...\.venv\Lib\site-packages\sphinx\builders\__init__.py", line 308, in build_update
    self.build(['__all__'], to_build)
  File "C:\...\.venv\Lib\site-packages\sphinx\builders\__init__.py", line 378, in build
    self.write(docnames, list(updated_docnames), method)
  File "C:\...\.venv\Lib\site-packages\sphinx\builders\latex\__init__.py", line 306, in write
    docwriter.write(doctree, destination)
  File "C:\...\.venv\Lib\site-packages\docutils\writers\__init__.py", line 78, in write
    self.translate()
  File "C:\...\.venv\Lib\site-packages\sphinx\writers\latex.py", line 88, in translate
    self.document.walkabout(visitor)
  File "C:\...\.venv\Lib\site-packages\docutils\nodes.py", line 199, in walkabout
    if child.walkabout(visitor):
       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\...\.venv\Lib\site-packages\docutils\nodes.py", line 199, in walkabout
    if child.walkabout(visitor):
       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\...\.venv\Lib\site-packages\docutils\nodes.py", line 199, in walkabout
    if child.walkabout(visitor):
       ^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 11 more times]
  File "C:\...\.venv\Lib\site-packages\docutils\nodes.py", line 212, in walkabout
    visitor.dispatch_departure(self)
  File "C:\...\.venv\Lib\site-packages\sphinx\util\docutils.py", line 601, in dispatch_departure
    super().dispatch_departure(node)
  File "C:\...\.venv\Lib\site-packages\docutils\nodes.py", line 2046, in dispatch_departure
    return method(node)
           ^^^^^^^^^^^^
  File "C:\...\.venv\Lib\site-packages\docutils\nodes.py", line 2068, in unknown_departure
    raise NotImplementedError(
NotImplementedError: <class 'sphinx.writers.latex.LaTeXTranslator'> departing unknown node type: NeedIncoming

[danwos]

Thanks for reporting the bug.
It looks like our routines for replacing Sphinx-Needs-specific nodes are not executed before the latex builder starts building the docs.

Does it work, when you do not use latex_documents, so creating a single document?

And what version of Sphinx-Needs are you using?

[marcel-22]

Yes, when I set latex_documents[] to output only one file (with the whole output), the build succeeds. I assume this would be identical to not using latex_documents[].

Sphinx version we use is currently 6.2.1.

[marcel-22]

Oh sorry, you asked for the Sphinx-Needs version. That is 1.3.0. With the previous version 1.2.2 we had the same error.

[staalb]

I'm having the same problem here. I think I've the code that causes this (without actually having any idea how the code works, got here by adding print statements and comparing output with a diff viewer).

The function process_caller() in sphinx_needs/needs.py returns early before doing any work at all if the current document is not the root document (or if it's in some kind of collection with 'needs' documents, which seems to always be empty in my case).

sphinx-needs/sphinx_needs/needs.py

Lines 278 to 283 in 150bcd6

	# We only need to analyse docs, which have Sphinx-Needs directives in it.
	if (
	fromdocname not in SphinxNeedsData(app.env).get_or_create_docs().get(doc_category, [])
	and fromdocname != f"{app.config.root_doc}"
	):
	return

If I just remove the if statement the problem is solved and everything works as expected. If I set latex_documents to only generate a single document, this branch is never taken at all.

[danwos]

Good finding.
But it's strange for me, why fromdocname not in SphinxNeedsData(app.env).get_or_create_docs().get(doc_category, []) seems to not take all docs into account, especially has this should not be builder specific?

@chrisjsewell : Do you maybe have an idea, what's going wrong here?

[chrisjsewell]

Do you maybe have an idea, what's going wrong here?

Not sure, I will have a look

[staalb]

Good finding. But it's strange for me, why fromdocname not in SphinxNeedsData(app.env).get_or_create_docs().get(doc_category, []) seems to not take all docs into account, especially has this should not be builder specific?

If I put a random sphinx-needs directive in the alternative index document, it is included in this collection. However, generating documentation from the normal document root fails with the error below. If I only generate the alternative document root the expected PDF document is generated without any errors.

Extension error (sphinx_needs.needs):
Handler <function process_creator.<locals>.process_caller at 0x7ffa7aac5b20> for event 'doctree-resolved' threw an exception (exception: ('index-alternative', None))

I catched the exception in process_caller to print the stack trace:

Traceback (most recent call last):
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx_needs/needs.py", line 299, in process_caller
    check_func(app, doctree, fromdocname, current_nodes[check_node])
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx_needs/debug.py", line 65, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx_needs/directives/needtable.py", line 244, in process_needtables
    row += row_col_maker(app, fromdocname, all_needs, temp_need, "id", make_ref=True, prefix=prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx_needs/utils.py", line 204, in row_col_maker
    ref_col["refuri"] = builder.get_relative_uri(fromdocname, need_info["docname"])
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/builders/latex/__init__.py", line 145, in get_relative_uri
    return self.get_target_uri(to, typ)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/builders/latex/__init__.py", line 140, in get_target_uri
    raise NoUri(docname, typ)
sphinx.errors.NoUri: ('index-alternative', None)

Then, if I include a reference to the alternative index in the normal document root index everything works! Except for the fact of course that the 'normal' generated documentation now contains two tables of contents.

[marcel-22]

I am sorry, I appreciate all the work everyone is doing, but is there a fix available? Right now I cannot generate multiple latex documents.

[marcel-22]

I have stripped my documentation files and done some testing.

The documentation files in my stripped project are:

docs/conf.py
docs/index.rst
docs/part.rst
docs/wow/requirements_documentation.rst

Both files index.rst and part.rst contain a toctree definition with wow/requirements_documentation.

Generating latex from index.rst works, but generating latex from part.rst fails.

I added two print statements before the statements pointed out by @staalb in the post above to see what data is in the variables.

The output was

DEBUG: fromdocname=part
DEBUG: SphinxNeedsData=[]
DEBUG: fromdocname=part
DEBUG: SphinxNeedsData=['wow/requirements_documentation']

The function process_caller() is called twice apparently. SphinxNeedsData contains a list of documents with sphinx-needs directives in the second call only.

I hope this helps. Please let me know if I can assist.

[marcel-22]

Hi, any updates on this issue? I desperately need to generate multiple PDF documents.

[danwos]

Hi all, sorry for the huge delay in the communication.

Is there a chance to get a test case from someone of you, which reproduces the problem?
I'm personally not working with latex and am also not familiar with how to set up a multi-document build for it.

This would also help us to test related PR #1208.

[marcel-22]

Hi,

I have stripped my documentation files even more now, so you can reproduce the problem. I flattened the directory structure as well. So all files can be recreated in the root of docs:

conf.py

project                 = 'test'
author                  = 'author'
extensions              = ['sphinx_needs']
templates_path          = ['_templates']
needs_id_regex          = '^[A-Z]{3,4}_[a-z0-9]{8}$'
latex_engine            = 'pdflatex'
latex_documents         = [
    ('index', 'main.tex', 'Main', author, 'manual', True),
    ('part',  'part.tex', 'Part', author, 'manual', True)
]

index.rst

Documentation
*************

Contents
========

.. toctree::
   :maxdepth: 3
   :caption: Contents

.. toctree::
   :maxdepth: 2

   requirements

part.rst

:orphan:

Part
****

.. toctree::
   :maxdepth: 3

   requirements

requirements.rst

Page
****

Section
=======

.. req:: Demo requirement
   :id: REQ_77bed70d

   Demo requirement.

.. req:: Link example
   :id: REQ_ae3d9643
   :links: REQ_77bed70d

   Example link.

I believe the error occurs during writing of the LaTeX output and before conversion from LaTeX to PDF. So, the following command should be sufficient to build the documention: sphinx-build -M latex docs docs/_build -W --keep-going -E

Note: I you disable the second document (part) in latex_documents[] you should notice the build succeeds.