embed markdown tables automatically generated

[gpagliuca]

I would like to embed tables and plots generated on the fly in markdown cells based on some (hidden) initialization cells.

This is possible in the jupyter notebook thanks to this extension python-markdown.

According to the extension documentation it is possible to have the same behavior exporting via nbconvert using a pre-processor (here the preprocessor source code) :

Exporting
In order to have nbconvert show the computed Python output when exporting to another format, use the pre_pymarkdown.py preprocessor. If you used the python setup.py install command to install the IPython-contrib extension package, this will already be installed.
For manual setup, you need to copy this file to a location within the Python path (or extend PYTHONPATH). Additionally, you need to add these two lines to your jupyter_nbconvert_config.py configuration file:
c = get_config()
c.Exporter.preprocessors = ['pre_pymarkdown.PyMarkdownPreprocessor']

This works when I manually convert a notebook to html using nbconvert, but not when I try to build the doc with sphinx.

Any suggestion? Is this the best approach? (related in end to issue #15)

Thanks ;-)

[mgeier]

For now, nbsphinx doesn't support adding custom preprocessors, but it shouldn't be too hard to implement this.

Would you like to make a PR for this?

[gpagliuca]

I am more a user... but I'll try.

Thanks for the answer.

[mgeier]

If you need help, don't hesitate to ask!

[gpagliuca]

Created a pull request with the hide_input option as cell metadata. It works but it is ugly.

I need your help for the pre-processor instead.
The pre-processor I'd like to add extends the nbconvert Preprocessor class and it is automatically called when using nbconvert with the jupyter nbextensions installed.

Now, in nbsphinx, if I have well understood, the only preprocessor used is HighlightMagicsPreprocessor in the Exporter class but if I simply try to add there PyMarkdownPreprocessor I see no change or error.

[mgeier]

AFAIK, the thing with the HighlightMagicsPreprocessor only works because it is in the list of default preprocessors:
https://github.com/jupyter/nbconvert/blob/d7c996c87db98c2bb80e72a0c22511a802064bc4/nbconvert/exporters/exporter.py#L70

A few lines above you can see the preprocessors attribute, maybe you can use this?

Is your PyMarkdownPreprocessor supposed to be run before or after the ExecutePreprocessor?
I'm invoking the ExecutePreprocessor manually (and conditionally) in nbsphinx. If your preprocessor has to run after that, I think the preprocessors attribute won't work, because those will be involked before nbsphinx.Exporter.from_notebook_node()is even called.

[keluc]

Has any one had success with this preprocessor yet? Tried it but without any success.

[krassowski]

It would be so awesome to be able to embed variables generated in the notebook within the Markdown cells for nbsphinx - this would help me a lot in documenting jupyterlab-lsp, and maybe it could be useful to jupyterlab in general in the future. I expect that the things changed in the last 4 years and maybe it would be worth to have a fresh look a this. Would you think that using a preprocessor is the best option, or would it be better to export the variables to the RST and display them using substitutions? Or would it be too difficult due to any potential escaping?

[krassowski]

So here is what I came up with: I write my markdown in a code cell but prepend it with %%markdown (which gives me linting of markdown as a bonus when using jupyterlab-lsp); however instead of using the default IPython markdown magic I redefine it as:

from IPython.display import Markdown
from IPython.core.magic import register_cell_magic


@register_cell_magic
def markdown(line, cell):
    return Markdown(cell.format(**globals()))

Then I hide the cell source in nbsphinx by setting {"hide_input": true} in cell metadata.

I find this solution very nice because I can provide installation instructions such as:

[mgeier]

Instead of going the Markdown detour, you can also simply use text output:

print(f"""\
pip install jupyter-lsp={JUPYTER_LSP_VERSION}
jupyter labextension install @krassowski/jupyterlab-lsp@{JUPYTERLAB_LSP_VERSION}
""")

Would you think that using a preprocessor is the best option,

I don't know whether it's the best option, but I still think it's worth trying.

or would it be better to export the variables to the RST and display them using substitutions?

I see two problems:

• nbsphinx internally using a reST representation is just an implementation detail which will be changed in the future (Convert directly to docutils without intermediate RST #36).
• the parsing of reST happens after the whole notebook is executed, so if a variable is overwritten during notebook execution, it will always be displayed with its final value, even if it is mentioned much earlier in the notebook.

[actsasgeek]

Did anything ever come of this? I would love to be able to do this and would be willing to help.

My main problem with any of the proposed alternatives is that--unless you can hide that specific code cell--you end up reading everything twice:

In [1]

answer = 42
print(f"This is the answer to life, universe, everything: {answer}")

Out[1]
This is the answer to life, the universe, everything: 42

unless you can hide the input code (but only hide that input code and not the input code for anything else you produce). Additionally, the output is in a different font than the rest of the text, which can be a disconcerting context switch.

I looked through the documentation for pre-processing and didn't see anything so I assume this wasn't solved but I'm not sure.

[krassowski]

I think that hide_input in metadata for specific cell does exactly that (see my comment above) - unless I misunderstood.

[actsasgeek]

@krassowski Thank you for your quick response.

I still have the qualm that, at least in the notebook itself, you have to read "it" twice...the code to generate it and the generated output. I provide both the notebooks and the PDF to my students.

OTOH, I do like the idea of something that works with Jupyter Lab as well as Jupyter Notebook.

The main issue I'm having right now is that {"hide_input": true} doesn't do anything (although "nbsphinx": "hidden" does but that hides both). I can't find documentation for just "hide_input" and looking through the issues, it's not clear how the whole "hide input" thing was ever resolved. Additionally, this is for PDF not HTML. (nbsphinx 0.7.1)

[mgeier]

I can't find documentation for just "hide_input" and looking through the issues, it's not clear how the whole "hide input" thing was ever resolved.

It wasn't.

There are several issues where this has been discussed and there are multiple open pull requests: #436, #185, #86.

If you want this to progress, please chime in!