Render pages concurrently

[naourass]

Is it currently possible to render pdf pages concurrently instead of a for loop? Something like:

pdf_pages = pdfium.PdfDocument("file.pdf")
with multiprocessing.Pool(multiprocessing.cpu_count()) as pool:
    pool.map(myRenderFunc, pdf_pages)

[mara004]

Yeah it is. Take a look at src/pypdfium2/_cli/render.py for a sample implementation. Key notes:

• pdfium/ctypes object handles can't be transferred across processes; instead you have to create them in workers. (i.e. pass filepath/input and page indices instead of PdfDocument or PdfPage objects)
• Use a multiprocessing initializer and a global variable to set up process-scope objects (e.g. the PdfDocument).
• Do all post-processing/saving (or whatever uses the bitmap) in workers. Do NOT transfer bitmaps back to the main process (this was a historical fault in pypdfium2).
• Do not use the fork multiprocessing strategy, it has unpredictable stability issues. Use spawn or maybe forkserver instead.
• For short documents (e.g. <4 pages), a linear loop may be more efficient, because subprocess creation causes some overhead.
• Be aware that rendering with N parallel jobs basically causes N times the memory load.

[naourass]

Is something like this okay?

def render_page(page_index, input_file, render_kwargs):
    pdf = pdfium.PdfDocument(input_file)
    page = pdf.get_page(page_index)
    bitmap = page.render(**render_kwargs)
    # do whatever you want with the image

def worker_initializer(input_file, render_kwargs):
    global worker_data
    worker_data = {
        "input_file": input_file,
        "render_kwargs": render_kwargs
    }

def worker_task(page_index):
    input_file = worker_data["input_file"]
    render_kwargs = worker_data["render_kwargs"]
    render_page(page_index, input_file, render_kwargs)

def main():
    input_file = "file.pdf"
    pdf = pdfium.PdfDocument(input_file)
    num_pages = len(pdf)
    
    render_kwargs = {
        "scale": 1,
        "grayscale": True,
        # ...
    }

    with multiprocessing.Pool(processes=4, initializer=worker_initializer, initargs=(input_file, render_kwargs)) as pool:
        pool.map(worker_task, range(num_pages))

[mara004]

Seems OK, basically. A few possible enhancements:

• Instead of the input filepath, you can directly put the PdfDocument into the worker_data, to set it up only once per process, not per job.
• You should not create more processes than there are pages, to avoid overhead from unnecessary process creation, so consider something like min(num_pages, 4), and maybe os.cpu_count() instead of static 4. Further, single-page documents can be rendered directly without process pool (also avoids overhead).
• You should yet make sure the pool uses spawn, not fork, e.g. by calling set_start_method().