Strategy to mimify HTML without compromising structure ?

[waltoss]

Everytime I want to fetch a page and send it to the LLM it exploses the token limit

As example I would like to pass documentation of this documentation page to generate a JSON schema. HTML to text or HTML to markdown breaks the table structure so LLM doesn't understand properly the documentation of fields

What strategy do you suggest to minify the HTML without compromising structure ?

[pelikhan]

You can try this one https://microsoft.github.io/genaiscript/reference/scripts/html/#converttablestojson

Or even use the playwright integration to load the page ina browser and "surgically" extract the tables you need.

https://microsoft.github.io/genaiscript/reference/scripts/browser/#_top

The default max token is 4000 because it is so easy for a tool to return a massive result and overflow the context window but it could be bumped up so that if you use a long context model, you get the full output.

[waltoss]

I suggest adding https://github.com/mixmark-io/turndown-plugin-gfm to add tables directly to the markdown converter, what do you think ?

export async function HTMLToMarkdown(
    html: string,
    options?: TraceOptions
): Promise<string> {
    if (!html) return html // Return original content if no HTML is provided
    const { trace } = options || {} // Extract trace for logging if available

    try {
        const Turndown = (await import("turndown")).default // Import Turndown library for HTML to Markdown conversion
        const res = new Turndown().turndown(html) // Use Turndown library to convert HTML to Markdown
        return res
    } catch (e) {
        trace?.error("HTML conversion failed", e) // Log error if conversion fails
        return undefined
    }
}

into

/**
 * Converts HTML content to Markdown format.
 *
 * @param html - The HTML content to convert.
 * @param options - Optional tracing parameters.
 * @returns The Markdown representation of the HTML.
 */
export async function HTMLToMarkdown(
    html: string,
    options?: TraceOptions
): Promise<string> {
    if (!html) return html // Return original content if no HTML is provided
    const { trace } = options || {} // Extract trace for logging if available

    try {
        const Turndown = (await import("turndown")).default // Import Turndown library for HTML to Markdown conversion
        const GFMPlugin = (await import("turndown-plugin-gfm")).default // 
        const service = new Turndown();
        service.use(GFMPlugin); // add GitHub Flavored Markdown (tables, striketrough, ...)
        const res = service.turndown(html) // Use Turndown library to convert HTML to Markdown
        return res
    } catch (e) {
        trace?.error("HTML conversion failed", e) // Log error if conversion fails
        return undefined
    }
}

[pelikhan]

sounds good. do you want to add it or i can push this in to a PR.

[pelikhan]

#916

[pelikhan]

in 1.81.0