Documentation website unable to expose all text pre-rendered

[AriaShishegaran]

Describe the bug
This is a big problem in my humble opinion but feel free to remove it if you think otherwise. I'd love to know your feedback nonetheless.

At the moment, when trying to quickly search through available attributes throughout your documentation, I thought to myself why not use CursorAI or Perplexity for some quick help on navigating things better. When I fed the specific URL or just generally asked regarding a specific thing I knew existed in your docs but these services weren't able to retrieve it, I started investigating and found out that the documentation platform you're using is not serving everything as pre-rendered SSR text and therefore some very important information is being lost here.

Here's my Perplexity link and screenshot:
BTW, this is absolutely incorrect. The data should be retrieved is attached below as well:

https://www.perplexity.ai/search/list-all-the-child-attributes-dxyqGf3oS0.uT0Ri1TZiWQ

https://r2r-docs.sciphi.ai/api-reference/endpoint/logs

Expected behavior
Expected behavior is absolute 0 information loss on information provided by the documentation website. I believe it is safe to assume that we're moving towards a future where people would more and more use these kind of services instead of heavily reading the actual docs. Besides, the crawlers that would aggregate these data to further train new models are also going to be missing these information and therefore the information provided by LLMs regarding your services are going to be inaccurate and therefore turn into a disadvantage.

[emrgnt-cmplxty]

Can you try the built-in AI agent to answer your queries?

Would it be helpful if we made it more obvious how you can use AI to interact with our codebase?

[AriaShishegaran]

@emrgnt-cmplxty I'll give it a try, but for sure that would help.

But still, i believe my argument holds. If your documentation website is holding you back from maximal exposure through AI answering engines, I believe you have a bigger problem to think about. With the AI chat tool, you're adding another specialized tool that the user needs to use. With a fully indexable and even downloadable documentation website, everyone can embed the latest version of your documentation into their workflow.
For this to happen, I even wrote a crawler to do so with your docs and even that is having problems dumping everything into MD files. All I'm saying is that it is drastically detracting from the overall experience, especially at this day and age.
For sure it is a beautiful website, but it is functional with modern use cases?

Cursor cannot fully read it, Perplexity cannot fully read it, for sure ChatGPT browsing cannot as well, and I would assume people that know what RAG is, wouldn't be preferring sticking to old methods where they find and implement things in an order of magnitude faster but you're platform is becoming the bottleneck of that.

[AriaShishegaran]

@emrgnt-cmplxty I added this logic to my script to even show the elements it can detect normally on the page and basically the most important part of the page is not even visible! It's super weird, it's like it cannot access it or something.