JavaScript modules parsing

[vbelouso]

During the VDB Embedding stage, the agent crashed while attempting to parse and load JavaScript files from the codebase, resulting in multiple exceptions (logs in the attachment)
Exceptions are logged, but they don't cause the pipeline to crash.
The issue occurs when the agent tries to parse JS modules

The current parsing logic is based on esprima-python=4.0 and langchain-community==0.2.17

Langchain JavaScriptSegmenter doesn't support module parsing (code source)

esprima-python only supports ECMAScript 2017, but not modern ES implementations

In our fork, we have added an extended version of JavaScriptSegmenter, which allows solving most of the issues with JS parsing.

Perhaps you will be interested in using it, then I will open a PR
js_logs.txt
output.json