Is there any way to skip the asset prefetching and other processing?

[Jocosityy]

I was giving this a try because I was under the impression that it would be more lightweight and better for performance, and I'm trying to back up a discord server that runs back several years which makes the html export from discord chat exporter perform badly (even in json format, our main discussion channel is still a little over 2 GB). But so far, I haven't gotten it to load at all; it just spends a while prefetching things and trying to load assets that aren't there, and then loops mongodb messages for upwards of an hour while steadily eating RAM (I have 64 GB so I can handle it eating 10, but most of the people in our server who might want to keep a backup don't). Is there any means of speeding the process up by disabling something? I wasn't sure if I should put this in issues or not since it's entirely possible it's just user error or my server just being too large.

[slatinsky]

Hello,

there is no way to skip the processing. This step is needed to make search working, which is in my opinion expected in rich viewer such as this one. This step runs only once (unless you update the program).

Now you need RAM the size of the largest export * 10. I didn't though that would be an issue, but multiple people are saying that their exports are really really large.

In the future I will rewrite message preprocessing to stream JSON files in, so it won't need to load the entire json in the RAM. But that will create another speed penalty (the fastest way to process things is to have them in RAM).

According to my testing, preprocessing step is still 50-times faster than export step from Discord.

[Jocosityy]

It is a lot faster than the exporting step, I won't deny that; it usually takes the majority of the day to export my server even into json. I'll just wait for that rewrite then. Thanks!

[slatinsky]

Then it should need only ~30 minutes to preprocess in your case. That needs to be done only this one time. Next time only new exports are preprocessed and you won't have to wait a long time