Front-end Log Table Sorting

[mbl-eea-sk]

• descending sorting of log table records was applied (most recent logs first)
• accordingly the first page of paged table is opened by default (still to show the most recent logs)
• resolves Front-end: inverted order of logs in the execution monitor #518

[mbl-eea-sk]

The same behavior for events table (records under tab Events) added.

[tomas-knap]

@mbl-eea-sk Did you test how it behaves when there is ~100 000 logs?

[mbl-eea-sk]

No I did not. How can I get such result?

[tomas-knap]

The best would be to ask @jindrichmynarz (who is preparing datasets for COMSODE) to give you fragment of pipeline which produces ~100 000 logs.

[jindrichmynarz]

Well, the amount of logs is why I usually don't run pipelines in the debug mode. I have a pipeline that creates 6226 pages of logs in normal execution mode. Would it be OK for testing? If not, @tomas-knap do you have tips for a DPU that is particularly noisy in the debug mode?

[tomas-knap]

We can start with this. @jindrichmynarz, can you please link here the export of such pipeline, so that @mbl-eea-sk can test it?

[mbl-eea-sk]

I have already prepared my own DPU that may produce thousands of logs. Anyway it's execution will probably take some time alike this one would. @jindrichmynarz if you still have this execution results, can you please test this front-end compilation on your system? It would be much quicker. I am going to do it with my results as well.

[jindrichmynarz]

Since I use only ODN releases on the VM where I have the pipeline, I cannot test it. If you already have a test pipeline, then I think it should suffice.

[mbl-eea-sk]

Actually I have ~400 000 logs and there is no problem with sorting. Only when I try to download the log file it takes too long but this is expectable I think.
And also the number of records could be written in another way (e. g. JavaScript) while it does not refresh automatically with increasing number of logs.

[mbl-eea-sk]

OK, the first 6 pages of logs are still sorted in ascendent order. Will wait until it finishes if the final result is correct.

[skrchnavy]

@mbl-eea-sk is performance of page load in GUI fast enough (similar to previous implementation)?

[mbl-eea-sk]

Yes, there is no visible difference.

[mbl-eea-sk]

Well, the execution finished having produced 842 080 logs. By random pages view it appears to be sorted correctly, causing no problem with request timeout.
Small delays observed when trying to skip by thousands of pages.

[skrchnavy]

@tomas-knap please comment if we can merge

[eea03]

I can confirm that it works as expected.
However, I am quite concerned about the performence of this solution.

I tested it with pipelines producing ~500k logs.
Before these changes, paging in logs table was done by SQL queries like

SELECT id, DPU, EXECUTION, log_level, message, relative_id, logger, stack_trace, timestmp FROM logging WHERE (((EXECUTION = 113) AND (relative_id > 400000)) AND (relative_id <= 400020));

In logs table with ~1.5M entries, query of any logs page is executed in ~0.100 ms

After the changes, paging is done by SQL queries like

SELECT id AS a1, DPU AS a2, EXECUTION AS a3, log_level AS a4, message AS a5, relative_id AS a6, logger AS a7, stack_trace AS a8, timestmp AS a9 FROM logging WHERE (EXECUTION = 113) ORDER BY id DESC LIMIT 20 OFFSET 45000;

In logs table with 1.5M entries, query of page lasts ~300 - 500 ms depending on offset, which is an great difference. And this time quickly increases as count of logs in database increases.

This difference can already be noticed on GUI and I think 1.5M is quite small logs table considering how much logs can be produced by some pipelines

Based on changed source code, I suspect this is some underlying implementation of paging table and probably is not so easy to change.

I would suggest to be careful with this feature and test it thoroughly with large log tables both for MySQL and PostgreSQL (I tested it for Postgres but I think it should be similar with MySQL)

[tomas-knap]

TODO test with mysql the performance degradation