pValue

[arj03]

I was wondering if we could do better on initial sync. #206 started out as that. That PR changed 1 initial jitdb query from around 17s to 15 something. Then I started wondering if we could do more and found that doing these extracting values (seq, timestamp, sequence) was significant. Meaning not only what we are looking for (opData.seek), but also the base values. Then I started playing around and found that actually finding the position of value inside the buffer is quite significant, so I refactored the code to get that once and pass that in. Note seekFromDesc allows a start (defaults to 0) and because of that we need 3 parameters to the seek function. I changed the seekers in ssb-db2 and now things are down to around 11s. This is on my laptop when it is not plugged in. Plugged in we are down to 7s :-)

This PR is against #206, I'll change that once we know about that PR.

[staltz]

Very good idea! I'll review properly tomorrow, but the idea is 💯

[staltz]

What about also pContent? It should be quite common to go into msg.value.content

[arj03]

It really depends on how often it is used. pValue is used in both timestamp, sequence and in most user specified queries. If you did a query for only author, then pContent would be wasted. pContent might make more sense in level indexes inside db2. I tried added it, but I don't have enough plugins to see any difference. In manyverse I have a feeling you are using a lot so might make more sense.

[staltz]

Exciting to squeeze more performance out of this!

[arj03]

I guess this should be 7.0?

[staltz]

I think there is no breaking change here. Correct me if I'm wrong.

The new argument is at the end, so any consumer of jitdb can be free to not use that argument and just recalculate pValue. Should be a new minor version.

[arj03]

Right, javascript will just throw away any extra arguments :)

[github-actions]

Benchmark results

Part	Speed	Heap Change	Samples
Count 1 big index (3rd run)	0.33ms ± 0.03ms	-10.05 kB ± 14.64 kB	53
Create an index twice concurrently	500.42ms ± 4.93ms	2.17 kB ± 29.91 kB	110
Load core indexes	0.88ms ± 0.01ms	84.47 B ± 191.71 B	9042
Load two indexes concurrently	508.79ms ± 6.84ms	-13.94 kB ± 242.89 kB	20
Paginate 10 results	23.88ms ± 1.04ms	5.15 kB ± 19.14 kB	29
Paginate 20000 msgs with pageSize=5	6359.22ms ± 161.72ms	172.43 kB ± 4058.36 kB	5
Paginate 20000 msgs with pageSize=500	621.99ms ± 14.09ms	-245.66 kB ± 547.21 kB	20
Query 1 big index (1st run)	764.07ms ± 7.48ms	9.91 kB ± 49.71 kB	72
Query 1 big index (2nd run)	321.05ms ± 3.17ms	-2.13 kB ± 16.26 kB	42
Query 3 indexes (1st run)	1002.68ms ± 12.99ms	28.5 kB ± 136.54 kB	54
Query 3 indexes (2nd run)	312.41ms ± 4.64ms	36.94 kB ± 56.06 kB	40
Query a prefix map (1st run)	330.21ms ± 4.94ms	-302.43 kB ± 607.77 kB	21
Query a prefix map (2nd run)	17.7ms ± 1.38ms	2.85 kB ± 46.1 kB	21

[github-actions]

Benchmark results

Part	Speed	Heap Change	Samples
Count 1 big index (3rd run)	0.45ms ± 0.15ms	5.75 kB ± 14.72 kB	55
Create an index twice concurrently	529.79ms ± 2.47ms	21.87 kB ± 32.6 kB	103
Load core indexes	1.08ms ± 0.01ms	91.56 B ± 205.3 B	7874
Load two indexes concurrently	448.9ms ± 3.29ms	-234.85 kB ± 134.5 kB	23
Paginate 10 results	31.95ms ± 1.27ms	23.33 kB ± 29.56 kB	24
Paginate 20000 msgs with pageSize=5	6208.74ms ± 70.73ms	57.41 kB ± 272.95 kB	5
Paginate 20000 msgs with pageSize=500	504.13ms ± 14.43ms	15.68 kB ± 603.56 kB	21
Query 1 big index (1st run)	748.22ms ± 9.21ms	-6.28 kB ± 32.82 kB	73
Query 1 big index (2nd run)	264.61ms ± 2.83ms	-1.69 kB ± 9.25 kB	56
Query 3 indexes (1st run)	966.35ms ± 9.9ms	-101.53 kB ± 128.64 kB	56
Query 3 indexes (2nd run)	280.32ms ± 3.52ms	118.95 kB ± 57.01 kB	44
Query a prefix map (1st run)	259.16ms ± 1.56ms	245.4 kB ± 417.82 kB	24
Query a prefix map (2nd run)	14.37ms ± 0.51ms	62.23 kB ± 59.76 kB	24