Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test and fix post compat #426

Merged
merged 3 commits into from
Nov 6, 2023
Merged

Test and fix post compat #426

merged 3 commits into from
Nov 6, 2023

Conversation

Powersource
Copy link
Collaborator

For ssbc/ssb-tribes#105

Trying to remove db1 completely from ssb-tribes, apparently some test runs replicate and that depends on sbot.post. But the compat plugin was only providing sbot.db.post. I also added a test.

@Powersource Powersource requested a review from staltz November 2, 2023 19:22
@Powersource
Copy link
Collaborator Author

@staltz review if you have time :) otherwise i could pass to mix probably

@Powersource
Copy link
Collaborator Author

damn that's not something i expected to fail

migrate keeps new log synced with ssb-db being updated
2023-11-02T19:33:14.129Z ssb:db2:private encrypted or decrypted indexes seem to be empty
2023-11-02T19:33:14.133Z ssb:db2 lowest offset for all indexes is -1
2023-11-02T19:33:14.133Z ssb:db2 updateIndexes() scan time: 0ms
2023-11-02T19:33:14.133Z ssb:db2 updateIndexes() live streaming
    ✔ migration done
    ✔ 1st query suceeded
2023-11-02T19:33:14.136Z ssb:db2 getAllLatest is done setting up initial validate state
2023-11-02T19:33:14.136Z ssb:db2 getAllLatest is done setting up initial validate state
    ✔ 10 msgs
    ✔ publish suceeded
    ✔ msg posted
    ✔ 2nd query suceeded
Uncaught Exception at  uncaughtException : TypeError: Cannot read properties of undefined (reading 'value')
    at /home/runner/work/ssb-db2/ssb-db2/test/migration.js:144:34
    at /home/runner/work/ssb-db2/ssb-db2/node_modules/jitdb/index.js:1417:13
    at CollectStream._cb (/home/runner/work/ssb-db2/ssb-db2/node_modules/jitdb/index.js:1247:11)
    at CollectStream.end (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/sinks/collect.js:16:15)
    at FilterStream.end (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/throughs/through.js:23:22)
    at AsyncMapStream.end (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/throughs/async-map.js:28:20)
    at ValueStream.resume (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/sources/values.js:19:51)
    at AsyncMapStream.resume (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/throughs/through.js:17:19)
    at FilterStream.resume (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/throughs/through.js:17:19)
    at FilterStream.pipe (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/pipe.js:18:26)
    ✖ 418) 11 msgs
      Expected 11 but got 10
      At: <anonymous> (/test/migration.js:143:19)

@Powersource
Copy link
Collaborator Author

so uugh code seems to be supposed to allow importing db1 and db2 at the same time? which breaks with this.

trying out removing post compat from the default imports

@Powersource
Copy link
Collaborator Author

yay works now

@Powersource
Copy link
Collaborator Author

i guess you can argue this is a major version since it changes the default imports but eeh no one uses those anyway

@Powersource
Copy link
Collaborator Author

@staltz done for real now

Copy link

github-actions bot commented Nov 2, 2023

Benchmark results

Part Duration
Create 5000 new messages 328.35ms
Validate 5000 messages 427.60ms
Native to db format 5000 messages 104.93ms
Db to native format 5000 messages 88.56ms
Add 1000 elements 361.98ms
Add 1000 box1 msgs 913.16ms
Unbox 1000 box1 msgs first run 177.58ms
Unbox 1000 box1 msgs second run 87.96ms
Add 1000 box1 msgs 859.41ms
Query 1000 msgs first run 41.25ms
Query 1000 msgs second run 10.58ms
Add 1000 box2 msgs 1317.45ms
Unbox 1000 box2 msgs first run 369.27ms
Unbox 1000 box2 msgs second run 174.19ms
Migrate (+db1) 10860.40ms
Migrate (alone) 3703.97ms
Migrate (+db1 +db2) 7600.90ms
Migrate (+db2) 5431.51ms
Migrate continuation (+db2) 1043.56ms
Memory usage without indexes 815.01 MB = 43.38 MB + etc
Initial indexing 1467.35ms
Initial indexing maxcpu=86 4830.18ms
Initial indexing compat 636.66ms
Two indexes updating concurrently 2517.12ms
Key one initial 86.17ms
Key two 1.26ms
Key one again 0.98ms
Reboot and key one again 79.09ms
Latest root posts 1629.58ms
Latest posts 10.25ms
Votes one initial 1290.67ms
Votes again 0.40ms
HasRoot 1169.69ms
HasRoot again 0.38ms
Author one posts 1251.42ms
Author two posts 26.00ms
Dedicated author one posts 1251.48ms
Dedicated author one posts again 0.92ms
DeleteFeed 5337.18ms
Maximum memory usage 861.89 MB = 59.41 MB + etc
Indexes folder size 10.01mb

Copy link
Member

@mixmix mixmix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@@ -18,7 +18,7 @@ mkdirp.sync(dir)

const keys = ssbKeys.loadOrCreateSync(path.join(dir, 'secret'))

const sbot = SecretStack({ appKey: caps.shs }).use(require('../')).call(null, {
const sbot = SecretStack({ appKey: caps.shs }).use(require('../')).use(require('../compat/post')).call(null, {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this one in here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because db.post.value is used further down in these tests

@@ -8,6 +8,5 @@ module.exports = [
require('ssb-box'),
require('ssb-box2'),
require('./compat/publish'),
require('./compat/post'),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a breaking change, so be careful to make a new major version.

Copy link
Member

@staltz staltz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems ok

@Powersource Powersource merged commit d955ea1 into master Nov 6, 2023
5 checks passed
@Powersource Powersource deleted the fix-post-compat branch November 6, 2023 13:42
@Powersource
Copy link
Collaborator Author

v8.0.0

Copy link

github-actions bot commented Nov 6, 2023

Benchmark results

Part Duration
Create 5000 new messages 358.80ms
Validate 5000 messages 456.62ms
Native to db format 5000 messages 92.58ms
Db to native format 5000 messages 90.28ms
Add 1000 elements 366.86ms
Add 1000 box1 msgs 980.34ms
Unbox 1000 box1 msgs first run 186.39ms
Unbox 1000 box1 msgs second run 89.45ms
Add 1000 box1 msgs 934.41ms
Query 1000 msgs first run 36.88ms
Query 1000 msgs second run 11.18ms
Add 1000 box2 msgs 1301.77ms
Unbox 1000 box2 msgs first run 312.52ms
Unbox 1000 box2 msgs second run 156.42ms
Migrate (+db1) 10587.18ms
Migrate (alone) 3374.11ms
Migrate (+db1 +db2) 7440.20ms
Migrate (+db2) 5249.44ms
Migrate continuation (+db2) 941.62ms
Memory usage without indexes 763.99 MB = 39.08 MB + etc
Initial indexing 1403.12ms
Initial indexing maxcpu=86 4740.16ms
Initial indexing compat 538.03ms
Two indexes updating concurrently 2459.00ms
Key one initial 65.35ms
Key two 1.10ms
Key one again 0.92ms
Reboot and key one again 79.46ms
Latest root posts 1567.41ms
Latest posts 25.31ms
Votes one initial 1254.46ms
Votes again 0.36ms
HasRoot 1198.69ms
HasRoot again 0.30ms
Author one posts 1266.28ms
Author two posts 29.32ms
Dedicated author one posts 1275.11ms
Dedicated author one posts again 0.49ms
DeleteFeed 5000.25ms
Maximum memory usage 910.52 MB = 60.65 MB + etc
Indexes folder size 10.01mb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants