Enable tanglePrune #8

Powersource · 2022-10-09T20:43:30Z

Based on #5

Powersource · 2022-10-12T13:47:34Z

A few issues for the future popped up
#9
#10 (this is maybe the next one i'll do)
#11

Also found a bug (fixed it here) that also exists in ssb-tribes ssbc/ssb-tribes#70

mixmix · 2022-10-13T00:02:41Z

test/tangle-prune.test.js

+
+test('tangle prune', async (t) => {
+  const ssb = Testbot()
+  const ssbId = ssb.id


🌶️

(emoji for potential spicy problem)

With different feedFormats, the "id" of an feedId is of different lengths, this means that if recps is a GroupId + 15 feedIds.... they could be "sigil" style or "ssb-uri" style.

We need to clarify in a spec "when you are DM'ing a person to add them to a group, which feedId do you use?"
Is it their rootFeedId (which is bendy butt ssb-uri?)

cc @arj03

EASY solution @Powersource - assume all ssb-uri of the longest length, which would currently be:

https://github.com/ssbc/ssb-uri2/blob/main/test/fixtures.js#L17-L26
this long:

ssb:feed/gabbygrove-v1/FY5OG311W4j_KPh8H9B2MZt4WSziy_p-ABkKERJdujQ=

or wait... what is this

ssb:feed/buttwoo-v1/FY5OG311W4j_KPh8H9B2MZt4WSziy_p-ABkKERJdujQ=/Z0rMVMDEO1Aj0uPl0_J2NlhFB2bbFLIHlty_YuqArFq=

Yeah we need a spec

Can they be URI style though? Have we allowed that anywhere actually?

ssb:feed/buttwoo-v1...

do we also know that this is the longest possible string?

Also worth considering this

ssb-tribes2/lib/tangle-prune.js

Lines 8 to 15 in dc4e7bf

// these variables are calculated in

// test/tangle-prune.test.js

// if these variables are out of date and

// * smaller than supposed to: we'll prune a bit much, tangles will converge a bit slower

// * bigger than supposed to: we'll prune less than we can. users might run into 'the message you want to publish is too big' more often

// but either way no catastrophe

const MAX_SIZE_16_recps = 5546

const MAX_SIZE_1_recps = 6041

mixmix · 2022-10-13T00:10:44Z

test/tangle-prune.test.js

+
+    let lower = 4000
+    let mid
+    let upper = 8000


🔥 set upper to 17000

https://github.com/ssbc/ssb-buttwoo-spec#validation

the content length in bytes. This number must not exceed 16384.

If we change this to 17000 now then later when I do #10 this test won't start failing. I sort of want it to fail so I know that I actually managed to swap the format. But maybe there are more solid ways to test which format we're using?

mixmix · 2022-10-13T00:12:39Z

test/tangle-prune.test.js

+  const content = (prevCount, numRecps) => ({
+    type: 'post',
+    text: 'hello!',
+    recps: new Array(numRecps).fill(ssbId),


technically, this need to follow the format

recps: [GroupId, FeedId, ... ] // you need to consider the length of a FeedId here

mixmix

We need to confirm the the recps we're gonna support. See 🌶️

Powersource · 2022-10-13T12:55:23Z

test/tangle-prune.test.js

+  //console.time('prune')
+  const result16 = tanglePrune(content(4000, 16))
+  //console.timeEnd('prune')
+  t.true(
+    encodedLength(result16) <= max16recps,
+    `pruned ${4000 - result16.tangles.group.previous.length}`
+  )
+
+  const result1 = tanglePrune(content(4000, 1))
+  t.true(
+    encodedLength(result1) <= max1recp,
+    `pruned ${4000 - result1.tangles.group.previous.length}`
+  )


I have a feeling it's a good idea to check we're in the right ballpark and not just under a certain number. So changing this to something like this instead

const e = encodedLength(result16) t.true(e > max16recps - 200 && e <= max16recps)

staltz · 2022-10-14T09:18:20Z

In a video call with Jacob, I suggested a different way of checking whether the content is too large, without hardcoding the choice of feedFormat: we can just catch the error emitted from ssb.db.create, and if that error matches "too large" (either matching the err.message string, or we can come up with error codes like err.code === 'ETOOLARGE'), then prune and retry ssb.db.create.

mixmix · 2022-10-17T22:06:53Z

yeah that could work, could make it less resource intensive by just cutting the number of previous in half, and keep doing that (stopping at 1) till it works.

The tricky case to test is whether it copes when you call create 5000 times in a row (fast), which is what happens when a person imports a whakapapa from CSV in Ahau

Powersource · 2022-10-20T09:21:46Z

i feel stuck, which of the options should i go with?

staltz · 2022-10-20T09:28:04Z

@Powersource I think it's the same approach (i.e. catch the error emitted from ssb.db.create then prune then retry), but Mix was suggesting that we should beware of the performance. Ahau uses ssb-db1, and ssb-db2 is faster, but the error-catch-prune-retry approach might be somewhat slow, so let's just write a test that adds 5000 messages with this API, and measure how long that takes, e.g. with console.time('measure') and console.timeEnd('measure')

mixmix · 2022-10-20T23:06:01Z

agree. Try Staltz's suggestion, and we see how bad it is!

Powersource · 2022-10-27T10:30:22Z

fixed with #19 instead

Powersource added 3 commits October 7, 2022 16:52

Import tangle-prune test

936a7ad

Move tanglePrune to own file

921d0d8

The tangle prune is broken

5aad75d

Powersource mentioned this pull request Oct 11, 2022

Make stuff work on feed encodings other than buttwoo #9

Closed

Use bipf instead of json.stringify

ff6905b

Powersource mentioned this pull request Oct 12, 2022

Actually run it on buttwoo #10

Closed

Powersource added 2 commits October 12, 2022 14:33

Get content size calculation to work

b618a94

Make tanglePrune work

8077af0

Powersource marked this pull request as ready for review October 12, 2022 13:48

Powersource requested review from staltz and mixmix October 12, 2022 13:49

Base automatically changed from list-authors to master October 12, 2022 15:02

Merge branch 'master' into tangle-prune

dc4e7bf

mixmix reviewed Oct 13, 2022

View reviewed changes

mixmix requested changes Oct 13, 2022

View reviewed changes

Powersource commented Oct 13, 2022

View reviewed changes

Powersource mentioned this pull request Oct 13, 2022

Run on buttwoo instead of classic #12

Closed

Powersource requested a review from mixmix October 14, 2022 06:36

Powersource mentioned this pull request Oct 21, 2022

Detect too big group tangle previous array by failing to publish, and prune it #19

Merged

3 tasks

Powersource closed this Oct 27, 2022

staltz deleted the tangle-prune branch October 27, 2022 11:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable tanglePrune #8

Enable tanglePrune #8

Powersource commented Oct 9, 2022

Powersource commented Oct 12, 2022

mixmix Oct 13, 2022 •

edited

Loading

mixmix Oct 13, 2022

Powersource Oct 13, 2022

Powersource Oct 13, 2022

Powersource Oct 13, 2022

mixmix Oct 13, 2022

Powersource Oct 13, 2022

mixmix Oct 13, 2022 •

edited

Loading

mixmix left a comment

Powersource Oct 13, 2022

staltz commented Oct 14, 2022

mixmix commented Oct 17, 2022

Powersource commented Oct 20, 2022

staltz commented Oct 20, 2022

mixmix commented Oct 20, 2022

Powersource commented Oct 27, 2022

	// these variables are calculated in
	// test/tangle-prune.test.js
	// if these variables are out of date and
	// * smaller than supposed to: we'll prune a bit much, tangles will converge a bit slower
	// * bigger than supposed to: we'll prune less than we can. users might run into 'the message you want to publish is too big' more often
	// but either way no catastrophe
	const MAX_SIZE_16_recps = 5546
	const MAX_SIZE_1_recps = 6041

Enable tanglePrune #8

Enable tanglePrune #8

Conversation

Powersource commented Oct 9, 2022

Powersource commented Oct 12, 2022

mixmix Oct 13, 2022 • edited Loading

Choose a reason for hiding this comment

🌶️

mixmix Oct 13, 2022

Choose a reason for hiding this comment

Powersource Oct 13, 2022

Choose a reason for hiding this comment

Powersource Oct 13, 2022

Choose a reason for hiding this comment

Powersource Oct 13, 2022

Choose a reason for hiding this comment

mixmix Oct 13, 2022

Choose a reason for hiding this comment

Powersource Oct 13, 2022

Choose a reason for hiding this comment

mixmix Oct 13, 2022 • edited Loading

Choose a reason for hiding this comment

mixmix left a comment

Choose a reason for hiding this comment

Powersource Oct 13, 2022

Choose a reason for hiding this comment

staltz commented Oct 14, 2022

mixmix commented Oct 17, 2022

Powersource commented Oct 20, 2022

staltz commented Oct 20, 2022

mixmix commented Oct 20, 2022

Powersource commented Oct 27, 2022

mixmix Oct 13, 2022 •

edited

Loading

mixmix Oct 13, 2022 •

edited

Loading