Feature: Ingest External SST #188

wangnengjie · 2022-08-23T06:41:19Z

Signed-off-by: wangnengjie [email protected]

See details in: #187

Signed-off-by: wangnengjie <[email protected]>

…heck test Signed-off-by: wangnengjie <[email protected]>

Signed-off-by: wangnengjie <[email protected]>

wangnengjie · 2022-08-24T06:01:14Z

The next_txn_ts won't update when reopening db? @GanZiheng

Signed-off-by: wangnengjie <[email protected]>

GanZiheng · 2022-08-24T09:20:22Z

The next_txn_ts won't update when reopening db? @GanZiheng

Yes... I remember it is set to 1.

wangnengjie · 2022-08-24T09:27:55Z

Yes... I remember it is set to 1.

Well, that's... amazing... I debug a while for my failed test. I think I should make another pr to make it works right.

Signed-off-by: wangnengjie <[email protected]>

wangnengjie · 2022-09-01T07:41:03Z

Generally done. You can have a look. @GanZiheng

BTW, the rustfmt.toml use many flags that are only available in nightly channel while CI use stable channel. So it just failed though I run make format under nightly channel.

src/ops/oracle.rs

GanZiheng · 2022-09-06T16:01:48Z

proto/src/proto/meta.proto

@@ -44,6 +44,7 @@ message ManifestChange {
  uint64 key_id  = 4;
  EncryptionAlgo encryption_algo = 5;
  uint32 compression = 6;   // Only used for CREATE Op.
+  uint64 global_version = 7; // Only used for file ingest, 0 means no global_version


Is it necessary to add global_version just for the ingest external file scenario?

Yes. For ingested files, version of every key is invalid. Or if you have any better idea to assign version for keys in ingested files? Iter on every key and issue random writes is not a good idea.

Yes. For ingested files, version of every key is invalid. Or if you have any better idea to assign version for keys in ingested files? Iter on every key and issue random writes is not a good idea.

How about rewriting files to be ingested? In this way, we can also deal with the large value cannot be stored in value log.

Yes. For ingested files, version of every key is invalid. Or if you have any better idea to assign version for keys in ingested files? Iter on every key and issue random writes is not a good idea.

How about rewriting files to be ingested? In this way, we can also deal with the large value cannot be stored in value log.

I think it really costs to rewrite. The bandwidth on cloud device is precious. Large value can store to value log when compactioning ingested files (do we impl this?). And, when rewriting, we are not sure these files can finally success to ingest and so split large values to value log needs more think.

Currently we only write value log when writing entry to LSM tree.

Currently we only write value log when writing entry to LSM tree.

Split during compaction is nice when the threshold changes dynamically or reopen db with different config.

Well, we can discuss this point tomorrow or offline.

Yes. For ingested files, version of every key is invalid. Or if you have any better idea to assign version for keys in ingested files? Iter on every key and issue random writes is not a good idea.

How about rewriting files to be ingested? In this way, we can also deal with the large value cannot be stored in value log.

I think this will block read process. We need to get commit_ts (version) before rewrite and due to SSI protection, every read_ts after that should wait for this txn to finish (watermark).

src/ingest.rs

GanZiheng · 2022-09-06T16:05:46Z

src/ingest.rs

+                    Bytes::copy_from_slice(user_key(file.table.as_ref().unwrap().smallest())),
+                    Bytes::copy_from_slice(user_key(file.table.as_ref().unwrap().biggest())),
+                );
+                let version = self.core.orc.new_ingest_commit_ts(vec![range], &self.opts);


Can we only get commit timestamp once for all external files?

I'd like to. But for files that overlap with each other, there might be same key. And if we get one commit_ts for all these files, there will be same key with same version in different SST which breaks the protection.

GanZiheng · 2022-09-06T16:09:02Z

Generally done. You can have a look. @GanZiheng

BTW, the rustfmt.toml use many flags that are only available in nightly channel while CI use stable channel. So it just failed though I run make format under nightly channel.

We currently use stable channel for the sake of avoiding CI failing too often. I think there is a format error in this version of code.

wangnengjie · 2022-09-06T16:30:49Z

Generally done. You can have a look. @GanZiheng
BTW, the rustfmt.toml use many flags that are only available in nightly channel while CI use stable channel. So it just failed though I run make format under nightly channel.

We currently use stable channel for the sake of avoiding CI failing too often. I think there is a format error in this version of code.

As what I have said, rustfmt.toml use many flags that are only available in nightly channel which makes the result of format different with stable channel. I will use stable channel to format the code later. So why not just remove all these flags that only available in nightly channel?

Does CI often failed in nightly channel? I saw that all tests (except the first one) run in nightly channel for the support of sanitizer...

GanZiheng · 2022-09-06T16:34:33Z

Does CI often failed in nightly channel? I saw that all tests (except the first one) run in nightly channel for the support of sanitizer...

Specifically, failed due to the clippy check. We can consider remove these flags or just turn back to nightly channel, I think that's not matter.

Signed-off-by: wangnengjie <[email protected]>

codecov · 2022-09-07T01:26:38Z

Codecov Report

Merging #188 (f241a88) into master (070ff80) will increase coverage by 1.16%.
The diff coverage is 94.85%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #188      +/-   ##
==========================================
+ Coverage   89.63%   90.80%   +1.16%     
==========================================
  Files          39       40       +1     
  Lines        8838     9679     +841     
==========================================
+ Hits         7922     8789     +867     
+ Misses        916      890      -26

wangnengjie added 4 commits August 22, 2022 20:08

feat: ingest external SST files

ce8b893

Signed-off-by: wangnengjie <[email protected]>

test(ingest): add basic tests

fce8f50

Signed-off-by: wangnengjie <[email protected]>

fix: fix ReadsTrace update for conflict check & add ingest conflict c…

ca86239

…heck test Signed-off-by: wangnengjie <[email protected]>

style: use unwrap,unwrap_err

15433a1

Signed-off-by: wangnengjie <[email protected]>

wangnengjie added 2 commits August 24, 2022 14:24

test: use DBTestWrapper & add reopen test (ignore currently)

54c5172

Signed-off-by: wangnengjie <[email protected]>

fix(ingest): alloc version for each file when overlap

60a7a39

Signed-off-by: wangnengjie <[email protected]>

wangnengjie added 3 commits August 30, 2022 15:50

test(ingest): add some tests

ad407bb

Signed-off-by: wangnengjie <[email protected]>

Merge branch 'master' into feat/ingest_sst

6e829c7

Signed-off-by: wangnengjie <[email protected]>

refactor: fit max_version

ea205ad

Signed-off-by: wangnengjie <[email protected]>

wangnengjie changed the title ~~[WIP] Feature: Ingest External SST~~ Feature: Ingest External SST Sep 1, 2022

style: run make format

1116138

Signed-off-by: wangnengjie <[email protected]>

GanZiheng reviewed Sep 6, 2022

View reviewed changes

wangnengjie added 2 commits September 7, 2022 08:47

docs: add some comments

a3e5264

Signed-off-by: wangnengjie <[email protected]>

style: format & clippy

f241a88

Signed-off-by: wangnengjie <[email protected]>

wangnengjie force-pushed the feat/ingest_sst branch from 36552fe to f241a88 Compare September 7, 2022 01:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Ingest External SST #188

Feature: Ingest External SST #188

wangnengjie commented Aug 23, 2022 •

edited

Loading

wangnengjie commented Aug 24, 2022

GanZiheng commented Aug 24, 2022 •

edited

Loading

wangnengjie commented Aug 24, 2022

wangnengjie commented Sep 1, 2022

GanZiheng Sep 6, 2022

wangnengjie Sep 6, 2022

GanZiheng Sep 6, 2022

wangnengjie Sep 6, 2022 •

edited

Loading

GanZiheng Sep 6, 2022

wangnengjie Sep 6, 2022

wangnengjie Sep 7, 2022

GanZiheng Sep 6, 2022

wangnengjie Sep 6, 2022

GanZiheng commented Sep 6, 2022

wangnengjie commented Sep 6, 2022

GanZiheng commented Sep 6, 2022 •

edited

Loading

codecov bot commented Sep 7, 2022

Feature: Ingest External SST #188

Are you sure you want to change the base?

Feature: Ingest External SST #188

Conversation

wangnengjie commented Aug 23, 2022 • edited Loading

wangnengjie commented Aug 24, 2022

GanZiheng commented Aug 24, 2022 • edited Loading

wangnengjie commented Aug 24, 2022

wangnengjie commented Sep 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangnengjie Sep 6, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GanZiheng commented Sep 6, 2022

wangnengjie commented Sep 6, 2022

GanZiheng commented Sep 6, 2022 • edited Loading

codecov bot commented Sep 7, 2022

Codecov Report

wangnengjie commented Aug 23, 2022 •

edited

Loading

GanZiheng commented Aug 24, 2022 •

edited

Loading

wangnengjie Sep 6, 2022 •

edited

Loading

GanZiheng commented Sep 6, 2022 •

edited

Loading