Support runtime chunk deduplication #1507

jiangliu · 2023-12-07T02:10:19Z

Details

This PR enhances nydusd to support runtime chunk deduplication. It works in this way:

Use a sqlite database to record information about decompressed/plaintext chunks available on local node.
When a chunk is not ready in the uncompressed data blob file, query the sqlite database whether a chunk with the same chunk digest is available. If a chunk with the same chunk digest exists, copy the decompressed from the source data blob file to the target data blob by using copy_file_range().
Otherwise download the compressed chunk from remote, uncompress it and write to the target data blob, and add a record for the chunk to the database.

So there are two types of chunk deduplication:

saving network bandwidth when the chunk is available on local node, because we don't need to download compressed chunk data from remote.
saving local disk space if the underlying filesystem supports reference. If the filesystem storing data blob files supports reference, copy_file_range() will optimize to use reference instead of data copy, thus reduce local storage consuption.

Types of changes

What types of changes does your PullRequest introduce? Put an x in all the boxes that apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation Update (if none of the other choices apply)

Checklist

Go over all the following points, and put an x in all the boxes that apply.

I have updated the documentation accordingly.
I have added tests to cover my changes.

codecov · 2023-12-07T05:18:55Z

Codecov Report

Attention: Patch coverage is 66.51163% with 72 lines in your changes missing coverage. Please review.

Project coverage is 62.74%. Comparing base (06755fe) to head (7d287c9).
Report is 121 commits behind head on master.

Files with missing lines	Patch %	Lines
storage/src/cache/dedup/mod.rs	82.75%	16 Missing and 4 partials ⚠️
src/bin/nydusd/main.rs	0.00%	18 Missing ⚠️
storage/src/cache/cachedfile.rs	0.00%	17 Missing ⚠️
storage/src/utils.rs	78.94%	6 Missing and 2 partials ⚠️
storage/src/cache/filecache/mod.rs	66.66%	2 Missing and 2 partials ⚠️
storage/src/cache/fscache/mod.rs	63.63%	2 Missing and 2 partials ⚠️
utils/src/digest.rs	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master    #1507    +/-   ##
========================================
  Coverage   62.73%   62.74%            
========================================
  Files         129      129            
  Lines       44153    44360   +207     
  Branches    44153    44360   +207     
========================================
+ Hits        27700    27834   +134     
- Misses      15087    15144    +57     
- Partials     1366     1382    +16

Files with missing lines	Coverage Δ
storage/src/cache/dedup/db.rs	`79.09% <100.00%> (+0.08%)`	⬆️
storage/src/cache/mod.rs	`57.84% <ø> (ø)`
utils/src/digest.rs	`91.53% <0.00%> (-0.53%)`	⬇️
storage/src/cache/filecache/mod.rs	`67.58% <66.66%> (+0.08%)`	⬆️
storage/src/cache/fscache/mod.rs	`75.92% <63.63%> (-0.47%)`	⬇️
storage/src/utils.rs	`93.59% <78.94%> (-2.30%)`	⬇️
storage/src/cache/cachedfile.rs	`33.14% <0.00%> (-0.44%)`	⬇️
src/bin/nydusd/main.rs	`0.18% <0.00%> (-0.01%)`	⬇️
storage/src/cache/dedup/mod.rs	`72.72% <82.75%> (+72.72%)`	⬆️

... and 2 files with indirect coverage changes

smoke/tests/chunk_dedup.go

storage/src/cache/filecache/mod.rs

ccx1024cc

NICE. LGTM.
copy_file_range is triggered in tests. It's effective.

Add helper copy_file_range() which: - avoid copy data into userspace - may support reflink on xfs etc Signed-off-by: Jiang Liu <[email protected]>

Implement CasManager to support chunk dedup at runtime. The manager provides to major interfaces: - add chunk data to the CAS database - check whether a chunk exists in CAS database and copy it to blob file by copy_file_range() if the chunk exists. Signed-off-by: Jiang Liu <[email protected]>

Enable chunk deduplication for file cache. It works in this way: - When a chunk is not in blob cache file yet, inquery CAS database whether other blob data files have the required chunk. If there's duplicated data chunk in other data files, copy the chunk data into current blob cache file by using copy_file_range(). - After downloading a data chunk from remote, save file/offset/chunk-id into CAS database, so it can be reused later. Signed-off-by: Jiang Liu <[email protected]>

Add smoking test case for chunk dedup. Signed-off-by: Jiang Liu <[email protected]>

Add documentation for cas. Signed-off-by: Jiang Liu <[email protected]>

docs/data-deduplication.md

imeoer · 2023-12-19T02:02:11Z

smoke/tests/chunk_dedup_test.go

+
+	// Verify lower layer mounted by nydusd
+	ctx.Env.BootstrapPath = lowerBootstrap
+	tool.Verify(t, ctx, lowerLayer.FileTree)


We may need a way to check if the CAS works.

imeoer · 2023-12-19T02:15:03Z

storage/src/cache/fscache/mod.rs

@@ -240,9 +240,16 @@ impl FileCacheEntry {
        };
        let blob_compressed_size = Self::get_blob_size(&reader, &blob_info)?;

+        // Turn off chunk deduplication in case of tarfs.
+        let cas_mgr = if is_tarfs {


Maybe give a warnning message about conflict feature here.

imeoer · 2023-12-19T02:15:34Z

storage/src/cache/filecache/mod.rs

@@ -208,11 +209,18 @@ impl FileCacheEntry {
        } else {
            reader.clone()
        };
+        // Turn off chunk deduplication in case of cache data encryption is enabled or is tarfs.
+        let cas_mgr = if mgr.cache_encrypted || mgr.cache_raw_data || is_tarfs {


Maybe give a warn message about conflict feature here.

imeoer · 2023-12-19T02:19:54Z

docs/data-deduplication.md

+If a record with the same chunk digest already exists, it will be reused.
+We call such a system as CAS (Content Addressable Storage).
+
+## Chunk Deduplication by Using CAS as L2 Cache


This still seems to be an experimental feature, do we still need to consider the cas.db record recycling?

jonoirwinrsa · 2024-05-28T18:48:31Z

Hi all, I tried out this feature and it seems to work as expected. Is there something preventing it from being merged?

imeoer · 2024-05-29T01:29:12Z

Hi all, I tried out this feature and it seems to work as expected. Is there something preventing it from being merged?

cc @jiangliu any updates we can continue? :)

elijah-rou · 2024-09-20T09:26:02Z

what is the status of this PR? anything to do to help get it merged? :)

Desiki-high · 2024-10-23T14:53:30Z

Complete in #1626.

jiangliu requested a review from a team as a code owner December 7, 2023 02:10

jiangliu requested review from liubogithub, luodw and changweige and removed request for a team December 7, 2023 02:10

jiangliu force-pushed the runtime-chunk-dedup branch 4 times, most recently from 7cc1aa1 to ac55d88 Compare December 7, 2023 05:13

jiangliu force-pushed the runtime-chunk-dedup branch 4 times, most recently from bc4403c to 946d8a0 Compare December 7, 2023 06:50

jiangliu mentioned this pull request Dec 7, 2023

local cas: achieve chunk deduplication for nydus. #1399

Open

3 tasks

jiangliu force-pushed the runtime-chunk-dedup branch 2 times, most recently from 13de7e8 to 40415e5 Compare December 8, 2023 04:17

Desiki-high reviewed Dec 9, 2023

View reviewed changes

smoke/tests/chunk_dedup.go Outdated Show resolved Hide resolved

jiangliu force-pushed the runtime-chunk-dedup branch 3 times, most recently from 5403291 to 1c695cf Compare December 11, 2023 10:51

ccx1024cc reviewed Dec 13, 2023

View reviewed changes

storage/src/cache/filecache/mod.rs Show resolved Hide resolved

ccx1024cc reviewed Dec 13, 2023

View reviewed changes

ccx1024cc requested review from ccx1024cc and imeoer December 13, 2023 09:42

jiangliu force-pushed the runtime-chunk-dedup branch from 1c695cf to d199ab8 Compare December 13, 2023 10:58

jiangliu added 4 commits December 15, 2023 15:19

storage: add helper copy_file_range()

f07f16c

Add helper copy_file_range() which: - avoid copy data into userspace - may support reflink on xfs etc Signed-off-by: Jiang Liu <[email protected]>

storage: add smoking test for chunk dedup

ef47e51

Add smoking test case for chunk dedup. Signed-off-by: Jiang Liu <[email protected]>

storage: add documentation for cas

7d287c9

Add documentation for cas. Signed-off-by: Jiang Liu <[email protected]>

jiangliu force-pushed the runtime-chunk-dedup branch from d199ab8 to 7d287c9 Compare December 15, 2023 07:19

imeoer reviewed Dec 19, 2023

View reviewed changes

jonoirwinrsa mentioned this pull request Jul 12, 2024

api.sock: not found #1600

Closed

Desiki-high mentioned this pull request Sep 21, 2024

storage: introduce CasManager to support chunk dedup at runtime #1626

Merged

6 tasks

Desiki-high closed this Oct 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support runtime chunk deduplication #1507

Support runtime chunk deduplication #1507

jiangliu commented Dec 7, 2023 •

edited

Loading

codecov bot commented Dec 7, 2023 •

edited

Loading

ccx1024cc left a comment

imeoer Dec 19, 2023

imeoer Dec 19, 2023

imeoer Dec 19, 2023

imeoer Dec 19, 2023

jonoirwinrsa commented May 28, 2024

imeoer commented May 29, 2024

elijah-rou commented Sep 20, 2024

Desiki-high commented Oct 23, 2024

Support runtime chunk deduplication #1507

Support runtime chunk deduplication #1507

Conversation

jiangliu commented Dec 7, 2023 • edited Loading

Details

Types of changes

Checklist

codecov bot commented Dec 7, 2023 • edited Loading

Codecov Report

ccx1024cc left a comment

Choose a reason for hiding this comment

imeoer Dec 19, 2023

Choose a reason for hiding this comment

imeoer Dec 19, 2023

Choose a reason for hiding this comment

imeoer Dec 19, 2023

Choose a reason for hiding this comment

imeoer Dec 19, 2023

Choose a reason for hiding this comment

jonoirwinrsa commented May 28, 2024

imeoer commented May 29, 2024

elijah-rou commented Sep 20, 2024

Desiki-high commented Oct 23, 2024

jiangliu commented Dec 7, 2023 •

edited

Loading

codecov bot commented Dec 7, 2023 •

edited

Loading