Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add caching #19

Merged
merged 15 commits into from
Feb 5, 2024
Merged

Add caching #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
f54488f
Initial cache infra
diondokter Jan 12, 2024
733cc49
Added the first real cache
diondokter Jan 12, 2024
07d940e
Better cache dealing in repair functions.
diondokter Jan 13, 2024
5cc580e
Allow false-positive lints
diondokter Jan 13, 2024
7b416fd
Remove defmt logging
diondokter Jan 13, 2024
a25c96b
Fully use the dirty flag to invallidate cache when required
diondokter Jan 13, 2024
e6616ff
Fix test + fmt
diondokter Jan 17, 2024
21ec76d
Redo the cache structure
diondokter Feb 5, 2024
07e58bd
Make cache module private
diondokter Feb 5, 2024
e68bb67
Merge branch 'master' into cache
diondokter Feb 5, 2024
455dc1e
Fix test
diondokter Feb 5, 2024
dfe81df
Fix fuzz
diondokter Feb 5, 2024
1f5a0b2
Add docs and more readme
diondokter Feb 5, 2024
fd71944
Fmt
diondokter Feb 5, 2024
2c2abdb
Reworked the public API again
diondokter Feb 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,21 @@ If you're looking for an alternative with different tradeoffs, take a look at [e
If for some reason an operation returns the corrupted error, then it might be repairable in many cases.
See the repair functions in the map and queue modules for more info.

## Caching

There are various cache options that speed up the operations.
By default (no cache) all state is stored in flash and the state has to be fully read every time.
Instead, we can optionally store some state in ram.

These numbers are taken from the test cases in the cache module:

| Name | Map # flash reads | Queue # flash reads |
| -------------: | ----------------: | ------------------: |
| NoCache | 100% | 100% |
| PageStateCache | 77% | 51% |

***Note:** These are the number of reads, not the amount of bytes.*

## Inner workings

To save on erase cycles, this crate only really appends data to the pages. Exactly how this is done depends
Expand Down Expand Up @@ -91,7 +106,11 @@ When using peek_many, you can look at all data from oldest to newest.
(DD-MM-YY)

### Unreleased

- *Breaking* The item to store is now passed by reference to Map `store_item`
- *Breaking* Added cache options to the functions to speed up reading the state of the flash.
To retain the old behaviour you can pass the `NoCache` type as the cache parameter.
- Removed defmt logging since that wasn't being maintained. The format impl for the errors remain.

### 0.7.0 10-01-24

Expand Down
8 changes: 8 additions & 0 deletions fuzz/fuzz_targets/map.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,12 @@ fn fuzz(ops: Input) {
let mut flash = MockFlashBase::<PAGES, WORD_SIZE, WORDS_PER_PAGE>::new(
WriteCountCheck::OnceOnly,
Some(ops.fuel as u32),
true,
);
const FLASH_RANGE: Range<u32> = 0x000..0x1000;

let mut cache = sequential_storage::cache::NoCache::new();

let mut map = HashMap::new();
#[repr(align(4))]
struct AlignedBuf([u8; 260]);
Expand All @@ -123,6 +126,7 @@ fn fuzz(ops: Input) {
match block_on(sequential_storage::map::store_item(
&mut flash,
FLASH_RANGE,
&mut cache,
&mut buf.0,
&item,
)) {
Expand All @@ -137,6 +141,7 @@ fn fuzz(ops: Input) {
match block_on(sequential_storage::map::fetch_item::<TestItem, _>(
&mut flash,
FLASH_RANGE,
&mut cache,
&mut buf.0,
item.key,
)) {
Expand Down Expand Up @@ -167,6 +172,7 @@ fn fuzz(ops: Input) {
block_on(sequential_storage::map::try_repair::<TestItem, _>(
&mut flash,
FLASH_RANGE,
&mut cache,
&mut buf.0,
))
.unwrap();
Expand All @@ -180,6 +186,7 @@ fn fuzz(ops: Input) {
match block_on(sequential_storage::map::fetch_item::<TestItem, _>(
&mut flash,
FLASH_RANGE,
&mut cache,
&mut buf.0,
key,
)) {
Expand Down Expand Up @@ -213,6 +220,7 @@ fn fuzz(ops: Input) {
block_on(sequential_storage::map::try_repair::<TestItem, _>(
&mut flash,
FLASH_RANGE,
&mut cache,
&mut buf.0,
))
.unwrap();
Expand Down
17 changes: 17 additions & 0 deletions fuzz/fuzz_targets/queue.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,12 @@ fn fuzz(ops: Input) {
let mut flash = MockFlashBase::<PAGES, WORD_SIZE, WORDS_PER_PAGE>::new(
WriteCountCheck::Twice,
Some(ops.fuel as u32),
true,
);
const FLASH_RANGE: Range<u32> = 0x000..0x1000;

let mut cache = sequential_storage::cache::NoCache::new();

let mut order = VecDeque::new();
let mut buf = AlignedBuf([0; MAX_VALUE_SIZE + 1]);

Expand All @@ -73,6 +76,7 @@ fn fuzz(ops: Input) {
let max_fit = match block_on(sequential_storage::queue::find_max_fit(
&mut flash,
FLASH_RANGE,
&mut cache,
)) {
Ok(val) => val,
Err(Error::Corrupted {
Expand All @@ -86,6 +90,7 @@ fn fuzz(ops: Input) {
block_on(sequential_storage::queue::try_repair(
&mut flash,
FLASH_RANGE,
&mut cache,
))
.unwrap();
corruption_repaired = true;
Expand All @@ -99,6 +104,7 @@ fn fuzz(ops: Input) {
match block_on(sequential_storage::queue::push(
&mut flash,
FLASH_RANGE,
&mut cache,
&buf.0[..val.len()],
false,
)) {
Expand Down Expand Up @@ -145,6 +151,7 @@ fn fuzz(ops: Input) {
block_on(sequential_storage::queue::try_repair(
&mut flash,
FLASH_RANGE,
&mut cache,
))
.unwrap();
corruption_repaired = true;
Expand All @@ -157,6 +164,7 @@ fn fuzz(ops: Input) {
match block_on(sequential_storage::queue::pop(
&mut flash,
FLASH_RANGE,
&mut cache,
&mut buf.0,
)) {
Ok(value) => {
Expand Down Expand Up @@ -193,6 +201,7 @@ fn fuzz(ops: Input) {
block_on(sequential_storage::queue::try_repair(
&mut flash,
FLASH_RANGE,
&mut cache,
))
.unwrap();
corruption_repaired = true;
Expand All @@ -205,6 +214,7 @@ fn fuzz(ops: Input) {
let mut popper = match block_on(sequential_storage::queue::pop_many(
&mut flash,
FLASH_RANGE,
&mut cache,
)) {
Ok(val) => val,
Err(Error::Corrupted {
Expand All @@ -218,6 +228,7 @@ fn fuzz(ops: Input) {
block_on(sequential_storage::queue::try_repair(
&mut flash,
FLASH_RANGE,
&mut cache,
))
.unwrap();
corruption_repaired = true;
Expand Down Expand Up @@ -267,6 +278,7 @@ fn fuzz(ops: Input) {
block_on(sequential_storage::queue::try_repair(
&mut flash,
FLASH_RANGE,
&mut cache,
))
.unwrap();
corruption_repaired = true;
Expand All @@ -282,6 +294,7 @@ fn fuzz(ops: Input) {
match block_on(sequential_storage::queue::peek(
&mut flash,
FLASH_RANGE,
&mut cache,
&mut buf.0,
)) {
Ok(value) => {
Expand All @@ -301,6 +314,7 @@ fn fuzz(ops: Input) {
block_on(sequential_storage::queue::try_repair(
&mut flash,
FLASH_RANGE,
&mut cache,
))
.unwrap();
corruption_repaired = true;
Expand All @@ -313,6 +327,7 @@ fn fuzz(ops: Input) {
let mut peeker = match block_on(sequential_storage::queue::peek_many(
&mut flash,
FLASH_RANGE,
&mut cache,
)) {
Ok(val) => val,
Err(Error::Corrupted {
Expand All @@ -326,6 +341,7 @@ fn fuzz(ops: Input) {
block_on(sequential_storage::queue::try_repair(
&mut flash,
FLASH_RANGE,
&mut cache,
))
.unwrap();
corruption_repaired = true;
Expand Down Expand Up @@ -357,6 +373,7 @@ fn fuzz(ops: Input) {
block_on(sequential_storage::queue::try_repair(
&mut flash,
FLASH_RANGE,
&mut cache,
))
.unwrap();
corruption_repaired = true;
Expand Down
Loading
Loading