Increase max stable memory read / write buffer size (#6440)

open-chat-labs · Sep 23, 2024 · 52a7a8b · 52a7a8b
1 parent 039012e
commit 52a7a8b
Show file tree

Hide file tree

Showing 23 changed files with 52 additions and 3 deletions.
diff --git a/backend/canisters/airdrop_bot/CHANGELOG.md b/backend/canisters/airdrop_bot/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1314](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1314-airdrop_bot)] - 2024-09-02
 

diff --git a/backend/canisters/community/CHANGELOG.md b/backend/canisters/community/CHANGELOG.md
@@ -5,6 +5,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [unreleased]
 
+### Changed
+
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
+
 ## [[2.0.1351](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1351-community)] - 2024-09-20
 
 ### Changed

diff --git a/backend/canisters/cycles_dispenser/CHANGELOG.md b/backend/canisters/cycles_dispenser/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1315](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1315-cycles_dispenser)] - 2024-09-02
 

diff --git a/backend/canisters/escrow/CHANGELOG.md b/backend/canisters/escrow/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1316](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1316-escrow)] - 2024-09-02
 

diff --git a/backend/canisters/event_relay/CHANGELOG.md b/backend/canisters/event_relay/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1317](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1317-event_relay)] - 2024-09-02
 

diff --git a/backend/canisters/group/CHANGELOG.md b/backend/canisters/group/CHANGELOG.md
@@ -5,6 +5,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [unreleased]
 
+### Changed
+
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
+
 ## [[2.0.1352](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1352-group)] - 2024-09-20
 
 ### Changed

diff --git a/backend/canisters/group_index/CHANGELOG.md b/backend/canisters/group_index/CHANGELOG.md
@@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [unreleased]
 
+### Changed
+
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
+
 ## [[2.0.1355](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1355-group_index)] - 2024-09-20
 
 ### Added

diff --git a/backend/canisters/identity/CHANGELOG.md b/backend/canisters/identity/CHANGELOG.md
@@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [unreleased]
 
+### Changed
+
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
+
 ## [[2.0.1344](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1344-identity)] - 2024-09-10
 
 ### Fixed

diff --git a/backend/canisters/local_group_index/CHANGELOG.md b/backend/canisters/local_group_index/CHANGELOG.md
@@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [unreleased]
 
+### Changed
+
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
+
 ## [[2.0.1356](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1356-local_group_index)] - 2024-09-20
 
 ### Added

diff --git a/backend/canisters/local_user_index/CHANGELOG.md b/backend/canisters/local_user_index/CHANGELOG.md
@@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [unreleased]
 
+### Changed
+
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
+
 ## [[2.0.1353](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1353-local_user_index)] - 2024-09-20
 
 ### Changed

diff --git a/backend/canisters/market_maker/CHANGELOG.md b/backend/canisters/market_maker/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1319](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1319-market_maker)] - 2024-09-02
 

diff --git a/backend/canisters/neuron_controller/CHANGELOG.md b/backend/canisters/neuron_controller/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1303](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1303-neuron_controller)] - 2024-08-22
 

diff --git a/backend/canisters/notifications/CHANGELOG.md b/backend/canisters/notifications/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1321](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1321-notifications)] - 2024-09-02
 

diff --git a/backend/canisters/notifications_index/CHANGELOG.md b/backend/canisters/notifications_index/CHANGELOG.md
@@ -5,6 +5,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [unreleased]
 
+### Changed
+
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
+
 ### Removed
 
 - Remove deprecated candid endpoints ([#6396](https://github.com/open-chat-labs/open-chat/pull/6396))

diff --git a/backend/canisters/online_users/CHANGELOG.md b/backend/canisters/online_users/CHANGELOG.md
@@ -5,6 +5,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [unreleased]
 
+### Changed
+
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
+
 ### Removed
 
 - Remove deprecated candid endpoints ([#6396](https://github.com/open-chat-labs/open-chat/pull/6396))

diff --git a/backend/canisters/proposals_bot/CHANGELOG.md b/backend/canisters/proposals_bot/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1307](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1307-proposals_bot)] - 2024-08-26
 

diff --git a/backend/canisters/registry/CHANGELOG.md b/backend/canisters/registry/CHANGELOG.md
@@ -9,6 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - When a token is added, disable existing token with the same ticker ([#6392](https://github.com/open-chat-labs/open-chat/pull/6392))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ### Removed
 

diff --git a/backend/canisters/storage_bucket/CHANGELOG.md b/backend/canisters/storage_bucket/CHANGELOG.md
@@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1325](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1325-storage_bucket)] - 2024-09-03
 

diff --git a/backend/canisters/storage_index/CHANGELOG.md b/backend/canisters/storage_index/CHANGELOG.md
@@ -12,6 +12,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1324](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1324-storage_index)] - 2024-09-03
 

diff --git a/backend/canisters/translations/CHANGELOG.md b/backend/canisters/translations/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 
 - Serialize large integers as strings when using MessagePack ([#6315](https://github.com/open-chat-labs/open-chat/pull/6315))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ## [[2.0.1326](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1326-translations)] - 2024-09-03
 

diff --git a/backend/canisters/user/CHANGELOG.md b/backend/canisters/user/CHANGELOG.md
@@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [unreleased]
 
+### Changed
+
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
+
 ## [[2.0.1357](https://github.com/open-chat-labs/open-chat/releases/tag/v2.0.1357-user)] - 2024-09-20
 
 ### Added

diff --git a/backend/canisters/user_index/CHANGELOG.md b/backend/canisters/user_index/CHANGELOG.md
@@ -8,7 +8,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ### Changed
 
-No auth check for `award_external_achievement` in test mode - ([#6397](https://github.com/open-chat-labs/open-chat/pull/6397))
+- No auth check for `award_external_achievement` in test mode ([#6397](https://github.com/open-chat-labs/open-chat/pull/6397))
+- Increase max stable memory read / write buffer size ([#6440](https://github.com/open-chat-labs/open-chat/pull/6440))
 
 ### Removed
 

diff --git a/backend/libraries/stable_memory/src/lib.rs b/backend/libraries/stable_memory/src/lib.rs
@@ -5,14 +5,14 @@ use ic_stable_structures::Memory;
 use std::cmp::min;
 use std::io::{Read, Write};
 
-const MAX_READER_WRITER_BUFFER_SIZE: usize = 1024 * 1024; // 1MB
+const MAX_READER_WRITER_BUFFER_SIZE: usize = 4 * 1024 * 1024; // 4MB
 
 pub fn get_reader<M: Memory>(memory: &M) -> impl Read + '_ {
     BufferedReader::new(buffer_size(memory), Reader::new(memory, 0))
 }
 
 pub fn get_writer<M: Memory>(memory: &mut M) -> impl Write + '_ {
-    BufferedWriter::new(MAX_READER_WRITER_BUFFER_SIZE, Writer::new(memory, 0))
+    BufferedWriter::new(buffer_size(memory), Writer::new(memory, 0))
 }
 
 fn buffer_size<M: Memory>(memory: &M) -> usize {