feat : added aws s3 adapter and data storage trait #27

ocdbytes · 2024-06-25T13:22:11Z

S3 Implementation

Added data storage crate for getting and storing data on cloud
Added AWS S3 trait and added .env values

Issue : #24

This commit fixes the style issues introduced in 7e65299 according to the output from Rustfmt. Details: #27

deepsource-io · 2024-06-25T13:22:59Z

Here's the code health analysis summary for commits a342034..cc54f2b. View details on DeepSource ↗.

Analysis Summary

Analyzer	Status	Summary	Link
Rust	✅ Success		View Check ↗

💡 If you’re a repository administrator, you can configure the quality gates from the settings.

apoorvsadana · 2024-06-25T13:58:00Z

crates/orchestrator/src/data_storage/aws_s3/mod.rs

we're using - instead of _ in file and crate names as a convention

apoorvsadana · 2024-06-25T13:59:08Z

crates/orchestrator/src/data_storage/aws_s3/mod.rs

+const SNOS_OUTPUT_FILE_NAME: &str = "snos_output.json";
+const KZG_FILE_NAME: &str = "kzg.txt";


s3 shouldn't be aware of this, these details can be in the snos job. also, do we need to store kzg now? i was thinking we can calculate the proof when we calculate state diffs in update state only

apoorvsadana · 2024-06-25T13:59:27Z

crates/orchestrator/src/data_storage/aws_s3/mod.rs

+
+#[async_trait]
+impl DataStorage for AWSS3 {
+    async fn get_snos_data_for_block(&self, block_number: u128) -> Result<StarknetOsOutput, Error> {


same as above, S3 should have no idea on what snos is. s3 should only have get and put functions.

This commit fixes the style issues introduced in 59ce0a9 according to the output from Rustfmt. Details: #27

…xyz/madara-orchestrator into feat/data-storage-adapter

apoorvsadana · 2024-06-26T08:32:27Z

.env.example

so, I think, as long as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are in the env variables, aws sdk automatically fetches the config. like, we never initialise the SQS config in the orchestrator, it just works. although we do use a wrapper library as well so maybe that handles it automatically but i mostly think its the aws sdk itself. and in that case, i think we can just have one aws access key and secret for all AWS services

apoorvsadana · 2024-06-26T08:33:57Z

crates/orchestrator/src/data_storage/aws_s3/mod.rs

+
+#[async_trait]
+impl DataStorage for AWSS3 {
+    async fn get_data_for_block(&self, key: &str) -> Result<StarknetOsOutput, Error> {


it should be just get_data and it should return just bytes. if we hardcode StarknetOsOutput, I can't use S3 later on with the DA layer for example in case I want to store it in s3

apoorvsadana · 2024-06-26T08:34:04Z

crates/orchestrator/src/data_storage/aws_s3/mod.rs

+        Ok(json)
+    }
+
+    async fn put_data_for_block(&self, data: StarknetOsOutput, key: &str) -> Result<usize, Error> {


same as above

apoorvsadana · 2024-06-26T08:34:11Z

crates/orchestrator/src/data_storage/aws_s3/mod.rs

+//
+//     #[rstest]
+//     #[tokio::test]
+//     async fn test_put_data_in_bucket() {


is this pending?

This commit fixes the style issues introduced in c47e330 according to the output from Rustfmt. Details: #27

…xyz/madara-orchestrator into feat/data-storage-adapter

* feat : added aws s3 adapter and data storage trait * updated .env * style: format code with Rustfmt This commit fixes the style issues introduced in 7e65299 according to the output from Rustfmt. Details: madara-alliance#27 * feat : removed redundant functions * style: format code with Rustfmt This commit fixes the style issues introduced in 59ce0a9 according to the output from Rustfmt. Details: madara-alliance#27 * feat : added bucket tests for local testing * feat : renamed funcs and cleanup * style: format code with Rustfmt This commit fixes the style issues introduced in c47e330 according to the output from Rustfmt. Details: madara-alliance#27 * feat : removed deepsource.toml file * feat : lint fix * feat : added comments for functions and impls as suggested by deepsource * fix : deepsource doc recommendation fix * refactor --------- Co-authored-by: Arun Jangra <[email protected]> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* feat : added aws s3 adapter and data storage trait * updated .env * style: format code with Rustfmt This commit fixes the style issues introduced in 7e65299 according to the output from Rustfmt. Details: #27 * feat : removed redundant functions * style: format code with Rustfmt This commit fixes the style issues introduced in 59ce0a9 according to the output from Rustfmt. Details: #27 * feat : added bucket tests for local testing * feat : renamed funcs and cleanup * style: format code with Rustfmt This commit fixes the style issues introduced in c47e330 according to the output from Rustfmt. Details: #27 * feat : removed deepsource.toml file * feat : lint fix * feat : added comments for functions and impls as suggested by deepsource * fix : deepsource doc recommendation fix * refactor --------- Co-authored-by: Arun Jangra <[email protected]> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

Arun Jangra and others added 3 commits June 25, 2024 18:49

feat : added aws s3 adapter and data storage trait

d8cc33d

updated .env

7e65299

style: format code with Rustfmt

da3f3f5

This commit fixes the style issues introduced in 7e65299 according to the output from Rustfmt. Details: #27

ocdbytes requested a review from apoorvsadana June 25, 2024 13:24

apoorvsadana reviewed Jun 25, 2024

View reviewed changes

Arun Jangra and others added 4 commits June 25, 2024 19:40

feat : removed redundant functions

59ce0a9

style: format code with Rustfmt

a39102f

This commit fixes the style issues introduced in 59ce0a9 according to the output from Rustfmt. Details: #27

feat : added bucket tests for local testing

952015c

Merge branch 'feat/data-storage-adapter' of https://github.com/karnot…

40a79b3

…xyz/madara-orchestrator into feat/data-storage-adapter

apoorvsadana reviewed Jun 26, 2024

View reviewed changes

Arun Jangra and others added 9 commits June 26, 2024 15:49

feat : renamed funcs and cleanup

c47e330

style: format code with Rustfmt

4545216

This commit fixes the style issues introduced in c47e330 according to the output from Rustfmt. Details: #27

feat : removed deepsource.toml file

d7dd81a

Merge branch 'feat/data-storage-adapter' of https://github.com/karnot…

f8064d3

…xyz/madara-orchestrator into feat/data-storage-adapter

feat : lint fix

bc2a549

temp

568bae4

feat : added comments for functions and impls as suggested by deepsource

e2d4cf8

fix : deepsource doc recommendation fix

e6e3f10

refactor

cc54f2b

ocdbytes merged commit a4e6cba into main Jun 27, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat : added aws s3 adapter and data storage trait #27

feat : added aws s3 adapter and data storage trait #27

ocdbytes commented Jun 25, 2024

deepsource-io bot commented Jun 25, 2024 •

edited

Loading

Analysis Summary

apoorvsadana Jun 25, 2024

apoorvsadana Jun 25, 2024

apoorvsadana Jun 25, 2024

apoorvsadana Jun 26, 2024

apoorvsadana Jun 26, 2024

apoorvsadana Jun 26, 2024

apoorvsadana Jun 26, 2024

		const SNOS_OUTPUT_FILE_NAME: &str = "snos_output.json";
		const KZG_FILE_NAME: &str = "kzg.txt";

feat : added aws s3 adapter and data storage trait #27

feat : added aws s3 adapter and data storage trait #27

Conversation

ocdbytes commented Jun 25, 2024

S3 Implementation

deepsource-io bot commented Jun 25, 2024 • edited Loading

Analysis Summary

apoorvsadana Jun 25, 2024

Choose a reason for hiding this comment

apoorvsadana Jun 25, 2024

Choose a reason for hiding this comment

apoorvsadana Jun 25, 2024

Choose a reason for hiding this comment

apoorvsadana Jun 26, 2024

Choose a reason for hiding this comment

apoorvsadana Jun 26, 2024

Choose a reason for hiding this comment

apoorvsadana Jun 26, 2024

Choose a reason for hiding this comment

apoorvsadana Jun 26, 2024

Choose a reason for hiding this comment

deepsource-io bot commented Jun 25, 2024 •

edited

Loading