Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement S3 bucket and Snowflake architecture for raw PeMS data #10

Merged
merged 4 commits into from
Oct 24, 2023

Conversation

ian-r-rose
Copy link
Member

@ian-r-rose ian-r-rose commented Oct 17, 2023

Fixes #7, fixes #8

This creates a publicly-readable bucket (I can change that for development if folks feel nervous about that), as well as a service account with write access to that bucket. I've also created our reference Snowflake architecture here.

@ian-r-rose ian-r-rose self-assigned this Oct 17, 2023
@ian-r-rose ian-r-rose changed the title Implement S3 bucket for raw PeMS data Implement S3 bucket and Snowflake architecture for raw PeMS data Oct 18, 2023
@ian-r-rose ian-r-rose marked this pull request as ready for review October 18, 2023 18:23
@ian-r-rose
Copy link
Member Author

ian-r-rose commented Oct 18, 2023

I should note: I've done everything here in AWS us-west-2 (both the S3 bucket and the Snowflake account)

Caltrans has expressed some interest in using Azure in the long-term, and we may tear down this infrastructure and rebuild it there. However, ODI is not currently set up with Azure, so in the short term I feel it's best to move forward with AWS with the knowledge that we can change that in the future. Does that seem reasonable @jasonlally?

@jasonlally
Copy link

Yeah that seems reasonable, what follow up should I have on the Azure request. Does that still need budget for approval?

@ian-r-rose
Copy link
Member Author

Yeah that seems reasonable, what follow up should I have on the Azure request. Does that still need budget for approval?

I pinged the relevant issue. It sounds like we still will need a budget estimate

@ian-r-rose ian-r-rose merged commit 7801dc5 into main Oct 24, 2023
1 check passed
@ian-r-rose ian-r-rose deleted the create-s3-infra branch February 7, 2024 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Set up this repository Create S3 infrastructure in preparation to receive data
2 participants