Skip to content

Latest commit

 

History

History
292 lines (259 loc) · 23.3 KB

s3.md

File metadata and controls

292 lines (259 loc) · 23.3 KB
Raw

S3

Storage Classes

  • S3 Standards (default):
    • The objects are stored in at least 3 AZs
    • Provides eleven nines of availability
    • The replication is using MD5 file checks together with CRCs to detect object issues
    • When objects are stored in S3 using the API, a HTTP 200 OK response is provided
    • Billing:
      • GB/month of data stored in S3
      • A dollar for GB charge transfer out (in is free)
      • Price per 1000 requests
      • No specific retrieval fee, no minimum duration, no minimum size
    • S3 standard makes data accessible immediately, can be used for static website hosting
    • Should be used for data frequently accessed
  • S3 Standard-IA:
    • Shares most of the characteristics of S3 standard: objects are replicated in 3 AZs, durability is the same, availability is the same, first byte latency is the same, objects can be made publicly available
    • Billing:
      • It is more cost effective for storing data
      • Data transfer fee is the same as S3 standard
      • Retrieval fee: for every GB of data there is a retrieval fee, overall cost may increase with frequent data access
      • Minimum duration charge: we will be billed for a minimum of 30 days, minimum capacity of the objects being 128KB (smaller objects will be billed as being 128 KB)
      • Should be used for long lived data where data access is infrequent
  • S3 One Zone-IA:
    • Similar to S3 standard, but cheaper. Also cheaper than S3 standard IA
    • Data stored using this class is only stored in one region
    • Billing:
      • Similar to S3 standard IA: similar minimum duration fee of 30 days, similar billing for smaller objects and also similar retrieval fee per GB
      • Same level of durability (if the AZ does not fail)
      • Data is replicated inside one AZ
    • Since data is not replicated between AZs, this storage class is not HA. It should be used for non-critical data or for data that can be reproduced easily
  • S3 Glacier Instant Retrieval:
    • It like S3 Standard-IA, but with cheaper storage, more expensive retrieval, longer minimums
    • Recommended for data that is infrequently accessed (once per quarter), but it still needs to be retrieved instantly
    • Minimum storage duration charge is 90 days
  • S3 Glacier Flexible Retrieval (formerly knowns as S3 Glacier):
    • Same data replication as S3 standard and S3 standard IA
    • Same durability characteristics
    • Storage cost is about 1/6 of S3 standard
    • S3 objects stored in Glacier should be considered cold objects (should not be accessed frequently)
    • Objects in Glacier class are just pointers to real objects and they can not be made public
    • In order to retrieve them, we have to perform a retrieval process:
      • A job that needs to be done to get access to objects
      • Retrievals processes are billed
      • When objects are retrieved for Glacier, they are temporarily stored in standard IA and they are removed after a while. We can retrieve them permanently as well
    • Retrieval job types:
      • Expedited: objects are retrieved in 1-5 minutes, retrieval process being the most expensive
      • Standard: data is accessible at 3-5 hours
      • Bulk: data is accessible at 5-12 hours at lower cost
    • Glacier has a 40KB minimum billable size and a 90 days minimum duration for storage
    • Glacier should be used for data archival (yearly access), where data can be retrieved in minutes to hours
  • S3 Glacier Deep Archive:
    • Deep Archive represents data in a frozen state
    • Has a 40KB minimum billable data size and a 180 days minimum duration for data storage
    • Objects can not be made publicly available, data access is similar to standard Glacier class
    • Restore jobs are longer:
      • Standard: up to 12 hours
      • Bulk: up to 48 hours
    • Should be used for archival which is very rarely accessed
  • S3 Intelligent-Tiering:
    • It is a storage class containing 5 different tiering a storage
    • Objects that are access frequently are stored in the Frequent Access tier, less frequently accessed objects are stored in the Infrequent Access tier. Objects accessed very infrequently will be stored in either Archive or Deep Archive tier
    • We don't have to worry for moving objects over tier, this is done by the storage class automatically
    • Intelligent tier can be configured, archiving data is optional and can be enabled/disabled
    • There is no retrieval cost for moving data between frequent and infrequent tiers, we will be billed based on the automation cost per 1000 objects
    • S3 Intelligent-Tiering is recommended for unknown or uncertain data access usage
  • Storage classes comparison:
S3 Standard S3 Intelligent-Tiering S3 Standard-IA S3 One Zone-IA S3 Glacier Instant S3 Glacier Flexible S3 Glacier Deep Archive
Designed for durability 99.999999999% (11 9's) 99.999999999% (11 9's) 99.999999999% (11 9's) 99.999999999% (11 9's) 99.999999999% (11 9's) 99.999999999% (11 9's) 99.999999999% (11 9's)
Designed for availability 99.99% 99.9% 99.9% 99.5% 99.9% 99.99% 99.99%
Availability SLA 99.9% 99% 99% 99% 99% 99.9% 99.9%
Availability Zones ≥3 ≥3 ≥3 1 ≥3 ≥3 ≥3
Minimum capacity charge per object N/A N/A 128KB 128KB 128KB 40KB 40KB
Minimum storage duration charge N/A 30 days 30 days 30 days 90 days 90 days 180 days
Retrieval fee N/A N/A per GB retrieved per GB retrieved per GB retrieved per GB retrieved per GB retrieved
First byte latency milliseconds milliseconds milliseconds milliseconds milliseconds select minutes or hours select hours
Storage type Object Object Object Object Object Object Object
Lifecycle transitions Yes Yes Yes Yes Yes Yes Yes

S3 Lifecycle Configuration

  • We can create lifecycle rules on S3 buckets which can move objects between tiers or expire objects automatically
  • A lifecycle configuration is a set of rules applied to a bucket or a group of objects in a bucket
  • Rules consist of actions:
    • Transition actions: move objects from one tier to another after a certain time
    • Expiration actions: delete objects or versions of objects
  • Objects can not be moved based on how much they are accessed, this can be done by the intelligent tiering. We can move objects based on time passed
  • By moving objects from one tier to another we can save costs, expiring objects also will help saving costs
  • Transitions between tiers: Transition between tiers
  • Considerations:
    • Files from One Zone-IA can transition to Glacier Flexible, or Deep Archive, NOT into Glacier Instant retrieval
    • Smaller objects cost more in Standard-IA, One Zone-IA, etc.
    • An object needs to remain for at least 30 days in standard tier before being able to be moved to infrequent tiers (objects can be uploaded manually in infrequent tiers)
    • A single rule can not move objects instantly from standard IA to infrequent tiers and then to Glacier tiers. Objects have to stay for at least 30 days in infrequent tiers before being able to be moved by one rule only. In order ot overcome this, we can define 2 different rules

S3 Replication

  • 2 types of replication are supported by S3:
    • Cross-Region Replication (CRR)
    • Same-Region Replication (SRR)
  • Both types of replication support same account replication and cross-account replication
  • If we configure cross-account replication, we have to define a policy on the destination account to allow replication from the source account
  • We can create replicate all objects from a bucket or we can create rules for a subset of objects. We can filter objects to replicate based on prefix or tags or both
  • We can specify which storage class to use for an object in the destination bucket (default: use the same class)
  • We can also define the ownership of the objects in the destination bucket. By default it will be the same as the owner in the source bucket
  • Replication Time Control (RTC): if enabled ensures a 15 minutes replication of objects
  • Replication consideration:
    • By default replication is not retroactive: only newer objects are replicated after the replication is enabled
    • Versioning needs to be enabled for both source and destination buckets
    • Batch replication can be used to replicate existing objects. It needs to be specifically configured. If it is not, replication wont be retroactive
    • Replication by default one-way only, source => destination. There is an option to use bi-directional replication, but this has to be configured
    • Replication is capable of handling objects encrypted with SSE-S3 and SSE-KMS (with extra configuration). SSE-C (customer managed keys) is also supported, historically it was incompatible
    • Replication requires for the owner of source bucket needs permissions on the objects which will be replicated
    • System events will not be replicated, only user events
    • Any objects in the Glacier and Glacier Deep Archive will not be replicated
    • By default, deletion are not replicated. We can enable replication for deletion events
  • Replication use cases:
    • SRR:
      • Log aggregation
      • PROD and Test sync
      • Resilience with strict sovereignty
    • CRR
      • Global resilience improvements
      • Latency reduction

S3 Encryption

  • Buckets aren't encrypted, objects inside buckets are encrypted
  • Encryption at rest types:
    • Client-Side encryption: data is encrypted before it leaves the client
    • Server-Side encryption: data is encrypted at the server side, it is sent on plain-text format from the client
  • Both encryption types use encryption in-transit for communication
  • Server-side encryption is mandatory, we cannot store data in S3 without being encrypted
  • There are 3 types of server-side encryption supported:
    • SSE-C: server-side encryption with customer-provided keys
      • Customer is responsible for managing the keys, S3 managed encryption/decryption
      • When an object is put into S3, we need to provide the key utilized
      • The object will be encrypted by the key, a hash is generated and stored for the key
      • The key will be discarded after the encryption is done
      • In case of object retrieval, we need to provide the key again
    • SSE-S3 (default): server-side encryption with Amazon S3-managed keys
      • AWS handles both the encryption/decryption and the key management
      • When using this method, S3 creates a master key for the encryption process (handled entirely by S3)
      • When an object is uploaded an unique key is used for encryption. After the encryption, this unique key is encrypted as well with the master key and the unencrypted key is discarded. Both the key and the object are stored together
      • For most situations, this is the default type of encryption. It uses AES-256 algorithm, they key management is entirely handled bt S3
    • SSE-KMS: Server-side encryption with customer-managed keys stored in AWS Key Management Service (KMS)
      • Similar to SSE-S3, but for this method the KMS handles stored keys
      • When an object is uploaded for the first time, S3 will communicate with KMS and creates a customer master key (CMK). This is default master key used in the future
      • When new objects are uploaded AWS uses the CMK to generate individual keys for encryption (data encryption keys). The data encryption key will be stored along with the object in encrypted format
      • We don't have to use the default CMK provided by AWS, we can use our own CMK. We can control the permission on it and how it is regulated
      • SSE-KMS provides role separation:
        • We can specify who can access the CMK from KMS
        • Administrators can administers buckets but they may not have access to KMS keys
  • Default Bucket Encryption:
    • When an object is uploaded, we can specify which server-side encryption to be used by adding a header to the request: x-amz-server-side-encryption
    • Values for the header:
      • To use SSE-S3: AES256
      • To use SSE-KMS: aws:kms
    • All Amazon S3 buckets have encryption configured by default, and all new objects that are uploaded to an S3 bucket are automatically encrypted at rest
    • Server-side encryption with Amazon S3 managed keys (SSE-S3) is the default encryption configuration for every bucket in Amazon S3, this can be overridden in a PUT request with x-amz-server-side-encryption header

S3 Bucket Keys

  • Each object in a bucket is using a uniq data-encryption key (DEK)
  • AWS uses the bucket's KMS key to generate this data-encryption key
  • Calls to KMS have a cost and levels where throttling occurs: 5500/10_000/50_000 PUT/sec depending on region
  • Bucket keys:
    • A time limited bucket key is used to generate DEKs within S3
    • KMS generates a bucket key and gives it to S3 to use to generate DEKs for each upload, offloading the load from KMS to S3
    • Reduces the number of KMS API calls => reduces the costs/increases scalability
  • Using bucket keys is not retroactive, it will only affect objects after bucket keys are enabled
  • Thing to keep in mind after enabling bucket keys:
    • CloudTrail KMS event logs will show the bucket ARN instead of the object ARN
    • Fewer CloudTrail events of KMS will be in the logs (since work is offloaded to S3)
    • Bucket keys work with SRR and CRR; the object encryption settings are maintained
    • If we replicate plaintext to a bucket using bucket keys, the object is encrypted at the destination side; this can result in ETAG changes on the object

S3 Presigned URLs

  • Is a way to give other people access to our buckets using our credentials
  • An IAM admin can generate a presigned URL for a specific object using his credentials. This URL will have an expiry date
  • The presigned URL can be given to unauthenticated uses in order to access the object
  • The user will interact with S3 using the presigned URL as if it was the person who generated the presigned URL
  • Presigned URLs can be used for downloads and for uploads
  • Presigned URLs can be used for giving direct access private files to an application user offloading load from the application. This approach will require a service account for the application which will generate the presigned URLs
  • Presigned URL considerations:
    • We can create a presigned ULR for objects we don't have access to
    • When using the URL, the permissions match the identity which generated it. The permissions are evaluated at the moment of accessing the object (it might happen the the identity had its permissions revoked, meaning we wont have access to the object either)
    • We should not generate presigned URLs generated on temporary credentials (assuming an IAM role). When the temporary credentials are expired, the presigned URL will stop working as well. Recommended to use long-term identities such as an IAM user

S3 Select and Glacier Select

  • Are ways to retrieve parts of objects instead of entire objects
  • S3 can store huge objects (up to 5 TB)
  • Retrieving a huge objects will take time and consume transfer capacity
  • S3/Glacier provides services to access partial objects using SQL-like statements to select parts of objects
  • Both S3 Select and Glacier selects supports the following formats: CSV, JSON, Parquet, BZIP2 compression for CSV and JSON

S3 Access Points

  • Improves the manageability of objects when buckets are used for many different teams or they contain objects for a large amount of functions
  • Access Points simplify the process of managing access to S3 buckets/objects
  • Rather than 1 bucket (1 bucket policy) access we can create many access points with different policies
  • Each access point can be limited from where it can be accessed, and each can have different network access controls
  • Each access point has its own endpoint address
  • We can create access point using the console or the CLI using aws s3control create-access-point --name < name > --account-id < account-id > --bucket < bucket-name >
  • Any permission defined on the access point needs to be defined on the bucket policy as well. We can do delegation, by defining wide access permissions in the bucket policy and granular permissions on the access point policy

S3 Block Public Access

  • The Amazon S3 Block Public Access feature provides settings for access points, buckets, and accounts to help manage public access to Amazon S3 resources
  • The settings we can configure with the Block Public Access Feature are:
    • IgnorePublicAcls: this prevents any new ACLs to be created or existing ACLs being modified which enable public access to the object. With this alone existing ACLs will not be affected
    • BlockPublicAcls: Any ACLs actions that exist with public access will be ignored, this does not prevent them being created but prevents their effects
    • BlockPublicPolicy: This prevents a bucket policy containing public actions from being created or modified on an S3 bucket, the bucket itself will still allow the existing policy
    • RestrictPublicBuckets: this will prevent non AWS services or authorized users (such as an IAM user or role) from being able to publicly access objects in the bucket

S3 Cost Saving Options

  • S3 Select and Glacier Select: save in network a CPU cost by retrieving ony the necessary data
  • S3 Lifecycle Rules: transition objects between tiers
  • Compress objects to save space
  • S3 Requester Pays:
    • In general, bucket owners pay for all Amazon S3 storage and data transfer costs associated with their bucket
    • With Requester Pays buckets, the requester instead of the bucket owner pays the cost of the request and the data download from the bucket
    • The bucket owner always pays the cost of storing data
    • Helpful when we want to share large datasets with other accounts
    • Requires a bucket policy
    • If an IAM role is assumed, the owner account of that role pays for the request!

S3 Object Lock

  • Object Lock can be enabled on newly created S3 buckets. For existing ones in order to enable Object Lock we have to contact AWS support
  • Versioning will be also enabled when Object Lock is enabled
  • Object Lock can not be disabled, versioning can not be suspended when Object Lock is active on the bucket
  • Object Lock is a Write-Once-Read-Many (WORM) architecture: when an object is written, can not be modified. Individual versions of objects are locked
  • There are 2 ways S3 managed object retention:
    • Retention Period
    • Legal Hold
  • Object versions can have both retention period and legal hold enabled; can have only one of those enabled or none of them
  • Object Lock retentions can be individually defined on object versions, a bucket can have default Object Lock settings

Retention Period

  • When a retention period is enabled on an object, we specify the days and years for the period
  • The retention period will end after the period
  • There are 2 types of retention period modes:
    • Compliance mode:
      • Object can not be adjusted, deleted or overwritten. The retention period can not be reduced, the retention mode can not be adjusted even by the account root user
      • Should be used for compliance reasons
    • Governance mode:
      • Objects can not be adjusted, deleted or overwritten, but special permissions can be added to some identities to allow for the lock setting to be adjusted
      • This identities should have the s3:BypassGovernanceRetention permission
      • The governance mode can be overwritten when passing x-amz-bypass-governance-retention:true header (header is default for console ui)

Legal Hold

  • We don't set a retention period for this type of retention, Legal Hold can be on or off for specific versions of an object
  • We can't delete or overwrite an object with Legal Hold
  • An extra permission is required when we want to add or remove the Legal Hold on an object: s3:PutObjectLegalHold
  • Legal Hold can be used for preventing accidental removals

S3 Transfer Accelerate

  • Used to transfer files into S3. Enables fast, easy, and secure transfers of files over long distances between our client and an S3 bucket
  • Takes advantage of the globally distributed edge locations in Amazon CloudFront
  • We might want to use Transfer Acceleration on a bucket for various reasons:
    • We upload to a centralized bucket from all over the world
    • We transfer gigabytes to terabytes of data on a regular basis across continents
    • We can't use all of our available bandwidth over the internet when uploading to Amazon S3
  • To use Transfer Accelerate, it must be enabled on the bucket. After we enable Transfer Acceleration on a bucket, it might take up to 20 minutes before the data transfer speed to the bucket increases

S3 Object Lambda

  • With Amazon S3 Object Lambda, we can add our own code to Amazon S3 GET, LIST, and HEAD requests to modify and process data as it is returned to an application
  • S3 Object Lambda uses AWS Lambda functions to automatically process the output of standard S3 GET, LIST, or HEAD requests
  • After we configure a Lambda function, we attach it to an S3 Object Lambda service endpoint, known as an Object Lambda Access Point
  • The Object Lambda Access Point uses a standard S3 access point
  • When we send a request to your Object Lambda Access Point, Amazon S3 automatically calls your Lambda function

Hosting Static Site on S3

  • We can host a static site on S3
  • To host a static site in a bucket we must enable static website hosting, configure an index document, and set permissions
  • We should also make a bucket content public:
    • We should turn of Block Public Access settings
    • We should attach a bucket policy which allows public read on the objects
  • Amazon S3 website endpoints do not support HTTPS! If we want to use HTTPS, we can use Amazon CloudFront to serve a static website hosted on Amazon S3