Andy
Andy DevOps professional. Always Learning. The Odin Project core team. Marathon Runner, Coffee Drinker, Day Hiker.

S3 (Simple Storage Service)

S3 (Simple Storage Service)

S3 is used to store flat files (objects) like PDFs and Word Documents. It can also be used to host static websites.

The Basics

S3 is Object based:

  • Key (Object Name)
  • Value (Data)
  • Version ID
  • Metadata
  • Subresources:
    • ACLs
    • Torrent
  • Built for 99.99% Availability, AWS guarantee 99.9% availability, 11x9s Durability
  • Read after write for PUTS of new Objects (New objects uploaded are available immediatly)
  • Eventual Consistency for overwrite PUTS and DELETES (May take some time for changes to a file to propogate)

Storage Tiers

  • S3 Standard => 99.99% availibility, 11x9 durability. Stored in multiple locations designed to sustain lost of 2 facilities concurrently
  • S3-IA (Infrequent Access) => Lower storage fee, but charged retrieval fee
  • S3 One Zone - IA => Not replicated across multiple AZs. Lower cost than S3-IA.
  • Glacier => Cheapest. Archival Only
    • Glacier Expedited => Restored within a few minutes
    • Glacier Standard => 3-5 hour restore
    • Glacier Bulk => 5-12 hour restore

StorageTierInfographic

Pricing

You’re charged for all of the following:

  • Storage
  • Requests
  • Storage Management
  • Data Transfer
  • Transfer Acceleration => Upload files to CloudFront Edge Location, then “syncs” to S3 via AWS Backbone

Security & Encryption

  • By default, all buckets are private.
  • You can set up access controls by using:
    • Bucket Policy => Rules apply to bucket
    • ALCs => Rules apply to users
  • Buckets can create access logs which can be saved in other S3 buckets

Encryption Types:

  • Client side encryption
  • In Transit => SSL/TLS
  • At Rest (Server Side Encryption):
    • Client-Side Encryption
      • SSE-S3 => Each object encrypted with AWS S3 Managed Keys (AES-256 bit encryption)
      • SSE-KMS => Provides audit trail, advanced usage
      • SSE-C => Customer Provided Keys

Versioning

Tracks and records all versions of an object. Similar to git.

  • Cannot be disabled, can only be suspended
  • MFA Authentication can be enabled for deleting an object

Cross Region Replication

Copies objects from one region to another for redundancy.

  • Versioning must be enabled for both buckets
  • Files in existing bucket ARE NOT replicated automatically
  • New files to existing buckets are replicated automatically
  • DELETE markers are not replicated
  • Deleting versions (or delete markers) are not replicated

Lifecycle Management (S3-IA & Glacier)

Set up rules to determine which storage tier to use. For example you can set it up such that after 30 days in S3, the object is moved to S3-IA, then, 30 days later (60 days after creation) the object is moved to Glacier.

  • Can be used to permanently delete objects

Storage Gateway

Service that connects on-prem software with cloud-based storage. It provides integration between on-prem datacenters and AWS S3.

You download the Storage Gateway VM in your datacenter, then configure it to your use cases

There are four types of Storage Gateways

  • File Gateway (NFS) => Files stored in S3. Gateway Appliance/Bucket is mounted on local OS as NFS Storage.
  • Volume Gateways => Block-Based Storage. Use iSCSI Adapter. Backed up to S3 as point-in-time snapshots. Snapshots are incremental. Two types of Volume Gateways:
    • Stored Volumes => All data stored on-prem while being backed up to AWS.
      • Use when low-latency access is needed to data
      • Can store 1GB to 16 TB
    • Cached Volumes => Use S3 as primary data storage. Only most frequent data is stored on-prem.
      • No need for large storage arrays on-prem
      • Stores 1GB to 32 TB
  • Tape Gateway (VTL) => Uses existing backup system (Veeam, NetBackup, BackupExec, etc) to store virtual tapes in S3

Transfer Acceleration

Utilizes CloudFront Edge Network to accelerate your S3 uploads.

myexample.s3-accelerate.amazonaws.com

User connects to closest edge location, then that edge location uses the AWS Backbone to optimize data transfer. The AWS backbone is a lot faster than the global internet.

Static Web hosting

S3 can be used to host static websites. You will need to enable public access to the bucket.

The URL of the website will be: bucketname.s3-website.amazonaws.com

To set up the hosting, click the “Properties” tab in the bucket. Then folow the “Static Website Hosting” dialog. You can also add a bucket policy so that everything in the bucket is public by default.

S3 Summary & Exam Tips

  • S3 is object based storage. It alows you to upload files.
  • Files can be 0Bytes to 5TB, Unlimited Storage. You’re charged for what you use
  • Fies are stored in buckets
  • S3 namespace is universal - Buckets must have unique name
  • URL Format: https://s3-region-1.amazonaws.com/bucketname
    • Example: https://s3-us-west-1.amazonaws.com/myfirstbucket
  • Read after write consistency for PUTS; Eventual consistancy for overwrite PUTS and DELETES
  • Storage Tiers => S3, S3-IA, S3-OneZone-IA, Glacier
  • Once enabled, versioning cannot be disabled. It can only be suspended.
  • By Defaul, all buckets are PRIVATE. You can make them public via Bucket Policies and ACLs
  • S3 buckets are be configured to create access logs
  • Encryption - In transit && At Rest
  • Storage Gateway - 4 types: File Gateway, Volume Gateway (2 types: Stored Volumes & Cached Volumes), Virtual Tape Libraries
  • Snowball
  • S3 Transfer Acceleration
  • Can host STATIC websites on S3
  • Successful write to S3 results in HTTP 200
  • Enable Multipart uploads to decrease upload time
  • Read the S3 FAQ