Skip to content

Latest commit

 

History

History
176 lines (144 loc) · 7.97 KB

2.2. Azure Storage.md

File metadata and controls

176 lines (144 loc) · 7.97 KB
Raw

Azure Storage

  • PaaS (Platform as a Service)
  • Store files, messages, tables, and other types of information.
    • Subservices: Blob, File, Queue, and Table.
  • It's also used by IaaS virtual machines, and other PaaS cloud services.
  • Three main roles:
    • Storage for Virtual Machines
      • Disks are persistent block storage for Azure IaaS virtual machines.
        • Azure disks handles creation of Storage account for you.
      • Files are fully managed file shares in the cloud.
    • Unstructured Data
      • Blobs are highly scaleable, REST based cloud object store.
      • Data Lake Store is Hadoop Distributed File System (HDFS) as a service.
      • Tables are a key/value, auto-scaling NoSQL store.
      • Cosmos DB is a globally distributed database service
    • Structured Data
      • Azure SQL DB is fully managed database-as-a-service built on SQL
  • A snapshot is a read-only version of a storage that's taken at a point in time.
    • Can be read, copied or deleted but not modified.
    • Provide a way to back up a blob as it appears at a moment in time

Azure Storage Explorer

  • Access multiple accounts and subscriptions.
  • Can manage Blob, Queue, Table, File, Cosmos DB and Data Lake storage.
    • Obtain shared access signature (SAS) keys.
  • Authorization: SAS or Azure account.

Storage types

Queue

  • More basic than ServiceBus without overhead
  • 💡 Good for >80GB

Blobs

  • Blob permissions
    • Private: Authentication (RBAC or storage key) to access
    • Blob: Anonymous access to blobs
    • Container: Anonymous access to blobs + containers
  • Blob types
    • Block blob is aimed at streaming and storing of objects such as media files and documents. Sequentially accessed.
    • Page blob is optimized for random reads and writes. E.g. SQL & VMs uses it.
    • Append blob is same as a block blob except data can only be added to the end of the blob. Blocks elsewhere cannot be deleted nor modified
  • Blob storage access tiers
    • Hot: More frequently accessed
    • Cool: Less frequently accessed
    • Archive: Minimum frequency. Only in v2

Azure Storage Accounts

  • Around 99.9% SLA
  • Kinds
    • Legacy => General purpose v1, Blob storage
      • 💡Can easily be migrated to v2
    • General purpose v2
      • Lowest price, pay per GB

General-purpose storage account

  • Tables, queues, files, blobs and Azure virtual machine disks under a single account
  • Performance tiers:
    • Standard: tables, queues, files, blobs, and Azure virtual machine disks
      • Magnetic drives (HDD) and provide the lowest cost per GB
      • Best for apps with bulk storage or where data is accessed infrequently.
    • Premium: Supports only Azure virtual machine disks
      • SSD, low latency performance
      • Best for I/O-intensive applications, like databases
    • ❗ Not possible to convert after deployment

Endpoints

  • Default domain name e.g. http://mystorageaccount.**blob**.core.windows.net
  • Two ways to configure
    • Direct CNAME mapping
      • Create DNS record (NS record as TXT or MX) for Azure URL
        • ❗ For subdomains you use A record.
      • Causes minor downtime as the domain is updated.
    • Intermediary mapping with asverify
      • No downtime
      • First create CNAME (in 3rd party) e.g. azverify.docs.amk.com
        • Put it in Azure and it gets verified
      • Then create CNAME (in 3rd party) docs.amk.com
      • You can now delete azverify CNAME record.

Creating accounts

  • You can use VNet to restrict the storage to only certain IP address subnets or VNets.
  • ❗ Location must match the location of the resource group

Data Replication

  • Replication strategies
    • LRS – Locally Redundant Storage
      • Copies: 1 copy in within a single facility in same region
    • ZRS – Zone Reduntant Storage
      • Copies: 3 copies in 3 different availability zones in same region
      • 📝 Replicates synchronously
    • Geo-redundant Storage (GRS)
      • Copies: In Another data center in a secondary (paired) region
      • 📝 Replicates asynchronously.
      • ❗ Not available in all regions
    • Read access geo redundant storage (RA-GRS)
      • Copies: As in GRS
      • Read access to the replicate.
        • Blob service is myaccount.blob.core.windows.net, then your secondary endpoint is myaccount-secondary.blob.core.windows.net.
        • The access keys for your storage account are the same for both the primary and secondary endpoints.
  • 💡 You can change your replication strategy later on.
    • From LRS: Free
    • GRS to RA-GRS: incur egress bandwidth charge (one time cost initially)
  • ❗ If you create availability sets for your virtual machines, then Azure uses Zone-redundant Storage (ZRS).
  • ❗ If you select Premium performance only LRS replication will be available.

Billing considerations

  • Storage costs (per-gigabyte), data access costs, and transaction costs increases as the tier gets cooler.
  • In GRS and RA-GRS, geo-replication data transfer incurs a per-gigabyte charge.
  • Outbound data transfer costs (outside of Azure region) incur billing for bandwidth usage on a per-gigabyte basis.
  • Changing tier from cool to hot costs as reading cool data, from hot to cool costs as writing data to cool tier.

Storing & accessing data

AzCopy

  • Command-line utility designed for copying data to/from Microsoft Azure Blob, File, and Table storage

Azure Import and Export Service

  • Import: Transfer to Azure Blob storage (block and page blobs) and Azure Files by shipping disk drives to an Azure data center.
    • Flow
      1. The customer prepares the hard drives using Import/Export Client Tool (WAImportExport Tool) that facilitates the process & encrypts the drive with BitLocker.
      2. The customer creates an import job using the Azure Portal / REST API.
      3. The customer ships the hard drives to the data center and enters tracking number in Azure.
      4. Hard drives are shipped back to the customer after the data is imported.
  • Export Transfer from Azure storage to hard disk & ship to on-promise sites. You can export Block, Page and Append blobs from Azure.
    • Requires you to ship empty disks.
    • Costs extra egress charges when data is exported.
    • Flow
      1. Customer creates export job with address & blob container path(s).
      2. The drives are encrypted with BitLocker; the keys are available via the Azure portal.
  • When?
    • Uploading/downloading is slow or costs much
    • E.g. migrating data to the cloud, content distribution to customers, backup, data recovery
  • ❗ Only specific types of disks are supported.
  • ❗ Many regions are not available (US only now).

Security

Network configuration

  • As default it accepts requests from any client, in Settings->Firewalls and virtual networks you can limit networks by choosing Selected Networks.
  • You can then add network rule in VNet to allow access to service endpoint of the Storage.
    • When planning for disaster recovery during a regional outage, you should create the VNets in the paired region in advance.
  • You can also set-up IP rules in Storage Account -> Firewalls and Virtual Networks -> Firewall -> Address range for cloud and hybrid scenarios.
    • ❗ Only allowed for public ip addresses

Shared access signature

  • Shared access policy includes a key (signatuer) to be used in query string to requests to the storage.
  • The policy can grant access to everything or can limit the access to specific operations (e.g. only read or writes).

Monitoring storage

  • Capacity metrics
    • Sub-service specific
      • E.g. for blob: Blob Capacity, Blob Container Count, and Blob Count
    • Sent every hour, refreshed daily.
  • Transaction metrics
    • Sent every minute from Azure Storage to Azure Monitor.
    • Available at both account and service level.
    • E.g. transactions, egress, ingress, availability
  • Metrics and logs are stored in the same storage account
  • Requires you to enable logging.
  • ❗ Not supported for premium storage accounts as they work differently and for Azure virtual machines