Enable AWS S3 Transfer acceleration for Amplify SDK

[Ankish]

State your question
Enable AWS S3 Transfer acceleration for Amplify SDK request or provide a alternative way to implement the same with AWS SDK.

Which AWS Services are you utilizing?
AWS / Amplify / iOS SDK

Provide code snippets (if applicable)
Currently using Amplify to upload file:

let option = StorageUploadFileRequest.Options(accessLevel: StorageAccessLevel.private, targetIdentityId: nil,
metadata: [Utils.createTimeStampKey : "(createTime)"], contentType: nil, pluginOptions:nil)

Amplify.Storage.uploadFile(key: videoId, local: localURL, options: option, progressListener: { progress

Environment(please complete the following information):

• Amplify SDK Version: 2.33.1
• Dependency Manager: Cocoapods
• Swift Version : 5.0

Device Information (please complete the following information):

• Device: All
• iOS Version: All
• Specific to simulators: No

How can I migrate this to use with S3 Acceleration code with access level and metadata information ?

If you need help with understanding how to implement something in particular then we suggest that you first look into our developer guide. You can also simplify your process of creating an application, as well as the associated backend setup by using the Amplify CLI.

[ruisebas]

Hi @Ankish!

Both Amplify and the AWS SDK for iOS support transfer acceleration.

Amplify support was introduced in version 2.15.1, while the SDK has had it since 2.4.3.

As it seems you're using Amplify, you just need to set "acceleratedEnabled": true in the pluginOptions when invoking your Storage operations. E.g.

let options = StorageUploadFileRequest.Options(
    accessLevel: .private,
    targetIdentityId: nil,
    metadata: [
        Utils.createTimeStampKey : "(createTime)"
    ],
    contentType: nil,
    pluginOptions: [
        "useAccelerateEndpoint": true
    ]
)
let task = Amplify.Storage.uploadFile(
    key: videoId,
    local: localURL,
    options: options
)

Note that this is only available in Amplify v2. If you do not wish to upgrade and want to use the iOS SDK instead, you can follow this instructions.

[ruisebas]

The Use transfer acceleration section has been added to the Amplify Documentation for Swift site.

[Ankish]

let options = StorageUploadFileRequest.Options(
    accessLevel: .private,
    targetIdentityId: nil,
    metadata: [
        Utils.createTimeStampKey : "(createTime)"
    ],
    contentType: nil,
    pluginOptions: [
        "useAccelerateEndpoint": true
    ]
)
let task = Amplify.Storage.uploadFile(
    key: videoId,
    local: localURL,
    options: options
)

Confirming, This is not supported in Amplify V1 ? I am using V1 and updated with above code and this does not throw any error.

[ruisebas]

Yes, this is only supported in Amplify v2.
Setting "useAccelerateEndpoint": true in pluginOptions has no effect in Amplify v1.

[Ankish]

The issue now is Amplify SPM and AWS SPM both does not work in same project, there is an open issue already on AWS.
So I need to use AWS Sdk for uploading via transfer acceleration. Do you have sample code or an example to migrate this code with AWS SDK with transfer acceleration? As stated in this ticket. Need to enable the acceleration while we work on complete migration to V2. Thanks !

let option = StorageUploadFileRequest.Options(accessLevel: StorageAccessLevel.private, targetIdentityId: nil,
metadata: [Utils.createTimeStampKey : "(createTime)"], contentType: nil, pluginOptions:nil)

Amplify.Storage.uploadFile(key: videoId, local: localURL, options: option, progressListener: { progress

This is how we initialise the configuration:

///configure the amplify if it is not done previously
    @discardableResult
    public func configureIfNeeded() -> Result<Any, Error>? {
        guard !AmplifyUtil.isAmplifyConfigured else {
            return nil
        }
        
        do {
            let apiPlugin = AWSAPIPlugin(apiAuthProviderFactory: MyAPIAuthProviderFactory())
            let storagePlugin = AWSS3StoragePlugin()
            let logging = AWSUnifiedLoggingPlugin()
            #if COACHDEBUG
            AWSDDLog.sharedInstance.logLevel = .debug
            AWSLogger.default().logLevel = .debug
            #endif
            try Amplify.add(plugin: apiPlugin)
            try Amplify.add(plugin: storagePlugin)
            try Amplify.add(plugin: logging)
            try Amplify.add(plugin: AWSCognitoAuthPlugin())
            
            ///can configure here target's configuration file from which configuaration need to read
            guard let path = Bundle.main.path(forResource: amplifyConfigurationFileName, ofType: "json") else {
                return Result.failure(GeneralError.generalError(NSLocalizedString("not able to read amplify configuration file", comment: ""),nil))
            }
            
            let configurationData = try Data(contentsOf: URL(fileURLWithPath: path), options: .mappedIfSafe)
            let configurationResult = try JSONSerialization.jsonObject(with: configurationData, options: .mutableLeaves)
            
            ///gets the api and storage plugins
            guard let result = configurationResult as? [String:Any],
                let apiConfig = getApiConfiguration(configuration: result),
                let storageConfig = getStorageConfiguration(configuration: result),
                let autConfig = getAuthConfiguration() else {
                    return Result.failure(GeneralError.generalError(NSLocalizedString("not able to read api or storage values from configuration file", comment: ""),nil))
            }
            
            let config = AmplifyConfiguration(api: apiConfig, auth: autConfig, storage: storageConfig)
            try Amplify.configure(config)
            
            AmplifyUtil.isAmplifyConfigured = true
        } catch let error {
            print(error.localizedDescription)
            return Result.failure(GeneralError.generalError(NSLocalizedString("unknown error occured during configuration. Please try again", comment: ""),nil))
        }
        
        return nil
    }
    ```

[Ankish]

@ruisebas : Help would be greatly appreciated. I know there are samples but I am not able to find a AWS replacement for my above amplify 1.0 code.

[ruisebas]

Hi @Ankish! First of all, to avoid any potential confusion for people finding this issue, let's clarify this statement:

Amplify SPM and AWS SPM both does not work in same project

This is incorrect. You cannot add Amplify v2 and the AWS SDK for iOS to the same project. But you can totally use Amplify v1 through SPM, which works fine with the AWS SDK for iOS.

---

Regarding your question, as Amplify v1 uses the TransferUtility from AWS SDK for iOS, you can also use it yourself and replicate what Amplify Storage does.

To upload using your own TransferUtility with accelerated mode enabled, you will need:

1. A AWSServiceConfiguration that defines your region and the credentials provider

Luckily, you can just retrieve the one that the Storage plugin uses, as it's the same.

// Retrieve the service configuration from Amplify
guard let plugin = try? Amplify.Storage.getPlugin(for: "awsS3StoragePlugin") as? AWSS3StoragePlugin else {
    fatalError("Unable to retrieve Storage plugin")
}
let serviceConfiguration = plugin.getEscapeHatch().configuration

This is needed in order to register the Transfer Utility

2. The bucket you want to upload to

You can grab this from the amplify configuration. As it seems you have your own configuration file, you can grab it from your storageConfig object.

3. A way to determine the correct object key path prefix according to the required access level.

Amplify uses the following prefixes for each access level:

• guest: "public/"
• protected: "protected/{identityId}/"
• private: "private/{identityId}/"

Each StorageAccessLevel already has a public serviceAccessPrefix property that returns its prefix, and you can get the identityId by calling Amplify.Auth.fetchAuthSession.

You can use this guide to then replicate the desired behaviour, resulting in something like this:

private func resolvePrefix(
    for accessLevel: StorageAccessLevel,
    targetIdentityId: String?,
    completion: @escaping (Result<String, StorageError>) -> Void
) {
    // Use "{prefix}/" for guest access levels
    if accessLevel == .guest {
        completion(.success("\(accessLevel.serviceAccessPrefix)/"))
        return
    }

    // Use "{prefix}/{targetIdentityId}/" for protected and private access levels if targetIdentityId is provided
    if let targetIdentityId = targetIdentityId {
        completion(.success("\(accessLevel.serviceAccessPrefix)/\(targetIdentityId)/"))
        return
    }

    // Use "{prefix}/{identityId}/" for protected and private access levels if targetIdentityId is nil
    Amplify.Auth.fetchAuthSession { result in
        do {
            guard let identityProvider = try result.get() as? AuthCognitoIdentityProvider else {
                completion(.failure(.authError("Unable to retrieve Cognito Identity Provider", "", nil)))
                return
            }

            let identityId = try identityProvider.getIdentityId().get()
            completion(.success("\(accessLevel.serviceAccessPrefix)/\(identityId)/"))
        } catch {
            completion(.failure(.authError("Failed to retrieve auth session", "", nil)))
        }
    }
}

4. Set the metadata to the request

You will need to append the x-amz-meta- prefix to all your keys, and then use AWSS3TransferUtilityUploadExpression.setValue(_:forRequestHeader:) to set each of them.

// Set metadata
let expression = AWSS3TransferUtilityUploadExpression()
for (key, value) in metadata {
    expression.setValue(value, forRequestHeader: "x-amz-meta-\(key)")
}

Going through the Using TransferUtility guide, you can create your own TransferAccelerationService class to unify everything in there:

import Amplify
import AWSPluginsCore
import AWSS3
import AWSS3StoragePlugin
import Foundation

class TransferAccelerationService {
    static let shared = TransferAccelerationService()
    private let transferUtilityWithAccelerationKey = "transfer-utility-with-acceleration"

    private init() {
        // Retrieve the service configuration from Amplify
        guard let plugin = try? Amplify.Storage.getPlugin(for: "awsS3StoragePlugin") as? AWSS3StoragePlugin else {
            fatalError("Unable to retrieve Storage plugin")
        }
        let serviceConfiguration = plugin.getEscapeHatch().configuration

        // Setup the transfer utility configuration
        let transferUtilityConfiguration = AWSS3TransferUtilityConfiguration()
        transferUtilityConfiguration.isAccelerateModeEnabled = true

        // Register a Transfer Utility object
        AWSS3TransferUtility.register(
            with: serviceConfiguration,
            transferUtilityConfiguration: transferUtilityConfiguration,
            forKey: transferUtilityWithAccelerationKey
        )
    }

    func uploadFile(
        key: String,
        local: URL,
        bucket: String,
        options: StorageUploadFileRequest.Options,
        progressListener: @escaping (Progress) -> Void,
        resultListener: @escaping (Result<String, Error>) -> Void
    ) {
        // Retrieve the Transfer Utility with Transfer Acceleration
        guard let transferUtility = AWSS3TransferUtility.s3TransferUtility(
            forKey: transferUtilityWithAccelerationKey
        ) else {
            resultListener(.failure(
                StorageError.configuration("Unable to retrieve Transfer Utility with Transfer Acceleration", "", nil)
            ))
            return
        }

        // Set metadata
        let expression = AWSS3TransferUtilityUploadExpression()
        for (key, value) in options.metadata ?? [:] {
            expression.setValue(value, forRequestHeader: "x-amz-meta-\(key)")
        }

        // Set progress listener
        expression.progressBlock = { _, progress in
            progressListener(progress)
        }

        resolvePrefix(
            for: options.accessLevel,
            targetIdentityId: options.targetIdentityId
        ) { result in
            do {
                let prefix = try result.get()
                transferUtility.uploadFile(
                    local,
                    bucket: bucket,
                    key: "\(prefix)\(key)",
                    contentType: options.contentType ?? "application/octet-stream",
                    expression: expression
                ) { task, error in
                    if let error = error {
                        resultListener(.failure(error))
                    } else {
                        resultListener(.success(task.key))
                    }
                }
            } catch {
                resultListener(.failure(error))
            }
        }
    }

    private func resolvePrefix(
        for accessLevel: StorageAccessLevel,
        targetIdentityId: String?,
        completion: @escaping (Result<String, StorageError>) -> Void
    ) {
        // Use "{prefix}/" for guest access levels
        if accessLevel == .guest {
            completion(.success("\(accessLevel.serviceAccessPrefix)/"))
            return
        }

        // Use "{prefix}/{targetIdentityId}/" for protected and private access levels if targetIdentityId is provided
        if let targetIdentityId = targetIdentityId {
            completion(.success("\(accessLevel.serviceAccessPrefix)/\(targetIdentityId)/"))
            return
        }

        // Use "{prefix}/{identityId}/" for protected and private access levels if targetIdentityId is nil
        Amplify.Auth.fetchAuthSession { result in
            do {
                guard let identityProvider = try result.get() as? AuthCognitoIdentityProvider else {
                    completion(.failure(.authError("Unable to retrieve Cognito Identity Provider", "", nil)))
                    return
                }

                let identityId = try identityProvider.getIdentityId().get()
                completion(.success("\(accessLevel.serviceAccessPrefix)/\(identityId)/"))
            } catch {
                completion(.failure(.authError("Failed to retrieve auth session", "", nil)))
            }
        }
    }
}

I've kept a very similar signature to the existing Storage method, so you can simply call it by doing:

let bucket = // your bucket
let options = StorageUploadFileRequest.Options(
    accessLevel: .private,
    targetIdentityId: nil,
    metadata: [
        Utils.createTimeStampKey: "(createTime)"
    ],
    contentType: nil,
    pluginOptions: nil
)

TransferAccelerationService.shared.uploadFile(
    key: videoId,
    local: localURL,
    bucket: bucket,
    options: options,
    progressListener: { progress in
        // Handle progress
    },
    resultListener: { result in
        // Handle result
    }
)

Please keep in mind that this is not an oficial feature that Amplify v1 supports, so you will need to handle any unexpected or special behaviour on your own.

Thanks!