+ Configuring Authentication +
+Customizing service client authentication.
+diff --git a/content/en/docs/faq/_index.md b/content/en/docs/faq/_index.md new file mode 100644 index 00000000000..8d057b49bb7 --- /dev/null +++ b/content/en/docs/faq/_index.md @@ -0,0 +1,97 @@ +--- +title: "Frequently Asked Questions" +linkTitle: "FAQ / Troubleshooting" +description: "Answers to some commonly-asked questions about the {{% alias sdk-go %}}" +weight: 9 +--- + +### How do I configure my SDK's HTTP client? Are there any guidelines or best practices? + +We are unable to provide guidance to customers on how to configure their HTTP +workflow in a manner that is most effective for their particular workload. The +answer to this is the product of a multivariate equation, with input factors +including but not limited to: + +* the network footprint of the application (TPS, throughput, etc.) +* the services being used +* the compute characteristics of the deployment +* the geographical nature of the deployment +* the desired application behavior or needs of the application itself (SLAs, + timings, etc.) + +### How should I configure operation timeouts? + +Much like the previous question, it depends. Elements to consider here include +the following: + +* All of the above factors concerning HTTP client config +* Your own application timing or SLA constraints (e.g. if you yourself serve + traffic to other consumers) + +**The answer to this question should almost NEVER be based on pure empirical +observation of upstream behavior** - e.g. "I made 1000 calls to this operation, +it took at most 5 seconds so I will set the timeout based on that with a safety +factor of 2x to 10 seconds". Environment conditions can change, services can +temporarily degrade, and these types of assumptions can become wrong without +warning. + +### Requests made by the SDK are timing out or taking too long, how do I fix this? + +We are unable to assist with extended or timed-out operation calls due to +extended time spent on the wire. "Wire time" in the SDK is defined as any of +the following: +* Time spent in an SDK client's `HTTPClient.Do()` method +* Time spent in `Read()`s on an HTTP response body that has been forwarded to + the caller (e.g. `GetObject`) + +If you are experiencing issues due to operation latency or timeouts, your first +course of action should be to obtain telemetry of the SDK operation lifecycle +to determine the timing breakdown between time spent on the wire and the +surrounding overhead of the operation. See the guide on +[timing SDK operations]({{< ref "/docs/faq/timing-operations.md" >}}), +which contains a reusable code snippet that can achieve this. + +### How do I fix a `read: connection reset` error? + +The SDK retries any errors matching the `connection reset` pattern by default. +This will cover error handling for most operations, where the operation's HTTP +response is fully consumed and deserialized into its modeled result type. + +However, this error can still occur in a context **outside** of the retry loop: +certain service operations directly forward the API's HTTP response body to the +caller to be consumed from the wire directly via `io.ReadCloser` (e.g. +`GetObject`'s object payload). You may encounter this error when performing a +`Read` on the response body. + +This error indicates that your host, the service or any intermediary party +(e.g. NAT gateways, proxies, load balancers) closed the connection while +attempting to read the response. + +This can occur for several reasons: +* You did not consume the response body for some time after the response itself + was received (after the service operation was called). **We recommend you + consume the HTTP response body as soon as possible for these types of + operations.** +* You did not close a previously-received response body. This can cause + connection resets on certain platforms. **You MUST close any `io.ReadCloser` + instances provided in an operation's response, regardless of whether you + consume its contents.** + +Beyond that, try running a tcpdump for an affected connection at the edge of +your network (e.g. after any proxies that you control). If you see that the AWS +endpoint seems to be sending a TCP RST, you should use the AWS support console +to open a case against the offending service. Be prepared to provide request +IDs and specific timestamps of when the issue occured. + +### Why am I getting "invalid signature" errors when using an HTTP proxy with the SDK? + +The signature algorithm for AWS services (generally sigv4) is tied to the +serialized request's headers, more specifically most headers prefixed with +`X-`. Proxies are prone to modifying the outgoing request by adding additional +forwarding information (often via an `X-Forwarded-For` header) which +effectively breaks the signature that the SDK calculated. + +If you're using an HTTP proxy and experiencing signature errors, you should +work to capture the request **as it appears outgoing from the proxy** and +determine whether it is different. + diff --git a/content/en/docs/faq/timing-operations.md b/content/en/docs/faq/timing-operations.md new file mode 100644 index 00000000000..5e1d4e2dbb0 --- /dev/null +++ b/content/en/docs/faq/timing-operations.md @@ -0,0 +1,250 @@ +--- +title: "Timing SDK operations" +linkTitle: "Timing Operations" +description: "How to perform basic instrumentation in the {{% alias sdk-go %}} to time SDK operations" +weight: 1 +--- + +When debugging timeout / latency issues in the SDK, it is critical to identify +the components of the operation lifecycle which are taking more time to execute +than expected. As a starting point, you will generally need to inspect the +timing breakdown between the overall operation call and the HTTP call itself. + +The following sample program implements a basic instrumentation probe in terms +of `smithy-go` middleware for SQS clients and demonstrates how it is used. The +probe emits the following information for each operation call: + +* AWS request ID +* service ID +* operation name +* operation invocation time +* http call time + +Each emitted message is prefixed with a unique (to a single operation) +"invocation ID" which is set at the beginning of the handler stack. + +The entry point for instrumentation is exposed as `WithOperationTiming`, which +is parameterized to accept a message handling function which will receive +instrumentation "events" as formatted strings. `PrintfMSGHandler` is provided +as a convenience which will simply dump messages to stdout. + +The service used here is interchangeable - ALL service client options accept +`APIOptions` and an `HTTPClient` as configuration. For example, +`WithOperationTiming` could instead be declared as: + +```go +func WithOperationTiming(msgHandler func(string)) func(*s3.Options) +func WithOperationTiming(msgHandler func(string)) func(*dynamodb.Options) +// etc. +``` + +If you change it, be sure to change the signature of the function it returns as +well. + +```go +import ( + "context" + "fmt" + "log" + "net/http" + "sync" + "time" + + awsmiddleware "github.com/aws/aws-sdk-go-v2/aws/middleware" + awshttp "github.com/aws/aws-sdk-go-v2/aws/transport/http" + "github.com/aws/aws-sdk-go-v2/config" + "github.com/aws/aws-sdk-go-v2/service/sqs" + "github.com/aws/smithy-go/middleware" + smithyrand "github.com/aws/smithy-go/rand" +) + +// WithOperationTiming instruments an SQS client to dump timing information for +// the following spans: +// - overall operation time +// - HTTPClient call time +// +// This instrumentation will also emit the request ID, service name, and +// operation name for each invocation. +// +// Accepts a message "handler" which is invoked with formatted messages to be +// handled externally, you can use the declared PrintfMSGHandler to simply dump +// these values to stdout. +func WithOperationTiming(msgHandler func(string)) func(*sqs.Options) { + return func(o *sqs.Options) { + o.APIOptions = append(o.APIOptions, addTimingMiddlewares(msgHandler)) + o.HTTPClient = &timedHTTPClient{ + client: awshttp.NewBuildableClient(), + msgHandler: msgHandler, + } + } +} + +// PrintfMSGHandler writes messages to stdout. +func PrintfMSGHandler(msg string) { + fmt.Printf("%s\n", msg) +} + +type invokeIDKey struct{} + +func setInvokeID(ctx context.Context, id string) context.Context { + return middleware.WithStackValue(ctx, invokeIDKey{}, id) +} + +func getInvokeID(ctx context.Context) string { + id, _ := middleware.GetStackValue(ctx, invokeIDKey{}).(string) + return id +} + +// Records the current time, and returns a function to be called when the +// target span of events is completed. The return function will emit the given +// span name and time elapsed to the given message consumer. +func timeSpan(ctx context.Context, name string, consumer func(string)) func() { + start := time.Now() + return func() { + elapsed := time.Now().Sub(start) + consumer(fmt.Sprintf("[%s] %s: %s", getInvokeID(ctx), name, elapsed)) + } +} + +type timedHTTPClient struct { + client *awshttp.BuildableClient + msgHandler func(string) +} + +func (c *timedHTTPClient) Do(r *http.Request) (*http.Response, error) { + defer timeSpan(r.Context(), "http", c.msgHandler)() + + resp, err := c.client.Do(r) + if err != nil { + return nil, fmt.Errorf("inner client do: %v", err) + } + + return resp, nil +} + +type addInvokeIDMiddleware struct { + msgHandler func(string) +} + +func (*addInvokeIDMiddleware) ID() string { return "addInvokeID" } + +func (*addInvokeIDMiddleware) HandleInitialize(ctx context.Context, in middleware.InitializeInput, next middleware.InitializeHandler) ( + out middleware.InitializeOutput, md middleware.Metadata, err error, +) { + id, err := smithyrand.NewUUID(smithyrand.Reader).GetUUID() + if err != nil { + return out, md, fmt.Errorf("new uuid: %v", err) + } + + return next.HandleInitialize(setInvokeID(ctx, id), in) +} + +type timeOperationMiddleware struct { + msgHandler func(string) +} + +func (*timeOperationMiddleware) ID() string { return "timeOperation" } + +func (m *timeOperationMiddleware) HandleInitialize(ctx context.Context, in middleware.InitializeInput, next middleware.InitializeHandler) ( + middleware.InitializeOutput, middleware.Metadata, error, +) { + defer timeSpan(ctx, "operation", m.msgHandler)() + return next.HandleInitialize(ctx, in) +} + +type emitMetadataMiddleware struct { + msgHandler func(string) +} + +func (*emitMetadataMiddleware) ID() string { return "emitMetadata" } + +func (m *emitMetadataMiddleware) HandleInitialize(ctx context.Context, in middleware.InitializeInput, next middleware.InitializeHandler) ( + middleware.InitializeOutput, middleware.Metadata, error, +) { + out, md, err := next.HandleInitialize(ctx, in) + + invokeID := getInvokeID(ctx) + requestID, _ := awsmiddleware.GetRequestIDMetadata(md) + service := awsmiddleware.GetServiceID(ctx) + operation := awsmiddleware.GetOperationName(ctx) + m.msgHandler(fmt.Sprintf(`[%s] requestID = "%s"`, invokeID, requestID)) + m.msgHandler(fmt.Sprintf(`[%s] service = "%s"`, invokeID, service)) + m.msgHandler(fmt.Sprintf(`[%s] operation = "%s"`, invokeID, operation)) + + return out, md, err +} + +func addTimingMiddlewares(mh func(string)) func(*middleware.Stack) error { + return func(s *middleware.Stack) error { + if err := s.Initialize.Add(&timeOperationMiddleware{msgHandler: mh}, middleware.Before); err != nil { + return fmt.Errorf("add time operation middleware: %v", err) + } + if err := s.Initialize.Add(&addInvokeIDMiddleware{msgHandler: mh}, middleware.Before); err != nil { + return fmt.Errorf("add invoke id middleware: %v", err) + } + if err := s.Initialize.Insert(&emitMetadataMiddleware{msgHandler: mh}, "RegisterServiceMetadata", middleware.After); err != nil { + return fmt.Errorf("add emit metadata middleware: %v", err) + } + return nil + } +} + +func main() { + cfg, err := config.LoadDefaultConfig(context.Background()) + if err != nil { + log.Fatal(fmt.Errorf("load default config: %v", err)) + } + + svc := sqs.NewFromConfig(cfg, WithOperationTiming(PrintfMSGHandler)) + + var wg sync.WaitGroup + + for i := 0; i < 6; i++ { + wg.Add(1) + go func() { + defer wg.Done() + + _, err = svc.ListQueues(context.Background(), nil) + if err != nil { + fmt.Println(fmt.Errorf("list queues: %v", err)) + } + }() + } + wg.Wait() +} +``` + +A sample output of this program: + +``` +[e9a801bb-c51d-45c8-8e9f-a202e263fde8] http: 192.24067ms +[e9a801bb-c51d-45c8-8e9f-a202e263fde8] requestID = "dbee3082-96a3-5b23-adca-6d005696fa94" +[e9a801bb-c51d-45c8-8e9f-a202e263fde8] service = "SQS" +[e9a801bb-c51d-45c8-8e9f-a202e263fde8] operation = "ListQueues" +[e9a801bb-c51d-45c8-8e9f-a202e263fde8] operation: 193.098393ms +[0740f0e0-953e-4328-94fc-830a5052e763] http: 195.185732ms +[0740f0e0-953e-4328-94fc-830a5052e763] requestID = "48b301fa-fc9f-5f1f-9007-5c783caa9322" +[0740f0e0-953e-4328-94fc-830a5052e763] service = "SQS" +[0740f0e0-953e-4328-94fc-830a5052e763] operation = "ListQueues" +[0740f0e0-953e-4328-94fc-830a5052e763] operation: 195.725491ms +[c0589832-f351-4cc7-84f1-c656eb79dbd7] http: 200.52383ms +[444030d0-6743-4de5-bd91-bc40b2b94c55] http: 200.525919ms +[c0589832-f351-4cc7-84f1-c656eb79dbd7] requestID = "4a73cc82-b47b-56e1-b327-9100744e1b1f" +[c0589832-f351-4cc7-84f1-c656eb79dbd7] service = "SQS" +[c0589832-f351-4cc7-84f1-c656eb79dbd7] operation = "ListQueues" +[c0589832-f351-4cc7-84f1-c656eb79dbd7] operation: 201.214365ms +[444030d0-6743-4de5-bd91-bc40b2b94c55] requestID = "ca1523ed-1879-5610-bf5d-7e6fd84cabee" +[444030d0-6743-4de5-bd91-bc40b2b94c55] service = "SQS" +[444030d0-6743-4de5-bd91-bc40b2b94c55] operation = "ListQueues" +[444030d0-6743-4de5-bd91-bc40b2b94c55] operation: 201.197071ms +[079e8dbd-bb93-43ab-89e5-a7bb392b86a5] http: 206.449568ms +[12b2b39d-df86-4648-a436-ff0482d13340] http: 206.526603ms +[079e8dbd-bb93-43ab-89e5-a7bb392b86a5] requestID = "64229710-b552-56ed-8f96-ca927567ec7b" +[079e8dbd-bb93-43ab-89e5-a7bb392b86a5] service = "SQS" +[079e8dbd-bb93-43ab-89e5-a7bb392b86a5] operation = "ListQueues" +[079e8dbd-bb93-43ab-89e5-a7bb392b86a5] operation: 207.252357ms +[12b2b39d-df86-4648-a436-ff0482d13340] requestID = "76d9cbc0-07aa-58aa-98b7-9642c79f9851" +[12b2b39d-df86-4648-a436-ff0482d13340] service = "SQS" +[12b2b39d-df86-4648-a436-ff0482d13340] operation = "ListQueues" +[12b2b39d-df86-4648-a436-ff0482d13340] operation: 207.360621ms +``` diff --git a/content/en/docs/making-requests.md b/content/en/docs/making-requests.md index 9fd244d95f8..93fe1245f91 100644 --- a/content/en/docs/making-requests.md +++ b/content/en/docs/making-requests.md @@ -233,14 +233,21 @@ if err != nil { For more information on error handling, including how to inspect for specific error types, see the [Handling Errors]({{% ref "handling-errors.md" %}}) documentation. -#### Responses with io.ReadCloser +#### Responses with `io.ReadCloser` -Some API operations return a response struct that contain an output member that is an `io.ReadCloser`. If you're making -requests with these operations, always be sure to call `io.ReadCloser` member's `Close` method after you've completed -reading the content. +Some API operations return a response struct that contain an output member that +is an `io.ReadCloser`. This will be the case for API operations that expose +some element of their output in the body of the HTTP response itself. -For example {{% alias service=S3 %}} `GetObject` operation returns a response -whose `Body` member is an `io.ReadCloser`: +For example, {{% alias service=S3 %}} `GetObject` operation returns a response +whose `Body` member is an `io.ReadCloser` for accessing the object payload. + +{{% pageinfo color="warning" %}} +**You MUST ALWAYS `Close()` any `io.ReadCloser` output members, regardless of +whether you've consumed its content. Failure to do so can leak resources and +potentially create issues with reading response bodies for operations called in +the future.** +{{% /pageinfo %}} ```go resp, err := s3svc.GetObject(context.TODO(), &s3.GetObjectInput{...}) diff --git a/docs/404.html b/docs/404.html index 722c7cda3ac..7aa2859227b 100644 --- a/docs/404.html +++ b/docs/404.html @@ -99,7 +99,7 @@ aria-label="Search this site…" autocomplete="off" - data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.810a5299d837b395cdd69648bfec2e99.json" + data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.ba2b7a89f1555398e1c5839c9aee3bc1.json" data-offline-search-base-href="/" data-offline-search-max-results="10" > diff --git a/docs/docs/cloud9-go/index.html b/docs/docs/cloud9-go/index.html index 825fc7959e4..e05c171ecba 100644 --- a/docs/docs/cloud9-go/index.html +++ b/docs/docs/cloud9-go/index.html @@ -105,7 +105,7 @@ aria-label="Search this site…" autocomplete="off" - data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.810a5299d837b395cdd69648bfec2e99.json" + data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.ba2b7a89f1555398e1c5839c9aee3bc1.json" data-offline-search-base-href="/" data-offline-search-max-results="10" > @@ -127,7 +127,7 @@ aria-label="Search this site…" autocomplete="off" - data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.810a5299d837b395cdd69648bfec2e99.json" + data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.ba2b7a89f1555398e1c5839c9aee3bc1.json" data-offline-search-base-href="/" data-offline-search-max-results="10" > @@ -145,7 +145,9 @@
BaseEndpoint
Customizing service client authentication.
+The AWS SDK for Go requires Go 1.15 or later. You can view your current version of Go by running the following command.
+The AWS SDK for Go requires Go 1.19 or later. You can view your current version of Go by running the following command:
go version
For information about installing or upgrading your version of Go, see https://golang.org/doc/install.
On October 31, 2023, the AWS SDK for Go (v1 and v2) will start following the Go release policy cadence. See the blog post for more information.
- -Welcome to the AWS SDK for Go. The AWS SDK for Go V2 provides APIs and utilities that developers can use to build Go +
Welcome to the AWS SDK for Go. The AWS SDK for Go V2 provides APIs and utilities that developers can use to build Go applications that use AWS services, such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).
The SDK removes the complexity of coding directly against a web service interface. It hides a lot of the lower-level @@ -571,6 +571,14 @@
Answers to some commonly-asked questions about the AWS SDK for Go V2
+For more information on error handling, including how to inspect for specific error types, see the Handling Errors documentation.
-Some API operations return a response struct that contain an output member that is an io.ReadCloser
. If you’re making
-requests with these operations, always be sure to call io.ReadCloser
member’s Close
method after you’ve completed
-reading the content.
For example Amazon S3 GetObject
operation returns a response
-whose Body
member is an io.ReadCloser
:
io.ReadCloser
Some API operations return a response struct that contain an output member that
+is an io.ReadCloser
. This will be the case for API operations that expose
+some element of their output in the body of the HTTP response itself.
For example, Amazon S3 GetObject
operation returns a response
+whose Body
member is an io.ReadCloser
for accessing the object payload.
You MUST ALWAYS Close()
any io.ReadCloser
output members, regardless of
+whether you’ve consumed its content. Failure to do so can leak resources and
+potentially create issues with reading response bodies for operations called in
+the future.
resp, err := s3svc.GetObject(context.TODO(), &s3.GetObjectInput{...})
if err != nil {
// handle error
@@ -703,7 +722,7 @@ Responses with io.ReadCloser
All service operation output structs include a ResultMetadata
member of type
middleware.Metadata. middleware.Metadata
is used by the SDK middleware
to provide additional information from a service response that is not modeled by the service. This includes metadata
-like the RequestID
. For example to retrieve the RequestID
associated with a service response to assit AWS Support in
+like the RequestID
. For example to retrieve the RequestID
associated with a service response to assist AWS Support in
troubleshooting a request:
import "fmt"
import "log"
@@ -1004,7 +1023,7 @@ Using Waiters
- Last modified October 6, 2022: Update making-requests.md (#1879) (4b56845262)
+ Last modified December 21, 2023: docs: add FAQ/troubleshooting and operation timing guide (aa70a99eac)
diff --git a/docs/docs/middleware/index.html b/docs/docs/middleware/index.html
index 74c9d2f0850..275a5b2e813 100644
--- a/docs/docs/middleware/index.html
+++ b/docs/docs/middleware/index.html
@@ -104,7 +104,7 @@
aria-label="Search this site…"
autocomplete="off"
- data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.810a5299d837b395cdd69648bfec2e99.json"
+ data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.ba2b7a89f1555398e1c5839c9aee3bc1.json"
data-offline-search-base-href="/"
data-offline-search-max-results="10"
>
@@ -126,7 +126,7 @@
aria-label="Search this site…"
autocomplete="off"
- data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.810a5299d837b395cdd69648bfec2e99.json"
+ data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.ba2b7a89f1555398e1c5839c9aee3bc1.json"
data-offline-search-base-href="/"
data-offline-search-max-results="10"
>
@@ -144,7 +144,9 @@
Configuring the SDK
-
+
+ FAQ / Troubleshooting
+
+
+
Testing
diff --git a/docs/docs/migrating/index.html b/docs/docs/migrating/index.html
index e684f009de5..f102be89260 100644
--- a/docs/docs/migrating/index.html
+++ b/docs/docs/migrating/index.html
@@ -100,7 +100,7 @@
aria-label="Search this site…"
autocomplete="off"
- data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.810a5299d837b395cdd69648bfec2e99.json"
+ data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.ba2b7a89f1555398e1c5839c9aee3bc1.json"
data-offline-search-base-href="/"
data-offline-search-max-results="10"
>
@@ -122,7 +122,7 @@
aria-label="Search this site…"
autocomplete="off"
- data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.810a5299d837b395cdd69648bfec2e99.json"
+ data-offline-search-index-json-src="/aws-sdk-go-v2/offline-search-index.ba2b7a89f1555398e1c5839c9aee3bc1.json"
data-offline-search-base-href="/"
data-offline-search-max-results="10"
>
@@ -140,7 +140,9 @@
Configuring the SDK
-
Middleware
+
+ FAQ / Troubleshooting
+
+
+
Testing
@@ -424,6 +433,7 @@
+ Mocking and *iface
Credentials & Credential Providers
+ Request customization
+
Features
@@ -450,6 +469,7 @@
Amazon EC2 Instance Metadata Service
Amazon S3 Transfer Manager
Amazon CloudFront Signing Utilities
+ Amazon S3 Encryption Client
@@ -514,8 +534,7 @@ Migrating to the AWS SDK for Go V2
Minimum Go Version
-The AWS SDK for Go V2 requires a minimum version of Go 1.15. Migration from AWS SDK for Go to AWS SDK for Go V2
-might require you to upgrade your application by one or more Go versions. The latest version of Go can be downloaded on
+
The AWS SDK for Go V2 requires a minimum Go version of 1.19. Migration from v1 to v2 The latest version of Go can be downloaded on
the Downloads page. See the Release History for
more information about each Go version release, and relevant information required for upgrading.
Modularization
@@ -670,7 +689,58 @@ Migrating from NewSessio
if err != nil {
// handle error
}
-
*iface
The *iface
packages and interfaces therein (e.g. s3iface.S3API)
+have been removed. These interface definitions are not stable since they are
+broken every time a service adds a new operation.
Usage of *iface
should be replaced by scoped caller-defined interfaces for
+the service operations being used:
// V1
+
+import "io"
+
+import "github.com/aws/aws-sdk-go/service/s3"
+import "github.com/aws/aws-sdk-go/service/s3/s3iface"
+
+func GetObjectBytes(client s3iface.S3API, bucket, key string) ([]byte, error) {
+ object, err := client.GetObject(&s3.GetObjectInput{
+ Bucket: &bucket,
+ Key: &key,
+ })
+ if err != nil {
+ return nil, err
+ }
+ defer object.Body.Close()
+
+ return io.ReadAll(object.Body)
+}
+
// V2
+
+import "context"
+import "io"
+
+import "github.com/aws/aws-sdk-go-v2/service/s3"
+
+
+type GetObjectAPIClient interface {
+ GetObject(context.Context, *s3.GetObjectInput, ...func(*s3.Options)) (*s3.GetObjectOutput, error)
+}
+
+func GetObjectBytes(ctx context.Context, client GetObjectAPIClient, bucket, key string) ([]byte, error) {
+ object, err := api.GetObject(ctx, &s3.GetObjectInput{
+ Bucket: &bucket,
+ Key: &key,
+ })
+ if err != nil {
+ return nil, err
+ }
+ defer object.Body.Close()
+
+ return io.ReadAll(object.Body)
+}
+
See the testing guide for more +information.
+The aws/credentials package and associated credential providers have been
relocated to the credentials package location. The credentials
package is a Go module that
you retrieve by using go get
.
The endpoints package no longer exists in the AWS SDK for Go V2. Each service client now embeds its required AWS endpoint metadata within the client package. This reduces the overall binary size of compiled applications by no longer including endpoint metadata for services not used by your application.
+Additionally, each service now exposes its own interface for endpoint
+resolution in EndpointResolverV2
. Each API takes a unique set of parameters
+for a service EndpointParameters
, the values of which are sourced by the SDK
+from various locations when an operation is invoked.
By default, service clients use their configured AWS Region to resolve the service endpoint for the target Region. If
-your application requires a custom endpoint to be specified for a particular service and region, you can specify
-a custom aws.EndpointResolver using the EndpointResolver
field on the
+your application requires a custom endpoint, you can specify custom behavior on EndpointResolverV2
field on the
aws.Config
structure. If your application implements a custom
-endpoints.Resolver you must migrate it to conform to the
-aws.EndpointResolver
interface. aws.EndpointResolverFunc is provided as
-a convenient way to wrap a resolver function to satisfy the aws.EndpointResolver
interface.
For more information on endpoints and implementing a custom resolver, see Configuring Client Endpoints.
+The AWS SDK for Go V2 supports more advanced authentication behavior, which +enables the use of newer AWS service features such as codecatalyst and S3 +Express One Zone. Additionally, this behavior can be customized on a per-client +basis.
The number of service client operation methods have been reduced significantly. The <OperationName>Request
,
<OperationName>WithContext
, and <OperationName>
methods have all been consolidated into single operation method, <OperationName>
.
s3.BucketExistsWaiter
provides a
Wait
method which can be used to wait for a bucket to become available.
-The V1 SDK technically supported presigning any AWS SDK operation, however, +this does not accurately represent what is actually supported at the service +level (and in reality most AWS service operations do not support presigning).
+AWS SDK for Go V2 resolves this by exposing specific PresignClient
+implementations in service packages with specific APIs for supported
+presignable operations.
Note: If a service is missing presigning support for an operation that you +were successfully using in SDK v1, please let us know by +filing an issue on GitHub.
+Uses of Presign and +PresignRequest must +be converted to use service-specific presigning clients.
+The following example shows how to migrate presigning of an S3 GetObject +request:
+// V1
+
+import (
+ "fmt"
+ "time"
+
+ "github.com/aws/aws-sdk-go/aws"
+ "github.com/aws/aws-sdk-go/aws/session"
+ "github.com/aws/aws-sdk-go/service/s3"
+)
+
+func main() {
+ sess := session.Must(session.NewSessionWithOptions(session.Options{
+ SharedConfigState: session.SharedConfigEnable,
+ }))
+
+ svc := s3.New(sess)
+ req, _ := svc.GetObjectRequest(&s3.GetObjectInput{
+ Bucket: aws.String("bucket"),
+ Key: aws.String("key"),
+ })
+
+ // pattern 1
+ url1, err := req.Presign(20 * time.Minute)
+ if err != nil {
+ panic(err)
+ }
+ fmt.Println(url1)
+
+ // pattern 2
+ url2, header, err := req.PresignRequest(20 * time.Minute)
+ if err != nil {
+ panic(err)
+ }
+ fmt.Println(url2, header)
+}
+
// V2
+
+import (
+ "context"
+ "fmt"
+ "time"
+
+ "github.com/aws/aws-sdk-go-v2/aws"
+ "github.com/aws/aws-sdk-go-v2/config"
+ "github.com/aws/aws-sdk-go-v2/service/s3"
+)
+
+func main() {
+ cfg, err := config.LoadDefaultConfig(context.Background())
+ if err != nil {
+ panic(err)
+ }
+
+ svc := s3.NewPresignClient(s3.NewFromConfig(cfg))
+ req, err := svc.PresignGetObject(context.Background(), &s3.GetObjectInput{
+ Bucket: aws.String("bucket"),
+ Key: aws.String("key"),
+ }, func(o *s3.PresignOptions) {
+ o.Expires = 20 * time.Minute
+ })
+ if err != nil {
+ panic(err)
+ }
+
+ fmt.Println(req.Method, req.URL, req.SignedHeader)
+}
+
The monolithic request.Request API +has been re-compartmentalized.
+The opaque Request
fields Params
and Data
, which hold the operation input
+and output structures respectively, are now accessible within specific
+middleware phases as input/output:
Request handlers which reference Request.Params
and Request.Data
must be migrated to middleware.
Params
// V1
+
+import (
+ "github.com/aws/aws-sdk-go/aws"
+ "github.com/aws/aws-sdk-go/aws/request"
+ "github.com/aws/aws-sdk-go/aws/session"
+ "github.com/aws/aws-sdk-go/service/s3"
+)
+
+func withPutObjectDefaultACL(acl string) request.Option {
+ return func(r *request.Request) {
+ in, ok := r.Params.(*s3.PutObjectInput)
+ if !ok {
+ return
+ }
+
+ if in.ACL == nil {
+ in.ACL = aws.String(acl)
+ }
+ r.Params = in
+ }
+}
+
+func main() {
+ sess := session.Must(session.NewSession())
+ sess.Handlers.Validate.PushBack(withPutObjectDefaultACL(s3.ObjectCannedACLBucketOwnerFullControl))
+
+ // ...
+}
+
// V2
+
+import (
+ "context"
+
+ "github.com/aws/aws-sdk-go-v2/service/s3"
+ "github.com/aws/aws-sdk-go-v2/service/s3/types"
+ "github.com/aws/smithy-go/middleware"
+ smithyhttp "github.com/aws/smithy-go/transport/http"
+)
+
+type withPutObjectDefaultACL struct {
+ acl types.ObjectCannedACL
+}
+
+// implements middleware.InitializeMiddleware, which runs BEFORE a request has
+// been serialized and can act on the operation input
+var _ middleware.InitializeMiddleware = (*withPutObjectDefaultACL)(nil)
+
+func (*withPutObjectDefaultACL) ID() string {
+ return "withPutObjectDefaultACL"
+}
+
+func (m *withPutObjectDefaultACL) HandleInitialize(ctx context.Context, in middleware.InitializeInput, next middleware.InitializeHandler) (
+ out middleware.InitializeOutput, metadata middleware.Metadata, err error,
+) {
+ input, ok := in.Parameters.(*s3.PutObjectInput)
+ if !ok {
+ return next.HandleInitialize(ctx, in)
+ }
+
+ if len(input.ACL) == 0 {
+ input.ACL = m.acl
+ }
+ in.Parameters = input
+ return next.HandleInitialize(ctx, in)
+}
+
+// create a helper function to simplify instrumentation of our middleware
+func WithPutObjectDefaultACL(acl types.ObjectCannedACL) func (*s3.Options) {
+ return func(o *s3.Options) {
+ o.APIOptions = append(o.APIOptions, func (s *middleware.Stack) error {
+ return s.Initialize.Add(&withPutObjectDefaultACL{acl: acl}, middleware.After)
+ })
+ }
+}
+
+func main() {
+ cfg, err := config.LoadDefaultConfig(context.Background())
+ if err != nil {
+ // ...
+ }
+
+ svc := s3.NewFromConfig(cfg, WithPutObjectDefaultACL(types.ObjectCannedACLBucketOwnerFullControl))
+ // ...
+}
+
Data
// V1
+
+import (
+ "github.com/aws/aws-sdk-go/aws"
+ "github.com/aws/aws-sdk-go/aws/request"
+ "github.com/aws/aws-sdk-go/aws/session"
+ "github.com/aws/aws-sdk-go/service/s3"
+)
+
+func readPutObjectOutput(r *request.Request) {
+ output, ok := r.Data.(*s3.PutObjectOutput)
+ if !ok {
+ return
+ }
+
+ // ...
+ }
+}
+
+func main() {
+ sess := session.Must(session.NewSession())
+ sess.Handlers.Unmarshal.PushBack(readPutObjectOutput)
+
+ svc := s3.New(sess)
+ // ...
+}
+
// V2
+
+import (
+ "context"
+
+ "github.com/aws/aws-sdk-go-v2/config"
+ "github.com/aws/aws-sdk-go-v2/service/s3"
+ "github.com/aws/smithy-go/middleware"
+ smithyhttp "github.com/aws/smithy-go/transport/http"
+)
+
+type readPutObjectOutput struct{}
+
+var _ middleware.DeserializeMiddleware = (*readPutObjectOutput)(nil)
+
+func (*readPutObjectOutput) ID() string {
+ return "readPutObjectOutput"
+}
+
+func (*readPutObjectOutput) HandleDeserialize(ctx context.Context, in middleware.DeserializeInput, next middleware.DeserializeHandler) (
+ out middleware.DeserializeOutput, metadata middleware.Metadata, err error,
+) {
+ out, metadata, err = next.HandleDeserialize(ctx, in)
+ if err != nil {
+ // ...
+ }
+
+ output, ok := in.Parameters.(*s3.PutObjectOutput)
+ if !ok {
+ return out, metadata, err
+ }
+
+ // inspect output...
+
+ return out, metadata, err
+}
+
+func WithReadPutObjectOutput(o *s3.Options) {
+ o.APIOptions = append(o.APIOptions, func (s *middleware.Stack) error {
+ return s.Initialize.Add(&withReadPutObjectOutput{}, middleware.Before)
+ })
+}
+
+func main() {
+ cfg, err := config.LoadDefaultConfig(context.Background())
+ if err != nil {
+ // ...
+ }
+
+ svc := s3.NewFromConfig(cfg, WithReadPutObjectOutput)
+ // ...
+}
+
The HTTPRequest
and HTTPResponse
fields from Request
are now exposed in
+specific middleware phases. Since middleware is transport-agnostic, you must
+perform a type assertion on the middleware input or output to reveal the
+underlying HTTP request or response.
Request handlers which reference Request.HTTPRequest
and
+Request.HTTPResponse
must be migrated to middleware.
HTTPRequest
// V1
+
+import (
+ "github.com/aws/aws-sdk-go/aws/request"
+ "github.com/aws/aws-sdk-go/aws/session"
+)
+
+func withHeader(header, val string) request.Option {
+ return func(r *request.Request) {
+ request.HTTPRequest.Header.Set(header, val)
+ }
+}
+
+func main() {
+ sess := session.Must(session.NewSession())
+ sess.Handlers.Build.PushBack(withHeader("x-user-header", "..."))
+
+ svc := s3.New(sess)
+ // ...
+}
+
// V2
+
+import (
+ "context"
+ "fmt"
+
+ "github.com/aws/aws-sdk-go-v2/config"
+ "github.com/aws/aws-sdk-go-v2/service/s3"
+ "github.com/aws/smithy-go/middleware"
+ smithyhttp "github.com/aws/smithy-go/transport/http"
+)
+
+type withHeader struct {
+ header, val string
+}
+
+// implements middleware.BuildMiddleware, which runs AFTER a request has been
+// serialized and can operate on the transport request
+var _ middleware.BuildMiddleware = (*withHeader)(nil)
+
+func (*withHeader) ID() string {
+ return "withHeader"
+}
+
+func (m *withHeader) HandleBuild(ctx context.Context, in middleware.BuildInput, next middleware.BuildHandler) (
+ out middleware.BuildOutput, metadata middleware.Metadata, err error,
+) {
+ req, ok := in.Request.(*smithyhttp.Request)
+ if !ok {
+ return out, metadata, fmt.Errorf("unrecognized transport type %T", in.Request)
+ }
+
+ req.Header.Set(m.header, m.val)
+ return next.HandleBuild(ctx, in)
+}
+
+func WithHeader(header, val string) func (*s3.Options) {
+ return func(o *s3.Options) {
+ o.APIOptions = append(o.APIOptions, func (s *middleware.Stack) error {
+ return s.Build.Add(&withHeader{
+ header: header,
+ val: val,
+ }, middleware.After)
+ })
+ }
+}
+
+func main() {
+ cfg, err := config.LoadDefaultConfig(context.Background())
+ if err != nil {
+ // ...
+ }
+
+ svc := s3.NewFromConfig(cfg, WithHeader("x-user-header", "..."))
+ // ...
+}
+
SDK v2 middleware phases are the successor to v1 handler phases.
+The following table provides a rough mapping of v1 handler phases to their +equivalent location within the V2 middleware stack:
+v1 handler name | +v2 middleware phase | +
---|---|
Validate | +Initialize | +
Build | +Serialize | +
Sign | +Finalize | +
Send | +n/a (1) | +
ValidateResponse | +Deserialize | +
Unmarshal | +Deserialize | +
UnmarshalMetadata | +Deserialize | +
UnmarshalError | +Deserialize | +
Retry | +Finalize, after "Retry" middleware (2) |
+
AfterRetry | +Finalize, before "Retry" middleware, post-next.HandleFinalize() (2,3) |
+
CompleteAttempt | +Finalize, end of step | +
Complete | +Initialize, start of step, post-next.HandleInitialize() (3) |
+
(1) The Send
phase in v1 is effectively the wrapped HTTP client round-trip in
+v2. This behavior is controlled by the HTTPClient
field on client options.
(2) Any middleware after the "Retry"
middleware in the Finalize step will be
+part of the retry loop.
(3) The middleware “stack” at operation time is built into a +repeatedly-decorated handler function. Each handler is responsible for calling +the next one in the chain. This implicitly means that a middleware step can +also take action AFTER its next step has been called.
+For example, for the Initialize step, which is at the top of the stack, this +means Initialize middlewares that take action after calling the next handler +effectively operate at the end of the request:
+// V2
+
+import (
+ "context"
+
+ "github.com/aws/smithy-go/middleware"
+)
+
+type onComplete struct{}
+
+var _ middleware.InitializeMiddleware = (*onComplete)(nil)
+
+func (*onComplete) ID() string {
+ return "onComplete"
+}
+
+func (*onComplete) HandleInitialize(ctx context.Context, in middleware.InitializeInput, next middleware.InitializeHandler) (
+ out middleware.InitializeOutput, metadata middleware.Metadata, err error,
+) {
+ out, metadata, err = next.HandleInitialize(ctx, in)
+
+ // the entire operation was invoked above - the deserialized response is
+ // available opaquely in out.Result, run post-op actions here...
+
+ return out, metadata, err
+}
+
The AWS SDK for Go V2 provides an Amazon EC2 Instance Metadata Service (IMDS) client that you can use to query the local IMDS when executing your application on an Amazon EC2 instance. The IMDS client is @@ -1283,7 +1798,32 @@
The AWS SDK for Go V2 provides Amazon CloudFront signing utilities in a Go module outside the service
client import path. This module can be retrieved by using go get
.
go get github.com/aws/aws-sdk-go-v2/feature/cloudfront/sign
-
Starting in AWS SDK for Go V2, the Amazon S3 encryption client is a separate
+module under AWS Crypto Tools. The latest version of the S3 encryption client
+for Go, 3.x, is now available at https://github.com/aws/amazon-s3-encryption-client-go.
+This module can be retrieved by using go get
:
go get github.com/aws/amazon-s3-encryption-client-go/v3
+
The separate EncryptionClient
+(v1, v2)
+and DecryptionClient
+(v1, v2)
+APIs have been replaced with a single client,
+S3EncryptionClientV3,
+that exposes both encrypt and decrypt functionality.
Like other service clients in AWS SDK for Go V2, the operation APIs have +been condensed:
+GetObject
, GetObjectRequest
, and GetObjectWithContext
decryption
+APIs are replaced by
+GetObject.PutObject
, PutObjectRequest
, and PutObjectWithContext
encryption
+APIs are replaced by
+PutObject.To learn how to migrate to the 3.x major version of the encryption client, see +this guide.
+