enable support within aws_bedrockagent_knowledge_base for embedding_model_configuration and supplemental_data_storage_configuration

[awgibbs]

• extend appropriate go schema
• add necessary go structs
• add acceptance tests
• extend docs as necessary
• fix broken dash characters found in docs

Description

Last month AWS introduced binary embedding support for Amazon Titan Text Embeddings V2. This PR makes it possible to choose that embedding data type as well as configure the dimensions. For good measure it also adds support for supplemental storage configuration.

I have created and successfully run a new acceptance test. I should note, however, that testing against OpenSearch Serverless Collections (what I was personally targeting) is difficult generally and a polished implementation for acceptance tests seems to have been deferred when this resource was created some months, resulting in a somewhat difficult and manually intensive situation for myself as there was no perfect/automated model to follow. You will note that the extant OSSC tests were set to "skip" and that is how I am committing my new one (though it was not skipped for my actual testing). To perform my testing I pointed at an appropriate extant/external OSSC and then had the KB created by the acceptance test runs point at that. The relevant parameters are XXX'd out in the acceptance test. I have successfully tested against both "BINARY" and "FLOAT32" index data types backed by real OSSC instances created out-of-band.

In real life fully automated environments I have used the aws_lambda_invocation resource to execute post-creation OSSC manipulations to get an index in place. Bedrock KB creation fails without this underlying index being in place because it blows up when doing validation. I think it would be reasonable to continue to defer this realm as tech debt but to circle back imminently to get these acceptance tests into a better place across the board (not just mine; and I am happy to help across the board). At the moment, however, I am in a rather urgent situation where the impetus to have done this is the need to convert vector DBs in an operational environment to the "BINARY" data type.

References

https://aws.amazon.com/blogs/machine-learning/build-cost-effective-rag-applications-with-binary-embeddings-in-amazon-titan-text-embeddings-v2-amazon-opensearch-serverless-and-amazon-bedrock-knowledge-bases/

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent/client/create_knowledge_base.html

Output from Acceptance Testing

Andrews-MBP:terraform-provider-aws awgibbs$ make testacc TESTS=TestAccBedrockAgent_serial/KnowledgeBase/fancyOpenSearch PKG=bedrockagent
make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.23.3 test ./internal/service/bedrockagent/... -v -count 1 -parallel 20 -run='TestAccBedrockAgent_serial/KnowledgeBase/fancyOpenSearch'  -timeout 360m
2024/12/29 21:47:37 Initializing Terraform AWS Provider...
=== RUN   TestAccBedrockAgent_serial
=== PAUSE TestAccBedrockAgent_serial
=== CONT  TestAccBedrockAgent_serial
=== RUN   TestAccBedrockAgent_serial/KnowledgeBase
=== RUN   TestAccBedrockAgent_serial/KnowledgeBase/fancyOpenSearch
--- PASS: TestAccBedrockAgent_serial (20.73s)
    --- PASS: TestAccBedrockAgent_serial/KnowledgeBase (20.73s)
        --- PASS: TestAccBedrockAgent_serial/KnowledgeBase/fancyOpenSearch (20.73s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/bedrockagent	34.217s

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

• Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
• Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

[awgibbs]

It's worth further noting that the build environment (make tools) was broken as of my pull of main on Christmas. I had to hand remediate that situation to be able to work at all. I documented my mitigation in the following post but I'm not sure how we navigate that separate issue.

#40485 (comment)

[github-actions]

Welcome @awgibbs 👋

It looks like this is your first Pull Request submission to the Terraform AWS Provider! If you haven’t already done so please make sure you have checked out our CONTRIBUTOR guide and FAQ to make sure your contribution is adhering to best practice and has all the necessary elements in place for a successful approval.

Also take a look at our FAQ which details how we prioritize Pull Requests for inclusion.

Thanks again, and welcome to the community! 😃