[DRAFT] Break functionality out of aws-elixir and aws-erlang and create a common aws_beam_core lib

[onno-vos-dev]

Description

Goal is to bring some of the common functionality between aws-elixir and aws-erlang under one umbrella.

The idea is to drop the need for including aws_signature in both SDKs as well as aws_credentials which is only documented in aws-erlang 🙂

For now, I call it aws_beam_core but I'm open for other naming suggestions 🙂

Reasoning

I've found too often that dependencies are different between the two, implementations vary ever so slightly and functionalities between the two diverges easily without a common place for common functionality. This slows down development and results in having to implement the same thing twice just in different languages. Considering Elixir's excellent capability to include Erlang code and not the other way around, the core bits are written in Erlang.

Furthermore, there are libraries that do not need the entirety of aws-elixir or aws-erlang for certain functionality and I'd like to give the opportunity to only bring in this "core" repo which can provide some of the more basic functionality such as aws_rds_iam_token and aws_s3_presigned_url without the need to bring in (relatively) fat libraries into the application.

One such example would be: klarna/system_monitor where I'd like to bring along the aws_rds_iam_token without the need to pull in all of aws-erlang.

TODO

• Update README
• Add aws_s3_presigned_url
• Extract HTTP transport layer stuff
• Test this branch in aws-erlang and check compatibility

[jadeallenx]

I've been down this road before and sometimes it makes sense to build a new library if you're pulling in a giant dependency tree for a 12 line function and some utility modules. But what I've found over time is that sometimes it's better to just embed those things downstream as a set of utility modules.

For example, if you need RDS tokens in downstream, then grab the functions from aws-erlang to implement it and vendor the utility files and selected dependencies from other libraries rather than make an entirely new tiny library for exactly one downstream project.

But like, aside from klarna, who else is going to use this?

[onno-vos-dev]

For example, if you need RDS tokens in downstream, then grab the functions from aws-erlang to implement it and vendor the utility files and selected dependencies from other libraries rather than make an entirely new tiny library for exactly one downstream project.

To me it's not just klarna/system_monitor that just happens to be one of the examples that Klarna has open sourced. 😉 I'd like to avoid any Erlang project that uses AWS Aurora/RDS to either a) have to pull in the entirety of aws-erlang or b) do exactly as you say and kinda handcraft the required functionality themselves. Sure it may not be overly complicated to copy-paste but it's functionality that I believe we as AWS-Beam should provide rather than rely on others to create this themselves. IMHO it's a bad abstraction to have those sorts of modules in a huge SDK to begin with.

Currently we have the situation where aws-elixir and aws-erlang share a lot of functionality but also there's functionality missing across those SDKs. RDS IAM Tokens, S3 Presigning, request retrying (was awfully late in aws-elixir for example) and would've been fantastic if instead it was all baked into a common lib. Simplifying and unifying the two SDKs.

[jadeallenx]

Currently we have the situation where aws-elixir and aws-erlang share a lot of functionality but also there's functionality missing across those SDKs. RDS IAM Tokens, S3 Presigning, request retrying (was awfully late in aws-elixir for example) and would've been fantastic if instead it was all baked into a common lib. Simplifying and unifying the two SDKs.

I agree it might be smart/helpful to decouple the HTTP transport stuff into its own thing, especially on areas around retries, retry jitter, and functions where its idempotent to retry and not safe to retry. It's true I don't have as much appreciation for what should be standardized across both the elixir and the erlang generated code bases than y'all.

Anyway, my main advice is try to avoid making it too specific for one or two downstream consumers.

[onno-vos-dev]

I agree it might be smart/helpful to decouple the HTTP transport stuff into its own thing, especially on areas around retries, retry jitter, and functions where its idempotent to retry and not safe to retry. It's true I don't have as much appreciation for what should be standardized across both the elixir and the erlang generated code bases than y'all.

Anyway, my main advice is try to avoid making it too specific for one or two downstream consumers.

I agree with you 👍 The challenge with the HTTP transport stuff is that aws-elixir has two optional http libraries Hackney and Finch and rebar doesn't support optional libraries but I believe it can be done through the use of callbacks 🙂 It'd just require a bit of surgery and thinking.

I've added it as a TODO but might postpone it as a follow up before pulling this into aws-elixir and aws-erlang 👍