How can I convert a DynamoDB view JSON to a normal JSON?

[rherrmannr]

For example, I want to convert:

{
 "id": {
  "N": "123"
 },
 "name": {
  "S": "some name"
 }
}

to this:

{
 "id": 123,
 "name": "some name"
}

I've found this question https://stackoverflow.com/questions/32712675/formatting-dynamodb-data-to-normal-json-in-aws-lambda with several answers for Node.js or Python.

Is there a converter like unmarshall for JavaScript or TypeDeserializer.deserialize for Python?

[johan-smits]

Use this: serde_dynamo

Edit: Changed the crate name to the correct one.

[rherrmannr]

This does not work for me using a dynamDB like this:

use lambda_runtime::{service_fn, Error, LambdaEvent};
use serde_json::Value;

async fn func(event: LambdaEvent<Value>) -> Result<(), Error> {
    let config = aws_config::load_from_env().await;
    let dynamodb = aws_sdk_dynamodb::Client::new(&config);
    let response = dynamodb.scan().table_name("Temperatures").send().await?;

    let items = response.items().unwrap();
    let item = items[0];

    let res = serde_dynamodb::from_hashmap(item);
    
    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let func = service_fn(func);
    lambda_runtime::run(func).await?;
    Ok(())
}

There is a type mismatch for AttributeValue:

    --> src/main.rs:73:44
     |
73   |     let res = serde_dynamodb::from_hashmap(item);
     |               ---------------------------- ^^^^ expected struct `rusoto_dynamodb::generated::AttributeValue`, found enum `AttributeValue`
     |               |
     |               arguments to this function are incorrect
     |
     = note: enum `AttributeValue` and struct `rusoto_dynamodb::generated::AttributeValue` have similar names, but are actually distinct types
note: enum `AttributeValue` is defined in crate `aws_sdk_dynamodb`
    --> /home/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/aws-sdk-dynamodb-0.21.0/src/model.rs:6677:1
     |
6677 | pub enum AttributeValue {
     | ^^^^^^^^^^^^^^^^^^^^^^^
note: struct `rusoto_dynamodb::generated::AttributeValue` is defined in crate `rusoto_dynamodb`
    --> /home/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/rusoto_dynamodb-0.47.0/src/generated.rs:84:1
     |
84   | pub struct AttributeValue {
     | ^^^^^^^^^^^^^^^^^^^^^^^^^
note: function defined here
    --> /home/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_dynamodb-0.9.0/src/dynamodb/de.rs:717:8
     |
717  | pub fn from_hashmap<'a, T, S: ::std::hash::BuildHasher + Clone>(
     |        ^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0308`.
error: could not compile `crud_temperatures` due to previous error

I don't want to use rusoto_dynamodb. I'd like to use aws_sdk_dynamodb.

[johan-smits]

Sorry I ment serde_dynamo. Also see #521

[rherrmannr]

Quite the same problem here. AttributeValues are defined in serde_dynamo::attribute_values and not using aws_sdk_dynamodb.

I'm trying this:

use lambda_runtime::{service_fn, Error, LambdaEvent};
use serde_json::Value;

async fn func(event: LambdaEvent<Value>) -> Result<(), Error> {
    let config = aws_config::load_from_env().await;
    let dynamodb = aws_sdk_dynamodb::Client::new(&config);
    let response = dynamodb.scan().table_name("Temperatures").send().await?;

    let items = response.items().unwrap();
    let res = serde_dynamo::from_items(items.to_owned()).unwrap();
    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let func = service_fn(func);
    lambda_runtime::run(func).await?;
    Ok(())
}

and get the following compile error:

error[E0277]: the trait bound `serde_dynamo::AttributeValue: From<aws_sdk_dynamodb::model::AttributeValue>` is not satisfied
  --> src/main.rs:70:40
   |
70 |     let res = serde_dynamo::from_items(items.to_owned()).unwrap();
   |               ------------------------ ^^^^^^^^^^^^^^^^ the trait `From<aws_sdk_dynamodb::model::AttributeValue>` is not implemented for `serde_dynamo::AttributeValue`
   |               |
   |               required by a bound introduced by this call
   |
   = note: required for `Item` to implement `From<HashMap<std::string::String, aws_sdk_dynamodb::model::AttributeValue>>`
   = note: 1 redundant requirement hidden
   = note: required for `Items` to implement `From<Vec<HashMap<std::string::String, aws_sdk_dynamodb::model::AttributeValue>>>`
   = note: required for `Vec<HashMap<std::string::String, aws_sdk_dynamodb::model::AttributeValue>>` to implement `Into<Items>`
note: required by a bound in `from_items`
  --> /home/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_dynamo-4.0.9/src/de/mod.rs:96:9
   |
96 |     Is: Into<Items>,
   |         ^^^^^^^^^^^ required by this bound in `from_items`

For more information about this error, try `rustc --explain E0277`.
error: could not compile `crud_temperatures` due to previous error

[jdisanti]

I think with serde_dynamo, you have to be sure to use the cargo feature that matches the aws-sdk-dynamodb version number. So if you're on aws-sdk-dynamodb-0.21.0, it would look like:

[dependencies]
aws-sdk-dynamodb = "0.21.0"
serde_dynamo = { version = "4.0.9", features = ["aws-sdk-dynamodb+0_21"] } 

If that doesn't work though, you should ask in the serde_dynamo GitHub repository.

[rherrmannr]

Does not work. I've asked in the serde_dynamo repo.
I'm wondering why I need to use a third-party library here, as other language SDKs support this. 😞

[Velfi]

The reason is because we don't currently support it. We're currently focused on implementing features so that we can someday release the 1.0 version of the SDK. This requires us to prioritize feature requests. Please create and upvote issues to let us know what's important to you. That we can build an SDK that meets your (and everyone else's) needs.

[ymwjbxxq]

You need manually convert each type to your struct.

Imagine this model

pub struct MyModel {
    pub prop1: String,
    pub prop2: String,
}

impl MyModel {
    pub fn from_dynamodb(value: HashMap<String, AttributeValue>) -> Result<MyModel, ApplicationError> {
        Ok(MyModel {
            prop1: value.get_string("prop1").unwrap(),
            prop1: value.get_string("prop1").unwrap(),
        })
    }
}

When you query:

let result = self
            .dynamo_db_client
            .get_item()
            .....
            .send()
            .await?;

 if let Some(item) = result.item {
  return Ok(Some(MyModel::from_dynamodb(item)?));
}

pub trait AttributeValuesExt {
    fn get_string(&self, key: &str) -> Option<String>;
    fn get_number(&self, key: &str) -> Option<u64>;
    fn get_bool(&self, key: &str) -> Option<bool>;
}

impl AttributeValuesExt for HashMap<String, AttributeValue> {
    fn get_number(&self, key: &str) -> Option<u64> {
        self.get(key)?.as_n().ok()?.parse().ok()
    }

    fn get_bool(&self, key: &str) -> Option<bool> {
        Some(self.get(key)?.as_bool().ok()?.to_owned())
    }
}

[evbo]

wouldn't it be good to add impls such as these to this repo? Isn't it common code that everyone would benefit from?

[Velfi]

It would be great to add these, it's just not a priority for us right now. We are working very hard to get our ducks in a row for a 1.0 release right now, and it's going to be a little while before we can focus on quality-of-life improvements like this.

That being said, we welcome PRs from outside contributors. If you can design, implement, test, and PR this feature, we'll review your PR as soon as we're able.