Why use Decimal?

[jinusean]

As the title states, why does this package convert all floats to Decimal? I understand Python has issues with floating point arithmetic, but I am not sure if Cryptofeed does any calculations on the data.

Binance sends its values as strings to preserve precision, wouldn't this be enough for Cryptofeed/Cryptostore's use case?

Awesome project btw!

[bmoscon]

had to standardize on something. At least with decimals you can do math operations and the like. Because the data comes as a string across the wire via the websocket, when we load the json data into a dictionary we can serialize it directly to a decimal. if we were to do it as a float and then to a string or decimal, precision would be lost

[yohplala]

had to standardize on something. At least with decimals you can do math operations and the like. Because the data comes as a string across the wire via the websocket, when we load the json data into a dictionary we can serialize it directly to a decimal. if we were to do it as a float and then to a string or decimal, precision would be lost

Hi Bryant,
As the question is raised, I have in mind that Redis users are impacted by this choicev(Redis not accepting Decimal).
Couldn't the data remain string while it is handled in Kafka or Redis, then converted to what is best according user data storage solution?

As far as I know, here is current data flow:

Kafka
string (websocket)
-> Decimal (python/cryptofeed)
-> Decimal (Kafka)
-> Decimal or float (python/user code/user data storage)

Redis
string (websocket)
-> Decimal (python/cryptofeed)
-> float (python/cryptofeed)
-> float (Redis)
-> float (python/user code/user data storage)

Couldn't it be (to keep Kafka and Redis workflow the same) ?

Redis/Kafka
string (websocket)
-> string (python/cryptofeed)
-> string (Kafka/Redis)
-> Decimal or float (python/depending user code or user data storage)

Is this relevant?

[bmoscon]

the end user can choose to use str/float for most of the backends, so you can choose to convert the Decimals to string for redis if you want to. https://github.com/bmoscon/cryptofeed/blob/master/cryptofeed/backends/redis.py#L23 <- set the numeric_type to str if you want str as opposed to float. The default type for kafka is also float, so not sure why you think its written as decimal in kafka: https://github.com/bmoscon/cryptofeed/blob/master/cryptofeed/backends/kafka.py#L18

[yohplala]

so not sure why you think its written as decimal in kafka

Sorry, my bad.
I thought it was Decimal as default because as you indicate, in the code of feeds, when json data is loaded into a dictionary, numeric is translated directly from string to Decimal.
I thought that it was done so because Kafka can manage them directly. Sorry.

[bmoscon]

ah, no thats from the exchanges. The data from websockets is bytes/string data so when we load json data to a dict we can parse it directly to a decimal (as opposed to a float) and not lose any precision. What I'm talking about here is exchange -> cryptofeed. It doesnt have anything to do with the backends

[yohplala]

What I'm talking about here is exchange -> cryptofeed. It doesnt have anything to do with the backends

I think I understand Bryant.
The point I was raising (but is probably completely pointless) is impact of performance of making a string -> Decimal conversion, while when transferred then to backends Decimal is not used, instead there is a conversion Decimal -> float.
As such, I was raising the question 'why not keeping it in string, and when handling it in backends, doing the conversion string -> float?'

I can completely understand the saving would be minimalist, and there might be people interested in current 'to Decimal' conversion.