Q1 2021 - Roadmap

[bmoscon]

Here are some things I have planned for the coming months. I'm curious if these will be useful to others, and what things others are interested in

• An orderbook extension in C - The orderbooks are currently a major source of latency, switching to a C version should save a lot of time, especially for calculating deltas, trimming orderbooks, etc (orderbook completed, need to migrate codebase to use and extensively test).
• A reworked connection architecture - this will allow exchanges to have as many websocket connections as needed. Will also provide a way to capture REST endpoint data (for the raw data capture and playback). EDIT: Completed
• In conjunction with the previous item, authenticated websockets and callbacks

[RezDaniel]

All points sound good but point number one (OB extension in C) would definitely be of value.

[bmoscon]

its happening already :D Been working on it for a month or so, hope to have it released in 2-4 weeks.

[RezDaniel]

Awesome! Looking forward to checking it out.

[olibre]

Hi @bmoscon
I have also started to improve the Feed / FeedHandler for a better REST & WebSocket handling.
Would you like I publish my draft ?

[bmoscon]

@olibre sure - I'd like to see what you have in mind. I've started on mine, but its not too late to stop and switch if needed. Basically I'm changing the exchanges to return a list of connection objects that the feed handler will then use to start separate tasks, calling the connect on each of these objects. The exchanges just need to create the correct number of objects, of the correct type (rest or websocket) with the correct addresses. So, its not a major change, but its a good amount of work and testing nonetheless.

[olibre]

Alright, my changes are not so ambitious. Your design feels more consistent. OK, I clean my work and push to a dedicated branch 👍

[olibre]

See #366

[bmoscon]

the orderbook implementation has been completed. Just doing more extensive testing and will migrate cryptofeed to use it (and do some thorough testing before merging). https://github.com/bmoscon/orderbook

[yohplala]

Hello @bmoscon,
Please, what future do you see to backfilling? Is this part of the roadmap?
I know it was in your plan in cryptostore, but my understanding is that it should be managed first in cryptofeed, right?

To be clearer about this feature in cryptofeed, let's take a practical example with BINANCE and trades.
So, Binance allows fetching trades back in time as indicated in their API documentation, Compressed/Aggregate Trades List chapter.
Input parameters are most notably:

symbol    | STRING | YES |  
--------- | ------ | --- | 
startTime | LONG   | NO  | Timestamp in ms to get aggregate trades from INCLUSIVE.
endTime   | LONG   | NO  | Timestamp in ms to get aggregate trades until INCLUSIVE.
limit     |  INT   | NO  | Default 500; max 1000.

In Cryptofeed / message handler, should there be a new channel added tradesByBackfilling, with an additional parameter startTime ?

To be noticed, other data like 'funding' also applies for instance, documentation here. Possibly, 'funding' is already obtained this way, I did not had a look.

Thanks for sharing your mind about this. Bests

[bmoscon]

backfill is already supported in cryptofeed, its in the rest/ directory