v1.9.0, v1.10+ Roadmap

[bmoscon]

Update: the last release of 1.9 has been put out (1.9.3), new code will be in the 2.x release line, major changes coming, see: #596

Planned features/Upgrades for 1.9.x

• Change REST calls to use new connection interface (done in 1.9.0)
• add code coverage, add integration testing (partially complete)
• Add more comprehensive unit tests - test all websocket exchanges (done in 1.9.0)

Planned for 1.10+

• Integrate in Orderbook C module
• Record and Publish (as documentation) performance metrics
• Complete Websocket Improvements - from open PR (done in 1.9.0)
• Investigate usability of Pyston (On Linux)

Completed in 1.8:

• Document authenticated channels and what feeds support what (done in 1.8)
• Run backends as separate tasks (one per queue), data will be fed to them via a queue. (done in 1.8)
• Investigate Cythonizing or integrating into C module slow sections of feedhandlers (usually book processing). (done in 1.8)

[yohplala]

hi @bmoscon,
Terrific!
I recently had a test with Cython, Pythran and Numba (I read about them in here)

I couldn't succeed to make Cython and Pythran work out of the box, but Numba has been great.
It decreased execution time of my python function in charge of iterating over a numpy array by a factor 35.
Great for me!

[bmoscon]

The speed up from processing the complete orderbook message in C and in Cython seems negligible. I can get orderbook updates down to about 27 microseconds by using the C orderbook (but parsing the json in Python), vs 23 microseconds if the entire orderbook update is handled in C vs 25 in Cython. The C version required nearly 100 lines of ugly C for just a single exchange. I'm just guessing here, but I'd guess that for the current list of supported exchanges there are roughly 8 ways of structuring orderbook updates in JSON, meaning someone (i.e. me) would need to write a solid 1000 lines of C, test it, maintain it, etc. If I was getting a very large speedup, it may be worth it, but at 27 microseconds an update, (assuming no other overhead) thats about 37k updates a second that could be processed. I don't think any exchange comes anywhere near that volume (i could be wrong) so I think just migrating to the C orderbook should suffice.

Without the C orderbook, the speed here is in about 75 microseconds an update, so getting to 27 is a good performance boost considering coinbase averages about 1100 updates a second (from my testing over the past few weeks). This leads to a time savings of about 50ms every second