Trying to build ta-lib from source if the library is not found

[mckelvin]

This is a failed attempt trying since there's a global variable in the c library, which make it impossible to share the global variable among multi extensions (which are compiled separately) if linked statically.

SEE ALSO: #111

---

Hi @mrjbq7

Thanks for developing and maintaining the Cython wrapper for ta-lib. The purpose of the PR is to make it easier to build and install ta-lib. It's quite hard to install the c library of ta-lib, especially for debian user as there's no system package and the project seems out of maintain. The PR may also be helpful to #88, #75.

One way to solve the problem is fallback to build from source if the library is not found. Like what https://github.com/redis/hiredis-py do.

Changes:

• git subtree add --prefix vendor/ta-lib https://github.com/TA-Lib/ta-lib.git v0.4.0 --squash (https://github.com/TA-Lib/ta-lib/releases/tag/v0.4.0)
• Update setup.py to make it fallback to build-from-source
• Move the testing related code to tests and refactor for py.test (a better alternative to nose)
• Support python setup.py test

[mrjbq7]

Hmm, thats a neat idea but I'm not sure I want to link to an "unofficial mirror"...

[mckelvin]

@mrjbq7 That's true. Sadly the upstream on sourceforge has no update for quite a few years. Would you like to take over the https://github.com/TA-Lib and make it part of mrjbq7/ta-lib ?

[mrjbq7]

Are you the owner of that org?

[mckelvin]

@mrjbq7 Yes. I've just invited you. The code is import from http://svn.code.sf.net/p/ta-lib/code/trunk/ta-lib/c/ using GitHub Importer. You may check using diff -rN svn_repo_dir git_repo_dir to make sure the code is clean without changes.

[mckelvin]

The PR above should work on OSX (At least the tests could pass) but it failed on Linux.

A few problems ahead: There's one global variable TA_Globals in the c library (ta_common). So we should have only one copy of the variable in memory, otherwise the test case: test_unstable_period will fail. But python extensions are built separately, I can't find a way to compile the c code into a static/dynamic library and then link it with every extensions. I guess it'll be easier if there's just only one extension in the project.

[mrjbq7]

Is the problem too many python C extensions? It's an open thing to resolve because of the extra memory that implies with keeping three copies around but I didn't realize it would be problematic for this PR.

[mckelvin]

The 3 Python C Extensions is dependent with each other:

• common.pyx: Which is clean and didn't depend on any pyxs but depend on the ta_common and ta_func in C.
• func.pyx: Depend on common.pyx and depend on ta_func, ta_common in C
• abstract.pyx: Depend on common.pyx and depend on ta_abstract, ta_common, ta_func in C.
• stream.pyx: Similar to func.pyx

First I tried to compile all the ".c" files and all the ".h" headers from the TA-Lib vendor with each of the 4 Python C Extensions(namely common, func, abstract, stream). As the TA_Globals is defined in ta_common, there'll be 4 copies of it in the memory, and test_unstable_period fails.

Later I tried compile each .pyx with minimum .c source files (like what the current PR is doing), but I found that they're still not sharing the ta_common. Somehow all the test case could on OS X but failed to build on Linux.

I think first we may need to make each of the C source code is used only by one Python C Extension (or just make sure there's only one Python C Extension), such that there'll be only one TA_Globals variable in runtime. @mrjbq7 What do you think of it ?

[mckelvin]

I just tried(https://github.com/mckelvin/python-ta-lib/tree/vendor-makefile-static) to compile the c library as a static library first, by running make command before build Python C extensions. The Python C extensions will link the vendor/ta-lib/lib/libta_libc_csr.a then.

The result is similar to what I tried at the very beginning. The test case failed.

$ for i in `ls *.so`; do ls $i && nm $i | grep Global; done
abstract.so
00000000000ad6a8 D _TA_Globals
00000000000b2478 S _ta_theGlobals
common.so
000000000000b038 D _TA_Globals
000000000000b4a8 S _ta_theGlobals
func.so
0000000000191c28 D _TA_Globals
0000000000195b68 S _ta_theGlobals
stream.so
0000000000165828 D _TA_Globals
00000000001691a8 S _ta_theGlobals

I think combine the .pyxs into just one extention will help a lot.

[mrjbq7]

One extension is on my todo list, but I have some travel coming up and can't get to it for a couple weeks.

[mckelvin]

@mrjbq7 Enjoy your journey! I've done that and I'll send another PR latter.