Huge memory usage

[gatopeich]

Hello I just ported it to Python 3.7 and made some tests to compare with Redis...

Performance is pretty good up to ~100,000 keys, however I am finding that the shared file seems unreasonably big, at ~500MB for that.

Is that expected?

I am also observing huge performance degradation at bigger sizes, probably because the file does not fit in RAM anymore...

[felix021]

Yes, the underlying hash table uses fixed key and value length in hashtable.c, therefore event if you only use a short key/value, it takes much space, maybe much more than expected. You can adjust bucket_size and max_key_size to reduce it, but be careful with your input.

This package is kind of experimental, and I didn't do much test on it, and it DO NOT use lock to avoid concurrency problems.

[felix021]

BE CAREFUL: this package is not for general purpose usage, it only accepts key < max_key_size and values < max_value_size. And although it's sharing memory based, it DO NOT use locks to avoid concurrency problem. It was designed for a former project, which had a write process, and many read process after the writer finished.

[gatopeich]

OK thanks anyway, it was an interesting experiment.
Now I am working on a more general solution based on Boost::interprocess...
If you don't mind, I will leave the issue opened so people do BE CAREFUL soon enough :-)

[felix021]

OK, and I've added the information in README.md