[WIP] Feature/add update delete support #195
Conversation
pool chain
pooling writers
csv writer detructor
process writer and csv pooling
airline.ontime table structure
memory consumtion
enhancements
logging enh
airline.ontime dataset
my.cnf options - make max_binlog_size bigger
initial commit - table builder
table builder
more details on required pip modules
minor doc fix - correct URL
--table-templates and --table-templates-json CLI options
import window
rename .sh files more clearly
rename main.py to clickhouse-mysql
rename pip to pip3 for Python 3.x
move shell scripts to examples/ folder
data migrator - initial commit
--mempool-max-rows-num option
prepare distribution structure for packaging
module executable
fix wheres -> where_clauses
dev: add shell quotation
new release
docs: fix src-tables
Docs - add known usage examples and extract manual
docs: fix links
docs: remove RPM sections
…ed to be generic. I've added this just to continue with the PoC
…one process or all
…hich actually change
|
Hello @ygnuss Thanks for the amazing work! Is there anything that I can help with to expedite this PR? |
|
Is there a way we can use this feature ? Currently we run ETL scripts at every 5mins to fetch data from OLTP(say MySQL) and put it to Clickhouse (Collapsing Merge Tree). Still by running ETLs we are not be able to match the realtimeness and the handling updates is not that easy in Clickhouse. Is there a good you would suggest ? Is there a way I can help in this feature addition, this is much needed. |
|
Hi!! We developed this feature in Tinybird and tried it with one of our use cases. Unfortunately, the performance was not the expected. Clickhouse, since it is designed for inserts not for update and deletes, does not perform well with this changes. We tried it with one of our use cases and after some time (weeks) the performance significantly degraded and finally blocked. Clickhouse was not able to apply the mutations fast enough. However, we have worked on other alternatives and we are currently able to handle this kind usage patterns (CDC) in real time. We already have it in production with several customers. You can try out Tinybird for free. You can check more information in our guides and our documentation documentation. Let me know if you need more help! |
alexon1234
force-pushed
the
feature/add-update-delete-support
branch
from
a9c0cfd to
f58e6b8
Compare
No description provided.