feat: HTTP SenderPool with asyncio support #66
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
A new API to make it easier to work with the sender asynchronously with true parallelism.
Details
flush
represents an atomic database transaction.API downsides
creates silent network-blocking operations in the API.
Thread safety and Parallelism
pool.next_buffer()
andpool.flush()
methods are thread safe.N:M
concurrencyN
buffer writer threadsM
threads responsible for concurrently writing to the database (inside the pool).Tasks
SenderPool
into newquestdb.ingress.pool
module.TransactionalBuffer.dataframe(..)
.pool.next_buffer()
andpool.flush
implementations.asyncio.wrap_future
indef flush()
directly (since that's how it's implemented anyway): https://github.com/python/cpython/blob/8ad88984200b2ccddc0a08229dd2f4c14d1a71fc/Lib/asyncio/base_events.py#L896 - this allows implementing.flush()
in terms of.flush_to_future()
and cut code duplication.Closes #64