Rushidave sync fork to upstream #28
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Create GH workflows to test code * Update tests for future Python versions
add imort json and msgpack into consumer and producer
* docs: Fix a few typos There are small typos in: - kafka/codec.py - kafka/coordinator/base.py - kafka/record/abc.py - kafka/record/legacy_records.py Fixes: - Should read `timestamp` rather than `typestamp`. - Should read `minimum` rather than `miniumum`. - Should read `encapsulated` rather than `incapsulates`. - Should read `callback` rather than `callbak`. * Update abc.py
Adding [ClusterMetadata] and [KafkaAdminClient]
I don't expect this to work yet since I know 3.12 is in an incomplete state, but here goes nothing.
In this commit, the del X is still commented out due to the fact that upstream benjaminp/six#176 is not merged.
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* [FIX] suitablt for the high vresion python. it won't import Mapping from collections at python3.11. tested it worked from python3.6 to 3.11.2. * Update selectors34.py to have conditional importing of Mapping from collections --------- Co-authored-by: William Barnhart <[email protected]>
…2381) I know that the typical release is uploaded to PyPi manually, however I figure I'd draft a PR with these changes because having the option to start doing this is worthwhile. More info can be found on https://github.com/pypa/gh-action-pypi-publish.
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v2...v3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4...v5) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](actions/setup-java@v3...v4) --- updated-dependencies: - dependency-name: actions/setup-java dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
After stop/start kafka service, kafka-python may use 100% CPU caused by busy-retry while the socket was closed. This fix the issue by sleep 0.1 second if the fd is negative.
Implement support for SOCKS5 proxies. Implement a new proxy wrapper that handles SOCKS5 connection, authentication and requesting connections to the actual Kafka broker endpoint. The proxy can be configured via a new keyword argument `socks5_proxy` to consumers, producers or admin client. The value is URL with optional username and password. E.g. `socks5://user:[email protected]:10800` The implementation is done in state machine that emulates repeated calls to connect_ex. The rationale with this design is minimal changes to the actual BrokerConnection object.
An infinite loop may happen with the following pattern:
self._send_request_to_node(self._client.least_loaded_node(), request)
The problem happens when `self._client`'s cluster metadata is out-of-date, and the
result of `least_loaded_node()` is a node that has been removed from the cluster but
the client is unware of it. When this happens `_send_request_to_node` will enter an
infinite loop waiting for the chosen node to become available, which won't happen,
resulting in an infinite loop.
This commit introduces a new method named `_send_request_to_least_loaded_node` which
handles the case above. This is done by regularly checking if the target node is
available in the cluster metadata, and if not, a new node is chosen.
Notes:
- This does not yet cover every call site to `_send_request_to_node`, there are some
other places were similar race conditions may happen.
- The code above does not guarantee that the request itself will be sucessful, since
it is still possible for the target node to exit, however, it does remove the
infinite loop which can render client code unusable.
If the value `_controller_id` is out-of-date and the node was removed from the cluster, `_send_request_to_node` would enter an infinite loop.
A call to `maybe_connect` can be performed while the cluster metadata is being updated. If that happens, the assumption that every entry in `_connecting` has metadata won't hold. The existing assert will then raise on every subsequent call to `poll` driving the client instance unusable. This fixes the issue by ignoring connetion request to nodes that do not have the metadata available anymore.
This fixes an issue in check_version where KeyError is raised if the broker is unavailable or an invalid node_id is used. Instead it will return BrokerNotAvailableError.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to: #24
Pushing directly to main
git push fork main --force-with-leaseresulted in permission errors. Discussion here to remove branch protection or have the ability to toggle