[ENH] add Rust pulsar and topic management

[HammadB]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Adds a proper binary for the worker
  • Add an entry point for the worker the binary can use
  • Allow returning of join handles in component handles
• New functionality
  • Adds pulsar subscriptions for the ingest, this watches the memberlist, and then spawns/removes sub-components that will subscribe to the correct stream and deserialize the message. Processing of this stream will be a following PR, this is big enough as is.
  • There are quite a few TODOs that are about error handling in here, I will do a clean up pass in a following PR.

Test plan

How are these changes tested?

• Tests pass locally with cargo test

Documentation Changes

None required.

[HammadB]

Current dependencies on/for this PR:

• main
  • PR [CHORE] Initialize rust #1499
    • PR [ENH] Add Rust Configuration #1500
      • PR [CHORE] Add Rust CI & Basic Dockerfile #1504
        • PR [ENH] Add rust rendezvous hashing and errors #1508
          • PR [ENH] Add rust assignmenment policy and config management #1510
            • PR [ENH] Add rust protobufs and conversion. Add build.rs, protobufs, and conversions #1513
              • PR [ENH] Add rust hnswlib bindings, index interface #1516
                • PR [ENH] Add component system + memberlist provider #1519
                  • PR [ENH] add Rust pulsar and topic management #1528 👈
                  • PR [ENH] Rust SysDB #1529
                  • PR [ENH] Add rust message id conversion for pulsar #1531
                  • PR [ENH] Refactor system to allow mutable handles, wrapped messages, single M recievers #1539
                  • PR [ENH] Add ingest dispatch + scheduler + segment #1542
                  • PR [ENH] Add rust grpc server #1548

This stack of pull requests is managed by Graphite.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readbility, Modularity, Intuitiveness)

[HammadB]

Generalizing this for any assignment