Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: kyvejs runtime containerization and enhancements #80

Closed
wants to merge 22 commits into from

Conversation

kchrist-rocketfueldev
Copy link

@kchrist-rocketfueldev kchrist-rocketfueldev commented Oct 17, 2023

Revamp the KYVE protocol node architecture, which currently consists of two main components: the core and the runtime. The primary focus of this overhaul is the enhancement of the runtime component, which, as of now, is exclusively developed by the KYVE team and implemented in TypeScript.

The primary goal is to introduce a standardized Docker interface, opening the door for the broader community to create their own runtimes in their preferred programming languages, as long as they adhere to the specified Docker interface standards. Consequently, the core component will communicate with the runtime via this Docker interface, simplifying the transfer of data between these two vital parts.

This pull request includes the following key updates:

  • Definition of gRPC protobuf file.
  • Porting of the Tendermint runtime into a standalone Docker container.
  • Development of a command-line tool enabling users to interact with core functions through the command line interface.

These changes collectively aim to make the KYVE protocol node more versatile, allowing a wider community of developers to contribute to and extend its functionality.

Resolves https://linear.app/kyve/issue/TECH-1077
Resolves https://linear.app/kyve/issue/TECH-1076
Resolves https://linear.app/kyve/issue/TECH-1034

@kchrist-rocketfueldev kchrist-rocketfueldev changed the title feat: kyvejs runtime containerization and enhancements #2Feat/runtime containerization feat: kyvejs runtime containerization and enhancements Oct 17, 2023
@harrytou harrytou added the enhancement New feature or request label Oct 17, 2023
@harrytou harrytou requested a review from troykessler October 17, 2023 17:53
@troykessler
Copy link
Member

Here my thoughts:

  • move runtime.proto from common/types to common/protocol since it is the "source of truth" of the runtime <> core communication and is definitely a core feature. Currently the source of truth for the runtime interface is the runtime.interface.ts: question here: what will be the truth of source in the future, do we compile the TS interface out of the proto file?
  • validateDataItem changed, instead of a boolean it returns a number now (indicating either valid, invalid or abstain), this has to be adjusted so @kyvejs/docker successfully builds again

kchrist-rocketfueldev and others added 15 commits October 26, 2023 09:40
- Create runtime gRPC proto definition
- gRPC implementation for tendermint runtime
- Implementation of calling runtime methods from core
- Add separate interface for gRPC runtime

Signed-off-by: Kostas Christopoulos <[email protected]>
- Add CLI commands for calling the runtime directly.
- Change dist path.
- Migrate tendrmint functionality to grpc runtime
- Update docker proxy calling interface
- Fix Dockerfile to allow runtime.proto.
- Reduce Dockerfile stages for optimization.
- Update documentation to include instructions on how to run the image.

Signed-off-by: Kostas Christopoulos <[email protected]>
- Removes second interface implementd for docker integrations
- Removes refactoring to core to call the new interface

Signed-off-by: Kostas Christopoulos <[email protected]>
- cleanup various stuff
- apply docker and code fixes to comply with new file structur

Signed-off-by: Kostas Christopoulos <[email protected]>
- Adapt proto file to needs, remove unneeed stuff, fix types
- Updated changelog and readme
- Add scripts exporting grpc typescript code

Signed-off-by: Kostas Christopoulos <[email protected]>
 * Place the 'runtime.proto' file in 'common/types/src/protos' and use
   it as the source of truth.
 * Change the 'export-grpc-stubs' script in the root of the repo to loop
   through all dirs under 'docker-integrations' and compile the TS files
   from the 'runtime.proto' which is defined in the first bullet.
 * In the index.ts of the 'tendermint-grpc' docker integration, remove
   the dependency of importing the runtime.proto and only use the
   compiled TS files.

Signed-off-by: Harry Touloupas <[email protected]>
The linter can be run from the root of the repo with 'yarn lint'

Signed-off-by: Harry Touloupas <[email protected]>
- remove not needed files

Signed-off-by: Kostas Christopoulos <[email protected]>
Attach the 'shutdown' function to be run when the process receives
SIGINT or SIGTERM. Also add some ESLint configuration files and change
the scripts to use them.

Signed-off-by: Harry Touloupas <[email protected]>
Signed-off-by: Kostas Christopoulos <[email protected]>
@kchrist-rocketfueldev kchrist-rocketfueldev force-pushed the feat/runtime-containerization branch from fcbce58 to 5f92860 Compare October 26, 2023 06:47
- move proto from types to protocal
- updated proto to work with new voting functionality
- updated client/server to work with new proto

Signed-off-by: Kostas Christopoulos <[email protected]>
@kchrist-rocketfueldev kchrist-rocketfueldev force-pushed the feat/runtime-containerization branch from 5f92860 to de2cbb2 Compare October 26, 2023 06:51
package.json Outdated Show resolved Hide resolved
package.json Outdated Show resolved Hide resolved
package.json Outdated Show resolved Hide resolved
@kchrist-rocketfueldev kchrist-rocketfueldev force-pushed the feat/runtime-containerization branch 3 times, most recently from 29c0250 to 1735455 Compare October 30, 2023 07:30
@kchrist-rocketfueldev kchrist-rocketfueldev force-pushed the feat/runtime-containerization branch 5 times, most recently from ef0b4e4 to 16410d4 Compare October 30, 2023 18:02
Signed-off-by: Kostas Christopoulos <[email protected]>
@kchrist-rocketfueldev kchrist-rocketfueldev force-pushed the feat/runtime-containerization branch 3 times, most recently from 06005a7 to cf3b4c3 Compare October 31, 2023 10:33
@kchrist-rocketfueldev kchrist-rocketfueldev force-pushed the feat/runtime-containerization branch from cf3b4c3 to 1bbc2e9 Compare October 31, 2023 10:33
The TARGETARCH build argument for Docker is available by default and can
 be 'amd64' or 'arm64' (between others). Use it to extract the correct
 binary architecture we need to build and copy that to the runtime stage
  of both core and the tendermint runtime.

Also add .dockerignore to all 'common/*' in order to minimize the build
context.

Signed-off-by: Harry Touloupas <[email protected]>
@kchrist-rocketfueldev kchrist-rocketfueldev force-pushed the feat/runtime-containerization branch from 96b3d52 to 003eb32 Compare November 2, 2023 09:15
@kchrist-rocketfueldev kchrist-rocketfueldev force-pushed the feat/runtime-containerization branch from 003eb32 to d4479e6 Compare November 2, 2023 09:16
harrytou
harrytou previously approved these changes Nov 2, 2023
Signed-off-by: Kostas Christopoulos <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants