Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bootstrapping docker container for sample datasets #803

Open
wants to merge 14 commits into
base: development
Choose a base branch
from

Conversation

krishnangovindraj
Copy link
Member

@krishnangovindraj krishnangovindraj commented Feb 21, 2024

What is the goal of this PR?

Introduces a docker image which downloads a config file from GitHub and loads the datasets into a typedb version specified in the config.

What are the changes implemented in this PR?

Adds

  • A Dockerfile for the image which contains:
  • boostrapper.py: Python script which sets up the
  • The gpg key for the cloudsmith apt repo, preloaded.
  • A config file for the bootstrapped at the expected location (this location is hard-coded in the bootstrapper)
  • A factory test which builds an image, containing the config file and datasets files and runs it + tests it is available.

Reasons for 'do not merge'

  • CI will fail currently, Since the datasets in config.yml are only on jameswhiteside/development. We can edit this out / Add a sample instead.
  • Once we're happy, we need to push version 1.0.0 of the docker image to dockerhub.
# From the root of the repo
> docker login # As someone who has rights to push
# Comment out the lines to copy the datasets in. That's only needed for CI.
> docker build  . -f typedb-samples-docker/Dockerfile  --tag vaticle/typedb-sample-datasets:1.0.0
> docker push vaticle/typedb-sample-datasets:1.0.0 

We should not need to release a new version unless the bootstrapping process (or something hardcoded in it) changes.

@typedb-bot
Copy link
Member

PR Review Checklist

Do not edit the content of this comment. The PR reviewer should simply update this comment by ticking each review item below, as they get completed.


Trivial Change

  • This change is trivial and does not require a code or architecture review.

Code

  • Packages, classes, and methods have a single domain of responsibility.
  • Packages, classes, and methods are grouped into cohesive and consistent domain model.
  • The code is canonical and the minimum required to achieve the goal.
  • Modules, libraries, and APIs are easy to use, robust (foolproof and not errorprone), and tested.
  • Logic and naming has clear narrative that communicates the accurate intent and responsibility of each module (e.g. method, class, etc.).
  • The code is algorithmically efficient and scalable for the whole application.

Architecture

  • Any required refactoring is completed, and the architecture does not introduce technical debt incidentally.
  • Any required build and release automations are updated and/or implemented.
  • Any new components follows a consistent style with respect to the pre-existing codebase.
  • The architecture intuitively reflects the application domain, and is easy to understand.
  • The architecture has a well-defined hierarchy of encapsulated components.
  • The architecture is extensible and scalable.

Copy link
Member Author

@krishnangovindraj krishnangovindraj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self review

sleep 30
docker exec test-typedb-samples-container typedb console --command="database list" && export TEST_SUCCESS=0 || export TEST_SUCCESS=1
docker stop test-typedb-samples-container; docker rm test-typedb-samples-container
exit $TEST_SUCCESS
Copy link
Member Author

@krishnangovindraj krishnangovindraj Feb 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How often would we want to run this test?
It shouldn't be significantly longer than testing it against a local installation of TypeDB, which is the alternative.

The test has run successfully in CI: https://factory.vaticle.com/krishnangovindraj/typedb-docs/1168c68815d9f9fa01580dbc4f8a9233607d677a/build/1/correctness/1/test-sample-bootstrapper/1

@@ -0,0 +1,12 @@
# Track version for breaking changes.
bootstrapper-version: 1.0.0
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bootstrapper-version for if/when we have to make breaking changes.

COPY typedb-samples-docker/bootstrapper.py /typedb-samples

# We need the 'exec' before typedb server, or the SIGTERM from `docker stop` will not forwarded to it
CMD python3 -u bootstrapper.py && exec typedb server
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lack of exec on the first also means we can't interrupt the container bootstrap

VERBOSE = os.environ.get("BOOTSTRAPPER_VERBOSE", "false").lower() != "false"
# Overrides for testing with local files
CONFIG_OVERRIDE = os.environ.get("BOOTSTRAPPER_CONFIG", None)
DATASET_ROOT_OVERRIDE = os.environ.get("BOOTSTRAPPER_DATASET_ROOT", None)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overrides for testing with local files. This can be used to test against a local typedb instance, or in a locally built docker container as the CI test does.

@izmalk izmalk removed their assignment Aug 15, 2024
@typedb-bot typedb-bot removed the request for review from izmalk August 15, 2024 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants