From bddbe19b9106aa633002f3682423785a7011998a Mon Sep 17 00:00:00 2001 From: Faiyaz Hasan Date: Tue, 16 May 2023 22:43:12 -0400 Subject: [PATCH] Expose GCP Batch RTD (#1650) * Expose GCP batch executor rtd * Update RTD * Expose GCP RTD * removing the tab spaces * removing few more tabs * hopefully final tab --------- Co-authored-by: Santosh kumar <29346072+santoshkumarradha@users.noreply.github.com> --- CHANGELOG.md | 4 ++- doc/source/api/executors/gcpbatch.rst | 47 +++++++++++++-------------- doc/source/api/executors/index.rst | 1 + doc/source/plugins.rst | 1 + 4 files changed, 27 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d9f52ce22..e9ad4ac28 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Docs - Redispatch feature page in Read the Docs. -- Clarify installation instructions for SLURM plugin in Read the Docs. +- Clarify installation instructions for SLURM plugin in Read the Docs (x2). +- Fix waiting order of electrons in docs inside snippet for adding a dependency when inputs and outputs are independent. +- Expose GCP Batch executor RTD. ### Fixed diff --git a/doc/source/api/executors/gcpbatch.rst b/doc/source/api/executors/gcpbatch.rst index 967fde43a..49c077af7 100644 --- a/doc/source/api/executors/gcpbatch.rst +++ b/doc/source/api/executors/gcpbatch.rst @@ -25,7 +25,7 @@ To use this plugin with Covalent, simply install it using :code:`pip`: 2. Usage Example =========================================== -Here we present an example on how a user can use the GCP Batch executor plugin in their Covalent workflows. In this example we train a simple SVM (support vector machine) model using the Google Batch executor. This executor is quite minimal in terms of the required cloud resoures that need to be provisioned prior to first use. The Google Batch executor needs the following cloud resources pre-configured +Here we present an example on how a user can use the GCP Batch executor plugin in their Covalent workflows. In this example we train a simple SVM (support vector machine) model using the Google Batch executor. This executor is quite minimal in terms of the required cloud resources that need to be provisioned prior to first use. The Google Batch executor needs the following cloud resources pre-configured: * A Google storage bucket * Cloud artifact registry for Docker images @@ -39,7 +39,7 @@ Here we present an example on how a user can use the GCP Batch executor plugin i .. note:: - Details about Google services accounts and how to use them properly can be found `here `_ + Details about Google services accounts and how to use them properly can be found `here `_. .. code-block:: python @@ -54,13 +54,14 @@ Here we present an example on how a user can use the GCP Batch executor plugin i executor = ct.executor.GCPBatchExecutor( bucket_name = "my-gcp-bucket", + region='us-east1', project_id = "my-gcp-project-id", container_image_uri = "my-executor-container-image-uri", service_account_email = "my-service-account-email", - vcpu = 2, # Number of vCPUs to allocate - memory = 512, # Memory in MB to allocate - time_limit = 300, # Time limit of job in seconds - poll_freq = 3 # Number of seconds to pause before polling for the job's status + vcpus = 2, # Number of vCPUs to allocate + memory = 512, # Memory in MB to allocate + time_limit = 300, # Time limit of job in seconds + poll_freq = 3 # Number of seconds to pause before polling for the job's status ) # Use executor plugin to train our SVM model. @@ -115,11 +116,7 @@ Here we present an example on how a user can use the GCP Batch executor plugin i print(result) -During the execution of the workflow one can navigate to the UI to see the status of the workflow, once completed however the above script should also output a value with the score of our model. - -.. code-block:: python - - 0.8666666666666667 +During the execution of the workflow the user can navigate to the web-based browser UI to see the status of the computations. =========================================== @@ -210,39 +207,39 @@ In order to successfully execute tasks using the Google Batch executor, some clo * Google storage bucket - The executor uses a storage bucket to store/cache exception/result objects that get generated during execution +The executor uses a storage bucket to store/cache exception/result objects that get generated during execution. * Google Docker artifact registry - The executor submits a container job whose image is pulled from the provided ``container_image_uri`` argument of the executor +The executor submits a container job whose image is pulled from the provided ``container_image_uri`` argument of the executor. * Service account - Keeping good security practices in mind, the jobs are executed using a service account that only has the necessary permissions attached to it that are required for the job to finish. +Keeping good security practices in mind, the jobs are executed using a service account that only has the necessary permissions attached to it that are required for the job to finish. - Users can free to provision these resources as they see fit or they can use Covalent to provision these for them. Covalent CLI can be used to deploy the required cloud resources. Covalent behind the scenes uses `Terraform `_ to provision the cloud resources. The terraform HCL scripts can be found in the plugin's Github repository `here `_. +Users can free to provision these resources as they see fit or they can use Covalent to provision these for them. Covalent CLI can be used to deploy the required cloud resources. Covalent behind the scenes uses `Terraform `_ to provision the cloud resources. The terraform HCL scripts can be found in the plugin's Github repository `here `_. - To run the scripts manually, users must first authenticate with Google cloud via their CLI +To run the scripts manually, users must first authenticate with Google cloud via their CLI and print out the access tokens with the following commands: - .. code:: shell +.. code:: shell - gcloud auth login + gcloud auth application-default login + gcloud auth print-access-token - Once the user has authenticated, the infrastructure can be stood up by simply apply the Terraform scripts i.e. +Once the user has authenticated, the infrastructure can be deployed by running the Terraform commands in the `infra` folder of the plugin's repository. - .. code:: shell +.. code:: shell terraform plan -out tf.plan - terrafrom apply tf.plan - + terrafrom apply tf.plan -var="access_token=" - .. note:: +.. note:: - For first time deployment, the terraform provides must be initialized properly via ``terraform init`` + For first time deployment, the terraform provides must be initialized properly via ``terraform init``. -The HCL scripts also build the base executor docker image and upload it to the artficat registry after it gets created. This way the user need not build and push an image separately as the process is fully automated via Covalent. +The HCL scripts also build the base executor docker image and upload it to the artifact registry after it gets created. This way the user need not build and push an image separately as the process is fully automated via Covalent. diff --git a/doc/source/api/executors/index.rst b/doc/source/api/executors/index.rst index 364a02202..33c3ef841 100644 --- a/doc/source/api/executors/index.rst +++ b/doc/source/api/executors/index.rst @@ -25,6 +25,7 @@ Plugins awsec2 awsecs awsbatch + gcpbatch Base executor ************* diff --git a/doc/source/plugins.rst b/doc/source/plugins.rst index 494d4b1d3..50388ac1f 100644 --- a/doc/source/plugins.rst +++ b/doc/source/plugins.rst @@ -19,3 +19,4 @@ Covalent offers various plugins, starting with executor plugins. Executors are u api/executors/awsbatch.rst api/executors/awsecs.rst api/executors/awsbraket.rst + api/executors/gcpbatch.rst