CALM-DSL changes for v3.7.2.2 #295
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change of string text from "existing_markeplace_runbook" to **existing_marketplace_runbook** (cherry picked from commit ddfd8edee8e31b8cde76bf9250ff4e2ccad21949)
Change of text coma to comma (cherry picked from commit c459a80c920647730539cb37e1b9874484321b93)
) Passing with_endpoints flag value to the publish_runbook functions. 1. --with_endpoints flag set and runbook published in marketplace <img width="613" alt="Screenshot 2023-10-04 at 2 44 59 PM" src="https://github.com/ideadevice/calm-dsl/assets/134352330/1024ece1-1bdd-4f7c-9ad0-aaee83f1bc07"> > (venv) [Wed Oct 04 11:25:28 nutanix@richa-kumari:~/calm-dsl ] calm publish runbook --name "My Runbook 1" --version 1.2.6 --project "test project" --with_secrets --with_endpoints --publish_to_marketplace "test_rb" [2023-10-04 11:25:50] [INFO] [calm.dsl.cli.marketplace:1636] Fetching existing marketplace runbooks with name My Runbook 1 [2023-10-04 11:25:51] [INFO] [calm.dsl.cli.runbooks:596] test_rb found [2023-10-04 11:25:52] [INFO] [calm.dsl.cli.marketplace:1615] Marketplace Runbook is published to marketplace manager successfully [2023-10-04 11:25:52] [INFO] [calm.dsl.cli.marketplace:1081] Fetching details of pending marketplace item My Runbook 1 with version 1.2.6 [2023-10-04 11:25:54] [INFO] [calm.dsl.cli.marketplace:1154] Marketplace Item My Runbook 1 with version 1.2.6 is approved successfully [2023-10-04 11:25:54] [INFO] [calm.dsl.cli.marketplace:1185] Fetching details of accepted marketplace item My Runbook 1 with version 1.2.6 [2023-10-04 11:25:55] [INFO] [calm.dsl.cli.marketplace:1265] Marketplace Item is published to marketplace successfully 2. --with_endpoints flag not set <img width="592" alt="Screenshot 2023-10-04 at 2 45 08 PM" src="https://github.com/ideadevice/calm-dsl/assets/134352330/77de168f-099a-4e7c-866a-df389fcee0e6"> > (venv) [Wed Oct 04 11:27:22 nutanix@richa-kumari:~/calm-dsl ] calm publish runbook --name "My Runbook 1" --version 1.2.7 --project "test project" --with_secrets -e "test_r b" [2023-10-04 11:28:12] [INFO] [calm.dsl.cli.marketplace:1703] Fetching existing marketplace runbooks with name My Runbook 1 [2023-10-04 11:28:13] [INFO] [calm.dsl.cli.marketplace:1727] Fetching existing versions of Marketplace Item My Runbook 1 [2023-10-04 11:28:13] [INFO] [calm.dsl.cli.runbooks:596] test_rb found [2023-10-04 11:28:13] [INFO] [calm.dsl.cli.marketplace:1615] Marketplace Runbook is published to marketplace manager successfully JIRA - https://jira.nutanix.com/browse/CALM-39483 (cherry picked from commit aeb3f630b8f1aa77412354659fd39771cca02444)
(cherry picked from commit 00686f6c26b7374a741ff3fbc968bfb8ef917bf5)
Added Backslash in the string before passing it to jinja (cherry picked from commit 1cb16dfbd476d9bd9b9fc8b02212b5c8f93f6200)
The secret variable strip was not happening before the upload API. Made changes to support it. (cherry picked from commit 349d81d472aade82fb92f676745878c93921ba3c)
https://jira.nutanix.com/browse/CALM-39476. As of now users doesn't have an option to update the Marketplace runbook or Marketplace blueprints with all_projects flag. As part of this review made changes in order to support the update of marketplace runbook or blueprints with all the projects in the system. --------- Co-authored-by: Prabhat Dwivedi <[email protected]> (cherry picked from commit 26255bc96ca0f133173cec154370a071104a16e3)
### Summary Include static ip address assosciated with nics in the schema open issue: #213 decompiled the blueprint with the following nic config: <img width="1182" alt="Screenshot 2023-10-08 at 3 06 55 PM" src="https://github.com/ideadevice/calm-dsl/assets/89129891/34eed26b-3a94-43df-a92f-c2a9d3fe492e"> following is the output after decompiling ``` # THIS FILE IS AUTOMATICALLY GENERATED. # Disclaimer: Please test this file before using in production. """ Generated blueprint DSL (.py) """ import json # no_qa import os # no_qa from calm.dsl.builtins import * # no_qa # Secret Variables BP_CRED_c1_PASSWORD = read_local_file("BP_CRED_c1_PASSWORD") # Credentials BP_CRED_c1 = basic_cred( "admin", BP_CRED_c1_PASSWORD, name="c1", type="PASSWORD", default=True, ) class Service1(Service): pass class vmcalm_timeResources(AhvVmResources): memory = 1 vCPUs = 1 cores_per_vCPU = 1 disks = [ AhvVmDisk.Disk.Scsi.cloneFromImageService( "Centos7HadoopMaster.qcow2", bootable=True ) ] nics = [ AhvVmNic.NormalNic.ingress( "abc", cluster="auto_cluster_prod_4f683249d95b", ip_endpoints=["10.1.0.9"] ), AhvVmNic.NormalNic.ingress( "vlan.154", cluster="auto_cluster_prod_4f683249d95b" ), ] class vmcalm_time(AhvVm): name = "vm-@@{calm_time}@@" resources = vmcalm_timeResources cluster = Ref.Cluster(name="auto_cluster_prod_4f683249d95b") class VM1(Substrate): account = Ref.Account("NTNX_LOCAL_AZ") os_type = "Linux" provider_type = "AHV_VM" provider_spec = vmcalm_time readiness_probe = readiness_probe( connection_type="SSH", disabled=True, retries="5", connection_port=22, address="@@{platform.status.resources.nic_list[0].ip_endpoint_list[0].ip}@@", delay_secs="0", ) class Package1(Package): services = [ref(Service1)] class d2d34557_deployment(Deployment): min_replicas = "1" max_replicas = "1" default_replicas = "1" packages = [ref(Package1)] substrate = ref(VM1) class Default(Profile): environments = [Ref.Environment(name="Default Environment")] deployments = [d2d34557_deployment] @action def UntitledAction1(name="Untitled Action1"): CalmTask.Exec.ssh( name="Task1", filename=os.path.join( "scripts", "Profile_Default_Action_UntitledAction1_Task_Task1.sh" ), target=ref(Service1), ) class test_nic_bp(Blueprint): services = [Service1] packages = [Package1] substrates = [VM1] profiles = [Default] credentials = [BP_CRED_c1] class BpMetadata(Metadata): categories = {"TemplateType": "Vm"} ``` also verified that the compile works correctly on the above decompiled bp, following is the output of compilation (cherry picked from commit acfb95a0ece7de6234347772fe2b350a26c43a81)
(cherry picked from commit 5211ab451afd31ca58b24427c1a10247db55dfc8)
Added for Endpoints, Blueprints, Runbooks and Accounts [CALM-39484](https://jira.nutanix.com/browse/CALM-39484) Addressed: #189 (cherry picked from commit 5a6216b6e4fe4887a28688e3e7a584181e6c2eb5)
1. `In case of invalid project: Project doesn't exist in system` <img width="1246" alt="Screenshot 2023-10-09 at 11 24 06 PM" src="https://github.com/ideadevice/calm-dsl/assets/52671728/b87d885b-d7a4-4cca-b4f5-5c13611f1388"> ```===========================================================================``` 2. `In case of invalid project: Project is not associated with MPI` <img width="1263" alt="Screenshot 2023-10-09 at 11 23 54 PM" src="https://github.com/ideadevice/calm-dsl/assets/52671728/54df9e38-21db-4082-964c-4483dc00c9bc"> ```===========================================================================``` 3. `In case of unpublish from all versions from specific project` <img width="1241" alt="Screenshot 2023-10-09 at 11 51 01 PM" src="https://github.com/ideadevice/calm-dsl/assets/52671728/a0a73f17-af0e-4c2b-bd09-bd99a549cfe7"> ```===========================================================================``` 4. `Incase of unpublish from specific version specific project` <img width="1277" alt="Screenshot 2023-10-09 at 11 25 03 PM" src="https://github.com/ideadevice/calm-dsl/assets/52671728/d73d4807-3fac-4ed1-b28f-09d18c1de204"> ```===========================================================================``` 5. `Incase of unpublish from all versions` <img width="1092" alt="Screenshot 2023-10-10 at 12 04 46 AM" src="https://github.com/ideadevice/calm-dsl/assets/52671728/928a09a9-74ef-444b-9da8-aa4ceab0eb9c"> ```===========================================================================``` 6. `Incase of unpublish from specific version` <img width="1257" alt="Screenshot 2023-10-09 at 11 25 36 PM" src="https://github.com/ideadevice/calm-dsl/assets/52671728/3754b817-9a12-4a0e-9356-09b9d80c65a3"> (cherry picked from commit 9ffa0b3f737f800fa6aaa21d0a5322a07659f47e)
(cherry picked from commit 4ea68ea1776536a60aec82a64c1e7940ce6a3f04)
#305) Allow read_local_file to read from default value if file does not exist at given path Bug link: https://jira.nutanix.com/browse/CALM-39470 Issue link: #183 (cherry picked from commit 5a8ba71bac7a9f599d9561b1ac5457be56374d66)
### Summary Added support in action tasks to render endpoints. When decompiling tasks, if they contain `exec_target_reference` we should use those instead of the `target_any_local_reference` to get the endpoint. open issue: #177 decompiled the below blueprint with Task1 having an explicit endpoint and Task2 having empty endpoint  result of decompile: ``` # THIS FILE IS AUTOMATICALLY GENERATED. # Disclaimer: Please test this file before using in production. """ Generated blueprint DSL (.py) """ import json # no_qa import os # no_qa from calm.dsl.builtins import * # no_qa from calm.dsl.runbooks import CalmEndpoint as Endpoint # Secret Variables BP_CRED_c1_PASSWORD = read_local_file("BP_CRED_c1_PASSWORD") # Credentials BP_CRED_c1 = basic_cred( "admin", BP_CRED_c1_PASSWORD, name="c1", type="PASSWORD", default=True, ) # Endpoints ep2 = Endpoint.use_existing("ep2") class Service1(Service): pass class vmcalm_timeResources(AhvVmResources): memory = 1 vCPUs = 1 cores_per_vCPU = 1 disks = [ AhvVmDisk.Disk.Scsi.cloneFromImageService( "Centos7HadoopMaster.qcow2", bootable=True ) ] nics = [ AhvVmNic.NormalNic.ingress( "abc", cluster="auto_cluster_prod_4f683249d95b", ip_endpoints=["10.1.0.9"] ), AhvVmNic.NormalNic.ingress( "vlan.154", cluster="auto_cluster_prod_4f683249d95b" ), ] class vmcalm_time(AhvVm): name = "vm-@@{calm_time}@@" resources = vmcalm_timeResources cluster = Ref.Cluster(name="auto_cluster_prod_4f683249d95b") class VM1(Substrate): account = Ref.Account("NTNX_LOCAL_AZ") os_type = "Linux" provider_type = "AHV_VM" provider_spec = vmcalm_time readiness_probe = readiness_probe( connection_type="SSH", disabled=True, retries="5", connection_port=22, address="@@{platform.status.resources.nic_list[0].ip_endpoint_list[0].ip}@@", delay_secs="0", ) class Package1(Package): services = [ref(Service1)] class d2d34557_deployment(Deployment): min_replicas = "1" max_replicas = "1" default_replicas = "1" packages = [ref(Package1)] substrate = ref(VM1) class Default(Profile): environments = [Ref.Environment(name="Default Environment")] deployments = [d2d34557_deployment] @action def UntitledAction1(name="Untitled Action1"): CalmTask.Exec.ssh( name="Task1", filename=os.path.join( "scripts", "Profile_Default_Action_UntitledAction1_Task_Task1.sh" ), target=ref(ep2), ) CalmTask.Exec.ssh( name="Task2", filename=os.path.join( "scripts", "Profile_Default_Action_UntitledAction1_Task_Task2.sh" ), target=ref(Service1), ) class test_nic_bp(Blueprint): services = [Service1] packages = [Package1] substrates = [VM1] profiles = [Default] credentials = [BP_CRED_c1] class BpMetadata(Metadata): categories = {"TemplateType": "Vm"} ``` (cherry picked from commit 2e2b06e61c2cf9bc52f725db0f83d46fbfd5d190)
### Summary Added support to decompile , compile and create dynamic variables that use http task with basic auth ### test decompiled below runbook with dynamic http variable having basic auth <img width="694" alt="Screenshot 2023-10-10 at 6 45 27 PM" src="https://github.com/ideadevice/calm-dsl/assets/89129891/1ddc5893-625d-4f68-b064-7b9a341ff19a"> verified that decompile works correctly. Local file for the basic auth is created and a reference to this is used in the runbook. When create is used the credential is read from the file and stripped, later on the secret is patched. (cherry picked from commit 14424f3e201f8f49b1568755bcf6c18bfbec118c)
If entities associated with project/environment, force create is not possible. So in this case showing the usage of projects/environments like shown in below screen <img width="1188" alt="Screenshot 2023-10-13 at 4 41 27 PM" src="https://github.com/ideadevice/calm-dsl/assets/52671728/74245941-ab01-4c0d-89f8-e7c56173f783"> If there are no entities associated with project/environment and if the project/environment is existed in system we try to delete them and create a new one. <img width="1397" alt="Screenshot 2023-10-13 at 4 55 32 PM" src="https://github.com/ideadevice/calm-dsl/assets/52671728/8feadd62-bee0-4de3-943f-9f70d2380036"> <img width="1512" alt="Screenshot 2023-10-13 at 5 49 03 PM" src="https://github.com/ideadevice/calm-dsl/assets/52671728/b691018f-5196-4be8-92f5-4de109330306"> **Readme Preview** <img width="1243" alt="Screenshot 2023-10-13 at 5 53 11 PM" src="https://github.com/ideadevice/calm-dsl/assets/52671728/e3056e2b-e241-4a4d-beb8-a74f0b882e2d"> (cherry picked from commit e6b223ce296ba06b43d0fd069f13a417b6040638)
…306) CALM-13856 : https://jira.nutanix.com/browse/CALM-13856 CALM-13793 : https://jira.nutanix.com/browse/CALM-13793 1. Verified support for basic auth with credentials is added through dsl while creating http task 2. Verified addition of secret headers for basic auth with credentials and basic auth (cherry picked from commit 9722c747e84c4e0eea6ea76ce1e8655633916584)
…309) Resource types list is fetched from accounts_uuid_resource_types_list instead of resource_types_uuid Cred dict using accounts_uuid_resource_types_list api - <img width="1109" alt="Screenshot 2023-10-16 at 10 42 24 AM" src="https://github.com/ideadevice/calm-dsl/assets/134352330/bb421125-b85f-4993-8aa5-430ad3919a5c"> Cred dict using resource_types_uuid api - <img width="1113" alt="Screenshot 2023-10-16 at 10 38 49 AM" src="https://github.com/ideadevice/calm-dsl/assets/134352330/e40f10c5-392a-476d-a07e-522c9d0ebc79"> Cred dict using accounts_uuid_resource_types_list api (Schema list with secret type and local type variables) - <img width="1136" alt="Screenshot 2023-10-16 at 12 26 19 PM" src="https://github.com/ideadevice/calm-dsl/assets/134352330/2e948d46-304a-49b8-8dc4-03dbce4cf822"> Regression link - http://erdinger.eng.nutanix.com/job/QA/job/DSL_Regression/518/#showFailuresLink (cherry picked from commit 11c45478b4308c4e8515ddb75c609df03c8f6ae3)
It was expecting cluster from overlay subnets which it should not Have made the changes and now vpc name will be displayed for overlay subnets (cherry picked from commit ea98956a302e1e482b07212142bac8b932219bc4)
### Summary Quick fix for when variable attrs don't have auth. (cherry picked from commit 7b9ff4b2f64a769e06b3c757ebd5638f521f4b2b)
Summary: NDB account absence was failing other tests to run during dsl regression. Renamed ndb files so that it doesn't get collected while executing pytest. (cherry picked from commit 090b0123028aa6e23aff5e634096dd8676da2697)
Summary: Add uuid to creds and add uuid readiness probe to environment payload credential reference Test Plan: Tested by creating a sample azure environement, and verifying the creation of the same in the UI Jira Link: https://jira.nutanix.com/browse/CALM-39478 (cherry picked from commit 0e73db57ed586c7d3fb44f4daab4a86227263a7b)
(cherry picked from commit b9ca92afa1f06153619f616504bbe97bccb8bb76)
Added secrets of NDB task to a local file. [CALM-39123](https://jira.nutanix.com/browse/CALM-39123) E.g. Runbook after changes: ``` import os import json from calm.dsl.runbooks import * from calm.dsl.runbooks import ( CalmEndpoint as Endpoint, RunbookTask as CalmTask, RunbookVariable as CalmVariable, ) from calm.dsl.builtins import CalmTask as CalmVarTask, Metadata # Secret Variables NDB_task_2_DatabaseServer_password = read_local_file( "NDB_task_2_DatabaseServer_password" ) NDB_task_2_DatabaseServer_ssh_public_key = read_local_file( "NDB_task_2_DatabaseServer_ssh_public_key" ) NDB_task_2_Database_initial_database_password = read_local_file( "NDB_task_2_Database_initial_database_password" ) # Runbook https://github.com/Runbook def utk_rb(): regex_var = CalmVariable.Simple( "", label="", regex="^(|[a-zA-Z][A-Za-z0-9_.-]+)$", validate_regex=True, is_mandatory=False, is_hidden=False, runtime=True, description="", ) # noqa CalmTask.SetVariable.escript.py2( name="Task 1", filename=os.path.join("scripts", "_Runbook_utk_rb_Task_Task1.py"), variables=["var1", "var2", "var3"], ) CalmTask.NutanixDB.PostgresDatabase.Create( name="Task 2", account=Ref.Account("ndb_acc"), database_server_config=DatabaseServer.Postgres.Create( password=NDB_task_2_DatabaseServer_password, software_profile=Ref.NutanixDB.Profile.Software(""), software_profile_version=Ref.NutanixDB.Profile.Software_Version(""), compute_profile=Ref.NutanixDB.Profile.Compute(""), cluster=Ref.NutanixDB.Cluster(""), ssh_public_key=NDB_task_2_DatabaseServer_ssh_public_key, name="'zds'jj\"nk\"", network_profile=Ref.NutanixDB.Profile.Network(""), ), instance_config=Database.Postgres.Create( name="", description="", database_parameter_profile=Ref.NutanixDB.Profile.Database_Parameter(""), listener_port="5432", size="200", initial_database_name="DB", initial_database_password=NDB_task_2_Database_initial_database_password, pre_create_script='"dsfds"', post_create_script="", ), timemachine_config=TimeMachine.Postgres.Create( name="", description="", sla=Ref.NutanixDB.SLA(""), snapshottimeofday__hours="5", snapshottimeofday__minutes="0", snapshottimeofday__seconds="0", logbackup_interval="30", snapshots_perday="1", weeklyschedule__dayofweek="MONDAY", monthlyschedule__dayofmonth="1", quartelyschedule__startmonth="JANUARY", ), tag_config=Tag.Create(), outargs=PostgresDatabaseOutputVariables.Create(), ) class RunbookMetadata(Metadata): project = Ref.Project("utk_proj") ``` --------- --------- Co-authored-by: utkarsh bairolia <[email protected]> Co-authored-by: kushjajal7 <[email protected]> Co-authored-by: naveenuk2411 <[email protected]> Co-authored-by: Abhishek Patwardhan <[email protected]> Co-authored-by: Gullapalli Akhil Sai <[email protected]> Co-authored-by: Prabhat Dwivedi <[email protected]> Co-authored-by: richakumarii <[email protected]> Co-authored-by: prakharsingh1000 <[email protected]> (cherry picked from commit d7606f930b49d69c3103408c78dd41f4a8efa38a)
Added OS env flag to compile with secrets. By default it is false. --------- Co-authored-by: Abhijeet Singh Kaurav <[email protected]> (cherry picked from commit cd15255aab95c0037ecf71130ecceea77ae655fa)
1. Added distributed_virtual_switch, vm_recovery_point to all the ACPs. 2. Added report config for Project admin. 3. Added collab independent 3rd index ACP for global entities within the project scope. 4. Moved user and use group to project scope. [CALM-40511](https://jira.nutanix.com/browse/CALM-40511) --------- (cherry picked from commit 09e570a3f1bbc8763e9ee6f878287c270e3b42f3)
(cherry picked from commit c8dccd8f7d5baf79ac43d6c44ffa4624ad6996fb)
Environment decoupling support in DSL. Added provider compilation templates for ntnx, vmware, gcp, aws. Use command: calm decompile environment -n <env_name> -p <project_name> --------- Co-authored-by: Abhijeet Kaurav <[email protected]> Co-authored-by: Utkarsh Bairolia <[email protected]> (cherry picked from commit 70896ec9005e3ee5c2a4a5fd7c9d824c1c1c85ab)
CALM-17414 : https://jira.nutanix.com/browse/CALM-17414 Summary: 1. Made changes in simple bp model to accept packages as list while creating bp. 2. Added tc to test downloadable image functionality.    Note: Merge this PR after https://github.com/ideadevice/jenkinsfile-scripts/pull/409/files (cherry picked from commit 4b522abd7e6d5115d42d999f4dafe99be5b6044f)
CALM-38760: Failed to create VMWare brownfield app Issue Link: https://jira.nutanix.com/browse/CALM-38760 Description: 1. Fixed guest.ipAddress field depending on calm version <img width="1160" alt="image" src="https://github.com/ideadevice/calm-dsl/assets/123161845/f3f47773-0ca9-42dd-9e1a-f3deabc7b27b"> (cherry picked from commit 3e4c58e75ca07d8bdc0bbf85d78147b9aa88a8ab)
…bp (#395) Description: RCA: Decompiling cluster macro was failing due to search in DSL cache for cluster macro variable. Fix: No need to search in cache and make cluster object directly from macro. Succesfull decompile: <img width="1178" alt="image" src="https://github.com/ideadevice/calm-dsl/assets/123161845/f9afdf82-854b-40a2-9536-c8144c967bb6"> (cherry picked from commit 253246d88cadbd4115fe367ab15d8b329106c761)
CALM-43323: https://jira.nutanix.com/browse/CALM-43323 Description: 1. deleted state filter doesn't work normally with parenthesis while making get api call to `api/nutanix/v3/apps/list` (sample params: {`"length": 100, "offset": 0, "filter": "(_state==deleted)"}`) 2. Above param works fine with other states 3. Moreover, `{"length": 100, "offset": 0, "filter": "_state==deleted"}` works with deleted state, hence using it as workaround till https://jira.nutanix.com/browse/CALM-43342 is resolved  (cherry picked from commit 05b5ca3d4877f8f01b67c613ffd8548aab675859)
Issue Link : https://jira.nutanix.com/browse/CALM-30977 Description: 1. Added `__post_create__ ` field to create tasks post vm creations. 2. Made changes in power state field to support creation of vm in both on and off phases in AHV, VMWARE. As currently only AHV and VMWARE support this thing. By default power state will be ON. 3. Added support for vm power actions(vm power on, vm power off, vm restart, vm check login) in service/profile level actions. 4. Added decompilation support for vm power actions. 5. Added UT to check power state and power actions functionality. 6. More details: https://docs.google.com/document/d/1-QznXHGfAEHxqx8GZT4GhRpHzeeox87R04ury_vAdxA/edit?usp=sharing 7. Scenarios tested: https://docs.google.com/spreadsheets/d/1Qsk6wNwHLtkA7oUu9-1Zkkn9fzSmPBC4uyRaY9M8cHE/edit?usp=sharing --------- Co-authored-by: Abhijeet Kaurav <[email protected]> (cherry picked from commit 7cf5ca3b6a8edaa2bad0198033c86141148e11db)
CALM-39474: Cache update in case if config_file cli option is passed Issue Link: https://jira.nutanix.com/browse/CALM-39474 Description: 1. Whenever config_file cli option is passed in any calm command. It updates the cache and context with file passes from cli option. 2. Update happens only if PC IP supplied from new config file is different from old PC. When new PC is supplied from config: update cache and context. <img width="1084" alt="image" src="https://github.com/ideadevice/calm-dsl/assets/123161845/e46e2d4e-e749-4f0c-912c-b6c5a91f92e5"> When same PC is supplied from config: no updation.  (cherry picked from commit 3e3657da7a392ba00ed8cf7d1dcf834b7110dcb8)
… (#380) CALM-42133: (https://jira.nutanix.com/browse/CALM-42133) Description: 1. Describe: added support to describe patch config actions using `calm describe app-migratable-entities "<app_name>"` 2. Compilation: Added support to create patch config actions in class `AhvUpdateConfigAttrs` 3. Decompilation: Added support to decompile patch config actions `calm decompile app-migratable-bp "<app_name>"` 4. Details and flow of commands updated in jira ticket 5. Added UT  --------- Co-authored-by: Abhijeet Kaurav <[email protected]> (cherry picked from commit 016f7f0b25971264ac8d9051671443a6e19b350f)
CALM-40213 - https://jira.nutanix.com/browse/CALM-40213 Description: 1. Modified protection policy model 2. Added support to create snapshot configs for vmware 3. Fixed decompilation after modifying app protection policy model 4. Modified decompilation of metadata project (as metadata project was not coming up in decompilation) 5. Added Ahv, Vmware class in SnapshotConfig, RestoreConfig class to make snapshot/restore of respective provider. To keep backward compatibility: by default Ahv class is pointed if no provider class is invoked. 6. Added UT and Integration Tests for snapshot configs 7. Other relevant logs and info available in ticket Test cases check:  Snapshot creation: <img width="992" alt="Screenshot 2024-01-17 at 6 28 58 PM" src="https://github.com/ideadevice/calm-dsl/assets/123161845/c705e954-2bde-4435-a51c-4f3f16255494"> Snapshot restore: <img width="1013" alt="Screenshot 2024-01-17 at 6 25 30 PM" src="https://github.com/ideadevice/calm-dsl/assets/123161845/76eec0e5-8879-4a24-bc2b-f18f772da6d8"> PC view: <img width="869" alt="Screenshot 2024-01-17 at 6 27 14 PM" src="https://github.com/ideadevice/calm-dsl/assets/123161845/0a646c77-e538-4045-bf40-e29c454705f0"> (cherry picked from commit c145c00f53f3a429311e95ba65bbd8cce6beb65c)
Description: 1. Encountered assertion failure in runbook jsons. RCA: Test failed due to conversion of runbook scripts to py3 through ideadevice/calm-dsl#386 but no corresponding changes in asserted json. Fix: Used updated json with py3 scripts instead of py2 in asserted json. 2. bp json assertion error when decompiled bp json is asserted. RCA: VM power actions were also decompiled even if they were not used in any custom actions. Resulting in difference of original and decompiled bp payload. Fix: Decompiling vm power actions only if they are used in other custom actions. 3. bp creation failing in some tests. RCA: config_type was getting updated in case of patch type also in profile.py. Fix: config_type should not update if it of type "patch" RP Link: http://rp.calm.nutanix.com/ui/#calm/launches/all/65f435d02ab79c0001d99488?page.page=1&page.size=50&page.sort=start_time%2CASC Build LInk: http://erdinger.eng.nutanix.com/blue/organizations/jenkins/QA%2FDSL_Regression/detail/DSL_Regression/581/pipeline/ (cherry picked from commit 56f9d82bc567660a6fdd9d4b9fc67c3159faad80)
Epsilon PR: ideadevice/epsilon#411 (cherry picked from commit 50fc730293f79a372aa86394896a78d934eb8a02)
Description: 1. Tests creating bp payload through json dumps were failing due to additional policy_reference key RCA: policy_reference key missing in known_json. Normal compile logic remove additional policy_reference key from bp payload. But, when we use json dumps then policy_reference was not removed FIX: Moved logic of policy reference removal to post_compile method of ConfigSpecType 2. Observed ui name being used to target deployment in decompiled bp for snapshot/restore/patchconfig. Fixed by mapping it to correct dsl class name. (cherry picked from commit 5c901f3ebeac592cf7ef46d9d49e6e3bbde48cb7)
Description: RCA: Power actions tasks that were at substrate level (post create) and package level (install, uninstall) were not decompiled due to no mapping of it to it's target substrate. Fix: Included tasks at these level while decompiling too Decompiled successfully after inclusion of power actions at all levels. (service/package/substrate/profile/patchconfig) <img width="1211" alt="image" src="https://github.com/ideadevice/calm-dsl/assets/123161845/029bf7c6-3a04-4515-b8c2-a0e1c8ebe94c"> (cherry picked from commit 84fff84c13cdf31f56c28339e19f970ead2175d8)
…ing version cache (#404) Issue Link: https://jira.nutanix.com/browse/CALM-43760 Description: RCA: Builds were failing due to version check on PC present on previous context. This is specific case while initialising DSL. Fix: Modified logic so that version cache is not synced when init command is invoked. Hence re-initialising version table is not required.  Succesfull initialisation:  Triggered regression: http://erdinger.eng.nutanix.com/blue/organizations/jenkins/QA%2FDSL_Regression/detail/DSL_Regression/584/pipeline RP Link: http://rp.calm.nutanix.com/ui/#calm/launches/64e490d4b3cb1f0001581be7%7Cpage.page=1&page.size=50&filter.cnt.name=DSL&page.sort=start_time,number%2CDESC/65fcbd9a2ab79c00014bc13d?page.page=1&page.size=50&page.sort=start_time%2CASC --------- Co-authored-by: Abhijeet Kaurav <[email protected]> (cherry picked from commit 747d3b0abdeb82e52ff0c042b7399e4f758620ce)
Issue Link: https://jira.nutanix.com/browse/CALM-43867 RCA: Update config of app is not having following in case of partial configs: - nic in pre_defined_nic_list - disks in pre_defined_disk_list - categories in pre_defined_categories Fix: 1. Modified compilation logic in patch config model to support partial configs 2. Modified decompilation logic We were encountering failure in app update when initiated multiple times. After this fix app update works fine: <img width="787" alt="image" src="https://github.com/ideadevice/calm-dsl/assets/123161845/8e339d04-8fb5-4abe-ad5d-239f18fda7b5"> (cherry picked from commit dc5ba19199be063099f39ffb1737cdc2b2f3b127)
This reverts commit db1fcc9.
This reverts commit 6741915.
Adding minor fix to resolve nonetype error if patch list is absent in blueprint payload (cherry picked from commit ad45abdd84087b1ec27b226027b0b95d51823f06)
Issue LInk: https://jira.nutanix.com/browse/CALM-44168 RCA: Decompile was failing due to max_value variable being used before assignment. This happens in case of decrease operation as we don't set any max_value. Therefore, it is not in present in blueprint payload. Fix: Assign empty value to max_value while declaring (this will prevent UnboundLocalError) and re-assign it to value present in blueprint payload. While rendering if empty value is present we won't render it in jinja template. (cherry picked from commit b55847df75781aabe191f13d455c567e6d8aa0e2)
Fixing pre_decompile part of config_spec model for better handling in case of undesirable state. (cherry picked from commit 9a855a00781636ddf42d0c6e764c6151a52cb235)
Description: 1. Multiple subnets were not rendering properly in provider list of `Environment` class. Fixed `calm/dsl/decompile/schemas/provider_ntnx.py.jinja2`. 2. After decompilation `os` module was used in environment but it was not included as import. Fixed it. 3. Gui substrate name was used in substrate list of decompiled `Environment` class instead of substrate class name. FIxed logic to use valid dsl class name for substrates. Remark: support for decompilation of azure, k8s is not present Test: 1. Ran decompile and then recompile over all environments in all projects of PC. (cherry picked from commit 0168f60401f5a3958de4ee0dfa2708c2fff0340a)
…mits Task/3722 revert acp related commits
…s in tasks (#419) changes target to target_endpoint when decompiling a blueprint containing action having task with endpoint target. decompiled following blueprint having a task with endpoint and a task with no endpoint  output of decompile: ``` """ Generated blueprint DSL (.py) """ import json # no_qa import os # no_qa from calm.dsl.builtins import * # no_qa from calm.dsl.runbooks import CalmEndpoint as Endpoint BP_CRED_cred1_PASSWORD = read_local_file("BP_CRED_cred1_PASSWORD") BP_CRED_cred1 = basic_cred( "root", BP_CRED_cred1_PASSWORD, name="cred1", type="PASSWORD", default=True, ) test_ep_1 = Endpoint.use_existing("test_ep_1") class Service1(Service): pass class vmcalm_timeResources(AhvVmResources): memory = 1 vCPUs = 1 cores_per_vCPU = 1 disks = [ AhvVmDisk.Disk.Scsi.cloneFromImageService("CentOS-7-cloudinit", bootable=True) ] nics = [ AhvVmNic.NormalNic.ingress("vlan1211", cluster="auto_cluster_prod_1a66414f258f") ] power_state = "ON" boot_type = "LEGACY" class vmcalm_time(AhvVm): name = "vm-@@{calm_time}@@" resources = vmcalm_timeResources cluster = Ref.Cluster(name="auto_cluster_prod_1a66414f258f") class VM1(Substrate): account = Ref.Account("NTNX_LOCAL_AZ") os_type = "Linux" provider_type = "AHV_VM" provider_spec = vmcalm_time readiness_probe = readiness_probe( connection_type="SSH", disabled=True, retries="5", connection_port=22, address="@@{platform.status.resources.nic_list[0].ip_endpoint_list[0].ip}@@", delay_secs="60", ) class Package1(Package): services = [ref(Service1)] class deployment_00d980ca(Deployment): min_replicas = "1" max_replicas = "1" default_replicas = "1" packages = [ref(Package1)] substrate = ref(VM1) class Default(Profile): environments = [Ref.Environment(name="test_env6f6123c7c9aa")] deployments = [deployment_00d980ca] @action def UntitledAction1(name="Untitled Action1"): CalmTask.Exec.ssh( name="Task1", filename=os.path.join( "scripts", "Profile_Default_Action_UntitledAction1_Task_Task1.sh" ), target=ref(Service1), ) CalmTask.Exec.ssh( name="Task2", filename=os.path.join( "scripts", "Profile_Default_Action_UntitledAction1_Task_Task2.sh" ), target_endpoint=ref(test_ep_1), ) class test_bp_endoint_decompile(Blueprint): services = [Service1] packages = [Package1] substrates = [VM1] profiles = [Default] credentials = [BP_CRED_cred1] class BpMetadata(Metadata): categories = {"TemplateType": "Vm"} project = Ref.Project("default") ``` Verified that calm create bp works for the above decompiled bp: ``` calm create bp -f ./test_bp_endoint_decompile/blueprint.py --name test_bp_endpoint_create --force [2024-04-17 09:03:42] [INFO] [calm.dsl.cli.bp_commands:251] Blueprint test_bp_endpoint_create created successfully. { "name": "test_bp_endpoint_create", "link": "https://10.44.76.124:9440/console/#page/explore/calm/blueprints/2b72675d-29a6-49fb-8d64-fbc8ed40e559", "state": "ACTIVE" } ```  --------- Co-authored-by: Prabhat Dwivedi <[email protected]> (cherry picked from commit 4d4c05c01bbcc4dbe238b552c1a9bc36df8345d6)
Issue Link: https://jira.nutanix.com/browse/CALM-44462 Fixed some python syntactical errors (cherry picked from commit 6926f263dbd67a9f107deedc755b63916c03da4d)
Will add these removed files back in next commit. So we can directly cherrypick next commit for 3.8.0 release. (cherry picked from commit ebb6a6a9d6dcdddf5d88bd280fd49df40fff491b)
…nto dsl-release-3722-beta
Description: Release note for DSL v3.7.2.2 (cherry picked from commit 53c8133d544c941801177ee685322c00c53361e9)
(cherry picked from commit 8a5ffdc85abd2c87d1d7aba2242fda850ec8fac2)
(cherry picked from commit 87273576dfdc03e35e1c365d5bbe3924fa64716e)
abhijeetkaurav1st
approved these changes
Apr 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.