Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add/keypair #8

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Add/keypair #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
95222fd
Isolate keypar in a new node. Preparation
victorsndvg Nov 22, 2018
b8c4fc8
restory sinchronous functioning on server node start operation
victorsndvg Nov 22, 2018
d025518
Autogenerate keypair with paramiko if not passed
victorsndvg Nov 22, 2018
fe49844
Fix public key (auto)generation
victorsndvg Nov 22, 2018
b6d006c
user/pass are optional
victorsndvg Nov 22, 2018
fba0011
update readme
victorsndvg Nov 22, 2018
feb1e3e
Fix default password as suggested in the code review
victorsndvg Nov 23, 2018
7edc94a
Fix im.yaml to be ready to merge
victorsndvg Nov 23, 2018
31dc315
update plugin acording to the current plugin status
victorsndvg Nov 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 31 additions & 13 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# cloudify-im-extension
Cloudify extension to work with Infrastructure Manager (Skeleton)

Note: This plugin has been tested against OCCI endpoints only. Minor modifications are required to support other cloud providers.

## Hello world

* Copy the repository contents into _[BLUEPRINT_ROOT]/cloudify-im-extension/_
Expand All @@ -13,11 +15,6 @@ imports:
- http://raw.githubusercontent.com/mso4sc/cloudify-hpc-plugin/master/resources/types/cfy_types.yaml
- http://raw.githubusercontent.com/MSO4SC/cloudify-im-extension/master/im.yaml

inputs:
blah:
description: Dummy input
default: "public"
type: string

node_templates:
vitual_machine:
Expand All @@ -27,8 +24,8 @@ node_templates:
id: im
host: 'http://im.srv.cesga.es:8800'
type: InfrastructureManager
user: ...
pass: ...
user: ... # IM user
pass: ... # IM password
endpoint:
id: occi
type: OCCI
Expand All @@ -41,6 +38,8 @@ node_templates:
target: network
- type: depends_on_setting
target: image
- type: depends_on_setting
target: keypair
- type: depends_on_setting
target: flavour

Expand All @@ -58,14 +57,24 @@ node_templates:
image:
type: im.nodes.Image
properties:
name: Centos-/
name: Centos-7
config:
id: 'https://fedcloud-services.egi.cesga.es:11443/51'
storage: 1024M
username: ...
password: ...
public_key: 'ssh-rsa ...'
private_key: |
username: ... # Optional
emepetres marked this conversation as resolved.
Show resolved Hide resolved
password: ... # Optional
use_external_resource: true
resource_id: default_image
simulate: False

keypair:
type: im.nodes.Keypair
properties:
name: Credentials
config:
id: 'user@laptop'
public_key: 'ssh-rsa ...' # Optional
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of optional, I understand that both keys doesn't have to be provided if use_existing_resource: False, and are mandatory if use_existing_resource: True.

Is that right? If so, please modify the docs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use_existing_resource is inherited from the template you provided at the begining. I was reading about it and I think is an OpenStack-plugin (and othe plugins) feature, but nothing I had implemented. I think it is not working and we can remove this property all along the plugin.

The plugin generates the keypair if public_key or private_key is not defined. As I explained before, use_existing_resource is not taked into account.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then remove use_existing_resource, because it seems that it is implemented.

Also, the generation of the keys if they are not defined should be reflected somehow in the README.

simulate property is implemented?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, simulate is implemented. If it's True it does nothing during the operation

private_key: | # Optional
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
Expand All @@ -88,12 +97,21 @@ node_templates:
software:
type: im.nodes.Software
properties:
name: singularity
name: singularity-openmpi
config:
packages: ["openmpi", "singularity"]
deploy: |
ln -s /usr/lib64/openmpi/bin/mpirun /usr/bin/mpirun
use_external_resource: true
resource_id: default_software
simulate: False

````

## Infrastructure Manager

This plugin connects to an [Infrastructure Manager](http://www.grycap.upv.es/im/index.php) instance to manage the deployment of virtual machines.

# Template

Previous template maps the *Resource and Application Description Language* (RADL) specification to cloudify node types. Basic RADL structure can be found in the official [RADL docs](https://imdocs.readthedocs.io/en/latest/radl.html#basic-structure). In this docs you can see how to reference OCCI endpoints and images.
22 changes: 16 additions & 6 deletions blueprint.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,14 +49,24 @@ node_templates:
image:
type: im.nodes.Image
properties:
name: Centos-/
name: Centos-7
config:
id: 'https://fedcloud-services.egi.cesga.es:11443/51'
storage: 1024M
username: ...
password: ...
public_key: 'ssh-rsa ...'
private_key: |
username: ... # Optional
password: ... # Optional
use_external_resource: true
resource_id: default_image
simulate: False

keypair:
type: im.nodes.Keypair
properties:
name: Credentials
config:
id: 'user@laptop'
public_key: 'ssh-rsa ...' # Optional
private_key: | # Optional
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
Expand All @@ -79,7 +89,7 @@ node_templates:
software:
type: im.nodes.Software
properties:
name: singularity
name: singularity-openmpi
config:
packages: ["openmpi", "singularity"]
deploy: |
Expand Down
32 changes: 32 additions & 0 deletions im.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,38 @@ node_types:
simulate:
default: { get_property: [SELF, simulate] }


im.nodes.Keypair:
derived_from: cloudify.nodes.Compute
properties:
name:
description: Name or ID of the resource
config:
description: Image config. See blueprint.yaml
resource_id:
description: id of the resource
use_external_resource:
description: true to use an already created server
type: boolean
default: false
simulate:
description: Set to true to simulate job without sending it
type: boolean
default: False
agent_config: # This is mandatory, to say cloudify that we are not installing an agent
default:
install_method: none
interfaces:
cloudify.interfaces.lifecycle:
configure:
implementation: im.plugin.nodes.keypair.configure
inputs:
config:
default: { get_property: [SELF, config] }
simulate:
default: { get_property: [SELF, simulate] }


im.nodes.Flavour:
derived_from: cloudify.nodes.Compute
properties:
Expand Down
12 changes: 4 additions & 8 deletions plugin/nodes/image.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,14 @@ def build_radl_image(config):

id = get_child(dictionary=config, key='id', required=True)
storage = get_child(dictionary=config, key='storage')
username = get_child(dictionary=config, key='username', required=True)
password = get_child(dictionary=config, key='password', required=True)
public_key = get_child(dictionary=config, key='public_key', required=True)
private_key = get_child(dictionary=config, key='private_key', required=True)
username = get_child(dictionary=config, key='username') or 'user'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said, this should always be defined in the dictionary.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think to provide a default user could help to simplify the plugin user experience by means of the blueprints

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is very unlikely that, if you left the username empty, the user in the image will be 'user'. So leaving it blank would lead to an error.

The idea is that, when you define an image, you need to know the default user that the image defines.

Therefore it should be mandatory (and reflect that on the README), unless you say that the plugin is creating the user when deploying the virtual machine. If it is the second case, that behaviour should be explained in the README as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the second assumption is the right case. The user is created during contextualization

password = get_child(dictionary=config, key='password') or ''

image_radl = ''
image_radl += " disk.0.image.url = '" + str(id) + "' and \n"
if storage: image_radl += " disk.0.image.size = '" + str(storage) + "' and \n"
image_radl += " disk.0.os.credentials.username = '" + str(username) + "' and \n"
image_radl += " disk.0.os.credentials.password = '" + str(password) + "' and \n"
image_radl += " disk.0.os.credentials.public_key = '" + str(public_key.strip()) + "' and \n"
image_radl += " disk.0.os.credentials.private_key = '" + str(private_key.strip()) + "' "
if storage: image_radl += " disk.0.image.size = '" + str(storage) + "' and \n"
if password: image_radl += " disk.0.os.credentials.password = '" + str(password) + "' and \n"

decrease_log_indentation()
return image_radl
Expand Down
47 changes: 47 additions & 0 deletions plugin/nodes/keypair.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
from cloudify import ctx
from cloudify.state import ctx_parameters as inputs
from cloudify.decorators import operation
from cloudify.exceptions import *
from plugin.nodes.utils import *
import paramiko
import StringIO

def build_radl_keypair(config):
ctx.logger.debug('{0} Infrastructure Manager deployment info:'.format(get_log_indentation()))
increase_log_indentation()

public_key = get_child(dictionary=config, key='public_key') or None
private_key = get_child(dictionary=config, key='private_key') or None

if not (public_key and private_key):
increase_log_indentation()
ctx.logger.debug('{0} Creating RSA keypair ...'.format(get_log_indentation()))
private_key_stream = StringIO.StringIO()
key = paramiko.RSAKey.generate(2048)
public_key = '{0} {1}'.format(key.get_name().strip(), key.get_base64().strip())
key.write_private_key(private_key_stream)
private_key = private_key_stream.getvalue().strip()
decrease_log_indentation()

keypair_radl = ''
keypair_radl += " disk.0.os.credentials.public_key = '" + str(public_key.strip()) + "' and \n"
keypair_radl += " disk.0.os.credentials.private_key = '" + str(private_key.strip()) + "' "

decrease_log_indentation()
return keypair_radl

@operation
def configure(config, simulate, **kwargs):
if (not simulate):
reset_log_indentation()
ctx.logger.debug('{0} Configure operation: Begin'.format(get_log_indentation()))
increase_log_indentation()
radl = get_child(ctx.instance.runtime_properties, key='settings')
if not radl:
radl = create_child(ctx.instance.runtime_properties, key='settings', value={})
radl_image = create_child(radl, key='keypair', value=build_radl_keypair(config))
decrease_log_indentation()
ctx.logger.debug('{0} Configure operation: End'.format(get_log_indentation()))



8 changes: 5 additions & 3 deletions plugin/nodes/server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,10 @@ def build_radl():


settings = get_child(ctx.instance.runtime_properties, key='settings', required=True)
network = get_child(settings, key='network') or ''
image = get_child(settings, key='image', required=True)
flavour = get_child(settings, key='flavour') or ''
network = get_child(settings, key='network') or ''
image = get_child(settings, key='image', required=True)
keypair = get_child(settings, key='keypair', required=True)
flavour = get_child(settings, key='flavour') or ''
software = get_child(settings, key='software') or ''

radl = \
Expand All @@ -53,6 +54,7 @@ def build_radl():
" net_interface.0.connection = 'net' and \n" + \
flavour + \
image + \
keypair + \
"""
)

Expand Down