Change automate methods to communicate via REST API

[Fryguy]

Much of this was discussed at the ManageIQ design summit for Botvinnik. However, @mkanoor and I recently sat down and brainstormed our way through many of the larger problems, and we think we've come to a cleaner design.

There may be more things I'm not thinking of, but I just wanted to start getting this information out there. Let's keep this document updated as we find more issues and/or come up with more ways to solve them.

---

Automate currently runs methods by

• starting a DRb server
• spawning the ruby method wrapped in a preamble and postamble, with the preamble starting a DRb client
• client uses service models and helper classes like $evm to communicate with the server

This causes a number of problems and headaches, which may be able to be addressed by moving to a REST API based implementation. Some notable issues:

• DRb requires server side object management so that the server doesn't garbage collect objects in use by the client. This is a huge headache which will just go away with REST.

• Every time we add a new object or method we must manually expose it via automate. This is prone to error. This would instead become an exercise in exposing via the REST API, which in nearly all cases should be done anyway.

• DRb is used to handle interactions with the automate workspace, which is in the server's memory, from the client. This can be handled by dumping the workspace into some format like JSON or YAML, and passing it on STDIN into the launched process.

  The automate method preamble would take STDIN and hold it in memory, lazy loading it on first access. If the method needs to modify the workspace, then it can be modified. $evm would no longer be special and can just be a simple object defined in the preamble that has the knowledge of manipulating the workspace in this manner. $evm.log could even just write directly to the log in question or could write to STDOUT.

  In the postamble we will dump the workspace to STDOUT. In order to prevent collision with user usage of STDOUT, we can prefix the dump with some sort of delimiter (e.g. "=" * 80). STDERR can be used to transfer any exception, or perhaps that becomes part of the workspace that's dumped into STDOUT. Again, a delimiter might be necessary.

  The server, when it detects that the child process has completed, will read from the STDOUT of the child process, find the last delimiter, and process the workspace back into memory, if changed.

  There may be optimizations here to prevent unnecessarily dumping and loading the workspace.

• DRb is also used for database interaction. The database interaction that is currently done via service models, can be done completely through the REST API. A Ruby gem needs to be created from the cfme_client code, which additionally has more of an OO design. (I'll call it manageiq_client for reference). Once that's available, then the $evm.vmdb can just be a front for accessing those objects from the manageiq_client gem. We could even deprecate $evm.vmdb in favor of having the automate methods use the normal manageiq_client gem directly.

  For example, currently $evm.vmdb["vm", 21] returns a MiqAeServiceVmVmware instance for VM 21. If the REST API gem had it's own OO based types, I could see $evm.vmdb["vm", 21] instead calling the manageiq_client gem, ultimately returning a ManageiqClient::Vm instance for VM 21

  With this in place, I think the entirety of the service models just goes away

• DRb is also used for built-in methods (like send_email). This can be handled by exposing them from the REST API, and passing the dumped workspace from the method as a parameter to the REST API. These may need some sort of system-only access rules, but then again, maybe not.

Other thoughts:

• With database interaction via the REST API, and the workspace passed via STDIN/STDOUT in a common format like JSON or YAML, then the support for other languages opens up. These formats are widely supported, so they could be passed to any kind of method whether it be Ruby, Python, Shell, etc. All that would be needed from the server is possibly some preamble/postamble code in the native language to simplify setting up helper objects and catching errors.

---

TODO

• Other parts of $evm object that we haven't even looked at like get/set_state_var, instance_*, etc
• What user would automate methods run as if they are accessing the REST API. For the system things like the built-in methods, it's some system account, but what about other methods? Can they just be system too?

cc @gmcculloug @mkanoor @tinaafitz @chessbyte
cc @abellotti (re: REST API stuff and cfme_client split out into manageiq_client gem)
cc @kbrock (re: system-only access rules)

[kbrock]

• I like REST for database access.
• I like not having automate code in vmdb rails server.
• I like a non-REST mechanism for the workspace.
  • Spawning a child process and passing JSON via STDIN/STDOUT is good. It could probably take a queue from the CGI interface.
  • Only passing deltas on STDOUT would cut down data transfer.
  • there are other wire protocols (nanomsg/zeromq/rabbitmq/cupid) that are very performant and cross platform, but going with CGI/STDOUT is simplest and requires no libraries and is very easy to debug.

[mkanoor]

One thing that we have to be careful is the $evm.instantiate that requires that an instance and tree of objects be added to the current workspace. This also provides for calling one automate method from another.

[kbrock]

So there are 3 types of data/interactions:

data	example	access
vmdb	VM, template, host, hard drives	REST
workspace		STDIN/STDOUT
rpc	$evm.instantiate	leave asis

Can we just focus on using REST for the first one?

[kbrock]

we could provide the automate process a $workspace variable ($stdout) and remap $stdout/STDOUT to a log file.

[mkanoor]

$evm.execute could also be a problem.
e.g.
$evm.execute('tag_exists?', category_name, tag_name)
$evm.execute('tag_create', category_name, :name => tag_name, :description => "#{tag}")

[miq-bot]

This issue has been automatically marked as stale because it has not been updated for at least 6 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.

Thank you for all your contributions!

[Fryguy]

@miq-bot move-issue manageiq-automation_engine

[miq-bot]

This issue has been moved to ManageIQ/manageiq-automation_engine#44