Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initial demo of new runner-adapter logic with fs adapter #74

Closed
wants to merge 51 commits into from

Conversation

amv
Copy link
Member

@amv amv commented Jul 5, 2014

THIS IS NOT READY TO MERGE.

It is meant to showcase the proposal for new runner-adapter logic and the new ejector logic - and acts as a place to discuss and make them better. The code is not tested through, but should give a good idea of the execution path.

The idea with the new logic would be the following:

  1. Runner's listenTask call provides adapter a callback( error, task, domain, db_state ) -parameter.
  2. Adapter fetches tasks, locks them for processing and calls callback(null, task_object, db_domain, { lock_or_transaction }) for each task individually.
  3. Runner guarantees that if it does not receive the first error parameter, it will always do it's best to end the callback execution to a call to one of the following adapter functions: disableListenedTask( task, db_state, callback ) OR updateListenedTask( task, db_state, callback ).
  4. Each of these adapter functions use the callback to signal that "disable" or "update" has been processed, and passes an Error as the first parameter if it did not.
  5. If runner receives a callback call without Error as the first parameter, it submits the task to the controller job.

A succesfull execution path could look like this:

Runner.startPolling()
  Runner --> Adapter.listenTask( function task_callback( error, task, domain, db_state ) {} )
    Adapter.start_poll_loop();

Adapter poll loop found something:
  LOCK = Adapter.get_lock( { TASK } )
  Adapter --> Runner.task_callback( null, { TASK }, 'default_db', { lock : LOCK }
    Runner.add_retry_timeout_to_task( { TASK } )
    Runner --> Adapter.updateListenedTask( { TASK }, { lock : LOCK }, function submit_task_callback( error ) {} )

Adapter database update succeeds:
  Adapter.release_lock( LOCK )
  Adapter --> Runner.submit_task_callback( null )
    Runner: TASK.eject_function = 'gearsloth_eject-' + domain
    Runner --> Client.submitControllerJobBg( { TASK } )

Controller gets and executes tasks:
  Controller.eject_task( TASK )
    Client.submitJob( TASK.eject_function, { TASK } )

@amv
Copy link
Member Author

amv commented Jul 14, 2014

Changed the interface of the adapter a bit to keep the eject task name generation out of the actual adapter implementations.

Adapters got a new getDomains( callback( error, db_name_list ) ) -call, which should resolve names of the of the separate databases this current adapter is connected to.

The database name of each task is also passed to the runner as part of the listenTask callback (as the parameter "domain") so that the eject_function name can be properly added to the task before it is passed on to the controller. This way runner does not need to parse the name from the id (which is good as id implementation can be highly adapter dependent).

None of the adapter specific tasks work yet but unit and e2e test should be updated to match the new spec. I'll go and update the original pull request description if I can to match the small changes made.

@amv
Copy link
Member Author

amv commented Jul 19, 2014

The current version does some initial work for #68 and implements #65.

DB initialization was also moved inside the constructor which should ease development on #66 and #25.

Make test runs now with a limited test coverage BUT if you run the unit tests before the integration tests (and not the other way around like we do now) inside the same mocha runtime, stuff starts to break... So there is still a lot to work with in the current tests.

Changes include:
 * whitespace changes per Googles style guide
 * breaking some long lines
 * explicit boolean conversions
 * jsdoc fixes
 * syntax fix, which prevented running the program
Directory for fs-adapter and dependencies for precise installation
From documentation: "fs.exists() is an anachronism and exists only
for historical reasons. There should almost never be a reason to
use it in your own code."
@amv
Copy link
Member Author

amv commented May 13, 2016

This pull request has gone stale. The sqlite branch has been built on top of this branch, and there should be a pull request to merge that instead.

@amv amv closed this May 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants