make evaluator script executable with command line arguments PDDL/SAS files for testing

[ClemensBuechner]

No description provided.

[FlorianPommerening]

We discussed this and would like evaluators to take the path to pickled state as input for the normal operation of the search rather than importing a function from the evaluator script and calling it. This will make it possible for the search to detect cases where the evaluator runs out of resources and will unify the grid and local environments more. The original goal of the issue was to also allow PDDL/SAS files as input for testing. We will handle this as optional inputs in convenience functions provided by the modules.

We discussed a few options on Discord and ended up with the following design. An evaluator without convenience functions, could look like this:

#!/usr/bin/env python3

from machetli import pddl, tools, EXIT_CODE_IMPROVING, EXIT_CODE_NONIMPROVING
import sys

def load_state(argv):
    try:
        return machetli.unpickle(argv[1])
    except:
        return pddl.create_state(argv[1], argv[2])

def evaluate(state):
    with pddl.temporary_files(state) as (domain_filename, problem_filename):
        return evaluate(domain_filename, problem_filename)

def evaluate(domain_filename, problem_filename):
    command = ["./bugged-planner/plan", f"{domain_filename}", f"{problem_filename}"]
    run = tools.Run(command, time_limit=20, memory_limit=3000)
    stdout, stderr, returncode = run.start()
    return "Wrong task encoding" in stdout

if __name__ == "__main__":
    state = load_state(sys.argv)
    if evaluate(state):
        return EXIT_CODE_IMPROVING
    else:
        return EXIT_CODE_NONIMPROVING

With convenience functions, we could boil it down to this:

from machetli import pddl, tools

def evaluate(domain_filename, problem_filename):
    command = ["./bugged-planner/plan", f"{domain_filename}", f"{problem_filename}"]
    run = tools.Run(command, time_limit=20, memory_limit=3000)
    stdout, stderr, returncode = run.start()
    return "Wrong task encoding" in stdout

if __name__ == "__main__":
    pddl.run_evaluator(evaluate)

[FlorianPommerening]

As discussed in #90, we want to work on #53 first. We might do this issue together with #53 if that makes things easier.

[FlorianPommerening]

Pull request #94 deals with this. There are still a couple of TODOs and we have to update the documentation.