Support for Automatic Task Execution from JSON Files

[Calvvnono]

Currently, UFO operates through command-line interaction, where users input commands one by one for the agent to execute tasks.

I would like to request support for enabling the agent to execute a predefined sequence of tasks from an, let's say JSON file. By defining tasks and their parameters in such a file, users could load it and allow the agent to execute the workflow automatically. This design would facilitate large-scale task automation for testing purposes, enabling comprehensive evaluation of UFO's performance across different types of tasks.

Does this feature align with the project’s scope? If so, I’d appreciate any guidance or suggestions on how best to implement it within the current architecture. Thank you for your time and support!

[vyokky]

Hi @Calvvnono , do you mean executing a sequence of actions, like click 1, click 2 ..., and the agent just do it without any intellgence?

[Calvvnono]

Yes but, It's more like a sequence of complete tasks, like task1, task2, the agent loads the task list and then executes all the tasks one by one.

We focus on whether each task is executed successfully or not. There is no sequential or logical dependency between the tasks, so after completing each task, it may be necessary to reset certain states (perhaps?).

[vyokky]

Yes, that is in our backlog we will release a version shortly. Now UFO supports to run task in follower mode in batch. Please check Step 3 in this doc:
https://microsoft.github.io/UFO/agents/follower_agent/
And if you want to reset the env (app state), you will need to implement by yourself.