-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
305 lines (282 loc) · 18.2 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<!-- Meta tags for social media banners, these should be filled in appropriatly as they are your "business card" -->
<!-- Replace the content tag with appropriate information -->
<meta name="description" content="GAME">
<meta property="og:title" content="GAME"/>
<meta property="og:description" content="GAME: Creating an Infinitely Interactive Roblox World with Autonomous Agents"/>
<meta property="og:url" content="URL OF THE WEBSITE"/>
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X630-->
<meta property="og:image" content="static/image/your_banner_image.png" />
<meta property="og:image:width" content="1200"/>
<meta property="og:image:height" content="630"/>
<meta name="twitter:title" content="GAME">
<meta name="twitter:description" content="GAME">
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X600-->
<meta name="twitter:image" content="static/images/your_twitter_banner_image.png">
<meta name="twitter:card" content="summary_large_image">
<!-- Keywords for your paper to be indexed by-->
<meta name="keywords" content="KEYWORDS SHOULD BE PLACED HERE">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>GAME</title>
<link rel="icon" type="image/x-icon" href="static/images/favicon.ico">
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
rel="stylesheet">
<link rel="stylesheet" href="static/css/bulma.min.css">
<link rel="stylesheet" href="static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="static/css/bulma-slider.min.css">
<link rel="stylesheet" href="static/css/fontawesome.all.min.css">
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="static/css/index.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script src="https://documentcloud.adobe.com/view-sdk/main.js"></script>
<script defer src="static/js/fontawesome.all.min.js"></script>
<script src="static/js/bulma-carousel.min.js"></script>
<script src="static/js/bulma-slider.min.js"></script>
<script src="static/js/index.js"></script>
</head>
<body>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">GAME: Creating an Infinitely Interactive Roblox World with Autonomous Agents</h1>
<div class="is-size-5 publication-authors">
<!-- Paper authors -->
<span class="author-block">
<a href="https://virtuals.io/" target="_blank">Virtuals Protocol</a><sup></sup></span>
<!--<span class="author-block">
<a href="SECOND AUTHOR PERSONAL LINK" target="_blank">Second Author</a><sup>*</sup>,</span>
<span class="author-block">
<a href="THIRD AUTHOR PERSONAL LINK" target="_blank">Third Author</a> -->
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- Teaser gif-->
<section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body has-text-centered">
<h2 class="subtitle has-text-centered">
We’ve created a platform and engine called GAME (Generative Autonomous Multimodal Entities), that powers AI agents to act and interact in virtual worlds and environments. We’ve used the GAME framework to power a multi-agent interactive simulation in Roblox called Project Westworld, where we are able to observe autonomous behaviour leading to emergent storylines.
</h2>
<img src="static/images/main-diagram.webp" alt="emergent gameplay" />
</div>
</div>
</section>
<!-- End teaser gif -->
<!-- Paper abstract -->
<!--section class="section hero is-light">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
This work reports the study of video generative models for interactive video game generation and simulation. We discuss and explore the use of available pre-trained open-sourced video generation models to create playable interactive video games. While being able to generate short clips of broad range of described scenes, such models still lack controllability and continuity. Given these limitations, we focus on producing and demonstrating a reliable and controllable video game generator on a single game domain. We present MarioVGG, a text-to-video diffusion model for controllable video generation on the Super Mario Bros game. MarioVGG demonstrates the ability to continously generate consistent and meaningful scenes and levels, as well as simulate the physics and movements of a controllable player all through video.
</p>
</div>
</div>
</div>
</div>
</section>
<!-- End paper abstract -->
<section class="section hero is-small">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-justified">
<div class="column is-four-fifths">
<h2 class="title is-3">Overview</h2>
<div class="content has-text-justified">
<p>
GAME is a modular framework built for agentic systems that is environment and game agnostic. The GAME framework builds on work done in generative agents and agentic systems [1, 3, 8] which can be composed of multiple building blocks and techniques such as prompting, planning and reasoning [12, 5, 6], search [7], self-reflection and self-refinement [2, 4], tool-use [10, 11] and memory [1, 9].
</p>
<p>
Games, virtual environments and worlds are designed to be immersive. Through the use of roleplay [13] in foundation models, interaction with and between GAME-powered generative agents result in novel, authentic, and completely unscripted conversation. GAME-powered agents however aren’t just static conversational chatbots or NPCs, they have the full ability to function and act in the environment as any other player or character. With chain-of-thought based prompting methods for planning and reasoning along with tool-use, our agents can plan, act and achieve goals which influence and affect the world. All of these methods are also brought together through the memory modules in GAME, which ensure that agents recall and consider past events, observations and conversations which all influence an agent's thoughts and actions.
</p>
</div>
<div class=content has-text-centred>
<img src="static/images/game-framework.png" alt="GAME Overview" class="center" />
<p>
<em>The GAME framework for agentic systems in virtual environments.</em>
</p>
</div>
<br><br>
</div>
</div>
</div>
</div>
</section>
<section class="section hero is-small">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-justified">
<div class="column is-four-fifths">
<h2 class="title is-3">Capabilities</h2>
<div class="content has-text-justified">
<p>
Today, we are releasing Project Westworld, an interactive simulation in Roblox that contains autonomous agents powered by our GAME framework. It is inspired by Westworld, a town inhabited by AI androids. In the Wild West, the player is dropped into the world where a hidden villain, The Bandit, lurks with the aim of seeking power and sowing chaos. The player’s goal is to identify The Bandit and influence the other agents to capture them.
</p>
</div>
<div class=content has-text-centred>
<img src="static/images/capabilities.png" alt="Capabilities" />
<p>
<em>Bird's eye view of the Westworld map in Roblox.</em>
</p>
</div>
<br><br>
<h3 class="title is-4">Agent Autonomy</h3>
<div class="content has-text-justified">
<p>
There are 10 agents inhabiting Westworld - each one with their own unique personalities, desires and goals.
</p>
</div>
<h4 class="title is-5">Personality</h4>
<div class=content has-text-centred>
<img src="static/images/personality.png" alt="Personality" />
<p>
<em>Example of an agent's character profile.</em>
</p>
</div>
<div class="content has-text-justified">
<p>
Take Ignacio Morales as an example - he is a raider who has faced betrayal in his past and now channels his hurt into seeking control, power, and wealth through ruthless and chaotic means. This backstory along with Ignacio’s goal of causing maximum chaos and robbing the rich has been added into his “personality” in the form of a character card.
</p>
</div>
<h4 class="title is-5">Autonomy</h4>
<div class="content has-text-justified">
<p>
Ignacio’s personality, along with several other variables such as the game state, his previous actions, and long term memory, is fed into the high level task planner. This generates a high-level task for him, which is then fed into the action planner and executor which is used to create a seqeunce of granular actions that are grounded and executable in the environment for Ignacio to take in the game. He also receives in-game feedback in the form of observations, rewards, successes and more which is used to inform and update his plans and following actions.
</p>
</div>
<div class=content has-text-centred>
<img src="static/images/autonomy.png" alt="Autonomy" />
<p>
<em>Illustration of how an agent creates plans and tasks based on input and feedback.</em>
</p>
</div>
<div class="content has-text-justified">
<p>
Having an autonomous character that has goals and strategises how to achieve these in an agentic manner greatly enhances NPCs in games today. Where a standard bot only has fixed paths and options, an autonomous agent is able to take input from its environment, other agents and surrounding player’s to independently develop plans and its actions in the world. This creates more immersive experiences as the agent becomes more unpredictable and humanlike.
</p>
</div>
<h4 class="title is-5">Environment Interaction</h4>
<div class=content has-text-centred>
<img src="static/images/environment.webp" alt="Environment" />
<p>
<em>Examples of interactions between an agent and its environment.</em>
</p>
</div>
<div class="content has-text-justified">
<p>
Agents in the game are also able to interact with the game environment - they can store/remove items in their inventory, they can use the items (e.g. throw a knife) and interact with objects in the environment (e.g. unlock a safe). Developers can utilise this ability to add more items in the action space which increases the storyline diversity.
</p>
</div>
</div>
<br><br>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="section hero is-small">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-justified">
<div class="column is-four-fifths">
<h2 class="title is-3">Infinite Replayability</h2>
<div class="content has-text-justified">
<p>
We find that introducing autonomous agents as NPCs result in more diverse storylines and outcomes in the same game compared to using standard NPCs. The more agents we introduce in the world, the more complex the interactions become and unexpected situations occur, with low repetition across multiple playthroughs. This results in the game essentially being an infinitely replayable world.
</p>
</div>
<div class="publication-video">
<!-- Youtube embed code here -->
<iframe src="https://www.youtube.com/embed/FUTG_xkmKlQ?si=TtIXwhGaVM904TgC" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
<p>
Autonomous agents create diverse storylines in a world with every playthrough.
</p>
</div>
<div class="content has-text-justified">
<br><br>
<p>
The Westworld playable demo in Roblox will be available to the public in the coming weeks. Follow <a href="https://x.com/virtuals_io" target="_blank">our X page</a> and join <a href="https://t.me/virtuals" target="_blank">our Telegram group</a> for updates.
</p>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- References -->
<section class="section hero is-small">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-justified">
<div class="column is-four-fifths">
<h2 class="title is-4">References</h2>
<div class="content has-text-justified">
<ol>
<li>Park JS, O'Brien J, Cai CJ, Morris MR, Liang P, Bernstein MS (2023). Generative agents: Interactive simulacra of human behavior. In <em>Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology 2023</em> (pp. 1-22).</li>
<li>Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). ReAct: Synergizing reasoning and acting in language models.</li>
<li>Chase, H. (2024). What is an agent? <a href="https://blog.langchain.dev/what-is-an-agent/">https://blog.langchain.dev/what-is-an-agent/</a></li>
<li>Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y. and Gupta, S. (2024). Self-refine: Iterative refinement with self-feedback. <em>Advances in Neural Information Processing Systems</em>, <em>36</em>.</li>
<li>Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V. and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. <em>Advances in neural information processing systems</em>, <em>35</em>, pp.24824-24837.</li>
<li>Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A. and Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. <em>arXiv preprint arXiv:2203.11171</em>.</li>
<li>Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, J.X., Kan, M.Y., He, J. and Xie, M. (2024). Self-evaluation guided beam search for reasoning. <em>Advances in Neural Information Processing Systems</em>, <em>36</em>.</li>
<li>Andrew Ng. Issue 253. <a href="https://www.deeplearning.ai/the-batch/issue-253/">https://www.deeplearning.ai/the-batch/issue-253/</a>, June 2024. Newsletter issue.</li>
<li>Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.T., Rocktäschel, T. and Riedel, S. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. <em>Advances in Neural Information Processing Systems</em>, <em>33</em>, pp.9459-9474.</li>
<li>Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N. and Scialom, T. (2024). Toolformer: Language models can teach themselves to use tools. <em>Advances in Neural Information Processing Systems</em>, <em>36</em>.</li>
<li>Qu, C., Dai, S., Wei, X., Cai, H., Wang, S., Yin, D., Xu, J. and Wen, J.R. (2024). Tool Learning with Large Language Models: A Survey. <em>arXiv preprint arXiv:2405.17935</em>.</li>
<li>Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K. and Herzog, A., 2022. Do as i can, not as i say: Grounding language in robotic affordances. <em>arXiv preprint arXiv:2204.01691</em>.</li>
<li>Shanahan, M., McDonell, K. and Reynolds, L. (2023). Role play with large language models. <em>Nature</em>, <em>623</em>(7987), pp.493-498.</li>
</ol>
</div>
</div>
</div>
</div>
</div>
</section>
<footer class="footer">
<div class="container">
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This page was built using the <a href="https://github.com/eliahuhorwitz/Academic-project-page-template" target="_blank">Academic Project Page Template</a> which was adopted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a> project page.
You are free to borrow the of this website, we just ask that you link back to this page in the footer. <br> This website is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative
Commons Attribution-ShareAlike 4.0 International License</a>.
</p>
</div>
</div>
</div>
</div>
</footer>
<!-- Statcounter tracking code -->
<!-- Default Statcounter code for GAME
https://virtual-protocol.github.io/mario-videogamegen/ -->
<script type="text/javascript">
var sc_project=13032676;
var sc_invisible=1;
var sc_security="74d282d4";
</script>
<script type="text/javascript"
src="https://www.statcounter.com/counter/counter.js" async></script>
<noscript><div class="statcounter"><a title="Web Analytics"
href="https://statcounter.com/" target="_blank"><img class="statcounter"
src="https://c.statcounter.com/13032676/0/74d282d4/1/" alt="Web Analytics"
referrerPolicy="no-referrer-when-downgrade"></a></div></noscript>
<!-- End of Statcounter Code -->
<!-- End of Statcounter Code -->
</body>
</html>