Title: MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft

URL Source: https://arxiv.org/html/2504.08388

Published Time: Mon, 14 Apr 2025 00:35:48 GMT

Markdown Content:
MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft
===============

1.   [1 Introduction](https://arxiv.org/html/2504.08388v1#S1 "In MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
2.   [2 Framework](https://arxiv.org/html/2504.08388v1#S2 "In MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
    1.   [2.1 Overview](https://arxiv.org/html/2504.08388v1#S2.SS1 "In 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
    2.   [2.2 Architectures](https://arxiv.org/html/2504.08388v1#S2.SS2 "In 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        1.   [Tokenizers](https://arxiv.org/html/2504.08388v1#S2.SS2.SSS0.Px1 "In 2.2 Architectures ‣ 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        2.   [Transformer Decoder](https://arxiv.org/html/2504.08388v1#S2.SS2.SSS0.Px2 "In 2.2 Architectures ‣ 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")

    3.   [2.3 Parallel Decoding](https://arxiv.org/html/2504.08388v1#S2.SS3 "In 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        1.   [Real-Time Interactive World Model](https://arxiv.org/html/2504.08388v1#S2.SS3.SSS0.Px1 "In 2.3 Parallel Decoding ‣ 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        2.   [Discussion with Related Works](https://arxiv.org/html/2504.08388v1#S2.SS3.SSS0.Px2 "In 2.3 Parallel Decoding ‣ 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")

3.   [3 Evaluation](https://arxiv.org/html/2504.08388v1#S3 "In MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
    1.   [3.1 Dataset Construction and Preprocessing](https://arxiv.org/html/2504.08388v1#S3.SS1 "In 3 Evaluation ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
    2.   [3.2 Evaluation Metrics](https://arxiv.org/html/2504.08388v1#S3.SS2 "In 3 Evaluation ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        1.   [Discrete Action Classification](https://arxiv.org/html/2504.08388v1#S3.SS2.SSS0.Px1 "In 3.2 Evaluation Metrics ‣ 3 Evaluation ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        2.   [Camera Movement.](https://arxiv.org/html/2504.08388v1#S3.SS2.SSS0.Px2 "In 3.2 Evaluation Metrics ‣ 3 Evaluation ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")

    3.   [3.3 Implementation Details](https://arxiv.org/html/2504.08388v1#S3.SS3 "In 3 Evaluation ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")

4.   [4 Experiments](https://arxiv.org/html/2504.08388v1#S4 "In MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
    1.   [4.1 Main Results](https://arxiv.org/html/2504.08388v1#S4.SS1 "In 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
    2.   [4.2 Analysis](https://arxiv.org/html/2504.08388v1#S4.SS2 "In 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        1.   [The Effect of Parallel Decoding](https://arxiv.org/html/2504.08388v1#S4.SS2.SSS0.Px1 "In 4.2 Analysis ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        2.   [The Metrics on Controllability](https://arxiv.org/html/2504.08388v1#S4.SS2.SSS0.Px2 "In 4.2 Analysis ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")

    3.   [4.3 Case Study](https://arxiv.org/html/2504.08388v1#S4.SS3 "In 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        1.   [General Capability](https://arxiv.org/html/2504.08388v1#S4.SS3.SSS0.Px1 "In 4.3 Case Study ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        2.   [Controllability](https://arxiv.org/html/2504.08388v1#S4.SS3.SSS0.Px2 "In 4.3 Case Study ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
        3.   [Serve as an Agent](https://arxiv.org/html/2504.08388v1#S4.SS3.SSS0.Px3 "In 4.3 Case Study ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")

5.   [5 Conclusion and Limitations](https://arxiv.org/html/2504.08388v1#S5 "In MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
    1.   [Limitations](https://arxiv.org/html/2504.08388v1#S5.SS0.SSS0.Px1 "In 5 Conclusion and Limitations ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")

6.   [A Minecraft Action Space](https://arxiv.org/html/2504.08388v1#A1 "In MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
    1.   [A.1 Details of Action Following Metric](https://arxiv.org/html/2504.08388v1#A1.SS1 "In Appendix A Minecraft Action Space ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
    2.   [A.2 Task-Specific Action Accuracy](https://arxiv.org/html/2504.08388v1#A1.SS2 "In Appendix A Minecraft Action Space ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")

7.   [B Model Configurations](https://arxiv.org/html/2504.08388v1#A2 "In MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
8.   [C More Results](https://arxiv.org/html/2504.08388v1#A3 "In MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")
    1.   [C.1 Reconstruction Results of Visual Tokenizer](https://arxiv.org/html/2504.08388v1#A3.SS1 "In Appendix C More Results ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")

MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft
===========================================================================

Junliang Guo∗, Yang Ye∗, Tianyu He∗, Haoyu Wu∗, Yushu Jiang, Tim Pearce, Jiang Bian∗Equal contribution. Correspondence to Junliang Guo. Microsoft Research 

{junliangguo,v-yangye,tianyuhe, v-haoywu}@microsoft.com

###### Abstract

World modeling is a crucial task for enabling intelligent agents to effectively interact with humans and operate in dynamic environments. In this work, we propose MineWorld, a real-time interactive world model on Minecraft, an open-ended sandbox game which has been utilized as a common testbed for world modeling. MineWorld is driven by a visual-action autoregressive Transformer, which takes paired game scenes and corresponding actions as input, and generates consequent new scenes following the actions. Specifically, by transforming visual game scenes and actions into discrete token ids with an image tokenizer and an action tokenizer correspondingly, we consist the model input with the concatenation of the two kinds of ids interleaved. The model is then trained with next token prediction to learn rich representations of game states as well as the conditions between states and actions simultaneously. In inference, we develop a novel parallel decoding algorithm that predicts the spatial redundant tokens in each frame at the same time, letting models in different scales generate 4 4 4 4 to 7 7 7 7 frames per second and enabling real-time interactions with game players. In evaluation, we propose new metrics to assess not only visual quality but also the action following capacity when generating new scenes, which is crucial for a world model. Our comprehensive evaluation shows the efficacy of MineWorld, outperforming SoTA open-sourced diffusion based world models significantly. The code and model have been released.

1 Introduction
--------------

World models have been extensively studied for their potential ability to simulate and interact with various environments and actions taken by humans or agents(Ha & Schmidhuber, [2018](https://arxiv.org/html/2504.08388v1#bib.bib10); Yang et al., [2023](https://arxiv.org/html/2504.08388v1#bib.bib34)). These models provide a computational framework that empowers intelligent agents to perceive surroundings, receive controls, and predict consequences. Thus, world models reduce reliance on real-world trials and automate complex tasks across industries, such as serving as a game engine(Valevski et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib27); Bruce et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib6); Decart et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib8); Kanervisto et al., [2025](https://arxiv.org/html/2504.08388v1#bib.bib14)) or a planner in a reinforcement learning system(Hafner et al., [2019](https://arxiv.org/html/2504.08388v1#bib.bib11); Wu et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib33); Agarwal et al., [2025](https://arxiv.org/html/2504.08388v1#bib.bib1)), illustrating their ability to improve decision-making, enable safe exploration, and facilitate scalable learning.

Recently, video generative models have shown a promising ability to learn commonsense knowledge from raw video datasets, ranging from physical laws in the real world(Brooks et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib4)) to object interactions in games(Parker-Holder et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib20); Kanervisto et al., [2025](https://arxiv.org/html/2504.08388v1#bib.bib14)), laying the foundation for their use as real-world simulators. However, foundamental challenges exist in ensuring the efficiency and controllability of these models, which are both crucial features for a desired world model.

The efficiency bottleneck lies in the generation target of these video generative models, _i.e_., the latent videos representation encoded by visual tokenizers, consists of a large number of tokens (_e.g_., 40 40 40 40 k to 160 160 160 160 k tokens for 16 16 16 16 frames with SoTA tokenizers(Yang et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib35); Tang et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib24); Wang et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib30))). That leads to a substantial computational costs during inference, making real-time interactions with the model a major challenge. Furthermore, the controllability requires the model to generate accurate outcomes based on a given control signal, which is challenging to evaluate due to the diverse nature of these signals. For instance, video generative models may be conditioned on textual descriptions(Brooks et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib4)), video demonstrations(Zhang et al., [2025](https://arxiv.org/html/2504.08388v1#bib.bib41)), and numeric features such as robotic arm movements(Wu et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib33)), yet a standardized metric to quantify how well the generated results adhere to the input signals remains lacking.

In this work, we propose MineWorld, a real-time, open-source interactive world model on the game Minecraft. It is built upon an autoregressive Transformer, designed to overcome the challenges of the efficiency and controllability in video-based world modeling. MineWorld explicitly learns the correlation between visual states and control signals by tokenizing both game scenes and actions into discrete representations, which are concatenated interleaved as the input to the model. To achieve real-time interactions between the model and humans 1 1 1 We provide the definition of real-time interaction for our world model in Section[2.3](https://arxiv.org/html/2504.08388v1#S2.SS3 "2.3 Parallel Decoding ‣ 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")., we introduce a novel parallel decoding algorithm that significantly accelerates the autoregressive generation of Transformer. Instead of sequentially predicting each token step by step, our method exploits the dependencies between spatially adjacent tokens, allowing certain token groups to be predicted in parallel. This optimization results in over a more than 3×3\times 3 × speedup compared to standard autoregressive decoding, without sacreficing the quality of results. Equipped with this decoding algorithm, MineWorld is able to generate 4 4 4 4 to 7 7 7 7 frames in one second, making the real-time interaction between human and world model feasible.

In addition, to assess the controllability of MineWorld, we propose new evaluation metrics that extend beyond conventional video quality assessments. We first transform the actions in Minecraft to discrete tokens in one vocabulary. Then, after output videos are generated conditioned on previous frames and actions, we utilize an inverse dynamic model(IDM)(Baker et al., [2022](https://arxiv.org/html/2504.08388v1#bib.bib3)) to predict an action between consecutive generated frames. This predicted action can be treated as the executed action according to generated videos, while the input action is the ground truth one which serves as the condition. Therefore, the accuracy between the two kinds of actions reflect the controllability of this generative model.

By addressing both efficiency and controllability, MineWorld advances the field of world modeling, offering the first high-quality and efficient framework for real-time simulation and interaction. To empower further research in this direction, we release our code and model weights. In summary, our contributions are threefold:

*   •MineWorld, an open-sourced, real-time interactive world model powered by an autoregressive Transformer, establishes a new benchmark for world modeling. 
*   •A novel parallel decoding algorithm that brings significant speedup in inference over standard autoregressive decoding, while maintaining high-quality generation results. 
*   •New evaluation metrics for assessing the controllability, in order to validate whether the generated sequences faithfully adhere to control signals. 

2 Framework
-----------

![Image 1: Refer to caption](https://arxiv.org/html/x1.png)

Figure 1: Illustrations of MineWorld model architecture. Visual and action tokenizers convert game states and actions into discrete tokens, which are concatenated and fed into a Transformer decoder as the input. The Transformer is then trained with an autoregressive objective.

### 2.1 Overview

We will introduce the proposed MineWorld framework in this section. We start with the problem definition of the studied task. Denote x i subscript 𝑥 𝑖 x_{i}italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT as the visual game state from Minecraft at the i 𝑖 i italic_i-th timestep, and a i subscript 𝑎 𝑖 a_{i}italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT as the action taken by users on the state, then the next game state x i+1 subscript 𝑥 𝑖 1 x_{i+1}italic_x start_POSTSUBSCRIPT italic_i + 1 end_POSTSUBSCRIPT represents the future when taking a i subscript 𝑎 𝑖 a_{i}italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT on x i subscript 𝑥 𝑖 x_{i}italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT. The objective of our world model is to predict future game states based on past observations x<i subscript 𝑥 absent 𝑖 x_{<i}italic_x start_POSTSUBSCRIPT < italic_i end_POSTSUBSCRIPT and the current action a i subscript 𝑎 𝑖 a_{i}italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, modeling the following conditional distribution:

p⁢(x i+1|x<i,a i).𝑝 conditional subscript 𝑥 𝑖 1 subscript 𝑥 absent 𝑖 subscript 𝑎 𝑖 p(x_{i+1}|x_{<i},a_{i}).italic_p ( italic_x start_POSTSUBSCRIPT italic_i + 1 end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT < italic_i end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) .(1)

We build MineWorld on the autoregressive Transformer(Vaswani et al., [2017](https://arxiv.org/html/2504.08388v1#bib.bib29)), considering its scaling behavior(Kaplan et al., [2020](https://arxiv.org/html/2504.08388v1#bib.bib15); Brown et al., [2020](https://arxiv.org/html/2504.08388v1#bib.bib5)) and good controllability (as will be shown in Section[4](https://arxiv.org/html/2504.08388v1#S4 "4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft")). As illustrated in Figure[1](https://arxiv.org/html/2504.08388v1#S2.F1 "Figure 1 ‣ 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"), the architecture consists of two modules, the tokenizers to convert videos and actions to discrete tokens, and a Transformer decoder to take the sequence of tokens as input and train in an autoregressive way. We introduce the details in the following.

### 2.2 Architectures

#### Tokenizers

The model input contains two different modalities, _i.e_., the game playing videos consist of sequences of states x i subscript 𝑥 𝑖 x_{i}italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, and actions consist of mouse and keyboard inputs. Therefore, we design different tokenizers to convert them into discrete tokens respectively.

For game videos, we train a VQ-VAE(Van Den Oord et al., [2017](https://arxiv.org/html/2504.08388v1#bib.bib28); Esser et al., [2021](https://arxiv.org/html/2504.08388v1#bib.bib9)) as the visual tokenizer. Considering the interleaved manner of the game states and actions, we utilize an image-level (_i.e_., with spatial compressions) VQ tokenizer to convert each state to tokens independently, and leave the utilization of video-level tokenizers (_i.e_., with both spatial and temporal compressions)(Tang et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib24)) for future work. Specifically, we initialize the VQ tokenizer from from a public pre-trained checkpoint(Patil et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib22)) and then fine-tune it on the Minecraft dataset. The tokenizer has 16×16\times 16 × compression rates on both the height and width. As a result, for a clip of game video x 𝑥 x italic_x that contains n 𝑛 n italic_n states, the VQ tokenizer converts it into a sequence of quantized ids t 𝑡 t italic_t, denoted as:

x 𝑥\displaystyle x italic_x=(x 1,⋯,x n),absent subscript 𝑥 1⋯subscript 𝑥 𝑛\displaystyle=(x_{1},\cdots,x_{n}),= ( italic_x start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , ⋯ , italic_x start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) ,(2)
t 𝑡\displaystyle t italic_t=(t 1,⋯,t c,t c+1,⋯,t 2⁢c,t 2⁢c+1,⋯⁢t N),absent subscript 𝑡 1⋯subscript 𝑡 𝑐 subscript 𝑡 𝑐 1⋯subscript 𝑡 2 𝑐 subscript 𝑡 2 𝑐 1⋯subscript 𝑡 𝑁\displaystyle=(t_{1},\cdots,t_{c},t_{c+1},\cdots,t_{2c},t_{2c+1},\cdots t_{N}),= ( italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , ⋯ , italic_t start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT , italic_t start_POSTSUBSCRIPT italic_c + 1 end_POSTSUBSCRIPT , ⋯ , italic_t start_POSTSUBSCRIPT 2 italic_c end_POSTSUBSCRIPT , italic_t start_POSTSUBSCRIPT 2 italic_c + 1 end_POSTSUBSCRIPT , ⋯ italic_t start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ) ,

where N=n⋅c 𝑁⋅𝑛 𝑐 N=n\cdot c italic_N = italic_n ⋅ italic_c is the total length of the sequence, and c 𝑐 c italic_c is the number of ids to represent each state.

The actions in Minecraft contains both continuous mouse movements that controls camera angles, and keyboard or mouse presses that represent discrete actions defined in the game engine such as forward and attack. To handle continuous movements, we follow previous practices(Baker et al., [2022](https://arxiv.org/html/2504.08388v1#bib.bib3)) and quantize camera angles into discrete bins. For discrete actions, considering the mutual exclusive nature of certain action pairs (e.g., forward and backward cannot occur simultaneously), we categorize the actions into 7 7 7 7 exclusive classes, each represented by a unique token. Additionally, we allocate 2 2 2 2 tokens to encode camera angles and introduce special tokens [aBOS] and [aEOS] to mark the boundaries of an action sequence. As a result, each action is represented by a sequence of 11 11 11 11 tokens, where each token corresponds to an action id from the complete action vocabulary. Dividing actions into exclusive classes not only helps reduce the sequence length of action tokens, but also makes it possible to validate the controllability of the model with classification based metrics, which will be introduced in Section[3](https://arxiv.org/html/2504.08388v1#S3 "3 Evaluation ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft").

In conclusion, for each pair of game state and actions in the original input (x i,a i)subscript 𝑥 𝑖 subscript 𝑎 𝑖(x_{i},a_{i})( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ), the tokenizers will transfer them into a flat sequence of discrete ids as:

(t i∗c+1,⋯,t(i+1)∗c,[aBOS],t 1 a i,⋯,t 9 a i,[aEOS]).subscript 𝑡 𝑖 𝑐 1⋯subscript 𝑡 𝑖 1 𝑐 delimited-[]aBOS subscript superscript 𝑡 subscript 𝑎 𝑖 1⋯subscript superscript 𝑡 subscript 𝑎 𝑖 9[aEOS](t_{i*c+1},\cdots,t_{(i+1)*c},[\texttt{aBOS}],t^{a_{i}}_{1},\cdots,t^{a_{i}}_{% 9},\texttt{[aEOS]}).( italic_t start_POSTSUBSCRIPT italic_i ∗ italic_c + 1 end_POSTSUBSCRIPT , ⋯ , italic_t start_POSTSUBSCRIPT ( italic_i + 1 ) ∗ italic_c end_POSTSUBSCRIPT , [ aBOS ] , italic_t start_POSTSUPERSCRIPT italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , ⋯ , italic_t start_POSTSUPERSCRIPT italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 9 end_POSTSUBSCRIPT , [aEOS] ) .(3)

#### Transformer Decoder

We design our Transformer following the LLaMA architecture(Touvron et al., [2023](https://arxiv.org/html/2504.08388v1#bib.bib25)) equipped with root mean square layer normalization(Zhang & Sennrich, [2019](https://arxiv.org/html/2504.08388v1#bib.bib39)) and Rotary Embeddings(Su et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib23)). We train the model as a traditional autoregressive decoder, where each token is predicted conditioned on all previous tokens,

f θ⁢(t)=∏i=1 N p⁢(t i|t<i).subscript 𝑓 𝜃 𝑡 superscript subscript product 𝑖 1 𝑁 𝑝 conditional subscript 𝑡 𝑖 subscript 𝑡 absent 𝑖 f_{\theta}(t)=\prod_{i=1}^{N}p(t_{i}|t_{<i}).italic_f start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_t ) = ∏ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT italic_p ( italic_t start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | italic_t start_POSTSUBSCRIPT < italic_i end_POSTSUBSCRIPT ) .(4)

Note that we treat the tokens of game states and actions equivalently, allowing the model to learn their interleaved structure. As a result, the model jointly captures the conditional relationships between states and actions, enabling it to function as both a policy model (_i.e_., predicting actions based on previous observations p⁢(a i+1|x<i+1)𝑝 conditional subscript 𝑎 𝑖 1 subscript 𝑥 absent 𝑖 1 p(a_{i+1}|x_{<i+1})italic_p ( italic_a start_POSTSUBSCRIPT italic_i + 1 end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT < italic_i + 1 end_POSTSUBSCRIPT )) and a world model (_i.e_., predict future states as described in Equ([1](https://arxiv.org/html/2504.08388v1#S2.E1 "In 2.1 Overview ‣ 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"))) in inference. While our primary focus in this paper is on the model’s performance as a world model, we also present case studies in the experiments to demonstrate its potential as a policy model.

### 2.3 Parallel Decoding

![Image 2: Refer to caption](https://arxiv.org/html/x2.png)

Figure 2: An illustration of two decoding algorithms. The game state is encoded into 12 12 12 12 tokens. The number in each token represents that in which decoding iteration it is generated. (a) Autoregressive decoding, which follows a raster scanned order and generates each token sequentially. (b) Our proposed parallel decoding. For each generated token, the tokens in the adjacent rows and columns will be generated simutaneously in the next iteration.

For a world model, it is crucial to generate real-time consequences w.r.t. the controling signals provided by the users. We provide the definition of “real-time” in our game scenario considering the Actions Per Minute(APM)(Wikipedia contributors, [2025](https://arxiv.org/html/2504.08388v1#bib.bib32)) of professional players.

#### Real-Time Interactive World Model

APM is a metric used to measure the number of in-game actions(_e.g_., keypresses, mouse clicks) a player executes within one minute. Professional players in the game StarCraft typically have APMs around 250 250 250 250 to 300 300 300 300 in average, and 50 50 50 50 for beginners(Wikipedia contributors, [2025](https://arxiv.org/html/2504.08388v1#bib.bib32)). We also conduct an in-house test on amateur game players and find that they have 150 150 150 150 APMs in average. Therefore, to develop a world model that achieves real-time interactions with users, it should generate more than 2 2 2 2 Frames Per Second(FPS) to keep up with amateur game players and 5 5 5 5 FPS for professional ones.

While visual autoregressive models(Liu et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib19); Yu et al., [2023](https://arxiv.org/html/2504.08388v1#bib.bib37); Kondratyuk et al., [2023](https://arxiv.org/html/2504.08388v1#bib.bib17)) can provide high-fidelity results for image and video generation tasks, their sequential nature poses a significant bottleneck on the efficiency when generating high-resolution, long-duration videos. Previous works(Bai et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib2); Yu et al., [2023](https://arxiv.org/html/2504.08388v1#bib.bib37)) generate visual tokens following a fixed raster-scan order (_i.e_., from left-to-right and top-to-bottom) or random orders(Chang et al., [2022](https://arxiv.org/html/2504.08388v1#bib.bib7); Yu et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib38)), without leveraging the inherent spatio and temporal dependencies in images and videos.

To facilitate the real-time interaction of our world model, we utilize Diagonal Decoding(Ye et al., [2025](https://arxiv.org/html/2504.08388v1#bib.bib36)), a training-free parallel decoding algorithm by leveraging the spatial dependencies between adjacent image tokens, providing significant speedup compared to naive autoregressive decoding with negligible quality degradation. Specifically, our method processes tokens across different rows and columns simultaneously, as shown in Figure[2](https://arxiv.org/html/2504.08388v1#S2.F2 "Figure 2 ‣ 2.3 Parallel Decoding ‣ 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"). Let x i,j subscript 𝑥 𝑖 𝑗 x_{i,j}italic_x start_POSTSUBSCRIPT italic_i , italic_j end_POSTSUBSCRIPT denote one of the generated tokens at row i 𝑖 i italic_i and column j 𝑗 j italic_j in the current game state, then in the next step, both tokens x i,j+1 subscript 𝑥 𝑖 𝑗 1 x_{i,j+1}italic_x start_POSTSUBSCRIPT italic_i , italic_j + 1 end_POSTSUBSCRIPT and x i+1,j subscript 𝑥 𝑖 1 𝑗 x_{i+1,j}italic_x start_POSTSUBSCRIPT italic_i + 1 , italic_j end_POSTSUBSCRIPT will be generated at the same time. Denote the height and width of the patchified game state as h ℎ h italic_h and w 𝑤 w italic_w, then the theoretical speedup ratio of our parallel decoding versus autoregressive decoding can be written as:

r=h×w h+w−1.𝑟 ℎ 𝑤 ℎ 𝑤 1 r=\frac{h\times w}{h+w-1}.italic_r = divide start_ARG italic_h × italic_w end_ARG start_ARG italic_h + italic_w - 1 end_ARG .(5)

From the above equation, we can observe that the larger the image resolution, the faster our decoding speed.

Our method utilizes the spatial redundency of adjacent tokens. However, due to the discrepency between training and inference, the speedup brings some performance degradation to the result. To solve the problem, we propose to fine-tune the autoregressively pre-trained model, by replacing the standard causal attention mask with the mask that aligns with our parallel decoding algorithm. As a result, our model is able to achieve real-time interaction frequency while maintaining good quality of generation results.

#### Discussion with Related Works

Several prior works have explored parallel decoding in visual generative models. Lformer(Li et al., [2023](https://arxiv.org/html/2504.08388v1#bib.bib18)) generate L-shaped token blocks in each iteration but requires training the model from scratch. More recently, ZipAR(He et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib12)) propose a parallel decoding algorithm for image generation by leveraging adjacent token dependencies. In contrast, our approach focuses on video generation, which presents additional challenges due to error accumulation across frames. Despite this increased complexity, our method achieves higher acceleration ratios relative to the autoregressive baseline, demonstrating the effectiveness of our parallel decoding strategy for real-time interactive applications.

3 Evaluation
------------

To comprehensively evaluate the performance of world models in the Minecraft environment, we assess both the video quality and controllability quality of generation results. We introduce the construction and preprocessing of the dataset first.

### 3.1 Dataset Construction and Preprocessing

We utilize the VPT dataset(Baker et al., [2022](https://arxiv.org/html/2504.08388v1#bib.bib3)) which consists of the pairs of recorded game playing videos and corresponding actions, where each frame is accompanied with the keyboard and mouse actions taken at the same time. We filter out the frames with no recorded action and also the ones when GUI is open to eliminate its influence of results. We split long videos into short clips with 16 16 16 16 frames considering the maximum context length of the model, and divide them into training, validation and test sets randomly. As a result, we train the model on 10 10 10 10 M video clips(_i.e_., 160 160 160 160 M frames) and validate / test on 0.5 0.5 0.5 0.5 k / 1 1 1 1 k clips.

For each original video, we resize its resolution from 360×640 360 640 360\times 640 360 × 640 to 224×384 224 384 224\times 384 224 × 384, to reduce the computation cost while keeping the original aspect ratio for better visual quality. As introduced in Section[2.2](https://arxiv.org/html/2504.08388v1#S2.SS2 "2.2 Architectures ‣ 2 Framework ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"), we adopt a VQ-VAE with 16×16\times 16 × spatial compression ratio and 8 8 8 8 k codebook size to transform each frame into 14×24 14 24 14\times 24 14 × 24 patches, and finally a sequence of 336 336 336 336 discrete tokens. In addtion to the 11 11 11 11 action tokens encoded by our action tokenizer, we represent each pair of game state and action as 347 347 347 347 tokens. As a result, for each training sample which contains 16 16 16 16 pairs of them, we transform it into 5.5 5.5 5.5 5.5 k discrete tokens after preprocessing. And in total there is 55 55 55 55 B tokens in the training set.

### 3.2 Evaluation Metrics

We utilize two types of metrics to assess the visual quality and the controllablility of the results generated by our model. For visual quality, we employ several common metrics including Fréchet Video Distance (FVD)(Unterthiner et al., [2018](https://arxiv.org/html/2504.08388v1#bib.bib26)), Peak Signal-to-Noise Ratio (PSNR)(Hore & Ziou, [2010](https://arxiv.org/html/2504.08388v1#bib.bib13)), Learned Perceptual Image Patch Similarity (LPIPS)(Zhang et al., [2018](https://arxiv.org/html/2504.08388v1#bib.bib40)), and Structural Similarity Index Measure (SSIM)(Wang et al., [2004](https://arxiv.org/html/2504.08388v1#bib.bib31)).

Controllablility indicates to what extent the generated game states align with the provided actions and previous states, which is a crucial feature for an accurate world model. To evaluate it, we utilize an Inverse Dynamics Model(IDM)(Baker et al., [2022](https://arxiv.org/html/2504.08388v1#bib.bib3)) to infer the generated actions from the generated states and previous states. Specifically, the IDM is a bidirectional model which takes game states as input and predicts the actions between them. We utilize a highly accurate pre-trained IDM(Baker et al., [2022](https://arxiv.org/html/2504.08388v1#bib.bib3)) which achieves 90.6%percent 90.6 90.6\%90.6 % accuracy on keypresses. However, due to the imbalanced nature of action labels (_e.g_., forward and attack appear much more frequently than use and drop) as well as the mixture of discrete and continuous actions (_e.g_., keyboard presses and camera movements), it is non-trival to design accurate and comprehensive evaluation metrics on the controllablility. We propose two kinds of metrics to solve this problem.

#### Discrete Action Classification

Following the dividing policy in the action tokenizer, actions can be grouped into 9 9 9 9 classes, where 7 7 7 7 of them represent discrete action classes and the other 2 2 2 2 represent camera movement angles. For discrete classes, each one of them contains two or three exclusive actions such as (forward,backward)forward backward(\texttt{forward},\texttt{backward})( forward , backward ) and (left,right)left right(\texttt{left},\texttt{right})( left , right ). We provide the full grouping results in Table[4](https://arxiv.org/html/2504.08388v1#A1.T4 "Table 4 ‣ A.1 Details of Action Following Metric ‣ Appendix A Minecraft Action Space ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"). Then, by taking the provided action as the ground truth and the predicted action from IDM as the prediction, we can utilize commonly utilized classification metrics including precision, recall and F1 score to evaluate the classification accuracy. We report both the macro scores to reduce the effect of imbalanced labels.

#### Camera Movement.

Camera movement is represented by the player’s rotational angle and handled separately from action prediction. Following the configuration in VPT(Baker et al., [2022](https://arxiv.org/html/2504.08388v1#bib.bib3)), we discretize the camera rotation along both the X and Y axes into 11 bins, each representing a specific range of angles. This design provides finer granularity for small movements and coarser bins for larger rotations, balancing precision and range. To evaluate the controllability of camera movements, we compute the L1 loss between the predicted and ground truth camera bins.

### 3.3 Implementation Details

For the visual tokenizer, we initialize it from a pre-trained checkpoint(Patil et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib22)) and then fine-tune it on the preprocessed VPT training data. We show the reconstruction results of visual tokenizers in Appendix[C](https://arxiv.org/html/2504.08388v1#A3 "Appendix C More Results ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"), and we can find that after fine-tuning, the tokenizer achieves good reconstrution quality on the validation set of VPT.

For the Transformer decoder, we adopt LLaMA(Touvron et al., [2023](https://arxiv.org/html/2504.08388v1#bib.bib25)) architecture considering its advantageous to large-scale modeling. We experiment with different sets of hyper-parameters which result in models with 300 300 300 300 M, 700 700 700 700 M and 1.2 1.2 1.2 1.2 B parameters separately. The vocabulary size of the image tokenizer is 8192 8192 8192 8192, plus 70 70 70 70 ids from the action vocabulary, and thus the final vocabulary size of our model is 8262 8262 8262 8262. We use the Adam optimizer(Kingma & Ba, [2015](https://arxiv.org/html/2504.08388v1#bib.bib16)) with a cosine decay learning rate scheduler to train the model. The training is conducted on 32 32 32 32 NVIDIA 40G A100 GPUs with PyTorch(Paszke et al., [2019](https://arxiv.org/html/2504.08388v1#bib.bib21)) for 200 200 200 200 k steps. We introduce more training details in Appendix[B](https://arxiv.org/html/2504.08388v1#A2 "Appendix B Model Configurations ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft").

4 Experiments
-------------

We present the results of MineWorld in this section. We compare our model with Oasis(Decart et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib8)), an open-sourced diffusion based world model on Minecraft.

### 4.1 Main Results

Table 1: Generation result of Oasis(Decart et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib8)) and different scales of our models.“FPS” indicates the number of frames generated per second by the model. “P, R, F1” denote the classification precision, recall and F1 scores on discrete actions. “L1” indicates the camera control loss.

Method Param.FPS↑↑\uparrow↑P↑↑\uparrow↑R↑↑\uparrow↑F1↑↑\uparrow↑L1↓↓\downarrow↓FVD↓↓\downarrow↓LPIPS↓↓\downarrow↓SSIM↑↑\uparrow↑PSNR↑↑\uparrow↑
Oasis 500M 2.58 0.49 0.44 0.41 2.60 377 0.53 0.36 14.38
MineWorld 300M 5.91 0.72 0.71 0.70 1.03 246 0.45 0.38 15.13
700M 3.18 0.72 0.71 0.70 1.04 231 0.44 0.38 15.32
1.2B 3.01 0.76 0.73 0.73 1.02 227 0.44 0.41 15.69

We provide the results of our models and the Oasis baseline in Table[1](https://arxiv.org/html/2504.08388v1#S4.T1 "Table 1 ‣ 4.1 Main Results ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"). Both the video quality and controllability accuracy scores are presented. We use the open-sourced 500 500 500 500 M model and inference code from Oasis to obtain its results. Compare with it, all of our models achieve better results on both aspects of evaluation metrics. Note that while they claim good video quality and fast inference speed in their blog 2 2 2[https://oasis-model.github.io](https://oasis-model.github.io/), it requires a larger model equipped with dedicated chips and inference engine, which are public inaccessible. On the other hand, our proposed parallel decoding algorithm is orthogonal to hardware and system level optimizations, and thus have the potential to achieve faster inference speed if strengths from both sides are combined.

From the comparisons between different model scales of MineWorld, we observe a clear scaling behaviour. Larger models achieve better performance on both the controllability and video quality. As for the inference speed, thanks to the proposed parallel decoding algorithm, even the largest 1.2 1.2 1.2 1.2 B model reaches a FPS of 3 3 3 3, _i.e_., generating 3 3 3 3 game states in one second and thus can respond to 180 180 180 180 Actions Per Minute(APM), an interaction frequence for most amateur game players. The most efficient 300 300 300 300 M model achieves an APM around 360 360 360 360, which is able to interact with top professional players.

### 4.2 Analysis

#### The Effect of Parallel Decoding

Table 2: The performance of different scales of MineWorld models with default autoregressive decoding and the proposed parallel decoding. “w/” or “w/o FT” indicate results after fine-tuning the autoregressive pre-trained model with the parallel attenion mask or not. We choose several main metrics on the efficiency and effectiveness to make the table compact.

Param.Decoding FPS↑↑\uparrow↑F1↑↑\uparrow↑PSNR↑↑\uparrow↑FVD↓↓\downarrow↓
300M AT 2.00 0.70 15.63 223
Parallel w/ FT 5.91 0.70 15.13 246
Parallel w/o FT 5.91 0.69 14.98 275
700M AT 1.08 0.73 15.74 210
Parallel w/ FT 3.18 0.71 15.32 231
Parallel w/o FT 3.18 0.70 15.27 247
1.2B AT 0.89 0.72 16.06 203
Parallel w/ FT 3.01 0.73 15.69 227
Parallel w/o FT 3.01 0.70 15.30 258

We compare the performance of autoregressive decoding and the proposed parallel decoding in Table[2](https://arxiv.org/html/2504.08388v1#S4.T2 "Table 2 ‣ The Effect of Parallel Decoding ‣ 4.2 Analysis ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"). The conclusions are twofold. Firstly, larger models perform better with parallel decoding even without fine-tuning. For example, the 1.2 1.2 1.2 1.2 B MineWorld model achieves a 3×3\times 3 × speedup in inference latency with parallel decoding, while preserving comparable performance in both controllability and video quality relative to autoregressive decoding.

Secondly, fine-tuning with the parallel attention mask consistently enhances the performance of parallel decoding. For smaller models, such as the 300 300 300 300 M variant, fine-tuning enables the model to match the autoregressive baseline, while still benefiting from a 3×3\times 3 × speedup in inference latency. These results suggest an exciting direction for future research: training the model from scratch with the parallel attention mask may lead to faster convergence and lower overall training costs.

#### The Metrics on Controllability

Table 3: The correlation between the proposed metrics and human evaluation on controllability.

F1 0.81 0.81 0.81 0.81
Human 4.21 4.21 4.21 4.21
r 𝑟 r italic_r 0.56 0.56 0.56 0.56
p-value 0.01 0.01 0.01 0.01

To validate that the metrics proposed in Section[3.2](https://arxiv.org/html/2504.08388v1#S3.SS2 "3.2 Evaluation Metrics ‣ 3 Evaluation ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft") accurately measure the controllability of the model, we conduct human evaluation and calculate the correlation coefficient between the proposed metrics and humane evaluation. Specifically, we sample 20 20 20 20 game video clips from the test set, and invite 5 5 5 5 experienced game players to score each video clip from the aspect of action following. Each participant was presented with the pair of game states and their corresponding actions, and asked to rate each sample on a scale of 1 to 5. We calculated the average score as the final result. The pearson correlation coefficient is calculated over all samples.

We conduct experiments on the 700 700 700 700 M model with autoregressive decoding, and list the results in Table[3](https://arxiv.org/html/2504.08388v1#S4.T3 "Table 3 ‣ The Metrics on Controllability ‣ 4.2 Analysis ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"), where r 𝑟 r italic_r represents the pearson correlation coefficient and p-value represents the corresponding probability value. As a result, the classification based evaluation metric has a significant positive correlation with human evaluation, showing the correctness of the proposed metrics.

### 4.3 Case Study

![Image 3: Refer to caption](https://arxiv.org/html/x3.png)

Figure 3: Case study on MineWorld 700 700 700 700 M model. The first game state and actions in following steps are provided as input, based on which the model generates consequent game states. For more cases and videos, please visit our project page.

We provide case studies on MineWorld in this section, to demonstrate the controllability and video quality of the model. All results are generated by the 700 700 700 700 M model with autoregressive decoding.

#### General Capability

We show several general capability of MineWorld in Figure[3](https://arxiv.org/html/2504.08388v1#S4.F3 "Figure 3 ‣ 4.3 Case Study ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"), where the initial game state is provided, and the model generates subsequent states based on corresponding actions. In the first example, the actions guide the model to open a door and walk outside. The model successfully generates the door-opening process and accurately renders the unseen outdoor environment. The second example depicts the process of chopping wood, where the model captures fine-grained details, such as the wood’s cross-section and the explosion effect when the chop is complete. In the third case, a house appears in the initial frame, and the camera first pans left and then returns to the right. The model correctly regenerates the same house with detailed fidelity when the camera returns.

These cases show that MineWorld learns foundamental physical knowledge of Minecraft, and can generate accurate reactions when receiving diverse actions. Besides, the strong video generation capacity empowers the model generates high-fidelity, coherent and consistent video results.

#### Controllability

![Image 4: Refer to caption](https://arxiv.org/html/x4.png)

Figure 4: Case study on the controllability of MineWorld. Providing the same initial game state and different actions, the model generates different results correspondingly.

We illustrate the controllability of MineWorld in Figure[4](https://arxiv.org/html/2504.08388v1#S4.F4 "Figure 4 ‣ Controllability ‣ 4.3 Case Study ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"), which shows the different generation results start from the same game state but with different actions. As a result, MineWorld accurately generates responses w.r.t each action.

#### Serve as an Agent

![Image 5: Refer to caption](https://arxiv.org/html/x5.png)

Figure 5: Case study on MineWorld as a gaming playing agent. By providing several initial game states and actions(splited by the dashed line), MineWorld continues to play the game by itself through generating future game states and actions in an iterative manner.

Since MineWorld predicts both game state tokens and action tokens during training, it naturally acquires the capabilities of both a world model and a policy model. This dual functionality enables MineWorld to serve as a self-contained game agent, capable of playing the game autonomously. Specifically, given a few initial game states and actions provided by users, the model can iteratively predict future states and actions, effectively simulating the long-horizon gameplay.

As shown in Figure[5](https://arxiv.org/html/2504.08388v1#S4.F5 "Figure 5 ‣ Serve as an Agent ‣ 4.3 Case Study ‣ 4 Experiments ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"), MineWorld generates diverse and contextually appropriate actions, as well as accurate, high-fidelity game states. This demonstrates its potential not only as an interactive world simulator but also as a foundation for developing game-playing agents.

5 Conclusion and Limitations
----------------------------

We present MineWorld, the first open-source, real-time interactive world model for Minecraft. Using an open dataset of game states and corresponding actions, we tokenize both modalities with separate tokenizers and train an autoregressive Transformer decoder via next-token prediction, using the interleaved sequence of state and action tokens as input. To support real-time interaction, we exploit the redundancy between adjacent image tokens and introduce a parallel decoding algorithm, achieving consistent speedups over standard autoregressive decoding without sacrifacing the generation quality. Through our comprehensive evaluation pipeline, we show that MineWorld achieves not only the strong controllability and video quality, but the fast inference speed of 2∼6 similar-to 2 6 2\sim 6 2 ∼ 6 FPS which enables real-time interaction with professional game players.

#### Limitations

MineWorld is trained exclusively on Minecraft data at a fixed, downsampled resolution, limiting its ability to generalize to other video domains (_e.g_., internet videos) or generate outputs at higher resolutions. The downsampling process may also lead to the loss of fine-grained details in game states. The maximum input length of the model is set to 5.5 5.5 5.5 5.5 k tokens, corresponding to 16 16 16 16 state-action pairs. While MineWorld demonstrates strong temporal consistency within this range, it is not guaranteed when the distance between game states exceeds this range.

References
----------

*   Agarwal et al. (2025) Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. _arXiv preprint arXiv:2501.03575_, 2025. 
*   Bai et al. (2024) Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp. 22861–22872, 2024. 
*   Baker et al. (2022) Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. _Advances in Neural Information Processing Systems_, 35:24639–24654, 2022. 
*   Brooks et al. (2024) Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL [https://openai.com/research/video-generation-models-as-world-simulators](https://openai.com/research/video-generation-models-as-world-simulators). 
*   Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877–1901, 2020. 
*   Bruce et al. (2024) Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In _Forty-first International Conference on Machine Learning_, 2024. 
*   Chang et al. (2022) Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pp. 11315–11325, 2022. 
*   Decart et al. (2024) Decart, Quevedo Julian, McIntyre Quinn, Campbell Spruce, Chen Xinlei, and Wachen Robert. Oasis: A universe in a transformer. 2024. URL [https://oasis-model.github.io/](https://oasis-model.github.io/). 
*   Esser et al. (2021) Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pp. 12873–12883, 2021. 
*   Ha & Schmidhuber (2018) David Ha and Jürgen Schmidhuber. World models. _arXiv preprint arXiv:1803.10122_, 2018. 
*   Hafner et al. (2019) Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. _arXiv preprint arXiv:1912.01603_, 2019. 
*   He et al. (2024) Yefei He, Feng Chen, Yuanyu He, Shaoxuan He, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipar: Accelerating autoregressive image generation through spatial locality. _arXiv preprint arXiv:2412.04062_, 2024. 
*   Hore & Ziou (2010) Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In _2010 20th international conference on pattern recognition_, pp. 2366–2369. IEEE, 2010. 
*   Kanervisto et al. (2025) Anssi Kanervisto, Dave Bignell, Linda Yilin Wen, Martin Grayson, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Tabish Rashid, Tim Pearce, Yuhan Cao, et al. World and human action models towards gameplay ideation. _Nature_, 638(8051):656–663, 2025. 
*   Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. _arXiv preprint arXiv:2001.08361_, 2020. 
*   Kingma & Ba (2015) Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In _International Conference on Learning Representations_, 2015. 
*   Kondratyuk et al. (2023) Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation. _arXiv preprint arXiv:2312.14125_, 2023. 
*   Li et al. (2023) Jiacheng Li, Longhui Wei, ZongYuan Zhan, Xin He, Siliang Tang, Qi Tian, and Yueting Zhuang. Lformer: Text-to-image generation with l-shape block parallel decoding. _arXiv preprint arXiv:2303.03800_, 2023. 
*   Liu et al. (2024) Xuantong Liu, Shaozhe Hao, Xianbiao Qi, Tianyang Hu, Jun Wang, Rong Xiao, and Yuan Yao. Elucidating the design space of language models for image generation. _arXiv preprint arXiv:2410.16257_, 2024. 
*   Parker-Holder et al. (2024) Jack Parker-Holder, Philip Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Christos Kaplanis, Alexandre Moufarek, Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Yung, Michael Dennis, Sultan Kenjeyev, Shangbang Long, Vlad Mnih, Harris Chan, Maxime Gazeau, Bonnie Li, Fabio Pardo, Luyu Wang, Lei Zhang, Frederic Besse, Tim Harley, Anna Mitenkova, Jane Wang, Jeff Clune, Demis Hassabis, Raia Hadsell, Adrian Bolton, Satinder Singh, and Tim Rocktäschel. Genie 2: A large-scale foundation world model. 2024. URL [https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/](https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/). 
*   Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. _Advances in neural information processing systems_, 32, 2019. 
*   Patil et al. (2024) Suraj Patil, William Berman, Robin Rombach, and Patrick von Platen. amused: An open muse reproduction. _arXiv preprint arXiv:2401.01808_, 2024. 
*   Su et al. (2024) Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. _Neurocomputing_, 568:127063, 2024. 
*   Tang et al. (2024) Anni Tang, Tianyu He, Junliang Guo, Xinle Cheng, Li Song, and Jiang Bian. Vidtok: A versatile and open-source video tokenizer. _arXiv preprint arXiv:2412.13061_, 2024. 
*   Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_, 2023. 
*   Unterthiner et al. (2018) Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. _arXiv preprint arXiv:1812.01717_, 2018. 
*   Valevski et al. (2024) Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. In _The Thirteenth International Conference on Learning Representations_, 2024. 
*   Van Den Oord et al. (2017) Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. _Advances in neural information processing systems_, 30, 2017. 
*   Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. _Advances in neural information processing systems_, 30, 2017. 
*   Wang et al. (2024) Yuchi Wang, Junliang Guo, Xinyi Xie, Tianyu He, Xu Sun, and Jiang Bian. Vidtwin: Video vae with decoupled structure and dynamics. _arXiv preprint arXiv:2412.17726_, 2024. 
*   Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. _IEEE transactions on image processing_, 13(4):600–612, 2004. 
*   Wikipedia contributors (2025) Wikipedia contributors. Actions per minute — Wikipedia, The Free Encyclopedia. [https://en.wikipedia.org/wiki/Actions_per_minute](https://en.wikipedia.org/wiki/Actions_per_minute), 2025. [Online; accessed 25-February-2025]. 
*   Wu et al. (2024) Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. _arXiv preprint arXiv:2405.15223_, 2024. 
*   Yang et al. (2023) Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. _arXiv preprint arXiv:2310.06114_, 1(2):6, 2023. 
*   Yang et al. (2024) Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. _arXiv preprint arXiv:2408.06072_, 2024. 
*   Ye et al. (2025) Yang Ye, Junliang Guo, Haoyu Wu, Tianyu He, Tim Pearce, Tabish Rashid, Katja Hofmann, and Jiang Bian. Fast autoregressive video generation with diagonal decoding. _arXiv preprint arXiv:2503.14070_, 2025. 
*   Yu et al. (2023) Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. _arXiv preprint arXiv:2310.05737_, 2023. 
*   Yu et al. (2024) Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Randomized autoregressive visual generation. _arXiv preprint arXiv:2411.00776_, 2024. 
*   Zhang & Sennrich (2019) Biao Zhang and Rico Sennrich. Root mean square layer normalization. _Advances in Neural Information Processing Systems_, 32, 2019. 
*   Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pp. 586–595, 2018. 
*   Zhang et al. (2025) Wentao Zhang, Junliang Guo, Tianyu He, Li Zhao, Linli Xu, and Jiang Bian. Video in-context learning: Autoregressive transformers are zero-shot video imitators. In _The Thirteenth International Conference on Learning Representations_, 2025. 

Appendix A Minecraft Action Space
---------------------------------

### A.1 Details of Action Following Metric

Table 4: Classification Tasks and Their Labels

Task Type Actions Labels
Triple Classification forward, backward forward, backward, null
left, right left, right, null
sprint, sneak sprint, sneak, null
Binary Classification use use, null
attack attack, null
jump jump, null
drop drop, null

The action space in Minecraft is inherently complex. To effectively validate the quality of action execution, we simplify the scenario by focusing on the 10 most common actions and camera movements. In addition, treating action prediction as a simple multi-class classification task does not fully capture the complexity of predicting sequential actions. To address this, we divide the 10 10 10 10 actions into 3 3 3 3 triple classification tasks and 4 4 4 4 binary classification tasks, based on the exclusion relationships between the actions. In cases where the IDM model predicts two mutually exclusive actions at the same frame, we classify it as ”no action” in line with common game behavior. Finally, we compute average macro precision, recall and F1 scores over all tasks.

### A.2 Task-Specific Action Accuracy

Table 5: Precision, Recall, and F1 scores for the 700 MineWorld model across different tasks.

Metric Forward-Backward Left-Right Sprint-Sneak Use Attack Jump Drop
Precision 0.807 0.808 0.765 0.742 0.729 0.760 0.500
Recall 0.773 0.745 0.788 0.648 0.749 0.847 0.498
F1 0.782 0.771 0.750 0.682 0.736 0.796 0.499

After defining the sub-classification tasks in Table [4](https://arxiv.org/html/2504.08388v1#A1.T4 "Table 4 ‣ A.1 Details of Action Following Metric ‣ Appendix A Minecraft Action Space ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"), we can analyze the model’s performance in more detail with respect to action prediction. The table [5](https://arxiv.org/html/2504.08388v1#A1.T5 "Table 5 ‣ A.2 Task-Specific Action Accuracy ‣ Appendix A Minecraft Action Space ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft") below shows the precision, recall, and F1 score for each task in our 700M MineWorld model. The results indicate that the ”drop” action performs significantly worse than the other actions, suggesting that it is more challenging for the model to learn.

Appendix B Model Configurations
-------------------------------

Table 6: The configuration of different size of models.

Hidden dim MLP dim Num. Heads Num. Layers
300M 1024 4096 16 20
700M 2048 4096 32 20
1.2B 2048 8192 32 20

Table 7: Optimization hyperparameters.

Hyperparameter Value
Learning rate scheduler cosine
Learning rate 3⁢e−4 3 superscript 𝑒 4 3e^{-4}3 italic_e start_POSTSUPERSCRIPT - 4 end_POSTSUPERSCRIPT
Warm up steps 10000
Weight decay 0.1
Optimizer AdamW
AdamW betas(0.9,0.95)0.9 0.95(0.9,0.95)( 0.9 , 0.95 )
Maximum Positions 5376 5376 5376 5376

To validate the scaling behavior of the Transformer decoder, we train three different sizes of the model within the LLaMA architecture: 300 300 300 300 M, 700 700 700 700 M, and 1.2 1.2 1.2 1.2 B. We tune the hidden dimension, intermediate dimension, and the number of layers to achieve different model sizes. The configuration of these models are listed in Table[6](https://arxiv.org/html/2504.08388v1#A2.T6 "Table 6 ‣ Appendix B Model Configurations ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft"). The hyperparameters of the optimizer used to train the model are listed in Table[7](https://arxiv.org/html/2504.08388v1#A2.T7 "Table 7 ‣ Appendix B Model Configurations ‣ MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft").

Appendix C More Results
-----------------------

### C.1 Reconstruction Results of Visual Tokenizer

Table 8: The reconstruction performance of the visual tokenizers on the validation set.

Visual Tokenizer PSNR↑↑\uparrow↑SSIM↑↑\uparrow↑LPIPS↓↓\downarrow↓rFID↓↓\downarrow↓
Amused(Patil et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib22))25.91 0.758 0.238 35.05
Ours 29.24 0.816 0.134 18.93

We evaluate the reconstruction performance of the pre-trained Amused VQ-VAE(Patil et al., [2024](https://arxiv.org/html/2504.08388v1#bib.bib22)) and the one after fine-tuned on the pre-processed VPT data. After fine-tuning, the performance is significantly improved, showing the necessity of this step.

Generated on Fri Apr 11 09:36:06 2025 by [L a T e XML![Image 6: Mascot Sammy](blob:http://localhost/70e087b9e50c3aa663763c3075b0d6c5)](http://dlmf.nist.gov/LaTeXML/)