Title: Trajectory Attention for Fine-grained Video Motion Control

URL Source: https://arxiv.org/html/2411.19324

Published Time: Mon, 02 Dec 2024 01:49:21 GMT

Markdown Content:
Zeqi Xiao 1, Wenqi Ouyang 1, Yifan Zhou 1, 

Shuai Yang 2, Lei Yang 3, Jianlou Si 3, Xingang Pan 1

1 S-Lab, Nanyang Technological University, 

2 Wangxuan Institute of Computer Technology, Peking University 

3 Sensetime Research 

{zeqi001, yifan006, wenqi.ouyang, xingang.pan}@ntu.edu.sg 

williamyang@pku.edu.cn 

{jianlousi,leiyang}@sensetime.com

###### Abstract

Recent advancements in video generation have been greatly driven by video diffusion models, with camera motion control emerging as a crucial challenge in creating view-customized visual content. This paper introduces trajectory attention, a novel approach that performs attention along available pixel trajectories for fine-grained camera motion control. Unlike existing methods that often yield imprecise outputs or neglect temporal correlations, our approach possesses a stronger inductive bias that seamlessly injects trajectory information into the video generation process. Importantly, our approach models trajectory attention as an auxiliary branch alongside traditional temporal attention. This design enables the original temporal attention and the trajectory attention to work in synergy, ensuring both precise motion control and new content generation capability, which is critical when the trajectory is only partially available. Experiments on camera motion control for images and videos demonstrate significant improvements in precision and long-range consistency while maintaining high-quality generation. Furthermore, we show that our approach can be extended to other video motion control tasks, such as first-frame-guided video editing, where it excels in maintaining content consistency over large spatial and temporal ranges.

††footnotetext: Project page at this [URL](https://xizaoqu.github.io/trajattn/).
1 Introduction
--------------

Video generation has experienced remarkable advancements in recent years, driven by sophisticated deep learning models such as video diffusion models and temporal attention mechanisms (OpenAI, [2024](https://arxiv.org/html/2411.19324v1#bib.bib30); Chen et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib4); Wang et al., [2023a](https://arxiv.org/html/2411.19324v1#bib.bib38); Guo et al., [2023b](https://arxiv.org/html/2411.19324v1#bib.bib14)). These innovations have enabled the synthesis of increasingly realistic videos, fueling fields in areas such as filmmaking (Zhao et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib58); Zhuang et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib61)) and world modeling (OpenAI, [2024](https://arxiv.org/html/2411.19324v1#bib.bib30); Valevski et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib36)). Video motion control, which aims to produce customized motion in video generation, has emerged as a crucial aspect (Yang et al., [2023b](https://arxiv.org/html/2411.19324v1#bib.bib48); Ling et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib27); Ouyang et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib31); Ku et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib23); Zhao et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib58)).

Among various control signals, camera motion control has garnered increasing attention due to its wide applications in creating view-customized visual content. However, effectively conditioning generation results on given camera trajectories remains non-trivial. Researchers have explored several approaches to address this challenge. One method involves encoding camera parameters into embeddings and injecting them into the model via cross-attention or addition (Wang et al., [2024c](https://arxiv.org/html/2411.19324v1#bib.bib41); He et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib15); Bahmani et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib1)). While straightforward, this approach often yields imprecise and ambiguous outputs due to the high-level constraints and implicit control mechanisms it employs. Another strategy involves rendering partial frames based on camera trajectories and using these either as direct input (Hu et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib20); Yu et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib54)) or as optimization targets (You et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib52)) for frame-wise conditioning. Although this method provides more explicit control, it often neglects temporal correlations across frames, leading to inconsistencies in the generated sequence.

In response to these limitations, recent methods have begun to address temporal relationships by leveraging 3D inductive biases (Xu et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib44); Li et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib26)). These approaches focus on narrowed domains, utilizing specific settings such as row-wise attention (Li et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib26)) or epipolar constraint attention (Xu et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib44)). As we consider the trajectory of a camera moving around scenes, it becomes apparent that certain parts of the moving trajectories of pixels, represented as a sequence of 2D coordinates across frames, are predictable due to 3D consistency constraints. This observation raises an intriguing question: can we exploit such trajectories as a strong inductive bias to achieve more fine-grained motion control?

Revisiting the temporal attention mechanism, which is central to video models for synthesizing dynamic motions with consistent content, we can view the dynamics as pixel trajectories across frames. The temporal attention mechanism, with its generic attention design, functions by implicitly synthesizing and attending to these trajectories. Building on this observation, when parts of the trajectories are available, the attention along these trajectories can be modeled explicitly as a strong inductive bias to produce controlled motion with consistent content.

To this end, we propose trajectory attention that performs attention along the available trajectories across frames for fine-grained camera motion control. Instead of directly adapting the temporal attention to operate on trajectories, which yields suboptimal results in practice, we model trajectory attention as an auxiliary branch alongside the original temporal attention. This design is critical due to the distinct goals of these two attention mechanisms. Temporal attention, which must balance motion synthesis and content consistency, typically focuses on short-range dynamics and attends to adjacent frames within a local window. In contrast, trajectory attention is designed to ensure long-range consistency across features along a trajectory (see Fig.[2](https://arxiv.org/html/2411.19324v1#S3.F2 "Figure 2 ‣ 3.1 Preliminary ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control")). The trajectory attention branch can inherit the parameters of the original temporal attention for efficient tuning, and its output is added to the output of temporal attention as residuals. This whole design offers several merits: 1) it allows better division of tasks: trajectory attention manages motion control and ensures long-range consistency along specified paths, while temporal attention synthesizes motion for the rest regions; 2) it can integrate seamlessly without modifying the original parameters; 3) it supports sparse trajectories, as the condition is injected moderately, meaning available trajectories do not have to cover all pixels.

Our experiments on camera motion control for images and videos demonstrate that our designs significantly enhance precision and long-range consistency. As shown in Fig. [1](https://arxiv.org/html/2411.19324v1#S1.F1 "Figure 1 ‣ 1 Introduction ‣ Trajectory Attention for Fine-grained Video Motion Control"), our approach leverages a stronger inductive bias that optimizes the attention mechanism. This results in improved control precision while maintaining high-quality generation. The proposed trajectory attention can be extended to other video motion control tasks, such as first-frame-guided video editing. Existing techniques often struggle to maintain content consistency over large spatial and temporal ranges (Ku et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib23); Ouyang et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib31)). In contrast, our method’s ability to model long-range, consistent correspondences achieves promising results in these challenging scenarios. Moreover, the efficiency of our design allows for training with limited data and computational resources, making it generalizable to diverse contexts and frame ranges.

![Image 1: Refer to caption](https://arxiv.org/html/2411.19324v1/x1.png)

Figure 1: Trajectory attention injects partial motion information by making content along trajectories consistent. It facilitates various tasks such as camera motion control on images and videos, and first-frame-guided video editing. Yellow boxes indicate reference contents. Green boxes indicate input frames. Blue boxes indicate output frames.

2 Related Works
---------------

Video Diffusion Models. The field of video generation has seen significant advancements in recent years, especially in the area of video diffusion models (Ho et al., [2022](https://arxiv.org/html/2411.19324v1#bib.bib17); Guo et al., [2023b](https://arxiv.org/html/2411.19324v1#bib.bib14); Chen et al., [2023a](https://arxiv.org/html/2411.19324v1#bib.bib3); Wang et al., [2023b](https://arxiv.org/html/2411.19324v1#bib.bib40); [a](https://arxiv.org/html/2411.19324v1#bib.bib38); OpenAI, [2024](https://arxiv.org/html/2411.19324v1#bib.bib30); Blattmann et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib2); Guo et al., [2023a](https://arxiv.org/html/2411.19324v1#bib.bib13); Chen et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib4); Hong et al., [2022](https://arxiv.org/html/2411.19324v1#bib.bib18)).

The core of motion modeling of video diffusion models is the temporal attention module. Some approaches (Guo et al., [2023b](https://arxiv.org/html/2411.19324v1#bib.bib14); Chen et al., [2023a](https://arxiv.org/html/2411.19324v1#bib.bib3); Wang et al., [2023b](https://arxiv.org/html/2411.19324v1#bib.bib40); [a](https://arxiv.org/html/2411.19324v1#bib.bib38)) decompose attention into spatial and temporal components, where temporal attention aligns features across different frames. Others (Yang et al., [2024c](https://arxiv.org/html/2411.19324v1#bib.bib49); OpenAI, [2024](https://arxiv.org/html/2411.19324v1#bib.bib30); Lab & etc., [2024](https://arxiv.org/html/2411.19324v1#bib.bib25)) integrate spatial and temporal attention into a unified mechanism, capturing both types of information simultaneously. While these methods rely on data-driven techniques to implicitly learn dynamic video priors within the attention mechanism, how to leverage such priors for explicit and precise motion control remains under-explored.

Motion Control in Video Generation. Prior works have explored various control signals for video motion control (Guo et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib12); Niu et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib29); Yu et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib53); Chen et al., [2023b](https://arxiv.org/html/2411.19324v1#bib.bib5); Yang et al., [2024b](https://arxiv.org/html/2411.19324v1#bib.bib47); Zuo et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib62); Zhu et al., [2024a](https://arxiv.org/html/2411.19324v1#bib.bib59); Zhao et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib58); Chen et al., [2023c](https://arxiv.org/html/2411.19324v1#bib.bib6); Zhang et al., [2023b](https://arxiv.org/html/2411.19324v1#bib.bib56)), including sketches (Wang et al., [2024b](https://arxiv.org/html/2411.19324v1#bib.bib39)), depth maps (Wang et al., [2024b](https://arxiv.org/html/2411.19324v1#bib.bib39)), drag vectors (Yin et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib51); Teng et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib35); Deng et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib8)), human pose (Zhang et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib57); Zhu et al., [2024b](https://arxiv.org/html/2411.19324v1#bib.bib60)), object trajectory (Qiu et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib33); Wang et al., [2024a](https://arxiv.org/html/2411.19324v1#bib.bib37); Wu et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib42); Gu et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib11)), and features extracted from reference videos (Yatim et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib50); Xiao et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib43); Yang et al., [2023b](https://arxiv.org/html/2411.19324v1#bib.bib48); Ouyang et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib31); Ku et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib23)).

One important branch of video motion control is camera motion control, also known as novel view synthesis. In this regard, Wang et al. ([2024c](https://arxiv.org/html/2411.19324v1#bib.bib41)); He et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib15)); Bahmani et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib1)); Wu et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib42)) utilize high-level condition signals by encoding camera pose parameters into conditional features. However, these methods often lack precision in capturing detailed temporal dynamics, as they impose weak constraints on the resulting motion. Hou et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib19)) enables camera control by rendering incomplete warped views followed by re-denoising.Müller et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib28)); Yu et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib54)); You et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib52)) render partial videos as guidance and leverage video generation models to inpaint the remaining frames. Despite these innovations, their approaches suffer from temporal inconsistency due to the lack of consideration for sequential coherence. Methods such as those proposed by Shi et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib34)); Xu et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib44)); Cong et al. ([2023](https://arxiv.org/html/2411.19324v1#bib.bib7)); Kuang et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib24)) explicitly modify attention using optical flow or epipolar constraints. These solutions can be viewed as a weaker variant of trajectory-consistent constraint. Our approach introduces a plug-and-play trajectory attention mechanism for motion information injection. Thanks to the strong inductive bias that makes the best use of the attention mechanism, our method offers precise control over video generation, improving efficiency without the need for specially annotated datasets (like camera pose annotations). It enables enhanced motion control throughout the generation process while maintaining the fidelity of temporal dynamics.

3 Methodology
-------------

This section introduces trajectory attention for fine-grained motion control. We first outline video diffusion models with a focus on temporal attention (Sec. [3.1](https://arxiv.org/html/2411.19324v1#S3.SS1 "3.1 Preliminary ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control")), then adapt it for trajectory attention and discuss its limitations (Sec. [3.2](https://arxiv.org/html/2411.19324v1#S3.SS2 "3.2 Taming temporal attention for trajectory attention ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control")). We present trajectory attention as an additional branch, with visualizations of its effectiveness (Sec. [3.3](https://arxiv.org/html/2411.19324v1#S3.SS3 "3.3 Modeling trajectory attention as an auxiliary branch ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control")), and describe an efficient training pipeline (Sec. [3.4](https://arxiv.org/html/2411.19324v1#S3.SS4 "3.4 Training trajectory attention efficiently ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control")).

### 3.1 Preliminary

The core of video motion modeling lies in the temporal attention mechanism within video diffusion models, whether applied through decomposed spatial and temporal attention or full 3D attention, to capture robust motion priors. This paper demonstrates the decomposed setting, which is more widely used and has greater open-source availability. However, our design is also adaptable to full 3D attention, as will shown in the experimental results and appendix.

A typical video diffusion architecture for decomposed spatial-temporal attention includes convolutional layers, spatial attention blocks, and temporal attention blocks. The temporal attention operates as follows. Given an input latent feature Z∈ℝ F×H×W×C Z superscript ℝ 𝐹 𝐻 𝑊 𝐶\textbf{Z}\in\mathbb{R}^{F\times H\times W\times C}Z ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_H × italic_W × italic_C end_POSTSUPERSCRIPT, where F 𝐹 F italic_F, H 𝐻 H italic_H, W 𝑊 W italic_W, and C 𝐶 C italic_C represent the number of frames, height, width, and channels, respectively, temporal attention operates along the frame dimension. The feature Z is first projected into query (Q), key (K), and value (V):

Q=p q⁢(Z),K=p k⁢(Z),V=p v⁢(Z),formulae-sequence Q subscript 𝑝 𝑞 Z formulae-sequence K subscript 𝑝 𝑘 Z V subscript 𝑝 𝑣 Z\textbf{Q}=p_{q}(\textbf{Z}),\textbf{K}=p_{k}(\textbf{Z}),\textbf{V}=p_{v}(% \textbf{Z}),Q = italic_p start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT ( Z ) , K = italic_p start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ( Z ) , V = italic_p start_POSTSUBSCRIPT italic_v end_POSTSUBSCRIPT ( Z ) ,(1)

where p q subscript 𝑝 𝑞 p_{q}italic_p start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT, p k subscript 𝑝 𝑘 p_{k}italic_p start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT, and p v subscript 𝑝 𝑣 p_{v}italic_p start_POSTSUBSCRIPT italic_v end_POSTSUBSCRIPT are learnable projection functions. Temporal attention is then applied along the frame dimension as:

Z′=Softmax⁢(Q K T)⁢V,superscript Z′Softmax superscript Q K 𝑇 V\textbf{Z}^{\prime}=\text{Softmax}(\textbf{Q}\textbf{K}^{T})\textbf{V},Z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = Softmax ( bold_Q bold_K start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT ) V ,(2)

yielding the output latent feature Z′superscript Z′\textbf{Z}^{\prime}Z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT. For simplicity, we omit the details like rescaling factor and multi-head operations. With large-scale training, temporal attention effectively captures dynamic and consistent video motions, making it a natural candidate for motion control in video models.

![Image 2: Refer to caption](https://arxiv.org/html/2411.19324v1/x2.png)

Figure 2: Attention map visualization of temporal attention and trajectory attention. (a) Temporal attention tends to concentrate its weight on a narrow, adjacent frame window. (b) In contrast, trajectory attention exhibits a broader attention window, highlighting its capacity to produce more consistent and controllable results. Here, the attention map is structured with the frame number as the side length. The attention weights are normalized within the range of 0 to 1, where higher values (indicated by light yellow) represent stronger attention.

![Image 3: Refer to caption](https://arxiv.org/html/2411.19324v1/x3.png)

Figure 3: Overview of the proposed motion control pipeline. Our method allows for conditioning on trajectories from various sources – such as camera motion derived from a single image, as shown in this figure. We inject these conditions into the model through trajectory attention, enabling explicit and fine-grained control over the motion in the generated video.

### 3.2 Taming temporal attention for trajectory attention

As shown in Fig. [4](https://arxiv.org/html/2411.19324v1#S3.F4 "Figure 4 ‣ 3.2 Taming temporal attention for trajectory attention ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control"), vanilla temporal attention operates on the same spatial position across different frames, where the coordinates in the attention form predefined trajectories across frames.

Since temporal attention has already learned to model motion along pre-defined trajectories, a natural extension is to tame temporal attention for additional trajectory attention. For example, given a set of trajectories Tr, where each trajectory is represented by a series of coordinates, we incorporate them into the temporal attention mechanism.

However, this straighwarpward adaptation often yields suboptimal results due to a conflict between temporal and trajectory attention. Temporal attention is designed to ensure consistency along the trajectory while preserving the dynamism of feature representations. However, achieving both perfectly is challenging. Consequently, temporal attention often prioritizes natural dynamics at the expense of long-range consistency. This is evident in the attention statistics: as shown in Fig. [2](https://arxiv.org/html/2411.19324v1#S3.F2 "Figure 2 ‣ 3.1 Preliminary ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control")(a), the learned temporal attention predominantly focuses on adjacent frames.

![Image 4: Refer to caption](https://arxiv.org/html/2411.19324v1/x4.png)

Figure 4: Visualization of vanilla temporal attention and trajectory attention.

In contrast, trajectory attention, given its known dynamics, aims solely to align features along the trajectory. This singular focus on alignment often clashes with the broader objectives of temporal attention. Simply adapting temporal attention to accommodate trajectory information can therefore introduce conflicts. Experimental results further demonstrate that, even with extensive training, the quality of motion control remains suboptimal when trajectory attention is naively integrated.

### 3.3 Modeling trajectory attention as an auxiliary branch

The above analysis reveals that temporal attention and trajectory attention should not share the same set of weights. Inspired by the recent success of Zhang et al. ([2023a](https://arxiv.org/html/2411.19324v1#bib.bib55)), we model temporal attention and trajectory attention into a two-branch structure, where trajectory attention is responsible for injecting fine-grained trajectory consistent signal to the origin generation process.

As illustrated in Fig.[3](https://arxiv.org/html/2411.19324v1#S3.F3 "Figure 3 ‣ 3.1 Preliminary ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control"), trajectory attention and temporal attention share the same structure, as well as identical input and output shapes. The key difference lies in the process: we first use the given trajectories to sample features from the hidden states (Algorithm [1](https://arxiv.org/html/2411.19324v1#algorithm1 "In 3.3 Modeling trajectory attention as an auxiliary branch ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control")), then apply multi-head attention with distinct parameters, and finally project the results back to the hidden state format after frame-wise attention (Algorithm [2](https://arxiv.org/html/2411.19324v1#algorithm2 "In 3.3 Modeling trajectory attention as an auxiliary branch ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control")).

To validate the purpose distinction, we compare the attention maps (softmax scores along the frame axis) of temporal and trajectory attention, based on the SVD model (Blattmann et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib2)). As shown in Fig. [2](https://arxiv.org/html/2411.19324v1#S3.F2 "Figure 2 ‣ 3.1 Preliminary ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control")(a) and (b), trajectory attention clearly provides a broader attention window, enabling more consistent and controllable results.

Input:Hidden states

Z∈ℝ F×H×W×C Z superscript ℝ 𝐹 𝐻 𝑊 𝐶\textbf{Z}\in\mathbb{R}^{F\times H\times W\times C}Z ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_H × italic_W × italic_C end_POSTSUPERSCRIPT
, where

F 𝐹 F italic_F
is the number of frames,

H,W 𝐻 𝑊 H,W italic_H , italic_W
are the spatial dimensions, and

C 𝐶 C italic_C
is the number of channels.

L 𝐿 L italic_L
trajectories

Tr∈ℝ L×F×2 Tr superscript ℝ 𝐿 𝐹 2\textbf{Tr}\in\mathbb{R}^{L\times F\times 2}Tr ∈ blackboard_R start_POSTSUPERSCRIPT italic_L × italic_F × 2 end_POSTSUPERSCRIPT
, where each trajectory specifies

F 𝐹 F italic_F
2D locations. Trajectory masks

M∈ℝ F×L M superscript ℝ 𝐹 𝐿\textbf{M}\in\mathbb{R}^{F\times L}M ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_L end_POSTSUPERSCRIPT
, where

M f,l∈{0,1}subscript 𝑀 𝑓 𝑙 0 1 M_{f,l}\in\{0,1\}italic_M start_POSTSUBSCRIPT italic_f , italic_l end_POSTSUBSCRIPT ∈ { 0 , 1 }
indicates whether a trajectory is valid at frame

f 𝑓 f italic_f
for trajectory

l 𝑙 l italic_l
.

1

2 foreach _trajectory i=1,…,L 𝑖 1…𝐿 i=1,\dots,L italic\_i = 1 , … , italic\_L_ do

3 Sample hidden states

Z i={Z f⁢(x f,i,y f,i)∣f=1,…,F}∈ℝ F×C subscript Z 𝑖 conditional-set subscript Z 𝑓 subscript 𝑥 𝑓 𝑖 subscript 𝑦 𝑓 𝑖 𝑓 1…𝐹 superscript ℝ 𝐹 𝐶\textbf{Z}_{i}=\{\textbf{Z}_{f}(x_{f,i},y_{f,i})\mid f=1,\dots,F\}\in\mathbb{R% }^{F\times C}Z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = { Z start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT ( italic_x start_POSTSUBSCRIPT italic_f , italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_f , italic_i end_POSTSUBSCRIPT ) ∣ italic_f = 1 , … , italic_F } ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_C end_POSTSUPERSCRIPT

4 where

(x f,i,y f,i)subscript 𝑥 𝑓 𝑖 subscript 𝑦 𝑓 𝑖(x_{f,i},y_{f,i})( italic_x start_POSTSUBSCRIPT italic_f , italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_f , italic_i end_POSTSUBSCRIPT )
are the 2D coordinates from

Tr⁢[i]Tr delimited-[]𝑖\textbf{Tr}[i]Tr [ italic_i ]
for each frame

f 𝑓 f italic_f
.

5 end foreach

6

7 Stack sampled hidden states:

Z s=Stack⁢(Z i∣i=1,…,L)∈ℝ F×L×C subscript Z 𝑠 Stack conditional subscript Z 𝑖 𝑖 1…𝐿 superscript ℝ 𝐹 𝐿 𝐶\textbf{Z}_{s}=\text{Stack}(\textbf{Z}_{i}\mid i=1,\dots,L)\in\mathbb{R}^{F% \times L\times C}Z start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT = Stack ( Z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∣ italic_i = 1 , … , italic_L ) ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_L × italic_C end_POSTSUPERSCRIPT

8 Mask out invalid hidden states using M:

Z t=Z s⊙M subscript Z 𝑡 direct-product subscript Z 𝑠 M\textbf{Z}_{t}=\textbf{Z}_{s}\odot\textbf{M}Z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = Z start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ⊙ M

Output:Masked sampled hidden states

Z t∈ℝ F×L×C subscript Z 𝑡 superscript ℝ 𝐹 𝐿 𝐶\textbf{Z}_{t}\in\mathbb{R}^{F\times L\times C}Z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_L × italic_C end_POSTSUPERSCRIPT

Algorithm 1 Trajectory-based sampling

Input:Hidden states after attention

Z t′∈ℝ F×L×C subscript superscript Z′𝑡 superscript ℝ 𝐹 𝐿 𝐶\textbf{Z}^{\prime}_{t}\in\mathbb{R}^{F\times L\times C}Z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_L × italic_C end_POSTSUPERSCRIPT
.

L 𝐿 L italic_L
trajectories

Tr∈ℝ L×F×2 Tr superscript ℝ 𝐿 𝐹 2\textbf{Tr}\in\mathbb{R}^{L\times F\times 2}Tr ∈ blackboard_R start_POSTSUPERSCRIPT italic_L × italic_F × 2 end_POSTSUPERSCRIPT
. Trajectory masks

M∈ℝ F×L M superscript ℝ 𝐹 𝐿\textbf{M}\in\mathbb{R}^{F\times L}M ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_L end_POSTSUPERSCRIPT
.

1

2 Initialize:

Z p∈ℝ F×H×W×C,U∈ℝ F×H×W,Z p=𝟎,U=𝟎 formulae-sequence subscript Z 𝑝 superscript ℝ 𝐹 𝐻 𝑊 𝐶 formulae-sequence U superscript ℝ 𝐹 𝐻 𝑊 formulae-sequence subscript Z 𝑝 0 U 0\textbf{Z}_{p}\in\mathbb{R}^{F\times H\times W\times C},\quad\textbf{U}\in% \mathbb{R}^{F\times H\times W},\quad\textbf{Z}_{p}=\mathbf{0},\quad\textbf{U}=% \mathbf{0}Z start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_H × italic_W × italic_C end_POSTSUPERSCRIPT , U ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_H × italic_W end_POSTSUPERSCRIPT , Z start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = bold_0 , U = bold_0

where

H 𝐻 H italic_H
and

W 𝑊 W italic_W
are the height and width of the spatial grid.

3 foreach _i=1,…,L 𝑖 1…𝐿 i=1,\dots,L italic\_i = 1 , … , italic\_L_ do

4 Add

Z t′⁢[i]∈ℝ F×C subscript superscript Z′𝑡 delimited-[]𝑖 superscript ℝ 𝐹 𝐶\textbf{Z}^{\prime}_{t}[i]\in\mathbb{R}^{F\times C}Z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT [ italic_i ] ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_C end_POSTSUPERSCRIPT
to

Z p subscript Z 𝑝\textbf{Z}_{p}Z start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT
at locations

(x f,i,y f,i)subscript 𝑥 𝑓 𝑖 subscript 𝑦 𝑓 𝑖(x_{f,i},y_{f,i})( italic_x start_POSTSUBSCRIPT italic_f , italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_f , italic_i end_POSTSUBSCRIPT )
from

Tr⁢[i]Tr delimited-[]𝑖\textbf{Tr}[i]Tr [ italic_i ]
:

Z p⁢(f,x f,i,y f,i,:)⁢+=Z t′⁢[i]⁢(f,:)subscript Z 𝑝 𝑓 subscript 𝑥 𝑓 𝑖 subscript 𝑦 𝑓 𝑖:subscript superscript+=Z′𝑡 delimited-[]𝑖 𝑓:\textbf{Z}_{p}(f,x_{f,i},y_{f,i},:)\text{+=}\textbf{Z}^{\prime}_{t}[i](f,:)Z start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_f , italic_x start_POSTSUBSCRIPT italic_f , italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_f , italic_i end_POSTSUBSCRIPT , : ) += bold_Z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT [ italic_i ] ( italic_f , : )

5 Update count table U at the same locations:

U⁢(f,x f,i,y f,i)⁢+=M⁢[f,i]U 𝑓 subscript 𝑥 𝑓 𝑖 subscript 𝑦 𝑓 𝑖+=M 𝑓 𝑖\textbf{U}(f,x_{f,i},y_{f,i})\text{+=}\textbf{M}[f,i]U ( italic_f , italic_x start_POSTSUBSCRIPT italic_f , italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_f , italic_i end_POSTSUBSCRIPT ) += bold_M [ italic_f , italic_i ]

6 end foreach

7

8 Normalize

Z p subscript Z 𝑝\textbf{Z}_{p}Z start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT
element-wise for valid positions (

U>0 U 0\textbf{U}>0 U > 0
):

Z p⁢(f,x,y,:)=Z p⁢(f,x,y,:)U⁢(f,x,y)for all(f,x,y)where U⁢(f,x,y)>0 formulae-sequence subscript Z 𝑝 𝑓 𝑥 𝑦:subscript Z 𝑝 𝑓 𝑥 𝑦:U 𝑓 𝑥 𝑦 for all 𝑓 𝑥 𝑦 where U 𝑓 𝑥 𝑦 0\textbf{Z}_{p}(f,x,y,:)=\frac{\textbf{Z}_{p}(f,x,y,:)}{\textbf{U}(f,x,y)}\quad% \text{for all}\quad(f,x,y)\quad\text{where}\quad\textbf{U}(f,x,y)>0 Z start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_f , italic_x , italic_y , : ) = divide start_ARG Z start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_f , italic_x , italic_y , : ) end_ARG start_ARG U ( italic_f , italic_x , italic_y ) end_ARG for all ( italic_f , italic_x , italic_y ) where U ( italic_f , italic_x , italic_y ) > 0

Output:Back-projected hidden states

Z p∈ℝ F×H×W×C subscript Z 𝑝 superscript ℝ 𝐹 𝐻 𝑊 𝐶\textbf{Z}_{p}\in\mathbb{R}^{F\times H\times W\times C}Z start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_H × italic_W × italic_C end_POSTSUPERSCRIPT

Algorithm 2 Back projection

### 3.4 Training trajectory attention efficiently

As illustrated in Fig.[5](https://arxiv.org/html/2411.19324v1#S3.F5 "Figure 5 ‣ 3.4 Training trajectory attention efficiently ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control"), we initialize the weights of the QKV projectors with those from temporal attention layers to harness the motion modeling capabilities learned from large-scale data. Additionally, the output projector is initialized with zero weights to ensure a gradual training process.

The training objective follows the standard approach used in fundamental generation models. For instance, in the case of Stable Video Diffusion (Blattmann et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib2)), the objective is:

𝔼⁢[‖D θ⁢(x 0+n;σ,c)−x 0‖2 2],𝔼 delimited-[]subscript superscript norm subscript 𝐷 𝜃 subscript x 0 n 𝜎 c subscript x 0 2 2\mathbb{E}[||D_{\theta}(\textbf{x}_{0}+\textbf{n};\sigma,\textbf{c})-\textbf{x% }_{0}||^{2}_{2}],blackboard_E [ | | italic_D start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + n ; italic_σ , c ) - x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT | | start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ] ,(3)

where D θ subscript 𝐷 𝜃 D_{\theta}italic_D start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT represents the neural network, x 0 subscript x 0\textbf{x}_{0}x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT denotes the latent features of the target videos, n is the noise, c is the condition signal, and σ 𝜎\sigma italic_σ is the variance parameter.

![Image 5: Refer to caption](https://arxiv.org/html/2411.19324v1/x5.png)

Figure 5: Training strategy for trajectory attention. To leverage the motion modeling capability learned from large-scale data, we initialize the weights of the QKV projectors with those from temporal attention layers. Additionally, the output projector is initialized with zero weights to ensure a smooth and gradual training process.

4 Fine-grained Control of Video Generation
------------------------------------------

This section delves into the process of extracting trajectories for different task settings. While our primary focus is on camera motion control for both static images and dynamic video content, we also showcase the process of trajectory extraction for video editing.

### 4.1 Camera Motion Control on Images

Algorithm [3](https://arxiv.org/html/2411.19324v1#algorithm3 "In 4.1 Camera Motion Control on Images ‣ 4 Fine-grained Control of Video Generation ‣ Trajectory Attention for Fine-grained Video Motion Control") outlines the process of extracting trajectories, denoted as Tr, along with the corresponding validity mask M from a single image. Unlike prior approaches (Wang et al., [2024c](https://arxiv.org/html/2411.19324v1#bib.bib41); He et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib15)), which rely on high-level control signals for video manipulation, our method explicitly models camera motion as trajectories across frames. This enables precise and accurate control of camera movement.

Input:Image

I∈ℝ H p×W p×3 I superscript ℝ subscript 𝐻 𝑝 subscript 𝑊 𝑝 3\textbf{I}\in\mathbb{R}^{{\color[rgb]{0,0,0}H_{p}}\times{\color[rgb]{0,0,0}W_{% p}}\times 3}I ∈ blackboard_R start_POSTSUPERSCRIPT italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × 3 end_POSTSUPERSCRIPT
, A set of camera pose with intrinsic and extrinsic parameters,

{𝐊∈ℝ 3×3}𝐊 superscript ℝ 3 3\{\mathbf{K}\in\mathbb{R}^{3\times 3}\}{ bold_K ∈ blackboard_R start_POSTSUPERSCRIPT 3 × 3 end_POSTSUPERSCRIPT }
and

{𝐄⁢[𝐑;𝐭]}𝐄 𝐑 𝐭\{\mathbf{E}[\mathbf{R};\mathbf{t}]\}{ bold_E [ bold_R ; bold_t ] }
, where

𝐑∈ℝ 3×3 𝐑 superscript ℝ 3 3\mathbf{R}\in\mathbb{R}^{3\times 3}bold_R ∈ blackboard_R start_POSTSUPERSCRIPT 3 × 3 end_POSTSUPERSCRIPT
representations the rotation part of the extrinsic parameters, and

𝐭∈ℝ 3×1 𝐭 superscript ℝ 3 1\mathbf{t}\in\mathbb{R}^{3\times 1}bold_t ∈ blackboard_R start_POSTSUPERSCRIPT 3 × 1 end_POSTSUPERSCRIPT
is the translation part. The length of the camera pose equals frame number

F 𝐹 F italic_F
.

H p subscript 𝐻 𝑝 H_{p}italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT
and W p subscript 𝑊 𝑝 W_{p}italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT are the height and width of the pixel space

1

2 Estimate the depth map

D∈ℝ H p×W p D superscript ℝ subscript 𝐻 𝑝 subscript 𝑊 𝑝\textbf{D}\in\mathbb{R}^{{\color[rgb]{0,0,0}H_{p}}\times{\color[rgb]{0,0,0}W_{% p}}}D ∈ blackboard_R start_POSTSUPERSCRIPT italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_POSTSUPERSCRIPT
from I given camera pose parameters.

3 Get the translation of pixels T∈ℝ F×H p×W p×2 T superscript ℝ 𝐹 subscript 𝐻 𝑝 subscript 𝑊 𝑝 2\textbf{T}\in\mathbb{R}^{{\color[rgb]{0,0,0}F}\times{\color[rgb]{0,0,0}H_{p}}% \times{\color[rgb]{0,0,0}W_{p}}\times 2}T ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × 2 end_POSTSUPERSCRIPT based on I using using D, K, and E.

4 Get trajecories

Tr=T+C Tr T C\textbf{Tr}=\textbf{T}+\textbf{C}Tr = T + C
, where

C∈ℝ H p×W p×2 C superscript ℝ subscript 𝐻 𝑝 subscript 𝑊 𝑝 2\textbf{C}\in\mathbb{R}^{{\color[rgb]{0,0,0}H_{p}}\times{\color[rgb]{0,0,0}W_{% p}}\times 2}C ∈ blackboard_R start_POSTSUPERSCRIPT italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × 2 end_POSTSUPERSCRIPT
is pixel-level grid coordinates of image with shape

H p×W p subscript 𝐻 𝑝 subscript 𝑊 𝑝{\color[rgb]{0,0,0}H_{p}}\times{\color[rgb]{0,0,0}W_{p}}italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT
.

5 Get valid trajectory mask M for pixels that within the image space.

Output:Trajectories Tr, Trajectory Masks M

Algorithm 3 Trajectory extraction from single image

### 4.2 Camera Motion Control on Videos

The process for camera motion control on videos is more complex than the process for images since the video itself has its own motion. We need to extract the original motion with point trajectory estimation methods like Karaev et al. ([2023](https://arxiv.org/html/2411.19324v1#bib.bib22)), then combine the original motion with camera motion to get the final trajectories. We show the details in Algorithm [4](https://arxiv.org/html/2411.19324v1#algorithm4 "In 4.2 Camera Motion Control on Videos ‣ 4 Fine-grained Control of Video Generation ‣ Trajectory Attention for Fine-grained Video Motion Control").

Input:Video Frames

V∈ℝ F×H p×W p×3 V superscript ℝ 𝐹 subscript 𝐻 𝑝 subscript 𝑊 𝑝 3\textbf{V}\in\mathbb{R}^{F\times{\color[rgb]{0,0,0}H_{p}}\times{\color[rgb]{% 0,0,0}W_{p}}\times 3}V ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × 3 end_POSTSUPERSCRIPT
, A set of camera pose with intrinsic and extrinsic parameters,

{𝐊∈ℝ 3×3}𝐊 superscript ℝ 3 3\{\mathbf{K}\in\mathbb{R}^{3\times 3}\}{ bold_K ∈ blackboard_R start_POSTSUPERSCRIPT 3 × 3 end_POSTSUPERSCRIPT }
and

{𝐄⁢[𝐑;𝐭]}𝐄 𝐑 𝐭\{\mathbf{E}[\mathbf{R};\mathbf{t}]\}{ bold_E [ bold_R ; bold_t ] }
. The lenght of camera pose equals to frame number

F 𝐹 F italic_F

1

2 Estimate the depth map

D∈ℝ F×H p×W p D superscript ℝ 𝐹 subscript 𝐻 𝑝 subscript 𝑊 𝑝\textbf{D}\in\mathbb{R}^{F\times{\color[rgb]{0,0,0}H_{p}}\times{\color[rgb]{% 0,0,0}W_{p}}}D ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_POSTSUPERSCRIPT
from V given camera pose parameters.

3 Estimate point trajecotries

P∈ℝ F×L×2 P superscript ℝ 𝐹 𝐿 2\textbf{P}\in\mathbb{R}^{F\times L\times 2}P ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_L × 2 end_POSTSUPERSCRIPT
and the corresponding occlusion masks

M o subscript M 𝑜\textbf{M}_{o}M start_POSTSUBSCRIPT italic_o end_POSTSUBSCRIPT
.

4 Get the translation of pixels T∈ℝ F×H p×W p×2 T superscript ℝ 𝐹 subscript 𝐻 𝑝 subscript 𝑊 𝑝 2\textbf{T}\in\mathbb{R}^{F\times{\color[rgb]{0,0,0}H_{p}}\times{\color[rgb]{% 0,0,0}W_{p}}\times 2}T ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × 2 end_POSTSUPERSCRIPT using D, K and E.

5 Sample the translation of point trajectories

P t∈ℝ F×L×2 subscript P 𝑡 superscript ℝ 𝐹 𝐿 2\textbf{P}_{t}\in\mathbb{R}^{F\times L\times 2}P start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_F × italic_L × 2 end_POSTSUPERSCRIPT
from T using P.

6 Get trajecories

Tr=P t+P Tr subscript P 𝑡 P\textbf{Tr}=\textbf{P}_{t}+\textbf{P}Tr = P start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT + P
.

7 Get valid trajectory mask

M=M i∧M o M subscript M 𝑖 subscript M 𝑜\textbf{M}=\textbf{M}_{i}\wedge\textbf{M}_{o}M = M start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∧ M start_POSTSUBSCRIPT italic_o end_POSTSUBSCRIPT
, where

M i subscript M 𝑖\textbf{M}_{i}M start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT
is for pixels that within the image space.

Output:Trajectories Tr, Trajectory Masks M

Algorithm 4 Trajectory extraction from video

### 4.3 Video Editing

Video editing based on an edited first frame has gained popularity recently (Ouyang et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib31); Ku et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib23)). The goal is to generate videos where the content of the first frame aligns with the edited version while inheriting motion from reference videos. Our method is well-suited for this task, as we leverage Image-to-Video generation models that use the edited first frame as a conditioning input while incorporating trajectories extracted from the original videos to guide the motion.

5 Experiments
-------------

Table 1: Qualitative comparison on image camera motion control. *: MotionI2V uses AnimateDiff (Guo et al., [2023b](https://arxiv.org/html/2411.19324v1#bib.bib14)) while we use SVD (Blattmann et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib2)) as the base models. Other methods use SVD as default.

Setting Methods ATE (m, ↓↓\downarrow↓)RPE trans (m, ↓↓\downarrow↓)RPE Rot (deg, ↓↓\downarrow↓)FID (↓↓\downarrow↓)
14 frames MotionCtrl 1.2151 0.5213 1.8372 101.3
Ours 0.0212 0.0221 0.1151 104.2
16 frames MotionI2V*0.0712 0.0471 0.2853 124.1
Ours 0.0413 0.0241 0.1231 108.7
25 frames CameraCtrl 0.0411 0.0268 0.3480 115.8
NVS_Solver 0.1216 0.0558 0.4785 108.5
Ours 0.0396 0.0232 0.1939 103.5

![Image 6: Refer to caption](https://arxiv.org/html/2411.19324v1/x6.png)

Figure 6: Qualitative comparisons for camera motion control on images. While other methods often exhibit significant quality degradation or inconsistencies in camera motion, our approach consistently delivers high-quality results with precise, fine-grained control over camera movements. Regions are highlighted in yellow boxes to reveal camera motion. For a more comprehensive understanding, we highly recommend viewing the accompanying videos in the supplementary materials.

Table 2: Qualitative comparison on video camera motion control.

Methods ATE (m, ↓↓\downarrow↓)RPE trans (m, ↓↓\downarrow↓)RPE Rot (deg, ↓↓\downarrow↓)FID (↓↓\downarrow↓)
NVS_Solver 0.5112 0.3442 1.3241 134.5
Ours 0.3572 0.1981 0.7889 129.3
Ours (w. NVS_Solver)0.3371 0.1972 0.6241 112.2

![Image 7: Refer to caption](https://arxiv.org/html/2411.19324v1/x7.png)

Figure 7: Qualitative comparisons for camera motion control on videos. In the second row, we provide video frames after view warping as a reference. Methods like NVS Solver (You et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib52)) use frame-wise information injection but overlook temporal continuity, leading to inconsistent motion control, especially in frames farther from the first one. In contrast, our approach explicitly models attention across frames, which significantly benefits control precision. We highlight the control precision with yellow boxes, where our method aligns better with the reference. *: we integrate NVS Solver’s capability to inject frame-wise information, achieving better video alignment with the original videos.

![Image 8: Refer to caption](https://arxiv.org/html/2411.19324v1/x8.png)

Figure 8: Results on first-frame guided video editing. We compare our method with those from Ouyang et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib31)); Ku et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib23)). The results show that other methods struggle to maintain consistency after editing. In contrast, our method successfully preserves the edited features across frames, thanks to its ability to model trajectory consistency throughout the video.

### 5.1 Experimental Settings

Datasets. We use MiraData (Ju et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib21)) for training, a large-scale video dataset with long-duration videos and structured captions, featuring realistic and dynamic scenes from games or daily life. We sample short video clips and apply Yang et al. ([2023a](https://arxiv.org/html/2411.19324v1#bib.bib45)) to extract optical flow as trajectory guidance. In total, we train with 10k video clips.

Implementation Details. We conducted our main experiments using SVD (Blattmann et al., [2023](https://arxiv.org/html/2411.19324v1#bib.bib2)), employing the Adam optimizer with a learning rate of 1e-5 per batch size, with mixed precision training of fp16. We only fine-tune the additional trajectory attention modules which inherit weights from the temporal modules. Our efficient training design allows for approximately 24 GPU hours of training (with a batch size of 1 on a single A100 GPU over the course of one day). We train trajectory attention on the 12-frame video generation modules and apply the learned trajectory attention to both 12-frame and 25-frame video generation models. Despite being trained on 12-frame videos, the trajectory attention performs effectively when integrated into the 25-frame model, demonstrating the strong generalization capability of our design.

Metrics. We assessed the conditional generation performance using four distinct metrics: (1) Absolute Trajectory Error (ATE) (Goel et al., [1999](https://arxiv.org/html/2411.19324v1#bib.bib10)), which quantifies the deviation between the estimated and actual trajectories of a camera or robot; and (2) Relative Pose Error (RPE) (Goel et al., [1999](https://arxiv.org/html/2411.19324v1#bib.bib10)), which captures the drift in the estimated pose by separately calculating the translation (RPE-T) and rotation (RPE-R) errors. (3) Fréchet Inception Distance (FID) (Heusel et al., [2017](https://arxiv.org/html/2411.19324v1#bib.bib16)), which evaluates the quality and variability of the generated views.

### 5.2 Camera Motion Control on Single Images

We compare the results of camera motion control on single images with the methods proposed by Wang et al. ([2024c](https://arxiv.org/html/2411.19324v1#bib.bib41)); Shi et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib34)); He et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib15)). The evaluation is based on 230 combinations of diverse scenes and camera trajectories. To ensure a fair comparison, our model is tested under varying settings due to the frame limitations of certain models (i.e., (Wang et al., [2024c](https://arxiv.org/html/2411.19324v1#bib.bib41)) only releases a 12-frame version).

Table [1](https://arxiv.org/html/2411.19324v1#S5.T1 "Table 1 ‣ 5 Experiments ‣ Trajectory Attention for Fine-grained Video Motion Control") summarizes the results, showing that our methods consistently achieve higher or comparable control precision in terms of ATE and RPE, along with strong fidelity as measured by FID, compared to other methods (Wang et al., [2024c](https://arxiv.org/html/2411.19324v1#bib.bib41); Shi et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib34); He et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib15); You et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib52)). Although MotionCtrl (Wang et al., [2024c](https://arxiv.org/html/2411.19324v1#bib.bib41)) generates slightly better results in terms of FID, it compromises significantly on control precision. Motion-I2V Shi et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib34)), which uses flow-based attention, only allows frames to attend to the first frame, leading to quality issues in some cases. In contrast, our approach maintains better control precision while preserving generation quality. It also performs better over longer time ranges than other recent methods (He et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib15); You et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib52)).

We further provide qualitative results in Fig.[6](https://arxiv.org/html/2411.19324v1#S5.F6 "Figure 6 ‣ 5 Experiments ‣ Trajectory Attention for Fine-grained Video Motion Control"), which is aligned with the conclusions in Table[1](https://arxiv.org/html/2411.19324v1#S5.T1 "Table 1 ‣ 5 Experiments ‣ Trajectory Attention for Fine-grained Video Motion Control").

### 5.3 Camera Motion Control on Videos

We compare the video synthesis performance of our method with You et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib52)), who employ a test-time optimization approach. Their method uses view-warped frames as optimization targets, injecting partial frame information into the generation process. However, it optimizes on a per-frame basis, neglecting temporal coherence. As a result, when large view changes occur, their method often struggles to follow the motion accurately and introduces spatial blur. In contrast, our method precisely handles large motions. Notably, the way You et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib52)) injects frame information is orthogonal to our approach. By combining their optimization technique with our trajectory attention, we achieve higher fidelity in the generated results, as demonstrated in Table[2](https://arxiv.org/html/2411.19324v1#S5.T2 "Table 2 ‣ 5 Experiments ‣ Trajectory Attention for Fine-grained Video Motion Control") and Fig.[7](https://arxiv.org/html/2411.19324v1#S5.F7 "Figure 7 ‣ 5 Experiments ‣ Trajectory Attention for Fine-grained Video Motion Control").

### 5.4 Video Editing

Compared to previous first-frame guided editing methods (Ku et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib23); Ouyang et al., [2024](https://arxiv.org/html/2411.19324v1#bib.bib31)), our approach explicitly models motion dynamics as trajectories across frames, enabling better content consistency over large spatial and temporal ranges. As shown in Fig. [8](https://arxiv.org/html/2411.19324v1#S5.F8 "Figure 8 ‣ 5 Experiments ‣ Trajectory Attention for Fine-grained Video Motion Control"), while other methods struggle to maintain consistency after editing, our method successfully preserves the edited features throughout the entire sequence.

### 5.5 Ablation on Trajectory Attention Designs

Table 3: Ablation on trajectory attention design.

Methods ATE (m, ↓↓\downarrow↓)RPE trans (m, ↓↓\downarrow↓)RPE Rot (deg, ↓↓\downarrow↓)FID (↓↓\downarrow↓)
Vanilla 1.7812 2.4258 13.2141 329.6
+ Tuning 0.3147 0.3169 1.5364 139.2
+ Add-on Branch 0.0724 0.1274 0.3824 112.4
+ Weight Inheriting 0.0396 0.0232 0.1939 103.5

To validate the effectiveness of our trajectory attention design, we conducted an ablation study, presented in Table [3](https://arxiv.org/html/2411.19324v1#S5.T3 "Table 3 ‣ 5.5 Ablation on Trajectory Attention Designs ‣ 5 Experiments ‣ Trajectory Attention for Fine-grained Video Motion Control") . We examined four types of implementations: 1) Directly applying temporal attention to trajectory attention, 2) Integrating trajectory attention into temporal attention with weight fine-tuning, 3) Utilizing an add-on branch for modeling trajectory attention, and 4) Inheriting weights from temporal attention (as illustrated in Fig. [5](https://arxiv.org/html/2411.19324v1#S3.F5 "Figure 5 ‣ 3.4 Training trajectory attention efficiently ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control"))

The results in Table[3](https://arxiv.org/html/2411.19324v1#S5.T3 "Table 3 ‣ 5.5 Ablation on Trajectory Attention Designs ‣ 5 Experiments ‣ Trajectory Attention for Fine-grained Video Motion Control") indicate that the vanilla adaptation leads to significantly poor motion tracking and video quality, with some outputs exhibiting complete noise (we omit such invalid results during evaluation, otherwise calculating the statistic results is not feasible.). After fine-tuning the temporal weights, the implementation functions better but remains suboptimal. In contrast, using an add-on branch for trajectory attention markedly improves both motion control precision and video quality. Additionally, inheriting weights from temporal attention facilitates faster convergence and better overall performance compared to simply initializing attention weights randomly.

![Image 9: Refer to caption](https://arxiv.org/html/2411.19324v1/x9.png)

Figure 9: Qualitative results on Open-Sora-Plan.(Lab & etc., [2024](https://arxiv.org/html/2411.19324v1#bib.bib25)) By incorporating trajectory attention into the 3D attention module, we successfully enable camera motion control.

### 5.6 Results on Full Attention Models.

Our method also has the potential to support full 3D attention using a similar pipeline as shown in Fig. [3](https://arxiv.org/html/2411.19324v1#S3.F3 "Figure 3 ‣ 3.1 Preliminary ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control") and Fig. [5](https://arxiv.org/html/2411.19324v1#S3.F5 "Figure 5 ‣ 3.4 Training trajectory attention efficiently ‣ 3 Methodology ‣ Trajectory Attention for Fine-grained Video Motion Control"), with the key difference being the application of trajectory attention to the 3D attention module instead of the temporal attention. As demonstrated in Fig. [9](https://arxiv.org/html/2411.19324v1#S5.F9 "Figure 9 ‣ 5.5 Ablation on Trajectory Attention Designs ‣ 5 Experiments ‣ Trajectory Attention for Fine-grained Video Motion Control"), this enables diverse camera motion control in the generated results. For detailed implementation, please refer to the supplementary materials.

6 Conclusion
------------

In conclusion, we introduced trajectory attention, a novel approach for fine-grained camera motion control in video generation. Our method, which models trajectory attention as an auxiliary branch alongside temporal attention, demonstrates significant improvements in precision and long-range consistency. Experiments show its effectiveness in camera motion control for both images and videos while maintaining high-quality generation. The approach’s extensibility to other video motion control tasks, such as first-frame-guided video editing, highlights its potential impact on the broader field of video generation and editing.

References
----------

*   Bahmani et al. (2024) Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. _arXiv preprint arXiv:2407.12781_, 2024. 
*   Blattmann et al. (2023) Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. _arXiv preprint arXiv:2311.15127_, 2023. 
*   Chen et al. (2023a) Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. _arXiv preprint arXiv:2310.19512_, 2023a. 
*   Chen et al. (2024) Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. _arXiv preprint arXiv:2401.09047_, 2024. 
*   Chen et al. (2023b) Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang. Motion-conditioned diffusion model for controllable video synthesis. _arXiv preprint arXiv:2304.14404_, 2023b. 
*   Chen et al. (2023c) Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models, 2023c. 
*   Cong et al. (2023) Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manuel Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flatten: optical flow-guided attention for consistent text-to-video editing. _arXiv preprint arXiv:2310.05922_, 2023. 
*   Deng et al. (2023) Yufan Deng, Ruida Wang, Yuhao Zhang, Yu-Wing Tai, and Chi-Keung Tang. Dragvideo: Interactive drag-style video editing. _arXiv preprint arXiv:2312.02216_, 2023. 
*   Geng & Owens (2024) Daniel Geng and Andrew Owens. Motion guidance: Diffusion-based image editing with differentiable motion estimators. _arXiv preprint arXiv:2401.18085_, 2024. 
*   Goel et al. (1999) Puneet Goel, Stergios I Roumeliotis, and Gaurav S Sukhatme. Robust localization using relative and absolute position estimates. In _Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No. 99CH36289)_, volume 2, pp. 1134–1140. IEEE, 1999. 
*   Gu et al. (2024) Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjie Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang. Videoswap: Customized video subject swapping with interactive semantic point correspondence. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp. 7621–7630, 2024. 
*   Guo et al. (2024) Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control. _arXiv preprint arXiv:2407.03168_, 2024. 
*   Guo et al. (2023a) Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. _arXiv preprint arXiv:2311.16933_, 2023a. 
*   Guo et al. (2023b) Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. _arXiv preprint arXiv:2307.04725_, 2023b. 
*   He et al. (2024) Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. _arXiv preprint arXiv:2404.02101_, 2024. 
*   Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. _Advances in neural information processing systems_, 30, 2017. 
*   Ho et al. (2022) Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. _Advances in Neural Information Processing Systems_, 35:8633–8646, 2022. 
*   Hong et al. (2022) Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. _arXiv preprint arXiv:2205.15868_, 2022. 
*   Hou et al. (2024) Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-free camera control for video generation. _arXiv preprint arXiv:2406.10126_, 2024. 
*   Hu et al. (2024) Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang, and Lizhuang Ma. Motionmaster: Training-free camera motion transfer for video generation, 2024. 
*   Ju et al. (2024) Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. _arXiv preprint arXiv:2407.06358_, 2024. 
*   Karaev et al. (2023) Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. _arXiv preprint arXiv:2307.07635_, 2023. 
*   Ku et al. (2024) Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen. Anyv2v: A plug-and-play framework for any video-to-video editing tasks. _arXiv preprint arXiv:2403.14468_, 2024. 
*   Kuang et al. (2024) Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. _arXiv preprint arXiv:2405.17414_, 2024. 
*   Lab & etc. (2024) PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, April 2024. URL [https://doi.org/10.5281/zenodo.10948109](https://doi.org/10.5281/zenodo.10948109). 
*   Li et al. (2024) Peng Li, Yuan Liu, Xiaoxiao Long, Feihu Zhang, Cheng Lin, Mengfei Li, Xingqun Qi, Shanghang Zhang, Wenhan Luo, Ping Tan, et al. Era3d: High-resolution multiview diffusion using efficient row-wise attention. _arXiv preprint arXiv:2405.11616_, 2024. 
*   Ling et al. (2024) Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation. _arXiv preprint arXiv:2406.05338_, 2024. 
*   Müller et al. (2024) Norman Müller, Katja Schwarz, Barbara Rössle, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, and Peter Kontschieder. Multidiff: Consistent novel view synthesis from a single image. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp. 10258–10268, 2024. 
*   Niu et al. (2024) Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model. _arXiv preprint arXiv:2405.20222_, 2024. 
*   OpenAI (2024) OpenAI. Video generation models as world simulators. [https://openai.com/research/video-generation-models-as-world-simulators](https://openai.com/research/video-generation-models-as-world-simulators), 2024. 
*   Ouyang et al. (2024) Wenqi Ouyang, Yi Dong, Lei Yang, Jianlou Si, and Xingang Pan. I2vedit: First-frame-guided video editing via image-to-video diffusion models. _arXiv preprint arXiv:2405.16537_, 2024. 
*   Peebles & Xie (2023) William Peebles and Saining Xie. Scalable diffusion models with transformers. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pp. 4195–4205, 2023. 
*   Qiu et al. (2024) Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning-free trajectory control in video diffusion models. _arXiv preprint arXiv:2406.16863_, 2024. 
*   Shi et al. (2024) Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In _ACM SIGGRAPH 2024 Conference Papers_, pp. 1–11, 2024. 
*   Teng et al. (2023) Yao Teng, Enze Xie, Yue Wu, Haoyu Han, Zhenguo Li, and Xihui Liu. Drag-a-video: Non-rigid video editing with point-based interaction. _arXiv preprint arXiv:2312.02936_, 2023. 
*   Valevski et al. (2024) Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines, 2024. URL [https://arxiv.org/abs/2408.14837](https://arxiv.org/abs/2408.14837). 
*   Wang et al. (2024a) Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. _arXiv preprint arXiv:2402.01566_, 2024a. 
*   Wang et al. (2023a) Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. _arXiv preprint arXiv:2308.06571_, 2023a. 
*   Wang et al. (2024b) Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. _Advances in Neural Information Processing Systems_, 36, 2024b. 
*   Wang et al. (2023b) Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. _arXiv preprint arXiv:2309.15103_, 2023b. 
*   Wang et al. (2024c) Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In _ACM SIGGRAPH 2024 Conference Papers_, pp. 1–11, 2024c. 
*   Wu et al. (2024) Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Motionbooth: Motion-aware customized text-to-video generation. _NeurIPS_, 2024. 
*   Xiao et al. (2024) Zeqi Xiao, Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-free motion interpreter and controller. _arXiv preprint arXiv:2405.14864_, 2024. 
*   Xu et al. (2024) Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera-controllable 3d-consistent image-to-video generation. _arXiv preprint arXiv:2406.02509_, 2024. 
*   Yang et al. (2023a) Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp. 7236–7246, 2023a. 
*   Yang et al. (2024a) Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. _arXiv preprint arXiv:2406.09414_, 2024a. 
*   Yang et al. (2024b) Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. _arXiv preprint arXiv:2402.03162_, 2024b. 
*   Yang et al. (2023b) Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In _SIGGRAPH Asia 2023 Conference Papers_, pp. 1–11, 2023b. 
*   Yang et al. (2024c) Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. _arXiv preprint arXiv:2408.06072_, 2024c. 
*   Yatim et al. (2023) Danah Yatim, Rafail Fridman, Omer Bar Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. _arXiv preprint arXiv:2311.17009_, 2023. 
*   Yin et al. (2023) Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. _arXiv preprint arXiv:2308.08089_, 2023. 
*   You et al. (2024) Meng You, Zhiyu Zhu, Hui Liu, and Junhui Hou. Nvs-solver: Video diffusion model as zero-shot novel view synthesizer. _arXiv preprint arXiv:2405.15364_, 2024. 
*   Yu et al. (2023) Jiwen Yu, Xiaodong Cun, Chenyang Qi, Yong Zhang, Xintao Wang, Ying Shan, and Jian Zhang. Animatezero: Video diffusion models are zero-shot image animators. _arXiv preprint arXiv:2312.03793_, 2023. 
*   Yu et al. (2024) Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis. _arXiv preprint arXiv:2409.02048_, 2024. 
*   Zhang et al. (2023a) Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pp. 3836–3847, 2023a. 
*   Zhang et al. (2023b) Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. _arXiv preprint arXiv:2305.13077_, 2023b. 
*   Zhang et al. (2024) Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mimicmotion: High-quality human motion video generation with confidence-aware pose guidance. _arXiv preprint arXiv:2406.19680_, 2024. 
*   Zhao et al. (2023) Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. _arXiv preprint arXiv:2310.08465_, 2023. 
*   Zhu et al. (2024a) Hanxin Zhu, Tianyu He, Anni Tang, Junliang Guo, Zhibo Chen, and Jiang Bian. Compositional 3d-aware video generation with llm director. _Advances in Neural Information Processing Systems_, 2024a. 
*   Zhu et al. (2024b) Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. In _European Conference on Computer Vision (ECCV)_, 2024b. 
*   Zhuang et al. (2024) Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, and Yali Wang. Vlogger: Make your dream a vlog. _arXiv preprint arXiv:2401.09414_, 2024. 
*   Zuo et al. (2024) Yi Zuo, Lingling Li, Licheng Jiao, Fang Liu, Xu Liu, Wenping Ma, Shuyuan Yang, and Yuwei Guo. Edit-your-motion: Space-time diffusion decoupling learning for video motion editing, 2024. URL [https://arxiv.org/abs/2405.04496](https://arxiv.org/abs/2405.04496). 

Appendix A Appendix
-------------------

### A.1 Limitations

As shown in Fig. [20](https://arxiv.org/html/2411.19324v1#A1.F20 "Figure 20 ‣ A.11 More Qualitative Result ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control"), our method has several limitations that require further exploration. Currently, it depends on external techniques such as Karaev et al. ([2023](https://arxiv.org/html/2411.19324v1#bib.bib22)); Yang et al. ([2023a](https://arxiv.org/html/2411.19324v1#bib.bib45)) for trajectory extraction. Investigating how to generate trajectories from more flexible inputs, such as text-based conditions, is a promising direction for future research. Moreover, our method depends on the generative capabilities of the underlying foundation models. If these models face difficulties handling rapid motions, the effectiveness of trajectory-based control diminishes. Additionally, our method encounters challenges with 3D cycle consistency. For instance, performing 360-degree rotations would require further design adaptations. Furthermore, the control precision diminishes when the trajectories become too sparse since the auxiliary branch does not provide enough control information.

### A.2 Details of full-attention implementation

We implement trajectory attention in Open-Sora-Plan Lab & etc. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib25)), a DiT (Peebles & Xie, [2023](https://arxiv.org/html/2411.19324v1#bib.bib32)) model with 3D attention. The trajectory attention is constructed and trained following the same procedure outlined in the main paper, with the key difference being that it is appended to the 3D full attention block instead of the temporal attention block (Fig. [10](https://arxiv.org/html/2411.19324v1#A1.F10 "Figure 10 ‣ A.2 Details of full-attention implementation ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control")). For other training details, we follow the setting in Lab & etc. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib25)).

![Image 10: Refer to caption](https://arxiv.org/html/2411.19324v1/x10.png)

Figure 10: Pipe for video diffusion models with 3D Attention. The key distinction with the pipeline in the main paper lies in applying trajectory attention to the 3D attention module, rather than to the temporal attention mechanism.

### A.3 Details of task process

Camera Parameters. The intrinsic parameters describe the internal characteristics of the camera. These parameters define how the camera transforms 3D points in its coordinate system to 2D points on the image plane. The matrix 𝐊∈ℝ 3×3 𝐊 superscript ℝ 3 3\mathbf{K}\in\mathbb{R}^{3\times 3}bold_K ∈ blackboard_R start_POSTSUPERSCRIPT 3 × 3 end_POSTSUPERSCRIPT typically has the following structure:

𝐊=[f x 0 c x 0 f y c y 0 0 1]𝐊 matrix subscript 𝑓 𝑥 0 subscript 𝑐 𝑥 0 subscript 𝑓 𝑦 subscript 𝑐 𝑦 0 0 1\mathbf{K}=\begin{bmatrix}f_{x}&0&c_{x}\\ 0&f_{y}&c_{y}\\ 0&0&1\end{bmatrix}bold_K = [ start_ARG start_ROW start_CELL italic_f start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT end_CELL start_CELL 0 end_CELL start_CELL italic_c start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL italic_f start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT end_CELL start_CELL italic_c start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 1 end_CELL end_ROW end_ARG ](4)

*   •f x subscript 𝑓 𝑥 f_{x}italic_f start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT and f y subscript 𝑓 𝑦 f_{y}italic_f start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT: Focal lengths in the x and y directions, often in pixel units. 
*   •c x subscript 𝑐 𝑥 c_{x}italic_c start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT and c y subscript 𝑐 𝑦 c_{y}italic_c start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT: Coordinates of the principal point (optical center) in the image plane. 

The intrinsic matrix 𝐊 𝐊\mathbf{K}bold_K encapsulates how pixel coordinates relate to normalized image coordinates. The extrinsic parameters define the camera’s position and orientation in the world coordinate system. This involves:

*   •Rotation matrix 𝐑∈ℝ 3×3 𝐑 superscript ℝ 3 3\mathbf{R}\in\mathbb{R}^{3\times 3}bold_R ∈ blackboard_R start_POSTSUPERSCRIPT 3 × 3 end_POSTSUPERSCRIPT: Represents the camera’s orientation by rotating the world coordinate system to align with the camera coordinate system. 
*   •Translation vector 𝐭∈ℝ 3×1 𝐭 superscript ℝ 3 1\mathbf{t}\in\mathbb{R}^{3\times 1}bold_t ∈ blackboard_R start_POSTSUPERSCRIPT 3 × 1 end_POSTSUPERSCRIPT: Represents the position of the camera in the world coordinate system. 

These two components are combined to form the extrinsic matrix, which can be represented as: 𝐄=[𝐑∣𝐭]𝐄 delimited-[]conditional 𝐑 𝐭\mathbf{E}=[\mathbf{R}\mid\mathbf{t}]bold_E = [ bold_R ∣ bold_t ] Here, 𝐑 𝐑\mathbf{R}bold_R defines how the 3D space is rotated relative to the camera, while 𝐭 𝐭\mathbf{t}bold_t indicates the displacement of the camera’s origin from the world coordinate origin. For a random input image, we predefine the intrinsic parameters and use the given camera trajectory to generate the extrinsic parameters.

Effects of intrinsic parameters. Since we cannot precisely estimate the intrinsic and extrinsic parameters from a single image, we use predefined intrinsic parameters and some hyperparameters for extrinsic parameters. From our observations, these predefined parameters with statistics can effectively generate reasonable results. We can also adjust them accordingly. Specifically, we set c x subscript 𝑐 𝑥 c_{x}italic_c start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT and c y subscript 𝑐 𝑦 c_{y}italic_c start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT in the center of the image plane, and f x subscript 𝑓 𝑥 f_{x}italic_f start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT and f y subscript 𝑓 𝑦 f_{y}italic_f start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT to 260 (relative). As shown in Fig. [21](https://arxiv.org/html/2411.19324v1#A1.F21 "Figure 21 ‣ A.11 More Qualitative Result ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control"), we illustrate cases with different focal lengths.

Depth estimation. We use DepthAnythingV2 Yang et al. ([2024a](https://arxiv.org/html/2411.19324v1#bib.bib46)) to estimate the depth maps from frames. Examples of estimated results are shown in Fig. [11](https://arxiv.org/html/2411.19324v1#A1.F11 "Figure 11 ‣ A.3 Details of task process ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control")

![Image 11: Refer to caption](https://arxiv.org/html/2411.19324v1/extracted/6032239/figures/depthmap.png)

Figure 11: Depth estimation results.

Translation computing. Based on the depth map, camera parameters of two views, we can get the translation of pixels shown in Alg. [5](https://arxiv.org/html/2411.19324v1#algorithm5 "In A.3 Details of task process ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control").

Input:

D∈ℝ H p×W p D superscript ℝ subscript 𝐻 𝑝 subscript 𝑊 𝑝\textbf{D}\in\mathbb{R}^{H_{p}\times W_{p}}D ∈ blackboard_R start_POSTSUPERSCRIPT italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_POSTSUPERSCRIPT
: Depth map of the first view

E 1∈ℝ 4×4 subscript E 1 superscript ℝ 4 4\textbf{E}_{1}\in\mathbb{R}^{4\times 4}E start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT 4 × 4 end_POSTSUPERSCRIPT
: Extrinsic matrix of the first view

E 2∈ℝ 4×4 subscript E 2 superscript ℝ 4 4\textbf{E}_{2}\in\mathbb{R}^{4\times 4}E start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT 4 × 4 end_POSTSUPERSCRIPT
: Extrinsic matrix of the second view

K∈ℝ 3×3 K superscript ℝ 3 3\textbf{K}\in\mathbb{R}^{3\times 3}K ∈ blackboard_R start_POSTSUPERSCRIPT 3 × 3 end_POSTSUPERSCRIPT
: Intrinsic matrix for both views

Output:

T 12∈ℝ H p×W p×2 subscript T 12 superscript ℝ subscript 𝐻 𝑝 subscript 𝑊 𝑝 2\textbf{T}_{12}\in\mathbb{R}^{H_{p}\times W_{p}\times 2}T start_POSTSUBSCRIPT 12 end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × 2 end_POSTSUPERSCRIPT
: Transformed pixel coordinates between views

1

T←E 2⋅E 1−1←T⋅subscript E 2 superscript subscript E 1 1\textbf{T}\leftarrow\textbf{E}_{2}\cdot\textbf{E}_{1}^{-1}T ← E start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ⋅ E start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT - 1 end_POSTSUPERSCRIPT
;

// Compute relative transformation

2

3

y←[0,…,H p−1]←y 0…subscript 𝐻 𝑝 1\textbf{y}\leftarrow[0,\ldots,H_{p}-1]y ← [ 0 , … , italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT - 1 ]
,

x←[0,…,W p−1]←x 0…subscript 𝑊 𝑝 1\textbf{x}\leftarrow[0,\ldots,W_{p}-1]x ← [ 0 , … , italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT - 1 ]

4

X,Y←meshgrid⁢(x,y)←X Y meshgrid x y\textbf{X},\textbf{Y}\leftarrow\text{meshgrid}(\textbf{x},\textbf{y})X , Y ← meshgrid ( x , y )

P homo←stack⁢([X,Y,𝟏 H p×W p])←subscript P homo stack X Y subscript 1 subscript 𝐻 𝑝 subscript 𝑊 𝑝\textbf{P}_{\text{homo}}\leftarrow\text{stack}([\textbf{X},\textbf{Y},\mathbf{% 1}_{H_{p}\times W_{p}}])P start_POSTSUBSCRIPT homo end_POSTSUBSCRIPT ← stack ( [ X , Y , bold_1 start_POSTSUBSCRIPT italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_POSTSUBSCRIPT ] )
;

// Homogeneous pixel positions

5

P~←K−1⋅P homo←~P⋅superscript K 1 subscript P homo\tilde{\textbf{P}}\leftarrow\textbf{K}^{-1}\cdot\textbf{P}_{\text{homo}}over~ start_ARG P end_ARG ← K start_POSTSUPERSCRIPT - 1 end_POSTSUPERSCRIPT ⋅ P start_POSTSUBSCRIPT homo end_POSTSUBSCRIPT
;

// Normalized positions in camera space

6

7

D reshaped←D←subscript D reshaped D\textbf{D}_{\text{reshaped}}\leftarrow\textbf{D}D start_POSTSUBSCRIPT reshaped end_POSTSUBSCRIPT ← D
reshaped to

(H p,W p,1,1)subscript 𝐻 𝑝 subscript 𝑊 𝑝 1 1(H_{p},W_{p},1,1)( italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , 1 , 1 )

8

P world←D reshaped⋅P~←subscript P world⋅subscript D reshaped~P\textbf{P}_{\text{world}}\leftarrow\textbf{D}_{\text{reshaped}}\cdot\tilde{% \textbf{P}}P start_POSTSUBSCRIPT world end_POSTSUBSCRIPT ← D start_POSTSUBSCRIPT reshaped end_POSTSUBSCRIPT ⋅ over~ start_ARG P end_ARG

9

P world, homo←concatenate⁢([P world,𝟏 H p×W p])←subscript P world, homo concatenate subscript P world subscript 1 subscript 𝐻 𝑝 subscript 𝑊 𝑝\textbf{P}_{\text{world, homo}}\leftarrow\text{concatenate}([\textbf{P}_{\text% {world}},\mathbf{1}_{H_{p}\times W_{p}}])P start_POSTSUBSCRIPT world, homo end_POSTSUBSCRIPT ← concatenate ( [ P start_POSTSUBSCRIPT world end_POSTSUBSCRIPT , bold_1 start_POSTSUBSCRIPT italic_H start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT × italic_W start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_POSTSUBSCRIPT ] )

10

P world, homo′←T⋅P world, homo←subscript superscript P′world, homo⋅T subscript P world, homo\textbf{P}^{\prime}_{\text{world, homo}}\leftarrow\textbf{T}\cdot\textbf{P}_{% \text{world, homo}}P start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT world, homo end_POSTSUBSCRIPT ← T ⋅ P start_POSTSUBSCRIPT world, homo end_POSTSUBSCRIPT

11

P world′←P world, homo′[:,:,:3]\textbf{P}^{\prime}_{\text{world}}\leftarrow\textbf{P}^{\prime}_{\text{world, % homo}}[:,:,:3]P start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT world end_POSTSUBSCRIPT ← P start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT world, homo end_POSTSUBSCRIPT [ : , : , : 3 ]

12

T 12←K⋅P world′←subscript T 12⋅K subscript superscript P′world\textbf{T}_{12}\leftarrow\textbf{K}\cdot\textbf{P}^{\prime}_{\text{world}}T start_POSTSUBSCRIPT 12 end_POSTSUBSCRIPT ← K ⋅ P start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT world end_POSTSUBSCRIPT

return

T 12 subscript T 12\textbf{T}_{12}T start_POSTSUBSCRIPT 12 end_POSTSUBSCRIPT

Algorithm 5 Compute pixel translation

Point trajectory extraction from videos. We use CoTracker Karaev et al. ([2023](https://arxiv.org/html/2411.19324v1#bib.bib22)) to extract point trajectories. An example is shown in Fig. [12](https://arxiv.org/html/2411.19324v1#A1.F12 "Figure 12 ‣ A.3 Details of task process ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control").

![Image 12: Refer to caption](https://arxiv.org/html/2411.19324v1/x11.png)

Figure 12: Point trajectory estimation results.

### A.4 Input process.

In Alg. 4 of the main paper, we present the process of combining the video point trajectories and camera motion trajectories. In Sec 4.3, we present the input process of video editing. To clarify this process, we further provide the visualization in Fig. [13](https://arxiv.org/html/2411.19324v1#A1.F13 "Figure 13 ‣ A.5 Ablations on training datasets. ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control")

### A.5 Ablations on training datasets.

We initially selected 10k short real-world videos from Miradata Ju et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib21)). To evaluate the impact of training domain diversity and dataset size, we conducted ablation experiments on various training datasets. As shown in Table [4](https://arxiv.org/html/2411.19324v1#A1.T4 "Table 4 ‣ A.5 Ablations on training datasets. ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control"), we did not observe substantial improvements when varying the training domains or increasing the dataset size. The evaluation setup matches the 25-frame version described in Table 1.

Table 4: Ablations on training datasets.

Dataset Setting Training steps ATE (m, ↓↓\downarrow↓)RPE trans (m, ↓↓\downarrow↓)RPE Rot (deg, ↓↓\downarrow↓)FID (↓↓\downarrow↓)
10k real-world 40k 0.0396 0.0232 0.1939 103.5
10k games 40k 0.0421 0.0211 0.2139 105.3
10k real-world+10k games 40k 0.0372 0.0233 0.1899 102.2

![Image 13: Refer to caption](https://arxiv.org/html/2411.19324v1/x12.png)

Figure 13: Input process visualization. For all tasks, the inputs to the network are the first frame and the extracted trajectories. The usage of the first frame and the trajectories are identical to Fig. 3 in the main paper. The wrapped frames and the reference frames will not be used as inputs to the generation network.

### A.6 Using synthetic optical flow as guidance

Our method directly leverages optical flow to guide the generation process. Unlike previous approaches Geng & Owens ([2024](https://arxiv.org/html/2411.19324v1#bib.bib9)) requiring inference-time optimization, our method seamlessly integrates this guidance into the attention mechanism. In addition to generating intermediate frames by interpolating the optical flow, our approach achieves improved consistency, as demonstrated in Fig. [14](https://arxiv.org/html/2411.19324v1#A1.F14 "Figure 14 ‣ A.6 Using synthetic optical flow as guidance ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control").

![Image 14: Refer to caption](https://arxiv.org/html/2411.19324v1/x13.png)

Figure 14: Using synthetic optical flow as guidance. Our method supports directly using optical flow to guide generation. Blue boxes indicate the optical flow. Yellow boxes indicate the reference image.

### A.7 More challenging cases.

As shown in Fig. [15](https://arxiv.org/html/2411.19324v1#A1.F15 "Figure 15 ‣ A.7 More challenging cases. ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control"), our method can also cover many challenging situations, like (a) video editing with multiple objects, (b) video editing with occlusions, and (c) diverse and rapid camera motions (i.e. zoom-in, zoom-out, and clockwise rotation.).

![Image 15: Refer to caption](https://arxiv.org/html/2411.19324v1/x14.png)

Figure 15: Examples of challenging situations. Our method effectively addresses complex scenarios, including (a) video editing involving multiple objects, (b) video editing in the presence of occlusions, (c) diverse and rapid camera movements, such as zooming in and out, as well as clockwise rotations, and (d) video editing with distinct object categories. Please note that yellow boxes indicate reference videos, green boxes indicate input frames and blue boxes indicate output results.

### A.8 Experiments on the sparsity of trajectory attention

To assess the generalization capability of trajectory attention under varying levels of sparsity, we conduct experiments in two scenarios: using trajectories with reduced density and applying a small region mask to the trajectories. As shown in Fig. [17](https://arxiv.org/html/2411.19324v1#A1.F17 "Figure 17 ‣ A.10 Training data construction ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control"), our method performs effectively in both settings. However, when the trajectory density is extremely sparse (below 1/32 resolution), the results become unstable.

Moreover, we also find trajectory attention can be generalized to sparse hand-crafter trajectories, as shown in Fig. [18](https://arxiv.org/html/2411.19324v1#A1.F18 "Figure 18 ‣ A.10 Training data construction ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control").

### A.9 Comparison on camera trajectories

For clearer understanding, we visualize the predicted trajectories in Fig.[16](https://arxiv.org/html/2411.19324v1#A1.F16 "Figure 16 ‣ A.9 Comparison on camera trajectories ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control"), illustrating results from five scenes with a single camera trajectory. The figure shows that, thanks to the explicit modeling of camera motion, our method’s estimated trajectories closely align with the ground truth. In contrast, methods like He et al. ([2024](https://arxiv.org/html/2411.19324v1#bib.bib15)) exhibit inconsistencies in some cases.

![Image 16: Refer to caption](https://arxiv.org/html/2411.19324v1/x15.png)

Figure 16: Visualization of camera trajectories. The first row displays the estimated trajectories from our generation alongside the ground truth trajectories. The second row presents the estimated trajectories from CameraCtrl (denoted as “CC”) compared to the ground truth. The results indicate that our method aligns significantly better with the ground truth camera motion trajectories.

### A.10 Training data construction

The training data consists of natural scenes with inherent camera and object movements. We estimate the optical flow and occlusion masks using the method proposed by (Yang et al., [2023a](https://arxiv.org/html/2411.19324v1#bib.bib45)). Examples are shown in Fig. [19](https://arxiv.org/html/2411.19324v1#A1.F19 "Figure 19 ‣ A.10 Training data construction ‣ Appendix A Appendix ‣ Trajectory Attention for Fine-grained Video Motion Control").

![Image 17: Refer to caption](https://arxiv.org/html/2411.19324v1/x16.png)

Figure 17: Results on sparse trajectories. In (a), we show that trajectory attention remains robust even with relatively sparse trajectories. Even when the trajectory density is reduced to 1/16 of the original video resolution, it still performs well in motion control. In (b), we apply the trajectory mask to selectively use only a portion of the trajectories, keeping the regions outside the mask static. The model accurately follows the motion within the small masked area. (c) If we apply sparse trajectories control to a specific region (i.e., the dog region), the output results are more dynamic. Best viewed on the attached HTML file.

![Image 18: Refer to caption](https://arxiv.org/html/2411.19324v1/x17.png)

Figure 18: Applications on drag signals. Trajectory attention supports hand-crafted dragging trajectory. Row 1: origin videos. Row 2: dragged results.

![Image 19: Refer to caption](https://arxiv.org/html/2411.19324v1/x18.png)

Figure 19: Visualization on the training data. The training data includes origin frames, predicted optical flow, and occlusion masks.

### A.11 More Qualitative Result

We strongly recommend viewing the attached webpage for a more intuitive visualization.

![Image 20: Refer to caption](https://arxiv.org/html/2411.19324v1/x19.png)

Figure 20: Visualization on failure cases. Our method encounters challenges when dealing with extremely fast motions as well as complex and difficult-to-estimate motion patterns.

![Image 21: Refer to caption](https://arxiv.org/html/2411.19324v1/x20.png)

Figure 21: Visualization on different intrinsic parameters.