Title: VASE: Object-Centric Appearance and Shape Manipulation of Real Videos

URL Source: https://arxiv.org/html/2401.02473

Published Time: Mon, 08 Jan 2024 02:00:29 GMT

Markdown Content:
Elia Peruzzo 1 1{}^{{\color[rgb]{0.21,0.49,0.74}1}}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Vidit Goel 2 2{}^{{\color[rgb]{0.21,0.49,0.74}2}}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Dejia Xu 3 3{}^{{\color[rgb]{0.21,0.49,0.74}3}}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT Xingqian Xu 2 2{}^{{\color[rgb]{0.21,0.49,0.74}2}}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Yifan Jiang 3 3{}^{{\color[rgb]{0.21,0.49,0.74}3}}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT

Zhangyang Wang 2,3 2 3{}^{{\color[rgb]{0.21,0.49,0.74}2,3}}start_FLOATSUPERSCRIPT 2 , 3 end_FLOATSUPERSCRIPT Humphrey Shi 2,4 2 4{}^{{\color[rgb]{0.21,0.49,0.74}2,4}}start_FLOATSUPERSCRIPT 2 , 4 end_FLOATSUPERSCRIPT Nicu Sebe 1 1{}^{{\color[rgb]{0.21,0.49,0.74}1}}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT

1 1{}^{{\color[rgb]{0.21,0.49,0.74}1}}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT University of Trento 2 2{}^{{\color[rgb]{0.21,0.49,0.74}2}}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Picasrt AI Research 3 3{}^{{\color[rgb]{0.21,0.49,0.74}3}}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT UT Austin 4 4{}^{{\color[rgb]{0.21,0.49,0.74}4}}start_FLOATSUPERSCRIPT 4 end_FLOATSUPERSCRIPT Georgia Tech

###### Abstract

Recently, several works tackled the video editing task fostered by the success of large-scale text-to-image generative models. However, most of these methods holistically edit the frame using the text, exploiting the prior given by foundation diffusion models and focusing on improving the temporal consistency across frames. In this work, we introduce a framework that is object-centric and is designed to control both the object’s appearance and, notably, to execute precise and explicit structural modifications on the object. We build our framework on a pre-trained image-conditioned diffusion model, integrate layers to handle the temporal dimension, and propose training strategies and architectural modifications to enable shape control. We evaluate our method on the image-driven video editing task showing similar performance to the state-of-the-art, and showcasing novel shape-editing capabilities. Further details, code and examples are available on our [project page](https://helia95.github.io/vase-website/).

![Image 1: [Uncaptioned image]](https://arxiv.org/html/2401.02473v1/x1.png)

Figure 1: With VASE, we can manipulate both the shape and appearance of an object within a real video. In the top row, we present the source video, followed by a series of edits showcasing precise adjustments in both attributes. In the first frame (1st column), the keyframe structure is overlayed to the object. Red areas represent regions that have been removed from the original shape, while green regions denote newly added parts. The driver image for appearance edits is showcased in the bottom left corner.

1 Introduction
--------------

In recent years, there has been a surge in both the quality and diversity of generated content, primarily due to the deployment of Diffusion Models (DM) [[45](https://arxiv.org/html/2401.02473v1/#bib.bib45), [46](https://arxiv.org/html/2401.02473v1/#bib.bib46), [17](https://arxiv.org/html/2401.02473v1/#bib.bib17)] trained on vast amounts of data [[43](https://arxiv.org/html/2401.02473v1/#bib.bib43)]. Remarkably, large-scale text-to-image (T2I) models [[39](https://arxiv.org/html/2401.02473v1/#bib.bib39), [29](https://arxiv.org/html/2401.02473v1/#bib.bib29), [42](https://arxiv.org/html/2401.02473v1/#bib.bib42), [37](https://arxiv.org/html/2401.02473v1/#bib.bib37)] enable inexperienced users to produce impressive results using only a textual prompt. A plethora of works have been developed ever since, tailoring these models for more specific and constrained use cases, enabling various editing capabilities and making them even more user-friendly [[6](https://arxiv.org/html/2401.02473v1/#bib.bib6), [40](https://arxiv.org/html/2401.02473v1/#bib.bib40), [41](https://arxiv.org/html/2401.02473v1/#bib.bib41), [19](https://arxiv.org/html/2401.02473v1/#bib.bib19)]. Following a similar trend, many works are extending these paradigms to videos. The majority of current methods rely on a text prompt to guide the editing process [[53](https://arxiv.org/html/2401.02473v1/#bib.bib53), [5](https://arxiv.org/html/2401.02473v1/#bib.bib5)] and introduce new techniques to improve the smoothness and temporal consistency across the generated frames [[35](https://arxiv.org/html/2401.02473v1/#bib.bib35), [57](https://arxiv.org/html/2401.02473v1/#bib.bib57), [12](https://arxiv.org/html/2401.02473v1/#bib.bib12)]. As highlighted in prior research [[40](https://arxiv.org/html/2401.02473v1/#bib.bib40), [10](https://arxiv.org/html/2401.02473v1/#bib.bib10), [13](https://arxiv.org/html/2401.02473v1/#bib.bib13)], text prompts, while intuitive, often fall short in capturing precise nuances, leading to potential mismatches with the user’s intentions. Additionally, existing approaches treat the video frames as a whole for editing [[35](https://arxiv.org/html/2401.02473v1/#bib.bib35), [12](https://arxiv.org/html/2401.02473v1/#bib.bib12), [53](https://arxiv.org/html/2401.02473v1/#bib.bib53)], lacking the granularity needed to constrain the changes to a specific region. Lastly, the majority of these approaches cannot modify the structure of the objects in the video explicitly, as they often rely on per-frame structural guidance from the source video [[9](https://arxiv.org/html/2401.02473v1/#bib.bib9), [6](https://arxiv.org/html/2401.02473v1/#bib.bib6), [51](https://arxiv.org/html/2401.02473v1/#bib.bib51)], making them not suitable for this application.

In this work, our goal is to explore other directions in the video editing domain to empower the user with comprehensive capabilities. Specifically, we propose VASE, a framework for video object-centric appearance and shape edits. Our approach is _object-centric_, designed to control one individual object at a time while preserving the rest of the video intact. This mirrors numerous real-world scenarios where users seek to confine edits to a particular region. Secondly, we drive the edit using another image, as an image can convey more details than a textual prompt. More significantly, we introduce the potential to alter the object’s structure, providing more comprehensive editing options. This capacity remains under-explored in previous methods, or if present, it lacks an explicit control mechanism [[23](https://arxiv.org/html/2401.02473v1/#bib.bib23), [35](https://arxiv.org/html/2401.02473v1/#bib.bib35)]. In this work, we argue that in a video editing setup, shape modifications should not alter the object’s shape drastically, as this pertains more to a pure generative setup than an editing scenario.

Our focus lies in enabling precise, user-driven shape adjustments by manipulating a _single keyframe_, which then extends across the entire video. It is worth noting that our approach is not confined to particular domains (_e.g_. humans); instead, we propose a generic framework that can modify both rigid and articulated objects for videos in the wild. Lastly, we exclude expensive per-video training or video-decomposition processes, which limit the practicality of editing methods.

We base our method on a pre-trained image-conditioned diffusion model [[56](https://arxiv.org/html/2401.02473v1/#bib.bib56)], inflated with temporal layers to process videos. In order to faithfully replicate the motion depicted in the original video, we incorporate a ControlNet [[6](https://arxiv.org/html/2401.02473v1/#bib.bib6)] that takes the optical flow derived from the source video as input. Additionally, we unlock the explicit control over the object’s shape by feeding the desired keyframe shape into the ControlNet.

In our initial implementation, we observed that the model could potentially solve the task by predominantly relying on the optical flow while disregarding the structural condition. This rendered the shape edits ineffective during inference. To address this, we propose a novel _Joint Flow-Structure Augmentation_ procedure to break the alignment between the optical flow and the shape conditioning. Additionally, we introduce an auxiliary _Segmentation Head_ at training time to force the model to rely on the structure information. Moreover, we introduce a _Flow-Completion Network_ to generate realistic motion in the edited regions where no displacement information is available from the source video.

To summarize, our key contributions are as follows:

*   •We enable object-centric video editing in the wild, controlling the shape and the appearance of individual objects in the input video while keeping the rest of the video unchanged. The user provides one driving image that specifies the desired appearance and the shape of a single edited keyframe. 
*   •We design a _Joint Flow-Structure Augmentation_ pipeline, and introduce a _Flow-Completion Network_ and a _Auxiliary Segmentation Head_. All the components are necessary to edit the shape of the objects effectively. 
*   •We perform challenging shape edits of the target object without relying on test-time optimization procedures. Our model is trained once and applied at inference time, enhancing its practicality for various editing scenarios. 

2 Related Works
---------------

Image Editing with Diffusion Models. Large-scale text-to-image (T2I) diffusion models [[39](https://arxiv.org/html/2401.02473v1/#bib.bib39), [29](https://arxiv.org/html/2401.02473v1/#bib.bib29), [42](https://arxiv.org/html/2401.02473v1/#bib.bib42), [37](https://arxiv.org/html/2401.02473v1/#bib.bib37)] have gained widespread adoption as powerful generative models. They serve as strong generative priors and can be tailored to accommodate various forms of conditioning, along with the ability to manipulate real images [[27](https://arxiv.org/html/2401.02473v1/#bib.bib27)]. One line of works, achieves editing capabilities in a zero-shot manner, by exploiting the internal representation of a pretrained T2I model. Hertz et al. [[15](https://arxiv.org/html/2401.02473v1/#bib.bib15)] investigate the role of cross-attention maps in the generated images and exploit them to bind edits to a specific region. Plug-and-Play [[48](https://arxiv.org/html/2401.02473v1/#bib.bib48)] instead processes a real image and a target text prompt, and exploits the features of self-attention blocks to preserve the structure from the original image while applying the editing operation. Epstein et al. [[8](https://arxiv.org/html/2401.02473v1/#bib.bib8)] recently applies similar observation in conjunction with classifier-free guidance [[16](https://arxiv.org/html/2401.02473v1/#bib.bib16)] to achieve more comprehensive editing capabilities. Differently, ControlNet [[6](https://arxiv.org/html/2401.02473v1/#bib.bib6)] opts to train a hypernetwork to control the generation of the main T2I model based on spatial cues, enabling control with various types of signals _e.g_. depth, semantic maps, or sketches. Concurrently, [[28](https://arxiv.org/html/2401.02473v1/#bib.bib28)] explores more lightweight designs for the hypernetwork and the combination of multiple conditioning signals. Other works explore image-driven conditioning as an alternative to text for specifying the desired appearance of the objects in the scene. In particular, personalization emerged as an important research direction [[40](https://arxiv.org/html/2401.02473v1/#bib.bib40), [10](https://arxiv.org/html/2401.02473v1/#bib.bib10), [1](https://arxiv.org/html/2401.02473v1/#bib.bib1)], where the goal is to inject a user-provided concept in the form of images which is not present in the training data. In a similar way, Paint-by-Example (PBE) [[56](https://arxiv.org/html/2401.02473v1/#bib.bib56)] proposed the first image-based inpainting method with coarse masks depicting the shape of the desired object, and [[7](https://arxiv.org/html/2401.02473v1/#bib.bib7)] enhances the fidelity to the driver image using features from strong self-supervised models. [[55](https://arxiv.org/html/2401.02473v1/#bib.bib55)] introduces a new encoder to capture more nuances of the driver image and implement a “prompt-free” method, while PAIR Diffusion [[13](https://arxiv.org/html/2401.02473v1/#bib.bib13)] extracts per-object appearance representation, which enables image-driven editing and precise control. We draw inspiration from these methods and aim for more comprehensive editing capabilities in the video domain.

Video Generation and Editing with Diffusion Models. Recently, there have been several efforts to expand the achievements of text-to-image models to the video domain [[44](https://arxiv.org/html/2401.02473v1/#bib.bib44), [11](https://arxiv.org/html/2401.02473v1/#bib.bib11), [18](https://arxiv.org/html/2401.02473v1/#bib.bib18), [4](https://arxiv.org/html/2401.02473v1/#bib.bib4)]. As foundational video diffusion models remain largely unavailable to the open-source community and prohibitive to train from scratch, many works explored available T2I models for video editing.

One line of work exploits pre-trained T2I models and adapts them to the task in a zero-shot manner [[35](https://arxiv.org/html/2401.02473v1/#bib.bib35), [5](https://arxiv.org/html/2401.02473v1/#bib.bib5), [21](https://arxiv.org/html/2401.02473v1/#bib.bib21), [12](https://arxiv.org/html/2401.02473v1/#bib.bib12), [57](https://arxiv.org/html/2401.02473v1/#bib.bib57)]. The temporal consistency of the generated frames is generally encouraged by inflating the self-attention blocks to process multiple frames [[21](https://arxiv.org/html/2401.02473v1/#bib.bib21), [53](https://arxiv.org/html/2401.02473v1/#bib.bib53)]. Tune-A-Video [[53](https://arxiv.org/html/2401.02473v1/#bib.bib53)] involves fine-tuning the model on the video to be edited, enabling test-time edits through text prompts or cross-attention control [[25](https://arxiv.org/html/2401.02473v1/#bib.bib25)]. In FateZero [[35](https://arxiv.org/html/2401.02473v1/#bib.bib35)], the attention maps are extracted during an initial inversion step and blended with those generated during the editing process, confining the edit to a specific region. FateZero allows for shape modification of the foreground object, but it lacks a mechanism to specify the desired edit explicitly. TokenFlow [[12](https://arxiv.org/html/2401.02473v1/#bib.bib12)] proposes to propagate features of the base T2I model leveraging the optical flow extracted from the source video. Differently, other methods opt for training on video datasets; these models employ factorized spatio-temporal layers, integrating pre-existing layers from a T2I model with newly initialized temporal blocks [[9](https://arxiv.org/html/2401.02473v1/#bib.bib9), [51](https://arxiv.org/html/2401.02473v1/#bib.bib51), [14](https://arxiv.org/html/2401.02473v1/#bib.bib14)]. While these methods demonstrate remarkable editing capabilities with satisfactory temporal consistency, they do not explore the domain of shape manipulation. Any potential changes in shape, if they occur, cannot be directly controlled, which poses a limitation on their applicability.

Neural Layered Atlases (NLA).Kasten et al. [[20](https://arxiv.org/html/2401.02473v1/#bib.bib20)] propose to decompose the video in Neural Layered Atlases (NLA), exploiting the high correlation across frames. This representation is inherently designed for editing purposes, as the edits applied to the canonical image are propagated to the rest of the video, with temporal consistency obtained by design. Text2Live [[3](https://arxiv.org/html/2401.02473v1/#bib.bib3)] introduces text conditioning to obtain both image and video appearance editing results through text, while [[32](https://arxiv.org/html/2401.02473v1/#bib.bib32)] improved the atlas decomposition both in terms of reconstruction quality and computational time. Recently, Lee et al. [[23](https://arxiv.org/html/2401.02473v1/#bib.bib23)] leverages NLA representation in conjunction with an image editor and designed a framework that propagates shape edits. A deformation field is obtained with a semantic correspondence method between the input keyframe and the edited keyframe. Following that, the canonical representation of the foreground object is updated using the guidance of a T2I model through an optimization process [[34](https://arxiv.org/html/2401.02473v1/#bib.bib34)]. Despite the remarkable results, the current version lacks direct control over the shape of the final object. Additionally, as pointed out by [[23](https://arxiv.org/html/2401.02473v1/#bib.bib23)], the pipeline is sensitive to the semantic correspondence method, which is inherently difficult to achieve with precision. Any discrepancies or inaccuracies in this process are reflected in the final video output. Furthermore, NLA-based methods have a high computational cost (approximately 10 hours for a 70-frame video) and may fail in case of complex motion and self-occlusions [[20](https://arxiv.org/html/2401.02473v1/#bib.bib20)]. Drawing inspiration from [[23](https://arxiv.org/html/2401.02473v1/#bib.bib23)], we aim to replicate similar capabilities in a model that doesn’t necessitate NLA decomposition, making it a more practical alternative.

3 Method
--------

![Image 2: Refer to caption](https://arxiv.org/html/2401.02473v1/x2.png)

Figure 2: We enhance video editing by conditioning the synthesis of the video on two branches, one that controls the appearance and the other responsible for the motion and the structure of the object. To enable shape modifications, we propose a Joint Flow-Structure Augmentation pipeline that outputs an augmented flow which is processed by a Flow-Completion Network, before going as input to the final ControlNet module. Furthermore, we introduce an auxiliary loss, used to enhance the model fidelity to the input segmentation map.

VASE is based on an image-conditioned video inpainting model. Our approach preserves the original video’s motion by employing optical flow, and we introduce explicit guidance to enhance structural manipulations. We recall that the structure conditioning is provided for a single keyframe, which for simplicity we assume to be the first frame of the sequence. An overview of our pipeline is shown in [Fig.2](https://arxiv.org/html/2401.02473v1/#S3.F2 "Figure 2 ‣ 3 Method ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos").

Formally, let 𝒳∈ℝ T×3×H×W 𝒳 superscript ℝ 𝑇 3 𝐻 𝑊\mathcal{X}\in\mathbb{R}^{T\times 3\times H\times W}caligraphic_X ∈ blackboard_R start_POSTSUPERSCRIPT italic_T × 3 × italic_H × italic_W end_POSTSUPERSCRIPT be the source video and ℳ∈{0,1}T×1×H×W ℳ superscript 0 1 𝑇 1 𝐻 𝑊\mathcal{M}\in\{0,1\}^{T\times 1\times H\times W}caligraphic_M ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_T × 1 × italic_H × italic_W end_POSTSUPERSCRIPT the segmentation mask of the foreground object targeted for editing. We denote with ℳ bb∈{0,1}T×1×H×W superscript ℳ bb superscript 0 1 𝑇 1 𝐻 𝑊\mathcal{M}^{\text{bb}}\in\{0,1\}^{T\times 1\times H\times W}caligraphic_M start_POSTSUPERSCRIPT bb end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_T × 1 × italic_H × italic_W end_POSTSUPERSCRIPT a bounding box mask enclosing it. Our goal is to synthesize an edited video 𝒴∈ℝ T×3×H×W 𝒴 superscript ℝ 𝑇 3 𝐻 𝑊\mathcal{Y}\in\mathbb{R}^{T\times 3\times H\times W}caligraphic_Y ∈ blackboard_R start_POSTSUPERSCRIPT italic_T × 3 × italic_H × italic_W end_POSTSUPERSCRIPT, where the appearance of the foreground object is driven by I ref∈ℝ 3×H×W subscript 𝐼 ref superscript ℝ 3 𝐻 𝑊 I_{\text{ref}}\in\mathbb{R}^{3\times H\times W}italic_I start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT 3 × italic_H × italic_W end_POSTSUPERSCRIPT, while the structure of the object is specified by M ref∈{0,1}1×H×W subscript 𝑀 ref superscript 0 1 1 𝐻 𝑊 M_{\text{ref}}\in\{0,1\}^{1\times H\times W}italic_M start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT 1 × italic_H × italic_W end_POSTSUPERSCRIPT, which represents the shape of the edited keyframe. We define the edited region as M edit≔M ref≠ℳ 0≔subscript 𝑀 edit subscript 𝑀 ref subscript ℳ 0 M_{\text{edit}}\coloneqq M_{\text{ref}}\neq\mathcal{M}_{0}italic_M start_POSTSUBSCRIPT edit end_POSTSUBSCRIPT ≔ italic_M start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT ≠ caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT, M edit∈{0,1}1×H×W subscript 𝑀 edit superscript 0 1 1 𝐻 𝑊 M_{\text{edit}}\in\{0,1\}^{1\times H\times W}italic_M start_POSTSUBSCRIPT edit end_POSTSUBSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT 1 × italic_H × italic_W end_POSTSUPERSCRIPT.

### 3.1 Backbone

Inflated UNet. A significant challenge in video generation lies in achieving temporal consistency, which involves minimizing the flickering and inconsistencies between frames. The first step of our method is designed to tackle this problem. We follow recent works [[9](https://arxiv.org/html/2401.02473v1/#bib.bib9), [4](https://arxiv.org/html/2401.02473v1/#bib.bib4)] and inflate a pretrained diffusion model by introducing temporal layers; we refer to this model as 3D Unet. Specifically, our temporal blocks are composed of a _1D-convolution_ followed by _temporal self-attention_. The temporal blocks are placed after the original spatial blocks, and the input features are reshaped to accommodate the temporal dimension. We base our framework on Paint-by-Example (PBE) [[56](https://arxiv.org/html/2401.02473v1/#bib.bib56)], which serves as our base image-conditioned diffusion model. We adopt a similar self-supervised training strategy extended to videos and pre-train the model to reconstruct the source video 𝒳 𝒳\mathcal{X}caligraphic_X. The input to the model consists of the masked source video 𝒳⊙(1−ℳ bb)direct-product 𝒳 1 superscript ℳ bb\mathcal{X}\odot(1-\mathcal{M}^{\text{bb}})caligraphic_X ⊙ ( 1 - caligraphic_M start_POSTSUPERSCRIPT bb end_POSTSUPERSCRIPT ), and the reference object extracted from the t 𝑡 t italic_t-th frame I ref≔𝒳 t⊙ℳ t bb≔subscript 𝐼 ref direct-product subscript 𝒳 𝑡 subscript superscript ℳ bb 𝑡 I_{\text{ref}}\coloneqq\mathcal{X}_{t}\odot\mathcal{M}^{\text{bb}}_{t}italic_I start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT ≔ caligraphic_X start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ⊙ caligraphic_M start_POSTSUPERSCRIPT bb end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT. The model is trained with the ℒ simple subscript ℒ simple\mathcal{L}_{\text{simple}}caligraphic_L start_POSTSUBSCRIPT simple end_POSTSUBSCRIPT loss [[39](https://arxiv.org/html/2401.02473v1/#bib.bib39)], defined as ℒ simple=∥ϵ−ϵ θ⁢(z t)∥2 2 subscript ℒ simple superscript subscript delimited-∥∥italic-ϵ subscript italic-ϵ 𝜃 subscript 𝑧 𝑡 2 2\mathcal{L}_{\text{simple}}=\lVert\epsilon-\epsilon_{\theta}(z_{t})\rVert_{2}^% {2}caligraphic_L start_POSTSUBSCRIPT simple end_POSTSUBSCRIPT = ∥ italic_ϵ - italic_ϵ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) ∥ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT. In this first stage, the model learns to generate a temporally consistent sequence of frames.

Inflated ControlNet. Next, we introduce a conditioning mechanism to guide the generation process, maintain the motion characteristics from the source video, and simultaneously add explicit control on the shape. We extract the forward optical flow ℱ∈ℝ T−1×2×H×W ℱ superscript ℝ 𝑇 1 2 𝐻 𝑊\mathcal{F}\in\mathbb{R}^{T-1\times 2\times H\times W}caligraphic_F ∈ blackboard_R start_POSTSUPERSCRIPT italic_T - 1 × 2 × italic_H × italic_W end_POSTSUPERSCRIPT from the source video using RAFT [[47](https://arxiv.org/html/2401.02473v1/#bib.bib47)]. Then, we input to the network the structure of the keyframe M ref=ℳ 0 subscript 𝑀 ref subscript ℳ 0 M_{\text{ref}}={\mathcal{M}}_{0}italic_M start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT = caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT, _i.e_. the segmentation mask of the first frame. We condition the model by incorporating a ControlNet [[58](https://arxiv.org/html/2401.02473v1/#bib.bib58)], which takes {ℱ,M ref}ℱ subscript 𝑀 ref\{\mathcal{F},M_{\text{ref}}\}{ caligraphic_F , italic_M start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT } as input, concatenated across the channel dimension. We pad the optical flow in the temporal dimension and repeat the structure conditioning T 𝑇 T italic_T times, to ensure compatibility of dimensions. During this stage, we train the ControlNet while freezing the Inflated UNet.

This model serves as our backbone. While it can achieve appearance editing operations, it fails to perform shape editing control. During inference, if M ref≠ℳ 0 subscript 𝑀 ref subscript ℳ 0 M_{\text{ref}}\neq\mathcal{M}_{0}italic_M start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT ≠ caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT, the model tends to overlook it. We impute this to the dominant signal coming from the optical flow ℱ ℱ\mathcal{F}caligraphic_F, which can be used to minimize ℒ simple subscript ℒ simple\mathcal{L}_{\text{simple}}caligraphic_L start_POSTSUBSCRIPT simple end_POSTSUBSCRIPT without considering the structure conditioning M ref subscript 𝑀 ref M_{\text{ref}}italic_M start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT.

In the next subsections, we describe our proposed solutions to solve this problem and enable shape control at inference time.

### 3.2 Joint Flow-Structure Augmentation (JFSA)

As a first step, we propose an augmentation procedure to jointly modify the shape and the flow conditioning. As outlined in [Sec.1](https://arxiv.org/html/2401.02473v1/#S1 "1 Introduction ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"), we are focusing on edits of the foreground object, which can be reduced to two main types: adding a region or removing a region from the foreground. With _JFSA_ we aim to simulate this behavior during training by modifying the optical flow ℱ ℱ\mathcal{F}caligraphic_F and the conditioning structural mask ℳ ℳ\mathcal{M}caligraphic_M in a systematic manner. Specifically, we approximate regions of the target object that exhibit consistent motion across the T 𝑇 T italic_T frames by clustering the optical flow along the temporal dimension:

E=𝙺𝙼𝚎𝚊𝚗𝚜⁢(ℱ+λ⋅X bias,N c)𝐸 𝙺𝙼𝚎𝚊𝚗𝚜 ℱ⋅𝜆 subscript 𝑋 bias subscript 𝑁 𝑐 E=\mathtt{KMeans}(\mathcal{F}+\lambda\cdot X_{\text{bias}},\,N_{c})italic_E = typewriter_KMeans ( caligraphic_F + italic_λ ⋅ italic_X start_POSTSUBSCRIPT bias end_POSTSUBSCRIPT , italic_N start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT )(1)

Let E={ℰ 0,…,ℰ N c}𝐸 superscript ℰ 0…superscript ℰ subscript 𝑁 𝑐 E=\{\mathcal{E}^{0},\ldots,\mathcal{E}^{N_{c}}\}italic_E = { caligraphic_E start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT , … , caligraphic_E start_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT end_POSTSUPERSCRIPT } represent the set of clustered regions, where ℰ k∈{0,1}T×1×H×W superscript ℰ 𝑘 superscript 0 1 𝑇 1 𝐻 𝑊\mathcal{E}^{k}\in\{0,1\}^{T\times 1\times H\times W}caligraphic_E start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_T × 1 × italic_H × italic_W end_POSTSUPERSCRIPT denotes the binary map associated with the k 𝑘 k italic_k-th region. Here, N c subscript 𝑁 𝑐 N_{c}italic_N start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT stands for the number of clusters, X bias subscript 𝑋 bias X_{\text{bias}}italic_X start_POSTSUBSCRIPT bias end_POSTSUBSCRIPT introduces a spatial bias, and λ 𝜆\lambda italic_λ governs the trade-off.

We consider E 𝐸 E italic_E as a collection of possible ground truth shape-edits, and define two operations:

ℱ~=ℱ;ℳ~=ℳ⊙(1−ℰ k)formulae-sequence~ℱ ℱ~ℳ direct-product ℳ 1 superscript ℰ 𝑘\begin{split}\tilde{\mathcal{F}}&=\mathcal{F};\\ \tilde{\mathcal{M}}&=\mathcal{M}\odot(1-\mathcal{E}^{k})\end{split}start_ROW start_CELL over~ start_ARG caligraphic_F end_ARG end_CELL start_CELL = caligraphic_F ; end_CELL end_ROW start_ROW start_CELL over~ start_ARG caligraphic_M end_ARG end_CELL start_CELL = caligraphic_M ⊙ ( 1 - caligraphic_E start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT ) end_CELL end_ROW(2)

and,

ℱ~=ℱ⊙(1−ℰ k)+(1 H⋅W⁢∑i,j ℱ⁢(i,j))⋅ℰ k;ℳ~=ℳ formulae-sequence~ℱ direct-product ℱ 1 superscript ℰ 𝑘⋅1⋅𝐻 𝑊 subscript 𝑖 𝑗 ℱ 𝑖 𝑗 superscript ℰ 𝑘~ℳ ℳ\begin{split}\tilde{\mathcal{F}}&=\mathcal{F}\odot(1-\mathcal{E}^{k})+\Big{(}% \dfrac{1}{H\cdot W}\sum_{i,j}\mathcal{F}(i,j)\Big{)}\cdot\mathcal{E}^{k};\\ \tilde{\mathcal{M}}&=\mathcal{M}\end{split}start_ROW start_CELL over~ start_ARG caligraphic_F end_ARG end_CELL start_CELL = caligraphic_F ⊙ ( 1 - caligraphic_E start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT ) + ( divide start_ARG 1 end_ARG start_ARG italic_H ⋅ italic_W end_ARG ∑ start_POSTSUBSCRIPT italic_i , italic_j end_POSTSUBSCRIPT caligraphic_F ( italic_i , italic_j ) ) ⋅ caligraphic_E start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT ; end_CELL end_ROW start_ROW start_CELL over~ start_ARG caligraphic_M end_ARG end_CELL start_CELL = caligraphic_M end_CELL end_ROW(3)

Respectively, in [Eq.2](https://arxiv.org/html/2401.02473v1/#S3.E2 "2 ‣ 3.2 Joint Flow-Structure Augmentation (JFSA) ‣ 3 Method ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"), we address the scenario where the user intends to remove a region from the foreground object. Meanwhile, in [Eq.3](https://arxiv.org/html/2401.02473v1/#S3.E3 "3 ‣ 3.2 Joint Flow-Structure Augmentation (JFSA) ‣ 3 Method ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"), we simulate the scenario where the user aims to include an additional region in the foreground object. This procedure is efficient and can be executed on the fly during training. We randomly choose an edit region ℰ k superscript ℰ 𝑘\mathcal{E}^{k}caligraphic_E start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT and perform one of the two operations with a specified probability p augm subscript 𝑝 augm p_{\text{augm}}italic_p start_POSTSUBSCRIPT augm end_POSTSUBSCRIPT. Note that we use ℱ~,ℳ~~ℱ~ℳ\tilde{\mathcal{F}},\tilde{\mathcal{M}}over~ start_ARG caligraphic_F end_ARG , over~ start_ARG caligraphic_M end_ARG as the updated signals, replacing the original ℱ,ℳ ℱ ℳ\mathcal{F},\mathcal{M}caligraphic_F , caligraphic_M, and use them as described in [Sec.3.1](https://arxiv.org/html/2401.02473v1/#S3.SS1 "3.1 Backbone ‣ 3 Method ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos").

### 3.3 Warping Flow-Completion Net (wFCN)

Inspired by the Video Inpainting literature [[60](https://arxiv.org/html/2401.02473v1/#bib.bib60)], we introduce an additional network that explicitly predicts a _complete_ optical flow before passing it to the ControlNet. The objective of this network is to estimate the optical flow within the editing area, thus simplifying the task for the subsequent synthesis module. Unlike Video Inpainting methods, our approach involves using a single mask for the initial keyframe instead of employing individual masks for each frame in the video. To bridge this gap, we propose to employ forward warping [[30](https://arxiv.org/html/2401.02473v1/#bib.bib30)] as an initial estimation for the edited region’s location. It is important to note that although forward warping may introduce holes or inconsistencies, in our case, it is applied to a binary mask, mitigating these issues. We repeatedly warp the edited region obtaining ℰ~k=W sum⁢(ℰ 0 k,ℱ)superscript~ℰ 𝑘 subscript 𝑊 sum subscript superscript ℰ 𝑘 0 ℱ\tilde{\mathcal{E}}^{k}=W_{\text{sum}}(\mathcal{E}^{k}_{0},\mathcal{F})over~ start_ARG caligraphic_E end_ARG start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT = italic_W start_POSTSUBSCRIPT sum end_POSTSUBSCRIPT ( caligraphic_E start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , caligraphic_F ), ℰ~k∈{0,1}T×1×H×W superscript~ℰ 𝑘 superscript 0 1 𝑇 1 𝐻 𝑊\tilde{\mathcal{E}}^{k}\in\{0,1\}^{T\times 1\times H\times W}over~ start_ARG caligraphic_E end_ARG start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_T × 1 × italic_H × italic_W end_POSTSUPERSCRIPT , and W sum subscript 𝑊 sum W_{\text{sum}}italic_W start_POSTSUBSCRIPT sum end_POSTSUBSCRIPT the summation splatting operation as defined in [[30](https://arxiv.org/html/2401.02473v1/#bib.bib30)]. Note that, while at training time the ground truth temporal edit operation ℰ k superscript ℰ 𝑘\mathcal{E}^{k}caligraphic_E start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT is available, we obtain better results by warping the mask instead, to reduce the gap with inference. Conversely, during the inference phase, the source optical flow ℱ ℱ\mathcal{F}caligraphic_F may be unavailable, particularly when adding a region to the foreground object. In such cases, infilling the missing region using the displacement of the nearest known point belonging to the object provides a reliable approximation, effectively resolving the issue (see _Supp.Mat._ for details). Our Warping Flow-Completion Network (wFCN) takes as input the concatenation of the optical flow, the warped edited region, and the structural mask, and predicts the complete optical flow ℱ complete=wFCN⁢(ℰ~k,ℱ,ℳ~)subscript ℱ complete wFCN superscript~ℰ 𝑘 ℱ~ℳ\mathcal{F}_{\text{complete}}=\text{wFCN}(\tilde{\mathcal{E}}^{k},\mathcal{F},% \tilde{\mathcal{M}})caligraphic_F start_POSTSUBSCRIPT complete end_POSTSUBSCRIPT = wFCN ( over~ start_ARG caligraphic_E end_ARG start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT , caligraphic_F , over~ start_ARG caligraphic_M end_ARG ). We pre-train this network offline using our _JFSA_ procedure to obtain pairs of editing regions and optical flow. During this phase, the primary driving loss is the reconstruction loss ℒ MSE⁢(ℱ complete,ℱ~)subscript ℒ MSE subscript ℱ complete~ℱ\mathcal{L}_{\text{MSE}}(\mathcal{F}_{\text{complete}},\tilde{\mathcal{F}})caligraphic_L start_POSTSUBSCRIPT MSE end_POSTSUBSCRIPT ( caligraphic_F start_POSTSUBSCRIPT complete end_POSTSUBSCRIPT , over~ start_ARG caligraphic_F end_ARG ), along with the other regularization losses defined in [[60](https://arxiv.org/html/2401.02473v1/#bib.bib60)]. The complete optical flow ℱ complete subscript ℱ complete\mathcal{F}_{\text{complete}}caligraphic_F start_POSTSUBSCRIPT complete end_POSTSUBSCRIPT is utilized in conjunction with ℳ~~ℳ\tilde{\mathcal{M}}over~ start_ARG caligraphic_M end_ARG as input for the ControlNet.

### 3.4 Segmentation Head (SH)

To emphasize the importance of shape conditioning, we incorporate a lightweight segmentation head during the training process. More specifically, we extract the second-to-last feature of the 3D UNet and pass it through a small convolutional decoder, which receives as additional input the embedding of the diffusion timestep. We supervise the model with binary cross-entropy loss ℒ bce⁢(ℳ pred,ℳ~)subscript ℒ bce subscript ℳ pred~ℳ\mathcal{L}_{\text{bce}}(\mathcal{M}_{\text{pred}},\tilde{\mathcal{M}})caligraphic_L start_POSTSUBSCRIPT bce end_POSTSUBSCRIPT ( caligraphic_M start_POSTSUBSCRIPT pred end_POSTSUBSCRIPT , over~ start_ARG caligraphic_M end_ARG ), with ℳ pred subscript ℳ pred\mathcal{M}_{\text{pred}}caligraphic_M start_POSTSUBSCRIPT pred end_POSTSUBSCRIPT the predicted segmentation maps. We optimize it together with the main loss of the diffusion model; the total loss thus becomes ℒ tot=ℒ simple+α⋅ℒ bce subscript ℒ tot subscript ℒ simple⋅𝛼 subscript ℒ bce\mathcal{L}_{\text{tot}}=\mathcal{L}_{\text{simple}}+\alpha\cdot\mathcal{L}_{% \text{bce}}caligraphic_L start_POSTSUBSCRIPT tot end_POSTSUBSCRIPT = caligraphic_L start_POSTSUBSCRIPT simple end_POSTSUBSCRIPT + italic_α ⋅ caligraphic_L start_POSTSUBSCRIPT bce end_POSTSUBSCRIPT. We further exploit the segmentation predictions ℳ pred subscript ℳ pred\mathcal{M}_{\text{pred}}caligraphic_M start_POSTSUBSCRIPT pred end_POSTSUBSCRIPT at inference time, to preserve the background within the unpainted area, performing masked DDIM sampling [[39](https://arxiv.org/html/2401.02473v1/#bib.bib39)]. We refer to the _Supp.Mat._ for additional details.

Table 1: Quantitative results for the task of Image Driven Appearance Editing. FVD and WE have been rescaled by a factor of 10 2 superscript 10 2 10^{2}10 start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT, 10−3 superscript 10 3 10^{-3}10 start_POSTSUPERSCRIPT - 3 end_POSTSUPERSCRIPT.

![Image 3: Refer to caption](https://arxiv.org/html/2401.02473v1/x3.png)

Figure 3: Results for the Image Driven Appearance Editing task. The driver image is displayed in the bottom-left corner of the initial frame in the source video. We refer the reader to the _Supp.Mat._ for video results that better showcase the differences.

### 3.5 Implementation Details

Our foundation model is Paint-by-Example (PBE) [[56](https://arxiv.org/html/2401.02473v1/#bib.bib56)], a latent diffusion model [[39](https://arxiv.org/html/2401.02473v1/#bib.bib39)] trained for the task of image-based inpainting. We inherit the appearance conditioning mechanism from PBE, _i.e_. the driver image is embedded through CLIP [[36](https://arxiv.org/html/2401.02473v1/#bib.bib36)]. We add temporal layers to the network in the form of 1d convolution and temporal self-attention, which are initialized following previous work [[4](https://arxiv.org/html/2401.02473v1/#bib.bib4)]. The architecture of the _Flow Completion Network_ is borrowed from ProPainter [[60](https://arxiv.org/html/2401.02473v1/#bib.bib60)], and adapted to accommodate one additional channel as input to process the segmentation map. Lastly, the _Segmentation Head_ is implemented as 4 convolutional layers, which upsample the second-to-last feature of the 3D Unet decoder (_i.e_. before the final conv-out layer) by a factor of 8 8 8 8. Our training procedure can be divided into three stages: initially, we train the 3D UNet on the video dataset and keep it frozen in the next steps. Subsequently, we introduce the ControlNet, conditioning on the original optical flow and segmentation map from the source video. Finally, we integrate the pretrained _wFCN_ and the _Segmentation Head_ into the model and jointly train them along with the ControlNet using our _JFSA_ procedure.

We use YouTube-VOS [[54](https://arxiv.org/html/2401.02473v1/#bib.bib54)] for training, and resize the frames at a resolution of 448×256 448 256 448\times 256 448 × 256. We train the model for a total of 400⁢K 400 𝐾 400K 400 italic_K iterations. During the inference stage, we employ DDIM sampling with 50 steps to generate the edited video. Additionally, we integrate classifier-free guidance [[16](https://arxiv.org/html/2401.02473v1/#bib.bib16)] to enhance the quality of the final output. We refer to the _Supp.Mat._ for more comprehensive implementation details.

4 Experiments
-------------

We showcase the performances of VASE both qualitatively and quantitatively. We study two scenarios: _Image-driven Appearance Editing_, wherein the structure of the source video remains unchanged, and the edit is driven by a reference image, and _Joint Appearance-Shape Editing_, where both the object’s structure and appearance are edited simultaneously.

Table 2: Quantitative comparison with Shape-NLA [[23](https://arxiv.org/html/2401.02473v1/#bib.bib23)] on the Joint Appearance-Shape Editing task. WE has been rescaled by 10−3 superscript 10 3 10^{-3}10 start_POSTSUPERSCRIPT - 3 end_POSTSUPERSCRIPT.

![Image 4: Refer to caption](https://arxiv.org/html/2401.02473v1/x4.png)

Figure 4: Results for the Joint Appearance-Shape Editing task. Please, note that the shape edit is provided only for the first keyframe, we show all of them only for visualization purposes. We highlight the structure modifications either in red or green.

### 4.1 Image-Driven Appearance Editing

The model is tasked with editing a target object in a source video based on the guidance from a driver image. It is important to emphasize that both the structure of the object and the background should remain unchanged.

Baselines. We compare our method with the following baselines. (i) _Frame Independent Editing_: we employ PAIR Diffusion [[13](https://arxiv.org/html/2401.02473v1/#bib.bib13)] to independently edit each frame of the source video. (ii) _Multi Frame Editing_: we use FILM [[38](https://arxiv.org/html/2401.02473v1/#bib.bib38)] to perform temporal interpolation between frames edited using the frame independent baseline [[13](https://arxiv.org/html/2401.02473v1/#bib.bib13)]. A stride of 4 is utilized in this process, obtaining videos with superior temporal consistency. (iii) _Text-based Editing_: we compare with TokenFlow [[12](https://arxiv.org/html/2401.02473v1/#bib.bib12)], and use BLIP2 [[24](https://arxiv.org/html/2401.02473v1/#bib.bib24)] to caption the driver image before editing.

Metrics. We evaluate the competitors on different aspects [[26](https://arxiv.org/html/2401.02473v1/#bib.bib26)]: (i) _Video Quality_: we rely on Fréchet Video Distance (FVD) [[49](https://arxiv.org/html/2401.02473v1/#bib.bib49), [50](https://arxiv.org/html/2401.02473v1/#bib.bib50)] to compute a score for the overall video quality. Moreover, we follow recent work and compute a per-video quality score using DOVER [[52](https://arxiv.org/html/2401.02473v1/#bib.bib52)]. (ii) _Temporal Consistency_: we compute the Warping Error (WE) [[22](https://arxiv.org/html/2401.02473v1/#bib.bib22)], as the pixel-wise difference between the warped next frame and the current frame. Additionally, we measure the CLIP similarity between adjacent frames (CLIP-T) [[9](https://arxiv.org/html/2401.02473v1/#bib.bib9), [12](https://arxiv.org/html/2401.02473v1/#bib.bib12)]. Given our focus on editing a particular foreground object, our computations are confined to the region that encloses it, without considering the background. (iii) _Image Alignment_: we evaluate the faithfulness of the edited frame to the driver image; we use the cosine similarity based on CLIP [[36](https://arxiv.org/html/2401.02473v1/#bib.bib36)] (CLIP-S) and DINO [[31](https://arxiv.org/html/2401.02473v1/#bib.bib31)] (DINO-S) features, and use the LPIPS to capture more low-level details [[59](https://arxiv.org/html/2401.02473v1/#bib.bib59)].

Dataset. We create a benchmark, using 30 videos from the DAVIS dataset [[33](https://arxiv.org/html/2401.02473v1/#bib.bib33)] and selecting 3 driver images to edit each video. All the methods are evaluated at a resolution of 448×256 448 256 448\times 256 448 × 256, with 24 24 24 24 fps.

Results. We report the quantitative results in [Tab.1](https://arxiv.org/html/2401.02473v1/#S3.T1 "Table 1 ‣ 3.4 Segmentation Head (SH) ‣ 3 Method ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"). We can observe that VASE archives good results in terms of overall visual quality. Furthermore, it is generally more aligned with the driver image, missing some low-level details compared to [[13](https://arxiv.org/html/2401.02473v1/#bib.bib13)]. Lastly, while VASE shows slightly lower numbers in terms of temporal consistency, we remark that it is not the main focus of the paper, and better consistency is expected by training on a higher quality training set both in terms of fps and visual quality (_e.g_. WebVid [[2](https://arxiv.org/html/2401.02473v1/#bib.bib2)]). Analyzing the qualitative results in [Fig.3](https://arxiv.org/html/2401.02473v1/#S3.F3 "Figure 3 ‣ 3.4 Segmentation Head (SH) ‣ 3 Method ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"), we can observe how the text-level conditioning of TokenFlow sometimes misses important aspects of the driver image (_e.g_. the color of the flamingo), and leaks the edit to the background (second column). Conversely, results obtained with PAIR Diffusion exhibit strong flickering artifacts, whereas FILM enhances the temporal consistency at the expense of introducing distortions in the videos. For a more comprehensive video comparison, we direct the reader to the _Supp.Mat._.

### 4.2 Joint Appearance-Shape Editing

The objective is to alter the structure of one object in a source video, guided by the initial edited keyframe; simultaneously, a driver image is employed to edit the appearance. Note that the edit is restricted to one specific object, and the background should remain unaffected.

Baselines. We compare our method with [[23](https://arxiv.org/html/2401.02473v1/#bib.bib23)], and denote it with Shape-NLA for brevity. Specifically, we edit the key-frame with [[13](https://arxiv.org/html/2401.02473v1/#bib.bib13)]. Next, we construct the dense semantic mapping, by assigning each point in the edited area to either the closest point in the background or the foreground.

Metrics. Following the previous setting, we compare the two methods on the _Video Quality_ and _Temporal Consistency_. We use the metrics defined in [Sec.4.1](https://arxiv.org/html/2401.02473v1/#S4.SS1 "4.1 Image-Driven Appearance Editing ‣ 4 Experiments ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"), excluding FVD, as it could not be reliably computed on a limited set of samples. Furthermore, we analyze two other aspects. (i) _Shape Control_: to assess if the shape edit is reflected in the final frame. Evaluating it for the entire sequence presents challenges, primarily due to the absence of ground truths for every frame. Consequently, our assessment focuses solely on the first frame, calculating the mIoU. (ii) _Throughput_: we quantify the inference cost of the different methods, by reporting the number of frames processed per second (Hz). It is noteworthy that, for Shape-NLA [[23](https://arxiv.org/html/2401.02473v1/#bib.bib23)], the NLA decomposition step is excluded from this computation. This step demands roughly an additional 10 hours per video [[20](https://arxiv.org/html/2401.02473v1/#bib.bib20)].

Dataset. We use a subset of our benchmark used in [Sec.4.1](https://arxiv.org/html/2401.02473v1/#S4.SS1 "4.1 Image-Driven Appearance Editing ‣ 4 Experiments ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") composed of 10 videos. For each video, we manually craft a structural edit for the initial frame.

Results. We report the quantitative results on [Tab.2](https://arxiv.org/html/2401.02473v1/#S4.T2 "Table 2 ‣ 4 Experiments ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"). We can observe that Shape-NLA [[23](https://arxiv.org/html/2401.02473v1/#bib.bib23)], obtain excellent performances in term of temporal consistency thanks to the NLA decomposition. However, the faithfulness to the edited shape is inferior compared to our method, as shown by a lower mIoU. We can observe it in the qualitative results displayed in [Fig.4](https://arxiv.org/html/2401.02473v1/#S4.F4 "Figure 4 ‣ 4 Experiments ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"). Shape-NLA struggles to precisely edit the shape _e.g_. in the blackswan and bus examples. At the same time, it produces artifacts around the edited shape, _e.g_.tiger. Lastly, the NLA decomposition introduces errors in the foreground object (for instance the legs in the horsejump-low case) and distortion in the background.

### 4.3 Ablation

Table 3: Ablation study on the components of VASE.

![Image 5: Refer to caption](https://arxiv.org/html/2401.02473v1/x5.png)

Figure 5: Qualitative results obtained with different versions of our model.

We showcase results obtained with different variations of our model in [Fig.5](https://arxiv.org/html/2401.02473v1/#S4.F5 "Figure 5 ‣ 4.3 Ablation ‣ 4 Experiments ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"), and compare them quantitatively in [Tab.3](https://arxiv.org/html/2401.02473v1/#S4.T3 "Table 3 ‣ 4.3 Ablation ‣ 4 Experiments ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"). We employ the experimental setup outlined in [Sec.4.2](https://arxiv.org/html/2401.02473v1/#S4.SS2 "4.2 Joint Appearance-Shape Editing ‣ 4 Experiments ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") for the comparison. Specifically, we train three versions of our full model. In the first experiments, we keep only the backbone model as described in [Sec.3.1](https://arxiv.org/html/2401.02473v1/#S3.SS1 "3.1 Backbone ‣ 3 Method ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos")(Vers. I). We can notice that the shape modification is ignored by the model, which focuses only on the optical flow to solve the task. Secondly, we add the Joint Flow-Structure Augmentation procedure [Sec.3.2](https://arxiv.org/html/2401.02473v1/#S3.SS2 "3.2 Joint Flow-Structure Augmentation (JFSA) ‣ 3 Method ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") (Vers. II). In this case, the model produces the desired edit in the first frame but generates artifacts around the object and the edit vanishes in the subsequent frames. Next, we introduce the Flow Completion Network [Sec.3.4](https://arxiv.org/html/2401.02473v1/#S3.SS4 "3.4 Segmentation Head (SH) ‣ 3 Method ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos")(Vers. III), which results in edits that are propagated to the whole sequence but with severe artifacts around the borders. Lastly, incorporating the Segmentation Head, we show the result of our Full model, which is the only one achieving satisfactory results. We refer the reader to the _Supp.Mat._ for further analysis and details.

5 Conclusions
-------------

In this paper, we present VASE, a framework that enables users with a high degree of control over real video editing. We propose an object-centric formulation, that enables appearance modifications through a driver image and precise structural modifications with a single keyframe. While our method is able to produce a variety of satisfactory edits, it presents limitations in the presence of strong occlusions or significant changes in perspective for the target object. Furthermore, achieving consistent edits in lengthy videos remains a challenge. We identify these limitations as opportunities for future exploration, possibly incorporating 3D information to make the framework more robust.

References
----------

*   Avrahami et al. [2023] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from a single image. _arXiv preprint_, 2023. 
*   Bain et al. [2021] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In _ICCV_, 2021. 
*   Bar-Tal et al. [2022] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In _ECCV_, 2022. 
*   Blattmann et al. [2023] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In _CVPR_, 2023. 
*   Ceylan et al. [2023] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In _ICCV_, 2023. 
*   Chen et al. [2023a] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. _arXiv preprint_, 2023a. 
*   Chen et al. [2023b] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. _arXiv preprint_, 2023b. 
*   Epstein et al. [2023] Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. _arXiv preprint_, 2023. 
*   Esser et al. [2023] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In _ICCV_, 2023. 
*   Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022. 
*   Ge et al. [2023] Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In _ICCV_, 2023. 
*   Geyer et al. [2023] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. _arXiv preprint_, 2023. 
*   Goel et al. [2023] Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair-diffusion: Object-level image editing with structure-and-appearance paired diffusion models. _arXiv preprint_, 2023. 
*   Guo et al. [2023] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. _arXiv preprint_, 2023. 
*   Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. _arXiv preprint_, 2022. 
*   Ho and Salimans [2022] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. _arXiv preprint_, 2022. 
*   Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. _NeurIPS_, 2020. 
*   Ho et al. [2022] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. _arXiv preprint_, 2022. 
*   Huang et al. [2023] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. _arXiv preprint_, 2023. 
*   Kasten et al. [2021] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. _ACM TOG_, 2021. 
*   Khachatryan et al. [2023] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. _arXiv preprint_, 2023. 
*   Lai et al. [2018] Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. Learning blind video temporal consistency. In _ECCV_, 2018. 
*   Lee et al. [2023] Yao-Chih Lee, Ji-Ze Genevieve Jang, Yi-Ting Chen, Elizabeth Qiu, and Jia-Bin Huang. Shape-aware text-driven layered video editing. In _CVPR_, 2023. 
*   Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In _ICML_, 2023. 
*   Liu et al. [2023a] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. _arXiv preprint_, 2023a. 
*   Liu et al. [2023b] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. _arXiv preprint_, 2023b. 
*   Meng et al. [2021] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. _arXiv preprint_, 2021. 
*   Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. _arXiv preprint_, 2023. 
*   Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. _arXiv preprint_, 2021. 
*   Niklaus and Liu [2020] Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In _CVPR_, 2020. 
*   Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. _arXiv preprint_, 2023. 
*   Ouyang et al. [2023] Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing. _arXiv preprint_, 2023. 
*   Perazzi et al. [2016] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In _CVPR_, 2016. 
*   Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. _arXiv preprint_, 2022. 
*   Qi et al. [2023] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. _arXiv preprint_, 2023. 
*   Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In _ICML_, 2021. 
*   Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. _arXiv preprint_, 2022. 
*   Reda et al. [2022] Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame interpolation for large motion. In _ECCV_, 2022. 
*   Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In _CVPR_, 2022. 
*   Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In _CVPR_, 2023. 
*   Saharia et al. [2022a] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In _ACM MM_. ACM MM, 2022a. 
*   Saharia et al. [2022b] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. _NeurIPS_, 2022b. 
*   Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. _arXiv preprint_, 2021. 
*   Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. _arXiv preprint_, 2022. 
*   Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In _ICML_, 2015. 
*   Song and Ermon [2019] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. _NeurIPS_, 2019. 
*   Teed and Deng [2020] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In _ECCV_, 2020. 
*   Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In _CVPR_, 2023. 
*   Unterthiner et al. [2018] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. _arXiv preprint_, 2018. 
*   Unterthiner et al. [2019] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019. 
*   Wang et al. [2023] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. _arXiv preprint_, 2023. 
*   Wu et al. [2023a] Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou Hou, Annan Wang, Wenxiu Sun Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In _ICCV_, 2023a. 
*   Wu et al. [2023b] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In _ICCV_, 2023b. 
*   Xu et al. [2018] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: Sequence-to-sequence video object segmentation. In _CVPR_, 2018. 
*   Xu et al. [2023] Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Irfan Essa, and Humphrey Shi. Prompt-free diffusion: Taking” text” out of text-to-image diffusion models. _arXiv preprint_, 2023. 
*   Yang et al. [2023a] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In _CVPR_, 2023a. 
*   Yang et al. [2023b] Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. _arXiv preprint_, 2023b. 
*   Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In _ICCV_, 2023. 
*   Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In _CVPR_, 2018. 
*   Zhou et al. [2023] Shangchen Zhou, Chongyi Li, Kelvin C.K Chan, and Chen Change Loy. ProPainter: Improving propagation and transformer for video inpainting. In _ICCV_, 2023. 

\thetitle

Supplementary Material

We present additional details and results for our method. Specifically, in [Appendix A](https://arxiv.org/html/2401.02473v1/#A1 "Appendix A VASE components ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"), we delve into a more comprehensive discussion of the components of VASE. We provide visualization for the _Joint Flow-Structure Augmentation_ procedure, along with the predictions of the _Segmentation Head_ and the _Warping Flow Completion Network_. In [Appendix B](https://arxiv.org/html/2401.02473v1/#A2 "Appendix B Model and Implementation Details ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") we collect the implementation details of our method. Lastly, in [Appendix C](https://arxiv.org/html/2401.02473v1/#A3 "Appendix C Qualitative Results ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") we provide additional qualitative results of our model, both for the image-driven appearance editing and for the join structure and appearance editing task. We refer the reader to the [project page](https://helia95.github.io/vase-website/) for full video results.

Appendix A VASE components
--------------------------

In this section, we provide details about the main components of VASE and show their respective effect with qualitative results. For a quantitative comparison and an examination of the specific contribution of each component, we refer the reader to [Tab.3](https://arxiv.org/html/2401.02473v1/#S4.T3 "Table 3 ‣ 4.3 Ablation ‣ 4 Experiments ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") of the main paper.

Joint Flow-Structure Augmentation. This augmentation procedure simulates structural modifications of the target object during training. To achieve realistic edits, our approach involves clustering the optical flow along the temporal dimension. This results in regions that exhibit consistent motion within the T 𝑇 T italic_T frames. As outlined in [Eq.1](https://arxiv.org/html/2401.02473v1/#S3.E1 "1 ‣ 3.2 Joint Flow-Structure Augmentation (JFSA) ‣ 3 Method ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos"), we introduce a regularization term in the form of a spatial bias, to promote the assignment of spatially neighbor pixels to the same region. The bias is computed as

X bias⁢(i,j)=(i,j)∀i∈{0,…,H−1},∀j∈{0,…,W−1}formulae-sequence subscript 𝑋 bias 𝑖 𝑗 𝑖 𝑗 formulae-sequence for-all 𝑖 0…𝐻 1 for-all 𝑗 0…𝑊 1 X_{\text{bias}}(i,j)=(i,j)\quad\forall i\in\{0,\ldots,H-1\},\,\forall j\in\{0,% \ldots,W-1\}italic_X start_POSTSUBSCRIPT bias end_POSTSUBSCRIPT ( italic_i , italic_j ) = ( italic_i , italic_j ) ∀ italic_i ∈ { 0 , … , italic_H - 1 } , ∀ italic_j ∈ { 0 , … , italic_W - 1 }

and replicated T−1 𝑇 1 T-1 italic_T - 1 times before adding it to the optical flow ℱ ℱ\mathcal{F}caligraphic_F. We employ KMeans clustering and vary the number of cluster centroids N c subscript 𝑁 𝑐 N_{c}italic_N start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT within a random range of 6 to 10. In [Fig.6](https://arxiv.org/html/2401.02473v1/#A1.F6 "Figure 6 ‣ Appendix A VASE components ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") (a), we present a qualitative example of the JFSA procedure. Here, distinct colors signify different regions clustered by our approach.

Warping Flow Completion Network. We present details about the warping operation conducted before the Flow Completion Network. This operation is introduced to align with the original use case of Video Inpainting [[60](https://arxiv.org/html/2401.02473v1/#bib.bib60)], where a mask is supplied for each frame. Additionally, while not entirely precise, this approach facilitates the task for the network. We use summation splitting [[30](https://arxiv.org/html/2401.02473v1/#bib.bib30)] prior to feeding the inputs to the flow completion network. During inference, two cases arise. If a region is removed from the object, the original optical flow is readily available. On the other hand, if we add a region to the object, the flow is absent, leading to potential warping errors. To address this issue, we employ a displacement determined by the closest point in the object for which the flow is available. The nearest neighborhood search is based on the Euclidean distance between the points. This provides us with an approximation for the missing flow region, enabling us to warp the edit region and facilitating the subsequent task for the wFCN.

We showcase an example of the complete optical flow predicted by the wFCN in [Fig.6](https://arxiv.org/html/2401.02473v1/#A1.F6 "Figure 6 ‣ Appendix A VASE components ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") (b).

Segmentation Head. We present qualitative results of the segmentation head’s predictions during inference in [Fig.6](https://arxiv.org/html/2401.02473v1/#A1.F6 "Figure 6 ‣ Appendix A VASE components ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") (c). These masks are utilized for masked DDIM sampling, with further details provided in the subsequent subsection.

![Image 6: Refer to caption](https://arxiv.org/html/2401.02473v1/x6.png)

Figure 6: Qualitative results showcasing components of VASE. (a) Illustrates JSFA, where distinct colors signify different clustered regions. (b) Compares the original optical flow extracted from the source video (top row), with the complete optical flow predicted by the wFCN (bottom row). The edit region is shown at the bottom of the original flow. (c) Shows the corresponding predicted segmentation map.

Appendix B Model and Implementation Details
-------------------------------------------

Architecture. Our foundation model is Paint-by-Example (PBE) [[56](https://arxiv.org/html/2401.02473v1/#bib.bib56)], a latent diffusion model [[39](https://arxiv.org/html/2401.02473v1/#bib.bib39)] trained for the task of image-based inpaining. We inherit the appearance conditioning mechanism from PBE, _i.e_. the driver image is embedded through CLIP, and the obtained representation is further compressed with a linear layer which reduces its dimension, to avoid copy-paste artifacts. Next, we inflate the UNet architecture by adding temporal layers between the original spatial layers. Specifically, the original spatial layers of PBE process each frame independently, producing features with dimensions f spatial∈ℝ(bs⋅T)×dim×h×w subscript 𝑓 spatial superscript ℝ⋅bs T dim h w f_{\text{spatial}}\in\mathbb{R}^{(\text{bs}\cdot\text{T})\times\text{dim}% \times\text{h}\times\text{w}}italic_f start_POSTSUBSCRIPT spatial end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT ( bs ⋅ T ) × dim × h × w end_POSTSUPERSCRIPT. The features are reshaped to expose the temporal dimension before being forwarded to the temporal layers, _i.e_.f temporal∈ℝ(bs⋅h⋅w)×dim×T subscript 𝑓 temporal superscript ℝ⋅bs h w dim T f_{\text{temporal}}\in\mathbb{R}^{(\text{bs}\cdot\text{h}\cdot\text{w})\times% \text{dim}\times\text{T}}italic_f start_POSTSUBSCRIPT temporal end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT ( bs ⋅ h ⋅ w ) × dim × T end_POSTSUPERSCRIPT. Finally, they are reshaped back to the original spatial structure. The temporal layers consist of a 1D-Convolution with a kernel size of 3, followed by a self-attention block, with relative positional encoding to capture the temporal order. The ControlNet architecture mirrors the encoder branch of the UNet, including the newly added temporal layers. We use a singular ControlNet, which receives the optical flow and the segmentation maps concatenated across the channel dimension.

The _Flow Completion Network_ architecture is borrowed from ProPainter [[60](https://arxiv.org/html/2401.02473v1/#bib.bib60)]. We adapt the architecture and add one channel to the input layer, which receives a concatenation of the source optical flow, the warped edited region, and the segmentation mask.

The last component of our framework is the Segmentation Head, which is implemented as 4 convolutional layers. It upsamples the second-to-last feature of the 3D Unet decoder by a factor of 8×8\times 8 × and produces an output that matches the dimensions of the input frames before the VQGAN encoding [[39](https://arxiv.org/html/2401.02473v1/#bib.bib39)]. This head is trained using binary cross-entropy (bce) loss, with a weighting factor set to λ=0.05 𝜆 0.05\lambda=0.05 italic_λ = 0.05.

Training. Our training procedure can be divided into three stages. (i) Train the 3D Unet on a video dataset, in this stage the network learns to model the temporal dimension and the consistency between adjacent frames. This stage is performed for 250K iterations. (ii) Introduce the ControlNet with flow conditioning and shape control while freezing the 3D Unet, in this way, the network learns to preserve the motion from the optical flow. In this stage, the model is trained for 100K iterations. (iii) Introduce _JFSA_, _wFCN_, and the _Segmentation Head_. The Flow Completion Network is pre-trained as a stand-alone module, before plugging it into our pipeline, for which we use the same training recipe as [[60](https://arxiv.org/html/2401.02473v1/#bib.bib60)]. We perform the final joint-finetuning of the whole pipeline for an additional 50K iterations. Additionally, during training we randomly drop the conditioning, to enable classifier-free guidance at inference time. Specifically, we drop the reference image I ref subscript 𝐼 ref I_{\text{ref}}italic_I start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT with a probability of p image=0.15 subscript 𝑝 image 0.15 p_{\text{image}}=0.15 italic_p start_POSTSUBSCRIPT image end_POSTSUBSCRIPT = 0.15. At the same time, we drop the flow conditioning and the mask with a probability of p flow=0.1 subscript 𝑝 flow 0.1 p_{\text{flow}}=0.1 italic_p start_POSTSUBSCRIPT flow end_POSTSUBSCRIPT = 0.1, p mask=0.1 subscript 𝑝 mask 0.1 p_{\text{mask}}=0.1 italic_p start_POSTSUBSCRIPT mask end_POSTSUBSCRIPT = 0.1.

We train the model with a batch size of 16, across 8 NVIDIA A100 GPUs. Training takes approximately 3 days. We keep all the other hyperparameters, including the noise schedule as from Paint-by-Example [[56](https://arxiv.org/html/2401.02473v1/#bib.bib56)].

Dataset. For training we use YouTube-VOS [[54](https://arxiv.org/html/2401.02473v1/#bib.bib54)], a dataset comprising 3471 videos at 6 fps, collected from YouTube. The dataset comes with annotations of the segmentation maps, which we use during training to build our masked source video and the conditioning segmentation map. We resize the frames to a resolution of 448×256 448 256 448\times 256 448 × 256 and use sequences of 8 8 8 8 frames to train our model (_i.e_.T=8 𝑇 8 T=8 italic_T = 8). The first frame of the sequence serves as the keyframe. For evaluation purposes, we rely on DAVIS [[33](https://arxiv.org/html/2401.02473v1/#bib.bib33)] and use it to build our evaluation benchmark.

Inference. At inference, we apply the DDIM [[17](https://arxiv.org/html/2401.02473v1/#bib.bib17)] algorithm with 50 steps for all the visualization in this paper. Moreover, we use the masks predicted by the Segmentation Heads to provide a coherent background in the regions that are not edited. This is achieved with the masked DDIM sampling procedure, as described in [[39](https://arxiv.org/html/2401.02473v1/#bib.bib39)].

Finally, we enhance the visual quality of the generated results by employing classifier-free guidance [[16](https://arxiv.org/html/2401.02473v1/#bib.bib16)]. VASE is conditioned on three signals: the driver image, the complete optical flow, and the structure conditioning. Let ∅\emptyset∅ denote the null conditioning signal. Our final prediction can be expressed as:

ϵ~θ⁢(z t;I ref,ℳ,ℱ)=ϵ θ⁢(z t;∅,∅,∅)++s image⋅(ϵ θ⁢(z t;I ref,∅,∅)−ϵ θ⁢(z t;∅,∅,∅))+s mask⋅(ϵ θ⁢(z t;I ref,ℳ,∅)−ϵ θ⁢(z t;I ref,∅,∅))+s flow⋅(ϵ θ⁢(z t;I ref,ℳ,ℱ)−ϵ θ⁢(z t;I ref,ℳ,∅))subscript~italic-ϵ 𝜃 subscript 𝑧 𝑡 subscript 𝐼 ref ℳ ℱ subscript italic-ϵ 𝜃 subscript 𝑧 𝑡⋅subscript 𝑠 image subscript italic-ϵ 𝜃 subscript 𝑧 𝑡 subscript 𝐼 ref subscript italic-ϵ 𝜃 subscript 𝑧 𝑡⋅subscript 𝑠 mask subscript italic-ϵ 𝜃 subscript 𝑧 𝑡 subscript 𝐼 ref ℳ subscript italic-ϵ 𝜃 subscript 𝑧 𝑡 subscript 𝐼 ref⋅subscript 𝑠 flow subscript italic-ϵ 𝜃 subscript 𝑧 𝑡 subscript 𝐼 ref ℳ ℱ subscript italic-ϵ 𝜃 subscript 𝑧 𝑡 subscript 𝐼 ref ℳ\begin{split}\tilde{\epsilon}_{\theta}(z_{t};I_{\text{ref}},\mathcal{M},% \mathcal{F})&=\epsilon_{\theta}(z_{t};\emptyset,\emptyset,\emptyset)+\\ +&s_{\text{image}}\cdot\bigl{(}\epsilon_{\theta}(z_{t};I_{\text{ref}},% \emptyset,\emptyset)-\epsilon_{\theta}(z_{t};\emptyset,\emptyset,\emptyset)% \bigr{)}\\ +&s_{\text{mask}}\cdot\bigl{(}\epsilon_{\theta}(z_{t};I_{\text{ref}},\mathcal{% M},\emptyset)-\epsilon_{\theta}(z_{t};I_{\text{ref}},\emptyset,\emptyset)\bigr% {)}\\ +&s_{\text{flow}}\cdot\bigl{(}\epsilon_{\theta}(z_{t};I_{\text{ref}},\mathcal{% M},\mathcal{F})-\epsilon_{\theta}(z_{t};I_{\text{ref}},\mathcal{M},\emptyset)% \bigr{)}\\ \end{split}start_ROW start_CELL over~ start_ARG italic_ϵ end_ARG start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ; italic_I start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT , caligraphic_M , caligraphic_F ) end_CELL start_CELL = italic_ϵ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ; ∅ , ∅ , ∅ ) + end_CELL end_ROW start_ROW start_CELL + end_CELL start_CELL italic_s start_POSTSUBSCRIPT image end_POSTSUBSCRIPT ⋅ ( italic_ϵ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ; italic_I start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT , ∅ , ∅ ) - italic_ϵ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ; ∅ , ∅ , ∅ ) ) end_CELL end_ROW start_ROW start_CELL + end_CELL start_CELL italic_s start_POSTSUBSCRIPT mask end_POSTSUBSCRIPT ⋅ ( italic_ϵ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ; italic_I start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT , caligraphic_M , ∅ ) - italic_ϵ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ; italic_I start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT , ∅ , ∅ ) ) end_CELL end_ROW start_ROW start_CELL + end_CELL start_CELL italic_s start_POSTSUBSCRIPT flow end_POSTSUBSCRIPT ⋅ ( italic_ϵ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ; italic_I start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT , caligraphic_M , caligraphic_F ) - italic_ϵ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ; italic_I start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT , caligraphic_M , ∅ ) ) end_CELL end_ROW(4)

Throughout this paper, we keep these values fixed to s image=5 subscript 𝑠 image 5 s_{\text{image}}=5 italic_s start_POSTSUBSCRIPT image end_POSTSUBSCRIPT = 5, s mask=7 subscript 𝑠 mask 7 s_{\text{mask}}=7 italic_s start_POSTSUBSCRIPT mask end_POSTSUBSCRIPT = 7, s flow=6 subscript 𝑠 flow 6 s_{\text{flow}}=6 italic_s start_POSTSUBSCRIPT flow end_POSTSUBSCRIPT = 6.

Longer Video Prediction. Predicting long videos remains an open problem, with many approaches facing challenges in achieving satisfactory results. In this paper, our emphasis is not on improving the length of the videos or enhancing temporal consistency but rather on introducing new tools to enhance editing capabilities. Even though the model is trained for video sequences of length T=8 𝑇 8 T=8 italic_T = 8, we generate longer videos using the following approach. We condition the next batch by taking the last frame of the current batch and concatenating it in the temporal dimension to the next batch. It’s important to note that in this case, we don’t mask the first frame but provide it clean as input to the 3D Unet. This behavior is replicated during training, where with a probability of p clean=0.1 subscript 𝑝 clean 0.1 p_{\text{clean}}=0.1 italic_p start_POSTSUBSCRIPT clean end_POSTSUBSCRIPT = 0.1, we condition input to the model a clean first frame, skipping the JFSA procedure.

Appendix C Qualitative Results
------------------------------

We report additional qualitative results of VASE and the other methods as shown in the main paper. In [Fig.7](https://arxiv.org/html/2401.02473v1/#A3.F7 "Figure 7 ‣ Appendix C Qualitative Results ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos")[Fig.8](https://arxiv.org/html/2401.02473v1/#A3.F8 "Figure 8 ‣ Appendix C Qualitative Results ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") we provide additional examples for the joint appearance shape editing, while in [Fig.9](https://arxiv.org/html/2401.02473v1/#A3.F9 "Figure 9 ‣ Appendix C Qualitative Results ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos")[Fig.10](https://arxiv.org/html/2401.02473v1/#A3.F10 "Figure 10 ‣ Appendix C Qualitative Results ‣ VASE: Object-Centric Appearance and Shape Manipulation of Real Videos") we show results for the task of image-driven appearance editing.

![Image 7: Refer to caption](https://arxiv.org/html/2401.02473v1/x7.png)

Figure 7: Results for the Joint Appearance-Shape Editing task. Please, note that the shape edit is provided only for the first keyframe, we show all of them only for visualization purposes.

![Image 8: Refer to caption](https://arxiv.org/html/2401.02473v1/x8.png)

Figure 8: Results for the Joint Appearance-Shape Editing task. Please, note that the shape edit is provided only for the first keyframe, we show all of them only for visualization purposes.

![Image 9: Refer to caption](https://arxiv.org/html/2401.02473v1/x9.png)

Figure 9: Results for the Image Driven Appearance Editing task. The driver image is displayed in the bottom-left corner of the initial frame in the source video.

![Image 10: Refer to caption](https://arxiv.org/html/2401.02473v1/x10.png)

Figure 10: Results for the Image Driven Appearance Editing task. The driver image is displayed in the bottom-left corner of the initial frame in the source video.