A joint team from Peking University, CUHK, Shanghai AI Lab and NTU released VGGT‑Edit, a framework that edits native 3D representations in about five seconds, claiming up to 120× speedup over prior pipelines. The paper shows strong benchmark numbers on DeltaScene, yet the approach depends on pre‑computed Gaussian splatting structures and does not yet support arbitrary geometry or texture changes without re‑training.
What the paper claims
VGGT‑Edit is presented as a native 3D editing system that can move, delete or recolor objects in a reconstructed scene in roughly 5 seconds. The authors report a 120× speed improvement compared with earlier methods that edit per‑view images and then re‑project the changes. Benchmarks on the DeltaScene suite show higher scores for semantic consistency, multi‑view stability and latency.
What is actually new
The core idea is to bypass 2D propagation entirely. Instead of editing each rendered view and stitching the results, VGGT‑Edit operates on the underlying 3D Gaussian Splatting representation introduced by recent works such as VGGT and pi‑cubed. By modifying the Gaussian parameters (position, opacity, feature vectors) directly, the system guarantees that a change applied at one viewpoint is automatically reflected from every angle. The pipeline can be summarised as:
- Pre‑process: build a Gaussian splatting model from a set of calibrated photographs (this step still takes several minutes).
- Edit command: user specifies a semantic operation (e.g., move chair to window).
- Optimization loop: a lightweight gradient‑based update adjusts the relevant Gaussians while keeping the rest of the scene fixed.
- Render: the updated splat set is rasterised in under five seconds.
The authors also introduce a semantic mask extractor trained on a modest indoor dataset to localise objects before editing. This mask guides the gradient updates, avoiding the need for manual point selection.
Practical implications
- Gaming asset pipelines could incorporate VGGT‑Edit for rapid iteration on level geometry without exporting to a full modelling suite.
- Architectural visualisation may benefit from quick “what‑if” rearrangements of furniture during client meetings.
- VR/AR prototyping gets a modest latency reduction, making on‑the‑fly scene tweaks feasible.
Limitations and open questions
- Heavy reliance on the Gaussian representation – the method works only when the scene is already expressed as a splatting model. Converting a mesh‑based asset or a point‑cloud into this format still requires the original multi‑view capture and optimisation stage, which can take several minutes to tens of minutes.
- Texture fidelity – changing material appearance is limited to the feature vectors stored in each Gaussian. Complex BRDF edits (e.g., adding a glossy coat to a sofa) often produce washed‑out results unless the underlying network is fine‑tuned, which re‑introduces latency.
- Scalability – the reported 5‑second edit time is measured on a NVIDIA RTX 4090. On more modest hardware (e.g., a laptop GPU) the latency climbs to 20‑30 seconds, still faster than prior methods but far from real‑time interaction.
- Generalisation – the semantic mask network was trained on indoor scenes with limited object categories. Applying the system to outdoor environments or industrial CAD models would likely require additional training data.
- User interface – the paper demonstrates command‑line style edits. A production‑ready UI that lets artists drag‑and‑drop objects or paint masks is not part of the release, so integration effort remains.
How it fits into the broader trend
Recent years have seen a surge in neural rendering techniques that turn sparse photos into dense 3D representations. VGGT‑Edit is the first to tackle editing directly on such representations rather than treating reconstruction and manipulation as separate stages. However, the approach does not eliminate the need for a high‑quality reconstruction step; it merely shortens the post‑reconstruction editing loop.
Verdict
VGGT‑Edit demonstrates that native 3D editing can be much faster than the traditional 2D‑propagation pipelines, and the benchmark results are convincing for the tested indoor scenarios. The claim of “120× speedup” holds under the specific experimental setup, but the overall workflow still depends on a costly preprocessing stage and on a representation that is not yet universal. For studios already using Gaussian splatting, the framework could shave minutes off iterative design cycles. For broader adoption, future work will need to address representation‑agnostic editing, richer material control, and a more ergonomic user interface.
Links
Comments
Please log in or register to join the discussion