Recent advances in Text-to-Video generation (T2V) have achieved remarkable success
in synthesizing high-quality general videos from textual descriptions. A largely
overlooked problem in T2V is that existing models have not adequately encoded physical
knowledge of the real world, thus generated videos tend to have limited motion and poor
variations. In this paper, we propose MagicTime, a metamorphic time-lapse video
generation model, which learns real-world physics knowledge from time-lapse videos and
implements metamorphic generation. First, we design a MagicAdapter scheme to decouple
spatial and temporal training, encode more physical knowledge from metamorphic videos,
and transform pre-trained T2V models to generate metamorphic videos. Second, we introduce
a Dynamic Frames Extraction strategy to adapt to metamorphic time-lapse videos, which have
a wider variation range and cover dramatic object metamorphic processes, thus embodying more
physical knowledge than general videos. Finally, we introduce a Magic Text-Encoder to improve
the understanding of metamorphic video prompts. Furthermore, we create a time-lapse video-text
dataset called ChronoMagic, specifically curated to unlock the metamorphic video
generation ability. Extensive experiments demonstrate the superiority and effectiveness of
MagicTime for generating high-quality and dynamic metamorphic videos, suggesting time-lapse
video generation is a promising path toward building metamorphic simulators of the physical world.
