Suits robotics and autonomous-driving developers and researchers who know Python and PyTorch and have access to a recent NVIDIA GPU.

What can you build with cosmos?

Generate a video of a robot task from a text prompt. Create synthetic driving scenes from a control layout with the Transfer cookbook. Run a robot policy such as the DROID example checkpoints on a manipulation task. Ask the Reasoner to spot hazards in dashcam footage. Serve a Cosmos model behind an OpenAI-compatible API and benchmark it.

How do I get started with cosmos?

Request access to the gated Cosmos-1.0-Guardrail repo on Hugging Face and log in with a read token. Then install the packages and run the short Diffusers script from the README. The first run downloads a 16B checkpoint, and it is slow. You can also try it in the browser at build.nvidia.com first. Commands from the README: uv venv --python 3.13 --seed --managed-python && source .venv/bin/activate; uv pip install --torch-backend=auto \ "diffusers @ git+https://github.com/huggingface/diffusers.git" \ accelerate av cosmos_guardrail huggingface_hub imageio imageio-ffmpeg \ torch torchvision transformers; uvx hf@latest auth login.

NVIDIA /

cosmos

Open world models that reason about and generate video, sound and robot actions

12.1k stars
140 stars since 3 Oct
Updated yesterday
Advanced
Models & local AI

Open on GitHub (opens in a new tab) Website (opens in a new tab)

Robots and self-driving cars need lots of realistic data and a way to predict what happens next. Cosmos offers world models for this. They can describe what they see, or generate new video, sound and actions from text, images and other inputs.

Cosmos 3 is one omnimodal model family with two surfaces. The Reasoner takes text and vision and outputs text, for understanding and planning. The Generator takes text, vision, sound and action and outputs vision, sound and action, for simulation, future prediction, synthetic data and policy learning.

This repo is where you explore, run and evaluate the models. Training lives in the separate Cosmos Framework repo, data curation in Cosmos Curator, and output evaluation in Cosmos Evaluator. The README says outputs can show artifacts like object morphing or implausible physics, and safety-critical uses need extra validation.

What it does

Three model sizes
Super (64B) targets top quality and synthetic data. Nano (16B) balances speed and quality. Edge (4B) targets real-time use on Jetson and RTX Pro 6000.
Reasoner and Generator
The Reasoner answers questions about images and video. The Generator makes video, sound and robot actions, including forward dynamics and control-to-video transfer.
Runnable cookbooks
Jupyter notebooks cover audiovisual generation, action policies, transfer and reasoning, with a prompt guide.
Many serving backends
Runs on Diffusers, Transformers, vLLM, vLLM-Omni, SGLang, TensorRT-LLM and NIM, including OpenAI-compatible APIs.
Evaluation suites
Includes PAIBench, Physics-IQ, RBench, UniGenBench and VLMEvalKit for checking model quality.

What you could build

Generate a video of a robot task from a text prompt.
Create synthetic driving scenes from a control layout with the Transfer cookbook.
Run a robot policy such as the DROID example checkpoints on a manipulation task.
Ask the Reasoner to spot hazards in dashcam footage.
Serve a Cosmos model behind an OpenAI-compatible API and benchmark it.

Get started

Request access to the gated Cosmos-1.0-Guardrail repo on Hugging Face and log in with a read token. Then install the packages and run the short Diffusers script from the README. The first run downloads a 16B checkpoint, and it is slow. You can also try it in the browser at build.nvidia.com first.

uv venv --python 3.13 --seed --managed-python && source .venv/bin/activate

uv pip install --torch-backend=auto \
  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
  accelerate av cosmos_guardrail huggingface_hub imageio imageio-ffmpeg \
  torch torchvision transformers

uvx hf@latest auth login

Commands are copied from the project's README. Read it on GitHub for the full setup.