# cosmos

> Open world models that reason about and generate video, sound and robot actions

- Repository: https://github.com/NVIDIA/cosmos
- Category: Models & local AI
- Stars: 12,107 · Forks: 924 · Language: n/a · License: not stated
- Experience: Advanced
- Page: https://ir.adyog.com/repos/nvidia/cosmos/

Robots and self-driving cars need lots of realistic data and a way to predict what happens next. Cosmos offers world models for this. They can describe what they see, or generate new video, sound and actions from text, images and other inputs.

Cosmos 3 is one omnimodal model family with two surfaces. The Reasoner takes text and vision and outputs text, for understanding and planning. The Generator takes text, vision, sound and action and outputs vision, sound and action, for simulation, future prediction, synthetic data and policy learning.

This repo is where you explore, run and evaluate the models. Training lives in the separate Cosmos Framework repo, data curation in Cosmos Curator, and output evaluation in Cosmos Evaluator. The README says outputs can show artifacts like object morphing or implausible physics, and safety-critical uses need extra validation.

## What it does

- **Three model sizes**: Super (64B) targets top quality and synthetic data. Nano (16B) balances speed and quality. Edge (4B) targets real-time use on Jetson and RTX Pro 6000.
- **Reasoner and Generator**: The Reasoner answers questions about images and video. The Generator makes video, sound and robot actions, including forward dynamics and control-to-video transfer.
- **Runnable cookbooks**: Jupyter notebooks cover audiovisual generation, action policies, transfer and reasoning, with a prompt guide.
- **Many serving backends**: Runs on Diffusers, Transformers, vLLM, vLLM-Omni, SGLang, TensorRT-LLM and NIM, including OpenAI-compatible APIs.
- **Evaluation suites**: Includes PAIBench, Physics-IQ, RBench, UniGenBench and VLMEvalKit for checking model quality.

## What you could build

- Generate a video of a robot task from a text prompt.
- Create synthetic driving scenes from a control layout with the Transfer cookbook.
- Run a robot policy such as the DROID example checkpoints on a manipulation task.
- Ask the Reasoner to spot hazards in dashcam footage.
- Serve a Cosmos model behind an OpenAI-compatible API and benchmark it.

## Who it's for

Suits robotics and autonomous-driving developers and researchers who know Python and PyTorch and have access to a recent NVIDIA GPU.

## Get started

Request access to the gated Cosmos-1.0-Guardrail repo on Hugging Face and log in with a read token. Then install the packages and run the short Diffusers script from the README. The first run downloads a 16B checkpoint, and it is slow. You can also try it in the browser at build.nvidia.com first.

```
uv venv --python 3.13 --seed --managed-python && source .venv/bin/activate
```
```
uv pip install --torch-backend=auto \
  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
  accelerate av cosmos_guardrail huggingface_hub imageio imageio-ffmpeg \
  torch torchvision transformers
```
```
uvx hf@latest auth login
```

Tags: world-models, physical-ai, robotics, video-generation, nvidia