# lingbot-map

> Feed-forward model that builds 3D scenes from video frames as they stream in

- Repository: https://github.com/Robbyant/lingbot-map
- Category: Models & local AI
- Stars: 18,178 · Forks: 1,934 · Language: Python · License: Apache-2.0
- Experience: Advanced
- Page: https://ir.adyog.com/repos/robbyant/lingbot-map/

LingBot-Map reconstructs 3D scenes from a stream of images or a video. It is a feed-forward model: each new frame goes through the network and updates the 3D result directly, instead of running a slow optimization over the whole sequence afterward.

To keep long sequences stable, it uses a Geometric Context Transformer. The README describes three kinds of memory: an anchor context for grounding coordinates, a pose-reference window for dense local cues, and a trajectory memory that corrects drift over time. Attention uses a paged KV cache through FlashInfer, with a PyTorch SDPA fallback.

The authors report about 20 FPS at 518x378 resolution on sequences over 10,000 frames, and say it beats other streaming and optimization-based methods on their benchmarks. There are limits: it was trained with video RoPE on 320 views, so longer inputs need a keyframe interval or windowed mode, and very long runs without state resetting can see pose collapse.

## What it does

- **Streaming 3D reconstruction**: Takes an image folder or a video and produces a point cloud and camera poses frame by frame, shown in a browser viewer built on viser.
- **Long-sequence controls**: A keyframe interval limits KV cache size, and a windowed mode with window size and overlap settings handles sequences beyond about 3,000 frames.
- **Sky masking**: An optional ONNX sky segmentation model removes sky points from outdoor scenes. Masks are cached for later runs.
- **Memory and speed options**: Offloading predictions to CPU, fewer scale frames, fewer camera refinement iterations and an SDPA fallback help on limited GPUs.
- **Benchmarks and offline rendering**: Includes evaluation scripts for datasets such as KITTI and Oxford Spires, and a batch pipeline for rendering very long videos.

## What you could build

- Reconstruct a 3D point cloud and camera path from a walkthrough video.
- Explore the bundled courthouse, university and loop example scenes in a browser viewer.
- Render a very long indoor or outdoor video offline with the batch pipeline.
- Compare streaming reconstruction against your own method using the KITTI and Oxford Spires evaluation scripts.
- Filter sky points out of outdoor reconstructions for cleaner visualization.

## Who it's for

Suits computer vision researchers and students with a CUDA GPU who are comfortable with PyTorch and conda, and who know basic ideas like camera pose and point clouds.

## Get started

Create a conda environment with Python 3.10, install PyTorch for CUDA 12.8, then install the repo with pip and optionally FlashInfer. Download the lingbot-map checkpoint from Hugging Face or ModelScope and run demo.py on an example folder, then open http://localhost:8080.

```
conda create -n lingbot-map python=3.10 -y
```
```
pip install -e .
```
```
python demo.py --model_path /path/to/lingbot-map.pt \
    --image_folder example/courthouse --mask_sky
```

Tags: 3d-reconstruction, computer-vision, transformer, slam, pytorch