Robbyant /
lingbot-map
Feed-forward model that builds 3D scenes from video frames as they stream in
- 18.2k stars
- 989 stars since 3 Oct
- Python
- Updated 8 days ago
- Advanced
- Models & local AI
LingBot-Map reconstructs 3D scenes from a stream of images or a video. It is a feed-forward model: each new frame goes through the network and updates the 3D result directly, instead of running a slow optimization over the whole sequence afterward.
To keep long sequences stable, it uses a Geometric Context Transformer. The README describes three kinds of memory: an anchor context for grounding coordinates, a pose-reference window for dense local cues, and a trajectory memory that corrects drift over time. Attention uses a paged KV cache through FlashInfer, with a PyTorch SDPA fallback.
The authors report about 20 FPS at 518x378 resolution on sequences over 10,000 frames, and say it beats other streaming and optimization-based methods on their benchmarks. There are limits: it was trained with video RoPE on 320 views, so longer inputs need a keyframe interval or windowed mode, and very long runs without state resetting can see pose collapse.
What it does
Streaming 3D reconstruction
Takes an image folder or a video and produces a point cloud and camera poses frame by frame, shown in a browser viewer built on viser.
Long-sequence controls
A keyframe interval limits KV cache size, and a windowed mode with window size and overlap settings handles sequences beyond about 3,000 frames.
Sky masking
An optional ONNX sky segmentation model removes sky points from outdoor scenes. Masks are cached for later runs.
Memory and speed options
Offloading predictions to CPU, fewer scale frames, fewer camera refinement iterations and an SDPA fallback help on limited GPUs.
Benchmarks and offline rendering
Includes evaluation scripts for datasets such as KITTI and Oxford Spires, and a batch pipeline for rendering very long videos.
What you could build
- Reconstruct a 3D point cloud and camera path from a walkthrough video.
- Explore the bundled courthouse, university and loop example scenes in a browser viewer.
- Render a very long indoor or outdoor video offline with the batch pipeline.
- Compare streaming reconstruction against your own method using the KITTI and Oxford Spires evaluation scripts.
- Filter sky points out of outdoor reconstructions for cleaner visualization.
Get started
Create a conda environment with Python 3.10, install PyTorch for CUDA 12.8, then install the repo with pip and optionally FlashInfer. Download the lingbot-map checkpoint from Hugging Face or ModelScope and run demo.py on an example folder, then open http://localhost:8080.
conda create -n lingbot-map python=3.10 -ypip install -e .python demo.py --model_path /path/to/lingbot-map.pt \
--image_folder example/courthouse --mask_skyCommands are copied from the project's README. Read it on GitHub for the full setup.
More in Models & local AI
transformers
Shared model definitions for text, vision, audio and multimodal AI models
open-webui
Self-hosted chat interface for Ollama and OpenAI-compatible models, with RAG and plugins
odysseus
Self-hosted workspace for AI chat, agents, research, documents, email and calendar
minimind
Train a tiny 64M-parameter language model from scratch on one consumer GPU
Kronos
Pre-trained Transformer models that forecast candlestick (K-line) price data
airllm
Run very large open language models on a small GPU by loading one layer at a time
Vane
Self-hosted AI answer engine that searches the web and cites its sources
timesfm
Pretrained Google model that forecasts time series without training on your data
Unlimited-OCR
Open OCR model that turns images and multi-page PDFs into structured text in one pass
gallery
Android and iOS app for running open LLMs offline on your phone