microsoft /
Florence-2-large
A small vision model that captions, detects objects and reads text from images using prompts.
- 1.9k likes
- 376.5k downloads a month
- 777M parameters
- Some experience
- Vision & language
Florence-2-large takes an image plus a short task prompt such as <CAPTION>, <OD> or <OCR> and writes out text, which the processor turns into labels and bounding boxes. It was trained on FLD-5B, a dataset of 5.4 billion annotations across 126 million images, and uses a sequence-to-sequence design so one model covers many vision tasks.
The card is upfront that this particular version is a continued pretrained one with a 4k context length, trained on only 0.1B extra samples, so it might not be trained well. The OCR task was also changed to include a line separator ('\n'), and the card lists COCO OD AP as 39.8. The card compares it to much larger models, and in the fine-tuned tables those specialists often score higher, so treat it as a compact, flexible option rather than the best at every task.
What you need to run it
777M parameters. The weights take about 2 GB at full 16-bit precision (fits a 8 GB Mac or a GPU with 8 GB), or about 966 MB as a 4-bit download, which runs on a 8 GB Mac or a GPU with 8 GB. Long inputs and images need more memory on top.
What it does well
One model, many tasks
You change the prompt to switch between captioning (three levels of detail), caption-to-phrase grounding, object detection, dense region captions, region proposals, OCR and OCR with regions.
Small for what it does
At 0.77B parameters it is far smaller than models like Flamingo (80B), yet the card reports zero-shot COCO captioning CIDEr of 135.6 for large versus 84.3 for Flamingo.
Zero-shot detection and grounding scores
Zero-shot COCO detection mAP is 37.5 for large (34.7 for base), and Flickr30k R@1 is 84.4 compared with 78.7 for Kosmos-2.
Structured outputs
Detection and region tasks return bounding boxes with labels, and OCR with region returns quad boxes with the text found, which is handy for building on top.
Sizes and fine-tuned variants
The family includes base (0.23B) and large (0.77B), plus fine-tuned -ft versions of each for a wider range of downstream tasks.
What you could build
- Generate captions of different lengths for a photo collection.
- Detect objects in an image and draw the returned bounding boxes with labels.
- Find the parts of an image that match a sentence, such as 'A green car parked in front of a yellow building.'
- Read text from images or screenshots with OCR, including where each piece of text sits.
- Experiment with a multi-task vision model in a class project using the sample notebook.
Get started
Install the usual Python libraries, then copy the card's example, which loads the model and processor, picks GPU if available and otherwise CPU, and runs the <OD> prompt on a sample car photo. Change the prompt to try other tasks, or open the linked sample notebook for more examples.
model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype, trust_remote_code=True).to(device)processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)prompt = "<OD>"Commands are copied from the model card. Read it on Hugging Face for the full setup.
More vision & language models
google/gemma-4-31B-it
Google DeepMind's 31B open model that reads text and images, reasons, writes code, and calls tools.
Qwen/Qwen3.6-35B-A3B
A 35B open-weight model with a vision encoder, built mainly for coding and agent-style tasks.
Qwen/Qwen3.6-27B
A 27B open-weight model from Qwen that reads text and images, built mainly for coding work.
Qwen/Qwen2.5-VL-7B-Instruct
A 7B model that reads images, charts, documents and video, and answers in text.