microsoft /

Phi-4-multimodal-instruct

A small open model that reads text, images and speech, and answers in text.

1.6k likes
111.5k downloads a month
5.6B parameters
Some experience
Speech

Open on Hugging Face (opens in a new tab)

Phi-4-multimodal-instruct is a lightweight open multimodal model from Microsoft. You can give it text, images or audio, and it replies with text. Earlier Phi models needed two steps for speech: one model to turn audio into text, and another to do the thinking. Here a single network handles text, vision and audio together, so the model sees the audio itself and not just a transcript.

It has limits worth knowing. Text works in 23 languages, but vision is English only, and audio covers English, Chinese, German, French, Italian, Japanese, Spanish and Portuguese. On speech question answering it still trails closed models such as Gemini-1.5-Flash and GPT-4o-realtime-preview, and Microsoft says it is working on that. The card also says the model is not evaluated for every use, so you should test accuracy, safety and fairness yourself, especially in high-risk settings.

What you need to run it

5.6B parameters. The weights take about 11 GB at full 16-bit precision (fits a 16 GB Mac or a GPU with 16 GB), or about 4 GB as a 4-bit download, which runs on a 8 GB Mac or a GPU with 8 GB. Long inputs and images need more memory on top.

What it does well

Strong speech recognition
The card reports a word error rate of 6.14% on the Hugging Face OpenASR leaderboard, ranking first as of March 04, 2025, against 6.5% for the next best model. It also says it beats WhisperV3 for speech recognition and SeamlessM4T-v2-Large for speech translation.
Speech summarization
The card calls it the first open-sourced model that can summarize speech, with performance close to GPT4o.
Reads charts, tables and documents
It scores 93.2 on DocVQA, 81.4 on ChartQA and 84.4 on OCR Bench, and the card lists OCR and chart and table understanding as intended uses.
Image and audio together
It can take an image plus a spoken question. On the card's speech-query chart and document tests it averages 72.2, against 62.6 for InternOmni-7B and 66.2 for Gemini-2.0-Flash.
Made for limited hardware
The card lists memory- and compute-constrained environments and latency-bound scenarios as intended uses. It also lists function and tool calling, and strong reasoning in math and logic.

What you could build

Transcribe recorded lectures or interviews, then ask the model to summarize them.
Build a speech translator that listens in one language and writes in another (the card's widget example asks for a transcript and a French translation, separated by <sep>).
Make a tool that reads photos of documents, charts or tables and answers questions about them.
Compare several images, or summarize a short video clip given as frames, in a small app.
Experiment with function and tool calling in an assistant that takes voice input.

Get started

The card excerpt I was given does not include any install or code commands. The quickest way to try it is one of the hosted playgrounds linked on the card (Azure, GitHub, Nvidia or Hugging Face), or the Hugging Face Space. To run it yourself, see the model's page on Hugging Face, which uses the transformers library.

Commands are copied from the model card. Read it on Hugging Face for the full setup.