cactus-compute /
needle
A tiny 8-29 MB model that calls tools and extracts fields on phones and microcontrollers
- 14.2k stars
- 1.3k stars since 3 Oct
- Python
- Updated 3 days ago
- Some experience
- Models & local AI
Needle is a foundation model for small devices. The whole model is one 8-29 MB file. It gives up general chat ability so it can do a narrow set of jobs well: choose which of your app's functions to call, fill in the arguments, and return typed data from unstructured text.
You declare your tools as Python functions or as a JSON file. The model reads the user's request and returns one JSON object with function calls, its reasoning and a confidence score. A grammar built from your schemas constrains each token, so the output always parses. If no tool fits the request, you get an empty list instead of a guess.
The README says it beats models 10x its size on mobile tool calls and matches models 2-3x larger on extraction. Those figures are the authors' own benchmarks. The package also includes Whistle, a small speech-to-text model that shares the same runtime, so audio can go in and tool calls come out. You can fine-tune Needle on your own tools, either locally or on the authors' hosted platform.
What it does
Tool calling
Given your functions, it selects the right ones, fills every argument and returns several calls in order. If no tool applies, it returns an empty list.
Grammar-constrained extraction
You declare a shape and pass in messy text. A grammar compiled from your schema guarantees the output parses. It can also be used for classification.
Confidence and embeddings
Every response carries a calibrated confidence score you can use to act, confirm or refuse. The same model also returns text embeddings for local search and routing.
Whistle speech-to-text
A 16.9 MB model on the same engine that transcribes up to 30 seconds of audio in seven languages. Audio in, tool calls out, in one call.
Fine-tuning and deployment
Fine-tune locally with LoRA, or on the hosted platform for every model depth. Prebuilt engines for each platform are fetched with needle build.
What you could build
- Build a voice assistant that turns a spoken request into calls to your app's functions, fully on the device.
- Extract invoice, booking or notification fields from messy text on a phone without a network call.
- Control smart-home devices or robots by mapping short commands to a fixed set of actions.
- Search and route user queries locally using the model's text embeddings.
- Fine-tune a small 2-to-8-layer model on one product's tools to fit a very constrained device.
Get started
Install the package, then decorate a Python function with @needle.tool and create a needle.Needle(tools=[...]) agent. Call agent.run with a request and it picks the function, runs it and returns the results. Note that telemetry is on by default; set NEEDLE_TELEMETRY=0 and DO_NOT_TRACK=1 to turn it off.
pip install cactus-needleCommands are copied from the project's README. Read it on GitHub for the full setup.
More in Models & local AI
transformers
Shared model definitions for text, vision, audio and multimodal AI models
open-webui
Self-hosted chat interface for Ollama and OpenAI-compatible models, with RAG and plugins
odysseus
Self-hosted workspace for AI chat, agents, research, documents, email and calendar
minimind
Train a tiny 64M-parameter language model from scratch on one consumer GPU
Kronos
Pre-trained Transformer models that forecast candlestick (K-line) price data
airllm
Run very large open language models on a small GPU by loading one layer at a time
Vane
Self-hosted AI answer engine that searches the web and cites its sources
timesfm
Pretrained Google model that forecasts time series without training on your data
Unlimited-OCR
Open OCR model that turns images and multi-page PDFs into structured text in one pass
gallery
Android and iOS app for running open LLMs offline on your phone