Cactus
A hybrid edge-cloud AI engine for mobile devices & wearables.
┌─────────────────┐
│ Cactus Engine │ ←── OpenAI-compatible APIs for text, speech, and vision.
└─────────────────┘
│
┌─────────────────┐
│ Cactus Graph │ ←── Zero-copy computation graph
└─────────────────┘
│
┌─────────────────┐
│ Cactus Kernels │ ←── CPU/GPU kernels for (Apple, Samsung, Pixel, etc.)
└─────────────────┘
│
┌─────────────────┐
│ Cactus Quants │ ←── Custom rotation-based quantization technique
└─────────────────┘
Quick Demo (Mac)
- Step 1:
brew install cactus-compute/cactus/cactus
- Step 2:
cactus run
Cactus Engine
#include "cactus_engine.h"
cactus_model_t model = cactus_init(
"path/to/weight/folder",
"path to txt or dir of txts for auto-rag",
false
);
const char* messages = R"([
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "My name is Henry Ndubuaku"}
])";
const char* options = R"({
"max_tokens": 50,
"stop_sequences": ["<|im_end|>"]
})";
char response[4096];
int result = cactus_complete(
model, // model handle
messages, // JSON chat messages
response, // response buffer
sizeof(response), // buffer size
options, // generation options
nullptr, // tools JSON
nullptr, // streaming callback
nullptr, // user data
nullptr, // pcm audio buffer
0 // pcm buffer size
);
Example response from Gemma4-E2B
{
"success": true, // generation succeeded
"error": null, // error details if failed
"cloud_handoff": false, // true if cloud model used
"response": "Hi there!",
"function_calls": [], // parsed tool calls
"segments": [], // transcription segments (empty for chat)
"confidence": 0.8193, // model confidence
"confidence_threshold": 0.7, // resolved handoff threshold (model-dependent)
"time_to_first_token_ms": 45.23,
"total_time_ms": 163.67,
"prefill_tps": 1621.89,
"decode_tps": 168.42,
"ram_usage_mb": 245.67,
"prefill_tokens": 28,
"decode_tokens": 50,
"total_tokens": 78
}
Cactus Graph
#include "cactus_graph.h"
CactusGraph graph;
auto a = graph.input({2, 3}, Precision::FP16);
auto b = graph.input({3, 4}, Precision::INT8);
auto x1 = graph.matmul(a, b, false);
auto x2 = graph.transpose(x1);
auto result = graph.matmul(b, x2, true);
float a_data[6] = {1.1f, 2.3f, 3.4f, 4.2f, 5.7f, 6.8f};
float b_data[12] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};
graph.set_input(a, a_data, Precision::FP16);
graph.set_input(b, b_data, Precision::INT8);
graph.execute();
void* output_data = graph.get_output(result);
graph.hard_reset();
Inference Speed
- LLM: Gemma-4-E2B-CQ4 (1k-context prefill / decode for 100 tokens)
- VLM: Gemma-4-E2B-CQ4 (256px image encode time / decode)
- Transcribe: Parakeet-TDT-0.6B-CQ4 (20s audio end-to-end transcribe time)
- 1k-Context RAM: peak MB during the LLM benchmark
- No speculative decode or MTP, pure decode
Command: cactus benchmark [optional --ios or --android]
| Device |
LLM |
VLM |
Transcribe |
RAM |
| Mac M5 Max |
2964tps / 154tps |
0.09s / 168tps |
0.15s |
1348MB |
| Mac M4 Pro |
1963tps / 101tps |
0.25s / 112tps |
0.21s |
1225MB |
| Mac M3 Pro |
1294tps / 64tps |
0.40s / 72tps |
0.37s |
735MB |
| iPad/Vision Pro M5 |
1336tps / 71tps |
0.25s / 80tps |
0.27s |
703MB |
| iPhone 17 Pro |
729tps / 37tps |
0.5s / 39tps |
0.51s |
644MB |
| iPhone 15 Pro |
517tps / 26tps |
1.15s / 27tps |
0.82s |
633MB |
N/B: With 1k-context prefill and decode for 100 runs on M5 Max
LFM2.5-VL-1.6B = 289toks/secQwen3-1.7B = 155toks/secLFM2.5-VL-450m = 472toks/sec, image encodes in 43msLFM22.5-VL-230m = 555toks/sec
Output Quality
- Gemma-4-E2B-it accuracy across bit widths, averaged over 3 seeds.
- CQ3.26 and CQ2.54 are mixed-precision, CQ2/CQ3/CQ4 are uniformly quantized.
- Full results in docs/cactus_quants.md:
| Task |
F16 (Original) |
CQ4 |
CQ3.26 |
CQ2.54 |
CQ2 |
| ARC-E |
73.80 |
73.73 |
74.20 |
68.20 |
50.80 |
| ARC-C |
56.47 |
52.47 |
51.53 |
37.20 |
24.73 |
| HellaSwag |
46.93 |
47.07 |
45.20 |
40.73 |
35.87 |
| WinoGrande |
61.00 |
61.13 |
59.60 |
60.13 |
51.27 |
| MMLU |
62.33 |
59.45 |
57.63 |
47.19 |
33.18 |
| GPQA |
34.34 |
34.34 |
31.82 |
30.81 |
23.23 |
| GSM8K |
73.67 |
71.20 |
66.20 |
22.00 |
0.40 |
| HumanEval |
54.88 |
57.11 |
53.66 |
15.24 |
1.02 |
| BFCL Simple |
92.00 |
92.42 |
91.50 |
82.25 |
18.75 |
| BFCL Multi |
89.00 |
88.33 |
89.00 |
52.50 |
13.67 |
| BFCL Parallel |
84.00 |
83.67 |
82.50 |
30.00 |
3.33 |
| BFCL Parallel-Multi |
78.00 |
83.33 |
82.00 |
37.00 |
1.33 |
Supported Models
- Any HuggingFace model can be converted using
cactus convert [HF-Name], though experimental.
- Liquid, Gemma. whisper. parakeet and Qwen model families are especially tested.
- Some models have been pre-uploaded here, just run
cactus download [HF-Name].
cactus run [HF-Name] albeit first downloads or convert the model if not found.
Needle
Needle is a 26m parameter model for on-device tool calling:
cactus run Cactus-Compute/needle [--tools my_tools.json] # OpenAI function-calling format; demo toolset by default
Learn More
| Reference |
Language |
Description |
| Cactus Engine |
C |
Chat completion, streaming, tool calling, transcription, embeddings, RAG, vision, vector index, cloud handoff |
| Cactus Graph |
C++ |
Tensor operations, matrix multiplication, attention, normalization, activation functions |
| Cactus Kernels |
C++ |
ARM NEON SIMD kernels for matmul, attention, convolution, quantization, DSP, image processing |
| Cactus Quants |
C++ |
Rotation-and-codebook quantization from 4-bit to 1-bit for all weight tensors |
| Cactus Hybrid |
C/Python |
Route hard queries to the cloud automatically based on local model confidence |
| Python Package |
Python |
Python package and CLI |
Bindings
Using this repo
┌────────────────────────────────────────────────────────────────────────────────┐
│ │
│ Step 0: if on Linux (Ubuntu/Debian) │
│ sudo apt-get install python3.12 python3.12-venv python3-pip cmake │
│ build-essential libcurl4-openssl-dev │
│ │
│ Step 1: clone and setup │
│ git clone https://github.com/cactus-compute/cactus && cd cactus │
│ source ./setup │
│ │
│ Step 2: use the commands │
│────────────────────────────────────────────────────────────────────────────────│
│ │
│ cactus auth manage cloud API key │
│ --status show key status │
│ --clear remove saved key │
│ │
│ cactus run [model|path] run a model (downloads if needed) │
│ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │
│ --backend cpu|metal inference backend (default: auto) │
│ --image <path> image file for VLM inference │
│ --audio <path> audio file for audio chat │
│ --system <prompt> system prompt │
│ --prompt <text> send prompt immediately │
│ --tools <json|file> tool definitions for tool calling │
│ --thinking enable thinking/reasoning mode │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ │
│ cactus transcribe [model] live microphone transcription with a model│
│ --file <audio.wav> audio file to transcribe (WAV) │
│ --language <code> language code (default: en) │
│ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ │
│ cactus download [model] get a bundle (prebuilt, else build) │
│ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert refresh cached bundle │
│ │
│ cactus convert <model> [dir] HuggingFace -> Cactus CQ weights │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force weight conversion from source │
│ --lora <path> merge a LoRA adapter before converting │
│ --weights-only stop after CQ weights (skip the graph) │
│ --artifact-dir <path> bundle output (default: weights/<model>) │
│ │
│ cactus serve [model] OpenAI-compatible local HTTP server │
│ --host <addr> bind address (default: 127.0.0.1) │
│ --port <port> port (default: 8080) │
│ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │
│ --backend cpu|metal inference backend (default: auto) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ --no-cloud-handoff disable automatic cloud handoff │
│ --confidence-threshold <0..1> handoff to cloud below this confidence │
│ --cloud-timeout-ms <n> max wait for cloud handoff │
│ │
│ cactus code run the AI coding agent (TUI / print) │
│ --serve-model <id> auto-start a server with this model │
│ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │
│ --backend cpu|metal inference backend (default: auto) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ --host <addr> server address (default: 127.0.0.1) │
│ --port <port> server port (default: 8080) │
│ --no-serve require a running server (no auto-start) │
│ --no-cloud-handoff disable automatic cloud handoff │
│ --confidence-threshold <0..1> handoff to cloud below this confidence │
│ --cloud-timeout-ms <n> max wait for cloud handoff │
│ -- <args...> pass remaining args to the agent │
│ │
│ cactus list list downloaded models │
│ │
│ cactus build build cactus libraries │
│ --apple Apple (iOS/macOS) │
│ --android Android │
│ --python shared lib for Python FFI │
│ │
│ cactus test run the test suite │
│ --component <name> kernels | graph | engine | all │
│ (default: all) │
│ --model <hf-id> default: google/gemma-4-E2B-it │
│ --transcription-model <hf-id> default: nvidia/parakeet-tdt-0.6b-v3 │
│ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │
│ --backend cpu|metal inference backend (default: auto) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild of test models │
│ --suite <name> run a single test suite by name │
│ (resolved across components, │
│ e.g. llm → engine) │
│ --list list components and suites │
│ --ios run on connected iPhone │
│ --android run on connected Android │
│ --enable-telemetry send cloud telemetry (off by default) │
│ │
│ cactus benchmark run the engine benchmark suite │
│ --model <hf-id> default: google/gemma-4-E2B-it │
│ --transcription-model <hf-id> default: nvidia/parakeet-tdt-0.6b-v3 │
│ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │
│ --backend cpu|metal inference backend (default: auto) │
│ --ios run on connected iPhone │
│ --android run on connected Android │
│ │
│ cactus clean delete build artifacts, weights, venv │
│ cactus --help show this help │
│ │
└────────────────────────────────────────────────────────────────────────────────┘
Citation
If you use Cactus in your research, please cite it as follows:
@software{cactus,
title = {Cactus: AI Inference Engine for Phones & Wearables},
author = {Ndubuaku, Henry and Cactus Team},
url = {https://github.com/cactus-compute/cactus},
year = {2025}
}
N/B: Scroll all the way up and click the shields link for resources!
评论