sam3.cpp

sam3.cpp

State-of-the-art image and video segmentation in portable C/C++

Why sam3.cpp?

Running Meta’s Segment Anything models typically requires Python, PyTorch, and a CUDA GPU. sam3.cpp eliminates all of that. It’s a single C++ library that runs SAM 2, SAM 2.1, SAM 3, and EdgeTAM inference on CPU and Apple Metal. No Python runtime, no GPU drivers, no heavyweight dependencies. Just compile and segment.

• 4 model families: SAM 2, SAM 2.1 (Hiera), SAM 3 (ViT + text detection), EdgeTAM (RepViT, 22x faster than SAM 2 on mobile)
• 4-bit quantization: EdgeTAM in 15 MB, SAM 2.1 Tiny in 22 MB at ~1 fps on Metal, SAM 3 down to 673 MB
• Apple Metal GPU acceleration for the full backbone and transformer decoder
• Text-prompted detection (SAM 3 only): type "cat" and get every cat in the image, no clicks needed
• Point/box segmentation + video tracking with memory bank across all models
• Single-file library: sam3.cpp + sam3.h, C++14, no exceptions, no inheritance
• Zero dependencies beyond ggml and stb

Quick Start

# Clone
git clone --recursive https://github.com/PABannier/sam3.cpp
cd sam3.cpp

# Build (Metal GPU enabled automatically on macOS)
mkdir build && cd build
cmake ..
make -j

# Download a model (SAM 2.1 Tiny, 75 MB)
# See "Model Zoo" below for all available models and download links
curl -L -o ../models/sam2.1_hiera_tiny_f16.ggml \
  https://huggingface.co/PABannier/sam3.cpp/resolve/main/sam2.1_hiera_tiny_f16.ggml

# Segment an image interactively (requires SDL2)
./examples/sam3_image --model ../models/sam2.1_hiera_tiny_f16.ggml --image ../data/test_image.jpg

# Track objects in a video interactively (requires SDL2)
./examples/sam3_video --model ../models/sam2.1_hiera_tiny_f16.ggml --video ../data/test_video.mp4

The interactive apps use SDL2 + ImGui. If SDL2 isn’t found, only the benchmark and quantize tools are built.

Benchmarks

Video object tracking latency on Apple M4 Pro (24 GB), 5 frames at 1008x1008 resolution, 4 threads. Each run is isolated in a forked subprocess.

SAM 3 (Full: text detection + visual tracking)

SAM 3 Visual-Only (no text encoder, tracking only)

EdgeTAM (RepViT backbone + Perceiver)

SAM 2 / SAM 2.1 (Hiera backbone)

# Full benchmark (all models, both backends)
./build/examples/sam3_benchmark

# GPU only, all models
./build/examples/sam3_benchmark --gpu-only

# Quick iteration (tiny models, 3 frames)
./build/examples/sam3_benchmark --filter tiny --n-frames 3 --gpu-only

# CPU only, specific model
./build/examples/sam3_benchmark --cpu-only --filter sam2.1_hiera_small

Model Zoo

All models are available in GGML format on Hugging Face:

PABannier/sam3.cpp: 52 model files covering 4 architectures x multiple sizes x up to 5 precisions.

SAM 3 (850M params, ViT-32 backbone + text encoder + DETR decoder)

SAM 2 / SAM 2.1 (Hiera backbone, 4 sizes)

EdgeTAM (RepViT-M1 backbone + Perceiver memory compressor)

Feature Matrix

Building from Source

Prerequisites

• C++14 compiler (Clang, GCC, MSVC)
• CMake 3.14+
• (Optional) SDL2 for the interactive image/video examples
• (Optional) ffmpeg for video frame decoding

Build

git clone --recursive https://github.com/PABannier/sam3.cpp
cd sam3.cpp
mkdir build && cd build
cmake ..
make -j

Metal is enabled automatically on macOS. To disable it:

cmake .. -DSAM3_METAL=OFF

To build tests:

cmake .. -DSAM3_BUILD_TESTS=ON
make -j

Usage

Image Segmentation (Interactive GUI)

# Point/box segmentation with any model
./sam3_image --model models/sam2.1_hiera_tiny_f16.ggml --image photo.jpg

# Text-prompted detection (SAM 3 only)
./sam3_image --model models/sam3-f16.ggml --image photo.jpg
# → Type "cat" in the text field, click [Segment]

Controls:

• Left-click: add positive point
• Right-click: add negative point
• Drag: draw bounding box
• Text field + Segment: detect all instances matching the text prompt (SAM 3 only)
• Export: save masks as PNG

Video Tracking (Interactive GUI)

# Visual tracking (SAM 2/2.1/3/EdgeTAM)
./sam3_video --model models/sam2.1_hiera_small_f16.ggml --video input.mp4

# Text-prompted tracking (SAM 3 only)
./sam3_video --model models/sam3-f16.ggml --video input.mp4

Controls:

• Click a point or draw a box on a paused frame to add an instance
• Click on an existing tracked mask to refine it
• Play/Pause/Step for playback
• Export per-frame mask PNGs

C++ API

#include "sam3.h"

// Load model
sam3_params params;
params.model_path = "models/sam2.1_hiera_tiny_f16.ggml";
params.use_gpu    = true;
params.n_threads  = 4;

auto model = sam3_load_model(params);
auto state = sam3_create_state(*model, params);

// Encode image (call once, reuse for multiple prompts)
auto image = sam3_load_image("photo.jpg");
sam3_encode_image(*state, *model, image);

// Segment with a point click
sam3_pvs_params pvs;
pvs.pos_points.push_back({315.0f, 250.0f});
sam3_result result = sam3_segment_pvs(*state, *model, pvs);

for (auto& det : result.detections) {
    sam3_save_mask(det.mask, "mask.png");
}

// Text-prompted detection (SAM 3 full model only)
sam3_pcs_params pcs;
pcs.text_prompt     = "yellow school bus";
pcs.score_threshold = 0.5f;
sam3_result result = sam3_segment_pcs(*state, *model, pcs);
// → result.detections contains every matching instance

// Video tracking
auto tracker = sam3_create_visual_tracker(*model, {});

// Frame 0: encode + add instance with a click
sam3_encode_image(*state, *model, frame0);
sam3_pvs_params pvs;
pvs.pos_points.push_back({315.0f, 250.0f});
sam3_tracker_add_instance(*tracker, *state, *model, pvs);

// Subsequent frames: propagate masks
for (int f = 1; f < n_frames; f++) {
    sam3_result result = sam3_propagate_frame(*tracker, *state, *model, frames[f]);
    // result.detections[i].mask - tracked mask for each instance
}

Quantization

Convert F32/F16 weights to smaller quantized formats:

./sam3_quantize models/sam3-f16.ggml models/sam3-q4_0.ggml q4_0
# Supported types: q4_0, q4_1, q8_0

Converting Weights

Convert official PyTorch checkpoints to GGML format:

# SAM 3
uv run python convert_sam3_to_ggml.py \
    --model sam3.pt \
    --output models/sam3-f16.ggml \
    --ftype 1 \
    --tokenizer /path/to/tokenizer

# SAM 3 visual-only (no text encoder, smaller file)
uv run python convert_sam3_to_ggml.py \
    --model sam3.pt \
    --output models/sam3-visual-f16.ggml \
    --ftype 1 \
    --visual-only

# SAM 2 / SAM 2.1
uv run python convert_sam2_to_ggml.py \
    --model sam2.1_hiera_large.pt \
    --config sam2.1_hiera_l.yaml \
    --output models/sam2.1_hiera_large_f16.ggml \
    --ftype 1

# EdgeTAM
uv run python convert_edgetam_to_ggml.py \
    --model edgetam.pt \
    --output models/edgetam_f16.ggml \
    --ftype 1

--ftype 0 = float32, --ftype 1 = float16 (recommended). Then quantize with sam3_quantize.

Acknowledgments

• Meta AI Research for SAM, SAM 2, SAM 3, and EdgeTAM
• ggml, the tensor computation library that makes this possible
• sam.cpp, the original SAM 1 C++ port that inspired this project’s architecture

License

MIT