optillm

OptiLLM

2-10x accuracy improvements on reasoning tasks with zero training

HuggingFace Space | Colab Demo | Discussions

OptiLLM is an OpenAI API-compatible optimizing inference proxy that implements 20+ state-of-the-art techniques to dramatically improve LLM accuracy and performance on reasoning tasks, without requiring any model training or fine-tuning.

It is possible to beat the frontier models using these techniques across diverse tasks by doing additional compute at inference time. A good example of how to combine such techniques is the CePO approach from Cerebras.

Key features

• Instant improvements: 2-10x better accuracy on math, coding, and logical reasoning
• Drop-in replacement: Works with any OpenAI-compatible API endpoint
• 20+ optimization techniques: From simple best-of-N to advanced MCTS and planning
• Zero training required: Just proxy your existing API calls through OptiLLM
• Production ready: Used in production by companies and researchers worldwide
• Multi-provider: Supports OpenAI, Anthropic, Google, Cerebras, and 100+ models via LiteLLM

Quick start

pip install optillm
export OPENAI_API_KEY="your-key-here"
optillm

Point any OpenAI client at the proxy and prefix the model name with a technique:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="moa-gpt-4o-mini",  # This gives you GPT-4o performance from GPT-4o-mini
    messages=[{"role": "user", "content": "Solve: If 2x + 3 = 7, what is x?"}],
)

Before OptiLLM: “x = 1”
After OptiLLM: “Let me work through this step by step: 2x + 3 = 7, so 2x = 4, therefore x = 2”

Results

See benchmark results for the full tables.

Installation

pip

pip install optillm

Docker

docker pull ghcr.io/algorithmicsuperintelligence/optillm:latest
docker run -p 8000:8000 ghcr.io/algorithmicsuperintelligence/optillm:latest

To build and run from the repository, set your keys in docker-compose.yaml or a .env file and run docker compose up -d.

From source

git clone https://github.com/algorithmicsuperintelligence/optillm.git
cd optillm
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python optillm.py

Providers

Set the environment variable for your provider and the proxy picks the matching client.

Other models work through LiteLLM. For example, set GEMINI_API_KEY and call moa-gemini/gemini-1.5-flash-002.

Usage

Selecting a technique

When the server runs with the default auto approach, a technique can be selected in three ways, in this order of preference:

# 1. Model name prefix
client.chat.completions.create(model="moa-gpt-4o-mini", messages=messages)

# 2. extra_body field
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    extra_body={"optillm_approach": "bon|moa|mcts"},
)

# 3. Tag in the system or user prompt
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "<optillm_approach>re2</optillm_approach> How many r's are there in strawberry?"}],
)

If the server is started with a specific --approach, set model to the model name only.

Combining techniques

• & runs techniques as a pipeline from left to right, using each response as the request to the next stage: cot_reflection&moa
• | runs techniques in parallel and returns the responses as a list: bon|moa|mcts

How requests flow

• A is an existing tool, framework, or your own code using any OpenAI client SDK.
• B is the optillm service, which sends requests to the configured base_url.
• C is any service providing an OpenAI-compatible chat completions endpoint.

For an LLM without an OpenAI-compatible endpoint, you can use the LiteLLM proxy server.

Local inference

Set OPTILLM_API_KEY to any value to use the built-in inference server, and use the same value in your client. Any HuggingFace model can be passed as the model name; set HF_TOKEN for private models. Add LoRA adapters with the + separator and choose one with active_adapter (the last one is used by default).

client = OpenAI(api_key="optillm", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-1B-Instruct+patched-codes/Llama-3.2-1B-FastApply+patched-codes/Llama-3.2-1B-FixVulns",
    messages=messages,
    temperature=0.2,
    logprobs=True,
    top_logprobs=3,
    extra_body={"active_adapter": "patched-codes/Llama-3.2-1B-FastApply"},
)

Alternate decoding techniques are available on the local server:

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-1B-Instruct",
    messages=messages,
    temperature=0.2,
    extra_body={
        "decoding": "cot_decoding",  # or "entropy_decoding"
        # CoT specific params
        "k": 10,
        "aggregate_paths": True,
        # OR entropy specific params
        "top_k": 27,
        "min_p": 0.03,
    },
)

The server also exposes the legacy POST /v1/completions endpoint. Streaming is not supported on it yet; use /v1/chat/completions for streaming.

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPTILLM_API_KEY" \
  -d '{"model": "meta-llama/Llama-3.2-1B-Instruct", "prompt": "Write a haiku about recursion.", "max_tokens": 64}'

External servers (llama.cpp, ollama)

export OPENAI_API_KEY="sk-no-key"  # placeholder
./llama-server -c 4096 -m path_to_model
python3 optillm.py --base-url http://localhost:8080/v1

Use as an LLM gateway

The proxy plugin turns OptiLLM into a self-hosted gateway with load balancing, failover, and health checks across any OpenAI-compatible providers, with no gateway fees. It can also apply any of the techniques below to the requests it routes, combining routing and optimization in one service. See Using OptiLLM as an LLM gateway for setup.

Techniques

Plugins

MCP plugin

The Model Context Protocol plugin connects OptiLLM to MCP servers, making their tools, resources, and prompts available to the model. When the model calls a tool, the plugin executes it and returns the result to the model.

Configure servers in ~/.optillm/mcp_config.json:

{
  "mcpServers": {
    "filesystem": {
      "transport": "stdio",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/docs"],
      "description": "Local filesystem access"
    },
    "github": {
      "transport": "sse",
      "url": "https://api.githubcopilot.com/mcp",
      "headers": {
        "Authorization": "Bearer ${GITHUB_TOKEN}",
        "Accept": "text/event-stream"
      },
      "description": "GitHub MCP server"
    },
    "remote-http": {
      "transport": "streamable_http",
      "url": "https://api.example.com/mcp",
      "headers": {
        "Authorization": "Bearer ${API_TOKEN}"
      },
      "description": "Remote Streamable HTTP MCP server"
    }
  },
  "log_level": "INFO"
}

All transports accept description and timeout (connection timeout in seconds, default 5.0). The websocket transport was removed in MCP Python SDK 2.x; use streamable_http or sse instead. String values such as headers support ${VARIABLE_NAME} expansion, and those variables must be set before starting OptiLLM.

Any of the official MCP servers can be used. Logs are written to ~/.optillm/logs/mcp_plugin.log; check there for connection and tool errors.

Configuration

Parameters can be passed on the command line. Most can also be set as environment variables prefixed with OPTILLM_, for example OPTILLM_APPROACH=mcts. Run optillm --help for the full list.

Security

Network binding and authentication. The server binds to 127.0.0.1 by default. Use --host 0.0.0.0 to allow external connections, and only on trusted networks or with authentication enabled. Set --optillm-api-key (or OPTILLM_API_KEY) to require clients to send Authorization: Bearer <key>.

Code execution. The executecode plugin runs Python from requests and LLM responses in a live Jupyter kernel with no sandboxing, so it is disabled unless you opt in. Only enable it in single-user, air-gapped environments where whoever submits requests is trusted with full access to the host.

optillm --enable-unsafe-plugins=executecode
# or: export OPTILLM_ENABLE_UNSAFE_PLUGINS=executecode

URL fetching. The readurls plugin blocks requests to non-public destinations (private, loopback, link-local, and reserved addresses such as the cloud metadata IP 169.254.169.254) to prevent SSRF. Every redirect hop is re-validated and unresolvable hosts are rejected. Only opt out on trusted networks that need internal URL fetching:

optillm --readurls-allow-internal
# or: export OPTILLM_READURLS_ALLOW_INTERNAL=true

The guard validates DNS resolution before each request, but the HTTP client resolves the host again when it connects, so a domain under attacker control could rebind to an internal address between the two lookups. For high-security deployments, also restrict the network egress of the optillm process with a firewall or service-mesh policy.

SSL verification. Use a custom CA bundle for self-signed certificates or corporate proxies. Disabling verification is insecure and intended for development only. See SSL_CONFIGURATION.md for details.

optillm --ssl-cert-path /path/to/ca-bundle.crt  # or: export OPTILLM_SSL_CERT_PATH=...
optillm --no-ssl-verify                         # or: export OPTILLM_SSL_VERIFY=false

Benchmark results

MARS on AIME 2025, IMO 2025, and LiveCodeBench (Oct 2025)

Model: google/gemini-2.5-flash-lite-preview-09-2025 via OpenRouter. Configuration: 3 agents, 2-pass verification, thinking tags disabled for proofs.

CePO on math and code benchmarks (Sep 2025)

AutoThink on GPQA-Diamond and MMLU-Pro (May 2025)

Testing

pip install -r tests/requirements.txt

# Approach tests
python tests/test.py
python tests/test.py --approaches moa bon mcts
python tests/test.py --single-test "Simple Math Problem"

# Unit and integration tests
./tests/run_tests.sh
pytest tests/test_plugins.py -v

Tests run on pull requests through GitHub Actions. See tests/README.md for the test structure and how to write new tests.

Contributing

• Report bugs by opening an issue.
• Share ideas by starting a discussion.
• Look for a place to start in the good first issues.

References

• Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques
• AutoThink: efficient inference for reasoning LLMs - Implementation
• Deep Think with Confidence: Confidence-guided reasoning and inference-time scaling - Implementation
• Self-Discover: Large Language Models Self-Compose Reasoning Structures - Implementation
• CePO: Empowering Llama with Reasoning using Test-Time Compute - Implementation
• LongCePO: Empowering LLMs to efficiently leverage infinite context - Implementation
• Chain of Code: Reasoning with a Language Model-Augmented Code Emulator - Inspired the implementation of coc plugin
• Entropy Based Sampling and Parallel CoT Decoding - Implementation
• Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation - Evaluation script
• Writing in the Margins: Better Inference Pattern for Long Context Retrieval - Inspired the implementation of the memory plugin
• Chain-of-Thought Reasoning Without Prompting - Implementation
• Re-Reading Improves Reasoning in Large Language Models - Implementation
• In-Context Principle Learning from Mistakes - Implementation
• Planning In Natural Language Improves LLM Search For Code Generation - Implementation
• Self-Consistency Improves Chain of Thought Reasoning in Language Models - Implementation
• Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers - Implementation
• Mixture-of-Agents Enhances Large Language Model Capabilities - Inspired the implementation of moa
• Prover-Verifier Games improve legibility of LLM outputs - Implementation
• Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning - Inspired the implementation of mcts
• Unsupervised Evaluation of Code LLMs with Round-Trip Correctness - Inspired the implementation of rto
• Patched MOA: optimizing inference for diverse software development tasks - Implementation
• Patched RTC: evaluating LLMs for diverse software development tasks - Implementation
• AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset - Implementation
• Test-Time Diffusion Deep Researcher (TTD-DR): Think More, Research More, Answer Better! - Implementation

Citation

If you use this library in your research, please cite:

@software{optillm,
  title = {OptiLLM: Optimizing inference proxy for LLMs},
  author = {Asankhaya Sharma},
  year = {2024},
  publisher = {GitHub},
  url = {https://github.com/algorithmicsuperintelligence/optillm}
}