letta-evals

Letta Evals

Letta Evals is a framework for evaluating Letta and Letta Code agents. It lets you define an evaluation suite with a dataset, target, extractors, graders, and a reward contract, then run that suite against one or more model configurations.

If you are building agentic systems, high-quality evals are one of the fastest ways to understand how model versions, prompts, tools, or agent configuration changes affect your product.

Requirements

• Python 3.11+
• A running Letta server, either:
  • Self-hosted: follow the Letta installation guide, or
  • Letta Cloud: create an account at app.letta.com and set:

    export LETTA_API_KEY=your-api-key
    

    Then use base_url: https://api.letta.com/ in your suite YAML, or pass --base-url https://api.letta.com/ on the CLI.
• Provider API keys for the models you use, such as OPENAI_API_KEY, ANTHROPIC_API_KEY, or GOOGLE_API_KEY.

Installation

For local development or custom eval authoring, clone this repository and install with dev dependencies:

uv sync --extra dev

To run existing evals without editing the repo:

pip install letta-evals

Quick start

1. Create a dataset (dataset.jsonl):

{"input": "What's the capital of France?", "ground_truth": "Paris"}
{"input": "Calculate 2+2", "ground_truth": "4"}

2. Create a suite (suite.yaml):

name: my-eval-suite
dataset: dataset.jsonl

target:
  kind: letta_code
  model_handles:
    - openai/gpt-4.1-mini
  base_url: http://localhost:8283

graders:
  correctness:
    kind: tool
    function: contains
    extractor: last_assistant

reward:
  kind: metric
  metric_key: correctness

3. Validate and run:

letta-evals validate suite.yaml
letta-evals run suite.yaml

Running evals

The core flow is:

Dataset → Target → Extractor → Grader → Reward → Result

Common commands:

# Run an evaluation suite with progress output
letta-evals run suite.yaml

# Save suite.json, summary.json, and per-model JSONL results
letta-evals run suite.yaml --output results/

# Run multiple times for aggregate statistics
letta-evals run suite.yaml --num-runs 5 --output results/

# Validate suite configuration and list built-ins
letta-evals validate suite.yaml
letta-evals list-extractors
letta-evals list-graders

You can also set run defaults in suite.yaml:

max_concurrent: 5
max_samples: 20
num_runs: 3
output: results/
cleanup: true

Relative paths in suite YAML are resolved from the suite file’s directory. CLI flags such as --max-concurrent, --output, --api-key, --base-url, and --num-runs override suite or environment defaults when provided.

letta-evals run loads a .env file from the directory you run it in, without overriding variables that are already set. When you call run_suite or Runner from Python, export the variables or load your own .env, e.g. with python-dotenv’s load_dotenv().

Writing suites

Datasets

Datasets can be JSONL or CSV. Each row should provide the user input and, for most graders, a ground_truth value:

{"input": "Draw a cat in ASCII", "ground_truth": "cat"}

For multi-turn evals, set input to a list of user messages; ground_truth stays a single string for the whole conversation.

Dataset rows may also include fields such as:

• extra_vars for custom graders, and extra_vars.cwd for the agent’s working directory (see Working directory)
• agent_args for programmatic agent factories
• rubric or rubric_path for per-sample model-judge rubric overrides

Instead of a local path, dataset: may point at a dataset hosted on the HuggingFace Hub. The single manifest file is fetched and cached on the host, then loaded exactly like a local file:

# pin a revision (tag / branch / commit SHA) for reproducible runs
dataset: https://huggingface.co/datasets/letta-ai/swe-chat-tagged/resolve/<revision>/train.jsonl

Private repos work with a standard HF_TOKEN / HUGGING_FACE_HUB_TOKEN in the environment. Fetching happens host-side, so nothing is downloaded inside the Modal sandbox. An unpinned revision (a bare repo URL, or .../resolve/main/...) warns and surfaces the resolved commit so the run stays reproducible from its logs; the resolved commit SHA is also recorded in suite.json under config.dataset_provenance. A bare repo URL is accepted only when the repo holds exactly one .jsonl/.csv manifest. Relative rubric_path values still resolve against the suite directory, not the HF cache.

Targets

The supported target is letta_code, which runs the Letta Code CLI against a Letta server. Important target fields include:

• base_url: Letta server URL; defaults to http://localhost:8283
• model_handles: one or more model handles to evaluate
• agent_script: optional file.py:function_name agent factory
• use_agent_model: run a factory-created agent on the model and model settings its factory gave it (context window, effort, output tokens) instead of applying the suite’s model handle with --model. The handle then only labels results, so exactly one is allowed. Requires agent_script
• flags: additional Letta Code CLI flags (including tool restrictions, e.g. --allowed-tools Bash Read)
• permission_mode: optional Letta Code permission mode, such as unrestricted, standard, or acceptEdits
• timeout and max_retries: target execution controls

Working directory

Each sample’s agent starts in a fresh, empty temporary directory rather than the suite directory, which holds the dataset, graders and past results. cleanup: true removes the directory after the sample.

To start the agent somewhere else, set cwd in the dataset row’s extra_vars or in the agent factory’s returned context (the factory wins). Relative paths resolve against the suite directory, and the directory must already exist. The {pwd} placeholder in a sample’s input is replaced with the agent’s working directory, and the directory is saved as context["cwd"] for extractors.

{"input": "Answer using the data files in {pwd}", "ground_truth": "...", "extra_vars": {"cwd": "files"}}

The working directory is where the agent starts and how its relative paths resolve; it does not confine the agent. Use sandbox: for filesystem isolation.

Memory directory

To give an agent a memory directory, set MEMORY_DIR in the rollout’s env, in the dataset row’s extra_vars or in the agent factory’s returned context (the factory wins). A relative path resolves against the agent’s working directory, so with the default working directory, {"env": {"MEMORY_DIR": "memory"}} gives each rollout its own. letta-evals creates the directory if it’s missing and sets MEMORY_DIR, LETTA_MEMORY_DIR and LETTA_MEMORY_DIR_EXPLICIT=1 for the subprocess. It never passes on a MEMORY_DIR from its own environment.

For agents with MemFS on, Letta Code points MEMORY_DIR at the agent’s own memory folder instead. Pass --stateless in flags to have the agent use the given directory.

Local backend

backend: local (or letta-evals run --backend local) runs Letta Code on its local backend (letta --backend local) with no Letta server. It needs a Letta Code release with letta agents create --body.

• Each sample gets a fresh local store in a temp directory, recorded on SampleResult.local_backend_dir. cleanup: true removes it after the sample.
• Agent factories receive a client whose only method is client.agents.create(...). It creates the agent in the sample’s store with the suite’s model handle (or, with use_agent_model, the factory’s own model), and the agent then runs with --agent.
• The trajectory comes from the CLI’s stream-json output. Token data, server-side pricing lookups, and server cleanup don’t apply.
• Model providers are configured the way Letta Code’s local backend reads them, e.g. LLAMA_CPP_BASE_URL for an OpenAI-compatible endpoint.
• Letta Code enables local memfs for its own agents, so a memory directory needs --stateless here too.

Graders and extractors

Suites can use deterministic tool graders or model-judge graders:

graders:
  exact:
    kind: tool
    function: exact_match
    extractor: last_assistant
  quality:
    kind: model_judge
    prompt_path: rubric.txt
    model: gpt-6-luna
    provider: openai
    extractor: last_assistant

A model-judge rubric is a Jinja template, rendered in Jinja’s sandbox and sent to the judge verbatim. It can use {{ input }}, {{ ground_truth }}, {{ submission }} and any key in the sample’s rubric_vars, including lists and dicts to loop over. Single braces are literal text, so JSON examples need no escaping. Before any agent runs, each sample’s rubric is checked: a variable the sample doesn’t provide, or a leftover {input}-style placeholder, stops the run.

Model-judge settings beyond the rubric:

• system_prompt_path: a file sent verbatim as the judge’s system prompt. By default no system prompt is sent.
• reasoning_effort: one of none, minimal, low, medium, high, xhigh or max, sent as reasoning.effort to OpenAI and output_config.effort to Anthropic. Each provider and model accepts a subset. Unset leaves the provider’s default.
• max_output_tokens: the most tokens the judge may generate, reasoning included (default 16000).
• max_retries and timeout: retries and per-request timeout in seconds for the judge’s API calls (defaults 5 and 120).

Judge replies and scorers

The judge replies in a fixed format, and a scorer turns the reply into the grade’s score. The judge’s full reply is saved in the grade’s metadata["response"]. Set scorer to one of:

• score (default): the judge replies {rationale, score} and the score is used as is.
• checklist: the judge gives a MET or UNMET verdict on each criterion in the sample’s rubric_vars["criteria"]. Each criterion is {id, text}, with an optional weight (default 1). The score is the weight of positive criteria MET over the total positive weight. A negative weight marks a criterion that describes an error: when it’s MET, its weight is subtracted. The score is clipped to [0, 1]. The rubric shows the judge the criteria, for example with {% for c in criteria %}[{{ c.id }}] {{ c.text }}{% endfor %}.
• file.py:function: your own scorer, taking (sample, response) and returning a score in [0, 1]. It needs response_format: file.py:ClassName, the Pydantic model the judge replies with.

graders:
  completion:
    kind: model_judge
    prompt_path: checklist_rubric.txt
    scorer: checklist

{"input": "...", "rubric_vars": {"criteria": [
  {"id": "doc", "text": "A 4-6 paragraph design document is created"},
  {"id": "seeds", "text": "Prev/next/random seed controls work", "weight": 2},
  {"id": "copied", "text": "The piece copies an existing artist's work", "weight": -1}
]}}

If the reply doesn’t match the format, the scorer raises ValueError (the checklist scorer does when a criterion is missing, repeated or unknown), or there’s no reply, the judge is asked again, up to 3 calls in all. A response_format can’t contain a dict field or refer to itself: Anthropic can’t serve either. Both are rejected before the run.

Use letta-evals list-graders and letta-evals list-extractors for built-ins. You can register custom Python graders, extractors, setup hooks, and agent factories with decorators; see examples/custom-tool-grader-and-extractor/ and examples/programmatic-agent-creation/.

Rewards

A reward turns grader outputs into the canonical per-sample scalar stored on SampleResult.reward. The framework owns the contract and persistence; suite authors own any custom composition logic.

For simple suites, use one grader directly:

reward:
  kind: metric
  metric_key: correctness

For suite-specific composition, point at a Python reward composer:

reward:
  kind: custom
  function: rewards.py:compose_reward

from letta_evals import RewardOutput, reward_composer


@reward_composer
def compose_reward(ctx):
    quality = ctx.grades["quality"].score
    valid = ctx.grades["validity_check"].score
    if valid < 1.0:
        return RewardOutput(score=0.0, metadata={"reason": "validity_check_failed"})
    return RewardOutput(score=quality)

grades remain the source of truth for raw grader outputs. Reward metadata should only contain derived composer decisions that are not already recoverable from grades, submissions, or the sample.

See examples/reward-composition/ for complete custom reward examples.

Setup scripts and agent factories

Use setup_script: file.py:function_name for one-time setup before a suite runs. Setup functions may have one of these signatures:

• () -> None
• (client: AsyncLetta) -> None
• (client: AsyncLetta, model_handle: str) -> None

Use target.agent_script: file.py:function_name to create or customize an agent per sample. Agent factories receive the Letta client and current Sample, and return either the agent ID or (agent_id, context).

context is a JSON-serializable dict of per-rollout values. context["env"] and context["cwd"] configure the Letta Code subprocess, layered over the same keys in sample.extra_vars. The target writes the agent’s working directory back to context["cwd"]. The dict is saved on SampleResult.context and passed to extractors that take a third context parameter and to reward composers as ctx.context:

@agent_factory
async def create_seeded_agent(client, sample):
    agent_id = ...
    return agent_id, {"env": {"MEMORY_DIR": str(seed_memory(sample))}}


@extractor
def memory_files(trajectory, config, context):
    root = Path(context["env"]["MEMORY_DIR"])  # per rollout
    return "\n".join(str(p.relative_to(root)) for p in sorted(root.glob(config["glob"])))  # config: extractor_config

Examples

The examples/ directory contains working suites:

• examples/custom-tool-grader-and-extractor/ — custom Python extractor and grader for structured JSON output
• examples/letta-code-simple-edit/ — Letta Code fixes buggy Python files, with a subprocess grader
• docs/modal-sandbox.md — per-sample Modal sandbox execution
• examples/reward-composition/ — custom reward composers across multiple graders
• examples/multi-model-simple-rubric-grader/ — compare multiple model handles in one suite
• examples/per-sample-rubric/ — override model-judge rubrics per dataset row
• examples/programmatic-agent-creation/ — create customized agents from Python before each sample

Modal sandbox execution

Add a suite-level sandbox block to run every sample inside a fresh Modal sandbox:

sandbox:
  kind: modal
  letta_evals_version: "0.25.0"
  letta_code_version: "0.30.5"
  cpu: 2
  memory_mb: 4096
  timeout_sec: 1800

The bundled runtime requires exact letta_evals_version and letta_code_version pins; mutable tags and version ranges are rejected. The host runner still owns the sample loop, concurrency, JSONL output, and reward aggregation. Each sample is uploaded to a sandbox along with the suite directory; the target, extractors, graders, and reward composer run in the sandbox; and the final SampleResult is returned to the host.

See docs/modal-sandbox.md for setup details, networking notes, and common failure modes.

FAQ

Can I write evals without Python code?

Yes. Many suites only need YAML plus JSONL/CSV data and built-in graders such as contains, exact_match, regex_match, or model-judge grading.

Can I test multiple models?

Yes. Set target.model_handles to a list. Letta Evals runs every sample for every model and writes per-model results. To run the complete suite against one model without editing its YAML, pass --model-handle, for example:

letta-evals run suite.yaml --model-handle baseten/dream-1@letta-research

Can I run evaluations repeatedly?

Yes. Use --num-runs N or num_runs: N to compute aggregate statistics across repeated suite runs.

Can I use this in CI/CD?

Yes. Letta Evals is designed for CI. See .github/workflows/e2e-tests.yml for an example of running suites in GitHub Actions.

Contributing

Contributions are welcome. See CONTRIBUTING.md for local setup, linting, testing, and PR guidelines.

License

This project is licensed under the Apache License 2.0. By contributing to this repository, you agree that your contributions are licensed under the repository’s license. You must have adequate rights to upload any data used in an eval. Letta reserves the right to use this data in future service improvements to our product.