terminal-bench

Introduction

Overview of the Terminal-Bench agent interface.

Creating a custom agent

Terminal-Bench tasks are complex environments with pre-configured lifecycles, include custom startup and teardown instructions. The Terminal-Bench harness handles this complexity for you, so you can focus on writing your agent.

If you haven't already, make sure to install Terminal-Bench by following our installation guide.

Integration Approaches

There are three approaches to integrating your own agent into Terminal-Bench:

  1. Install and run in the task container
  2. Implement the BaseAgent interface
  3. Implement the MCPAgent interface

Install and run in the task container via AbstractInstalledAgent

If your agent is installable from the command line, then the most straightforward approach is to implement the AbstractInstalledAgent interface. Implementing the interface involves defining an installation script, the execution command, and environment variables.

To get started, you'll need to implement the _install_agent_script_path, _run_agent_commands, _env, and name methods.

Below is a minimal example of installing Claude Code inside the harness. Note that this is an illustrative example - we already support Claude Code (see our supported agents), so you don't need to implement this yourself.

claude_code.py
import os
import shlex
from pathlib import Path
 
from terminal_bench.agents.installed_agents.abstract_installed_agent import (
    AbstractInstalledAgent,
)
from terminal_bench.harness_models import TerminalCommand
 
class ClaudeCodeDemoAgent(AbstractInstalledAgent):
 
    @staticmethod
    def name() -> str:
        # Define a name for your agent
        return "Claude Code Demo"
 
    def __init__(self, model_name: str | None = None, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._model_name = model_name
 
    @property
    def _env(self) -> dict[str, str]:
        # Pass the Anthropic API key from your local environment to the agent
        env = {
            "ANTHROPIC_API_KEY": os.environ["ANTHROPIC_API_KEY"],
        }
        if self._model_name:
            env["ANTHROPIC_MODEL"] = self._model_name
        elif "ANTHROPIC_MODEL" in os.environ:
            env["ANTHROPIC_MODEL"] = os.environ["ANTHROPIC_MODEL"]
        return env
 
    @property
    def _install_agent_script_path(self) -> os.PathLike:
        # Returns the local path to the script that will install the agent.
        # The harness will copy this script into the task container and run it.
        # In this case, the script is in the same directory as the agent class.
        return Path(__file__).parent / "setup.sh"
 
    def _run_agent_commands(self, task_description: str) -> list[TerminalCommand]:
        # Returns a list of commands that will run the agent with the given task description
 
        escaped_description = shlex.quote(task_description)
        return [
            TerminalCommand(
                command=f"claude -p {escaped_description} --allowedTools "
                "Bash Edit Write Read Glob Grep Agent", # For Claude Code we need to specify the tools we want to use
                max_timeout_sec=float("inf"), # Run indefinitely
                block=True, # Wait for the agent to finish
            )
        ]

We will also need to supply a setup.sh script that will be copied into the task container and run to install the agent: In order to maximize compatibility, your script shouldn't make any assumptions about dependencies (including that curl is pre-installed) other than that they are running on debian linux.

setup.sh
#!/bin/bash
 
apt-get update
apt-get install -y curl
 
# Install npm, nvm, and node
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.2/install.sh | bash
 
source "$HOME/.nvm/nvm.sh"
 
nvm install 22
npm -v
 
# Install Claude Code
npm install -g @anthropic-ai/claude-code

You can then run your agent by passing its import path to the harness:

export ANTHROPIC_API_KEY="your-anthropic-api-key" # Set the Anthropic API key in your environment
tb run --agent-import-path claude_code:ClaudeCodeDemoAgent  --task-id hello-world

Implement the BaseAgent interface

Implementing the BaseAgent interface is more robust than installing your agent in the task container, but, depending on your agent's implementation, can require a bit of surgery to implement.

The BaseAgent interface only requires you to implement one method:

def perform_task(
    self,
    task_description: str,
    session: TmuxSession,
    logging_dir: Path | None = None,
) -> AgentResult:
    """
    Perform the task described by `task_description`.
 
    Args:
        task_description: The description of the task to perform.
        session: Optional tmux session to send commands to.
        logging_dir: The directory to optionally log your agent's output.
 
    Returns:
        An `AgentResult` object with the agent's token counts and optionally a
        failure mode for debugging.
    """
    pass

To integrate your agent by implementing the BaseAgent interface, you'll need to

Want help?

We are also willing to pair-program with you to integrate your agent. Email mikeam@cs.stanford.edu or alex@laude.org (or both!) if you'd like help.

Implement the MCPAgent interface

Coming soon!

Running a custom agent

Once you have implemented your custom agent, you can run it by using the --agent-import-path flag when running the harness.

tb run --agent-import-path path.to.your.agent:YourCustomAgent ...

Submitting to the leaderboard

See our submitting to the leaderboard guide for instructions on how to submit your agent results to the leaderboard.

Creating Private Sets

How to create a private set of tasks.

Submitting to the Leaderboard

How to submit your agent results to the Terminal-Bench leaderboard.

On this page