Skip to main content

​
Required env vars

Set these on your dashboard:
Env VarRequiredDescription
LLM_API_KEYYesYour LLM provider API key
LLM_MODELYesModel to use (e.g., claude-haiku-4-5-20251001)

​
Usage

# Run against SWEbench
benchspan run --benchmark swebench --agent openhands --instances 10

# Quick healthcheck
benchspan run --benchmark agent-healthcheck.quick --agent openhands

​
What it does

The OpenHands runner:
  1. Installs the OpenHands CLI binary via their install script
  2. Sets OPENHANDS_WORK_DIR to point at the benchmark task directory
  3. Runs OpenHands in headless mode with --override-with-envs

​
Supported models

OpenHands supports multiple LLM providers. Set LLM_API_KEY and LLM_MODEL accordingly:
ProviderLLM_MODEL exampleLLM_API_KEY
Anthropicclaude-haiku-4-5-20251001Your Anthropic key
OpenAIgpt-4oYour OpenAI key
See the OpenHands docs for the full list of supported models.