Skip to main content

Version: v2.0

aixplain.v2.agent_evaluator

Agent evaluation utilities for aiXplain v2 SDK.

Provides a minimal executor that runs a Dataset of EvalCase rows through one or more Agent instances, runs optional Metric instances, and returns a structured AgentEvaluationRun. Use AgentEvaluationRun.to_dataframe for tabular export and Eval.load_from_csv to reload from disk.

MetricResponse Objects

@dataclass_json

@dataclass(repr=False)
class MetricResponse(Result)

Result for a metric tool run after validation and cleanup.

Extends Result with optional metric-specific fields populated by post-processing (response validation and cleanup).

AgentResponseDataFields Objects

@dataclass_json

@dataclass
class AgentResponseDataFields()

Fields that are required from AgentResponseData.

give_codes

def give_codes() -> Dict[str, str]

Return placeholder codes for query, trace, and output fields.

give_metric_input

def give_metric_input(agent_response: AgentResponseData) -> str

Build metric input string from agent response fields.

Metric Objects

@dataclass_json

@dataclass(repr=False)
class Metric(Tool)

Tool wrapper for creating a tool from a metric integration.

Adds optional pre-processing before creation (placeholder) and post-processing (response validation and cleanup) when running.

Optional threshold marks each evaluated row with metric_pass when set: for string/enum scores use a list of passing values; for numeric scores use a single float (pass when score > threshold).

__post_init__

def __post_init__() -> None

Initialize metric and validate threshold.

create

@classmethod
def create(cls,
           name: str,
           llm_path: str,
           metric_description: str = "",
           prompt_template: Optional[str] = None,
           score_type: Optional[str] = None,
           instruction: Optional[str] = None,
           start_number: Optional[float] = None,
           end_number: Optional[float] = None,
           categories: Optional[list[str]] = None,
           detailed_rubric: Optional[dict] = None,
           auto_complete: bool = False,
           allowed_actions: Optional[List[str]] = None,
           **kwargs: Any) -> "Metric"

Create and persist a Metric backed by the custom LLM prompt integration.

Provide either a ready-made prompt_template or generation parameters (score_type, instruction, and type-specific fields). When prompt_template is a non-empty string, it is used as-is and generation parameters are ignored.

Arguments:

name - Name of the metric tool.
llm_path - The path or ID of the LLM to use.
metric_description - Optional description of the metric tool.
prompt_template - Full prompt template for the LLM. If omitted or blank, a template is built via _generate_prompt_template.
score_type - One of numeric, categorical, or boolean (required when prompt_template is not set).
instruction - Task instruction embedded in the generated template (required when prompt_template is not set).
start_number - Scale lower bound for numeric metrics.
end_number - Scale upper bound for numeric metrics.
categories - Allowed labels for categorical metrics.
detailed_rubric - Optional extra rubric lines appended to the rubric section.
auto_complete - Reserved for future use; passed through to template generation.
allowed_actions - Optional list of allowed actions (currently unused).
**kwargs - Reserved for future Tool construction options.

Returns:

The saved Metric instance.

Raises:

ValidationError - When neither a usable template nor valid generation inputs are given.

initialize

@classmethod
def initialize(cls,
               name: str,
               prompt_template: str,
               llm_path: str,
               metric_description: str = "",
               allowed_actions: Optional[List[str]] = None,
               **kwargs: Any) -> "Metric"

Deprecated. Use create instead.

Preserves the historical argument order (name, prompt_template, llm_path).

trim_and_load_json

@staticmethod
def trim_and_load_json(input_string: str) -> dict

Extract and parse JSON from a string response.

measure

def measure(agent_response: AgentResponseData) -> MetricResponse

Run metric tool with agent response data.

handle_run_response

def handle_run_response(response: MetricResponse,
                        **kwargs: Any) -> MetricResponse

Validate and cleanup response, then return a MetricResponse.

metric_pass_rates_from_rows

def metric_pass_rates_from_rows(
        rows: Sequence[AgentEvaluationRow]) -> Dict[str, Any]

Aggregate pass counts and rates for each metric prefix that has metric_pass.

Only rows with a coercible boolean metric_pass under a prefix are counted. Structure:

{
    "<prefix>": {
        "passed": int,
        "evaluated": int,
        "pass_rate": float,
        "by_agent": {
            "<agent_name>": {"passed": int, "evaluated": int, "pass_rate": float},
            ...
        },
    },
    ...
}

Arguments:

rows - Evaluation rows (typically AgentEvaluationRun.rows).

Returns:

Empty dict when no metric_pass fields are present.

EvalCase Objects

@dataclass
class EvalCase()

One evaluation example (input plus optional reference and metadata).

Attributes:

query - Passed to agent.run(query, **agent_run_kwargs).
reference - Optional ground truth or expected value for metrics.
metadata - Optional extra fields merged into the result row as case_meta__<key>.

Dataset Objects

@dataclass
class Dataset()

Named evaluation dataset: a list of EvalCase with optional description.

Use from_csv or from_queries to build common shapes, or construct with Dataset(name=..., cases=[EvalCase(...), ...]).

iter

def __iter__() -> Iterator[EvalCase]

Iterate over evaluation cases.

len

def __len__() -> int

Return number of evaluation cases.

from_queries

@classmethod
def from_queries(cls,
                 queries: Sequence[str],
                 *,
                 name: str,
                 description: Optional[str] = None) -> Dataset

Build a dataset from plain query strings (one EvalCase per string).

from_csv

@classmethod
def from_csv(cls,
             path: Union[str, Path, Any],
             *,
             name: Optional[str] = None,
             description: Optional[str] = None,
             query_column: str = "query",
             reference_column: Optional[str] = "reference",
             metadata_columns: Optional[Sequence[str]] = None,
             **read_csv_kwargs: Any) -> Dataset

Build a Dataset from a CSV with at least a query column.

Arguments:

path - CSV path or file-like accepted by pandas.read_csv.
name - Human-readable name; defaults to the path stem when path is a str or pathlib.Path, otherwise "dataset".
description - Optional longer description of the dataset.
query_column - Column name used as EvalCase.query.
reference_column - Column for EvalCase.reference, or None to skip.
metadata_columns - Optional column names merged into each case's metadata.
**read_csv_kwargs - Forwarded to pandas.read_csv.

Returns:

Dataset with cases populated; header-only CSV yields an empty cases list.

Raises:

ValidationError - If the query column is missing or a row has an empty query.

AgentEvaluationRow Objects

@dataclass
class AgentEvaluationRow()

One evaluated (case, agent) pair including nested metric tool fields.

metrics maps each metric tool prefix (see _metric_prefix) to a dict of flattened keys (for example metric_status, score, metric_pass when the tool defines a threshold) matching the former <prefix>__<key> column names without the <prefix>__ prefix.

per_asset_stats maps a stable asset label (type and name from each step's unit, joined as type:name, or breakdown keys from execution_stats when steps are absent) to run_time, used_credits, and n_steps aggregates.

metric_value

def metric_value(tool_prefix: str, key: str) -> Any

Return metrics[tool_prefix][key] when present, else None.

AgentEvaluationRun Objects

@dataclass
class AgentEvaluationRun()

Structured output of Eval.evaluate.

Convenience methods (filtering, LLM-ready text, summaries, HTML, optional plots, and chatbot) build on to_dataframe and aixplain.v2.eval_results_display.

Default LLM-backed insight features (executive_summary, chatbot without model=) resolve the model at DEFAULT_INSIGHT_MODEL_PATH using the client bound by configure_insights. After creating aix = Aixplain(...), call AgentEvaluationRun.configure_insights(aix) so aix.Model.get with the path-style model id uses the same API key and URLs as your agents. DEFAULT_INSIGHT_MODEL stays None until the first successful resolution; call ensure_insight_model_loaded to populate it without generating a summary.

configure_insights

@classmethod
def configure_insights(cls, client: Any) -> None

Bind default insight model resolution to an Aixplain client.

Clears any cached default model so the next resolution uses client.Model.

Arguments:

client - An initialized Aixplain instance (same one used for client.Agent.get, client.Model, etc.).

ensure_insight_model_loaded

@classmethod
def ensure_insight_model_loaded(cls) -> Optional[Model]

Resolve and cache DEFAULT_INSIGHT_MODEL from DEFAULT_INSIGHT_MODEL_PATH.

Call after configure_insights. Idempotent when a model is already cached.

Returns:

The cached Model, or None if get/search fails.

iter

def __iter__() -> Iterator[AgentEvaluationRow]

Iterate over evaluation rows.

len

def __len__() -> int

Return number of evaluation rows.

bool

def __bool__() -> bool

Return True if there are evaluation rows.

to_dataframe

def to_dataframe() -> pd.DataFrame

Materialize rows into a long-format pandas.DataFrame (CSV / pivot helpers).

compare_agents_side_by_side

def compare_agents_side_by_side(
        *,
        value_columns: Optional[Sequence[str]] = None,
        include_query: bool = True,
        include_reference: bool = False) -> pd.DataFrame

Pivot to one row per case with agents in columns; see compare_agents_side_by_side.

filter_base

def filter_base(*,
                case_indices: Optional[Sequence[int]] = None,
                agent_names: Optional[Sequence[str]] = None,
                agent_run_failed: Optional[bool] = None) -> AgentEvaluationRun

Return a new run containing only rows matching structural filters.

Filters by evaluation case index, agent name, and/or agent run failure flag. For metric score, latency, or credits filters use filter instead.

filter

def filter(*,
           case_indices: Optional[Sequence[int]] = None,
           agent_names: Optional[Sequence[str]] = None,
           agent_run_failed: Optional[bool] = None,
           metric: Optional[str] = None,
           op: Optional[str] = None,
           value: Any = _FILTER_VALUE_UNSPECIFIED,
           inner_key: str = "score") -> AgentEvaluationRun

Return a new run after structural filters and an optional metric clause.

Structural arguments (case_indices, agent_names, agent_run_failed) are applied first via filter_base, then rows are kept that satisfy the metric clause when metric is set.

metric is either a key under AgentEvaluationRow.metrics (the same prefix used by diff and metrics[prefix]['score'] in _read_metric_score), or a reserved per-row field alias: run_time / latency (row latency), used_credits / credits_used / cost (row credits).

op supports lt, le, gt, ge, eq, ne, and in. For in, pass value as a non-empty list or tuple (numeric membership for numeric scores and row-level metrics; string membership otherwise).

Arguments:

case_indices - Optional set of case indices to keep.
agent_names - Optional set of agent names to keep.
agent_run_failed - When set, keep only rows with this failure flag.
metric - Metric tool prefix or reserved row field name.
op - Comparison operator (required when metric is set).
value - Right-hand side: scalar for numeric/string compare, or sequence for in.
inner_key - Bucket field to read when metric is a tool prefix (default score).

Returns:

New AgentEvaluationRun with matching rows.

Raises:

ValidationError - When metric, op, and value are inconsistent.

filter_where

def filter_where(
        predicate: Callable[[AgentEvaluationRow], bool]) -> AgentEvaluationRun

Return a new run with rows for which predicate(row) is true.

subset_for_case

def subset_for_case(case_index: int) -> AgentEvaluationRun

Return a new run with only rows for case_index.

metric_pass_rates

def metric_pass_rates() -> Dict[str, Any]

Aggregate pass counts and rates for metrics that recorded metric_pass.

Returns:

Mapping from metric tool prefix to passed, evaluated, pass_rate, and nested by_agent (same keys per agent). Empty when no thresholds were applied. See metric_pass_rates_from_rows.

evaluate_quality_gates

def evaluate_quality_gates(
        *,
        metric_score_criteria: Optional[Mapping[str, Any]] = None,
        run_aggregate_gates: Optional[Mapping[str,
                                              Any]] = None) -> Dict[str, Any]

Assess pass/fail against custom metric score rules and run-level aggregates.

Metric scores (per Metric prefix in rows metrics):

A bare number uses the same rule as Metric.threshold for numeric scores (pass when score > threshold).
A list/tuple of strings passes when the score string is in that set (enum-style).
A dict supports threshold, optional operator (lt / le / gt / ge / eq), and optional score_key (defaults to "score").

Rows with metric_skipped or missing score_key are omitted from that metric's evaluation count. If every row is omitted, that metric gate fails (nothing to verify).

Per-sample latency and cost use the same criterion shapes as numeric metrics but reserved criterion names (not metric prefixes): run_time / latency compare AgentEvaluationRow.run_time; used_credits / credits_used / cost compare AgentEvaluationRow.used_credits. Every row is evaluated; list/enum criteria are not allowed for these fields.

Run aggregate gates use the same field names as overall run_summary (agent_failure_rate, total_time_seconds, total_cost, n_agent_failures, total_tool_calls, rows_evaluated, plus aliases credits_used, run_time). They are evaluated per agent against that agent's slice (sums / counts / failure rate for that agent's rows only).

Each gate is {"bound": float, "operator": "lt"} (default operator lt).

Arguments:

metric_score_criteria - Map metric prefix or reserved per-row field name → criterion.
run_aggregate_gates - Map run-summary-style field → {"bound", "operator"}.

Returns:

Dict with by_agent (each agent's overall_pass, metric_gates, aggregate_gates), all_agents_pass (conjunction across agents), agents, criteria echoing inputs, and debug_dataframe — a long-format pandas.DataFrame with query / input, agent_name, output, agent_response, core row fields, <metric_prefix>__<metric_field> columns (score, metric_pass, metric_status, metric_error, skip fields, etc.), plus <key>__criteria_pass / <key>__criteria_reason when the corresponding metric_score_criteria entry applies to that row. Rows with missing agent_name are grouped under "__unnamed__" in by_agent; the debug frame lists raw agent_name values.

to_llm_context

def to_llm_context(*,
                   layout: str = "markdown",
                   max_output_chars: Optional[int] = 8000,
                   case_indices: Optional[Sequence[int]] = None) -> str

Build a single string suitable for pasting into an LLM prompt (review / compare).

Includes per-row fields needed for analysis: reference, run_time, used_credits, request_id, assets_used, total_tool_calls, per_asset_stats, status, completion and failure flags, errors, case_metadata, metrics, output, and agent_response (both long text fields respect max_output_chars when set). When any row has metric_pass under a metric prefix, a leading section summarizes overall and per-agent pass rates (same data as metric_pass_rates).

Arguments:

layout - markdown (headings and bullets) or text (plain lines).
max_output_chars - Truncate output and agent_response; None for no limit.
case_indices - If set, only include rows whose case_index is listed.

to_json_records

def to_json_records() -> List[Dict[str, Any]]

One JSON-serializable dict per row (metrics flattened as prefix__key).

Run-level pass-rate aggregates live under run_summary's metric_pass_rates (not duplicated on each record).

executive_summary

def executive_summary(model: Optional[Model] = None,
                      *,
                      prompt_input_kw: Optional[str] = None,
                      max_context_chars: int = 24_000,
                      quality_gates_report: Optional[Mapping[str, Any]] = None,
                      **model_run_kwargs: Any) -> str

Generate an executive summary, optionally using an LLM for dynamic insights.

Arguments:

model - Optional Model used to generate richer narrative insights from run statistics and compact context. If omitted, AgentEvaluationRun.DEFAULT_INSIGHT_MODEL_PATH is resolved via AgentEvaluationRun.configure_insights (bound client.Model); otherwise a deterministic template summary is returned.
prompt_input_kw - Optional explicit keyword for model.run (for example "text" or "data"). When omitted it is inferred from model parameters.
max_context_chars - Maximum size of embedded evaluation context passed to the model.
quality_gates_report - Optional mapping from evaluate_quality_gates (or its by_agent slice only); embedded as JSON for LLM/template context. debug_dataframe is stripped automatically.
**model_run_kwargs - Extra kwargs forwarded to model.run when model is provided.

run_summary

def run_summary(*,
                include_executive_summary: bool = True,
                summary_model: Optional[Model] = None,
                summary_prompt_input_kw: Optional[str] = None,
                summary_max_context_chars: int = 24_000,
                quality_gates_report: Optional[Mapping[str, Any]] = None,
                **summary_model_run_kwargs: Any) -> Dict[str, Any]

Return aggregate run statistics and optional executive summary text.

When include_executive_summary is true and summary_model is not provided, AgentEvaluationRun.DEFAULT_INSIGHT_MODEL_PATH is resolved using the client from AgentEvaluationRun.configure_insights when set.

Arguments:

include_executive_summary - Generate an LLM-backed executive summary.
summary_model - Optional model for executive summary; defaults to configured insight model.
summary_prompt_input_kw - Explicit keyword for model run input.
summary_max_context_chars - Truncate embedded evaluation context to this size.
quality_gates_report - Optional evaluate_quality_gates result (or by_agent only); copied into quality_gates on the returned dict (without debug_dataframe) and passed into executive_summary when enabled.
**summary_model_run_kwargs - Additional kwargs passed to summary_model.run().

Returns:

Dict with aggregate stats and optional executive_summary text.

summarize_by_agent

def summarize_by_agent() -> pd.DataFrame

Per-agent counts, failures, numeric metric means, and __metric_pass pass rates.

See summarize_by_agent.

pivot_agents_wide

def pivot_agents_wide(value_columns: Optional[Sequence[str]] = None,
                      *,
                      include_query: bool = True,
                      include_reference: bool = True) -> pd.DataFrame

Wide pivot with MultiIndex columns; see pivot_agents_wide.

case_comparison_html

def case_comparison_html(case_index: int,
                         *,
                         max_output_chars: Optional[int] = 8000) -> str

HTML table comparing agents for one case; see case_comparison_html.

case_rows

def case_rows(case_index: int) -> pd.DataFrame

Long-format pandas.DataFrame for a single case_index; see case_rows.

metric_prefixes

def metric_prefixes() -> List[str]

Sorted union of metric tool prefixes present across rows.

metric_inner_key_is_numeric

def metric_inner_key_is_numeric(inner_key: str = "score",
                                *,
                                tool_prefix: Optional[str] = None) -> bool

Return True if every non-null inner_key value coerces to a number via pandas.to_numeric.

inner_key defaults to "score". Use this to choose between plot_mean_metric_by_agent (numeric) and plot_enum_metric_by_agent (string / enum-like categories).

Raises:

ValidationError - If there is no data for this metric key after resolving tool_prefix.

plot_mean_metric_by_agent

def plot_mean_metric_by_agent(
        inner_key: str = "score",
        *,
        tool_prefix: Optional[str] = None,
        title: Optional[str] = None,
        figsize: Optional[tuple[float, float]] = None) -> Any

Draw a bar chart of mean inner_key per agent_name (numeric metrics only).

inner_key is a key inside AgentEvaluationRow.metrics[tool_prefix]; it defaults to "score". If tool_prefix is omitted and exactly one metric prefix exists on the run, it is used; otherwise pass tool_prefix explicitly.

Requires plotly (pip install plotly). Jupyter / Cursor notebook inline display also needs nbformat>=4.2.0 (pip install nbformat or pip install -e ".[notebook]" from this repo).

Returns:

A plotly.graph_objects.Figure.

plot_enum_metric_by_agent

def plot_enum_metric_by_agent(inner_key: str = "score",
                              *,
                              tool_prefix: Optional[str] = None,
                              title: Optional[str] = None,
                              figsize: Optional[tuple[float, float]] = None,
                              normalize: bool = True) -> Any

Draw a grouped bar chart of categorical inner_key counts or shares per agent_name.

inner_key defaults to "score". Values are treated as discrete categories (strings or enum.Enum members). When normalize is True (default), values are row-normalized (proportion per agent).

Requires plotly (pip install plotly). Jupyter / Cursor notebook inline display also needs nbformat>=4.2.0 (pip install nbformat or pip install -e ".[notebook]" from this repo).

Returns:

A plotly.graph_objects.Figure.

plot_metric_by_agent

def plot_metric_by_agent(inner_key: str = "score",
                         *,
                         tool_prefix: Optional[str] = None,
                         title: Optional[str] = None,
                         figsize: Optional[tuple[float, float]] = None,
                         normalize_enum: bool = True) -> Any

Plot inner_key by agent, dispatching on numeric vs categorical values.

inner_key defaults to "score". Calls metric_inner_key_is_numeric; numeric metrics use plot_mean_metric_by_agent, otherwise plot_enum_metric_by_agent. normalize_enum is passed only to the enum path.

Requires plotly (pip install plotly). Jupyter / Cursor notebook inline display also needs nbformat>=4.2.0 (pip install nbformat or pip install -e ".[notebook]" from this repo).

Returns:

A plotly.graph_objects.Figure.

chatbot

def chatbot(
    model: Optional[Model] = None,
    *,
    system_prompt: Optional[str] = None,
    max_context_chars: int = 48_000,
    prompt_input_kw: Optional[str] = None,
    quality_gates_report: Optional[Mapping[str, Any]] = None
) -> AgentEvaluationResultsChatbot

Build an LLM-backed helper that answers questions about this evaluation run.

The underlying Model is invoked with one text payload per question. The run keyword (text, data, etc.) is taken from prompt_input_kw when provided; otherwise it is inferred from params (required fields first), then "text" if the model declares no parameters.

Arguments:

model - Optional loaded Model. If omitted, the default insight model is resolved via AgentEvaluationRun.configure_insights when set.
system_prompt - Override the default analyst instructions.
max_context_chars - Truncate embedded evaluation context to this size.
prompt_input_kw - Explicit keyword for run (e.g. "data" for some utilities). None selects automatically.
quality_gates_report - Optional evaluate_quality_gates result (or by_agent only); appended as JSON after the evaluation excerpt on each turn. debug_dataframe is stripped automatically.

Returns:

AgentEvaluationResultsChatbot with ask.

AgentEvaluationResultsChatbot Objects

@dataclass
class AgentEvaluationResultsChatbot()

LLM-backed Q&A over a single AgentEvaluationRun.

Call ask with natural-language questions; prior turns are kept in conversation_history until reset_conversation.

reset_conversation

def reset_conversation() -> None

Clear prior user/assistant turns (evaluation context is re-injected each call).

ask_with_result

def ask_with_result(question: str, **model_run_kwargs: Any) -> ModelResult

Run the model on question and return the raw ModelResult.

ask

def ask(question: str, **model_run_kwargs: Any) -> str

Ask a question about the run; returns assistant text only.

normalize_eval_results_dataframe

def normalize_eval_results_dataframe(df: pd.DataFrame) -> pd.DataFrame

Return a copy of evaluator results with stable dtypes after CSV round-trip.

AgentEvaluationRun.to_dataframe (or CSV from to_csv) followed by pandas.read_csv often yields object dtypes for booleans and occasionally mis-typed integers. Use this on frames loaded from disk before calling helpers such as pivot_agents_wide or summarize_by_agent.

Only columns that exist are touched; unknown columns are left unchanged.

Arguments:

df - Long-format evaluation results.

Returns:

A new DataFrame; the input is not modified.

compare_agents_side_by_side

def compare_agents_side_by_side(
        results: Union[pd.DataFrame, AgentEvaluationRun],
        *,
        value_columns: Optional[Sequence[str]] = None,
        include_query: bool = True,
        include_reference: bool = False) -> pd.DataFrame

Pivot long evaluator results so each case is one row and agents are columns.

Typical long output (one row per case per agent, from AgentEvaluationRun or results.csv from to_dataframe().to_csv) is normalized with normalize_eval_results_dataframe, then pivoted. By default only output and metric score fields are included (columns ending with __score / __scores, or other numeric metric payload columns such as m1__bleu). Pass value_columns to override.

Result columns look like output__<agent_name> and aws-correctness__score__<agent_name>. Optional query / reference are one column each per case (not split by agent).

Arguments:

results - Long-format pandas.DataFrame or AgentEvaluationRun.
value_columns - Fields to spread by agent_name; default is score-like columns only plus output.
include_query - If True and query is present, add a single query column per case_index.
include_reference - Same for reference.

Returns:

Wide DataFrame with case_index as the first column. Empty input yields an empty DataFrame.

Raises:

ValidationError - If required columns are missing or value_columns references absent columns.

Eval Objects

class Eval()

Runs eval cases across agents, runs metric tools, returns AgentEvaluationRun.

For each pair of (case, agent) the executor calls agent.run with the case's query. Each Metric is invoked with run payload data containing at least output (agent output) and reference (from the case, may be None). Metric results are nested under AgentEvaluationRow.metrics[prefix] using the tool's name, id, or metric_<n> as prefix (see AgentEvaluationRun.to_dataframe for the legacy <prefix>__<key> flat layout).

Set cache_experiments=False to skip writing Experiment snapshots to the local cache after each Experiment.run. Use experiment_cache_dir to override the default cache directory.

init

def __init__(*,
             cache_experiments: bool = True,
             experiment_cache_dir: Optional[Union[str, Path]] = None,
             autosave_eval_runs: Optional[bool] = None) -> None

Configure optional local persistence for Experiment.

Arguments:

cache_experiments - When True (default), experiments created via create_experiment are saved to disk after run.
experiment_cache_dir - Root directory for experiment JSON files; defaults to a platform-appropriate user cache path (see default_experiment_cache_dir).
autosave_eval_runs - Deprecated alias for cache_experiments when not None.

create_experiment

def create_experiment(agents: Union[Agent, Sequence[Agent]],
                      dataset: Dataset,
                      metrics: Optional[Sequence[Metric]] = None,
                      *,
                      metadata: Optional[Dict[str, Any]] = None) -> Experiment

Build an Experiment bound to this executor.

Snapshots agents and metrics via to_dict() for provenance and cache reload. Call run to execute and append an ExperimentRun.

Arguments:

agents - Agent or sequence evaluated against dataset.
dataset - Named evaluation dataset (Dataset).
metrics - Optional metric tools.
metadata - Arbitrary JSON-serializable metadata stored on the experiment.

Returns:

A new experiment with a unique id and creation timestamp.

list_cached_experiments

def list_cached_experiments() -> List[Dict[str, Any]]

List experiments on disk under this executor's cache directory.

load_cached_experiment

def load_cached_experiment(experiment_id: str) -> Experiment

Load a cached experiment and bind this executor for subsequent run calls.

load_from_csv

@classmethod
def load_from_csv(cls,
                  path: Union[str, Path, Any],
                  *,
                  normalize: bool = True,
                  **read_csv_kwargs: Any) -> AgentEvaluationRun

Load a CSV written by AgentEvaluationRun.to_dataframe into a structured run.

Unknown columns (for example a legacy agent_id column) are ignored. Flat <metric_prefix>__<field> columns are split into AgentEvaluationRow.metrics.

Arguments:

path - Path to the CSV file, or a file-like object accepted by pandas.read_csv.
normalize - If True, run normalize_eval_results_dataframe so dtypes match in-memory evaluation results.
**read_csv_kwargs - Forwarded to pandas.read_csv.

Returns:

AgentEvaluationRun with one row per CSV record.

Raises:

ValidationError - If the CSV is non-empty but missing case_index or agent_name.

evaluate

def evaluate(agents: Union[Agent, Sequence[Agent]],
             dataset: Dataset,
             metrics: Optional[Sequence[Metric]] = None,
             **agent_run_kwargs: Any) -> AgentEvaluationRun

Execute all cases against all agents and build a structured result.

Arguments:

agents - A single Agent or a sequence of agents.
dataset - Named evaluation dataset whose Dataset.cases are executed.
metrics - Optional sequence of Metric instances. When a tool sets Metric.threshold, each successful metric row includes metric_pass (boolean) from the score and threshold.
**agent_run_kwargs - Forwarded to each agent.run call.

Returns:

AgentEvaluationRun with one AgentEvaluationRow per (case, agent). Agent or metric failures are recorded per row instead of aborting the batch. Empty dataset.cases yields an empty run.

MetricResponse Objects
AgentResponseDataFields Objects
Metric Objects
EvalCase Objects
Dataset Objects
AgentEvaluationRow Objects
AgentEvaluationRun Objects
AgentEvaluationResultsChatbot Objects
Eval Objects