aixplain.v2.benchmark

BenchmarkCreateParams Objects

class BenchmarkCreateParams(BareCreateParams)

[view_source]

Parameters for creating a benchmark.

Attributes:

• name - str: The name of the benchmark.
• dataset_list - List["Dataset"]: The list of datasets.
• model_list - List["Model"]: The list of models.
• metric_list - List["Metric"]: The list of metrics.

Benchmark Objects

class Benchmark(BaseResource, GetResourceMixin[BareGetParams, "Benchmark"],
                CreateResourceMixin[BenchmarkCreateParams, "Benchmark"])

[view_source]

Resource for benchmarks.

list_normalization_options

@classmethod
def list_normalization_options(cls, metric: "Metric",
                               model: "Model") -> List[str]

[view_source]

List the normalization options for a metric and a model.

Arguments:

• metric - "Metric": The metric.
• model - "Model": The model.

Returns:

• List[str] - The list of normalization options.

BenchmarkJob Objects

class BenchmarkJob(BaseResource, GetResourceMixin[BareGetParams,
                                                  "BenchmarkJob"])

[view_source]

Resource for benchmark jobs.

get_scores

def get_scores() -> dict

[view_source]

Get the scores for a benchmark job.

Returns:

• dict - The scores.