Skip to main content

module aixplain.factories.benchmark_factory

class BenchmarkFactory​

A static class for creating and managing the Benchmarking experience.

Attributes:

  • backend_url (str): The URL for the backend.

classmethod create​

create(
    name: str,
    dataset_list: List[Dataset],
    model_list: List[Model],
    metric_list: List[Metric]
) → Benchmark

Creates a benchmark based on the information provided like name, dataset list, model list and score list. Note: This only creates a benchmark. It needs to run seperately using start_benchmark_job.

Args:

  • name (str): Unique Name of benchmark
  • dataset_list (List[Dataset]): List of Datasets to be used for benchmarking
  • model_list (List[Model]): List of Models to be used for benchmarking
  • metric_list (List[Metric]): List of Metrics to be used for benchmarking

Returns:

  • Benchmark: description

classmethod get​

get(benchmark_id: str) → Benchmark

Create a 'Benchmark' object from Benchmark id

Args:

  • benchmark_id (Text): Benchmark ID of required Benchmark.

Returns:

  • Benchmark: Created 'Benchmark' object

classmethod get_benchmark_job_scores​

get_benchmark_job_scores(job_id)

classmethod get_job​

get_job(job_id: str) → BenchmarkJob

Create a 'BenchmarkJob' object from job id

Args:

  • job_id (Text): ID of the required BenchmarkJob.

Returns:

  • BenchmarkJob: Created 'BenchmarkJob' object

classmethod list_normalization_options​

list_normalization_options(metric: Metric, model: Model) → List[str]

Get list of supported normalization options for a metric and model to be used in benchmarking

Args:

  • metric (Metric): Metric for which normalization options are to be listed
  • model (Model): Model to be used in benchmarking

Returns:

  • List[str]: List of supported normalization options