Skip to main content
Version: 1.0

aixplain.modules.benchmark

__author__​

Copyright 2022 The aiXplain SDK authors

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Author: Duraikrishna Selvaraju, Thiago Castro Ferreira, Shreyas Sharma and Lucas Pavanelli Date: October 25th 2022 Description: Benchmark Class

Benchmark Objects​

class Benchmark(Asset)

[view_source]

Benchmark is a powerful tool for benchmarking machine learning models and evaluating their performance on specific tasks. It represents a collection of Models, Datasets and Metrics to run associated Benchmark Jobs.

Attributes:

  • id str - ID of the Benchmark.
  • name str - Name of the Benchmark.
  • model_list List[Model] - List of Models to be used for benchmarking.
  • dataset_list List[Dataset] - List of Datasets to be used for benchmarking.
  • metric_list List[Metric] - List of Metrics to be used for benchmarking.
  • job_list List[BenchmarkJob] - List of associated Benchmark Jobs.
  • additional_info dict - Any additional information to be saved with the Benchmark.

__init__​

def __init__(id: Text,
             name: Text,
             dataset_list: List[Dataset],
             model_list: List[Model],
             metric_list: List[Metric],
             job_list: List[BenchmarkJob],
             description: Text = "",
             supplier: Text = "aiXplain",
             version: Text = "1.0",
             **additional_info) -> None

[view_source]

Create a Benchmark with the necessary information.

Arguments:

  • id Text - ID of the Benchmark.
  • name Text - Name of the Benchmark.
  • dataset_list List[Dataset] - List of Datasets to be used for benchmarking.
  • model_list List[Model] - List of Models to be used for benchmarking.
  • metric_list List[Metric] - List of Metrics to be used for benchmarking.
  • job_list List[BenchmarkJob] - List of associated Benchmark Jobs.
  • description Text, optional - Description of the Benchmark. Defaults to "".
  • supplier Text, optional - Author of the Benchmark. Defaults to "aiXplain".
  • version Text, optional - Benchmark version. Defaults to "1.0".
  • **additional_info - Any additional Benchmark info to be saved.

__repr__​

def __repr__() -> str

[view_source]

Return a string representation of the Benchmark instance.

Returns:

  • str - A string in the format "<Benchmark name>".

start​

def start() -> BenchmarkJob

[view_source]

Start a new benchmark job (run) for the current benchmark.

This method initiates a new benchmark job using the configured models, datasets, and metrics. It communicates with the backend API to create and start the job.

Returns:

  • BenchmarkJob - A new BenchmarkJob instance representing the started job. Returns None if the job creation fails.

Raises:

  • Exception - If there's an error creating or starting the benchmark job. The error is logged and None is returned.