Benchmark AI Models
How to benchmark a model
This guide explains how to benchmark models using the aiXplain SDK. You'll learn to select datasets, models, and metrics, and create benchmarks for tasks.
Generic Example (Template)
from aixplain.factories import (
BenchmarkFactory,
DatasetFactory,
MetricFactory,
ModelFactory,
)
datasets = DatasetFactory.list("...") # specify Data ID
metrics = MetricFactory.list("...") # specify Metric ID
models = ModelFactory.list("...") # specify Model ID
benchmark = BenchmarkFactory.create(
"benchmark_name", dataset_list=datasets, model_list=models, metric_list=metrics
)
benchmark_job = benchmark.start()
status = benchmark_job.check_status()
results_path = benchmark_job.download_results_as_csv()
Benchmark Examples
The following examples show Benchmarking applied to Text Generation, Translation and Speech Recognition using differing approaches.