Monitoring

Monitoring agent performance is crucial for understanding usage, efficiency, and cost metrics. aiXplain provides built-in tools to track agent statistics, including:

• Execution status
• Agent-level metrics (credits used, runtime, number of calls)
• Tool-level metrics (credits used, runtime, number of calls)

This guide explains how to access and analyze these statistics.

Generating Agent and Tool Statistics

from collections import defaultdict

def analyze_agent_and_tool_metrics(agent_response):
    data = agent_response["data"]

    # Initialize dictionaries for statistics
    agent_metrics = defaultdict(lambda: {'credits': 0, 'runtime': 0, 'calls': 0})
    tool_metrics = defaultdict(lambda: {'credits': 0, 'runtime': 0, 'calls': 0})

    total_agent_credits = 0
    total_agent_runtime = 0
    total_agent_calls = 0

    # Process agent statistics
    for step in data['intermediate_steps']:
        agent = step['agent']
        credits = step.get('usedCredits', 0) or 0
        runtime = step.get('runTime', 0) or 0

        agent_metrics[agent]['credits'] += credits
        agent_metrics[agent]['runtime'] += runtime
        agent_metrics[agent]['calls'] += 1

        total_agent_credits += credits
        total_agent_runtime += runtime
        total_agent_calls += 1

        # Process tool statistics if present
        if step.get('tool_steps'):
            for tool_step in step['tool_steps']:
                tool = tool_step['tool']
                tool_credits = tool_step.get('usedCredits', 0) or 0
                tool_runtime = tool_step.get('runTime', 0) or 0

                tool_metrics[tool]['credits'] += tool_credits
                tool_metrics[tool]['runtime'] += tool_runtime
                tool_metrics[tool]['calls'] += 1

    # Print Agent Metrics
    print("Agent Metrics:")
    print("-" * 80)
    print(f"{'Agent':<20} {'Total Credits':<20} {'Total Runtime':<20} {'Total Calls':<15}")
    print("-" * 80)
    for agent, data in agent_metrics.items():
        print(f"{agent:<20} {data['credits']:<20.6f} {data['runtime']:<20.3f} {data['calls']:<15}")
    print("-" * 80)
    print(f"{'Total':<20} {total_agent_credits:<20.6f} {total_agent_runtime:<20.3f} {total_agent_calls:<15}")

    # Print Tool Metrics
    print("\nTool Metrics:")
    print("-" * 80)
    print(f"{'Tool':<20} {'Total Credits':<20} {'Total Runtime':<20} {'Total Calls':<15}")
    print("-" * 80)
    for tool, data in tool_metrics.items():
        print(f"{tool:<20} {data['credits']:<20.6f} {data['runtime']:<20.3f} {data['calls']:<15}")

    return agent_metrics, tool_metrics

# Example Usage
agent_metrics, tool_metrics = analyze_agent_and_tool_metrics(result)

Understanding the Statistics

Metric	Description
Status	Indicates whether the agent execution was successful or encountered an error.
Input	The query or request sent to the agent.
Output	The final response generated by the agent.
Agent Name	The specific agent performing the task.
Total Credits	The number of processing credits consumed by the agent or tool.
Total Runtime	The total execution time in seconds.
Total Calls	The number of times an agent or tool was used during execution.

Monitoring agent statistics provide detailed insights into agent performance, resource consumption, and execution flow. By tracking agent and tool statistics, you can:

• Optimize agent efficiency by analyzing runtime and credit usage.
• Identify performance bottlenecks in complex workflows.
• Debug errors by examining execution steps and resource consumption.

By leveraging these monitoring techniques, you can ensure that your aiXplain agents run efficiently and effectively.