Telemetry

Telemetry

FabrCore includes comprehensive OpenTelemetry instrumentation across server and client packages for full observability of your AI agent systems.

OpenTelemetry Setup

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource("FabrCore.Sdk.*")
        .AddSource("FabrCore.Host.*")
        .AddSource("FabrCore.Client.*"))
    .WithMetrics(metrics => metrics
        .AddMeter("FabrCore.Sdk.*")
        .AddMeter("FabrCore.Host.*")
        .AddMeter("FabrCore.Client.*"));

Metrics

Server Metrics

WebSocket (FabrCore.Host.WebSocket)

Agent Grain (FabrCore.Host.AgentGrain)

SDK (FabrCore.Sdk.FabrCoreAgentProxy)

Client Metrics

Connection (FabrCore.Client.Connection)

API Client (FabrCore.Client.FabrHostApiClient)

Client Agent Proxy (FabrCore.Client.FabrCoreClientAgentProxy)

Distributed Tracing

Activity Sources

Structured Logging

Use the built-in logger in your agents for structured logging that integrates with OpenTelemetry:

public override async Task<AgentMessage> OnMessage(AgentMessage message)
{
    logger.LogInformation(
        "Processing message from {FromHandle}",
        message.FromHandle);

    try
    {
        // Process message
    }
    catch (Exception ex)
    {
        logger.LogError(ex,
            "Error processing message {MessageId}",
            message.Id);
        throw;
    }
}

Tool Invocation Logging

Use Microsoft.Extensions.AI's UseLogging() middleware to capture tool invocations, arguments, and results through the standard ILogger pipeline:

var chatClient = new ChatClientBuilder(innerClient)
    .UseLogging(loggerFactory)
    .Build();

This logs tool names, arguments, results, and duration through your existing logging infrastructure — no custom log aggregation agents needed. Combine with your observability stack (Application Insights, Seq, ELK, Grafana) for production monitoring.

Health Endpoints

FabrCore exposes diagnostics endpoints for monitoring. Combine these with ASP.NET Core health checks for Kubernetes probes and alerting:

ASP.NET Core Health Check Integration

// Register health checks that query FabrCore diagnostics
builder.Services.AddHealthChecks()
    .AddCheck<FabrCoreAgentHealthCheck>("fabrcore-agents");

var app = builder.Build();

// Map the standard health endpoint
app.MapHealthChecks("/health");

This integrates with Kubernetes liveness/readiness probes, Azure App Service health checks, and standard monitoring infrastructure.

LLM Token Tracking

FabrCore automatically tracks LLM token usage across all agent interactions through the monitoring system. The TokenTrackingChatClient wraps chat client calls to capture input and output token counts, while LlmUsageScope aggregates usage across multiple LLM calls within a single message processing cycle.

Token counts are automatically attached to agent response messages via Args (e.g., _tokens_input, _tokens_output, _llm_calls), making usage data available to clients without additional configuration.

See full token tracking documentation in the Monitoring guide for details on configuring token tracking, building custom usage dashboards, and integrating with cost monitoring systems.