Monitoring

Agent Monitoring

FabrCore provides a pluggable monitoring system for observing message traffic, event streams, and internal LLM calls across your agent system. Messages, events, and LLM calls live in separate buffers with independent query methods and notification events.

IAgentMessageMonitor

The IAgentMessageMonitor interface is the core abstraction for agent monitoring. FabrCore ships with three implementations:

Enabling Monitoring

Monitoring is opt-in. Enable it in AddFabrCoreServer:

using FabrCore.Host;

// Call once during host setup. Metadata-only capture is the default.
builder.AddFabrCoreServer(new FabrCoreServerOptions()
    .UseInMemoryAgentMessageMonitor());

To use a custom implementation:

// MyAgentMessageMonitor is your implementation of IAgentMessageMonitor.
builder.AddFabrCoreServer(new FabrCoreServerOptions()
    .UseAgentMessageMonitor<MyAgentMessageMonitor>());

Monitored Types

MonitoredMessage

Captures every message an agent receives (inbound) and every response it sends (outbound).

MonitoredEvent

Captures events that reach an agent's OnEvent handler (fire-and-forget).

MonitoredLlmCall

Captures calls made through FabrCore's tracked chat clients. Independently constructed clients need their own instrumentation.

OriginContext values:

Querying Monitored Data

Inject IAgentMessageMonitor and use the three separate query methods:

public class MonitorDashboard
{
    private readonly IAgentMessageMonitor _monitor;

    public MonitorDashboard(IAgentMessageMonitor monitor)
    {
        _monitor = monitor;
    }

    public async Task ShowActivity(string agentHandle)
    {
        // Messages (most recent first)
        var messages = await _monitor.GetMessagesAsync(
            agentHandle: agentHandle, limit: 50);

        // Events delivered to the agent
        var events = await _monitor.GetEventsAsync(
            agentHandle: agentHandle, limit: 50);

        // LLM calls made by the agent
        var llmCalls = await _monitor.GetLlmCallsAsync(
            agentHandle: agentHandle, limit: 20);

        foreach (var call in llmCalls)
        {
            Console.WriteLine(
                $"[{call.Timestamp:HH:mm:ss}] {call.OriginContext} " +
                $"model={call.Model} tokens={call.InputTokens}/{call.OutputTokens} " +
                $"dur={call.DurationMs}ms");
        }
    }
}

Token Tracking

FabrCore tracks LLM token usage at two levels:

Per-Call Tracking

Every LLM call captured by the monitor includes InputTokens and OutputTokens. The TokenTrackingChatClient wraps any IChatClient to automatically record each call to the monitor.

Per-Agent Totals

The AgentTokenSummary provides accumulated totals per agent:

var summary = await _monitor.GetAgentTokenSummaryAsync("user1:my-agent");
if (summary is not null)
{
    Console.WriteLine($"Total input tokens: {summary.TotalInputTokens}");
    Console.WriteLine($"Total output tokens: {summary.TotalOutputTokens}");
    Console.WriteLine($"Total calls: {summary.TotalLlmCalls}");
}

LlmUsageScope

LlmUsageScope is an AsyncLocal scope automatically set by OnMessage, carrying the agent handle, parent message ID, trace ID, and origin. This ensures LLM calls made during message processing are correctly attributed.

LlmCallContext

For LLM calls outside of OnMessage (timers, events, background work), use LlmCallContext to tag the origin:

using (LlmCallContext.Set("Timer:cleanup"))
{
    // LLM calls here will be attributed to "Timer:cleanup"
    var response = await chatClient.GetResponseAsync(messages);
}

Real-Time Notifications

Subscribe to events for real-time updates as messages, events, and LLM calls are recorded:

_monitor.OnMessageRecorded += message =>
{
    // Push to UI when a message is captured
};

_monitor.OnEventRecorded += evt =>
{
    // Push to UI when an event is captured
};

_monitor.OnLlmCallRecorded += call =>
{
    // Push to UI when an LLM call completes
};

Custom Monitor Implementations

Implement IAgentMessageMonitor to build custom monitoring backends (SQL database, external logging service, etc.):

// Implement the complete contract from FabrCore.Core.Monitoring.
// It includes recording, queries, summaries, notifications and ClearAsync.
// Query collections are List<T>; per-agent summaries can be null.

LlmCaptureOptions

Retained cloud queries and optional SQL storage

FabrCore 2.0 adds query interfaces alongside recording providers. Filter by principal, agent, trace, time, channel, kind and execution category before paging (100 default, 1,000 maximum), with byte limits and opaque cursors. Fetch captured payloads separately in bounded chunks. Read cloud monitoring for source identity, gaps, retention and hidden-view polling behavior.

Opt in with FabrCore:Monitoring:Provider=sql and the operational database. Defaults are seven-day retention, 10,000 queued records / 64 MiB, batches up to 250 and one-second flushing. Recording does not await SQL; queue saturation and write failures appear as drops and persistence health. Disabled, in-memory and custom providers remain supported. SQL mode alone does not enable monitoring.

Insights refreshes manually by default, optionally every ten seconds while visible, with at most four concurrent host requests. Shared stores are queried once by opaque identity; silo-local sources are queried separately. Missing capture and offline hosts remain visible. Existing local notifications and SSE do not require a continuous cloud stream.

Implement custom providers against the complete current interface. For durable monitoring, retention, queue limits and outage behavior, see cloud administration.