MODULE 12 · LESSON 12.2
Inspect messages, events, LLM calls, and usage
Inspect retained messages, events, model calls and token usage.
Lesson 69 of 86 · FabrCore 2.0
Overview
Capture has a cost and a coverage boundary. In-memory monitoring is bounded and silo-local; SQL retention/shared queries have different behavior. Captured prompts/results can be truncated or redacted. A retained monitor row is not signed evidence.
Capture is bounded and can be incomplete
The monitor can retain messages, events, model calls and usage, but payload capture has size/privacy controls. In-memory providers are bounded and can be silo-local. SQL monitoring is separately enabled and has its own queue/flush behavior; selecting SQL runtime mode does not automatically make every monitor row durable.
Coverage belongs in the result
A query is scoped to its source, filters and retention window. Opaque cursors belong to that query/source, and a reported gap means continuity was lost. A dashboard should identify offline silos, truncated payloads or dropped records rather than claim an empty query proves no activity. Prompt/token details are operational data, not cryptographic evidence.
Query captured messages and usage
- Enable the mode-appropriate monitor and configure payload/tool-argument capture limits intentionally.
- Query messages, events and LLM calls with paging/filtering. Inspect token totals, busy-routing and child-agent attribution; subscribe to live notifications only when needed.
- Set retention and implement a custom provider if required. Report incomplete capture and store/silo coverage in dashboards.
{
"FabrCore": {
"Monitoring": {
"Provider": "sql",
"RetentionDays": 7
}
}
}
// Use the built-in in-memory monitor — metadata-only LLM capture by default.
builder.AddFabrCoreServer(options =>
{
options.UseInMemoryAgentMessageMonitor();
});
// Enable full LLM payload capture (prompts, responses, tool args) with redaction and size caps.
builder.AddFabrCoreServer(options =>
{
options.UseInMemoryAgentMessageMonitor(capture =>
{
capture.CapturePayloads = true; // capture prompts/responses/tool args
capture.MaxPayloadChars = 4_000; // per-field char cap
capture.MaxToolArgsChars = 2_000; // separate cap for tool args
capture.MaxBufferedCalls = 1_000; // lower than the default 2000 since payloads are larger
capture.Redact = s =>
System.Text.RegularExpressions.Regex.Replace(s, "sk-[A-Za-z0-9]+", "***");
});
});
// Or use a custom implementation
builder.AddFabrCoreServer(options =>
{
options.UseAgentMessageMonitor<MyAgentMessageMonitor>();
});
// Get token summary for one agent
var summary = await _monitor.GetAgentTokenSummaryAsync("user1:my-agent");
if (summary != null)
{
Console.WriteLine($"Total input tokens: {summary.TotalInputTokens}");
Console.WriteLine($"Total output tokens: {summary.TotalOutputTokens}");
Console.WriteLine($"Total LLM calls: {summary.TotalLlmCalls}");
}
// Get summaries for all agents
var allSummaries = await _monitor.GetAllAgentTokenSummariesAsync();
The Cloud Server can distribute monitor settings and query retained records through the administration API. A dedicated monitoring screen is optional; the requests below are the shared contract.
The settings fragment opts into SQL monitoring in an already SQL-ready environment. It does not provision the required migration or replace the feature connection. Check the settings catalog and use a deliberate restart where required.
- Prepare the required monitoring schema and publish the provider/retention fragment alongside the existing environment configuration.
- Verify applied/pending state and restart as needed. Run a known request and allow the provider to flush.
- Query the target records with stable filters, inspect source/coverage metadata and follow opaque cursors from that same source.
{
"settings": {
"FabrCore:Monitoring:Provider": "sql",
"FabrCore:Monitoring:RetentionDays": "7"
}
}
GET /fabrcoreapi/admin/v1/observability/monitor?principal=dev&agentHandle=researcher&limit=20&includePayload=false
Authorization: Bearer <administration-credential>
| Property | Default | Description |
|---|---|---|
| Enabled | true | Master switch for the LLM call track. When false, no LLM calls are recorded. |
| CapturePayloads | false | When true, captures prompts, responses, and tool args. Off by default for privacy/memory reasons. |
| MaxPayloadChars | 8000 | Per-field character cap on captured message text |
| MaxToolArgsChars | 4000 | Character cap on captured tool call arguments |
| Redact | null | Optional Func<string,string> applied to every captured string (use for secret/PII scrubbing) |
| MaxBufferedCalls | 2000 | FIFO cap on the in-memory LLM call buffer |
Locate a call and inspect its coverage
- Run a recognizable request, then query monitoring using its principal/agent and time range. Inspect the associated model/tool records and available token usage.
- Lower a capture limit in a controlled environment and repeat with a larger result. Verify truncation/limits are represented; the UI must not present a partial payload as complete.
- Inspect provider/source identity and health. If testing SQL monitoring, wait for successful flush and verify retention after restart rather than assuming enqueue already made the record durable.
The result should explain both observed work and capture limits. Tokens and payloads are useful only when their attribution and coverage are understood.
If the result is different
An empty query can mean eviction, filters or another silo, not “nothing happened.” Keep credentials and sensitive payloads out of uncontrolled capture.
Go deeper
Explore the related documentation.