Testing Agents

Testing Agents

FabrCore agents are tested with a lightweight in-memory host — no Orleans silo, no running Host API, no containers. Your agent runs its real production code against a substituted IFabrCoreAgentHost and a chat client you control, so a full agent test executes in milliseconds.

What Gets Substituted

Only the layer beneath your agent is replaced. The agent itself — OnInitialize, OnMessage, tool resolution, state access — is the same code that runs in production.

Mock Mode and Live Mode

Mock mode uses FakeChatClient. There are no network calls, so tests are fast, offline, and deterministic. This is where routing logic, JSON parsing, tool selection, and error handling belong.

Live mode reads model configuration and API keys directly from fabrcore.json and creates chat clients locally against Azure, OpenAI, Grok, Gemini, or OpenRouter. No FabrCore Host needs to be running. Tag these [TestCategory("Integration")] so they can be excluded from the fast loop.

Test Project Setup

A standard MSTest project. fabrcore.json must be copied to the output directory, because the harness reads it from there at run time.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="MSTest" Version="3.*" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.*" />
    <PackageReference Include="NSubstitute" Version="5.*" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\YourAgentProject\YourAgentProject.csproj" />
  </ItemGroup>
  <ItemGroup>
    <None Update="fabrcore.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>
</Project>

Copy the four infrastructure files — TestFabrCoreAgentHost.cs, FakeChatClient.cs, TestChatClientService.cs, and FabrCoreTestHarness.cs — into an Infrastructure/ folder in the test project. To reach internal members, add to the agent project:

<ItemGroup>
  <InternalsVisibleTo Include="YourTestProject" />
</ItemGroup>

Writing a Mock Test

WithSequentialResponses is the important one. Agents that make several LLM calls per message — a routing or effort-estimation call, then the answer — need each turn scripted in order, which is exactly what makes agent logic deterministically testable:

[TestClass]
public class MyAgentTests
{
    [TestMethod]
    public async Task OnMessage_ReturnsExpectedResponse()
    {
        using var harness = new FabrCoreTestHarness();

        // First call is the agent's routing decision, second is the answer.
        var chatClient = FakeChatClient.WithSequentialResponses(
            """{"effort": "small", "reasoning": "Simple question"}""",
            "The answer is 42.");

        var agent = harness.CreateMockAgent<MyAgent>(chatClient);
        await harness.InitializeAgent(agent);

        var response = await harness.SendMessage(agent, "What is the answer?");

        Assert.IsNotNull(response.Message);
        Assert.IsTrue(response.Message.Contains("42"));
    }
}

Writing a Live Test

[TestClass]
[TestCategory("Integration")]
public class MyAgentIntegrationTests
{
    [TestMethod]
    public async Task OnMessage_ProducesCoherentResponse()
    {
        using var harness = new FabrCoreTestHarness();
        var agent = harness.CreateLiveAgent<MyAgent>();

        if (agent is null)
        {
            Assert.Inconclusive("Requires fabrcore.json with valid API keys.");
            return;
        }

        await harness.InitializeAgent(agent);
        var response = await harness.SendMessage(agent, "What is the capital of France?");

        Assert.IsTrue(response.Message!.Contains("Paris", StringComparison.OrdinalIgnoreCase));
    }
}

Harness API

FakeChatClient exposes three factories: WithTextResponse(text) always returns the same text, WithJsonResponse(json) is an alias for it, and WithSequentialResponses(r1, r2, …) returns each response in turn.

Asserting Side Effects

Most agent behaviour worth testing is not the returned string — it is what the agent did along the way. TestFabrCoreAgentHost records it:

// Messages the agent sent on to other agents
Assert.AreEqual(1, harness.AgentHost.SentMessages.Count);

// Events published
Assert.AreEqual(0, harness.AgentHost.SentEvents.Count);

// Timers and reminders registered during OnInitialize
CollectionAssert.Contains(harness.AgentHost.RegisteredTimers, "my-timer");

// Status message set by the agent or one of its plugins
Assert.AreEqual("Processing..", harness.AgentHost.CurrentStatusMessage);

Principal-scoped handles

The default handle is test-agent with no principal. Pass a qualified handle to exercise principal-scoped behaviour:

var harness = new FabrCoreTestHarness(new() { Handle = "principal1:my-agent" });
var host = harness.AgentHost;

host.GetHandle();       // "principal1:my-agent"
host.GetAgentHandle();  // "my-agent"
host.GetUserHandle();   // "principal1"
host.HasUserHandle();   // true

Custom State Resilience

Private agent state outlives restarts, package upgrades, and schema changes, so the interesting tests are the ones where stored state is wrong. Seed state on the in-memory host before initialization and cover the cases that actually occur in production:

• Missing values, JsonValueKind.Null, and JsonValueKind.Undefined should all read as default through GetStateAsync<T>.
• Malformed or type-incompatible JSON belongs in TryGetStateAsync<T> tests when the agent is expected to self-heal, migrate, or reset.
• If the agent deliberately calls GetStateAsync<T> on unreadable state, assert that the InvalidOperationException names the state key, agent handle and type, target type, and stored value kind.
• An agent that owns private state should remove only its own bad key, initialize fresh, and keep running.

State APIs are protected, so expose them through a small test subclass rather than loosening production accessibility:

private sealed class TestAgentProxy : MyAgent
{
    public TestAgentProxy(
        AgentConfiguration config,
        IServiceProvider services,
        IFabrCoreAgentHost host)
        : base(config, services, host) { }

    public Task<StateReadResult<T>> TryRead<T>(string key) => TryGetStateAsync<T>(key);
    public Task<T?> Read<T>(string key) => GetStateAsync<T>(key);
}

What the Harness Cannot Cover

Agent eviction and Blueprint application are Host and Orleans lifecycle features. They do not exist in the in-memory host, so cover them with integration tests against a running Host rather than pretending FabrCoreTestHarness exercises them.

For eviction, verify against DELETE /fabrcoreapi/Agent/{handle} that persisted chat and custom state are gone, timers are disposed and persistent reminders unregistered, stream subscriptions are removed, and the agent leaves GetTrackedAgents(). A later health call should reactivate the virtual grain as NotConfigured, discovery should still list the agent type because that comes from assembly metadata, deleting during an active OnMessage should return 409 Conflict, and a second delete should be idempotent.

For Blueprints, verify against POST /fabrcoreapi/Agent/blueprint that a missing x-user-handle or empty agents returns 400, bare handles are scoped to the calling principal, cross-principal qualified handles are rejected before any agent is processed, tracked-but-NotConfigured agents get configured, healthy ones are left alone, and an incoming ForceReconfigure = true is ignored. Blueprints are caller-driven — a Host restart does not reapply them, so the test has to post the blueprint itself.

Running the Suite

# Fast loop: no API key needed
dotnet test --filter "TestCategory!=Integration"

# Integration only: requires fabrcore.json with a real key
dotnet test --filter "TestCategory=Integration"

# Everything
dotnet test

fabrcore.json for live tests

{
  "ModelConfigurations": [
    {
      "Name": "default",
      "Provider": "Azure",
      "Uri": "https://your-resource.cognitiveservices.azure.com/",
      "Model": "gpt-4o",
      "ApiKeyAlias": "default-key",
      "TimeoutSeconds": 180,
      "ContextWindowTokens": 128000
    }
  ],
  "ApiKeys": [
    { "Alias": "default-key", "Value": "your-api-key-here" }
  ]
}

Supported providers are Azure, OpenAI, OpenRouter, Grok, and Gemini. Keep this file out of source control, and never point live tests at a production model deployment — the suite will call it for real.

LLM Evaluation

Assertions like Contains("Paris") only reach so far. Microsoft.Extensions.AI.Evaluation scores real responses with a second LLM acting as judge, turning "is this answer any good" into a threshold a build can fail on.

[TestClass]
[TestCategory("Evaluation")]
public class MyAgentEvalTests
{
    [TestMethod]
    public async Task Agent_Response_MeetsQualityThresholds()
    {
        using var harness = new FabrCoreTestHarness();
        var agent = harness.CreateLiveAgent<MyAgent>();
        if (agent is null)
        {
            Assert.Inconclusive("Requires fabrcore.json with valid API keys.");
            return;
        }

        const string question = "Explain how photosynthesis works.";
        await harness.InitializeAgent(agent);
        var response = await harness.SendMessage(agent, question);

        // The judge LLM is separate from the agent's own model. The harness resolves it
        // from its own container, so call this after CreateMockAgent/CreateLiveAgent.
        IChatClient judge = await harness.GetChatClient("default");
        var chatConfig = new ChatConfiguration(judge);

        var evaluator = new CompositeEvaluator(
            new RelevanceEvaluator(),
            new FluencyEvaluator(),
            new CoherenceEvaluator());

        var result = await evaluator.EvaluateAsync(question, response.Message!, chatConfig);

        var relevance = result.Get<NumericMetric>("Relevance");
        Assert.IsFalse(relevance.Interpretation!.Failed, relevance.Reason);
    }
}

Quality evaluators are LLM-judged and score 1–5; NLP evaluators are algorithmic and score 0–1; safety evaluators return severity 0–7 where lower is safer. Default interpretation thresholds are:

Prefer asserting on Interpretation.Failed over a hand-picked number: it carries the evaluator's own threshold plus a Reason string, so a failing build tells you why the response scored badly instead of just printing 2.4.

• Use CompositeEvaluator — it runs its evaluators concurrently in one call.
• Tag evals [TestCategory("Evaluation")]; they make real LLM calls and are slow.
• Add DiskBasedResponseCacheProvider from the reporting pipeline to stop re-paying the judge on unchanged inputs.
• For RAG agents, pass retrieved context to GroundednessEvaluator via GroundednessEvaluatorContext so grounding is measured against what was actually retrieved.

dotnet test --filter "TestCategory=Evaluation"