Evaluate real models, Memory, and GraphRAG

Overview

Deterministic tests answer “did the contract work?” Evaluation asks whether real model answers are grounded, relevant and useful. Memory corrections and GraphRAG evidence retrieval need representative cases, not a single successful demo.

Quality needs representative cases

A deterministic test can prove the tool returned a record while the model still misreads it. An evaluation set pairs realistic requests with expected evidence/criteria, then measures answer quality under a recorded model/configuration baseline. Include supported questions, missing evidence, stale facts and partial investigations rather than only easy success examples.

Scores need inspectable examples

Model-judged metrics are signals, not facts guaranteed by the framework. Preserve the retrieved context, final answer and reason for a failure so a reviewer can diagnose retrieval versus generation errors. Measure latency and token cost beside quality; changing an extraction model or prompt can alter all three.

Evaluate grounded answer quality

1. Use Microsoft.Extensions.AI.Evaluation with quality, safety, groundedness or task metrics suited to the application. Store reports and cache appropriate repeat inputs.
2. Include a supported policy question, missing evidence, stale preference correction, tool selection and multi-step partial completion.
3. Measure latency/tokens as well as scores, and record exact models/configuration/datasets. Run real SQL and intended Microsoft tenant checks separately.

[TestMethod]
public async Task Agent_Response_IsGroundedInContext()
{
    // ... set up agent and get response ...

    var groundingContext = """
        Photosynthesis is a process used by plants to convert light energy
        into chemical energy. It occurs primarily in chloroplasts using
        chlorophyll. The process converts CO2 and water into glucose and oxygen.
        """;

    var evaluator = new GroundednessEvaluator();
    var chatConfig = new ChatConfiguration(evaluatorChatClient);

    var result = await evaluator.EvaluateAsync(
        "Explain how photosynthesis works.",
        response.Message!,
        chatConfig,
        additionalContext: [new GroundednessEvaluatorContext(groundingContext)]);

    var groundedness = result.Get<NumericMetric>("Groundedness");
    Assert.IsTrue(groundedness.Value >= 3.0,
        $"Groundedness: {groundedness.Value}/5 — {groundedness.Reason}");
}

Go deeper

Explore the related documentation.

• FabrCore Telemetry
• FabrCore Monitoring
• FabrCore Verifiable Execution
• FabrCore Testing
• Evaluation with Grounding Context (RAG Scenarios)