Cloud administration

FabrCore 2.0 lets a cloud server manage clusters and investigate agent behavior through authenticated, versioned APIs. Vulcan365 Insights provides an operator experience; other servers can implement the same public contracts.

Contracts ship in FabrCore.Core, cluster services in FabrCore.Host, and the typed FabrCoreAdministrationClient in FabrCore.Sdk. None requires Insights identities, packages or storage. These capabilities are part of the 2.0 release.

Connect on your terms

The cluster keeps its outbound HTTP long-poll connection. Configuration uses /fabrcore-cloud/v1, command delivery uses /fabrcore-cloud/v2/connect, and Host administration uses /fabrcoreapi/admin/v1. Requests and responses are bounded; longer actions return receipts that clients poll. Cloud WebSockets and streaming diagnostic tokens are deferred. The existing agent WebSocket client remains available for ordinary application chat.

Enable cloud enrollment and remote administration in Host configuration. The trusted server authenticates operators, authorizes their cluster access and establishes X-FabrCore-Admin-Actor. Never accept a public caller's actor header as authority. A channel name or message argument cannot grant administrative access.

Preserve conditional headers, status codes, command IDs, operation IDs, lease tokens, expiry and body limits through the connection. A lost mutation response can mean uncertain execution: read its receipt before deciding on an explicit retry.

Discover what the cluster supports

Read GET /fabrcoreapi/admin/v1/capabilities before enabling controls. Older hosts retain their APIs; hide unsupported management controls and label legacy monitoring as a bounded snapshot. The authenticated /fabrcoreapi/admin/v1/openapi.json describes installed routes and request schemas; the SDK provides typed response contracts.

The installed agent catalog exposes agent types, model aliases, plugins, tools, skills and extension schemas without credentials. Agent building configures these installed components. It does not compile arbitrary code or install assemblies.

Manage principals and ACLs

GET /fabrcoreapi/admin/v1/principals returns registered and runtime-discovered principals, including inactive registrations, with source information. Page with offset, limit and the first page's revision. Runtime discovery alone is not the full principal directory.

In SQL mode, collection and individual APIs support principal, role, group and grant CRUD, membership changes, effective grants with origins, protected entities and enforcement reset. Read the numeric ACL version from /access/metadata; send it in If-Match for writes to /access/entities/{kind}/{id}. The directory revision is a different value. Conditions are checked inside the serialized registry mutation. Cached authorization evaluation adds no cloud I/O to ordinary messages. See access control.

Review and deploy blueprints

Typed summary pages support browsing without confusing legacy blueprint-name lists with summary objects. Create, edit, clone, import, export and delete definitions while preserving extension JSON. Surface squads remain an optional package extension.

1. Read the current definition and revision. Save with If-Match; use quoted * when creating a new definition.
2. Validate and preview expansion. Missing dependencies, duplicate expanded handles, unsupported extensions and cross-principal targets are rejected. Preview performs no provisioning or tool execution; custom extensions explicitly implement IBlueprintPreviewExpander.
3. Deploy the reviewed revision and expansion digest. Ensure missing creates missing agents. Update existing also reconfigures existing agents. Neither mode deletes agents removed from a definition.
4. Inspect persisted per-agent results and explicitly retry selected failed items. Cluster coordination deduplicates matching deliveries across silos; interrupted work is not silently replayed.

Cloud envelopes use the same service and support optional DeploymentId and ApplyMode fields with compatible defaults. Removing an envelope entry does not delete definitions or agents. Insights distinguishes cloud-managed definitions from runtime-only definitions and shows deployment status, deployed revisions and definition drift. That drift compares saved and deployed definitions; automatic comparison with every live agent configuration is not implemented.

Build and manage agents

Create or ensure agents individually or in batches, inspect configuration and health, and manage custom state and threads. Editable configuration/state uses revisions; disruptive actions check both normal and admin activity. Stale writes return 412, missing required conditions 428, and conflicting activity 409.

Use stable client submission IDs and poll persistent operation receipts for create, ensure, configure, restart, reset, tests and deployment. Completed, failed and incomplete execution must remain distinguishable. Persistence across process restart depends on the storage provider; default standalone memory is not durable.

Ask about this agent

Start an operator-owned diagnostic session from an agent, a selected thread or a captured error. Ask which tools failed, whether retained messages contain problematic content, or what records support a disputed response. Answers cite available thread, message, tool and trace identifiers, distinguish observations from hypotheses, and identify missing capture.

The diagnostic agent reads a fixed source-history snapshot or starts with an empty fork for state-only diagnosis. Explicit tools can inspect other target threads or newer records. It can read configuration, registered capabilities, captured errors, persisted custom state, runtime health and execution evidence. Applications can expose read-only snapshots through GetAdminDiagnosticSnapshotAsync(CancellationToken); arbitrary private-field reflection is excluded.

One admin turn per target agent can overlap normal user processing. Other admin turns and disruptive lifecycle changes conflict while it runs. The reserved _admin channel is reachable only through privileged dispatch; ordinary chat/events reject both _admin and _debug. Diagnostic execution has separate processing, cancellation, run-safety, model usage and response paths.

Sessions and diagnostic tool results are stored separately from normal threads, scoped to principal, agent and authenticated operator. Source history is copied at creation and never altered by diagnostic persistence or compaction. Server-generated session IDs and stable client turn IDs prevent duplicate completed turns; interrupted turns are marked incomplete instead of replayed. Other operators start their own sessions.

Configure FabrCore:Administration:Model to override the target model alias. TimeoutSeconds under that section defaults to 120 (bounded to 1–600). A missing model is an explicit error. Limits are 100 sessions per operator/agent, 100 turns per session, 16,000 input characters per turn and a 1 MiB source snapshot. Insights lets operators resume sessions and stop waiting; stopping polling does not cancel an accepted turn. Mutations remain separate management workflows.

Read monitoring without a constant stream

Provider query interfaces complement existing recording interfaces. Filter by principal, agent, trace, time, kind, channel and execution category before paging. Pages default to 100 records (maximum 1,000), enforce byte limits and carry opaque cursors and source attribution. Fetch larger captured payloads separately in bounded chunks. Missing or redacted capture cannot be recovered by polling.

Query silo-local stores separately and shared stores once per opaque source identity. Show offline hosts, retention gaps, capture settings, queue depth, lag, drops and persistence failures. A successful query from one source does not prove complete cluster coverage. Token/cost summaries identify their scope; admin model use is attributed separately to actor, target, session and _admin. Diagnostic transcripts require administration access.

Administration clients should bound polling and concurrent source queries, pause refresh for hidden views, and expose partial coverage. The Host API does not impose a particular client UI refresh schedule.

{
  "FabrCore": {
    "Monitoring": {
      "Provider": "sql",
      "RetentionDays": 7,
      "QueueRecords": 10000,
      "QueueBytes": 67108864,
      "BatchSize": 250
    }
  }
}

SQL monitoring requires the configured operational database. It buffers up to 10,000 records / 64 MiB, flushes batches of up to 250 every second and retains seven days by default. Recording enqueues without awaiting SQL. Bounded retries and drop/failure reporting keep SQL persistence off the agent's awaited write path; durability begins after a successful flush. Disabled, in-memory and custom providers remain supported. Apply the additive monitoring migration first in manual-schema mode. Existing audit functionality and APIs remain unchanged.

Retrieve verifiable execution evidence

Discover retained traces and create an immutable export of a selected trace. Read its bounded base64 chunks, concatenate decoded bytes in order and check the manifest's SHA-256 revision and byte count. New records do not change an existing export. Preserve records, signatures, certificates and attestations exactly; delete the stored export after retrieval.

Distinguish verified, unsigned, tampered and incomplete results. Transport integrity, trusted signatures and source coverage are separate questions. Signing remains opt-in, and a local export is not a unified export of every cluster store. See verifiable execution.

API quick reference

Paths below are relative to /fabrcoreapi/admin/v1. {p} is the explicit principal and {a} is a local agent handle; encode each segment once.

Build a compatible server

Start with the wire protocol, administration contract and independent reference cloud server. The sample's bounded in-memory command queue demonstrates conformance; production services need their own durable broker and operator authentication.

Copy the entire cloud administration skill folder, including references, SDK example and migration, for your coding assistant. The distribution catalog also covers agents, ACLs, monitoring and evidence.

The validation report separates deterministic and SQL checks from remaining multi-silo stress/failover, sustained SQL and real-model evaluations. The less-than-5% throughput/p95 regression target requires representative workload measurement; it is not a production performance guarantee.

Connection administration and user handoffs

Manage profiles, exact agent grants and authorization through the existing cloud channel. Conditional profile writes use revisions. User consent remains in the client application; enable encrypted handoffs when relaying it through a cloud broker. Read the integration guide.