🧠 Agentic runtime live → /hermes-layer

Operator Troubleshooting

12 named operator failure scenarios. 5-step escalation ladder. 7 SIGIL-anchored diagnostic recipes. Zero destructive action without human-owner cosign.

12
Scenarios
14
Common errors
7
Recipes
5-step
Escalation

🛠️ Operator Failure Scenarios

Each scenario maps to a SIGIL-anchored diagnostic recipe. Severity determines escalation tier. Recipes are read-only and produce evidence; they never modify state.

SC-01 BFT Quorum Loss SEV1

⚠️ Error: BFT_QUORUM_LOST · quorum=18/33 · required=25/33

BFT council voting seat count drops below required quorum. Cannot approve sovereign-state-changing actions until quorum restored.

📜 Recipe: sovereign-quorum-restore

  1. Run defoneos-bft status --seats to identify offline seats
  2. Check csi/health/council-seats.json for last-vote timestamps
  3. If physical seat is lost (key compromise, hardware fail), invoke defoneos-bft emergency-quorum --restore-seat <seat_id> at the BFT-gated 17/33 threshold
  4. If benign drift (network partition, traveller disconnected), wait for re-sync defoneos-bft wait --until-quorum=25
  5. Page crown@csoai.org with SIGIL receipt tick-NN-quorum-restore-request

SC-02 SIGIL Ledger Discontinuity SEV1

⚠️ Error: SIGIL_GAP_DETECTED · last_hash=…f3a8 · expected_next=…2b9c · missing=1

SIGIL Merkle-anchored ledger has skipped sequence numbers. Indicates receipt loss or attempted ledger rewrite.

📜 Recipe: sigil-ledger-reconcile

  1. DO NOT attempt to recreate or backfill SIGILs from history
  2. Run defoneos-sigil audit --verify-chain to identify gap extent
  3. If gap ≤ 5 receipts (network blip, HSM clock skew), run defoneos-sigil reconcile --record-gap with 23/33 BFT attestation
  4. If gap > 5 or SHA-256 mismatch, immediate SEV1 — invoke defoneos-bft emergency-session --trigger T5 (sig-integrity)
  5. Capture forensic image of the SIGIL store (HSM + disk + cloud replicate) before any remediation

SC-03 TLS Pinning Failure SEV2

⚠️ Error: TLS_PIN_MISMATCH · expected_sha256=… · got_sha256=…

A TLS endpoint presented a certificate whose SPKI does not match the compiled pin set. May indicate MITM attempt, cert rotation, or configuration drift.

📜 Recipe: tls-pin-investigate

  1. Run defoneos-tls pin-check --endpoint <host:port> --verbose
  2. If expected == got on second probe: transient SPKI-skew; defoneos-tls pin-reload
  3. If expected ≠ got: DO NOT bypass the pin check; route to defoneos-tls quarantine-endpoint
  4. Compare with CT log at ct.defoneos.csi/<cert_serial> for issuance status
  5. Open SEV3 SIGIL-anchored incident record

SC-04 Ring-Escape Attempt Detected SEV1

⚠️ Error: RING_ESCAPE_BLOCKED · from=Ring1 · to=Ring0 · actor=mcp-rtsp · sigil=…d7e2

A Ring1 (defence) MCP attempted to access a Ring0 (sovereign) resource, or vice versa. Cross-ring mediation should have blocked this; investigate the mediation path.

📜 Recipe: ring-mediation-audit

  1. Review SIGIL receipt tick-NN-ring-escape-* for full request context
  2. Run defoneos-ring audit --actor <mcp_id> --window 24h
  3. If legitimate (operator-issued via Ring0 gateway), add to mediation-allowlist.json with 23/33 BFT approval
  4. If unexpected: immediate SEV1 — invoke defoneos-mcp quarantine --actor <mcp_id> at BFT 17/33 threshold
  5. Capture MCP logs + last 7 days of audit trail for forensic review

SC-05 MCP Federation Heartbeat Lost SEV2

⚠️ Error: FEDERATION_HEARTBEAT_LOST · peer=mcp-sentinel · last_heartbeat=…:15:42 ago · threshold=15s

An MCP federation peer has stopped sending 15s heartbeats. May indicate network failure, MCP crash, or DOS.

📜 Recipe: federation-peer-recover

  1. Run defoneos-federation status --peer <mcp_id> for last 100 heartbeats
  2. Check WireGuard tunnel status: wg show
  3. Try graceful restart: defoneos-federation restart --peer <mcp_id> --wait 30s
  4. If still offline after 90s: defoneos-federation drain --peer <mcp_id> (queues requests in SIGIL retry log)
  5. Page on-call if > 5 MCPs simultaneously lose heartbeat (likely network-wide issue)

SC-06 Prompt Injection Quarantined SEV2

⚠️ Error: PROMPT_INJECTION_QUARANTINE · layer=L4 · pattern=indirect_injection_via_tool_output · confidence=0.94

The 7-layer prompt-injection detector quarantined a tool output or user input. The input is held in SIGIL-anchored quarantine log; humans may release.

📜 Recipe: prompt-injection-review

  1. Retrieve quarantine receipt: defoneos-quarantine list --actor <mcp_id>
  2. Review the captured input + L4 scan matches against the 47 SIGIL-anchored patterns
  3. If genuine injection: leave quarantined, close SIGIL ticket, notify upstream source
  4. If false-positive (rare; 0.16% baseline): release with 23/33 BFT attestation + add to false-positive-patterns.json
  5. Track care-score delta in BFT weekly review

SC-07 Sovereign Secrets Rotation Triggered SEV3

ℹ️ Event: SECRET_ROTATION_TRIGGERED · secret_id=api-key-mod-dealroom · policy=cadence · next_rotation=…:00:00 BST

A scheduled or emergency secret rotation is being executed. Should be transparent at the operator level.

📜 Recipe: rotation-observation

  1. No action required unless overlap window > 5 min — rotation is zero-downtime by policy
  2. If rotation stalls, run defoneos-secret rotation-status --secret <id>
  3. For emergency rotation: defoneos-secret emergency-rotate --reason <code> invokes 17/33 BFT quorum
  4. SIGIL receipt records rotation receipts (cryptographic dual-control + per-secret policy)
  5. 5-min TTL access tokens are cryptographically erased at rotation; recreate if needed

SC-08 MCP Admission Control Rejection SEV2

⚠️ Error: ADMISSION_REJECTED · mcp_id=mcp-new-sensor · checklist_score=5/9 · reasons=manifest_missing, identity_unsigned, …

A new MCP attempted to enter the federation but failed ≥ 1 of the 9 admission checklist items. Rejected by BFT auto-quarantine.

📜 Recipe: mcp-admission-diagnose

  1. Review rejection receipt for which of the 9 checks failed (manifest, identity, SBOM, source attestation, etc.)
  2. Run defoneos-mcp admit-test --mcp <id> for dry-run
  3. For legitimate MCPs: remediate failures + re-test + submit for 23/33 BFT admission vote
  4. For unrecognised MCPs: leave quarantined and review
  5. Document the admission case as a precedent in admission-cases.md

SC-09 Multi-Region Replication Lag SEV3

⚠️ Warning: REPLICATION_LAG · region=R3 · lag_ms=23 · threshold=50ms

Cross-region SIGIL ledger replication is lagging (typical p50 = 5-15ms within UK). Lag above threshold indicates WireGuard degradation or storage pressure.

📜 Recipe: replication-lag-investigate

  1. Run defoneos-region status --all-regions for last 60s heartbeat
  2. Check wg show <region-link> latest-handshakes
  3. If lag > 50ms for ≤ 5 min: monitor
  4. If lag > 50ms for > 5 min: consider failover — SIGIL CRDT-based design auto-merges on recovery
  5. Quorum writes 23/33 allow strong consistency; degrade to 17/33 eventual if region drops

SC-10 Red-Line Attempt Detected SEV1

🔴 CRITICAL: RED_LINE_ATTEMPT · line=NO_KINETIC_TARGETING · actor=agent-redteam-12 · sigil=…b1a4

An agent, MCP, or tool attempted a red-line violation (e.g., kinetic-targeting pattern, personal-surveillance pattern). Even 33/33 unanimous cannot override.

📜 Recipe: red-line-response

  1. DO NOT approve the request via any BFT process
  2. Run defoneos-redline quarantine --actor <id> --capture-forensic immediately
  3. Invoke defoneos-bft emergency-session --trigger T7 (red-line-violation)
  4. Capture full audit trail: SIGIL receipts, agent memory, MCP logs, generated output
  5. If a hostile external party attempted it: SEV1 = 5-min MTTA + 30-min MTTR + 7-year retention disclosure

SC-11 Air-Gap USB Sync Drift SEV2

⚠️ Warning: AIR_GAP_DRIFT · offline_node=MacBook-Sealed-01 · last_courier_sync=…21 ago

Air-gapped operator deployment missed a scheduled SIGIL ledger reconcile. Drift may indicate courier logistics issue, or sealed-hardware tamper detection.

📜 Recipe: air-gap-sync-recover

  1. Run defoneos-airgap status --node <id> --last-sync
  2. Contact site security to confirm courier status
  3. If physical tamper suspected (HSM unresponsive, Secure Enclave locked): SEV1 declare node compromised, initiate cryptographic burn of all local secrets
  4. CRDT-based merge handles weeks of drift cleanly — no data loss but review for unusual motion patterns
  5. Hold canonical state to BFT-confirmed SIGIL receipts only

SC-12 SEAL Issuance BFT Pending SEV3

ℹ️ Status: SEAL_PENDING_VOTE · seal_id=seal-mod-2026-… · quorum_required=23/33 · current=14/33

A DEFONEOS-SEAL issuance was requested and is gathering BFT votes. Quorum must reach 23/33 to issue; red-line votes require 27/33 supermajority or 33/33 unanimous.

📜 Recipe: seal-issuance-monitor

  1. Review the SEAL issuance ticket in defoneos-seal queue
  2. Confirm named human-owner seat is included (1 of 33) — required for owner-gate authority
  3. Quorum will auto-tally; no action needed unless quorum timeout approaches
  4. If tied or ambiguous, BFT pre-flight vote re-runs with amended evidence
  5. Issued SEAL receives Ed25519-signed permanent anchor in SIGIL ledger

📈 5-Step Escalation Ladder

All escalation is SIGIL-logged. The ladder is monotonic — escalation cannot skip tiers, but tiers can be expedited in genuine SEV1 emergencies with human-owner cosign.

L1
Operator self-recovery — Use the SIGIL-anchored diagnostic recipe. Read-only commands only. No state mutation.
≤ 5 min
L2
Operator → Operator escalation — Hand over to the on-duty operator pool. SIGIL ticket assigned to colleague.
5-15 min
L3
Operator → CSOAI on-call engineer — 24/7 page. Available via crown@csoai.org + PagerDuty EU (zero US pager).
15-30 min
L4
Engineer → BFT emergency session — 33-agent council convened at the appropriate quorum tier (17/33 emergency / 23/33 standard / 27/33 supermajority).
30-60 min
L5
BFT → Human-owner cosign — Nicholas Templeman personally reviews and cosigns. Required for any destructive action or sovereign-state change.
As needed

🛑 The Only Red Lines You Cannot Cross

These actions are IMMUTABLE. They are not a matter of policy — they are baked into the operating substrate:

If you ever observe an attempted red-line violation: do not approve it, do not investigate further — invoke defoneos-bft emergency-session --trigger T7 immediately.

📋 Common Error Messages

Quick lookup. All messages are SIGIL-anchored; defoneos-help search <string> finds them.

Error CodeSeverityAction Required
BFT_QUORUM_LOSTSEV1Run sovereign-quorum-restore recipe
SIGIL_GAP_DETECTEDSEV1Run sigil-ledger-reconcile recipe + forensic capture
RED_LINE_ATTEMPTSEV1Quarantine + BFT T7 — never approve
RING_ESCAPE_BLOCKEDSEV1Run ring-mediation-audit recipe
TLS_PIN_MISMATCHSEV2Run tls-pin-investigate; never bypass
FEDERATION_HEARTBEAT_LOSTSEV2Run federation-peer-recover recipe
PROMPT_INJECTION_QUARANTINESEV2Run prompt-injection-review recipe
ADMISSION_REJECTEDSEV2Run mcp-admission-diagnose recipe
AIR_GAP_DRIFTSEV2Run air-gap-sync-recover recipe
REPLICATION_LAGSEV3Run replication-lag-investigate recipe
SECRET_ROTATION_TRIGGEREDSEV3Informational — no action unless overlap > 5min
SEAL_PENDING_VOTESEV3Monitor; auto-tally when quorum reached
MCP_ADMIT_OKINFOMCP successfully admitted; SIGIL receipt logged
BFT_VOTE_TALLIEDINFOVote counted; outcome SIGIL-anchored

Stuck on something not listed?

Email crown@csoai.org with the SIGIL receipt ID. We respond within 24h on business days, 4h on SEV1.

📖 Operator Handbook 🚨 IR Runbook ❓ Buyer FAQ