> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tracectrl.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Architecture

> How TraceCtrl's components fit together

## System Overview

TraceCtrl has five layers:

1. **Python SDK** (in your agent) — emits OpenTelemetry spans enriched with `tracectrl.*` security attributes, plus optional in-process guardrails.
2. **Protector Plus** (optional external HTTP firewall) — called from `tracectrl.protector`, emits guardrail evaluation spans.
3. **OTel Collector** — receives OTLP spans and writes them to ClickHouse.
4. **Intelligence Engine** — FastAPI service that runs the pipeline and serves the REST API.
5. **Dashboard** — React SPA that visualizes agents, traces, alerts, guardrails, and attack paths.

![TraceCtrl architecture: a Python agent process runs the SDK (Framework Instrumentor + TraceCtrl SDK core + Guardrails), exports OTLP spans on :4317/:4318 to the OTel Collector. The Collector forwards to ClickHouse (:8123/:9000) which holds otel\_traces, agent\_inventory, topology edges, attack paths, guardrail registry + violations, scan results, and the protector\_plus\_config. The Intelligence Engine (FastAPI :8000) runs a 60-second pipeline that builds inventory + topology, ingests guardrail spans, and computes the attack graph; it serves /api/v1/\* REST + SSE. The React Dashboard (:3000) renders Monitor (agents/sessions/topology), Security (alerts/guardrails/scan/risk/attacks), and Configure (settings). The Guardrails block also talks HTTP to an external Protector Plus firewall (optional); both SDK and Protector Plus violations flow to the same engine pipeline.](https://mintcdn.com/tracectrl/CrL8lse7B3o3gfNI/images/tracectrl-architecture.svg?fit=max\&auto=format\&n=CrL8lse7B3o3gfNI\&q=85\&s=6feb1bba7bd91eb9664fba867459b30f)

## Component Details

### Python SDK

The SDK wraps [OpenInference](https://github.com/Arize-ai/openinference) framework instrumentors with a `TraceCtrlSpanProcessor` that enriches every span with security attributes. It uses OpenTelemetry as the transport — no custom protocols, no vendor lock-in.

**Key modules:**

* `tracectrl.config` — `configure()` sets up the TracerProvider with OTLP gRPC exporter.
* `tracectrl.processor` — `TraceCtrlSpanProcessor` adds agent identity, tool categories, session IDs, ingress markers.
* `tracectrl.inference` — Classifies tools into risk categories.
* `tracectrl.session` — Session ID management via Python `contextvars`.
* `tracectrl.context` — W3C traceparent propagation for cross-service agents.
* `tracectrl.guardrails` — In-process LLM-judge guardrails. Emits `tracectrl.guardrail.registered` (on registration) and `tracectrl.guardrail.evaluation` (per evaluation) spans. {/* Sources: sdk/tracectrl/src/tracectrl/guardrails/ */}
* `tracectrl.protector` — TraceCtrl Guards integration with external Protector Plus HTTP firewall. Emits the same guardrail span types, tagged with `tracectrl.guardrail.provider = "protector_plus"`. {/* Sources: sdk/tracectrl/src/tracectrl/protector.py */}

### Protector Plus (optional)

An external HTTP firewall called from `tracectrl.protector`. Exposes seven sub-guardrails (`llm`, `keyword`, `regex`, `pii`, `vector`, `content_moderation`, `system_prompt_protection`) via `/apikey/api/protectorplus/v1/input-check` and `.../output-check`. The SDK calls it fire-and-forget; flagged sub-guardrails are emitted as `tracectrl.guardrail.evaluation` spans (`decision=fail`) for the engine to ingest. Config (endpoint URL, API key, enabled guardrails) is persisted in `protector_plus_config` and configured from the Settings page. {/* Sources: sdk/tracectrl/src/tracectrl/protector.py, engine/db/client.py:133-140 */}

### OTel Collector

A standard OpenTelemetry Collector configured to receive OTLP spans on gRPC (`:4317`) and HTTP (`:4318`), then export them to ClickHouse via the `clickhouseexporter`. Configuration lives in `config/otel-collector.yaml`. {/* Sources: docker-compose.yml:7-9 */}

### Intelligence Engine

A FastAPI application that:

1. **Runs the pipeline** every `PIPELINE_INTERVAL_SECONDS` (default 60). Each tick re-reads all spans, refreshes the agent inventory, topology, guardrail registry, guardrail violations, and the attack-graph analysis. Idempotency comes from ClickHouse `ReplacingMergeTree` rather than a watermark. {/* Sources: engine/scheduler.py:23, engine/pipeline/runner.py */}
2. **Serves a REST API** under `/api/v1` — system, topology, sessions, agents, risk, scan, violations, guardrails. {/* Sources: engine/main.py:37-44 */}
3. **Owns the ClickHouse schema** — `ensure_schema()` runs the CREATE/ALTER statements at startup. {/* Sources: engine/db/client.py:44-146 */}

### Dashboard

A React + Vite + TypeScript SPA. The sidebar groups pages into three sections (**Monitor**, **Security**, **Configure**) and the route table lives in `ui/src/App.tsx:145-158`. See the [Dashboard page](https://docs.tracectrl.ai/platform/dashboard.md) for the per-page reference.

## Data Pipeline (one tick)

**1. Fetch spans**

Read all spans from `otel_traces`.

**2. Inventory**

Upsert agents with cumulative observation/run counts and tool lists.

**3. Topology**

Upsert agent-to-agent and agent-to-tool edges with confidence scoring.

**4. Violations**

Ingest `tracectrl.guardrail.evaluation` spans with `decision in ('fail','error')` into `guardrail_violations`.

**5. Guardrail registry**

Ingest `tracectrl.guardrail.registered` spans into `guardrail_registry`; flip health to `error` when the last hour shows error-decision violations.

**6. Attack graph**

Evaluate TAGAAI rules, generate attack paths, score risk, write `attack_paths` / `agent_risk_scores` / `system_risk`.

## ClickHouse Tables

| Table | Engine | Purpose |
| - | - | - |
| `otel_traces` | Auto-created by Collector | Raw OpenTelemetry spans |
| `agent_inventory` | ReplacingMergeTree | Deduplicated agent records |
| `topology_agent_edges` | ReplacingMergeTree | Agent-to-agent connections |
| `topology_tool_edges` | ReplacingMergeTree | Agent-to-tool connections |
| `pipeline_state` | ReplacingMergeTree | Pipeline state (legacy watermark key) |
| `attack_paths` | ReplacingMergeTree | TAGAAI-detected attack paths |
| `agent_risk_scores` | ReplacingMergeTree | Per-agent risk |
| `system_risk` | ReplacingMergeTree | System-wide risk summary |
| `scan_results` / `scan_topology` / `scan_runs` | MergeTree / ReplacingMergeTree | Static OpenClaw scan output |
| `guardrail_violations` | ReplacingMergeTree | Guardrail evaluation failures (judge-LLM + Protector Plus) |
| `guardrail_registry` | ReplacingMergeTree | Registered guardrails (per agent) |
| `protector_plus_config` | ReplacingMergeTree | Single-row Protector Plus config |

> [!NOTE]
>
> ReplacingMergeTree deduplicates rows during background merges. Always query with `FINAL` for correct results: `SELECT * FROM agent_inventory FINAL`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.