Skip to main content

​
Overview

The TraceCtrl Engine is a FastAPI application that:
  1. Runs a background pipeline every 60 seconds to process new spans
  2. Serves a REST API for the dashboard and external integrations
  3. Maintains cumulative state in ClickHouse using ReplacingMergeTree tables

​
Pipeline

The pipeline is scheduled via APScheduler and runs at the interval set by PIPELINE_INTERVAL_SECONDS (default: 60).
1

Fetch new spans

Reads from otel_traces where Timestamp > last_watermark. Extracts 20+ attributes per span including all tracectrl.*, openinference.*, and framework-specific fields.
2

Update agent inventory

Groups spans by tracectrl.agent.id and upserts agent records. Tracks cumulative observation_count, run_count, tools_observed, and computes maturity (LEARNING → MATURE after 10 observations).
3

Build topology

Creates agent-to-agent edges (via tracectrl.caller.agent_id or parent span resolution) and agent-to-tool edges. Edges carry cumulative counts and confidence levels (LOW → MEDIUM → HIGH).
4

Advance watermark

Updates pipeline_state.last_processed_at. Only advances on success — if the pipeline fails or finds no new spans, the watermark stays put so no data is ever skipped.

​
Configuration

VariableDefaultDescription
CLICKHOUSE_HOSTlocalhostClickHouse host
CLICKHOUSE_PORT9000ClickHouse native port
CLICKHOUSE_DBtracectrlDatabase name
PIPELINE_INTERVAL_SECONDS60Pipeline cycle interval

​
Running Locally

cd tracectrl
source .venv/bin/activate
pip install -r engine/requirements.txt
uvicorn engine.main:app --reload --port 8000
When running locally, ensure ClickHouse is accessible at the configured host/port and the tracectrl database exists with the schema from config/schema.sql.