AI Observability

How to Build an AI Agent Observability Dashboard for Enterprise Fintech Teams

Scott Miller

19 Jul 2026 — 12 min read

If you are running AI agents in a production fintech environment today, you are almost certainly flying partially blind. You may know your models are responding, but do you know exactly what they cost per transaction, how their latency distributes across your user base, and whether their accuracy is drifting as market conditions shift? More urgently: can you prove all of that to a regulator?

With the EU AI Act's high-risk financial system provisions entering their enforcement phase and the SEC's AI model governance guidance tightening throughout H2 2026, the window to build robust observability infrastructure before it becomes a compliance requirement is closing fast. This guide walks enterprise fintech engineering and ML platform teams through building a production-grade AI agent observability dashboard from the ground up, covering instrumentation, telemetry collection, pipeline architecture, and the specific metrics that will matter most when auditors come knocking.

This is not a toy demo. Every pattern described here is designed for multi-model inference pipelines running at scale, handling sensitive financial data, and operating under strict latency and accuracy SLAs.

Why Fintech AI Observability Is a Different Problem

General-purpose LLM observability tools are proliferating rapidly in 2026. Tools like Langfuse, Arize Phoenix, Weights and Biases Weave, and OpenLLMetry have matured significantly. But fintech teams face a fundamentally different observability challenge for several reasons:

Regulatory audit trails: Every inference decision that touches a credit score, fraud flag, trade recommendation, or KYC outcome may need to be reconstructed months later with full fidelity.
Multi-model complexity: Production fintech pipelines rarely use a single model. A typical workflow might chain a routing classifier, a retrieval-augmented generation (RAG) layer, a specialized financial reasoning model, and a guardrails model, each with its own cost and latency profile.
Cost attribution: Token costs must be attributed to specific business units, products, and customer segments for accurate P&L reporting.
Accuracy drift in adversarial conditions: Financial markets and fraud patterns shift rapidly. Model accuracy degradation can be both a performance issue and a compliance issue simultaneously.
Data residency constraints: Telemetry pipelines themselves must respect data sovereignty rules, meaning you often cannot simply ship raw prompt/completion pairs to a third-party SaaS observability tool.

With that context established, let's build the system.

Step 1: Define Your Telemetry Schema Before You Write a Single Line of Instrumentation Code

The single biggest mistake teams make is instrumenting first and designing their schema later. In a fintech context, your telemetry schema is effectively a legal document. Design it with that gravity.

Your core telemetry event for each model call should capture the following fields:

Required Fields per Inference Event

trace_id: A globally unique identifier that links every model call in a single agent workflow together. Use UUID v7 (time-ordered) for easy chronological sorting.
span_id: Identifies the specific model call within a trace. This maps directly to OpenTelemetry span semantics.
model_id: The exact model version called, including provider, family, and version tag (e.g., anthropic/claude-4-sonnet-20260301). Never log just "Claude" or "GPT." Version specificity is non-negotiable for audit purposes.
pipeline_stage: Which stage in your multi-model chain this call represents (e.g., intent_classifier, rag_synthesizer, compliance_guardrail).
input_token_count / output_token_count: Raw token counts, not just cost estimates, since pricing changes over time and you need to be able to recompute costs retrospectively.
latency_ms: Wall-clock latency from request dispatch to first token received (TTFT) and to full completion (total latency). Both matter for user experience and SLA reporting.
cost_usd: Computed cost at time of call, stored alongside the token counts.
business_context: A structured metadata object carrying non-PII business identifiers: product line, business unit, customer segment tier, and workflow type.
outcome_label: For pipelines where ground truth is eventually available (fraud outcomes, credit decisions), a nullable field to be populated asynchronously for accuracy tracking.
guardrail_triggered: Boolean plus category if a safety or compliance guardrail intercepted the response.
data_residency_region: Where the inference was processed, critical for GDPR and data sovereignty compliance.

Define this schema in a shared Protobuf or Avro definition file stored in your internal schema registry. Every team instrumenting a model call must use the same schema. This is infrastructure, not a suggestion.

Step 2: Instrument Your Inference Pipeline with OpenTelemetry

OpenTelemetry (OTel) has become the de facto standard for distributed tracing in 2026, and it is the right foundation for AI agent observability in enterprise environments because it is vendor-neutral, has first-class support in every major cloud provider, and produces audit-friendly structured data.

The key is to treat each model call as an OTel span and each end-to-end agent workflow as an OTel trace. Here is a practical Python instrumentation pattern for a multi-model pipeline:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
import time, uuid

# Initialize the tracer
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://your-otel-collector:4317")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("fintech-ai-pipeline", "1.0.0")

def instrument_model_call(
    model_id: str,
    pipeline_stage: str,
    prompt: str,
    business_context: dict,
    model_fn: callable
):
    with tracer.start_as_current_span(f"model_call.{pipeline_stage}") as span:
        span.set_attribute("ai.model.id", model_id)
        span.set_attribute("ai.pipeline.stage", pipeline_stage)
        span.set_attribute("ai.business.unit", business_context.get("unit"))
        span.set_attribute("ai.business.product", business_context.get("product"))
        span.set_attribute("ai.customer.segment", business_context.get("segment"))

        start_time = time.monotonic_ns()

        try:
            response = model_fn(prompt)
            latency_ms = (time.monotonic_ns() - start_time) / 1_000_000

            # Extract token counts from provider response metadata
            input_tokens = response.usage.input_tokens
            output_tokens = response.usage.output_tokens
            cost_usd = compute_cost(model_id, input_tokens, output_tokens)

            span.set_attribute("ai.tokens.input", input_tokens)
            span.set_attribute("ai.tokens.output", output_tokens)
            span.set_attribute("ai.cost.usd", cost_usd)
            span.set_attribute("ai.latency.ms", latency_ms)
            span.set_attribute("ai.guardrail.triggered", False)
            span.set_status(trace.StatusCode.OK)

            return response

        except GuardrailException as e:
            span.set_attribute("ai.guardrail.triggered", True)
            span.set_attribute("ai.guardrail.category", e.category)
            span.set_status(trace.StatusCode.ERROR, str(e))
            raise

Wrap every model call in your pipeline with this pattern. The business_context dictionary is the bridge between your AI telemetry and your financial reporting systems. Never skip it, even in development environments, because you want your observability data to be consistent across all stages.

Handling the Multi-Model Chain

In a chained pipeline (classifier to RAG to reasoning model to guardrail), the parent trace must propagate through all child spans automatically. OTel's context propagation handles this, but you need to ensure your async task queues and message brokers (Kafka, RabbitMQ, Celery) are configured to carry OTel context headers. For Kafka specifically, use the opentelemetry-instrumentation-kafka-python package and inject span context into message headers at produce time, extracting it at consume time.

Step 3: Build the Telemetry Collection and Storage Layer

Raw OTel spans flow into an OTel Collector, which is your central routing and processing hub. Configure it with the following pipeline:

OTel Collector Configuration Strategy

Receivers: Accept OTLP over gRPC (port 4317) and HTTP (port 4318) from all instrumented services.
Processors: Apply a batch processor for efficiency, a memory_limiter to prevent OOM crashes under spike load, and a custom attributes processor to hash or redact any PII that may have leaked into span attributes before storage.
Exporters: Fan out to two destinations: (1) a time-series database for real-time dashboarding and (2) a cold storage layer for long-term audit retention.

For the time-series database, ClickHouse has emerged as the dominant choice for AI telemetry in 2026 for good reason. Its columnar storage handles the high-cardinality, high-volume nature of inference telemetry far better than Prometheus or InfluxDB, and its SQL interface makes it accessible to data analysts who need to build compliance reports without learning PromQL. A single ClickHouse node can comfortably ingest millions of span events per day on modest hardware.

For long-term audit retention, export a daily snapshot of all inference events to your data lake (S3-compatible object storage in Parquet format). Retain for a minimum of 7 years to align with standard financial record-keeping requirements. Compress with Zstandard (Zstd) for a typical 6:1 compression ratio on telemetry data.

ClickHouse Schema for AI Inference Events

CREATE TABLE ai_inference_events (
    event_time       DateTime64(3, 'UTC'),
    trace_id         String,
    span_id          String,
    model_id         LowCardinality(String),
    pipeline_stage   LowCardinality(String),
    business_unit    LowCardinality(String),
    product_line     LowCardinality(String),
    customer_segment LowCardinality(String),
    input_tokens     UInt32,
    output_tokens    UInt32,
    cost_usd         Float64,
    latency_ms       Float64,
    ttft_ms          Float64,
    guardrail_triggered UInt8,
    guardrail_category  LowCardinality(String),
    outcome_label    Nullable(String),
    data_region      LowCardinality(String),
    status_code      LowCardinality(String)
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(event_time)
ORDER BY (business_unit, product_line, event_time)
TTL event_time + INTERVAL 90 DAY TO DISK 'cold_storage';

The LowCardinality type on repeated string fields like model_id and pipeline_stage dramatically reduces storage and query costs. The TTL clause automatically migrates data older than 90 days to cheaper cold storage while keeping it queryable.

Step 4: Implement Asynchronous Accuracy Feedback Loops

Cost and latency telemetry are synchronous: you know them the moment a call completes. Accuracy is fundamentally asynchronous in financial applications. A fraud detection model's accuracy on a given transaction may not be known until days or weeks later when chargebacks are processed. A credit underwriting model's accuracy unfolds over the loan's lifetime.

Build an outcome ingestion service that accepts delayed ground truth labels and joins them back to the original inference events by trace_id:

class OutcomeIngestionService:
    def __init__(self, clickhouse_client, event_store):
        self.ch = clickhouse_client
        self.store = event_store

    def record_outcome(
        self,
        trace_id: str,
        outcome_label: str,
        outcome_timestamp: datetime,
        outcome_source: str  # e.g., "chargeback_processor", "loan_servicing"
    ):
        # Update the inference event with the ground truth label
        self.ch.execute("""
            ALTER TABLE ai_inference_events
            UPDATE
                outcome_label = %(label)s,
                outcome_timestamp = %(ts)s,
                outcome_source = %(source)s
            WHERE trace_id = %(trace_id)s
        """, {
            "label": outcome_label,
            "ts": outcome_timestamp,
            "trace_id": trace_id,
            "source": outcome_source
        })

        # Trigger accuracy metric recomputation for the relevant model/segment
        self._recompute_rolling_accuracy(trace_id)

This closes the feedback loop between your AI system and your business outcomes, giving you the accuracy telemetry that regulators will increasingly require as evidence that your models are performing as declared in their model risk management (MRM) documentation.

Step 5: Build the Grafana Dashboard Layer

With ClickHouse as your backend, Grafana with the ClickHouse data source plugin is the fastest path to a production-grade dashboard. Structure your dashboard into four focused panels:

Panel 1: Real-Time Cost Telemetry

This panel answers the question: "What is this AI infrastructure costing us right now, and where is the money going?"

Time-series chart of cost per hour by model_id, stacked by pipeline stage.
Bar chart of cost by business unit for the current billing period, with budget threshold lines.
Stat panel showing cost per 1,000 inferences by product line, updated every 5 minutes. This is your unit economics metric.
Alert rule: Trigger a PagerDuty notification if hourly cost for any single model exceeds a configurable threshold (start at 2x the 7-day rolling average).

Panel 2: Latency Distribution

Do not use average latency. Averages lie. Use percentiles.

Heatmap of latency distribution (p50, p90, p95, p99) per pipeline stage, updated in real time.
Time-series of TTFT (time to first token) for streaming endpoints, which directly maps to user-perceived responsiveness.
SLA breach counter: Number of inferences per hour that exceeded your defined latency SLA per product tier.

Panel 3: Accuracy and Model Health

Rolling 7-day and 30-day accuracy by model and customer segment, plotted as a time series to reveal drift trends.
Confusion matrix breakdown for classification models (fraud detection, intent routing), refreshed daily as outcome labels arrive.
Guardrail trigger rate over time: A sudden spike in guardrail activations is an early warning signal of either adversarial activity or model drift.
Population Stability Index (PSI) chart for input feature distributions, critical for detecting data drift in credit and fraud models.

Panel 4: Compliance and Audit Readiness

This panel is specifically designed for your Chief Risk Officer and compliance team, not your engineers.

Model version inventory: A live table showing every model version currently in production, its deployment date, its inference volume in the current period, and a link to its MRM documentation.
Data residency compliance map: Inference volume by data_region, flagging any volume that may have been processed outside approved jurisdictions.
Explainability coverage rate: Percentage of high-stakes inferences (credit decisions, fraud flags) for which an explanation record was generated and stored.
Audit log export button: A one-click Grafana action that triggers a ClickHouse query exporting all inference events for a specified trace, model, or time range to a signed, tamper-evident CSV for regulatory submission.

Step 6: Implement Cost Anomaly Detection with Statistical Process Control

Real-time dashboards show you what is happening. Anomaly detection tells you when something unexpected is happening before it becomes a crisis. For AI cost telemetry, Statistical Process Control (SPC) using CUSUM (Cumulative Sum) charts is more appropriate than simple threshold alerting because it detects sustained shifts in cost or latency that individually stay below alert thresholds but collectively signal a problem.

import numpy as np
from collections import deque

class CUSUMCostAnomalyDetector:
    def __init__(self, target_cost_per_1k: float, k_factor: float = 0.5, h_threshold: float = 5.0):
        """
        target_cost_per_1k: Expected cost per 1,000 inferences (your baseline)
        k_factor: Allowance factor (typically 0.5 * sigma)
        h_threshold: Decision threshold (typically 4-5 * sigma)
        """
        self.target = target_cost_per_1k
        self.k = k_factor
        self.h = h_threshold
        self.cusum_high = 0.0
        self.cusum_low = 0.0
        self.history = deque(maxlen=1000)

    def update(self, observed_cost_per_1k: float) -> dict:
        self.history.append(observed_cost_per_1k)
        sigma = np.std(self.history) if len(self.history) > 10 else 1.0

        normalized = (observed_cost_per_1k - self.target) / sigma
        self.cusum_high = max(0, self.cusum_high + normalized - self.k)
        self.cusum_low = min(0, self.cusum_low + normalized + self.k)

        return {
            "anomaly_high": self.cusum_high > self.h,   # Costs trending up
            "anomaly_low": self.cusum_low < -self.h,     # Costs trending down (unusual too)
            "cusum_high": self.cusum_high,
            "cusum_low": self.cusum_low,
            "signal": "ALERT" if (self.cusum_high > self.h or self.cusum_low < -self.h) else "NORMAL"
        }

Run this detector per model per business unit, feeding it 5-minute cost-per-1k-inferences aggregates from ClickHouse. Connect the output to your alerting system. A cost anomaly in a fraud detection pipeline might mean a prompt injection attack is inflating token usage; catching it in minutes rather than hours can save thousands of dollars and prevent a security incident from escalating.

Step 7: Prepare Your Audit Export and Regulatory Reporting Package

As H2 2026 regulatory requirements tighten, your observability infrastructure needs to produce structured compliance artifacts on demand. Build the following into your platform now:

The Model Performance Report (MPR)

A scheduled weekly report (auto-generated every Monday, delivered to your MRM team) covering: model version in production, inference volume, mean and p99 latency, cost per unit, accuracy metrics with confidence intervals, guardrail activation rate, and any SLA breaches. Generate it as a PDF from a headless Grafana render and store it in your document management system with an immutable timestamp.

The Inference Audit Trail

For any individual customer decision (loan denial, fraud flag, trade alert), your compliance team must be able to reconstruct the full inference chain within minutes. Build a simple internal tool that accepts a trace_id or a customer decision reference ID and returns: the complete sequence of model calls with their inputs (redacted per your PII policy), outputs, costs, latencies, model versions, and the final decision with its confidence score. Store this reconstruction capability as a tested, documented runbook, not just an ad-hoc query.

Data Residency Certification

Generate a monthly certification report showing that all inferences involving EU resident data were processed within EU-approved regions, all inferences involving data subject to US financial privacy law were processed within US infrastructure, and no cross-border data transfers occurred outside approved mechanisms. This is a ClickHouse query away if you instrumented data_residency_region correctly from day one.

Step 8: Operationalize and Maintain

An observability system that nobody trusts is worse than no system at all. Operational discipline is what separates a dashboard that collects dust from one that drives decisions.

Run weekly observability reviews: A 30-minute standing meeting where ML engineers, product owners, and a risk representative review the prior week's cost, latency, and accuracy trends together. Decisions made here should be logged.
Test your instrumentation in CI/CD: Write integration tests that assert your instrumentation is emitting the correct span attributes. A model call that produces no telemetry is a silent compliance gap.
Version your telemetry schema: When you add new fields or change semantics, version the schema in your registry and maintain backward compatibility. Regulators may ask you to re-run analysis on historical data using current schema definitions.
Practice your audit response: Quarterly, run a tabletop exercise where your team simulates a regulatory inquiry. How long does it take to produce a full inference audit trail for a specific customer decision? Your target should be under 15 minutes.

The Regulatory Timeline You Are Working Against

To sharpen the urgency: the EU AI Act's obligations for high-risk AI systems in credit scoring, fraud detection, and financial advice are in active enforcement as of 2026. The Basel Committee's guidance on model risk for AI systems (BCBS d586 and subsequent updates) now explicitly references real-time monitoring as a supervisory expectation, not just a best practice. The SEC's updated guidance on AI use in investment advisory contexts requires firms to maintain records of model behavior that are "sufficient to reconstruct the basis for any automated recommendation." In the UK, the FCA's AI and Data Science Regulatory Sandbox graduates are feeding directly into binding guidance expected in Q4 2026.

None of these requirements can be met retroactively. You cannot instrument your pipeline after an audit inquiry and produce historical telemetry that does not exist. The infrastructure described in this guide takes a focused team approximately 6 to 10 weeks to implement properly. That timeline, against a Q4 2026 enforcement horizon, means the work needs to start now.

Conclusion: Observability Is Your Competitive Moat, Not Just a Compliance Tax

It is tempting to frame AI observability purely as a regulatory burden. Resist that framing. Teams that build deep observability infrastructure consistently discover that it pays for itself in cost savings alone within the first quarter of operation. Knowing your exact cost per inference by product line enables model selection decisions that can cut AI infrastructure spend by 30 to 50 percent without sacrificing accuracy. Knowing your latency distribution by pipeline stage reveals optimization opportunities that directly improve user experience metrics. And knowing your accuracy trends before your business stakeholders do gives your ML team the credibility and lead time to address drift proactively rather than reactively.

The fintech teams that will have the most freedom to deploy ambitious AI systems in 2027 and beyond are the ones building the trust infrastructure today: the telemetry pipelines, the audit trails, the accuracy feedback loops, and the compliance reports that demonstrate their systems are behaving exactly as declared. Observability is not the opposite of innovation. It is what makes sustained innovation possible in a regulated industry.

Start with Step 1. Define your schema. Everything else follows from that.