Design and Architecture

Telemetry Ingest

Ingest and processing of OpenTelemetry data from agent harnesses.

Summary

Zeeq ingests and processes OpenTelemetry (OTEL) data from agent harnesses which carry information about:

  1. Token usage
  2. Tool usage
  3. Prompts
  4. Pricing (Claude)

Flow

Telemetry volume can easily overwhelm the system, so Zeeq uses a multi-stage ingest pipeline to process telemetry data.

While Zeeq can directly ingest OTEL logs and spans, it is recommended to use the open source OTEL collector to pre-process and filter high volume telemetry data streams to retain only the relevant logs and spans for Zeeq.

It can be used suitably as a standalone collector in low-volume scenarios

OTEL Collector Configuration

Local Development

For local development, the OTEL collector should be configured to send telemetry data to both Zeeq and Aspire as this will allow troubleshooting of the telemetry flow with better visibility since agents can directly dump the telemetry from Aspire.

Aspire Only (For Full Telemetry Flow)

# Configuration to use for an OTEL collector forwarding to Aspire
# This is a basic setup that allows sending traces and logs to Aspire which makes
# it easier to understand the real shape of the telemetry flow.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp/aspire:
    endpoint: "${env:OTEL_EXPORTER_OTLP_ENDPOINT}"
    headers:
      x-otlp-api-key: "${env:ASPIRE_DASHBOARD_OTLP_API_KEY}"
    tls:
      insecure: true

processors:
  batch:
    send_batch_max_size: 100
    send_batch_size: 10
    timeout: 10s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/aspire]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/aspire]
Use this setup when testing canonical behavior since it can show all messages or add a filter to verify filter behavior.

Aspire and Zeeq (For Simulation of Upstream Filtering)

The collector can be configured to forward to multiple endpoints simultaneously which allows sending to Aspire as well as Zeeq. This is useful for simulating the upstream filtering that would occur in production.

This is the actual config used locally (host/otel-collector-config.filtered.yaml), bind-mounted into the otel-collector Aspire resource:

# Configured to forward to multiple targets with JWT auth and header forwarding

extensions:
  # Validates the caller's Bearer JWT against Zeeq's OpenIddict issuer via
  # standard OIDC discovery (/.well-known/openid-configuration + JWKS).
  # This is a cheap outer gate that keeps unauthenticated noise off Zeeq.
  oidc:
    issuer_url: "${env:ZEEQ_ISSUER_URL}"              # e.g. https://app.zeeq.ai/auth
    # No dedicated telemetry OpenIddict resource/scope exists yet, so this
    # points at the same resource already registered for MCP user tokens
    # (AppSettings:Auth:Resource) - an existing Zeeq user token satisfies
    # this check without new server-side provisioning.
    audience: "${env:ZEEQ_TELEMETRY_AUDIENCE}"
    username_claim: sub

  # Forwards the original caller's Authorization header from the inbound request
  # onto the exporter request, so Zeeq sees the end user's token, not a
  # collector service identity. Requires include_metadata: true on the receiver
  # and metadata_keys on the batch processor (see below).
  headers_setter:
    headers:
      - action: upsert
        key: authorization
        from_context: authorization

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        auth:
          authenticator: oidc
        include_metadata: true                           # required for from_context in headers_setter
      http:
        endpoint: 0.0.0.0:4318
        auth:
          authenticator: oidc
        include_metadata: true

exporters:
  # Forward to Zeeq with per-user JWT pass-through via headers_setter.
  # The Authorization header from the inbound request is forwarded; if no
  # user token is present, the collector can fall back to a service-level
  # client_credentials identity via additional_auth on headers_setter.
  # otlphttp (not gRPC): Zeeq's Kestrel endpoint accepts both HTTP/1.1 and
  # HTTP/2 locally, so a plain HTTP exporter/receiver pair is simplest.
  # otlphttp auto-appends /v1/logs and /v1/traces to this base endpoint.
  otlphttp/zeeq:
    endpoint: "${env:ZEEQ_OTLP_HTTP_ENDPOINT}"
    auth:
      authenticator: headers_setter
    sending_queue:
      enabled: true
    tls:
      insecure: true                                     # Local

  # Fan-out to Aspire dashboard for full-telemetry visibility.
  otlp/aspire:
    endpoint: "${env:OTEL_EXPORTER_OTLP_ENDPOINT}"
    headers:
      x-otlp-api-key: "${env:ASPIRE_DASHBOARD_OTLP_API_KEY}"
    tls:
      insecure: true

processors:
  # ---- LOGS: keep only Claude Code + Codex session events (drop-if-true) ----
  filter/agent_session_logs:
    error_mode: ignore
    logs:
      log_record:
        - |
          not (
            resource.attributes["service.name"] == "claude-code" and
            (log.body == "claude_code.user_prompt" or
             log.body == "claude_code.tool_result" or
             log.body == "claude_code.api_request" or
             log.body == "claude_code.tool_decision")
          ) and not (
            IsMatch(resource.attributes["service.name"], ".*codex.*") and
            (log.attributes["prompt_length"] != nil or
             (log.attributes["tool_name"] != nil and log.attributes["call_id"] != nil) or
             log.attributes["event.kind"] == "response.completed" or
             (log.attributes["mcp_servers"] != nil and log.attributes["approval_policy"] != nil))
          )

  # ---- TRACES: keep only Copilot Chat business spans (drop-if-true) ----
  filter/agent_session_traces:
    error_mode: ignore
    traces:
      span:
        - |
          not (
            resource.attributes["service.name"] == "copilot-chat" and
            span.attributes["copilot_chat.chat_session_id"] != nil and
            (span.attributes["gen_ai.operation.name"] == "invoke_agent" or
             span.attributes["gen_ai.operation.name"] == "chat" or
             span.attributes["gen_ai.operation.name"] == "execute_tool")
          )

  batch:
    send_batch_size: 128
    send_batch_max_size: 512
    timeout: 5s
    # One batcher per distinct caller token: preserves per-user Authorization
    # forwarding correctness. Without this, records from two users could land
    # in one exported batch carrying a single Authorization header.
    metadata_keys: [authorization]
    metadata_cardinality_limit: 1000

service:
  extensions: [oidc, headers_setter]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/agent_session_traces, batch]
      exporters: [otlp/aspire, otlphttp/zeeq]
    logs:
      receivers: [otlp]
      processors: [filter/agent_session_logs, batch]
      exporters: [otlp/aspire, otlphttp/zeeq]

Authentication

The OTLP receivers validate incoming JWTs using the OIDC extension (configured with Zeeq's OpenIddict issuer). Only authenticated requests are accepted; invalid or missing bearer tokens are rejected at the collector edge.

Note: Aspire health checks (172.18.0.1) will log "Authentication failed: missing or empty header" warnings in the collector logs. These are expected and benign—health checks intentionally have no bearer token. Use the separate otlp/health receivers (ports 4319/4320) if cleaner logs are preferred, though they add complexity for what is fundamentally a deployment detail.

Local Networking Notes

The oidc extension fetches discovery/JWKS once at collector startup and fails the whole container (no retry) if that fetch errors, so a few local-dev-only details matter:

  • Issuer trailing slash. OpenIddict's SetIssuer canonicalizes to a root-path URI (http://zeeq-web.localhost:8095/), and the oidc extension does an exact string match between issuer_url and the discovery document's self-reported issuer field. Omitting the trailing slash fails collector startup with a did not match the issuer URL returned by provider error.
  • DNS resolution inside the container. zeeq-web.localhost is a browser/OS-resolved loopback hostname, not a Docker network name, and does not resolve inside the collector container by default. The AppHost maps it via --add-host zeeq-web.localhost:host-gateway (Docker 20.10+), which routes back through the host machine to YARP's published port, preserving the hostname the oidc extension needs verbatim.
  • Startup ordering. Discovery routes through zeeq-web.localhost -> yarp-reverse-proxy -> Vite dev server -> zeeq-server, not directly to the backend, so the collector resource waits on the full chain (WaitFor(backend/frontend/proxy)), and the frontend resource has an explicit WithHttpHealthCheck("/") so that wait blocks on Vite actually serving, not just the process having started.

Agent Harness Configuration

Verified live against Claude Code 2.1.197, Codex CLI 0.144.3, and VS Code Copilot Chat. Each harness needs its OTLP exporter pointed at the collector's HTTP endpoint (http://localhost:44318 locally), with the caller's Zeeq user token attached, matching what the oidc/filter blocks above expect. Claude Code and Copilot Chat use a base URL that gets /v1/logs//v1/traces auto-appended; Codex does not auto-append and needs the full path (see its section below).

Claude Code

Configured via env in .claude/settings.json (or settings.local.json):

{
  "env": {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:44318",
    "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer <zeeq-user-token>",
    "OTEL_LOGS_EXPORTER": "otlp",
    "OTEL_LOG_USER_PROMPTS": "1",
    "OTEL_LOG_TOOL_DETAILS": "1"
  }
}
Deliberately no OTEL_TRACES_EXPORTER/CLAUDE_CODE_ENHANCED_TELEMETRY_BETA here. Claude Code is logs-only: filter/agent_session_traces only has an allow condition for service.name == "copilot-chat", so any Claude Code spans would be emitted and immediately dropped at the collector. Cost (cost_usd) is a log-only field on Claude Code's wire anyway - there's nothing spans would add here that logs don't already carry.

The filter above keeps claude_code.user_prompt, claude_code.tool_result, claude_code.api_request, and claude_code.tool_decision, identified by log.body (Claude Code emits the event type as the log body, not as an event.name attribute despite what Anthropic's docs claim).

Codex

Configured via [otel] in ~/.codex/config.toml:

[otel]
exporter = "otlp-http"
log_user_prompt = true

[otel.exporter."otlp-http"]
endpoint = "http://localhost:44318/v1/logs"
headers = { authorization = "Bearer <zeeq-user-token>" }
Unlike Claude Code and Copilot Chat (both of which use the generic OTEL_EXPORTER_OTLP_ENDPOINT convention and auto-append /v1/logs//v1/traces to a base URL), Codex's endpoint is used verbatim with no path appended - confirmed directly from Codex's Rust source (codex-rs/otel/tests/suite/otlp_http_loopback.rs), where every OtlpHttp test endpoint is constructed with an explicit /v1/logs or /v1/traces suffix. Omitting it here means every export 404s silently at the collector.

The filter above keeps Codex session events by attribute shape, not by an event.name attribute (also absent on Codex's wire despite docs claiming otherwise): a prompt_length attribute identifies a prompt, tool_name+call_id identifies a tool result, and event.kind == "response.completed" identifies a completion.

Copilot Chat

Configured via VS Code settings.json:

{
  "github.copilot.chat.otel.enabled": true,
  "github.copilot.chat.otel.exporterType": "otlp-http",
  "github.copilot.chat.otel.otlpEndpoint": "http://localhost:44318",
  "github.copilot.chat.otel.captureContent": true
}

Auth header via environment instead, since VS Code settings don't support one directly:

export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <zeeq-user-token>"
captureContent is required for prompt text and tool arguments/results to appear at all - it is off by default, and without it only token counts and identity metadata are captured.

Copilot Chat is traces-only for practical purposes: cost (copilot_chat.copilot_usage_nano_aiu), MCP tool identity, and prompt/response content are all trace-only fields, and Copilot's logs (when they arrive - often delayed relative to the matching trace spans) are a strict subset with none of that. The filter above keeps invoke_agent/chat/execute_tool spans, gated on copilot_chat.chat_session_id being present to exclude Next Edit Suggestions completions and other sessionless housekeeping spans.