Telemetry Ingest
Summary
Zeeq ingests and processes OpenTelemetry (OTEL) data from agent harnesses which carry information about:
- Token usage
- Tool usage
- Prompts
- Pricing (Claude)
Flow
Telemetry volume can easily overwhelm the system, so Zeeq uses a multi-stage ingest pipeline to process telemetry data.
While Zeeq can directly ingest OTEL logs and spans, it is recommended to use the open source OTEL collector to pre-process and filter high volume telemetry data streams to retain only the relevant logs and spans for Zeeq.
OTEL Collector Configuration
Local Development
For local development, the OTEL collector should be configured to send telemetry data to both Zeeq and Aspire as this will allow troubleshooting of the telemetry flow with better visibility since agents can directly dump the telemetry from Aspire.
Aspire Only (For Full Telemetry Flow)
# Configuration to use for an OTEL collector forwarding to Aspire
# This is a basic setup that allows sending traces and logs to Aspire which makes
# it easier to understand the real shape of the telemetry flow.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
otlp/aspire:
endpoint: "${env:OTEL_EXPORTER_OTLP_ENDPOINT}"
headers:
x-otlp-api-key: "${env:ASPIRE_DASHBOARD_OTLP_API_KEY}"
tls:
insecure: true
processors:
batch:
send_batch_max_size: 100
send_batch_size: 10
timeout: 10s
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/aspire]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp/aspire]
Aspire and Zeeq (For Simulation of Upstream Filtering)
The collector can be configured to forward to multiple endpoints simultaneously which allows sending to Aspire as well as Zeeq. This is useful for simulating the upstream filtering that would occur in production.
This is the actual config used locally (host/otel-collector-config.filtered.yaml), bind-mounted into the otel-collector Aspire resource:
# Configured to forward to multiple targets with JWT auth and header forwarding
extensions:
# Validates the caller's Bearer JWT against Zeeq's OpenIddict issuer via
# standard OIDC discovery (/.well-known/openid-configuration + JWKS).
# This is a cheap outer gate that keeps unauthenticated noise off Zeeq.
oidc:
issuer_url: "${env:ZEEQ_ISSUER_URL}" # e.g. https://app.zeeq.ai/auth
# No dedicated telemetry OpenIddict resource/scope exists yet, so this
# points at the same resource already registered for MCP user tokens
# (AppSettings:Auth:Resource) - an existing Zeeq user token satisfies
# this check without new server-side provisioning.
audience: "${env:ZEEQ_TELEMETRY_AUDIENCE}"
username_claim: sub
# Forwards the original caller's Authorization header from the inbound request
# onto the exporter request, so Zeeq sees the end user's token, not a
# collector service identity. Requires include_metadata: true on the receiver
# and metadata_keys on the batch processor (see below).
headers_setter:
headers:
- action: upsert
key: authorization
from_context: authorization
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
auth:
authenticator: oidc
include_metadata: true # required for from_context in headers_setter
http:
endpoint: 0.0.0.0:4318
auth:
authenticator: oidc
include_metadata: true
exporters:
# Forward to Zeeq with per-user JWT pass-through via headers_setter.
# The Authorization header from the inbound request is forwarded; if no
# user token is present, the collector can fall back to a service-level
# client_credentials identity via additional_auth on headers_setter.
# otlphttp (not gRPC): Zeeq's Kestrel endpoint accepts both HTTP/1.1 and
# HTTP/2 locally, so a plain HTTP exporter/receiver pair is simplest.
# otlphttp auto-appends /v1/logs and /v1/traces to this base endpoint.
otlphttp/zeeq:
endpoint: "${env:ZEEQ_OTLP_HTTP_ENDPOINT}"
auth:
authenticator: headers_setter
sending_queue:
enabled: true
tls:
insecure: true # Local
# Fan-out to Aspire dashboard for full-telemetry visibility.
otlp/aspire:
endpoint: "${env:OTEL_EXPORTER_OTLP_ENDPOINT}"
headers:
x-otlp-api-key: "${env:ASPIRE_DASHBOARD_OTLP_API_KEY}"
tls:
insecure: true
processors:
# ---- LOGS: keep only Claude Code + Codex session events (drop-if-true) ----
filter/agent_session_logs:
error_mode: ignore
logs:
log_record:
- |
not (
resource.attributes["service.name"] == "claude-code" and
(log.body == "claude_code.user_prompt" or
log.body == "claude_code.tool_result" or
log.body == "claude_code.api_request" or
log.body == "claude_code.tool_decision")
) and not (
IsMatch(resource.attributes["service.name"], ".*codex.*") and
(log.attributes["prompt_length"] != nil or
(log.attributes["tool_name"] != nil and log.attributes["call_id"] != nil) or
log.attributes["event.kind"] == "response.completed" or
(log.attributes["mcp_servers"] != nil and log.attributes["approval_policy"] != nil))
)
# ---- TRACES: keep only Copilot Chat business spans (drop-if-true) ----
filter/agent_session_traces:
error_mode: ignore
traces:
span:
- |
not (
resource.attributes["service.name"] == "copilot-chat" and
span.attributes["copilot_chat.chat_session_id"] != nil and
(span.attributes["gen_ai.operation.name"] == "invoke_agent" or
span.attributes["gen_ai.operation.name"] == "chat" or
span.attributes["gen_ai.operation.name"] == "execute_tool")
)
batch:
send_batch_size: 128
send_batch_max_size: 512
timeout: 5s
# One batcher per distinct caller token: preserves per-user Authorization
# forwarding correctness. Without this, records from two users could land
# in one exported batch carrying a single Authorization header.
metadata_keys: [authorization]
metadata_cardinality_limit: 1000
service:
extensions: [oidc, headers_setter]
pipelines:
traces:
receivers: [otlp]
processors: [filter/agent_session_traces, batch]
exporters: [otlp/aspire, otlphttp/zeeq]
logs:
receivers: [otlp]
processors: [filter/agent_session_logs, batch]
exporters: [otlp/aspire, otlphttp/zeeq]
Authentication
The OTLP receivers validate incoming JWTs using the OIDC extension (configured with Zeeq's OpenIddict issuer). Only authenticated requests are accepted; invalid or missing bearer tokens are rejected at the collector edge.
Note: Aspire health checks (172.18.0.1) will log "Authentication failed: missing or empty header" warnings in the collector logs. These are expected and benign—health checks intentionally have no bearer token. Use the separate otlp/health receivers (ports 4319/4320) if cleaner logs are preferred, though they add complexity for what is fundamentally a deployment detail.
Local Networking Notes
The oidc extension fetches discovery/JWKS once at collector startup and fails the whole container (no retry) if that fetch errors, so a few local-dev-only details matter:
- Issuer trailing slash. OpenIddict's
SetIssuercanonicalizes to a root-path URI (http://zeeq-web.localhost:8095/), and theoidcextension does an exact string match betweenissuer_urland the discovery document's self-reportedissuerfield. Omitting the trailing slash fails collector startup with adid not match the issuer URL returned by providererror. - DNS resolution inside the container.
zeeq-web.localhostis a browser/OS-resolved loopback hostname, not a Docker network name, and does not resolve inside the collector container by default. The AppHost maps it via--add-host zeeq-web.localhost:host-gateway(Docker 20.10+), which routes back through the host machine to YARP's published port, preserving the hostname theoidcextension needs verbatim. - Startup ordering. Discovery routes through
zeeq-web.localhost -> yarp-reverse-proxy -> Vite dev server -> zeeq-server, not directly to the backend, so the collector resource waits on the full chain (WaitFor(backend/frontend/proxy)), and the frontend resource has an explicitWithHttpHealthCheck("/")so that wait blocks on Vite actually serving, not just the process having started.
Agent Harness Configuration
Verified live against Claude Code 2.1.197, Codex CLI 0.144.3, and VS Code Copilot Chat. Each harness needs its OTLP exporter pointed at the collector's HTTP endpoint (http://localhost:44318 locally), with the caller's Zeeq user token attached, matching what the oidc/filter blocks above expect. Claude Code and Copilot Chat use a base URL that gets /v1/logs//v1/traces auto-appended; Codex does not auto-append and needs the full path (see its section below).
Claude Code
Configured via env in .claude/settings.json (or settings.local.json):
{
"env": {
"CLAUDE_CODE_ENABLE_TELEMETRY": "1",
"OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
"OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:44318",
"OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer <zeeq-user-token>",
"OTEL_LOGS_EXPORTER": "otlp",
"OTEL_LOG_USER_PROMPTS": "1",
"OTEL_LOG_TOOL_DETAILS": "1"
}
}
OTEL_TRACES_EXPORTER/CLAUDE_CODE_ENHANCED_TELEMETRY_BETA here. Claude Code is logs-only: filter/agent_session_traces only has an allow condition for service.name == "copilot-chat", so any Claude Code spans would be emitted and immediately dropped at the collector. Cost (cost_usd) is a log-only field on Claude Code's wire anyway - there's nothing spans would add here that logs don't already carry.The filter above keeps claude_code.user_prompt, claude_code.tool_result, claude_code.api_request, and claude_code.tool_decision, identified by log.body (Claude Code emits the event type as the log body, not as an event.name attribute despite what Anthropic's docs claim).
Codex
Configured via [otel] in ~/.codex/config.toml:
[otel]
exporter = "otlp-http"
log_user_prompt = true
[otel.exporter."otlp-http"]
endpoint = "http://localhost:44318/v1/logs"
headers = { authorization = "Bearer <zeeq-user-token>" }
OTEL_EXPORTER_OTLP_ENDPOINT convention and auto-append /v1/logs//v1/traces to a base URL), Codex's endpoint is used verbatim with no path appended - confirmed directly from Codex's Rust source (codex-rs/otel/tests/suite/otlp_http_loopback.rs), where every OtlpHttp test endpoint is constructed with an explicit /v1/logs or /v1/traces suffix. Omitting it here means every export 404s silently at the collector.The filter above keeps Codex session events by attribute shape, not by an event.name attribute (also absent on Codex's wire despite docs claiming otherwise): a prompt_length attribute identifies a prompt, tool_name+call_id identifies a tool result, and event.kind == "response.completed" identifies a completion.
Copilot Chat
Configured via VS Code settings.json:
{
"github.copilot.chat.otel.enabled": true,
"github.copilot.chat.otel.exporterType": "otlp-http",
"github.copilot.chat.otel.otlpEndpoint": "http://localhost:44318",
"github.copilot.chat.otel.captureContent": true
}
Auth header via environment instead, since VS Code settings don't support one directly:
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <zeeq-user-token>"
captureContent is required for prompt text and tool arguments/results to appear at all - it is off by default, and without it only token counts and identity metadata are captured.Copilot Chat is traces-only for practical purposes: cost (copilot_chat.copilot_usage_nano_aiu), MCP tool identity, and prompt/response content are all trace-only fields, and Copilot's logs (when they arrive - often delayed relative to the matching trace spans) are a strict subset with none of that. The filter above keeps invoke_agent/chat/execute_tool spans, gated on copilot_chat.chat_session_id being present to exclude Next Edit Suggestions completions and other sessionless housekeeping spans.