Runtime Configuration

Google Cloud Configuration

How to configure Zeeq to run on Google Cloud Platform (GCP).

This work is early and incomplete

Services

KMS
SQL
Secret Manager
Pub/Sub
Cloud Storage
Cloud Run
OAuth

Service account roles

Role	Purpose
Cloud KMS Viewer	Allows the service account to read the KMS key for decrypting secrets.
Cloud SQL Client	Allows the service account to connect to Cloud SQL instances.
Secret Manager Secret Accessor	Allows the service account to access secrets stored in Secret Manager.
Pub/Sub Editor	Allows the service account to publish and subscribe to Pub/Sub topics.

Google OAuth

Required to allow login with Google

Settings	Value	Description
Application type	Web application	Select "Web application" as the application type.
Name	`zeeq-oauth-prod`	Name your application.
Authorized JavaScript origins	`https://app.zeeq.ai`	The web application domain for valid origins
Authorized redirect URIs	`https://app.zeeq.ai/auth/callback/google`	The post-authorization redirect target

Secrets required

Several secrets are required for OpenIddict to generate secure tokens.

# Generate the certificates used for signing the tokens
./build/certs/gen-openiddict-certs.sh

# Create the secrets in GCP Secret Manager (use the cert password)
ZEEQ_OPENIDDICT_ENCRYPTION_PASSWORD="..." ZEEQ_OPENIDDICT_SIGNING_PASSWORD="..." ./build/certs/upload-openiddict-secrets.sh

Additional secrets:

# The GitHub App Private Key (PEM); used to access GitHub API as app.
printf "SECRET_VALUE" | gcloud secrets create AppSettings__GitHub__PrivateKeyPem \
  --replication-policy="automatic" \
  --data-file=- \
  --project=$GCP_PROJECT_ID

# The GitHub App Webhook Secret (used to validate the incoming webhook requests)
printf "SECRET_VALUE" | gcloud secrets create AppSettings__GitHub__WebhookSecret \
  --replication-policy="automatic" \
  --data-file=- \
  --project=$GCP_PROJECT_ID

# Used when encrypting the contents of the review request link.
openssl rand -base64 48 | tr -d '\n' | \
  gcloud secrets create AppSettings__CodeReview__ReviewRequestLinkEncryptionKey \
    --replication-policy=automatic \
    --data-file=- \
    --project=$GCP_PROJECT_ID

# Used to sign Zeeq library export packages before they can be imported.
openssl rand -base64 48 | tr -d '\n' | \
  gcloud secrets create AppSettings__Documents__LibraryExportSigningKey \
    --replication-policy=automatic \
    --data-file=- \
    --project=$GCP_PROJECT_ID

# System-admin allow-list entry in provider:subject format. Create as empty and
# Then update after the first accounts are created.
printf "SECRET_VALUE" | gcloud secrets create AppSettings__Platform__SystemAdminSubjects__0 \
  --replication-policy="automatic" \
  --data-file=- \
  --project=$GCP_PROJECT_ID

# Default Fireworks API Key for the LLM model. The default model is Fireworks GLM 5.2
# (accounts/fireworks/models/glm-5p2). High and Max tiers inherit the Fast key when blank.
printf "SECRET_VALUE" | gcloud secrets create AppSettings__Llm__Models__Fast__ApiKey \
  --replication-policy="automatic" \
  --data-file=- \
  --project=$GCP_PROJECT_ID

# Fireworks API key for snippet embeddings (separate from the chat-completion key —
# different model, and rotated independently since only the sweep and the search API use it).
printf "SECRET_VALUE" | gcloud secrets create AppSettings__Llm__Embeddings__ApiKey \
  --replication-policy="automatic" \
  --data-file=- \
  --project=$GCP_PROJECT_ID

# Auth client secret for Google OAuth (login)
printf "SECRET_VALUE" | gcloud secrets create AppSettings__Auth__Providers__0__ClientSecret \
  --replication-policy="automatic" \
  --data-file=- \
  --project=$GCP_PROJECT_ID

# Auth client secret for GitHub OAuth (login)
printf "SECRET_VALUE" | gcloud secrets create AppSettings__Auth__Providers__1__ClientSecret \
  --replication-policy="automatic" \
  --data-file=- \
  --project=$GCP_PROJECT_ID

# Auth client secret for Microsoft (login)
printf "SECRET_VALUE" | gcloud secrets create AppSettings__Auth__Providers__2__ClientSecret \
  --replication-policy="automatic" \
  --data-file=- \
  --project=$GCP_PROJECT_ID

# The database connection string for the production database
# Host=/cloudsql/zeeq-ai-prod:us-central1:zeeq-pg-prod;Database=zeeq;Username=zeeq-app;Password=<password>;SSL Mode=Prefer;Pooling=true;Maximum Pool Size=10;Include Error Detail=true;Connection Idle Lifetime=60;Connection Lifetime=1800
printf "SECRET_VALUE" | gcloud secrets create AppSettings__Database__ConnectionString \
  --replication-policy="automatic" \
  --data-file=- \
  --project=$GCP_PROJECT_ID

# OTEL API key (e.g. Datadog, Honeycomb) injected as header. e.g.: `x-honeycomb-team=abc1234...`
printf "SECRET_VALUE" | gcloud secrets create zeeq-otel-api-key-header \
  --replication-policy="automatic" \
  --data-file=- \
  --project=$GCP_PROJECT_ID

# (Updating a secret)
printf "SECRET_VALUE" | gcloud secrets versions add AppSettings__Llm__Models__Fast__ApiKey \
  --data-file=- \
  --project=$GCP_PROJECT_ID

KMS setup

KMS with key rotation is used to manage encryption and decryption of user-entered API keys and secrets.

This ensures that secrets are securely encrypted at rest and can be rotated as needed.

Ensure that the service account is already provisioned.

# GCP_RUNTIME_SA=... ./build/init/gcp-kms.sh
PROJECT=$GCP_PROJECT_ID # Set in .config/mise.toml
RUNTIME_SA=$GCP_RUNTIME_SA # Set via env var
REGION=us-central1
KEYRING=zeeq-runtime
KEY=zeeq-llm-secrets

gcloud services enable cloudkms.googleapis.com \
  --project="$PROJECT"

gcloud kms keyrings create "$KEYRING" \
  --location="$REGION" \
  --project="$PROJECT"

gcloud kms keys create "$KEY" \
  --keyring="$KEYRING" \
  --location="$REGION" \
  --purpose=encryption \
  --rotation-period=90d \
  --next-rotation-time="$(date -u -v+90d '+%Y-%m-%dT%H:%M:%SZ')" \
  --project="$PROJECT"

gcloud kms keys add-iam-policy-binding "$KEY" \
  --keyring="$KEYRING" \
  --location="$REGION" \
  --member="serviceAccount:${RUNTIME_SA}" \
  --role="roles/cloudkms.cryptoKeyEncrypterDecrypter" \
  --project="$PROJECT"

Cloud SQL

Any CloudSQL Postgres configuration is suitable.

Two flags need to be enabled:

cloudsql.enable_pg_cron=on
cron.database_name=zeeq

These flags enable pg_partman cleanup and self provisioning of partitions to work.

# Enable pg_cron extension.
PROJECT=zeeq-ai-prod
INSTANCE=zeeq-pg-prod

gcloud sql instances patch "$INSTANCE" \
  --project="$PROJECT" \
  --database-flags=cloudsql.enable_pg_cron=on,cron.database_name=zeeq

# Access
brew install cloud-sql-proxy
gcloud auth application-default login
gcloud auth login
gcloud services enable sqladmin.googleapis.com

cloud-sql-proxy \
  --port 55432 \
  zeeq-ai-prod:us-central1:zeeq-pg-prod

The worker pool clones repositories for repository content ingest under AppSettings__Ingest__ContentRootPath - a Cloud Run ephemeral disk mounted at /mnt/ingest, rather than the OS temp directory local dev uses by default. LocalTempWorkspaceProvider needs no code change for this: it already reads its workspace root from that setting.

Cloud Run ephemeral disk currently requires at least 10Gi. Zeeq sets ZEEQ_INGEST_EPHEMERAL_DISK_SIZE=10Gi by default even though the ingest workspace usually needs much less.

build/ship-worker.sh mounts this disk at /mnt/ingest and points AppSettings__Ingest__ContentRootPath there:

./build/ship-worker.sh

--add-volume="name=ingest,type=ephemeral-disk,size=${ZEEQ_INGEST_EPHEMERAL_DISK_SIZE}" \
--add-volume-mount="volume=ingest,mount-path=${ZEEQ_INGEST_MOUNT_PATH}"

Only the worker pool mounts this disk - the web service never runs the ingest pipeline, so ship.sh is unaffected. Cloud Storage FUSE remains available for other runtime use cases, but git clone/checkout no longer writes through the GCS API-backed mount.

ZeeqIngestWorkspaceStartupCheck (a hosted service registered only by ZeeqWorkerHost, before message-consumer registration) runs a synthetic write/read/delete against the configured Ingest.ContentRootPath before the worker starts consuming any messages - a broken mount or wrong path fails loud at startup and crash-loops the Cloud Run worker instance.

Each ingest run still deletes its own workspace directory on dispose, regardless of storage medium (LocalIngestWorkspace's local-temp semantics are unchanged) - a repeat sync re-clones rather than reuses a prior pull. Ephemeral disk data is also discarded when the worker instance terminates.

Diagnostic Logging

# Tail the logs of the Cloud Run web service
gcloud run services logs tail zeeq-runtime \
  --project zeeq-ai-prod \
  --region us-central1

# Tail the logs of the Cloud Run worker
gcloud run worker-pools logs tail zeeq-worker \
  --project zeeq-ai-prod \
  --region us-central1

OTEL Collector VM

Agent telemetry is filtered by a standalone OpenTelemetry Collector before it is forwarded to Zeeq. The production deployment uses a single Compute Engine VM instead of Cloud Run so the collector can keep exporter queues, batch timers, and the Caddy TLS proxy running continuously.

The public endpoint is normal HTTPS:

https://otel.zeeq.ai

The collector still listens on the standard OTLP/HTTP port inside the private Docker network:

zeeq-otel-collector:4318

Caddy terminates public TLS on 80/443 and reverse proxies to the private collector listener. Do not expose 4318 publicly. OTLP/HTTP is the wire protocol; public clients should use HTTPS on 443.

Command order

# 1. Authenticate Docker to Artifact Registry.
gcloud auth configure-docker us-central1-docker.pkg.dev

# 2. Build and push the collector image.
mise exec -- ./build/pack-otel-collector.sh

# 3. Create or update the VM infrastructure.
mise exec -- ./build/init/gcp-otel-collector-vm.sh

# 4. Configure registrar DNS after the init script prints the static IP.
dig +short otel.zeeq.ai A

# 5. Redeploy only the collector container after later image changes.
mise exec -- ./build/ship-otel-collector.sh

The DNS step is manual at the registrar or DNS provider: create an A record from otel.zeeq.ai to the static IP printed by the init script. If the zone already uses CAA records, confirm letsencrypt.org is allowed before debugging Caddy certificate issuance.

Build and push the collector image

mise exec -- ./build/pack-otel-collector.sh

The image is pushed to Artifact Registry as:

us-central1-docker.pkg.dev/zeeq-ai-prod/zeeq/zeeq-otel-collector

The script always pushes an immutable sha-<shortsha> tag and either latest or the release tag supplied by ZEEQ_VERSION_TAG.

Create the VM

mise exec -- ./build/init/gcp-otel-collector-vm.sh

The script creates or reuses:

a VM service account;
the Artifact Registry Docker repository;
a regional static external IPv4 address;
a firewall rule allowing only tcp:80 and tcp:443;
a Container-Optimized OS VM running the collector and Caddy containers.

The VM uses a reserved static external IPv4 address. Do not use an ephemeral address for production because DNS for otel.zeeq.ai must remain stable across VM lifecycle operations.

Registrar DNS

After the init script prints the static IP, configure the registrar/DNS provider:

Type: A
Name/Host: otel
Value: <reserved static external IPv4 address>
TTL: 300 or provider default

DNS does not include ports. Do not configure 4318 at the registrar. Clients connect to https://otel.zeeq.ai; Caddy forwards internally to the collector on the private zeeq-otel Docker network.

If the zone already uses CAA records, allow Let's Encrypt:

Type: CAA
Name/Host: zeeq.ai
Value: 0 issue "letsencrypt.org"

Only add CAA when the zone intentionally manages certificate authorities. Existing CAA records that omit Let's Encrypt can block Caddy certificate issuance even when DNS and firewall rules are correct.

Verify DNS propagation:

dig +short otel.zeeq.ai A

Caddy certificate state

Caddy stores certificate/private-key/OCSP/ACME state under /data. On the VM, that is mounted from:

/var/lib/zeeq-caddy/data

Caddy autosaved config state is mounted from:

/var/lib/zeeq-caddy/config

These paths live on the VM persistent boot disk, so they survive VM reboots and container restarts. Do not treat them as caches, and do not move them to Cloud Storage FUSE: Cloud Storage FUSE is not POSIX-compliant and does not support file locking, which is a poor fit for TLS certificate state.

Redeploy the collector

mise exec -- ./build/ship-otel-collector.sh

This pulls the configured collector image and restarts only the collector container. It does not replace the VM, static IP, firewall rule, Caddy container, or Caddy certificate state.

Troubleshooting

# 1. Confirm DNS points at the reserved static IP printed by the init script.
dig +short otel.zeeq.ai A

# 2. Confirm the VM is running, has the reserved static IP, and has the firewall target tag.
gcloud compute instances describe zeeq-otel-collector \
  --project zeeq-ai-prod \
  --zone us-central1-a \
  --format='table(name,status,tags.items,networkInterfaces[0].accessConfigs[0].natIP)'

# 3. Confirm the firewall rule allows only public HTTP/HTTPS to that target tag.
gcloud compute firewall-rules describe allow-zeeq-otel-collector-https \
  --project zeeq-ai-prod \
  --format='yaml(network,direction,allowed,sourceRanges,targetTags,disabled)'

# 4. SSH into the VM for container and startup-script diagnostics.
gcloud compute ssh zeeq-otel-collector \
  --project zeeq-ai-prod \
  --zone us-central1-a

# 5. On the VM, inspect the COS startup-script service. This is where first boot
# container startup failures appear.
sudo journalctl -u google-startup-scripts.service --no-pager -n 300
sudo journalctl -u google-startup-scripts.service --no-pager | grep -i -E 'zeeq|docker|startup|error|failed'

# 6. On the VM, rerun the metadata startup script after fixing metadata or IAM.
sudo google_metadata_script_runner startup

# 7. On the VM, confirm containers exist and check their logs.
sudo docker ps -a
sudo docker logs --tail 100 zeeq-otel-caddy
sudo docker logs --tail 100 zeeq-otel-collector

# 8. On the VM, confirm expected listeners. Caddy should own 80/443 through
# published Docker ports. The collector should not publish 4318 to the VM.
sudo ss -ltnp | grep -E ':80|:443|:4318'

# 8a. On Container-Optimized OS, Caddy should use bridge networking with
# explicit 80/443 port publishing. If Caddy is reachable on 127.0.0.1 but
# external curl hangs, restart Caddy with published ports rather than host
# networking and update the init script before the next VM metadata rerun.
docker inspect zeeq-otel-caddy --format '{{.HostConfig.NetworkMode}}'
docker inspect zeeq-otel-collector --format '{{.HostConfig.NetworkMode}}'

# 9. If startup logs show "/root/.docker: read-only file system", update to the
# script version that uses DOCKER_CONFIG=/var/lib/zeeq-docker-config, rerun init
# locally, then rerun the metadata startup script.
exit
mise exec -- ./build/init/gcp-otel-collector-vm.sh

# 10. After DNS and Caddy are healthy, TLS should connect. An unauthenticated
# request should be rejected; the useful signal here is a valid TLS handshake.
curl -v https://otel.zeeq.ai/v1/logs

# 11. If the zone uses CAA records, confirm letsencrypt.org is allowed before
# debugging Caddy certificate issuance.

# 12. If telemetry exports get 401/403, confirm each harness sends
# Authorization=Bearer <zeeq-user-token> and the collector forwards it.

# 13. If the collector fails with an issuer mismatch, confirm the production
# issuer includes the trailing slash.
echo 'ZEEQ_ISSUER_URL=https://app.zeeq.ai/'

# 14. If ingest backlog grows, check VM collector logs, Zeeq Cloud Run logs, and
# telemetry_raw_requests in Postgres.

Cloud Run Runtime Roles

Zeeq uses the same container image for the HTTP web runtime and the background worker runtime. The process behavior is selected with environment variables. ZEEQ_MESSAGING_ROLE is required so a production web service cannot accidentally start consumers when the role is omitted. Production Cloud Run deployments also run with the GCP Pub/Sub transport, so the Pub/Sub project id must be available before startup topology reconciliation runs. The deployment scripts pass the explicit ZeeqMessaging__GcpPubSub__ProjectId setting from GCP_PROJECT_ID and also set GCP_PROJECT_ID for fallback resolution.

Setting	Values	Purpose
`ZEEQ_RUN_MODE`	`web`, `worker`	Selects the host shape. `web` starts ASP.NET Core; `worker` starts the generic-host worker without HTTP middleware.
`ZEEQ_MESSAGING_ROLE`	`producer`, `consumer`, `producer-consumer`	Selects which Brighter messaging services the process registers.
`ZeeqMessaging__Provider`	`GcpPubSub`	Selects the production messaging transport. CI forces Postgres regardless of this setting.
`ZeeqMessaging__GcpPubSub__ProjectId`	GCP project id	Required when the provider is `GcpPubSub`; used to create and validate topics and subscriptions.
`GCP_PROJECT_ID`	GCP project id	Passed by the deployment scripts and used as a runtime fallback for Pub/Sub project resolution.

Production uses two Cloud Run runtimes:

Runtime	Required env vars	Behavior
Web service	`ZEEQ_RUN_MODE=web`, `ZEEQ_MESSAGING_ROLE=producer`, `ZeeqMessaging__Provider=GcpPubSub`, `ZeeqMessaging__GcpPubSub__ProjectId=$GCP_PROJECT_ID`	Accepts HTTP traffic and publishes messages only.
Worker pool	`ZEEQ_RUN_MODE=worker`, `ZEEQ_MESSAGING_ROLE=producer-consumer`, `ZeeqMessaging__Provider=GcpPubSub`, `ZeeqMessaging__GcpPubSub__ProjectId=$GCP_PROJECT_ID`	Runs message consumers and can publish follow-up messages from handlers.

Only the web service mounts the OpenIddict certificate .pfx secrets as files. Cloud Run worker pools do not support Secret Manager volume mounts, and worker mode exits before ASP.NET Core/OpenIddict setup.

The local Aspire app uses ZEEQ_MESSAGING_ROLE=producer-consumer for the single-process development shape. When ZEEQ_ASPIRE_MODE=split, the web resource switches to producer and the worker resource uses producer-consumer.

GCP_PROJECT_ID defaults to zeeq-ai-prod in build/ship.sh and build/ship-worker.sh when it is not set by the caller. Set GCP_PROJECT_ID explicitly before running build/send.sh to deploy the same image to another project.

Deployment scripts:

Script	Runtime
`build/ship.sh`	Deploys the production web service as `producer`.
`build/ship-worker.sh`	Deploys the production worker pool as `producer-consumer`.
`build/send.sh`	Builds/pushes the image and deploys both runtimes.

Edit this pageorReport an issue

LLM Settings Configuration

How organization owners configure shared LLM tiers and tenant-managed API keys.

GitHub Configuration

Configure Zeeq to with permissions to interact with applications in GitHub.