Key Architecture Principles
Summary
Zeeq's system architecture is guided by a few key principles:
1. Prefer minimal infrastructure sprawl
Zeeq's core features can all run on a single Postgres database instance including:
- Postgres for the message queue (via Brighter)
- Postgres for the cache layer (via
HybridCacheconnecting to Postgres using unlogged tables) - Postgres for the main application data
- Postgres for the telemetry
- Postgres for lock coordination and leases
- Postgres for data protection backing
The design intentionally does not depend on Redis, but maintains the ability and option to add Redis via interfaces to replace the components.
The goal is to allow organizations to self-host with minimal additional infrastructure beyond deploying the container and connecting to an existing Postgres resource
Some optimizations are required for this to work:
| Approach | Reasoning |
|---|---|
| Unlogged tables | Use when the data is disposable and can be regenerated. This reduces WAL volume and write overhead. |
| Partitioned tables | Use when the data grows unbounded and read scenarios will be generally scoped to a single partition or the data can be truncated (e.g. telemetry) |
That said, Zeeq's production runtime has variants:
- Pub/Sub for message queue (instead of Postgres)
- Cloud KMS for key management (instead of .NET's default data protection provider)
2. Differentiate configuration and secrets
Secrets should be stored in Secret Manager or a secure vault while configuration should be placed into the appsettings.*.json files.
This makes configuration easy to manage and highly visible and easy to understand while true secrets are stored in secure vaults.
3. Partitioning of data by organization ID
The organization_id must be a part of all primary keys for organization scoped data.
This ensures:
- Data is always scoped to the organization and cannot be accidentally shared across organizations.
- Indexes are optimized for filtering by organization ID, which is the most common query pattern.
4. Use cursors for pagination
The system uses date-based cursors in on many of the tables like the PRs, the code review findings, the telemetry since this data is generally read in sets only by date.
A key note here is that because the partitions are date-based, many indexes include the date as part of the index.
The API layer uses an encoded token which makes querying by cursor a bit more ergonomic.