Polytomic can expose its own operational state to your Prometheus, on a private endpoint inside your cluster. It answers the questions such as: is replication still running, did the last sync execution succeed, is a connection's credential expired, and is anything silently stuck.

Nothing is exposed until you turn it on, and no data leaves your cluster.

Enabling the endpoint

This requires version 1.9.0 or later of the Polytomic Helm chart, and Polytomic release rel2026.09.10 or later. Set one value and upgrade:

polytomic:
  prometheus:
    enabled: true

The endpoint serves /metrics and nothing else, on port 9090 by default. Change it with polytomic.prometheus.port if 9090 is already in use.

Where the endpoint is

The endpoint is not on your Polytomic URL. Your ingress and DNS route to Polytomic's web pods, and the metrics are served by a different pod, the scheduler. Instead, the chart creates a Service of its own in front of the scheduler, named with your release's full name plus a -metrics suffix. Kubernetes gives every Service a DNS name inside the cluster, so the scrape address is:

http://<service name>.<namespace>.svc:9090/metrics

For a release named polytomic installed in the polytomic namespace, that is polytomic-metrics.polytomic.svc:9090. The output of helm install and helm upgrade prints the exact address for your installation, and you can list the Service with:

kubectl get svc -n <namespace> -l app.kubernetes.io/component=metrics

The Service is ClusterIP-only and is never routed through your ingress, because the endpoint reports on every connection and sync in the deployment. Its address resolves only inside your cluster.

Checking the endpoint

Before connecting Prometheus, confirm the endpoint responds:

kubectl port-forward -n <namespace> svc/<service name> 9090:9090
curl -s http://127.0.0.1:9090/metrics | grep query_success

polytomic_ingestion_metrics_query_success 1 means Polytomic is serving its metrics. If the port-forward reports connection refused, your Polytomic release predates the endpoint; upgrade to rel2026.09.10 or later.

Choosing how Prometheus finds the endpoint

Use one of the three options below. Each one is a separate scrape job, so enabling two makes Prometheus scrape the endpoint twice.

Scrape the Service address

Add the Service address to a static scrape config. You don't need to set anything else in the chart.

scrape_configs:
  - job_name: polytomic
    scrape_interval: 60s
    static_configs:
      - targets: ["polytomic-metrics.polytomic.svc:9090"]

Pod annotations

If your Prometheus discovers targets by prometheus.io/scrape annotation, turn the annotations on:

polytomic:
  prometheus:
    enabled: true
    scrapeAnnotations: true

ServiceMonitor

If you run the Prometheus Operator, create a ServiceMonitor:

polytomic:
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
      # Must match your Prometheus resource's serviceMonitorSelector.
      labels:
        release: kube-prometheus-stack

This requires the Prometheus Operator's monitoring.coreos.com CRDs, which the chart does not install. serviceMonitor also accepts interval (default 60s), scrapeTimeout (default 10s), relabelings, and metricRelabelings.

With pod annotations or a ServiceMonitor, Prometheus looks up the scheduler pod through the Kubernetes API and scrapes its IP directly, so it doesn't use a DNS name at all.

Prometheus outside your cluster

A Prometheus server or hosted monitoring service running outside your cluster can't reach the endpoint, because its address exists only inside the cluster. Run an agent inside the cluster that scrapes the endpoint using one of the options above and forwards the samples, such as Prometheus in agent mode with remote_write, Grafana Alloy, or the Datadog Agent's OpenMetrics check.

Don't expose the endpoint through a load balancer or your ingress instead. It has no authentication of its own, and it reports on every connection and sync in the deployment.

If you enable it and see no data

Check the endpoint directly first. Follow Checking the endpoint. If it responds there, Polytomic is serving metrics and the problem is in how Prometheus finds them.

Don't point Prometheus at your Polytomic URL. That URL reaches the web pods, which don't serve these metrics. Use the -metrics Service address or one of the discovery options above.

Check the ServiceMonitor's labels first. The Prometheus Operator only scrapes ServiceMonitors that match its Prometheus resource's serviceMonitorSelector, which in a kube-prometheus-stack install is usually release: <stack release name>. A ServiceMonitor without matching labels is created without error and then never scraped, so the target never appears at all. This looks identical to the endpoint never having been enabled. Print the selector with kubectl get prometheus -A -o jsonpath='{..serviceMonitorSelector}'.

Pod-annotation discovery needs scrapeAnnotations: true. Setting enabled: true alone doesn't add any annotations.

Restricting access

The chart's optional NetworkPolicy (networkPolicy.enabled) does not cover the metrics endpoint: it doesn't block a Prometheus in another namespace, and it doesn't limit which pods can reach the port. To restrict access, add a NetworkPolicy of your own that selects the pods behind the -metrics Service. To print the labels to select on, run kubectl get svc <service name> -n <namespace> -o jsonpath='{.spec.selector}'.

What is exposed

The sync-level metrics — the sync inventory, executions, and schedules — cover bulk syncs only. Model syncs are not reported yet. Connection health, schema discovery, and exporter health are reported per connection or for the whole deployment, so they apply whichever kind of sync uses a connection.

Exporter health

If Polytomic cannot read its own state, the endpoint still returns HTTP 200, sets query_success to 0, and emits no object series — never zero-valued gauges, because a zero would read as "healthy and caught up."

Always alert on query_success. Otherwise a failure that removes every object series would silently silence all your other alerts.

build_info is for correlation rather than alerting. Overlaying deploy boundaries on a lag or backlog chart answers "did this start when we upgraded?", and changes() on it can suppress other alerts during a deploy window.

Inventory and metadata

All human-readable and changeable metadata lives only on these two series, so renaming a sync or a connection does not break the continuity of the operational time series. See Adding names to alerts for joining them back in.

Bulk sync executions

An execution is one run of a bulk sync writing to your destination — the layer whose failure you normally notice first. Label: sync_id.

What each status means

errors is a partial failure: data is still moving, but part of your pipeline is broken. That usually warrants a different response from failed — a ticket rather than a page — and a partial failure reported as succeeded would be invisible while a table quietly stopped updating. If you want one alert covering both, match status=~"failed|errors".

failed can mean ingestion fell behind

For a sync that replicates changes from its source, Polytomic checks how far behind ingestion is when each execution finishes. If it's past the sync's threshold, the execution is marked failed even if every schema completed, and its status message in the Polytomic UI says how far behind it was.

• Sources that report event timestamps, such as change-data-capture databases, are measured by lag. The default threshold is 12 hours.
• Queue-based sources such as Amazon S3 are measured by backlog, in messages. They're checked only when the sync sets its own threshold.

A threshold set on the sync replaces the default.

execution_active describes the sync

A bulk sync can run more than one execution at a time. polytomic_sync_execution_active is 1 while any of the sync's executions is running, which isn't necessarily the one the last_* series describe: a short execution can finish while an older one is still running.

last_attempt_count above 1

The execution began, stopped before finishing, and started over — most often because the process was killed, typically by a memory limit. That pattern is otherwise close to invisible: it does not necessarily produce a failed execution, and container restart counts are not reliable in every cluster.

The reported execution matches the sync page

For a sync that is backfilling, the reported execution is the primary one, not one of its continuations — the same rule the execution list in the Polytomic UI uses, so the status on the endpoint and the status in the UI for the same sync agree.

Test runs are the exception. The UI lists them, but these metrics skip them: a test reads only a sample and doesn't advance replication, so it says nothing about whether scheduled syncing is working.

The schedule interval is absent for some syncs

polytomic_sync_schedule_interval_seconds reports a cadence only where one is well defined: hourly, daily, weekly, and continuous schedules. Manual, run-after, dbt Cloud, custom schedule, and cron expression syncs have no interval and produce no series, so the generic "hasn't run lately" alert below produces no result for them.

For manual, run-after, and dbt Cloud syncs, that is correct rather than a gap: a manual sync that has not run is not late. Custom and cron schedules are left out because the gap between their runs isn't necessarily constant — a schedule that runs on the 1st and 2nd of each month waits a day and then about a month — so no single interval describes them. Alert on those syncs with a threshold of your own on polytomic_sync_execution_last_completion_timestamp_seconds.

A sync with several schedules reports the smallest interval among them.

polytomic_sync_schedule_effective_timestamp_seconds is reported alongside the interval. It's when the sync could first have been expected to run on that schedule: the later of when the schedule was created and when the sync was last activated. Creating or activating a sync doesn't run it — the first execution comes at the next scheduled time — so the example alert below uses it to give a new or reactivated sync its grace period, and to catch a sync that has never completed.

Connection health

Credential expiry and revoked permissions are among the most common causes of a silently broken pipeline, and among the few whose fix is entirely on your side. This covers every connection, including ones no sync currently uses — which are exactly the ones whose credential expiry goes unnoticed.

unknown means the connection has not been health-checked yet.

There is deliberately no "time since last health check" series. Polytomic records health transitions rather than every check, so the age of that timestamp would grow without bound on a perfectly healthy connection and any staleness alert built on it would be wrong. The state itself is maintained on the connection's health-check interval and is trustworthy.

Schema discovery

This catches a specific and genuinely silent failure: schema discovery can break while existing syncs keep running normally, because a sync only refreshes the schemas it already uses. New tables stop appearing and nothing else looks wrong. The connection's own health can read healthy throughout.

An alert on the age of the last successful refresh surfaces it in hours rather than weeks.

After you upgrade to this release

The success timestamp is new. On upgrade, Polytomic fills it in for every connection whose schema list is currently cached. A connection whose most recent refresh had already failed before the upgrade has no successful refresh to record, so it emits no success series until discovery works again.

Until then, ..._last_attempt_timestamp_seconds and ..._errors cover it: a connection with a rising error count and no success series is one whose discovery was already broken when you upgraded.

Ingestion from queue-based sources

Queue-driven sources such as Amazon S3 consume a notification queue. For these, being behind is a count, and the queue belongs to the connection rather than to any one sync. Label: connection_id.

A connection reports these only while at least one active bulk sync still consumes it. Deactivate every sync using a queue connection and its series disappear, which is correct: nobody is draining that queue on purpose.

Sources covered

Everything else on this page — bulk sync executions and schedules, connection health, schema discovery, and exporter health — covers all of your bulk syncs and connections, regardless of source type.

The timestamp-based family, which would report how far behind a CDC source is as a duration, is not yet available. Each reader implementation has to be validated before we publish its series, because the family is only useful if you can tell a quiet source from a stuck one, and that depends on behavior we verify per implementation rather than assume. We would rather publish nothing than publish a signal whose alerting semantics we have not established.

How to read these signals

Backlog is a message count, not a duration

polytomic_connection_ingestion_backlog_messages counts outstanding queue messages, including those waiting, in flight, and delayed. It cannot be converted into a duration: the queue does not report the age of its oldest message, so there is no "minutes behind" figure for a queue-based source. It also counts messages, not rows — one message can describe more than one object.

Backlog depth updates once per ingestion run

Depth is measured when a run finishes, not continuously; between runs the gauge holds its last value. Use for: durations and rate windows spanning several runs, and note that a burst arriving and fully draining inside one run never appears. The gauge answers "what was left over", which is the right question for "are we falling behind". A failed run still reports the backlog it leaves, so a failing ingestion does not appear caught up.

Queue depth is not always readable

Reading queue depth requires a permission some credentials do not grant. When Polytomic cannot read it, the depth gauge is omitted and backlog_known reports 0 rather than a fabricated zero. Alert on backlog_known == 0 so you know when you are in this mode — otherwise a blind queue is indistinguishable from a drained one.

Liveness is not the same as success

Status-update timestamps advance whenever Polytomic records progress, including periodic checkpoints while a healthy source sits idle — which is what makes them the right "is it still running" signal. They also update when an attempt starts, so a source failing and retrying in a loop can keep them fresh. That is what the two error flags are for.

Zero records written is often normal

A sync execution that writes zero records usually means there was nothing new to write, not that something is broken. This is especially common when a sync polls more frequently than its source ingestion completes. Alert on execution failure, on attempt counts above one, and on schedule staleness — not on a zero record count by itself.

Series appear and disappear as objects change

Deactivating or deleting a sync removes its series on the next successful scrape; reactivating restores them. Prefer alerting on exporter health over absent() checks on individual objects, which would fire every time someone intentionally turns something off.

Example alerts

Thresholds are starting points. Tune backlog thresholds to your queue's normal steady state.

The monitoring itself

- alert: PolytomicMetricsQueryFailing
  expr: polytomic_ingestion_metrics_query_success == 0
  for: 5m
  annotations:
    summary: "Polytomic cannot read its own state; other alerts are unreliable"

- alert: PolytomicMetricsMissing
  expr: absent_over_time(polytomic_ingestion_metrics_query_success[10m])
  for: 5m
  annotations:
    summary: "Polytomic metrics have not been scraped in 10 minutes"

Pair these with an alert on Prometheus's own up series for the scrape job, such as up{job="polytomic"} == 0. The job name depends on how Prometheus finds the endpoint: it's the job_name you chose in a static scrape config, the Service name (polytomic-metrics for a release named polytomic) with a ServiceMonitor, and your annotation scrape job's name with pod annotations.

Sync executions are failing

- alert: PolytomicSyncExecutionFailed
  expr: polytomic_sync_execution_last_status{status="failed"} == 1
  for: 10m

- alert: PolytomicSyncExecutionPartiallyFailed
  expr: polytomic_sync_execution_last_status{status="errors"} == 1
  for: 15m
  annotations:
    summary: "Some schemas failed while others succeeded; part of this sync is not updating"

- alert: PolytomicSyncExecutionRestarting
  expr: polytomic_sync_execution_last_attempt_count > 1
  for: 10m
  annotations:
    summary: "Sync execution restarted mid-run; check for memory limits or evictions"

A sync stopped running, or is running too long

# Generic: no completed execution in more than 3x the sync's own schedule.
# Counts from the later of the last completion and the schedule's effective
# time, so a sync that has never completed still alerts, and one that was just
# reactivated gets its grace period instead of alerting on a stale completion.
# Skips syncs that are running right now; the next rule covers a run that goes
# on too long. Produces no result for syncs without an interval -- that is
# intended.
- alert: PolytomicSyncNotRunning
  expr: |
    (
      time()
        - max by (sync_id) ({__name__=~"polytomic_sync_execution_last_completion_timestamp_seconds|polytomic_sync_schedule_effective_timestamp_seconds"})
    )
      > on (sync_id) (3 * polytomic_sync_schedule_interval_seconds)
    unless on (sync_id) (polytomic_sync_execution_active == 1)
  for: 30m

# Measures from the start of the most recent execution. If executions overlap,
# an older one still running behind a newer one isn't covered.
- alert: PolytomicSyncExecutionRunningLong
  expr: |
    polytomic_sync_execution_active == 1
      and
    (time() - polytomic_sync_execution_last_start_timestamp_seconds) > 14400
  for: 15m

A connection is unhealthy, or schema discovery is failing

- alert: PolytomicConnectionUnhealthy
  expr: polytomic_connection_health{status="error"} == 1
  for: 15m

- alert: PolytomicSchemaRefreshStale
  expr: |
    (time() - polytomic_connection_schema_refresh_last_success_timestamp_seconds)
      > 172800
  for: 1h
  annotations:
    summary: "Schema discovery has not succeeded in 48h; new tables will not appear"

Backlog is deep and not draining

min_over_time requires depth to have stayed above the threshold for the whole
window, so a burst that drains does not fire.

- alert: PolytomicQueueBacklogNotDraining
  expr: min_over_time(polytomic_connection_ingestion_backlog_messages[2h]) > 10000
  for: 30m

- alert: PolytomicQueueDepthUnavailable
  expr: polytomic_connection_ingestion_backlog_known == 0
  for: 1h

- alert: PolytomicQueueIngestionStalled
  expr: (time() - polytomic_connection_ingestion_status_updated_timestamp_seconds) > 1800
  for: 15m

Queue sources report on run boundaries, so their normal gap between updates is much larger than you might expect. Set that threshold from your own observed steady state.

Adding names to alerts

Operational series carry IDs only, so alert text would otherwise read as UUIDs. Multiply by the matching info series with group_left to pull in readable labels. Note that and on (...) deliberately drops right-hand labels, so apply the join at the end of the expression:

expr: |
  (
    polytomic_sync_execution_last_status{status="failed"} == 1
  )
  * on (sync_id) group_left(sync_name, organization_id) polytomic_sync_info
annotations:
  summary: "Polytomic sync {{ $labels.sync_name }} failed"

The same pattern works for connection-scoped series with polytomic_connection_info on connection_id.

Finding which syncs a backlog affects

Because queue depth belongs to the connection, one deep backlog can affect several syncs. Rewrite the sync inventory's connection label to match:

label_replace(
  polytomic_sync_info, "connection_id", "$1", "source_connection_id", "(.+)"
)
  * on (connection_id) group_left()
    polytomic_connection_ingestion_backlog_messages

The result is backlog depth, one series per affected sync, labeled with sync_name. Use it for dashboards and ticket routing rather than paging — page once on the connection, not once per sync.

Comparing notes with Polytomic support

Polytomic reports the same facts to its own monitoring, from the same queries, on an independent schedule. That means a value you see and a value support sees are both correct but may have been sampled at different moments.

polytomic_ingestion_metrics_snapshot_timestamp_seconds is the time at which the state you are looking at was observed. When comparing a number with support, quote that timestamp alongside it.

Scope

Metrics are reported per bulk sync and per connection, never per schema. A single connection can carry hundreds of schemas, and per-schema series would multiply the export by that factor.

The endpoint exposes operational state only: identifiers, timestamps, counts,
and statuses. It contains none of your synced data.